software – Alice, Eve and Bob

The 9 development stages for software (and kids)

We can draw parallels in the stages through which software projects and children tend to progress.

This week, one of my kids turned 18, and is therefore an adult – at least in the eyes of law in the UK, where we live. This is scary. For me, and probably for the rest of the UK.

It also got me thinking about how there are similarities between the development lifecycle for software and kids, and that we can probably draw some parallels in the stages through which they tend to progress. Here are the ones that occurred to me.

1. Creation

Creating a new software project is very easy, and a relatively quick process, though sometimes you have a huge number of false starts and things don’t go as planned. The same, it turns out, applies when creating children. Another similarity is that many software projects are created by people who don’t really know what they’re doing, and shouldn’t be allowed anywhere near the process at all. Equally useless at the process are people who have only a theoretical understanding of how it should work, having studied it at school, college or university, but who feel that they are perfectly qualified.

The people who are best qualified – those who have done it before – are either rather blasé about it and go about creating new ones all over the place, or are so damaged by it all that they swear they’ll never do it again. Beware: there are also numerous incidents of people starting software processes at very young ages, or when they had no intention of doing so.

2. Naming

Naming a software project – or a baby – is an important step, as it’s notoriously difficult to change once you’ve assigned a name. While there’s always a temptation to create a “clever” or “funny” name for your project, or come up with an alternative spelling of a well-known word, you creation will suffer if you allow yourself to be so tempted. Use of non-ASCII characters will either be considered silly, or, for non-Anglophone names, lead to complications when your project (or child) is exposed to other cultures.

3. Ownership

When you create a software project, you need to be careful that your employer (or educational establishment) doesn’t lay claim to it. This is rarely a problem for human progeny (if it is, you really need to check your employment contract), but issues can arise when two contributors are involved and wish to go their separate ways, or if other contributors – e.g. mothers-in-law – feel that their input should have more recognition, and expect more of their commits to be merged into main.

4. Language choice

The choice of language may be constrained by the main contributors’ expertise, but support for multiple languages can be very beneficial to a project or child. Be aware that confusion can occur, particularly at early stages, and it is generally worthwhile trying to avoid contributors attempting to use languages in which they are not fluent.

5. Documentation

While it is always worthwhile being aware of the available documentation, there are many self-proclaimed “experts” out there, and much conflicting advice. Some classic texts, e.g. The C Programming Language by Brian Kernighan and Dennis Ritchie or Baby and Child Care by Dr Benjamin Spock, are generally considered outdated in some circles, while yet others may lead to theological arguments. Some older non-core contributors (see, for example, “mothers-in-law”, above), may have particular attachments to approaches which are not considered “safe” in modern software or child development.

6. Maintenance

While the initial creation step is generally considered the most enjoyable in both software and child development processes, the vast majority of the development lifecycle revolves around maintenance. Keeping your project or child secure, resilient and operational or enabling them to scale outside the confines of the originally expected environment, where they come into contact with other projects, can quickly become a full-time job. Many contributors, at this point, will consider outside help to manage these complexities.

7. Scope creep

Software projects don’t always go in the direction you intend (or would like), discovering a mind of their own as they come into contact with external forces and interacting in contexts which are not considered appropriate by the original creators. Once a project reaches this stage, however, there is little that can be done, and community popularity – considered by most contributors as a positive attribute at earlier stages of lifecycle – can lead to some unexpected and possibly negative impacts on the focus of the project as competing interests vie to influence the project’s direction. Careful management of resources (see below) is the traditionally approach to dealing with this issue, but can backfire (withdrawal of privileges can have unexpected side effects in both software and human contexts).

8. Resource management

Any software project always expands to available resources. The same goes for children. In both cases, in fact, there will always appear to be insufficient resources to meet the “needs” of the project/child. Be strong. Don’t give in. Consider your other projects and how they could flourish if provided with sufficient resources. Not to mention your relationships with other contributors. And your own health/sanity.

9. Hand-over

At some point, it becomes time to hand over your project. Whether this is to new lead maintainer (or multiple maintainers – we should be open-minded), to an academic, government or commercial institution, letting go can be difficult. But you have to let go. Software projects – and children – can rarely grow and fulfil their potential under the control of the initial creators. When you do manage to let go, it can be a liberating experience for you and your creation. Just don’t expect that you’ll be entirely free of them: for some reason, as the initial creator, you may well be expected to arrange continued resources well past the time you were expecting. Be generous, and enjoy the nostalgia, but you’re not in charge, so don’t expect the resources to be applied as you might prefer.

Conclusion

I’m aware that there are times when children – and even software projects – can actually cause pain and hurt, and I don’t want to minimise the negative impact that the inability to have children, their illness, injury or loss can have on individuals and families. Please accept this light-hearted offering in the spirit it is meant, and if you are negatively affected by this article, please consider accessing help and external support.

Saving one life

Scratching the surface of the technologies which led to the saving of a life

When a loved one calls you from the bathroom at 3.30 in the morning, and you find them collapsed, unconscious on the floor, what does technology do for you? I’ve had the opportunity to consider this over the past few days after a family member was rushed to hospital for an emergency operation which, I’m very pleased to say, seems to have been completely successful. Without it, or if it had failed (the success rate is around 50%), they would, quite simply, be dead now.

We are eternally grateful to all those directly involved in my family member’s care, and to the NHS, which means that there are no bills to pay, just continued National Insurance taken as tax from our monthly pay packets, and which we begrudge not one jot. But I thought it might be worth spending a few minutes just scratching the surface of the sets of technologies which led to the saving of a life, from the obvious to the less obvious. I have missed out many: our lives are so complex and interconnected that it is impossible to list everything, and it is only when they are missing that we realise how it all fits together. But I want to say a huge – a HUGE – thank you to anyone who has ever been involved in any of the systems or technologies, and to ask you to remind yourself that even if you are seldom thanked, your work saves lives every day.

The obvious

The combined ECG and blood pressure unit attached to the patient which allows the ambulance crew to react quickly enough to save the patient’s life
The satellite navigation systems which guided the crew to the patient’s door
The landline which allowed the call to the emergency systems
The triage and dispatch system which prioritised the sending of the crew
The mobile phone system which allowed a remote member of the family to talk to the crew before they transported the patient

The visible (and audible)

The anaesthesiology and monitoring equipment which kept the patient alive during the operation
The various scanning equipment at the hospital which allowed a diagnosis to be reached in time
The sirens and flashing lights on the ambulances
The technology behind the training (increasingly delivered at least partly online) for all of those involved in the patient’s care

The invisible

The drugs and medicines used in the patient’s care
Equipment: batteries for ambulances, scalpels for operating theatres, paper for charts, keyboards, CPUs and motherboards for computers, soles for shoes, soap for hand-washing, paint for hospital corridors, pillows and pillow cases for beds and everything else that allows the healthcare system to keep running
The infrastructure to get fuel to the ambulances and into the cars, trains and buses which transported the medical staff to hospital
The maintenance schedules and processes for the ambulances
The processes behind the ordering of PPE for all involved
The supply chains which allowed those involved to access the tea, coffee, milk, sugar and other (hopefully legal) stimulants to keep staff going through the day and night
Staff timetabling software for everyone from cleaners to theatre managers, maintenance people to on-call surgeons
The music, art, videos, TV shows and other entertainment that kept everyone involved sufficiently energised to function

The infrastructure

Clean water
Roads
Electricity
Internet access and routing
Safety processes and culture in healthcare
… and everything else I’ve neglected to mention.

A final note

I hope it’s clear that I’m aware that the technology is all interconnected, and too complex to allow every piece to be noted: I’m sorry if I missed your piece out. The same, however, goes for the people. I come from a family containing some medical professionals and volunteers, and I’m aware of the sacrifices made not only by them, but also by the people around them who they know and love, and who see less of them than they might like, or how have to work around difficult shift patterns, or see them come back home after a long shift, worn out or traumatised by what they’ve seen and experienced. The same goes for ancillary workers and services worked in other, supporting industries.

I thank you all, both those involved directly and those involved in any of the technologies which save lives, those I’ve noted and those I’ve missed. In a few days, I hope to see a member of my family who, without your involvement, I would not ever be seeing again in this life. That is down to you.

Enarx end-to-end complete!

We now have a fully working end-to-end proof of concept, with no smoke and mirrors.

I’ve written lots about the Enarx project, a completely open source project around deploying workloads to Trusted Execution Environments, and you can find a few of the articles here:

I have some very exciting news to announce.

A team effort

Yesterday was a huge day for the Enarx project, in that we now have a fully working end-to-end proof of concept, with no smoke and mirrors (we don’t believe in those). The engineers on the team have been working really hard on getting all of the low-level pieces in place, with support from other members on CI/CD, infrastructure, documentation, community outreach and beyond. I won’t mention everyone, as I don’t want to miss anyone out, and I also don’t have their permission, but it’s been fantastic working with everyone. We’ve been edging closer and closer to having all the main pieces ready to go, and just before Christmas/New Year we got attested AMD SEV Keeps working, with the ability to access information from that attestation within the Keep. This allowed us to move to the final step, which is creating an end-to-end client-server architecture. It is this that we got running yesterday.

I happened to be the lucky person to be able to complete this part of the puzzle, building on work by the rest of the team. I don’t have the low-level expertise that many of the team have, but my background is in client-server and peer-to-peer distributed systems, and after I started learning Rust around March 2020, I decided to see if I could do something useful for the project code base: this is my contribution to the engineering. To give you an idea of what we’ve implemented, let’s look at a simple architectural diagram of an Enarx deployment.

Much of the work that’s been going on has been concentrated in the Enarx runtime component, getting WebAssembly working in SGX and SEV Trusted Execution Environments, working on syscall implementations and attestation. There’s also been quite a lot of work on glue – how we transfer information around the system in a standards-compliant way (we’re using CBOR encoding throughout). The pieces that I’ve been putting together have been the Enarx client agent, the Enarx host agent (or Enarx Keep Manager) and two pieces which aren’t visible in this diagram (but are in the more detailed one below): the Enarx Keep Loader and Enarx Wasm Loader (“App loader” in the detailed view).

The components

Let’s look at what these components do, and then explain exactly what we’ve achieved. The name in bold refers to the diagram, the name in italics relates to the Rust crate (and, where already merged, the github repository) associated with the component.

Enarx Client Agent (client) – responsible to talking to the enarx-keepmgr and requesting a Keep. It checks that the Keep is correctly set up and attested and then sends the workload (a WebAssembly package) to the enarx-wasmldr component, using HTTPS with a one-use certificate derived from the attestation process.
Enarx Keep Manager (enarx-keepmgr) – creates enarx-keepldr components at the request of the client, proxying communications to them from the client as required (for certain attestation flows, for instance). It is untrusted by the client.
Enarx Keep Loader (enarx-keepldr) – there is an enarx-keepldr per Keep, and it performs the loading of components into the Trusted Execution Environment itself. It sits outside the TEE instance, and is therefore untrusted by the client.
Enarx App Loader (enarx-wasmldr) – the enarx-wasmldr component resides within the TEE instance, and is therefore has confidentiality and integrity protection from the rest of the host. It receives the WebAssembly (Wasm) workload from the client component and may access secret information provisioned into the Keep during the attestation process.

Here’s the post I made to the Enarx chat #development channel yesterday to announce what we managed to achieve:

client -> keepmgr: “create sev keep”
keepmgr launches sev keep via systemd
client -> keepmgr: “perform attestation, include this private key” (note – private key is encrypted from keepmgr)
keepmgr -> keepldr: “attestation + private key”
keepldr creates keep, passes private key to it
wasmldr creates certificate from private key
wasmldr waits for workload
client sends workload of HTTPS to wasmldr
wasmldr accepts workload over HTTPS
wasmldr executes workload

WE HAVE A FULLY WORKING END-TO-END DEMO! Thank you everyone

What does this mean? Well, everything works! The client requests a Keep using with an AMD SEV instance, it’s created, attested, listens for an incoming connection over HTTPS, and the client sends the workload, which then executes. The workload was written in Rust and compiled to WebAssembly – it’s a real application, in other words, and not a hand-crafted piece of WebAssembly for the purposes of testing.

What’s next?

There’s lots left to do, including:

merging all of the code into the main repositories (I was working in a separate set to avoid undue impact on other efforts)
tidying it to make it more presentable (both what the demo shows and the quality of the code!)
add SGX support – we hope that we’re closing in on this very soon
make the various components production-ready (the keepmgr, for instance, doesn’t manage multiple enarx-keepldr components very well yet)
define the wire protocol fully (somewhere other than in my head)
document everything!

But most of that’s easy: it’s just engineering. 🙂

We’d love you to become involved. If you’re interested, read some of my articles, visit project home page and repositories, hang out on our chat server or watch some of our videos on YouTube. We really welcome involvement – and not just from engineers, either. Come and have a play!

Formal verification … or Ken Thompson?

“You can’t trust code that you did not totally create yourself” – Ken Thompson.

This article is an edited excerpt from my forthcoming book on Trust in Computing and the Cloud for Wiley.

How can we be sure that the code we’re running does what we think it does? One of the answers – or partial answers – to that question is “formal verification.” Formal verification is an important field of study, applying mathematics to computing, and it aims to start with proofs – at best, with an equivalent level of assurance to that of formal mathematical proofs – of the correctness of algorithms to be implemented in code to ensure that they perform the operations expected and set forth in a set of requirements. Though implementation of code can often fall down in the actual instructions created by a developer or set of developers – the programming – mistakes are equally possible at the level of the design of the code to be implemented in the first place, and so this must be a minimum step before looking at any actual implementations. What is more, these types of mistakes can be all the more hard to spot, as even if the developer has introduced no bugs in the work they have done, the implementation will be flawed by virtue of it being incorrectly defined in the first place. It is with an acknowledgement of this type of error, and an intention of reducing or eliminating it, that formal verification starts, but some areas go much further, with methods to examine concrete implementations and make statements about their correctness with regards to the algorithms which they are implementing.

Where we can make these work, they are extremely valuable, and the sort of places that they are applied are exactly where we would expect: for systems where security is paramount, and to prove the correctness of cryptographic designs and implementations. Another major focus of formal verification is software for safety systems, where the “correct” operation of the system – by which we mean “as designed and expected” – is vital. Examples might include oil refineries, fire suppression systems, nuclear power station management, aircraft flight systems and electrical grid management – unsurprisingly, given the composition of such systems, formal verification of hardware is also an important field of study. The practical application of formal verification methods to software is, however, more limited than we might like. As Alessandro Abate notes in a paper on formal verification of software:

“Two known shortcomings of standard techniques in formal verification are the limited capability to provide system-level assertions, and the scalability to large, complex models.”

To these shortcomings we can add another, extremely significant one: how sure can you be that what you are running is what you think you are running? Surely knowing what you are running is exactly why we write software, look at the source, and then compile it under our control? That, certainly, is the basic starting point for software that we care about.

The problem is arguably one of layers and dependencies, and was outlined by Ken Thompson, one of the founders or modern computing, in the lecture he gave at his acceptance of the Turing Award in 1983. It is short, stands as one of the establishing artefacts of computing security, and has weathered the tests of time: I have no hesitation in recommending that all readers of this blog read it: Reflections on Trusting Trust. In it, he describes how careful placing of malicious code in the C standard compiler could lead to vulnerabilities (his specific example is in account login code) which are not only undetectable by those without access to the source code, but also not removable. The final section of the paper is entitled “Moral”, and Thompson starts with these words:

“The moral is obvious. You can’t trust code that you did not totally create yourself. (Especially code from companies that employ people like me.) No amount of source-level verification or scrutiny will protect you from using untrusted code.”

However, as he goes on to point out, here is nothing special about the compiler:

“I could have picked on any program-handling program such as an assembler, a loader, or even hardware microcode. As the level of program gets lower, these bugs will be harder and harder to detect. A well-installed microcode bug will be almost impossible to detect.”

It is for this the reasons noted by Thompson that open source software – and hardware – is so vital to the field of computer security, and to our task of defining and understanding what “trust” means in the context of computing. Just relying on the “open source-ness” of your code is not enough: there is more work to be done in understanding your stack, the community and your requirements, but without the ability to look at the source code of all the layers of software and hardware on which you are running code, then you can have only reduced trust that what you are running is what you think you should be running, whether you have performed formal verification on it or not.