How to hire an open source developer

Our view was that a pure “algorithm coding” exercise was pretty much useless for what we wanted.

We’ve recently been hiring developers to work on the Enarx project, a security project, written almost exclusively in Rust (with a bit of Assembly), dealing with Confidential Computing. By “we”, I mean Profian, the start-up for which I’m the CEO and co-founder. We’ve now found all the people we’re looking for initially on the team (with a couple due to start in the next few weeks), though we absolutely welcome contributors to Enarx, and, if things continue to go well, we’ll definitely want to hire some more folks in the future.

Hiring people is not easy, and we were hit with a set of interesting requirements which made the task even more difficult. I thought it would be useful and interesting for the community to share how we approached the problem.

What were we looking for?

I mentioned above some interesting requirements. Here’s what the main ones were:

  • systems programming – we mainly need people who are happy programming at the systems layer. This is pretty far down the stack, with lots of interactions directly with hardware or the OS. Where we are creating client-server pieces, for instance, we’re having to write quite a lot of the protocols, manage the crypto, etc., and the tools we’re using aren’t all very mature (see “Rust” below).
  • Rust – almost all of the project is written in Rust, and what isn’t is written in Assembly language (currently exclusively x86, though that may change as we add more platforms). Rust is new, cool and exciting, but it’s still quite young, and some areas don’t have all the support you might like, or aren’t as mature as you might hope – everything from cryptography through multi-threading libraries and compiler/build infrastructure.
  • distributed team – we’re building a team of folks where can find them: we have developers in Germany, Finland, the Netherlands, North Carolina (US), Massachusetts (US), Virginia (US) and Georgia (US), I’m in the UK, our community manager is in Brazil and we have interns in India and Nigeria. We knew from the beginning that we wouldn’t have everyone in one place, and this required people who we were happy would be able to communicate and collaborate with people via video, chat and (at worst) email.
  • security – Enarx is a security project, and although we weren’t specifically looking for security experts, we do need people who are able to think and work with security top of mind, and design and write code which is applicable and appropriate for the environment.
  • git – all of our code is stored in git (mainly GitHub, with a little bit of GitLab thrown in), and so much of our interaction around code revolves around git that anybody joining us would need to be very comfortable using it as a standard tool in their day-to-day work.
  • open source – open source isn’t just a licence, it’s a mindset, and, equally important, a way of collaborating. A great deal of open source software is created by people who aren’t geographically co-located, and who might not even see themselves as a team. We needed to be sure that the people we were hiring, while gelling as a close team within the company, will also be able to collaborate with people outside the organisation and be able to embrace Profian’s “open by default” culture not just for code, but for discussions, communications and documentation.

How did we find them?

As I’ve mentioned before, in Recruiting is hard. We ended up using a variety of means to find candidates, with varying levels of success:

  • LinkedIn job adverts
  • LinkedIn searches
  • Language-specific discussion boards and hiring boards (e.g. Reddit)
  • An external recruiter (shout out to Gerald at Interstem)
  • Word-of-mouth/personal recommendations

It’s difficult to judge between them in terms of quality, but without an external recruiter, we’d certainly have struggled with quantity (and we had some great candidates from that pathway, too).

How did we select them?

We needed to measure all of the candidates against all of the requirements noted above, but not all of them were equal. For instance, although we were keen to hire Rust programmers, we were pretty sure that someone with strong C/C++ skills at the systems level would be able to pick up Rust quickly enough to be useful. On the other hand, a good knowledge of using git was absolutely vital, as we couldn’t spend time working with new team members to bring them up-to-speed on our way of working. A strong open source background was, possibly surprisingly, not a requirement, but the mindset to work in that sort of model was, and anyone with a history of open source involvement is likely to have a good knowledge of git. The same goes for the ability to work in a distributed team: so much of open source is distributed that involvement in almost any open source community was a positive indicator. Security we decided was a “nice-to-have”.

How to proceed? We wanted to keep the process simple and quick – we don’t have a dedicated HR or People function, and we’re busy trying to get code written. What we ended up was this (with slight variations), which we tried to get complete within 1-2 weeks:

  1. Initial CV/resume/github/gitlab/LinkedIn review – this to decide whether to interview
  2. 30-40 minute discussion with me as CEO, to find out if they might be a good cultural fit, to give them a chance to find out about us, and get an idea if they were as technically adept as they appeared from the first step
  3. Deep dive technical discussion led by Nathaniel, usually with me there
  4. Chat with other members of the team
  5. Coding exercise
  6. Quick decision (usually within 24 hours)

The coding exercise was key, but we decided against the usual approach. Our view was that a pure “algorithm coding” exercise of the type so beloved by many tech companies was pretty much useless for what we wanted. What we wanted to understand was whether candidates could quickly understand a piece of code, fix some problems and work with the team to do so. We created a github repository (in fact, we ended up using two – one for people a little higher up the stack) with some almost-working Rust code in it, some instructions to fix it, perform some git-related processes on it, and then improve it slightly, adding tests along the way. A very important part of the test was to get candidates to interact with the team via our chat room(s). We scheduled 15 minutes on a video call for set up and initial questions, 2 hours for the exercise (“open book” – as well as talking to the team, candidates were encouraged to use all resources available to them on the Internet), followed by a 30 minute wrap-up session where the team could ask questions and the candidate could also reflect on the task. This also allowed us to get an idea of how well the candidate was able to communicate with the team (combined with the chat interactions during the exercise). Afterwards, the candidate would drop off the call, and we’d generally make a decision within 5-10 minutes as to whether we wanted to hire them.

This generally worked very well. Some candidates struggled with the task, some didn’t communicate well, some failed to do well with the git interactions – these were the people we didn’t hire. It doesn’t mean they’re not good coders, or that they might not be a good fit for the project or the company later on, but they didn’t immediate meet the criteria we need now. Of the ones we hired, the levels of Rust experience and need for interaction with the team varied, but the level of git expertise and their reactions to our discussions afterwards was always sufficient for us to decide to take them.

Reflections

On the whole, I don’t think we’d change a huge amount about the selection process – though I’m pretty sure we could do better with the search process. The route through to the coding exercise allowed us to filter out quite a few candidates, and the coding exercise did a great job of helping us pick the right people. Hopefully everyone who’s come through the process will be a great fit and will produce great code (and tests and documentation and …) for the project. Time will tell!

Cloud security asymmetry

We in the security world have to make people understand this issue.

My book, Trust in Computer Systems and the Cloud, is due out in the next few weeks, and I was wondering as I walked the dogs today (a key part of the day for thinking!) what the most important message in the book is. I did a bit of thinking and a bit of searching, and decided that the following two paragraphs expose the core thesis of the book. I’ll quote them below and then explain briefly why (the long explanation would require me to post most of the book here!). The paragraph is italicised in the book.

A CSP [Cloud Service Provider] can have computational assurances that a tenant’s workloads cannot affect its hosts’ normal operation, but no such computational assurances are available to a tenant that a CSP’s hosts will not affect their workloads’ normal operation.

In other words, the tenant has to rely on commercial relationships for trust establishment, whereas the CSP can rely on both commercial relationships and computational techniques. Worse yet, the tenant has no way to monitor the actions of the CSP and its host machines to establish whether the confidentiality of its workloads has been compromised (though integrity compromise may be detectable in some situations): so even the “trust, but verify” approach is not available to them.”

What does this mean? There is, in cloud computing, a fundamental asymmetry: CSPs can protect themselves from you (their customer), but you can’t protect yourself from them.

Without Confidential Computing – the use of Trusted Execution Environments to protect your workloads – there are no technical measures that you can take which will stop Cloud Service Providers from looking into and/or altering not only your application, but also the data it is processing, storing and transmitting. CSPs can stop you from doing the same to them using standard virtualisation techniques, but those techniques provide you with no protection from a malicious or compromised host, or a malicious or compromised CSP.

I attended a conference recently attended by lots of people whose job it is to manage and process data for their customers. Many of them do so in the public cloud. And a scary number of them did not understand that all of this data is vulnerable, and that the only assurances they have are commercial and process-based.

We in the security world have to make people understand this issue, and realise that if they are looking after our data, they need to find ways to protect it with strong technical controls. These controls are few:

  • architectural: never deploy sensitive data to the public cloud, ever.
  • HSMs: use Hardware Security Modules. These are expensive, difficult to use and don’t scale, but they are appropriate for some sensitive data.
  • Confidential Computing: use Trusted Execution Environments (TEEs) to protect data and applications in use[1].

Given my interest – and my drive to write and publish my book – it will probably come as no surprise that this is something I care about: I’m co-founder of the Enarx Project (an open source Confidential Computing project) and co-founder and CEO of Profian (a start-up based on Enarx). But I’m not alone: the industry is waking up to the issue, and you can find lots more about the subject at the Confidential Computing Consortium‘s website (including a list of members of the consortium). If this matters to you – and if you’re an enterprise company who uses the cloud, it almost certainly already does, or will do so – then please do your research and consider joining as well. And my book is available for pre-order!

Logs – good or bad for Confidential Computing?

I wrote a simple workload for testing. It didn’t work.

A few weeks ago, we had a conversation on one of the Enarx calls about logging. We’re at the stage now (excitingly!) where people can write applications and run them using Enarx, in an unprotected Keep, or in an SEV or SGX Keep. This is great, and almost as soon as we got to this stage, I wrote a simple workload to test it all.

It didn’t work.

This is to be expected. First, I’m really not that good a software engineer, but also, software is buggy, and this was our very first release. Everyone expects bugs, and it appeared that I’d found one. My problem was tracing where the issue lay, and whether it was in my code, or the Enarx code. I was able to rule out some possibilities by trying the application in an unprotected (“plain KVM”) Keep, and I also discovered that it ran under SEV, but not SGX. It seemed, then, that the problem might be SGX-specific. But what could I do to look any closer? Well, with very little logging available from within a Keep, there was little I could do.

Which is good. And bad.

It’s good because one of the major points about using Confidential Computing (Enarx is a Confidential Computing framework) is that you don’t want to leak information to untrusted parties. Since logs and error messages can leak lots and lots of information, you want to restrict what’s made available, and to whom. Safe operation dictates that you should make as little information available as you possibly can: preferably none.

It’s bad because there are times when (like me) you need to work out what’s gone wrong, and find out whether it’s in your code or the environment that you’re running your application in.

This is where the conversation about logging came in. We’d started talking about it before this issue came up, but this made me realise how important it was. I started writing a short blog post about it, and then stopped when I realised that there are some really complex issues to consider. That’s why this article doesn’t go into them in depth: you can find a much more detailed discussion over on the Enarx blog. But I’m not going to leave you hanging: below, you’ll find the final paragraph of the Enarx blog article. I hope it piques your interest enough to go and find out more.

In a standard cloud deployment, there is little incentive to consider strong security controls around logging and debugging, simply because the host has access not only to all communications to and from a hosted workload, but also to all the code and data associated with the workload at runtime.  For Confidential Computing workloads, the situation is very different, and designers and architects of the TEE infrastructure (e.g. the Enarx projects) and even, to a lesser extent, of potential workloads themselves, need to consider very carefully the impact of host gaining access to messages associated with the workload and the infrastructure components.  It is, realistically, infeasible to restrict all communication to levels appropriate for deployment, so it is recommended that various profiles are created which can be applied to different stages of a deployment, and whose use is carefully monitored, logged (!) and controlled by process.


Header image by philm1310 from Pixabay.

Enarx first release

Write an application, compile it to WebAssembly, and then run it in one of three Keeps types.

I was on holiday last week, and I took the opportunity not to write a blog post, but while I was sunning myself[1] at the seaside, the team did a brilliant thing: we have our first release of Enarx, and a new look for the website, to boot.

To see the new website, head over to https://enarx.dev. There, you’ll find new updated information about the project, details of how to get involved, and – here’s the big news – instructions for how to download and use Enarx. If you’re a keen Rustacean, you can also go straight to crates.io (https://crates.io/crates/enarx) and start off there. Up until now, in order to run Enarx, you’ve had to do quite a lot of low level work to get things running, run your own github branches, understand how everything fits together and manage your own development environment. This has now all changed.

This first release, version 0.1.1, is codenamed Alamo, and provides an easy way in to using Enarx. As always, it’s completely open source: you can look at every single line of our code. It doesn’t provide a full feature set, but what it does do is allow you, for the first time, to write an application, compile it to WebAssembly, and then run it in one of three Keep[2] types:

  1. KVM – this is basically a debugging Keep, in that it doesn’t provide any confidentiality or integrity protection, but it does allow you to get running and to try things even if you don’t have access to specialist hardware. A standard Linux machine should do you fine.
  2. SEV – this is a Keep using AMD’s SEV technology, specifically the newer version, SEV-SNP. This requires access to a machine which supports it[3].
  3. SGX – this is a Keep using Intel’s SGX technology. Again, this requires access to a machine which supports it[3].

The really important point here is that you’re running the same binary on each of these architectures. No recompilation for different architectures: just plain old WebAssembly[4].

Current support

There’s a lot more work to do, but what do we support at the moment?

  • running WebAssembly
  • KVM, SEV and SGX Keeps (see above)
  • stdin and stdout from/to the host – this is temporary, as the host is untrusted in the Enarx model, but until we have networking support (see below), we wanted to provide a simple way to manage input and output from a Keep.

There’s lots more to come – networking and attestation are both high on the list – but now anyone can start playing with Enarx. And, we hope, submitting enhancement and feature requests, not to mention filing bugs (we know there will be some!): to do so, hop over to https://github.com/enarx/enarx/issues.

To find out more, please head over to the website – there’s loads to see – or join us on chat channel over at https://chat.enarx.dev to find out more and get involved.


1 – it’s the British seaside, in October, so “sunning” might be a little inaccurate.

2 – a Keep is what we call a TEE instance set up for you to run an application in.

3 – we have AMD and SGX machines available for people who contribute to the project – get in touch!

4 – WebAssembly is actually rather new, but “plain old” sounds better than “vanilla”. Not my favourite ice cream flavour[5].

5 – my favourite basic ice cream flavour is strawberry. Same for milkshakes.

Announcing Profian

Profian, a security start-up in the Confidential Computing space

I’m very excited to announce Profian, a security start-up in the Confidential Computing space that I co-founded with Nathaniel McCallum, came out of stealth mode today to announce that we’ve completed our Seed Round – you can find the press release here. This is the culmination of months of hard work and about two years of a vision that we’ve shared and developed since coming up with the idea of Enarx. Profian will be creating products and services around Enarx, and we’re committed to keeping everything we do open source: not just because we believe in open source as an ethical choice, but also because we believe that it’s best for security.

Enarx grew out of a vision that we had to simplify use of Trusted Execution Environments like AMD’s SEV and Intel’s SGX[1], while not compromising on the security that we believe the industry wants and needs. Enarx aims to allow you to deploy applications to any of the supported platforms without needing to recompile for each one, and to simplify both the development and deployment process. It supports WebAssembly as its runtime, allowing a seamless execution environment across multiple hardware types. Engineering for Enarx was initially funded by Red Hat, and towards the end of 2020, we started looking for a way to ensure long-term resourcing: out of this Profian was born. We managed to secure funding from two VC funds – Project A (lead investor) and Illuminate Financial – and four amazing angel investors. Coming out of stealth means that we can now tell more people about what we’re doing.

Profian is a member of two great industry bodies: the Confidential Computing Consortium (a Linux Foundation project to promote open source around Trusted Execution Environments) and the Bytecode Alliance (an industry group to promote and nurture WebAssembly, the runtime which Enarx supports).

The other important thing to announce is that with funding of Profian comes our chance to develop Enarx and its community into something really special.

If it’s your thing, you can find the press release on Business Wire, and more information on the company press page.

A few questions and answers

What’s confidential computing?

I tend to follow the Confidential Computing Consortium’s definition: “Confidential Computing protects data in use by performing computation in a hardware-based Trusted Execution Environment”.

What does Profian mean?

It’s Anglo-Saxon, the language also sometimes called “Old English”, which was spoken in (modern day) England and parts of Scotland from around the mid-5th century BCE to 1066, when Norman French had such an impact on the language that it changed (to Middle English).

One online Anglo-Saxon dictionary defines profian thus:

profian - 1. to esteem; regard as 2. to test ; try ; prove 3. to show evidence of ; evince

It’s the root of the English word “to prove”, from which we also get “proof” and “proven”. We felt that this summed up much of what we want to be doing, and is nicely complementary to Enarx.

How is Profian pronounced?

Not the way most pre-Conquest Anglo-Saxons would probably have pronounced it, to be honest. We (well, I) thought about trying to go with a more “authentic” pronunciation, and decided (or was convinced…) that it was too much trouble. We’re going with “PROH-vee-uhn”[2].

What does Enarx mean?

You’ll find more information about this (and how to pronounce Enarx), over at the Enarx FAQ. TL;DR – we made it up.

Who’s part of the company?

Well, there’s me (I’m the CEO), Nathaniel McCallum (the CTO) and a small team of developers. We also have Nick Vidal, who we recruited as Community Manager for Enarx. By the beginning of October, we expect to have six employees in five different countries spread across three separate continents[3].

What’s next?

Well, lots of stuff. There’s so much to do when running a company of which I knew next to nothing when we started. You would not believe the amount of work involved with registering a company, setting up bank accounts, recruiting people, paying people, paying invoices, etc. – and that’s not even about creating products. We absolutely plan to do this (or the investors are not going to be happy).

No – what’s next for this blog?

Ah, right. Well, I plan to keep it going. There will be more articles about my book on trust, security, open source and probably VCs, funding and the rest. There have been quite a few topics I’ve just not felt safe blogging about until Profian came out of stealth mode. Keep an eye out.


1 – there are more coming, such as Arm CCA (also known as “Realms”), and Intel’s TDX – we plan to support these are they become available.

2 – Anglo-Saxons would probably have gone with something more like “PRO-fee-an”, where the “o” has sound like “pop”.

3 – yes, I know we’ve not made it easy on ourselves.

Arm joins the Confidential Computing party

Arm’s announcement of Realms isn’t just about the Edge

The Confidential Computing Consortium is a Linux Project designed to encourage open source projects around confidential computing. Arm has been part of the consortium for a while – in fact, the company is Premier Member – but things got interesting on the 30th March, 2021. That’s when Arm announced their latest architecture: Arm 9. Arm 9 includes a new set of features, called Realms. There’s not a huge amount of information in the announcement about Realms, but Arm is clear that this is their big play into Confidential Computing:

To address the greatest technology challenge today – securing the world’s data – the Armv9 roadmap introduces the Arm Confidential Compute Architecture (CCA).

I happen to live about 30 minutes’ drive from the main Arm campus in Cambridge (UK, of course), and know a number of Arm folks professionally and socially – I think I may even have interviewed for a job with them many moons ago – but I don’t want to write a puff piece about the company or the technology[1]. What I’m interested in, instead, is the impact this announcement is likely to have on the Confidential Computing landscape.

Arm has had an element in their architecture for a while called TrustZone which provides a number of capabilities around security, but TrustZone isn’t a TEE (Trusted Execution Environment) on its own. A TEE is the generally accepted unit of confidential computing – the minimum building block on which you can build. It is arguably possible to construct TEEs using TrustZone, but that’s not what it’s designed for, and Arm’s decision to introduce Realms strongly suggests that they want to address this. This is borne out by the press release.

Why is all this important? I suspect that few of you have laptops or desktops that run on Arm (Raspberry Pi machines apart – see below). Few of the servers in the public cloud run Arm, and Realms are probably not aimed particularly at your mobile phone (for which TrustZone is a better fit). Why, then, is Arm bothering to make a fuss about this and to put such an enormous design effort into this new technology? There are two answers, it seems to me, one of which is probably pretty much a sure thing, and the other of which is more of a competitive gamble.

Answer 1 – the Edge

Despite recent intrusions by both AMD and Intel into the Edge space, the market is dominated by Arm-based[3] devices. And Edge security is huge, partly because we’re just seeing a large increase in the number of Edge devices, and partly because security is really hard at the Edge, where devices are more difficult to defend, both logically (they’re on remote networks, more vulnerable to malicious attack) and physically (many are out of the control of their owners, living on customer premises, up utility poles, on gas pipelines or in sports stadia, just to give a few examples). One of the problems that confidential computing aims to solve is the issue that, traditionally, once an attacker has physical access to a system, it should be considered compromised. TEEs allow some strong mitigations against that problem (at least against most attackers and timeframes), so making it easy to create and use TEEs on the Edge makes a lot of sense. With the addition of Realms to the Arm 9 architecture, Arm is signally its intent to address security on the Edge, and to defend and consolidate its position as leader in the market.

Answer 2 – the Cloud

I mentioned above that few public cloud hosts run Arm – this is true, but it’s likely to change. Arm would certainly like to see it change, and to see its chipsets move into the cloud mainstream. There has been a lot of work to improve support for server-scale Arm within Linux (in fact, open source support for Arm is generally excellent, not least because of the success of Arm-based chips in Raspberry Pi machines). Amazon Cloud Services (AWS) started offering Arm-based servers to customers as long ago as 2018. This is a market in which Arm would clearly love to be more active and carve out a larger share, and the growing importance of confidential computing in the cloud (and public and private) means that having a strong story in this space was important: Realms are Arm’s answer to this.

What next?

An announcement of an architecture is not the same as availability of hardware or software to run on it. We can expect it to be quite a few months before we see production chips running Arm 9, though evaluation hardware should be available to trusted partners well before that, and software emulation for various components of the architecture will probably come even sooner. This means that those interested in working with Realms should be able to get things moving and have something ready pretty much by the time of availability of production hardware. We’ll need to see how easy they are to use, what performance impact they have, etc., but Arm do have an advantage here: as they are not the first into the confidential computing space, they’ve had the opportunity to watch Intel and AMD and see what has worked, and what hasn’t, both technically and in terms of what the market seems to like. I have high hopes for Arm Realms, and Enarx, the open source confidential computing project with which I’m closely involved, has plans to support them when we can: our architecture was designed with multi-platform support from the beginning.


1 – I should also note that I participated in a panel session on Confidential Computing which was put together by Arm for their “Arm Vision Day”, but I was in no way compensated for this[2].

2 -in fact, the still for the video is such a terrible picture of me that I think maybe I have grounds to sue for it to be taken down.

3 – Arm doesn’t manufacture chips itself: it licenses its designs to other companies, who create, manufacture and ship devices themselves.

GET/SET methods for open source projects

Or – how to connect with open source

I’m aware that many of the folks who read my blog already know lots about open source, but I’m also aware that there are many who know little if anything about it. I’m a big, big proponent or open source software (and beyond, such as open hardware), and there are lots of great resources you can find to learn more about it: a very good starting point is Opensource.com. It’s run by a bunch of brilliant people for the broader community by my current employer, Red Hat (I should add a disclaimer that I’m not only employed by Red Hat, but also a “Correspondent” at Opensource.com – a kind of frequent contributor/Elder Thing), and has articles on pretty much every aspect of open source that you can imagine.

I was thinking about APIs today (they’re in the news this week after a US Supreme Court Judgment on an argument between Google and Oracle), and it occurred to me that if I were interested in understanding how to interacting with open source at the project level, but didn’t know much about it, then a quick guide might be useful. The same goes if I were involved in an open source project (such as Enarx) which was interested in attracting contributors (particularly techie contributors) who aren’t already knowledgeable about open source. Given that most programmers will understand what GET and SET methods do (one reads data, the other writes data), I thought this might be useful way to consider engagement[1]. I’ll start with GET, as that’s how you’re likely to be starting off, as well – finding out more about the project – and then move to SET. This is far from an exhaustive list, but I hope that I’ve hit most of the key ways you’re most likely to start getting involved/encourage others to get involved. The order I’ve chosen reflects what I suspect is a fairly typical approach to finding out more about a project, particularly for those who aren’t open source savvy already, but, as they say, YMMV[2].

I’ve managed to stop myself using Enarx as the sole source of examples, but have tried to find a variety of projects to give you a taster. Disclaimer: their inclusion here does not mean that I am a user or contributor to the project, nor is it any guarantee of their open source credentials, code quality, up-to-date-ness, project maturity or community health[4].

GET methods

  • Landing page – the first encounter that you may have with a project will probably be its landing page. Some projects go for something basic, others apply more design, but you should be able to use this as the starting point for your adventures around the project. You’d generally hope to find a link various of the other resources listed below from this page. Sigstore
  • Wiki – in many cases, the project will have a wiki. This could be simple, it could be complex. It may allow editing by anyone, or only by a select band of contributors to the project, and its relevance as source-of-truth may be impacted by how up to date it is, but the wiki is usually an excellent place to start. Fedora Project
  • Videos – some projects maintain a set of videos about their project. These may include introductions to the concepts, talking head interviews with team members, conference sessions, demos, HOW-TOs and more. It’s also worth looking for videos put up my contributors to the project, but which aren’t necessarily officially owned by the project. Rust Language
  • Code of Conduct – many projects insist that their project members follow a code of conduct, to reduce harassment, reduce friction and generally make the project a friendly, more inclusive and more diverse place to be. Linux kernel
  • Binary downloads – as projects get more mature, they may choose to provide pre-compiled binary downloads for users. More technically-inclined users may choose to compile their own from the code base (see below) even before this, but binary downloads can be a quick way to try out a project and see whether it does what you want. Chocolate Doom (a Doom port)
  • Design documentation – without design documentation, it can be very difficult to get really into a project (I’ve written about the importance of architecture diagrams on this blog before). This documentation is likely to include everything from an API definition up to complex use cases and threat models. Kubernetes
  • Code base – you’ve found out all you need to get going: it’s time to look at the code! This may vary from a few lines to many thousands, may include documentation in comments, may include test cases: but if it’s not there, then the project can’t legitimately call itself open source. Rocket Rust web framework[5]
  • Email/chat – most projects like to have a way for contributors to discuss matters asynchronously. The preferred medium varies between projects, but most will choose an email list, a chat server or both. Here’s where to go to get to know other users and contributors, ask questions, celebrate successful compiles, and just hang out. Enarx chat
  • Meet-ups, video conferences, calls, etc. – though physically meetings are tricky for many at the moment (I’m writing as Covid-19 still reduces travel opportunities for many), having ways for community members and contributors to get together synchronously can be really helpful for everybody. Sometimes these are scheduled on a daily, weekly or monthly basis, sometimes they coincide with other, larger meet-ups, sometimes a project gets big enough to have its own meet-ups, and sometimes so big that there are meet-ups of sub-projects or internal interest groups. Linux Security Summit Europe

SET methods

  • Bug reports – for many of us, the first time we contribute anything substantive back to an open source project is when we file a bug report. These types of bug reports – from new users – can be really helpful for projects, as they not only expose bugs which may not already be known to the project, but they also give clues as to how actual users of the project are trying to use the code. If the project already publishes binary downloads (see above), then you don’t even need to have compiled the code to try it and submit a bug report, but bug reports related to compilation and build can also be extremely useful to the project. Sometimes, the mechanism for bug reporting also provides a way to ask more general questions about the project, or to ask for new features. exa (replacement for the ls command)
  • Tests – once you’ve starting using the project, another way to get involved (particularly once you start contributing code) can be to design and submit tests for how the project ought to work. This can be a great way to unearth both your assumptions (and lack of knowledge!) about the project, but also the project’s design assumptions (some of which may well be flawed). Tests are often part of the code repository, but not always. Gnome Shell
  • Wiki – the wiki can be a great way to contribute to the project whether you’re coding or not. Many projects don’t have as much information available as they should do, and that information may often not be aimed at people coming to the project “fresh”. If this is what you’ve done, then you’re in a great position to write material which will help other “newbs” to get into the project faster, as you’ll know what would have helped you if it had been there. Wine (Windows Emulator for Linux)
  • Code – last, but not least, you can write code. You may take hours, months or years to get to this stage – or may never reach it – but open source software is nothing without its code. If you’ve paid enough attention to the other steps, got involved in the community, understood what the project aims to do, and have the technical expertise (which you may well develop as you go!), then writing code may be way to you want to do. Enarx (again)

1 – I did consider standard RESTful verbs – GET, PUT, POST and DELETE, but that felt rather contrived[2].

2 – And I don’t like the idea of DELETE in this context!

3 – “Your Mileage May Vary”, meaning, basically, that your experience may be different, and that’s to be expected.

4 – that said, I do use lots of them!

5 – I included this one because I’ve spent far too much of my time look at this over the past few months…

Enarx end-to-end complete!

We now have a fully working end-to-end proof of concept, with no smoke and mirrors.

I’ve written lots about the Enarx project, a completely open source project around deploying workloads to Trusted Execution Environments, and you can find a few of the articles here:

I have some very exciting news to announce.

A team effort

Yesterday was a huge day for the Enarx project, in that we now have a fully working end-to-end proof of concept, with no smoke and mirrors (we don’t believe in those). The engineers on the team have been working really hard on getting all of the low-level pieces in place, with support from other members on CI/CD, infrastructure, documentation, community outreach and beyond. I won’t mention everyone, as I don’t want to miss anyone out, and I also don’t have their permission, but it’s been fantastic working with everyone. We’ve been edging closer and closer to having all the main pieces ready to go, and just before Christmas/New Year we got attested AMD SEV Keeps working, with the ability to access information from that attestation within the Keep. This allowed us to move to the final step, which is creating an end-to-end client-server architecture. It is this that we got running yesterday.

I happened to be the lucky person to be able to complete this part of the puzzle, building on work by the rest of the team. I don’t have the low-level expertise that many of the team have, but my background is in client-server and peer-to-peer distributed systems, and after I started learning Rust around March 2020, I decided to see if I could do something useful for the project code base: this is my contribution to the engineering. To give you an idea of what we’ve implemented, let’s look at a simple architectural diagram of an Enarx deployment.

Simple Enarx architectural diagram

Much of the work that’s been going on has been concentrated in the Enarx runtime component, getting WebAssembly working in SGX and SEV Trusted Execution Environments, working on syscall implementations and attestation. There’s also been quite a lot of work on glue – how we transfer information around the system in a standards-compliant way (we’re using CBOR encoding throughout). The pieces that I’ve been putting together have been the Enarx client agent, the Enarx host agent (or Enarx Keep Manager) and two pieces which aren’t visible in this diagram (but are in the more detailed one below): the Enarx Keep Loader and Enarx Wasm Loader (“App loader” in the detailed view).

Detailed Enarx architectural diagram

The components

Let’s look at what these components do, and then explain exactly what we’ve achieved. The name in bold refers to the diagram, the name in italics relates to the Rust crate (and, where already merged, the github repository) associated with the component.

  • Enarx Client Agent (client) – responsible to talking to the enarx-keepmgr and requesting a Keep. It checks that the Keep is correctly set up and attested and then sends the workload (a WebAssembly package) to the enarx-wasmldr component, using HTTPS with a one-use certificate derived from the attestation process.
  • Enarx Keep Manager (enarx-keepmgr) – creates enarx-keepldr components at the request of the client, proxying communications to them from the client as required (for certain attestation flows, for instance). It is untrusted by the client.
  • Enarx Keep Loader (enarx-keepldr) – there is an enarx-keepldr per Keep, and it performs the loading of components into the Trusted Execution Environment itself. It sits outside the TEE instance, and is therefore untrusted by the client.
  • Enarx App Loader (enarx-wasmldr) – the enarx-wasmldr component resides within the TEE instance, and is therefore has confidentiality and integrity protection from the rest of the host. It receives the WebAssembly (Wasm) workload from the client component and may access secret information provisioned into the Keep during the attestation process.

Here’s the post I made to the Enarx chat #development channel yesterday to announce what we managed to achieve:

  1. client -> keepmgr: “create sev keep”
  2. keepmgr launches sev keep via systemd
  3. client -> keepmgr: “perform attestation, include this private key” (note – private key is encrypted from keepmgr)
  4. keepmgr -> keepldr: “attestation + private key”
  5. keepldr creates keep, passes private key to it
  6. wasmldr creates certificate from private key
  7. wasmldr waits for workload
  8. client sends workload of HTTPS to wasmldr
  9. wasmldr accepts workload over HTTPS
  10. wasmldr executes workload

WE HAVE A FULLY WORKING END-TO-END DEMO! Thank you everyone

What does this mean? Well, everything works! The client requests a Keep using with an AMD SEV instance, it’s created, attested, listens for an incoming connection over HTTPS, and the client sends the workload, which then executes. The workload was written in Rust and compiled to WebAssembly – it’s a real application, in other words, and not a hand-crafted piece of WebAssembly for the purposes of testing.

What’s next?

There’s lots left to do, including:

  • merging all of the code into the main repositories (I was working in a separate set to avoid undue impact on other efforts)
  • tidying it to make it more presentable (both what the demo shows and the quality of the code!)
  • add SGX support – we hope that we’re closing in on this very soon
  • make the various components production-ready (the keepmgr, for instance, doesn’t manage multiple enarx-keepldr components very well yet)
  • define the wire protocol fully (somewhere other than in my head)
  • document everything!

But most of that’s easy: it’s just engineering. 🙂

We’d love you to become involved. If you’re interested, read some of my articles, visit project home page and repositories, hang out on our chat server or watch some of our videos on YouTube. We really welcome involvement – and not just from engineers, either. Come and have a play!

Vint Cerf’s “game changer”

I’m really proud to be involved with a movement which I believe can change the way we do computing.

Today’s article is a little self-indulgent, but please bear with me, as I’m a little excited. Vint Cerf is one of a small handful of people who have a claim to being called “greats”. He’s one of the co-developers of TCP/IP protocol with Bob Kahn in 1974, and has been working on technology – much of it pretty cool technology – since then. I turned 50 recently, and if I’d achieved half of what he had by his 50th birthday, I’d be feeling more accomplished than I do right now! As well as his work in technology, he’s also an advocate for accessibility, which is something which is also dear to my heart.

What does this have to do with Alice, Eve and Bob – a security blog? Well, last week, Dark Reading[1], an influential technology security site, published a commentary piece by Cerf under its “Cloud” heading: Why Confidential Computing is a Game Changer. I could hardly have been more pleased: this is an area which I’m very excited about, and which the Enarx project, of which I’m co-founder, addresses. The Enarx project is part of the Confidential Computing Consortium (mentioned in Cerf’s article), a Linux Foundation project to increase use of confidential computing through open source projects.

So, what is confidential computing? Cerf describes it as “a breakthrough technology that encrypts data in use, while it is being processed”. He goes on to give a good description of the technology, noting that Google (his employer[2]) has recently released a product using confidential computing. Google is actually far from the first cloud service provider to do this, but it’s only fair that Cerf should mention his employer’s services from time to time: I’m going to forgive him, given how enthusiastic he is about the technology more generally. He describes it as a transformational technology which “will and should be a part of every enterprise cloud deployment”.

I agree, and it’s really exciting to see such a luminary embracing the possibilities the confidential computing presents. For those readers who aren’t aware of what it is, confidential computing allows you to keep data and processes secret in the cloud, on private servers, on the Edge, IoT, etc. – even from administrators, hypervisors and the host kernel. It uses TEEs – Trusted Execution Environments – to protect the confidentiality and integrity of the workloads (application, programs) that you want to run. If you’re not sure you trust your cloud provider, if your regulatory body won’t let you run your applications in certain places, if you want to deploy to machines which are vulnerable to attack – physical or logical – then TEEs and confidential computing can help.

You can find a more information in some of my articles:

You can always visit the Confidential Computing Consortium[3] or visit the Enarx project (links above): all of our code and documentation is open, and we’d love to see you. I’m really proud to be involved with – in fact, deeply embedded in – a movement which I believe can change the way we do computing. And really excited that someone like Vint Cerf agrees.


1 – I have no affiliation with Dark Reading, though I do recommend it to readers of this blog.

2- neither do I have any affiliation with Google or Alphabet, its parent!

3 – I am, however, a member of both the Governing Board and the Technical Advisory Council of the Confidential Computing Consortium. I’m also the Treasurer.

Why Enarx is open

It’s not just our coding that we do in the open.

When Nathaniel McCallum and I embarked on the project which is now called Enarx, we made one decision right at the beginning: the code for Enarx would be open source, a stance fully supported by our employer Red Hat (see standard disclaimer). All of it, and for ever. That’s a decision that we’ve not regretted at any point, and it’s something we stand behind. As soon as we had enough code for a demo, and were ready to show it, we created a repository on github and made it public. There’s a very small exception, which is that there are some details of upcoming chip features that are shared with us under NDA[1] where if we write code for them, publishing that code would be a breach of the NDA. But where this applied (which is rarely) we are absolutely clear with the various vendors that we intend to make the code open as soon as possible, and lobby them to release details as early as they can (which may be earlier than they might prefer), so that more experts can look over both their designs and our code.

Auditability and trust

This brings us to possibly the most important reasons for making Enarx open source: auditability and trust. Enarx is a security-related project, and I believe passionately not only that security should be done in the open, but that if anybody is actually going to trust their sensitive data, algorithms and workloads to a piece of software, then they want to be in a position where as many experts as possible have looked at it, scrutinised it, criticised it and improved it: whether that is the people running the software, their employees, contractors or (even better) the wider security community. The more people who check the code, the happier you should be to trust it. This is important for any piece of security software, but vital for software such as Enarx which is designed to protect your more most sensitive workloads.

Bug-catching

There are bugs in Enarx. I know: I’m writing some of the code[2] and I found one yesterday (which I’d put in), just as I was about to give a demo[3]. It is very, very difficult to write perfect code, and we know that if we make our source open, then more people can help us fix issues.

Commonwealth

For Nathaniel and me, open source is an ethical issue, and we make no apologies for that. I think it’s the same for most, if not all, of the team working on Enarx. This include a number of Red Hat employees (see standard disclaimer), so shouldn’t come as a surprise, but we have non-Red Hat contributors from a number of backgrounds, and we feel that Enarx should be a Common Good, and contribute to the commonwealth of intellectual property out there.

More brain power

Making something open source doesn’t just make it easier to fix bugs: it can improve the quality of what you produce in general. The more brain power you have to apply to the problem, but better your chances of making something great – assuming that the brain power is applied efficiently (not always an easy task!). We had a design meeting yesterday where one of the participants said towards the end, “I’m sure I could implement some of this, but don’t know a huge amount about this topic, and I’m worried that I’m not contributing to this discussion.” In fact, they had, by asking questions and clarifying some points, and we assured them that we wanted to include experienced, senior developers for their expertise and knowledge, and to pull out assumptions and to validate the design, and not because we expected everybody to be experts in all parts of the project. Having bright people around, involved in design and coding, spreads expertise and knowledge, and helps keep the work from becoming an insulated, isolated “ivory tower” construction, understood by few, and almost impossible to validate.

Not just code

It’s not just our coding that we do in the open. We manage our architecture in the open, our design meetings, our protocol design, our design methodology[4], our documentation, our bug-tracking, our chat, our CI/CD processes: all of it is open. The one exception is our vulnerability management process, which needs to have the opportunity for confidential exposure for a limited time.

We also take diversity seriously, and the project contributors are subject to the Contributor Covenant Code of Conduct.

In short, Enarx is an open project. I’m sure we could do better, and we’ll strive for that, but our underlying principles are that open is good in general, and vital for security. If you agree, please come and visit!


1 – Non-Disclosure Agreement.

2 – to the surprise of many of the team, including myself. At least it’s not in Perl.

3 – I fixed it. Admittedly after the demo.

4 – we’ve just moved to a Sprint pattern – the details of which we designed and agreed in the open.