How to hire an open source developer

Our view was that a pure “algorithm coding” exercise was pretty much useless for what we wanted.

We’ve recently been hiring developers to work on the Enarx project, a security project, written almost exclusively in Rust (with a bit of Assembly), dealing with Confidential Computing. By “we”, I mean Profian, the start-up for which I’m the CEO and co-founder. We’ve now found all the people we’re looking for initially on the team (with a couple due to start in the next few weeks), though we absolutely welcome contributors to Enarx, and, if things continue to go well, we’ll definitely want to hire some more folks in the future.

Hiring people is not easy, and we were hit with a set of interesting requirements which made the task even more difficult. I thought it would be useful and interesting for the community to share how we approached the problem.

What were we looking for?

I mentioned above some interesting requirements. Here’s what the main ones were:

  • systems programming – we mainly need people who are happy programming at the systems layer. This is pretty far down the stack, with lots of interactions directly with hardware or the OS. Where we are creating client-server pieces, for instance, we’re having to write quite a lot of the protocols, manage the crypto, etc., and the tools we’re using aren’t all very mature (see “Rust” below).
  • Rust – almost all of the project is written in Rust, and what isn’t is written in Assembly language (currently exclusively x86, though that may change as we add more platforms). Rust is new, cool and exciting, but it’s still quite young, and some areas don’t have all the support you might like, or aren’t as mature as you might hope – everything from cryptography through multi-threading libraries and compiler/build infrastructure.
  • distributed team – we’re building a team of folks where can find them: we have developers in Germany, Finland, the Netherlands, North Carolina (US), Massachusetts (US), Virginia (US) and Georgia (US), I’m in the UK, our community manager is in Brazil and we have interns in India and Nigeria. We knew from the beginning that we wouldn’t have everyone in one place, and this required people who we were happy would be able to communicate and collaborate with people via video, chat and (at worst) email.
  • security – Enarx is a security project, and although we weren’t specifically looking for security experts, we do need people who are able to think and work with security top of mind, and design and write code which is applicable and appropriate for the environment.
  • git – all of our code is stored in git (mainly GitHub, with a little bit of GitLab thrown in), and so much of our interaction around code revolves around git that anybody joining us would need to be very comfortable using it as a standard tool in their day-to-day work.
  • open source – open source isn’t just a licence, it’s a mindset, and, equally important, a way of collaborating. A great deal of open source software is created by people who aren’t geographically co-located, and who might not even see themselves as a team. We needed to be sure that the people we were hiring, while gelling as a close team within the company, will also be able to collaborate with people outside the organisation and be able to embrace Profian’s “open by default” culture not just for code, but for discussions, communications and documentation.

How did we find them?

As I’ve mentioned before, in Recruiting is hard. We ended up using a variety of means to find candidates, with varying levels of success:

  • LinkedIn job adverts
  • LinkedIn searches
  • Language-specific discussion boards and hiring boards (e.g. Reddit)
  • An external recruiter (shout out to Gerald at Interstem)
  • Word-of-mouth/personal recommendations

It’s difficult to judge between them in terms of quality, but without an external recruiter, we’d certainly have struggled with quantity (and we had some great candidates from that pathway, too).

How did we select them?

We needed to measure all of the candidates against all of the requirements noted above, but not all of them were equal. For instance, although we were keen to hire Rust programmers, we were pretty sure that someone with strong C/C++ skills at the systems level would be able to pick up Rust quickly enough to be useful. On the other hand, a good knowledge of using git was absolutely vital, as we couldn’t spend time working with new team members to bring them up-to-speed on our way of working. A strong open source background was, possibly surprisingly, not a requirement, but the mindset to work in that sort of model was, and anyone with a history of open source involvement is likely to have a good knowledge of git. The same goes for the ability to work in a distributed team: so much of open source is distributed that involvement in almost any open source community was a positive indicator. Security we decided was a “nice-to-have”.

How to proceed? We wanted to keep the process simple and quick – we don’t have a dedicated HR or People function, and we’re busy trying to get code written. What we ended up was this (with slight variations), which we tried to get complete within 1-2 weeks:

  1. Initial CV/resume/github/gitlab/LinkedIn review – this to decide whether to interview
  2. 30-40 minute discussion with me as CEO, to find out if they might be a good cultural fit, to give them a chance to find out about us, and get an idea if they were as technically adept as they appeared from the first step
  3. Deep dive technical discussion led by Nathaniel, usually with me there
  4. Chat with other members of the team
  5. Coding exercise
  6. Quick decision (usually within 24 hours)

The coding exercise was key, but we decided against the usual approach. Our view was that a pure “algorithm coding” exercise of the type so beloved by many tech companies was pretty much useless for what we wanted. What we wanted to understand was whether candidates could quickly understand a piece of code, fix some problems and work with the team to do so. We created a github repository (in fact, we ended up using two – one for people a little higher up the stack) with some almost-working Rust code in it, some instructions to fix it, perform some git-related processes on it, and then improve it slightly, adding tests along the way. A very important part of the test was to get candidates to interact with the team via our chat room(s). We scheduled 15 minutes on a video call for set up and initial questions, 2 hours for the exercise (“open book” – as well as talking to the team, candidates were encouraged to use all resources available to them on the Internet), followed by a 30 minute wrap-up session where the team could ask questions and the candidate could also reflect on the task. This also allowed us to get an idea of how well the candidate was able to communicate with the team (combined with the chat interactions during the exercise). Afterwards, the candidate would drop off the call, and we’d generally make a decision within 5-10 minutes as to whether we wanted to hire them.

This generally worked very well. Some candidates struggled with the task, some didn’t communicate well, some failed to do well with the git interactions – these were the people we didn’t hire. It doesn’t mean they’re not good coders, or that they might not be a good fit for the project or the company later on, but they didn’t immediate meet the criteria we need now. Of the ones we hired, the levels of Rust experience and need for interaction with the team varied, but the level of git expertise and their reactions to our discussions afterwards was always sufficient for us to decide to take them.

Reflections

On the whole, I don’t think we’d change a huge amount about the selection process – though I’m pretty sure we could do better with the search process. The route through to the coding exercise allowed us to filter out quite a few candidates, and the coding exercise did a great job of helping us pick the right people. Hopefully everyone who’s come through the process will be a great fit and will produce great code (and tests and documentation and …) for the project. Time will tell!

Trust book – playlist!

A playlist of music to which I’d listened and which I’d enjoyed over the months it took to write the book.

I had probably more fun than I deserved to have writing the acknowledgements section of my book, Trust in Computer Systems and the Cloud (published by Wiley at the end of December 2021). There was another section which I decided to add to the book purely for fun: a playlist of music to which I’d listened and which I’d enjoyed over the months it took to write. I listen to a lot of music, and the list is very far from a complete one, but it does represent a fair cross-section of my general listening tastes. Here’s the list, with a few words about each one.

One thing that’s missing is any of the classical music that I listen to. I decided against including this, as I’d rarely choose single tracks, but adding full albums seemed to miss the point. I do listen to lots of classical music, in particular sacred choral and organ music – happy to let people have some suggestions if they’d like.

  • Secret Messages – ELO – I just had to have something related (or that could be considered to be related) to cryptography and security. This song isn’t, really, but it’s a good song, and I like it.
  • Bleed to Love Her – Fleetwood Mac – Choosing just one Fleetwood Mac song was a challenge, but I settled on this one. I particularly like the harmonics in the version recorded live at Warner Brothers Studio in Burbank.
  • Alone in Kyoto – Air – This is a song that I put on when I want to relax. Chiiiiilllll.
  • She’s So Lovely – Scouting for Girls – Canonically, this song is known as “She’s A Lovely” in our family, as that’s what we discovered our daughters singing along to when we played it in the car many years ago.
  • Prime – Shearwater – This is much more of an “up” song when I want to get an edge on. Shearwater have a broad range of output, but this is particular favourite.
  • Stay – Gabrielle Aplin – I like the way this song flips expectations on its head. A great song by a talented artist.
  • The Way I Feel – Keane – A song about mental health.
  • Come On, Dreamer – Tom Adams – Adams has an amazing voice, and this is a haunting song about hope.
  • Congregation – Low – I discovered this song watching DEVS on Amazon Prime (it was originally on Hulu). Low write (and perform) some astonishing songs, and it’s really worth going through their discography if you like this one.
  • Go! – Public Service Broadcasting – You either love this or hate it, but I’m in the “love” camp. It takes original audio from the Apollo 11 moon landing and puts it to energising, exciting music.
  • The Son of Flynn (From “TRON: Legacy”/Score) – Daft Punk – TRON:Legacy may not be not the best film ever released, but the soundtrack from Daft Punk is outstanding Electronica.
  • Lilo – The Japanese House – A song about loss? About hope? Another one to chill to (and tha band are great live, too).
  • Scooby Snacks – Fun Lovin’ Criminals – Warning: explicit lyrics (from the very beginning!) A ridiculous song which makes me smile every time I listen to it.
  • My Own Worth Enemy – Stereophonics – I slightly surprised myself by choosing this song from the Stereophonics, as I love so many of their songs, but it really does represent much of what I love about their oeuvre.
  • All Night – Parov Stelar – If you ever needed a song to dance to as if nobody’s watching, this is the one.
  • Long Tall Sally (The Thing) – Little Richard – Sometimes you need some classic Rock ‘n’ Roll in your life, and who better to provide it?
  • Shart Dressed Man – ZZ Top – “Black tie…” An all-time classic by men with beards. Mostly.
  • Dueling Banjos – Eric Weissberg – I first heard this song at university. It still calls out to me. There are some good versions out there, but original from the songtrack to Deliverance is the canonical one. And what a film.
  • The Starship Avalon (Main Title) – Thomas Newman – This (with some of the others above) is on a playlist I have called “Architecting”, designed to get me in the zone. Another great film.
  • A Change is Gonna Come – Sam Cooke – A song of sadness, pain and hope.
  • This Place – Jamie Webster – A song about Liverpool, and a family favourite. Listen and enjoy (the accent and the song!).

If you’d like to listen to these tracks yourself, I’ve made playlists on my two preferred audio streaming sites: I hope you enjoy.

Spotify – Trust in Computer Systems and the Cloud – Bursell

Qobuz – Trust in Computer Systems and the Cloud – Bursell

As always, I love to get feedback from readers – do let me know what you think, or suggest other tracks or artists I or other readers might appreciate.

10(+1) plans for 2022

I’m not a big fan of New Year’s resolutions, as I don’t like to set myself up to fail.

This week’s song: Bleed to Love Her by Fleetwood Mac.

I’m not a big fan of New Year’s resolutions, as I don’t like to set myself up to fail. Instead, here are a few things – professional and personal – that I hope or expect to be doing this year. Call them resolutions if you want, but words have power, and I’m avoiding the opportunity

  1. Spend lots of time shepherding Enarx to greater maturity. At Profian, we see our future as closely ties to that of Enarx, and we’ll be growing the project’s capabilities and functionality significantly over this year. Keep an eye out for announcements!
  2. Get fit(ter) again. Yeah, that.
  3. Promote my book. I’m really proud of my book Trust in Computer Systems and the Cloud, which was published right at the end of the year. It aims to raise the standard of knowledge within the industry by proposing a framework for discussion, and I want to make that happen.
  4. Start travelling again. I miss conferences, I miss seeing colleagues, I miss meeting new people. Hopefully it’s going to be easier and safer to travel this year.
  5. Delegate better (and more). As the CEO of a startup, there’s lots I need to make happen. I’m not always the best person actually to be doing it all, and learning to help other people take some (more!) of it over is actually really important not just dot me, but for the business.
  6. Drink lots of tea. No real change here.
  7. Drjnk good whisky. In moderation.
  8. Keep gaming. Possibly a weird one, but gaming is an important downtime activity for me, and helps me relax.
  9. Make the most of music. I listen to lots of music whilst working, travelling, driving, relaxing, etc.. Watch out for a link to the playlist associated with my book – I also plan to list a song or track a week on my blog (see the top of this article for this week’s offering!).
  10. Enjoy reading. One of the benefits of having completed the book is that I now have more time to read; more specifically, more time when I don’t feel guilty that I’m reading rather than doing book-work.
  11. A bonus one: spend more time over at Opensource.com. I’m a Correspondent over there, and enjoy both writing for them and reading other people’s contributions. A great way to get into – or keep up-to-date with – the open source community.

So – not the most inspiring list, but if I can manage most of these this year, I’ll be happy.

Open source Christmas presents

Give the gift of open source to more people.

If you find this post interesting, you’ll find a lot more about how community and open source are important in my book Trust in Computer Systems and the Cloud, published by Wiley.

Whether you celebrate Christmas or not (our family does, as it happens), this time of year is one where presents are often given and received. I thought it might be nice to think about what presents we could give in the spirit of open source. Now, there are lots of open source projects out there, and you could always use one to create something for a friend, colleague or loved one (video, audio, blog post, image, website) or go deeper with a project which combines open source software and hardware, such as Mycroft or Crowdsupply. Or you could go in the other direction, and get people involved in projects you’re part of or enjoy. That’s what I’d like to suggest in this article: give the gift of open source to more people, or just make open source more accessible to more people: that’s a gift in itself (to them and to the project!).

Invite

First of all, people need to know about projects. “Evangelism” is a word that’s often used around open source projects, because people need to be told about them before they can get involved. Everyone can do evangelism, whether it’s word of mouth, laptop stickers, blog posts, videos, speaking at conferences, LinkedIn mentions, podcasts, Slack, IRC, TikTok[1], Twitter, ICQ[2] or Reddit. Whatever is your preferred medium to talk to the world, use it. Tell people why it’s important. Tell people why it’s fun. Share the social side of the project. Explain some of the tricky design issues that face it. Tell people why it’s written in the language(s) it’s in. Point people at the sections of code you’ve written and are proud of. Even better, point people at the sections of code you’ve written and are ashamed of, but don’t have time to fix as you’re too busy at the moment. But most of all, invite them to look around, meet the contributors, read the code, test the executables, read the documentation. Make it easy for them to find the project. Once we get back to a world where in-person conferences are re-emerging, arrange meet-ups, provide swag and get together (safely!) IRL[3].

Include

Once your invitees have started looking around, interacting with the community, submitting issues, documentation or patches, find ways to include them. There’s nothing more alienating than, well, being alienated. I think the very worst thing anyone can say to a person new to a project is something along the lines of “go and read the documentation – this is a ridiculous question/terrible piece of documentation/truly horrible piece of code”. It may be all of those things, but how does that help anyone? If you find people giving these reactions – if you find yourself giving these reactions – you need to sort it out. Everyone was a n00b once, and everyone has a different learning style, way of interacting, cultural background and level of expertise. If there are concerns that senior project members’ time is being “wasted” by interactions, nominate (and agree!) that someone will take time to mentor newcomers. Better yet, take turns mentoring, so that information and expertise is spread widely and experts in the project get to see the questions and concerns that non-experts are having. There are limits to this, of course, but you need to find ways not just to welcome people into the project, but actually include them in the functioning, processes, social interactions and day-to-day working of the project which make it a community.

You should also strongly consider a code of conduct such as the Contributor Covenant to model, encourage and, if necessary enforce appropriate and inclusive behaviour. Diversity and Inclusion are complex topics, but there’s a wealth of material out there if you want to take engage – and you should.

Encourage

Encouragement is a little different to inclusion. It’s possible to feel part of a community, but not actually to be participating to the development and growth of the project. Encouragement may be what people need to move into active engagement, contributing more than lurking. And there’s a difference between avoiding negative comments (as outlined above) and promoting positive interactions. The former discourage, and the latter can encourage. If someone contributes their first patch, and gets an “accepted, merged” message, that’s great, but it’s pretty clear that they’re much more likely to contribute again if, instead, they receive a message along the lines of “thanks for this: great to see. We need more contributions in this area: have you looked at issues #452, #599 and #1023?”.

These sorts of interactions are time-consuming, and it may not always be the maintainers who are providing them: as above, the project may need to have someone whose role includes this sort of encouragement. If you’re using something like Github, you may be able to automate notifications of first-time contributions so that you know that it’s time to send an encouraging message. The same could go for someone who was making a few contributions, but has slowed down or dropped off: a quick message or two might be enough to get them involved in the project again.

Celebrate

I see celebration as as step on again from simple encouragement – though it can certainly reinforce it. Celebration isn’t just about acknowledging something positive, but is also a broader social interaction. When somebody’s achievements are celebrated, other people in the community come together to say well done and congratulate them. This is great for the person whose work is being celebrated, as the acknowledgement from others reinforces the network of people with whom they’re connected, bringing them closer into the community.

Celebrating a project-related event like a release and including new members of the community in that celebration can be even more powerful. When new members are part of a celebration, and are made to feel that their contributions, though small, have made up part of what’s being celebrated, their engagement in the project is likely to increase. Their feelings of inclusion in the community are also likely to go up. Celebrations in person (again, when possible) allow for better network-building and closer ties, but even virtual meet-ups can bring peripherally-involved or new members closer to the core of the project.

Summary

Getting people involved in your open source project is important for its health and its growth, but telling people about it isn’t enough. You need to take conscious steps to increase involvement and ensure that initial contributions to a project are followed up, tying people into the project and making them part of the community.


1 – I’m going to be honest: I wouldn’t know where to start with TikTok. My kids will probably be appalled that I even mentioned it, but hey, why not? The chances are that you, dear reader, are younger and (almost certainly) cooler than I am.

2 – I’m guessing the take up will be a bit lower here.

3 – In Real Life. It seems odd to be re-using this term, which had all but disappeared from what I could tell, but which seems to need to re-popularised.

“Trust in Computer Systems and the Cloud” published

I’ll probably have a glass or two of something tonight.

It’s official: my book is now published and available in the US! What’s more, my author copies have arrived, so I’ve actually got physical copies that I can hold in my hand.

You can buy the book at Wiley’s site here, and pre-order with Amazon (the US site lists is as “currently unavailable”, and the UK site lists is as available from 22nd Feb, 2022. ,though hopefully it’ll be a little earlier than that). Other bookstores are also stocking it.

I’m over the moon: it’s been a long slog, and I’d like to acknowledge not only those I mentioned in last week’s post (Who gets acknowledged?), but everybody else. Particularly, at this point, everyone at Wiley, calling out specifically Jim Minatel, my commissioning editor. I’m currently basking in the glow of something completed before getting back to my actual job, as CEO of Profian. I’ll probably have a glass or two of something tonight. In the meantime, here’s a quote from Bruce Schneier to get you thinking about reading the book.

Trust is a complex and important concept in network security. Bursell neatly unpacks it in this detailed and readable book.

Bruce Schneier, author of Liars and Outliers: Enabling the Trust that Society Needs to Thrive

At least you know what to buy your techy friends for Christmas!

Who gets acknowledged?

Some of the less obvious folks who get a mention in my book, and why

After last week’s post, noting that my book was likely to be delayed, it turns out that it may be available sooner than I’d thought. Those of you in the US should be able to get hold of a copy first – possibly sooner than I do. The rest of the world should have availability soon after. While you’re all waiting for your copy, however, I thought it might be fun for me to reveal a little about the acknowledgements: specifically, some of the less obvious folks who get a mention, and why they get a mention.

So, without further ado, here’s a list of some of them:

  • David Braben – in September 1984, not long after my 14th birthday, the game Elite came out on the BBC micro. I was hooked, playing for as long and as often as I was allowed (which wasn’t as much as I would have liked, as we had no monitor, and I had to hook the BBC up to the family TV). I first had the game on cassette, and then convinced my parents that a (5.25″) floppy drive would be a good educational investment for me, thereby giving me the ability to play the extended (and much quicker loading) version of the game. Fast forward to now, and I’m still playing the game which, though it has changed and expanded in many ways, is still recognisably the same one that came out 37 years ago. David Braben was the initial author, and still runs the company (Frontier Developments) which creates, runs and supports the game. Elite excited me, back in the 80s, with what computers could do, leading me to look into wireframes, animation and graphics.
  • Richard D’Silva – Richard was the “head of computers” at the school I attended from 1984-1989. He encouraged me (and many others) to learn what computers could do, all the way up to learning Pascal and Assembly language to supplement the (excellent) BASIC available BBC Bs and BBC Masters which the school had (and, latterly, some RISC machines). There was a basic network, too, an “Econet”, and this brought me to initial research into security – mainly as a few of us tried (generally unsuccessfully) to access machines and accounts were weren’t supposed to.
  • William Gibson – Gibson wrote Neuromancer – and then many other novels (and short stories) – in the cyberpunk genre. His vision of engagement with technology – always flawed, often leading to disaster, has yielded some of the most exciting and memorable situations and characters in scifi (Molly, we love you!).
  • Nick Harkaway – I met Nick Cornwell (who writes as Nick Harkaway) at the university Jiu Jitsu club, but he became a firm friend beyond that. Always a little wacky, interested maybe more about the social impacts of technology than tech for tech’s sake (more my style then), he always had lots of interesting opinions to share. When he started writing, his wackiness and thoughtfulness around how technology shapes us informed his fiction (and non-fiction). If you haven’t read The Gone-Away World, order it now (and read it after you’ve finished my book!).
  • Anne McCaffrey – while I’m not an enormous fan of fantasy fiction (and why do scifi and fantasy always seem to be combined in the same section in bookshops?), Anne McCaffrey’s work was a staple in my teenage years. I devoured her DragonRiders of Pern series and also enjoyed her (scifi) Talents series as that emerged. One of the defining characteristics of her books was always strong female characters – a refreshing change for a genre which, at the time, seemed dominated by male protagonists. McCaffrey also got me writing fiction – at one point, my school report in English advised that “Michael has probably written enough science fiction for now”.
  • Mrs Macquarrie (Jenny) – Mrs Macquarrie was my Maths teacher from around 1978-1984. She was a redoubtable Scot, known to the wider world as wife of the eminent theologian John Macquarrie, but, in the universe of the boarding school I attended, she was the strict but fair teacher who not only gave me a good underpinning in Maths, but also provided a “computer club” at the weekends (for those of us who were boarding), with her ZX Spectrum.
  • Sid Meier – I’ve played most of the “Sid Meier’s Civilization” (sic) series of games over the past several decades(!) from the first, released in 1991. In my university years, there would be late night sessions with a bunch of us grouped around the monitor, eating snacks and drinking whatever we could afford. These days, having a game running on a different monitor can still be rewarding when there’s a boring meeting you have to attend…
  • Bishop Nick – this one is a trick, and shouldn’t really appear in this section, as Bishop Nick isn’t a person, but a local brewery. They brew some great beers, however, including “Heresy” and “Divine”. Strongly recommended if you’re in the Northeast Essex/West Suffolk area.
  • Melissa Scott – Scott’s work probably took the place of McCaffrey’s as my reading tastes matured. Night Sky Mine, Trouble and her Friends and The Jazz provided complex and nuanced futures, again with strong female protagonists. The queer undercurrents in her books – most if not all of her books have some connection with queer themes and cultures – were for me introduction to a different viewpoint on writing and sexuality in “popular” fiction, beyond the more obvious and “worthy” literary treatments with which I was already fairly familiar.
  • Neal Stephenson – Nick Harkaway/Cornwell (see above) introduced me to Snowcrash when it first came out, and I managed to get a UK trade paperback copy. Stephenson’s view of a cyberpunk future, different from Gibson’s and full of linguistic and cultural craziness, hooked me, and I’ve devoured all of his work since. You can’t lose with Snowcrash, but my other favourite of his is Cryptonomicon, a book which zig-zags between present day (well, early 2000s, probably) and the Second World War, embracing cryptography, religion, computing, gold, civil engineering and start-up culture. It’s on the list of books I suggest for anyone considering getting into security because the mindset shown by a couple of the characters really nails what it’s all about.

There are more people mentioned, but these are the ones most far removed either in time from or direct relevance to the writing of the book. I’ll leave those more directly involved, or just a little more random, for you to discover as you read.

Don’t forget: if you follow this blog, you’re in for a chance to win a free copy of the Trust in Computer Systems and the Cloud!

Book delay

(You can still win a free copy)

I’m sorry to have to announce that the availability of my book, Trust in Computrer Systems and the Cloud, is likely to be delayed. Wiley, my publisher, had hoped to get copies in the US for early December, and to Europe a month or so after that, but problems getting hold of paper (a core component of physical books, for the uninitiated) mean that these dates will be delayed.

I’m obviously disappointed about this, but it’s really not Wiley’s fault (the paper shortage is wide-spread across the US, it appears). Travel rules permitting, I intend to attend the RSA Conference in San Francisco in February 2022, and we hope to have copies of the book available there (book your signed copy now[1]).

Anyway, sorry to announce this, but it does give you more time to follow this blog, giving you a chance of a free copy when they are available.


1 – I will, actually, sign[2] your copy if you like: do feel free to contact me!

2 – I’m hoping we don’t get to the stage where, as in the film[3] Notting Hill, unsigned copies are worth more than signed ones!

3 – yeah, yeah, “movie” if you must.

Image by Peggychoucair from Pixabay

Cloud security asymmetry

We in the security world have to make people understand this issue.

My book, Trust in Computer Systems and the Cloud, is due out in the next few weeks, and I was wondering as I walked the dogs today (a key part of the day for thinking!) what the most important message in the book is. I did a bit of thinking and a bit of searching, and decided that the following two paragraphs expose the core thesis of the book. I’ll quote them below and then explain briefly why (the long explanation would require me to post most of the book here!). The paragraph is italicised in the book.

A CSP [Cloud Service Provider] can have computational assurances that a tenant’s workloads cannot affect its hosts’ normal operation, but no such computational assurances are available to a tenant that a CSP’s hosts will not affect their workloads’ normal operation.

In other words, the tenant has to rely on commercial relationships for trust establishment, whereas the CSP can rely on both commercial relationships and computational techniques. Worse yet, the tenant has no way to monitor the actions of the CSP and its host machines to establish whether the confidentiality of its workloads has been compromised (though integrity compromise may be detectable in some situations): so even the “trust, but verify” approach is not available to them.”

What does this mean? There is, in cloud computing, a fundamental asymmetry: CSPs can protect themselves from you (their customer), but you can’t protect yourself from them.

Without Confidential Computing – the use of Trusted Execution Environments to protect your workloads – there are no technical measures that you can take which will stop Cloud Service Providers from looking into and/or altering not only your application, but also the data it is processing, storing and transmitting. CSPs can stop you from doing the same to them using standard virtualisation techniques, but those techniques provide you with no protection from a malicious or compromised host, or a malicious or compromised CSP.

I attended a conference recently attended by lots of people whose job it is to manage and process data for their customers. Many of them do so in the public cloud. And a scary number of them did not understand that all of this data is vulnerable, and that the only assurances they have are commercial and process-based.

We in the security world have to make people understand this issue, and realise that if they are looking after our data, they need to find ways to protect it with strong technical controls. These controls are few:

  • architectural: never deploy sensitive data to the public cloud, ever.
  • HSMs: use Hardware Security Modules. These are expensive, difficult to use and don’t scale, but they are appropriate for some sensitive data.
  • Confidential Computing: use Trusted Execution Environments (TEEs) to protect data and applications in use[1].

Given my interest – and my drive to write and publish my book – it will probably come as no surprise that this is something I care about: I’m co-founder of the Enarx Project (an open source Confidential Computing project) and co-founder and CEO of Profian (a start-up based on Enarx). But I’m not alone: the industry is waking up to the issue, and you can find lots more about the subject at the Confidential Computing Consortium‘s website (including a list of members of the consortium). If this matters to you – and if you’re an enterprise company who uses the cloud, it almost certainly already does, or will do so – then please do your research and consider joining as well. And my book is available for pre-order!

Logs – good or bad for Confidential Computing?

I wrote a simple workload for testing. It didn’t work.

A few weeks ago, we had a conversation on one of the Enarx calls about logging. We’re at the stage now (excitingly!) where people can write applications and run them using Enarx, in an unprotected Keep, or in an SEV or SGX Keep. This is great, and almost as soon as we got to this stage, I wrote a simple workload to test it all.

It didn’t work.

This is to be expected. First, I’m really not that good a software engineer, but also, software is buggy, and this was our very first release. Everyone expects bugs, and it appeared that I’d found one. My problem was tracing where the issue lay, and whether it was in my code, or the Enarx code. I was able to rule out some possibilities by trying the application in an unprotected (“plain KVM”) Keep, and I also discovered that it ran under SEV, but not SGX. It seemed, then, that the problem might be SGX-specific. But what could I do to look any closer? Well, with very little logging available from within a Keep, there was little I could do.

Which is good. And bad.

It’s good because one of the major points about using Confidential Computing (Enarx is a Confidential Computing framework) is that you don’t want to leak information to untrusted parties. Since logs and error messages can leak lots and lots of information, you want to restrict what’s made available, and to whom. Safe operation dictates that you should make as little information available as you possibly can: preferably none.

It’s bad because there are times when (like me) you need to work out what’s gone wrong, and find out whether it’s in your code or the environment that you’re running your application in.

This is where the conversation about logging came in. We’d started talking about it before this issue came up, but this made me realise how important it was. I started writing a short blog post about it, and then stopped when I realised that there are some really complex issues to consider. That’s why this article doesn’t go into them in depth: you can find a much more detailed discussion over on the Enarx blog. But I’m not going to leave you hanging: below, you’ll find the final paragraph of the Enarx blog article. I hope it piques your interest enough to go and find out more.

In a standard cloud deployment, there is little incentive to consider strong security controls around logging and debugging, simply because the host has access not only to all communications to and from a hosted workload, but also to all the code and data associated with the workload at runtime.  For Confidential Computing workloads, the situation is very different, and designers and architects of the TEE infrastructure (e.g. the Enarx projects) and even, to a lesser extent, of potential workloads themselves, need to consider very carefully the impact of host gaining access to messages associated with the workload and the infrastructure components.  It is, realistically, infeasible to restrict all communication to levels appropriate for deployment, so it is recommended that various profiles are created which can be applied to different stages of a deployment, and whose use is carefully monitored, logged (!) and controlled by process.


Header image by philm1310 from Pixabay.

Recruiting is hard

It’s going to be easier to outsource this work to somebody who is more of an expert than I’ll ever be, would ever want to be, or could ever be.

We (Profian) are currently looking to recruit some software engineers. Now, I’ve been involved in hiring people before – on the interviewing side, at least – but actually doing the recruiting is a completely new experience for me. And it’s difficult. As the CEO of a start-up, however, it turns out that it’s pretty much down to me to manage the process, from identifying the right sort of person, to writing a job advert (see above), to finding places to place it, to short-listing candidates, interviewing them and then introducing them to the rest of the team. Not to mention agreeing a start date, “compensation package” (how much they get paid) and all that. Then there’s the process of on-boarding them (getting contracts sorted, getting them email addresses, etc.), and least some of which I’m pleased to say I have some help with.

The actual recruiting stuff is difficult, though. Recruitment consultants get a bad rap, and there are some dodgy ones, but I’m sure most of them are doing the best they can and are honest people. You might even be happy to introduce some of them to your family. Just a few. But, like so many other things about being start-up founder, it turns out that there comes a time when you have to say to yourself: “well, I could probably learn to do this – maybe not well, but with some degree of competence – but it’s just not worth my time. It’s going to be easier, and actually cheaper in the long run, to outsource this work to somebody who is, frankly, more of an expert than I’ll ever be, would ever want to be, or could ever be. And so I’ve found someone to work with.

What’s really interesting when you find somebody to help you with a new task is the time it takes to mesh your two worlds. I’m a software guy, a we’re looking for software people. I need to explain to the recruitment consultant not only what skills we’re looking for, but what phrases, when they appear on a LinkedIn page or CV[1], are actually red flags. In terms of phrases we’re looking for (or are nice to haves), I’d already mentioned “open source” to the recruitment consultant, but it was only on looking over some possible candidates that I realised that “FOSS” should be in there, too. A person whose current role is “Tech lead” is much more likely to be a fit than “Technical manager”. What’s the difference between a “cloud architect” and a “systems architect”? Is “Assembly” different to “WebAssembly” (yes! – oh, and the latter is sometimes shortened to “Wasm”).

There are, of course, recruitment consultants who specialise in particular technical fields, but what we’re doing (see the Enarx project) is so specialised and so new that I really don’t think that there are likely to be any specialist recruiters anywhere in the world (yet).

So, I feel lucky that I’ve managed to find someone who seems to get not only where we’re coming from as a company, but also the sorts of people we’re looking for. He wisely suggested that we spend some time going over some possible candidates so he could watch me identifying people who were a definite “no” – as useful for him as a definite “must interview”. Hopefully we’ll start to find some really strong candidates soon. If you think you might be one of them, please get in touch!

(Oh – and yes, I’ve invited him to meet my family.)


1 – that’s “resume” for our US friends.