Category: expertise

How to hire an open source developer

Our view was that a pure “algorithm coding” exercise was pretty much useless for what we wanted.

We’ve recently been hiring developers to work on the Enarx project, a security project, written almost exclusively in Rust (with a bit of Assembly), dealing with Confidential Computing. By “we”, I mean Profian, the start-up for which I’m the CEO and co-founder. We’ve now found all the people we’re looking for initially on the team (with a couple due to start in the next few weeks), though we absolutely welcome contributors to Enarx, and, if things continue to go well, we’ll definitely want to hire some more folks in the future.

Hiring people is not easy, and we were hit with a set of interesting requirements which made the task even more difficult. I thought it would be useful and interesting for the community to share how we approached the problem.

What were we looking for?

I mentioned above some interesting requirements. Here’s what the main ones were:

  • systems programming – we mainly need people who are happy programming at the systems layer. This is pretty far down the stack, with lots of interactions directly with hardware or the OS. Where we are creating client-server pieces, for instance, we’re having to write quite a lot of the protocols, manage the crypto, etc., and the tools we’re using aren’t all very mature (see “Rust” below).
  • Rust – almost all of the project is written in Rust, and what isn’t is written in Assembly language (currently exclusively x86, though that may change as we add more platforms). Rust is new, cool and exciting, but it’s still quite young, and some areas don’t have all the support you might like, or aren’t as mature as you might hope – everything from cryptography through multi-threading libraries and compiler/build infrastructure.
  • distributed team – we’re building a team of folks where can find them: we have developers in Germany, Finland, the Netherlands, North Carolina (US), Massachusetts (US), Virginia (US) and Georgia (US), I’m in the UK, our community manager is in Brazil and we have interns in India and Nigeria. We knew from the beginning that we wouldn’t have everyone in one place, and this required people who we were happy would be able to communicate and collaborate with people via video, chat and (at worst) email.
  • security – Enarx is a security project, and although we weren’t specifically looking for security experts, we do need people who are able to think and work with security top of mind, and design and write code which is applicable and appropriate for the environment.
  • git – all of our code is stored in git (mainly GitHub, with a little bit of GitLab thrown in), and so much of our interaction around code revolves around git that anybody joining us would need to be very comfortable using it as a standard tool in their day-to-day work.
  • open source – open source isn’t just a licence, it’s a mindset, and, equally important, a way of collaborating. A great deal of open source software is created by people who aren’t geographically co-located, and who might not even see themselves as a team. We needed to be sure that the people we were hiring, while gelling as a close team within the company, will also be able to collaborate with people outside the organisation and be able to embrace Profian’s “open by default” culture not just for code, but for discussions, communications and documentation.

How did we find them?

As I’ve mentioned before, in Recruiting is hard. We ended up using a variety of means to find candidates, with varying levels of success:

  • LinkedIn job adverts
  • LinkedIn searches
  • Language-specific discussion boards and hiring boards (e.g. Reddit)
  • An external recruiter (shout out to Gerald at Interstem)
  • Word-of-mouth/personal recommendations

It’s difficult to judge between them in terms of quality, but without an external recruiter, we’d certainly have struggled with quantity (and we had some great candidates from that pathway, too).

How did we select them?

We needed to measure all of the candidates against all of the requirements noted above, but not all of them were equal. For instance, although we were keen to hire Rust programmers, we were pretty sure that someone with strong C/C++ skills at the systems level would be able to pick up Rust quickly enough to be useful. On the other hand, a good knowledge of using git was absolutely vital, as we couldn’t spend time working with new team members to bring them up-to-speed on our way of working. A strong open source background was, possibly surprisingly, not a requirement, but the mindset to work in that sort of model was, and anyone with a history of open source involvement is likely to have a good knowledge of git. The same goes for the ability to work in a distributed team: so much of open source is distributed that involvement in almost any open source community was a positive indicator. Security we decided was a “nice-to-have”.

How to proceed? We wanted to keep the process simple and quick – we don’t have a dedicated HR or People function, and we’re busy trying to get code written. What we ended up was this (with slight variations), which we tried to get complete within 1-2 weeks:

  1. Initial CV/resume/github/gitlab/LinkedIn review – this to decide whether to interview
  2. 30-40 minute discussion with me as CEO, to find out if they might be a good cultural fit, to give them a chance to find out about us, and get an idea if they were as technically adept as they appeared from the first step
  3. Deep dive technical discussion led by Nathaniel, usually with me there
  4. Chat with other members of the team
  5. Coding exercise
  6. Quick decision (usually within 24 hours)

The coding exercise was key, but we decided against the usual approach. Our view was that a pure “algorithm coding” exercise of the type so beloved by many tech companies was pretty much useless for what we wanted. What we wanted to understand was whether candidates could quickly understand a piece of code, fix some problems and work with the team to do so. We created a github repository (in fact, we ended up using two – one for people a little higher up the stack) with some almost-working Rust code in it, some instructions to fix it, perform some git-related processes on it, and then improve it slightly, adding tests along the way. A very important part of the test was to get candidates to interact with the team via our chat room(s). We scheduled 15 minutes on a video call for set up and initial questions, 2 hours for the exercise (“open book” – as well as talking to the team, candidates were encouraged to use all resources available to them on the Internet), followed by a 30 minute wrap-up session where the team could ask questions and the candidate could also reflect on the task. This also allowed us to get an idea of how well the candidate was able to communicate with the team (combined with the chat interactions during the exercise). Afterwards, the candidate would drop off the call, and we’d generally make a decision within 5-10 minutes as to whether we wanted to hire them.

This generally worked very well. Some candidates struggled with the task, some didn’t communicate well, some failed to do well with the git interactions – these were the people we didn’t hire. It doesn’t mean they’re not good coders, or that they might not be a good fit for the project or the company later on, but they didn’t immediate meet the criteria we need now. Of the ones we hired, the levels of Rust experience and need for interaction with the team varied, but the level of git expertise and their reactions to our discussions afterwards was always sufficient for us to decide to take them.

Reflections

On the whole, I don’t think we’d change a huge amount about the selection process – though I’m pretty sure we could do better with the search process. The route through to the coding exercise allowed us to filter out quite a few candidates, and the coding exercise did a great job of helping us pick the right people. Hopefully everyone who’s come through the process will be a great fit and will produce great code (and tests and documentation and …) for the project. Time will tell!

Saving one life

Scratching the surface of the technologies which led to the saving of a life

When a loved one calls you from the bathroom at 3.30 in the morning, and you find them collapsed, unconscious on the floor, what does technology do for you? I’ve had the opportunity to consider this over the past few days after a family member was rushed to hospital for an emergency operation which, I’m very pleased to say, seems to have been completely successful. Without it, or if it had failed (the success rate is around 50%), they would, quite simply, be dead now.

We are eternally grateful to all those directly involved in my family member’s care, and to the NHS, which means that there are no bills to pay, just continued National Insurance taken as tax from our monthly pay packets, and which we begrudge not one jot. But I thought it might be worth spending a few minutes just scratching the surface of the sets of technologies which led to the saving of a life, from the obvious to the less obvious. I have missed out many: our lives are so complex and interconnected that it is impossible to list everything, and it is only when they are missing that we realise how it all fits together. But I want to say a huge – a HUGE – thank you to anyone who has ever been involved in any of the systems or technologies, and to ask you to remind yourself that even if you are seldom thanked, your work saves lives every day.

The obvious

  • The combined ECG and blood pressure unit attached to the patient which allows the ambulance crew to react quickly enough to save the patient’s life
  • The satellite navigation systems which guided the crew to the patient’s door
  • The landline which allowed the call to the emergency systems
  • The triage and dispatch system which prioritised the sending of the crew
  • The mobile phone system which allowed a remote member of the family to talk to the crew before they transported the patient

The visible (and audible)

  • The anaesthesiology and monitoring equipment which kept the patient alive during the operation
  • The various scanning equipment at the hospital which allowed a diagnosis to be reached in time
  • The sirens and flashing lights on the ambulances
  • The technology behind the training (increasingly delivered at least partly online) for all of those involved in the patient’s care

The invisible

  • The drugs and medicines used in the patient’s care
  • Equipment: batteries for ambulances, scalpels for operating theatres, paper for charts, keyboards, CPUs and motherboards for computers, soles for shoes, soap for hand-washing, paint for hospital corridors, pillows and pillow cases for beds and everything else that allows the healthcare system to keep running
  • The infrastructure to get fuel to the ambulances and into the cars, trains and buses which transported the medical staff to hospital
  • The maintenance schedules and processes for the ambulances
  • The processes behind the ordering of PPE for all involved
  • The supply chains which allowed those involved to access the tea, coffee, milk, sugar and other (hopefully legal) stimulants to keep staff going through the day and night
  • Staff timetabling software for everyone from cleaners to theatre managers, maintenance people to on-call surgeons
  • The music, art, videos, TV shows and other entertainment that kept everyone involved sufficiently energised to function

The infrastructure

  • Clean water
  • Roads
  • Electricity
  • Internet access and routing
  • Safety processes and culture in healthcare
  • … and everything else I’ve neglected to mention.

A final note

I hope it’s clear that I’m aware that the technology is all interconnected, and too complex to allow every piece to be noted: I’m sorry if I missed your piece out. The same, however, goes for the people. I come from a family containing some medical professionals and volunteers, and I’m aware of the sacrifices made not only by them, but also by the people around them who they know and love, and who see less of them than they might like, or how have to work around difficult shift patterns, or see them come back home after a long shift, worn out or traumatised by what they’ve seen and experienced. The same goes for ancillary workers and services worked in other, supporting industries.

I thank you all, both those involved directly and those involved in any of the technologies which save lives, those I’ve noted and those I’ve missed. In a few days, I hope to see a member of my family who, without your involvement, I would not ever be seeing again in this life. That is down to you.

Coming to you in Japanese

We are now multi-lingual.

I have an exciting announcement, which is that starting this week, some of the articles on this blog will also be in Japanese.  My very talented Red Hat colleague Yuki Kubota showed an interest in translating some which she thought might be of interest to Japanese readers, and I jumped at the chance.  I’m very thrilled and humbled.

We’re still ironing out the process, but hopefully (if you already read Japanese), you’ll be able to read the following articles.

We’ll try to add the tag “Japanese” to each of these, as well.

So, a huge thank you to Yuki: we’d love comments – in English or Japanese!

5 (Professional) development tips for security folks

… write a review of “Sneakers” or “Hackers”…

To my wife’s surprise[1], I’m a manager these days.  I only have one report, true, but he hasn’t quit[2], so I assume that I’ve not messed this management thing up completely[2].  One of the “joys” of management is that you get to perform performance and development (“P&D”) reviews, and it’s that time of year at the wonderful Red Hat (my employer).  In my department, we’re being encouraged (Red Hat generally isn’t in favour of actually forcing people to do things) to move to “OKRs”, which are “Objectives and Key Results”.  Like any management tool, they’re imperfect, but they’re better than some.  You’re supposed to choose a small number of objectives (“learn a (specific) new language”), and then have some key results for each objective that can be measured somehow (“be able to check into a hotel”, “be able to order a round of drinks”) after a period of time (“by the end of the quarter”).  I’m simplifying slightly, but that’s the general idea.

Anyway, I sometimes get asked by people looking to move into security for pointers to how to get into the field.  My background and route to where I am is fairly atypical, so I’m very sensitive to the fact that some people won’t have taken Computer Science at university or college, and may be pursuing alternative tracks into the profession[3].  As a service to those, here are a few suggestions as to what they can do which take a more “OKR” approach than I provided in my previous article Getting started in IT security – an in/outsider’s view.

1. Learn a new language

And do it with security in mind.  I’m not going to be horribly prescriptive about this: although there’s a lot to be said for languages which are aimed a security use cases (Rust is an obvious example), learning any new programming language, and thinking about how it handles (or fails to handle) security is going to benefit you.  You’re going to want to choose key results that:

  • show that you understand what’s going on with key language constructs to do with security;
  • show that you understand some of what the advantages and disadvantages of the language;
  • (advanced) show how to misuse the language (so that you can spot similar mistakes in future).

2. Learn a new language (2)

This isn’t a typo.  This time, I mean learn about how other functions within your organisations talk.  All of these are useful:

  • risk and compliance
  • legal (contracts)
  • legal (Intellectual Property Rights)
  • marketing
  • strategy
  • human resources
  • sales
  • development
  • testing
  • UX (User Experience)
  • IT
  • workplace services

Who am I kidding?  They’re all useful.  You’re learning somebody else’s mode of thinking, what matters to them, and what makes them tick.  Next time you design something, make a decision which touches on their world, or consider installing a new app, you’ll have another point of view to consider, and that’s got to be good.  Key results might include:

  • giving a 15 minute presentation to the group about your work;
  • arranging a 15 minute presentation to your group about the other group’s work;
  • (advanced) giving a 15 minute presentation yourself to your group about the other group’s work.

3. Learning more about cryptography

So much of what we do as security people comes down to or includes some cryptography.  Understanding how it should be used is important, but equally, being able to understand how it shouldn’t be used is something we should all understand.  Most important, from my point of view, however, is to know the limits of your knowledge, and to be wise enough to call in a real cryptographic expert when you’re approaching those limits.  Different people’s interests and abilities (in mathematics, apart from anything else) vary widely, so here is a broad list of different possible key results to consider:

  • learn when to use asymmetric cryptography, and when to use symmetric cryptography;
  • understand the basics of public key infrastructure (PKI);
  • understand what one-way functions are, and why they’re important;
  • understand the mathematics behind public key cryptography;
  • understand the various expiry and revocation options for certificates, their advantages and disadvantages.
  • (advanced) design a protocol using cryptographic primitives AND GET IT TORN APART BY AN EXPERT[4].

4. Learn to think about systems

Nothing that we manage, write, design or test exists on its own: it’s all part of a larger system.  That system involves nasty awkwardnesses like managers, users, attackers, backhoes and tornadoes.  Think about the larger context of what you’re doing, and you’ll be a better security person for it.  Here are some suggestions for key results:

  • read a book about systems, e.g.:
    • Security Engineering: A Guide to Building Dependable Distributed Systems, by Ross Anderson;
    • Beautiful Architecture: Leading Thinkers Reveal the Hidden Beauty in Software Design, ed. Diomidis Spinellis and Georgios Gousios;
    • Building Evolutionary Architectures: Support Constant Change by Neal Ford, Rebecca Parsons & Patrick Kua[5].
  • arrange for the operations folks in your organisation to give a 15 minute presentation to your group (I can pretty much guarantee that they think about security differently to you – unless you’re in the operations group already, of course);
  • map out a system you think you know well, and then consider all the different “external” factors that could negatively impact its security;
  • write a review of “Sneakers” or “Hackers”, highlighting how unrealistic the film[6] is, and how, equally, how right on the money it is.

5. Read a blog regularly

THIS blog, of course, would be my preference (I try to post every Tuesday), but getting into the habit of reading something security-related[7] on a regular basis means that you’re going to keep thinking about security from a point of view other than your own (which is a bit of a theme for this article).  Alternatively, you can listen to a podcast, but as I don’t have a podcast myself, I clearly can’t endorse that[8].  Key results might include:

  • read a security blog once a week;
  • listen to a security podcast once a month;
  • write an article for a site such as (the brilliant) OpenSource.com[9].

Conclusion

I’m aware that I’ve abused the OKR approach somewhat by making a number of the key results non-measureable: sorry.  Exactly what you choose will depend on you, your situation, how long the objectives last for, and a multitude of other factors, so adjust for your situation.  Remember – you’re trying to develop yourself and your knowledge.

1 – and mine.

2 – yet.

3 – yes, I called it a profession.  Feel free to chortle.

4 – the bit in CAPS is vitally, vitally important.  If you ignore that, you’re missing the point.

5 – I’m currently reading this after hearing Dr Parsons speak at a conference.  It’s good.

6 – movie.

7 – this blog is supposed to meet that criterion, and quite often does…

8 – smiley face.  Ish.

9 – if you’re interested, please contact me – I’m a community moderator there.

Getting started in IT security – an in/outsider’s view

… a basic grounding in cryptography is vital …

I am, by many measures, almost uniquely badly qualified* to talk about IT security, given that my degree is in English Literature and Theology (I did two years of each, finishing with the latter), and the only other formal university qualification I have is an MBA.  Neither of these seem to be great starting points for a career in IT security.  Along the way, admittedly, I did pick up a CISSP qualification and took an excellent SANS course on Linux and UNIX security, but that’s pretty much it.  I should also point out in my defence that I was always pretty much a geek at school***, learning Pascal and Assembly to optimise my Mandelbrot set generator**** and spending countless hours trying to create simple stickman animations.

The rest of it was learnt on the job, at seminars, meetings, from colleagues or from books.  What prompted me to write this particular post was a post over at IT Security guru, 9 out of 10 IT Security Pros Surveyed Favour Experience over Qualifications – FireMon, a brief analysis of a survey disclosed on Firemon’s site.

This cheered me, I have to say, given my background, but it also occurred to me that I sometimes get asked what advice I have for people who are interested in getting involved in IT Security.  I’m wary providing a one-size-fits-all answer, but there’s one action, and three books, that I tend to suggest, so I thought I’d share them here, in case they’re useful to anyone.

An action:

  • get involved in an Open Source project, preferably related to security.  Honestly, this is partly because I’m passionate about Open Source, but also because it’s something that I know I and others look for on an CV*****.  You don’t even need to be writing code, necessarily: there’s a huge need for documentation, testing, UI design, evangelism****** and the rest, but it’s great exposure, and can give you a great taster of what’s going on.  You can even choose a non-security project, but considering getting involved in security-related work for that project.

Three books******* to give you a taste of the field, and a broad grounding:

  1. Security Engineering: A Guide to Building Dependable Distributed Systems, by Ross Anderson. I learned more about security systems from this book than any other. I think it gives a very good overview of the field from a point of view that makes sense to me.  There’s deep technical detail in here, but you don’t need to understand all of it on first reading in order to get a lot of benefit.
  2. Practical Cryptography, by Bruce Schneier. Schneier has been in the field of security for a long time (many of his books are worth reading, as is his monthly email, CRYPTO-GRAM), and this book is a follow-up to his classic “Applied Cryptography”. In Practical Cryptography, he admitted that security was more than just mathematics, and that the human element is also important. This book goes into quite a lot of technical depth, but again, you don’t have to follow all of it to benefit.
  3. Cryptonomicon, by Neal Stephenson. This is a (very long!) work of fiction, but it has a lot of security background and history in it, and also gives a good view into the mindset of how many security people think – or used to think!  I love it, and re-read it every few years.

I’m aware that the second and third are unashamedly crypto-related (though there’s a lot more general security in Cryptonomicon than the title suggests), and I make no apology for that.  I think that a basic grounding in cryptography is vital for anyone wishing to make a serious career in IT Security.  You don’t need to understand the mathematics, but you do need to understand, if not how to use crypto correctly, then at least the impact of using it incorrectly********.

So, that’s my lot.  If anyone has other suggestions, feel free to post them in comments.  I have some thoughts on some more advanced books around architecture which I may share at some point, but I wanted to keep it pretty simple for now.

*we could almost stop the sentence here**, to be honest.

**or maybe the entire article.

***by which I mean “before university”.  When Americans ask Brits “are you at school?”, we get upset if we’ve already started university (do we really look that young?).

****the Pascal didn’t help, because BBC BASIC was so fast already, and floating point was so difficult in Assembly that I frankly gave up.

*****”Curriculum Vitae”.  If you’re from North America, think “Resumé”, but it’s Latin, not French.

******I know quite a lot about evangelism, given my degree in Theology, but that’s a story for another time.

*******All of these should be available from a decent library.  If your university/college/town/city library doesn’t have these, I’d lobby for them.  You should also be able to find them online.  Please consume them legally: authors deserve to be paid for their work.

********Spoiler: it’s bad.  Very bad.

Ignorance as a virtue: being proud to say “I don’t know”

“I am the wisest man alive, for I know one thing, and that is that I know nothing.” Socrates

In order to be considered an expert in any field, you have to spend a lot of time learning things.  In fact, I’d argue that one of the distinguishing traits of someone who is – or could become – an expert is their willingness and enthusiasm to learn, and keep learning.  The ability to communicate that knowledge is another of those traits: you can’t really be an expert if you have no way to communicate that knowledge.  Though that doesn’t mean that you need to be a great speaker, or even a great writer: by “communicate” I’m thinking of something much broader.  In the field of security and IT, that communication may be by architecture diagram, by code writing, by firewall rule instantiation, or by GUI, database or kernel module design, to name just a few examples.  These are all ways by which expertise can be communicated, instantiated or realised: the key is that the knowledge that has been gained is not contained, but can be externalised.

There’s another trait that, for me, betrays a true expert, and that’s the ability to say “I don’t know”.  And it’s difficult.  We enjoy and cultivate our expert status and other’s recognition of it: it’s part of our career progression, and it hits the “esteem” block in Maslow’s Hierarchy of Needs[1].  We like people asking our opinion, and we like being able to enlighten them: we take pride in our expertise, and why wouldn’t we?  We’ve earned it, after all, with all that hard graft and studying.  What’s more, we’ve all seen what happens when people get asked a question to which they don’t know the answer to something – they can become flustered, embarrassed, and they can be labelled stupid.*  Why would we want that for ourselves?

The problem, and very particularly in the security field, is that you’ll always get found out if you fake it.  In my experience, you’ll go into a customer meeting, for instance, and there’s either the sandal-wearing grey-beard, the recently-graduated genius or just the subject matter expert who’s been there for fifteen years and knows this specific topic better than … well, possibly anybody else on the planet, but certainly better than you.  They may not be there in the first meeting, but you can bet your bottom dollar*** that they’ll be in the second meeting, or the third – and you’ll get busted.  And when that happens, everything else you’ve said is called into question.  That may not seem fair, but that’s the way it goes.  Your credibility is dented, possibly irreparably.

The alternative to faking it is to accept that awkward question and simply to say, “I don’t know”.  You may want to give the question a moment’s thought – there have been times when I’ve plunged into an response and then stopped myself to admit that I just can’t give a full or knowledgeable answer, and when I could have saved myself bother by just pausing and considering it for a few seconds.  And you may want to follow up that initial acknowledgement of ignorance by saying that you know somebody else who does (if that happens to be true), or “I can find out” (if you think you can) or even “do you have any experts who might be able to help with that?”

This may not impress people who think you should know, but they’re generally either asking because they don’t (in which case they need a real answer) or because they’re trying to trip you up (in which case you don’t want to oblige them).  But it will impress those who are experts, because they know that nobody knows everything, and it’s much better to have that level of self-awareness than to dig yourself an enormous hole from which it’s difficult to recover.  But they’ll also understand, from your follow-up, that you want to find out: you want to learn.  And that is how one expert recognises another.

* it’s always annoyed me when people mock Donald Rumsfeld for pointing out that there are “unknown unknowns”: it’s probably one of the wisest soundbites in recent history**, for my money.

** and for an equivalently wise soundbite in ancient history, how about “I am the wisest man alive, for I know one thing, and that is that I know nothing”, by Socrates
*** other currencies and systems of exchange are available