programming – Alice, Eve and Bob

How to hire an open source developer

Our view was that a pure “algorithm coding” exercise was pretty much useless for what we wanted.

We’ve recently been hiring developers to work on the Enarx project, a security project, written almost exclusively in Rust (with a bit of Assembly), dealing with Confidential Computing. By “we”, I mean Profian, the start-up for which I’m the CEO and co-founder. We’ve now found all the people we’re looking for initially on the team (with a couple due to start in the next few weeks), though we absolutely welcome contributors to Enarx, and, if things continue to go well, we’ll definitely want to hire some more folks in the future.

Hiring people is not easy, and we were hit with a set of interesting requirements which made the task even more difficult. I thought it would be useful and interesting for the community to share how we approached the problem.

What were we looking for?

I mentioned above some interesting requirements. Here’s what the main ones were:

systems programming – we mainly need people who are happy programming at the systems layer. This is pretty far down the stack, with lots of interactions directly with hardware or the OS. Where we are creating client-server pieces, for instance, we’re having to write quite a lot of the protocols, manage the crypto, etc., and the tools we’re using aren’t all very mature (see “Rust” below).
Rust – almost all of the project is written in Rust, and what isn’t is written in Assembly language (currently exclusively x86, though that may change as we add more platforms). Rust is new, cool and exciting, but it’s still quite young, and some areas don’t have all the support you might like, or aren’t as mature as you might hope – everything from cryptography through multi-threading libraries and compiler/build infrastructure.
distributed team – we’re building a team of folks where can find them: we have developers in Germany, Finland, the Netherlands, North Carolina (US), Massachusetts (US), Virginia (US) and Georgia (US), I’m in the UK, our community manager is in Brazil and we have interns in India and Nigeria. We knew from the beginning that we wouldn’t have everyone in one place, and this required people who we were happy would be able to communicate and collaborate with people via video, chat and (at worst) email.
security – Enarx is a security project, and although we weren’t specifically looking for security experts, we do need people who are able to think and work with security top of mind, and design and write code which is applicable and appropriate for the environment.
git – all of our code is stored in git (mainly GitHub, with a little bit of GitLab thrown in), and so much of our interaction around code revolves around git that anybody joining us would need to be very comfortable using it as a standard tool in their day-to-day work.
open source – open source isn’t just a licence, it’s a mindset, and, equally important, a way of collaborating. A great deal of open source software is created by people who aren’t geographically co-located, and who might not even see themselves as a team. We needed to be sure that the people we were hiring, while gelling as a close team within the company, will also be able to collaborate with people outside the organisation and be able to embrace Profian’s “open by default” culture not just for code, but for discussions, communications and documentation.

How did we find them?

As I’ve mentioned before, in Recruiting is hard. We ended up using a variety of means to find candidates, with varying levels of success:

LinkedIn job adverts
LinkedIn searches
Language-specific discussion boards and hiring boards (e.g. Reddit)
An external recruiter (shout out to Gerald at Interstem)
Word-of-mouth/personal recommendations

It’s difficult to judge between them in terms of quality, but without an external recruiter, we’d certainly have struggled with quantity (and we had some great candidates from that pathway, too).

How did we select them?

We needed to measure all of the candidates against all of the requirements noted above, but not all of them were equal. For instance, although we were keen to hire Rust programmers, we were pretty sure that someone with strong C/C++ skills at the systems level would be able to pick up Rust quickly enough to be useful. On the other hand, a good knowledge of using git was absolutely vital, as we couldn’t spend time working with new team members to bring them up-to-speed on our way of working. A strong open source background was, possibly surprisingly, not a requirement, but the mindset to work in that sort of model was, and anyone with a history of open source involvement is likely to have a good knowledge of git. The same goes for the ability to work in a distributed team: so much of open source is distributed that involvement in almost any open source community was a positive indicator. Security we decided was a “nice-to-have”.

How to proceed? We wanted to keep the process simple and quick – we don’t have a dedicated HR or People function, and we’re busy trying to get code written. What we ended up was this (with slight variations), which we tried to get complete within 1-2 weeks:

Initial CV/resume/github/gitlab/LinkedIn review – this to decide whether to interview
30-40 minute discussion with me as CEO, to find out if they might be a good cultural fit, to give them a chance to find out about us, and get an idea if they were as technically adept as they appeared from the first step
Deep dive technical discussion led by Nathaniel, usually with me there
Chat with other members of the team
Coding exercise
Quick decision (usually within 24 hours)

The coding exercise was key, but we decided against the usual approach. Our view was that a pure “algorithm coding” exercise of the type so beloved by many tech companies was pretty much useless for what we wanted. What we wanted to understand was whether candidates could quickly understand a piece of code, fix some problems and work with the team to do so. We created a github repository (in fact, we ended up using two – one for people a little higher up the stack) with some almost-working Rust code in it, some instructions to fix it, perform some git-related processes on it, and then improve it slightly, adding tests along the way. A very important part of the test was to get candidates to interact with the team via our chat room(s). We scheduled 15 minutes on a video call for set up and initial questions, 2 hours for the exercise (“open book” – as well as talking to the team, candidates were encouraged to use all resources available to them on the Internet), followed by a 30 minute wrap-up session where the team could ask questions and the candidate could also reflect on the task. This also allowed us to get an idea of how well the candidate was able to communicate with the team (combined with the chat interactions during the exercise). Afterwards, the candidate would drop off the call, and we’d generally make a decision within 5-10 minutes as to whether we wanted to hire them.

This generally worked very well. Some candidates struggled with the task, some didn’t communicate well, some failed to do well with the git interactions – these were the people we didn’t hire. It doesn’t mean they’re not good coders, or that they might not be a good fit for the project or the company later on, but they didn’t immediate meet the criteria we need now. Of the ones we hired, the levels of Rust experience and need for interaction with the team varied, but the level of git expertise and their reactions to our discussions afterwards was always sufficient for us to decide to take them.

Reflections

On the whole, I don’t think we’d change a huge amount about the selection process – though I’m pretty sure we could do better with the search process. The route through to the coding exercise allowed us to filter out quite a few candidates, and the coding exercise did a great job of helping us pick the right people. Hopefully everyone who’s come through the process will be a great fit and will produce great code (and tests and documentation and …) for the project. Time will tell!

Rust – my top 7 keywords

A few useful keywords from the Rust standard library.

I’ve been using Rust for a few months now, writing rather more of it than I expected – though quite a lot of that has been thrown away as I’ve learnt improved what I’m writing and taken some more complex tasks on beyond what I’d originally intended. I still love it, and thought that today might be a good day to talk about some of the important keywords that come up again and again in Rust, and provide my personal summary of what they do, why you need to think about how you use them, and anything else that’s useful, particularly for people who are new to Rust, or coming from another language (such as Java – see my previous article on the subject, 5 Rust reflections (from Java)). Without further ado, let’s get going. A good place for further information is always the official Rust documentation – you’ll probably want to start with the std library.

const – you get to declare constants with “const”, and you should. This isn’t rocket science, but do declare with const, and if you’re going to use constants across different modules, then do the right thing and create a lib.rs file (the Rust default) into which you can put all of these, with a nicely name module. I’ve had clashes of const variable names (and values!) across different files in different modules, simply because I was too lazy to do anything other than cut and paste across files, when I could have save myself lots of work by simply creating a shared module.
let – you don’t always need to declare a variable with a let statement, but your code will be clearer when you do. What’s more, always add the type if you can. Rust will do its very best to guess what it should be, but may not always be able to do so at runtime (in which case Cargo, the compiler, will tell you), or may even not necessarily do what you expect. In the latter case, it’s always simpler for Cargo to complain that the function you’re assigning from (for instance) doesn’t match the declaration than for Rust to try to help you do the wrong thing, only for you to have to spend ages debugging elsewhere.
match – match was new to me, and I love it. It’s not dissimilar to “switch” in other languages, but is used extensively in Rust. It makes for legible code, and Cargo will have a good go at warning you if you do something foolish (such as miss out possible cases). My general rule of thumb, where I’m managing different options or doing branching, is to ask whether I can use match. If I can, I will.
mut – when declaring a variable, if it’s going to change after its initialisation, then you need to declare it mutable. A common mistake is to declare something as mutable when it isn’t changed – but the compiler will warn you about that. If you get a warning from Cargo that a mutable variable isn’t changed when you think it is, then you may wish to check the scope of the variable, and check that you’re using the right version.
return – I actually very rarely use return, which is for returning a value from a function, because it’s usually simpler and clearer to read if you just provide the value (or function providing the return value) at the end of the function, as the last line. Warning: you will forget to omit the semicolon at the end of this line on many occasions: if you do, the compiler won’t be happy.
unsafe – does what it says on the tin: if you want to do things where Rust can’t guarantee memory safety, then you’re going to need to use this keyword. I have absolutely no intention of declaring any of my Rust code unsafe now or at any point in the future: one of the reasons Rust is so friendly is because it stops this sort of hackery. If you really need to do this, think again, think yet again, and then redesign. Unless you’re a seriously low-level systems programmer, avoid.
use – when you want to use an item – struct, variable, function, etc. from another crate, then you need to declare it at the beginning of the block where you’ll be using it. Another common mistake is to do this, but fail to add the crate (preferably with minimum version number) to the Cargo.toml file.

This isn’t the most complicated article I’ve ever written, I know, but it’s the sort of article which I would have appreciated finding when I was starting to learn Rust. I plan to create similar articles on key functions and other Rust must-knows: let me know if you have any requests!

More Rusty thoughts

I do feel that I’m now actually programming in Rust. And I like it.

I wrote an article a couple of weeks ago – 5 Rust reflections (from Java) – about learning Rust, and specifically, about moving to Rust from Java. I talked about five particular points:

Rust feels familiar
References make sense
Ownership will make sense
Cargo is helpful
The compiler is amazing

I absolutely stand by all of these, but I’ve got a little more to say, because I now feel like a Rustacean[1], in that:

I don’t feel like programming in anything else ever again;
I’ve moved away from simple incantations.

What do I mean by these two statemetents? Well, the first is pretty simple: Rust feels like the place to be. It’s well-structured, it’s expressive, it helps you do the right thing[2], it’s got great documentation and tools, and there’s a fantastic community. And, of course, it’s all open source, which is something that I care about deeply.

And the second thing? Well, I decided that in order to learn Rust properly, I should take an existing project that I had originally written in Java and reimplement it in hopefully fairly idiomatic Rust. Sometime in the middle of last week, I started fixing mistakes – and making mistakes – around implementation, rather than around syntax. And I wasn’t just copying text from tutorials or making minor, seemingly random changes to my code based on the compiler output. In other words, I was getting things to compile, understanding why they compiled, and then just making programming mistakes[3].

This is a big step forward. When you start learning a language, it’s easy just to copy and paste text that you’ve seen elsewhere, or fiddle with unfamiliar constructs until they – sort of – work. Using code – or producing code -that you don’t really understand, but seems to work, is sometimes referred to as “using incantations” (from the idea that most magicians in fiction, film and gaming reciti collections of magic words which “just work” without really understanding what they’re doing or what the combination of words actually means). Some languages[4] are particularly prone to this sort of approach, but many – most? – people learning a new language will be prone to doing this when they start out, just because they want things to work.

And last night, I was up till 1am implementing a new feature – accepting command-line input – which I really couldn’t get my head round. I’d spent quite a lot of time on it (including looking for, and failing to find, some appropriate incantations), and then asked for some help on a rust-lang channel inhabited by some people I know. A number of people had made some suggestions about what had been going wrong, and one person in particular was enormously helpful in picking apart some of the suggestions so that I understood them better. He explained quite a lot, but finished with “I don’t know the return type of the hash function you’re calling – I think this is a good spot for you to figure this piece out on your own.”

This was just what I needed – and any learner of anything, including programming languages, needs. So when I had to go downstairs at midnight to let the dog out, I decided to stay down and see if I could work things out for myself. And I did. I took the suggestions that people had made, understood out what they were doing, tried to divine what they should be doing, worked out how they should be doing it, and then found the right way of making it happen.

I’ve still got lots to learn, and I’ll make lots of mistakes still, but I now feel that I’m in a place to find my way through those mistakes (with a little help along the way, probably – thanks to everyone who’s already pointed me in the right direction). But I do feel that I’m now actually programming in Rust. And I like it.

1 – this is what Rust programmers call themselves.

2 – it’s almost impossible to stop people doing the wrong thing entirely, but encouraging people do to the right thing is great. In fact, Rust goes further, and actually makes it difficult to do the wrong thing in many situations. You really have to try quite hard to do bad things in Rust.

3 – I found a particularly egregious off-by-one error in my code, for instance, which had nothing to do with Rust, and everything to do with my not paying enough attention to the program flow.

4 – *cough* Perl *cough*