Organisational suppleness

Growing the ability to react to the unexpected is a valuable skill.

“In preparing for battle I have always found that plans are useless but planning is indispensable.”

Dwight D. Eisenhower

Much of this blog is about security – cybersecurity – in one way or another, but on occasion I do try to take a broader view. Cybersecurity is often modelled or described in military terms, talking about “fighting battles”, “wars of attrition” and “arms races” with “attackers”. These can be useful metaphors (and it’s why I started this article with a quote from a general), but there is a broader set of responsibilities that many of us in the sector need to consider, which is the continued (and hopefully healthy) functioning of our businesses and organisations. In particular, I like to talk about risk and how it relates not just to security, but to how businesses work and plan. One theme that I’ve visited before is that known or planned degradation of a service is often significantly better than failure, or even planned closure (see Service degradation: actually a good thing). My argument is that there are many occasions where keeping a service or business function running, albeit at reduced capacity, or with reductions in known capabilities, allows for better continuity than just stopping it.

Keeping a service running requires work. You can’t just hope that everything is installed and will run as you expect: what happens when your administrator is ill, your fibre-optic cable gets severed by a back hoe, or a DDoS attack is directed at you? You need to plan and practice what to do in these situations. What I’d like to explore in this article goes somewhat beyond the expectation of that planning in three directions. Let’s call them scenario coverage, muscle memory and organisational suppleness.

Scenario coverage

The first, and most obvious of the three directions, is about understanding eventualities. The more scenarios that we model and practice, the more we reduce our risk, simply because we have reduced the number of unknown eventualities in the probability space. There is a actually a side benefit to modelling lost of scenarios, which is that the more situations you consider, the more will come to mind. Every situation involves sets of choices or probabilities – “after the door closes, will it lock or not?” or “if the coolant fails, will the system turn off or burst into flames?” – and the more scenarios you consider, the more questions will arise. This can be daunting – and it’s almost impossible to consider every eventuality – but the more options are covered, the better your opportunities to mitigate the various risks they present.

Muscle memory

Muscle memory is what comes with training and practice. Assuming that you are including your teams in the scenario planning

And I’m assuming here that the planning isn’t solely a paper exercise. Theoretical planning, while useful, only goes so far, for a couple of important reasons:

  • systems will always fails in unexpected ways
  • people will do unexpected things.

What the first of these means is that however much you assume that your back-up generator will kick in if there’s a power outage, until you test it, you can’t be sure that it will. The second of these relates to the fact that however much you tell people what to do, when it actually comes to the doing of it, they’re unlikely to as you expect. This is likely to be even worse if there’s been no training, and you’re just assuming that person X will know how to operate a fire extinguisher, or that team Y will, of course, exit the building in an orderly manner via exit Z (rather than find fourteen different exits, or not even leave the building at all).

For both of these reasons, getting people together to work through possible scenarios, and then, where possible, actually practising what to do, means that you have a higher assurance that when one of the situations you’ve considered does arrive, that they will know what to do, and act as you expect.

Organisational suppleness

While you cannot, as we’ve noted, plan for every eventuality or know exactly how an employee or team will react when things go wrong, there is another benefit to involving a broad group of people in your scenario planning and training. This is that their very involvement gives them practice in dealing with uncertainty, working out how they will react, and giving them experience in how those around them will act. While I may not know exactly what to do if the payroll system goes down an hour before it is due to run, if I have worked with colleagues on scenarios where the sales processing system fails, I’ve got a better chance of making some sensible choices about who to contact, initial steps to take and information to collect than if this is the first time I’ve ever seen anything like it. Likewise, we may not have modelled our response to a physical failure of one of our network links, but our shared experience of practising our response to a DDoS attack means that we have an idea of what to do.

And it is not just having an idea of what to do that is important, but also having gathered and practised the cognitive skills associated with investigating failures, collating data, sharing information and working with others to ameliorate the situation which allows a team or an organisation to respond better to new, maybe unexpected situations. We can think of this as suppleness, as it means that rather than just failing, or cracking, an organisation can react as a tree does to strong winds, or a gymnast does to a new exercise. Growing the ability to react to the unexpected is a valuable skill for an organisation, and knowing that it is supple allows its leaders to plan with more certainty and mitigate more risk.

Buying my own t-shirts, OR “what I miss about conferences”

I can buy my own t-shirts, but friendships need nurturing.

A typical work year would involve my attending maybe six to eight conferences in person and speaking at quite a few of them. A few years ago, I stopped raiding random booths at the exhibitions usually associated with these for t-shirts for the simple reason that I had too many of them. That’s not to say that I wouldn’t accept one here or there if it was particularly nice, or an open source project which I esteemed particularly, for instance. Or ones which I thought my kids would like – they’re not “cool”, but are at least useful for sleepwear, apparently. I also picked up a lot of pens, and enough notebooks to keep me going for a while.

And then, at the beginning of 2020, Covid hit, I left San Francisco, where I’d been attending meetings co-located with RSA North America (my employer at the time, Red Hat, made the somewhat prescient decision not to allow us to go to the main conference), and I’ve not attended any in-person conferences since.

There are some good things about this, the most obvious being less travel, though, of late, my family has been dropping an increasing number of not-so-subtle hints about how it would be good if I let them alone for a few days so they can eat food I don’t like (pizza and macaroni cheese, mainly) and watch films that I don’t enjoy (largely, but not exclusively, romcoms on Disney+). The downsides are manifold. Having to buy my own t-shirts and notebooks, obviously, though it turns out that I’d squirrelled away enough pens for the duration. It also turned out that the move to USB-C connectors hadn’t sufficiently hit the conference swag industry by the end of 2019 for me to have enough of those to keep me going, so I’ve had to purchase some of those. That’s the silly,minor stuff though – what about areas where there’s real impact?

Virtual conferences aren’t honestly too bad and the technology has definitely improved over the past few months. I’ve attended some very good sessions online (and given my share of sessions and panels, whose quality I won’t presume to judge), but I’ve realised that I’m much more likely to attend borderline-interesting talks not on my main list of “must-sees” (some of which turn out to be very valuable) if I’ve actually travelled to get to a venue. The same goes for attention. I’m much less likely to be checking email, writing emails and responding to chat messages in an in-person conference than a virtual one. It’s partly about the venue, moving between rooms, and not bothering to get my laptop out all the time – not to mention the politeness factor of giving your attention to the speaker(s) or panellists. When I’m sitting at my desk at home, none of these is relevant, and the pull of the laptop (which is open anyway, to watch the session) is generally irresistible.

Two areas which have really suffered, though, are the booth experience the “hall-way track”. I’ve had some very fruitful conversations both from dropping by booths (sometimes mainly for a t-shirt – see above) or from staffing a booth and meeting those who visit. I’ve yet to any virtual conferences where the booth experience has worked, particularly for small projects and organisations (many of the conferences I attend are open source-related). Online chat isn’t the same, and the serendipitous aspect of wandering past a booth and seeing something you’d like to talk about is pretty much entirely missing if you have to navigate a set of webpages of menu options with actual intent.

The hall-way track is meeting people outside the main sessions of a conference, either people you know already, or as conversations spill out of sessions that you’ve been attending. Knots of people asking questions of presenters or panellists can reveal shared interests, opposing but thought-provoking points of view or just similar approaches to a topic which can lead to valuable professional relationships and even long-term friendships. I’m not a particularly gregarious person – particularly if I’m tired and jetlagged – but I really enjoy catching up with colleagues and friends over a drink or a meal from time to time. While that’s often difficult given the distributed nature of the companies and industries I’ve been involved with, conferences have presented great opportunities to meet up, have a chinwag and discuss the latest tech trends, mergers and acquisitions and fashion failures of our fellow attendees. This is what I miss most: I can buy my own t-shirts, but friendships need nurturing. and I hope that we can safely start attending conferences again so that I can meet up with friends and share a drink. I just hope I’m not the one making the fashion mistakes (this time).

Eat, Sleep, Wake (nothing but…)

At least I’m not checking my email every minute of every hour of every day.

If your mind just filled in the ellipsis (the “…”) in the title of this article with “you”, then you may have been listening to the Bombay Bicycle Club, a British band. I’ve recently seen them live, and then were good – what’s more, it’s a great (and very catch) song. “You” is probably healthy. If, on the other hand, your mind filled in the ellipsis with “work”, then, well we – or rather, you – have a problem.

When I wake up in the morning, one of the first things I do – like many of you, my dear readers, I suspect – is reach for my mobile phone. One of the first things I do on unlocking it is check my email. Specifically, my work email. Like many of us, I find it convenient to keep my work email account on my personal phone. I enjoy the flexibility of not being tied to my desk throughout the working day, and fancy myself important enough that I feel that people may want to contact me during the day and expect a fairly quick reply. Equally, I live in the UK and work with people across CET (an hour earlier than me) to Eastern US time (5 hours after me), often correspond with people on Pacific US time (8 hours after me), and sometimes in other timezones, too. In order to be able to keep up with them, and not spend 12 hours or so at my desk, I choose to be able to check for incoming emails wherever I am – which is wherever my phone is. So I check email through the day – and to almost last thing at night.

This is not healthy. I know this – as do my family. It is also not required. I know this – as do my colleagues. In fact, my colleagues and my family all know that it’s neither healthy nor required. I also know that I have a mildly addictive personality, and that, if I allowed myself to do so, I would drown in my work, always checking email, always writing new documents, always reviewing other people’s work, always, always, always on my phone: eat, sleep, wake…

In order to stop myself doing this, I make myself do other things. These aren’t things I don’t want to do – it’s just that I would find excuses not to do them if I could. I run (slowly and badly, up to 5 kilometres) 2-3 times a week. I read (mainly, but not exclusively, science fiction). I game (Elite Dangerous, TitanFall 2 (when it’s not being DDoSed), Overwatch, Civilization (mainly V, Call to Power), and various games on my phone), I listen to, and occasionally watch, cricket. And recently, I’ve restarted a hobby from my early teenage years: I’m assembling a model airplane (badly, though not as badly as I did when I was younger). I force myself to take time to do these things. I’m careful to ensure that they don’t interfere with work calls, and that I have time to get “actual” work done. I keep block of time where I can concentrate on longer tasks, requiring bouts of concentration. But I know that my other work actually benefits when I force myself to take time out, because a few minutes away from the screen, at judicious points, allows me to step back and recharge a bit.

I know that I’m a little odd in having lots of activities – hobbies, I guess – that I enjoy (I’ve only listed a few above). Other people concentrate on one, and rather than interspersing blocks of non-work time into their day, have these blocks of time scheduled outside their core working hours. One friend I know cycles for hours at a time (his last Strava entry was a little over 100km (60 miles) and a little under 3 and a half hours) – an activity which would be difficult to fit in between meetings for most working routines. Others make the most of their commute (yes, some people do commute still) to listen to podcasts, for instance. What’s in common here is a commitment to the practice of not working.

I realise that being able to do this is a luxury not shared by all. I likewise realise that I work in an industry (IT) where there is an expectation that senior people will be available at short notice for many hours of the day – something we should resist. But finding ways of not working through the day is, for me, a really important part of my working – it makes me a more attentive, better worker. I hesitate to call this “work-life balance”, because, honestly, I’m not sure that it is a balance, and I need to keep tweaking it. But at least I’m not checking my email every minute of every hour of every day.

Recruiting on ability and potential

Let’s not just hire people who look and think and talk (and take exams) like us

This is one of those more open-ended posts, in that I don’t have any good answers, but I’ve got a bunch of questions. I’d love to have feedback, comments and thoughts if you have any.

This week, it turns out is “results week” in the UK. Here, there are two major sets of exams which most students take: GCSEs and A levels. The former generally are taken by 16 year olds, with around 8-12 in total, and the latter by 18 year olds, with 2-4 in total (3 being the strong median). A level results are one of the major criteria for entry into universities, and GCSE results typically determine what A levels students will take, and therefore have an impact on university course choice in the future. In normal years, A level results are release a week before the GCSE results, but (Covid having messed things up), A level results day is Tuesday, 2021-08-10, and GCSE results day is Thursday, 2021-08-12.

Both GCSEs and A levels are determined, in most cases, mainly by examination, with even subjects like PE (Physical Education), Dance and Food Technology having fairly large examination loads. This was not always the case, particularly for GCSEs, which, when they were launched in the mid 1980s, had much larger coursework components than now, aiming to provide opportunities to those who may have learned retained lots of information, but for whom examinations are not a good way of showing their expertise or acquired knowledge base.

At this point, I need to come out with an admission: exams suit me just fine. I’ve always been able to extricate information from my memory in these sorts of situations and put it down on paper, whether that’s in a school setting, university setting or professional setting (I took the CISSP examination ages ago). But this isn’t true for everybody. Exams are very artificial situations, and they just don’t work for many people. The same is true for other types of attempts to extract information out of people, such as classic coding tests. Now, I know that coding tests, when designed and applied well, can do a good job in finding out how people think, as well as what they know – in fact, that should be true for many well-designed examinations – but not only are they not always well administered, but the very context in which they are taken can severely disadvantage folks whose preferred mechanism of coding is more solitary and interactive than performative.

A family friend, currently applying for university places, expressed surprise the other day that some of the Computer Science courses for which he was applying didn’t require any previous experience of programming. I reflected that for some candidates, access to computing resources might be very limited, putting them at a disadvantage. I’ve made a similar argument when discussing how to improve diversity in security (see Is homogeneity bad for security?): we mustn’t make assumptions about people’s potential based on their previous ability to access resources which would allow that ability to be expressed and visible.

What does this have to do with recruitment, particularly? Everything. I’ve been involved in recruiting some folks recently, and the more I think about it, the more complex I realise the task is. Finding the right people – the people who will grow into roles, and become really strong players – is really tricky. I’m not just talking about finding candidates and encouraging them to apply (both difficult tasks in their own rights), but choosing from the candidates that do apply. I don’t want to just people who examine well (looking solely at their academic grades), people who perform well (who shine at coding exercises), people who interview well (those who are confident in face-to-face or video conference settings). I want to find people who have the ability to work in a team, grow the projects they’re working on, contribute creatively to what they’re doing. That may include people who fall apart in exams, who freeze during coding exercises, or whose social anxiety leads them to interview “badly”.

I’m not sure how to address these problems completely, but there are some things we can try. Here are my thoughts, and I’d love to hear more:

  1. Look beyond exams, and rate experience equally with academic attainment
  2. Accommodate different working styles
  3. Be aware that the candidate may not share your preferred interaction style
  4. Think “first language” and work with those whose native language is not yours
  5. Look beyond the neurotypical

None of these is rocket science, and I hope that the recent changes in working habits brought around by the Covid pandemic have already started people thinking about these issues, but let’s work harder to find the right people – not just the people who look and think and talk (and take exams) just like us.

In praise of … the Community Manager

I am not – and could never be – a community manager

This is my first post in a while. Since Hanging up my Red Hat I’ve been busy doing … stuff. Stuff which I hope to be able to speak about soon. But in the meantime, I wanted to start blogging regularly again. Here’s my first post back, a celebration of an important role associated with open source projects: the community manager.

Open source communities don’t just happen. They require work. Sometimes the technical interest in an open source project is enough to attract a group of people to get involved, but after some time, things are going to get too big for those with a particular bent (documentation, coding, testing) to manage the interactions between the various participants, moderate awkward (or downright aggressive) communications, help encourage new members to contribute, raise the visibility of the project into new areas or market sectors and all the other pieces that go into keeping a project healthy.

Enter the Community Manager. The typical community manager is in that awkward position of having lots of responsibility, but no direct authority. Open source projects being what they are, few of them have empowered “officers”, and even when there are governance structures, they tend to operate by consent of those involved – by negotiated, rather than direct, authority. That said, by the point a community manager is appointed for a community manager, it’s likely that at least one commercial entity is sufficiently deep into the project to fund or part-fund the community manager position. This means that the community manager will hopefully have some support from at least one set of contributors, but will still need to build consensus across the rest of the community. There may also be tricky times, also, when the community manager will need to decide whether their loyalties lie with their employer or with the community. A wise employer should set expectations about how to deal with such situations before they arise!

What does the community manager need to do, then? The answer to this will depend on a number of issues, and there is likely to be a balance between these tasks, but here’s a list of some that come to mind[1].

  • marketing/outreach – this is about raising visibility of the project, either in areas where it is already known, or new markets/sectors, but there are lots of sub-tasks such as a branding, swag ordering (and distribution!), analyst and press relations.
  • event management – setting up meetups, hackathons, booths at larger events or, for really big projects, organising conferences.
  • community growth – spotting areas where the project could use more help (docs, testing, outreach, coding, diverse and inclusive representation, etc.) and finding ways to recruit contributors to help improve the project.
  • community lubrication – this is about finding ways to keep community members talking to each other, celebrate successes, mourn losses and generally keep conversations civil at least and enthusiastically friendly at best.
  • project strategy – there are times in a project when new pastures may beckon (a new piece of functionality might make the project exciting to the healthcare or the academic astronomy community for instance), and the community manager needs to recognise such opportunities, present them to the community, and help the community steer a path.
  • product management – in conjunction with project strategy, situations are likely to occur when a set of features or functionality are presented to the community which require decisions about their priority or the ability of the community to resource them. These may even create tensions between various parts of the community, including involved commercial interests. The community manager needs to help the community reason about how to make choices, and may even be called upon to lead the decision-making process.
  • partner management – as a project grows, partners (open source projects, academic institutions, charities, industry consortia, government departments or commercial organisations) may wish to be associated with the project. Managing expectations, understanding the benefits (or dangers) and relative value can be a complex and time-consuming task, and the community manager is likely to be the first person involved.
  • documentation management – while documentation is only one part of a project, it can often be overlooked by the core code contributors. It is, however, a vital resource when considering many of the tasks associated with the points above. Managing strategy, working with partners, creating press releases: all of these need good documentation, and while it’s unlikely that the community manager will need to write it (well, hopefully not all of it!), making sure that it’s there is likely to be their responsibility.
  • developer enablement – this is providing resources (including, but not restricted to, documentation) to help developers (particularly those new to the project) to get involved in the project. It is often considered a good idea to separate this set of tasks out, rather than expecting a separate role to that of a community manager, partly because it may require a deeper technical focus than is required for many of the other responsibilities associated with the role. This is probably sensible, but the community manager is likely to want to ensure that developer enablement is well-managed, as without new developers, almost any project will eventually calcify and die.
  • cat herding – programmers (who make up the core of any project) are notoriously difficult to manage. Working with them – particularly encouraging them to work to a specific set of goals – has been likened to herding cats. If you can’t herd cats, you’re likely to struggle as a community manager!

Nobody (well almost nobody) is going to be an expert in all of these sets of tasks, and many projects won’t need all of them at the same time. Two of the attributes of a well-established community manager are an awareness of the gaps in their expertise and a network of contacts who they can call on for advice or services to fill out those gaps.

I am not – and could never be – a community manager. I don’t have the skills (or the patience), and one of the joys of gaining experience and expertise in the world is realising when others do have skills that you lack, and being able to recognise and celebrate what they can bring to your world that you can’t. So thank you, community managers!


1 – as always, I welcome comments and suggestions for how to improve or extend this list.

3 vital traits for an open source leader

The world is not all about you.

I’ve written a few articles on how to be do something badly, or how not to do things, as I think they’re a great way of adding a little humour to the process of presenting something. Some examples include:

The last, in particular, was very popular, and ended up causing so much furore on a mailing list that the mailing list had to be deleted. Not my fault, arguably (I wasn’t the one who posted it to the list), but some measure of fame (infamy?) anyway. I considered writing this article in a similar vein, but decided that although humour can work as a great mechanism to get engaged, it can also sometimes confuse the message, or muddy the waters of what I’m trying to say. I don’t want that: this is too important.

I’m writing this in the midst of a continuing storm around the re-appointment to the board of the Free Software Foundation (FSF) of Richard Stallman. We’re also only a couple of years on from Linus Torvalds deciding to make some changes to his leadership style, and apologising for his past behaviour. Beyond noting those events, I’m not going to relate them to any specific points in this article, though I will note that they have both informed parts of it.

The first thing I should say about these tips is that they’re not specific to open source, but are maybe particularly important to it. Open source, though much more “professional” than it used to be, with more people paid to work on it, is still about voluntary involvement. If people don’t like how a project is being run, they can leave, fork it, or the organisation for which they work may decide to withdraw funding (and/or its employees’ time). This is different to most other modes of engagement in projects. Many open source projects also require maintenance, and lots of it: you don’t just finish it, and then hand it over. In order for it to continue to grow, or even to continue to be safe and relevant to use, it needs to keep running, preferably with a core group of long-term contributors and maintainers. This isn’t unique to open source, but it is key to the model.

What all of the above means is that for an open source project to thrive in the long-term, it needs a community. The broader open source world (community in the larger sense) is moving to models of engagement and representation which more closely model broader society, acknowledging the importance of women, neuro-diverse members, older, younger, disabled members and other under-represented groups (in particular some ethnic groups). It has become clear to most, I believe, that individual projects need to embrace this shift if they are to thrive. What, then, does it mean to be a leader in this environment?

1. Empathise

The world is not all about you. The project (however much it’s “your baby”) isn’t all about you. If you want your project to succeed, you need other people, and if you want them to contribute, and to continue to contribute to your project, you need to think about why they might want to do so, and what actions might cause them to stop. When you do something, say something or write something, think not just about how you feel about it, but about how other people may feel about it.

This is hard. Putting yourself in other people’s shoes can be really, really difficult, the more so when you don’t have much in common with them, or feel that your differences (ethnicity, gender, political outlook, sexuality, nationality, etc.) define your relationship more than your commonalities. Remember, however, that you do share something, in fact, the most important thing in this context, which is a desire to work on this project. Ask others for their viewpoints on tricky problems – no, strike that – just ask others for the viewpoints, as often as possible, particularly when you assume that there’s no need to do so. If you can see things at least slightly from other people’s point of view, you can empathise with them, and even the attempt to do so shows that you’re making an effort, and that helps other people make an effort to empathise, too, and you’re already partway to meeting in the middle.

2. Apologise

You will get things wrong. Others will get things wrong. Apologise. Even if you’re not quite sure what you’ve done wrong. Even if you think you’re in the right. If you’ve hurt someone, whether you meant to or not, apologise. This can be even harder than empathising, because once two (or more) parties have entrenched themselves in positions on a particular topic, if they’re upset or angry, then the impulse to empathise will be significantly reduced. If you can empathise, it will become easier to apologise, because you will be able to see others’ points of view. But even if you can’t see their point of view, at least realise that they have another point of view, even if you don’t agree with it, or think it’s rational. Apologising for upsetting someone is only a start, but it’s an important one.

3. Don’t rely on technical brilliance and vision

You may be the acknowledged expert in the field. You may have written the core of the project. It may be that no-one will ever understand what you have done, and its brilliance, quite like you. Your vision may be a guiding star, bringing onlookers from near and far to gaze on your project.

Tough.

That’s not enough. People may come to your project to bask in the glory of your technical brilliance, or to wrap themselves in the vision you have outlined. Some may even stay. But if you can’t empathise, if you can’t apologise when you upset them, those people will represent only a fraction of the possible community that you could have had. The people who stay may be brilliant and visionary, too, but your project is the weaker for not encouraging (not to mention possibly actually discouraging) broader, more inclusive involvement of those who are not like you, in that they don’t value brilliance and genius sufficiently to overlook the deficits in your leadership. It’s not just that you won’t get people who aren’t like you: you will even lose people who are like you, but are unwilling to accept a leadership style which excludes and alienates.

Conclusion

It’s important, I think, to note that the two first points above require active work. Fostering a friendly environment, encouraging involvement, removing barriers: these are all important. They’re also (at least in theory) fairly simple, and don’t require hard choices and emotional investment. Arguably, the third point also requires work, in that, for many, there is an assumption that if your project is technically exciting enough (and, by extension, so is your leadership), then that’s enough: casting away this fallacy can be difficult to do.

Also, I’m aware that there’s something of an irony that I, a white, fairly neuro-typical, educated, middle-aged, Anglo adult male in a long-term heterosexual relationship, is writing about this, because many – too many! – of the leaders in this (as with many other spaces) are very much like me in many of their attributes. And I need to do a better job of following my own advice above. But I can try to model it, and I can shout about how important it is, and I can be an ally to those who want to change, and to those worst affected when that change does not come. I cannot pretend that inertia, a lack of change and a resistance to it, affects me as much as it does others, due to my position of privilege within society (and the communities about which I’ve been writing), but I can (and must) stand up when I can.

There are also times to be quiet and leave space for other voices (despite the fact that even the ability to grant that space is another example of privilege). I invite others to point me at other voices, and if I get enough feedback to do so, I’ll compile an article in the next few weeks designed to point at them from this blog.

In the meantime, one final piece of advice for leaders: be kind.

A User Advisory Council for the CCC

The CCC is currently working to create a User Advisory Council (UAC)

Disclaimer: the views expressed in this article (and this blog) do not necessarily reflect those of any of the organisations or companies mentioned, including my employer (Red Hat) or the Confidential Computing Consortium.

The Confidential Computing Consortium was officially formed in October 2019, nearly a year and a half ago now. Despite not setting out to be a high membership organisation, nor going out of its way to recruit members, there are, at time of writing, 9 Premier members (of which Red Hat, my employer, is one), 22 General members, and 3 Associate members. You can find a list of each here, and a brief analysis I did of their business interests a few weeks ago in this article: Review of CCC members by business interests.

The CCC has two major committees (beyond the Governing Board):

  • Technical Advisory Board (TAC) – this coordinates all technical areas in which the CCC is involved. It recommends whether software projects should be accepted into the CCC (no hardware projects have been introduced so far, thought it’s possible they might be), coordinates activities like special interest groups (we expect one on Attestation to start very soon), encourages work across projects, manages conversations with other technical bodies, and produces material such as the technical white paper listed here.
  • Outreach Committee – when we started the CCC, we decided against going with the title “Marketing Committee”, as we didn’t think it represented the work we hoped this committee would be doing, and this was a good decision. Though there are activities which might fall under this heading, the work of the Outreach Committee is much wider, including analyst and press relations, creation of other materials, community outreach, cross-project discussions, encouraging community discussions, event planning, webinar series and beyond.

These two committees have served the CCC well, but now that it’s fairly well established, and has a fairly broad industry membership of hardware manufacturers, CSPs, service providers and ISVs (see my other article), we decided that there was one set of interested parties who were not well-represented, and which the current organisational structure did not do a sufficient job of encouraging to get involved: end-users.

It’s all very well the industry doing amazing innovation, coming up with astonishingly well-designed, easy to integrate, security-optimised hardware-software systems for confidential computing if nobody wants to use them. Don’t get me wrong: we know from many conversations with organisations across multiple sectors that users absolutely want to be able to make use of TEEs and confidential computing. That is not that same, however, as understanding their use cases in detail and ensuring that what we – the members of the CCC, who are focussed mainly on creating services and software – actually provide what users need. These users are across many sectors – finance, government, healthcare, pharmaceutical, Edge, to name but a few – and their use cases and requirements are going to be different.

This is why the CCC is currently working to create a User Advisory Council (UAC). The details are being worked out at the moment, but the idea is that potential and existing users of confidential computing technologies should have a forum in which they can connect with the leaders in the space (which hopefully describes the CCC members), share their use cases, find out more about the projects which are part of the CCC, and even take a close look at those projects most relevant to them and their needs. This sort of engagement isn’t likely, on the whole, to require attendance at lots of meetings, or to have frequent input into the sorts of discussions which the TAC and the Outreach Committee typically consider, and the general feeling is that as we (the CCC) are aiming to service these users, we shouldn’t be asking them to pay for the privilege (!) of talking to us. The intention, then, is to allow a low bar for involvement in the UAC, and for there to be no membership fee required. That’s not to stop UAC members from joining the CCC as members if they wish – it would be a great outcome if some felt that they were so keen to become more involved that membership was appropriate – but there should be no expectation of that level of commitment.

I should be clear that the plans for the UAC are not complete yet, and some of the above may change. Nor should you consider this a formal announcement – I’m writing this article because I think it’s interesting, and because I believe that this is a vital next step in how those involved with confidential computing engages with the broader world, not because I represent the CCC in this context. But there’s always a danger that “cool” new technologies develop into something which fits only the fundamentally imaginary needs of technologists (and I’ll put my hand up and say that I’m one of those), rather than the actual needs of businesses and organisations which are struggling to operate around difficult issues in the real world. The User Advisory Council, if it works as we hope, should allow the techies (me, again) to hear from people and organisations about what they want our technologies to do, and to allow the CCC to steer its efforts in these directions.

Managing by exception

What I want, as a human, is interesting opportunities to apply my expertise

I’ve visited a family member this week (see why in last week’s article), in a way which is allowed under the UK Covid-19 lockdown rules as an “exceptional circumstance”. When governments and civil authorities are managing what their citizens are allowed to do, many jurisdictions (including the UK, where I live), follow the general principle of “Everything which is not forbidden is allowed“. This becomes complicated when you’re putting in (hopefully short-term) restrictions on civil liberties such as disallowing general movement to visit family and friends, but in the general case, it makes a lot of sense. It allows for the principle of “management by exception”: rather than taking the approach that you check that every journey is allowed, you look out for disallowed journeys (taking an unnecessary trip to a castle in the north of England, for instance) and (hopefully) punish those who have undertaken them.

What astonishes me about the world of IT – and security in particular – is how often we take the opposite approach. We record every single log result and transfer it across the network just in case there’s a problem. We have humans in the chain for every software build, checking that the correct versions of containers have been used. When what we should be doing is involving humans – the expensive parts of the chain – only when they’re needed, and only sending lots of results across the network – the expensive part of that chain – when the system which is generating the logs is under attack, or has registered a fault.

That’s not to say that we shouldn’t be recording information, but that we should be intelligent about how we use it: which means that we should be automating. Automation allows us to manage – that is, apply the expensive operations in a chain – only when it is relevant. Having a list of allowed container images, and then asking the developer why she has chosen a non-standard option, is so, so much cheaper for the organisation, not to mention more interesting for the container expert, than monitoring every single build. Allowing the system generating logs to increase the amount of information sent when it realises its under attack, or to send it a command to up what it sends when a possible problem is noticed remotely – is more efficient than the alternative.

The other thing I’m not saying is that we should just ignore information that’s generated in normal cases, where operation is “nominal“. The growing opportunities to apply AI/ML techniques to this to allow us to realise what is outside normal operation, and become more sensitive to when we need to apply those expensive components in a system where appropriate, makes a lot of sense. Sometimes, statistical sampling is required, where we can’t expect all of the data to be provided to those systems (in the remote logging case, for instance), or designs of distributed systems with remote agents need to be designed.

What I want, as a human, is interesting opportunities to apply my expertise, where I can make a difference, rather than routine problems (if you have routine problems, you have broader, more concerning issues) which don’t test me, and which don’t make a broader difference to how the systems and processes I’m involved with run. That won’t happen unless I can be part of an organisation where management by exception is the norm.

One final thing that I should be clear about is that I’m also not talking about an approach where “everything which isn’t explicitly allowed is disallowed” – that doesn’t sound like a great approach for security (I may not be a huge fan of the term zero-trust, but I’m not that opposed to it). It’s the results of the decisions that we care about, on the whole, and where we can manage it, we just have to automate, given the amount of information that’s becoming available. Even worse than not managing by exception is doing nothing with the data at all!

It doesn’t happen often, but let’s realise that, on this occasion, we have something to learn from our governments, and manage by exception.

Saving one life

Scratching the surface of the technologies which led to the saving of a life

When a loved one calls you from the bathroom at 3.30 in the morning, and you find them collapsed, unconscious on the floor, what does technology do for you? I’ve had the opportunity to consider this over the past few days after a family member was rushed to hospital for an emergency operation which, I’m very pleased to say, seems to have been completely successful. Without it, or if it had failed (the success rate is around 50%), they would, quite simply, be dead now.

We are eternally grateful to all those directly involved in my family member’s care, and to the NHS, which means that there are no bills to pay, just continued National Insurance taken as tax from our monthly pay packets, and which we begrudge not one jot. But I thought it might be worth spending a few minutes just scratching the surface of the sets of technologies which led to the saving of a life, from the obvious to the less obvious. I have missed out many: our lives are so complex and interconnected that it is impossible to list everything, and it is only when they are missing that we realise how it all fits together. But I want to say a huge – a HUGE – thank you to anyone who has ever been involved in any of the systems or technologies, and to ask you to remind yourself that even if you are seldom thanked, your work saves lives every day.

The obvious

  • The combined ECG and blood pressure unit attached to the patient which allows the ambulance crew to react quickly enough to save the patient’s life
  • The satellite navigation systems which guided the crew to the patient’s door
  • The landline which allowed the call to the emergency systems
  • The triage and dispatch system which prioritised the sending of the crew
  • The mobile phone system which allowed a remote member of the family to talk to the crew before they transported the patient

The visible (and audible)

  • The anaesthesiology and monitoring equipment which kept the patient alive during the operation
  • The various scanning equipment at the hospital which allowed a diagnosis to be reached in time
  • The sirens and flashing lights on the ambulances
  • The technology behind the training (increasingly delivered at least partly online) for all of those involved in the patient’s care

The invisible

  • The drugs and medicines used in the patient’s care
  • Equipment: batteries for ambulances, scalpels for operating theatres, paper for charts, keyboards, CPUs and motherboards for computers, soles for shoes, soap for hand-washing, paint for hospital corridors, pillows and pillow cases for beds and everything else that allows the healthcare system to keep running
  • The infrastructure to get fuel to the ambulances and into the cars, trains and buses which transported the medical staff to hospital
  • The maintenance schedules and processes for the ambulances
  • The processes behind the ordering of PPE for all involved
  • The supply chains which allowed those involved to access the tea, coffee, milk, sugar and other (hopefully legal) stimulants to keep staff going through the day and night
  • Staff timetabling software for everyone from cleaners to theatre managers, maintenance people to on-call surgeons
  • The music, art, videos, TV shows and other entertainment that kept everyone involved sufficiently energised to function

The infrastructure

  • Clean water
  • Roads
  • Electricity
  • Internet access and routing
  • Safety processes and culture in healthcare
  • … and everything else I’ve neglected to mention.

A final note

I hope it’s clear that I’m aware that the technology is all interconnected, and too complex to allow every piece to be noted: I’m sorry if I missed your piece out. The same, however, goes for the people. I come from a family containing some medical professionals and volunteers, and I’m aware of the sacrifices made not only by them, but also by the people around them who they know and love, and who see less of them than they might like, or how have to work around difficult shift patterns, or see them come back home after a long shift, worn out or traumatised by what they’ve seen and experienced. The same goes for ancillary workers and services worked in other, supporting industries.

I thank you all, both those involved directly and those involved in any of the technologies which save lives, those I’ve noted and those I’ve missed. In a few days, I hope to see a member of my family who, without your involvement, I would not ever be seeing again in this life. That is down to you.

Acting (and coding) your age

With seniority comes perks, but it also comes with responsibilities.

I dropped a post on LinkedIn a few days ago:

I’m now 50 years old and writing the most complex code in my career (for Enarx) in a language (Rust) that I only started learning 9 months ago and I’ve just finished the first draft of a book (for Wiley). Not sure what’s going on (and I wouldn’t have believed you if you’d told me this 25 years ago). #codingtips #writing #security #confidentialcomputing #rustlang

I’ve never received such attention. Lots of comments, lots of “likes” and other reactions, lots of people wanting to connect. It was supposed to be a throw-away comment, and I certainly had no intention either to boast or elicit sympathy: I am genuinely surprised by all of the facts mentioned – including my age, given that I feel that I’m somewhere between 23 and 31 (both primes, of course).

I remember in my mid- to late-twenties thinking “this business stuff is pretty simple: why don’t the oldies move aside and let talented youngsters[1] take over, or at least provide them some inspired advice?” Even at the time I realised that this was a little naive, and that there is something to be said for breadth of experience and decades of acquired knowledge, but I’m pretty certain that this set of questions has been asked by pretty much every generation since Ogg looked at the failings in his elders’ flint spear-head knapping technique and later got into a huff when his mum wouldn’t let him lead the mammoth hunt that afternoon.

Why expertise matters

Sadly (for young people), there really are benefits associated with praxis (actually doing things), even if you’ve absorbed all of the theory (and you haven’t, which is one of the things you learn with age). Of course, there’s also the Dunning-Kruger effect, which is a cognitive bias (Trust you? I can’t trust myself.) which leads the inexperienced to overestimate their own ability and experts to underestimate theirs.

Given this, there are some interesting and bizarre myths around about software/coding being a “young man’s game”. Leaving aside the glaring gender bias in that statement[2], this is rather odd. I know some extremely talented over-40 and over-50 software engineers, and I’m sure that you can think of quite a few if you try. There are probably a few factors at play here:

  • the lionisation of the “start-ups in the garage” young (mainly white) coders turning their company into “unicorn” trope;
  • the (over-)association of programming with mathematical ability, where a certain set of mathematicians are considered to have done their best work in their twenties;
  • the relative scarcity of roles (particularly in organisations which aren’t tech-specific) of “individual contributor” career tracks with roles where it’s possible to rise in seniority (and pay) without managing other people;
  • a possible tendency (which I’m positing without much evidence) for a sizeable proportion of senior software folks to take a broader view of the discipline and to move into architectural roles which are required by the industry but are difficult to perform without a good grounding in engineering basics.

In my case, I moved away from writing software maybe 15 years ago, and honestly never thought I’d do any serious coding again, only to discover a gap in the project I’m working on (Enarx) which nobody else had the time to fill, but which I felt merited some attention. That, and a continuous desire to learn new things, which had led me to starting to learn Rust, brought me to some serious programming, which I’ve really enjoyed.

We need old coders: people who have been around the block a few times, have made the mistakes and learned from them. People who can look at competing technologies and make reasoned decisions about which is the best fit for a project, rather than just choosing the newest and “coolest”[3].

Why old people should step aside

Having got all of the above out of my system, I’m now going to put forward an extremely important counter-argument. First, some context. I volunteer for the East of England Ambulance Service Trust as a Community First Responder, a role where I attend patients in (possible) emergency situations and work with ambulance staff, paramedics, etc.. I’ve become very interested in some of the theory around patient safety, which it turns out is currently being strongly influenced by lessons learned over the past few decades from transport safety, particularly aviation safety[5].

I need to do more study around this topic, as there are some really interesting lessons that can be applied to our sector (in fact, some are already be learned from our sector, particularly in how DevOps/WebOps respond to incidents), but there are two points that have really hit home for me this week, and which are relevant to the point at hand. They are specifically discussed with relation to high-intensity, stressful situations, but I think there’s broader applicability.

1. With experience comes expectation

While experience is enormously useful – bringing insights and knowledge that others may not have, or will find difficult to synthesise – it can also lead you down paths which are incorrect. If you’ve seen the same thing 99 times, you’re likely to assume that the 100th will be the same: bringing in other voices, including less experienced ones, allows other views to be considered, giving a better chance that the correct conclusion will be met. You increase diversity of opinion and allow alternatives to be brought into the mix. The less experience team members may be wrong, but from time to time, you’ll be wrong, and everyone will benefit from this. By allowing other people a voice, you’re also setting an example that speaking up and offering alternative views is not only acceptable, but valued. You and the team get to learn from each other, whether it’s when you’re wrong, or when you’re right, but you get to discuss with others how you came to your conclusions, and welcome their probing and questions around how you got there.

2. Sometimes you need to step aside to apply yourself elsewhere

Perhaps equally important is that sometimes, tempting as it may be to get your hands dirty and apply your expertise to a particular problem (particularly one which is possibly trivial to you), there are times when it’s best to step aside and let someone less experienced than you do it. Not only because they need the experience themselves, but also because your skills may be better applied at a systems level or dealing with other problems in other contexts (such as funding or resource management). The example sometimes given in healthcare is when a senior clinician arrives on scene at an incident: rather than their taking over the treatment of patients (however skilled the senior clinician may be), their role is to see the larger situation, to prioritise patients for treatment, assess risks to staff on scene, manage transport and the rest. Sometimes they may need to knuckle down and apply their clinical skills directly (much as senior techies may end up coding to meet a demo deadline, for instance), but most of the time, they are best deployed in stepping aside.

Conclusion

With seniority comes perks: getting to do the interesting stuff, taking decisions, having junior folks make the tea and bring the doughnuts in[6]. But it also comes with responsibilities: helping other people learn, seeing the bigger picture, giving less experienced team members the chance to make mistakes, removing barriers imposed by organisational hierarchy and getting the first round in at the pub[7]. Look back at what you were thinking about the beginning of your career, and give your successors (because they will be your successors) the chances that you were so keen for back then. Show them respect, and you (and your organisation) will benefit.


1 – I think that the “like me” is pretty implicit here, yes?

2 – which, sadly, reflects another bias in the market.

3 – there’s an important point here: many of us older folks love new shiny things just as much as the youngsters, and are aware of the problems of the old approaches and languages – but we’re also aware that there are risks and pain points associated with the new, which need to be taken into account[4].

4 – that really made me sound old, didn’t it?

5 – in large part influenced by the work of Martin Bromiley, a civil aviation pilot whose wife Elaine died in a “routine” operation in 2005 and who has worked (and is working) to help the health care sector transition to a no-blame, “just” culture around patient safety.

6 – this is a joke: if you have ever, ever find yourself in an office or team where this is the norm, and hierarchy shows in this sort of way, either get out or change that culture just as soon as you can. It’s toxic.

7 – I’m writing this in the middle of the UK’s second Covid-19 lockdown, and can barely remember what a “pub” or a “round” even is.