Tag: expertise

Managing by exception

What I want, as a human, is interesting opportunities to apply my expertise

I’ve visited a family member this week (see why in last week’s article), in a way which is allowed under the UK Covid-19 lockdown rules as an “exceptional circumstance”. When governments and civil authorities are managing what their citizens are allowed to do, many jurisdictions (including the UK, where I live), follow the general principle of “Everything which is not forbidden is allowed“. This becomes complicated when you’re putting in (hopefully short-term) restrictions on civil liberties such as disallowing general movement to visit family and friends, but in the general case, it makes a lot of sense. It allows for the principle of “management by exception”: rather than taking the approach that you check that every journey is allowed, you look out for disallowed journeys (taking an unnecessary trip to a castle in the north of England, for instance) and (hopefully) punish those who have undertaken them.

What astonishes me about the world of IT – and security in particular – is how often we take the opposite approach. We record every single log result and transfer it across the network just in case there’s a problem. We have humans in the chain for every software build, checking that the correct versions of containers have been used. When what we should be doing is involving humans – the expensive parts of the chain – only when they’re needed, and only sending lots of results across the network – the expensive part of that chain – when the system which is generating the logs is under attack, or has registered a fault.

That’s not to say that we shouldn’t be recording information, but that we should be intelligent about how we use it: which means that we should be automating. Automation allows us to manage – that is, apply the expensive operations in a chain – only when it is relevant. Having a list of allowed container images, and then asking the developer why she has chosen a non-standard option, is so, so much cheaper for the organisation, not to mention more interesting for the container expert, than monitoring every single build. Allowing the system generating logs to increase the amount of information sent when it realises its under attack, or to send it a command to up what it sends when a possible problem is noticed remotely – is more efficient than the alternative.

The other thing I’m not saying is that we should just ignore information that’s generated in normal cases, where operation is “nominal“. The growing opportunities to apply AI/ML techniques to this to allow us to realise what is outside normal operation, and become more sensitive to when we need to apply those expensive components in a system where appropriate, makes a lot of sense. Sometimes, statistical sampling is required, where we can’t expect all of the data to be provided to those systems (in the remote logging case, for instance), or designs of distributed systems with remote agents need to be designed.

What I want, as a human, is interesting opportunities to apply my expertise, where I can make a difference, rather than routine problems (if you have routine problems, you have broader, more concerning issues) which don’t test me, and which don’t make a broader difference to how the systems and processes I’m involved with run. That won’t happen unless I can be part of an organisation where management by exception is the norm.

One final thing that I should be clear about is that I’m also not talking about an approach where “everything which isn’t explicitly allowed is disallowed” – that doesn’t sound like a great approach for security (I may not be a huge fan of the term zero-trust, but I’m not that opposed to it). It’s the results of the decisions that we care about, on the whole, and where we can manage it, we just have to automate, given the amount of information that’s becoming available. Even worse than not managing by exception is doing nothing with the data at all!

It doesn’t happen often, but let’s realise that, on this occasion, we have something to learn from our governments, and manage by exception.

Acting (and coding) your age

With seniority comes perks, but it also comes with responsibilities.

I dropped a post on LinkedIn a few days ago:

I’m now 50 years old and writing the most complex code in my career (for Enarx) in a language (Rust) that I only started learning 9 months ago and I’ve just finished the first draft of a book (for Wiley). Not sure what’s going on (and I wouldn’t have believed you if you’d told me this 25 years ago). #codingtips #writing #security #confidentialcomputing #rustlang

I’ve never received such attention. Lots of comments, lots of “likes” and other reactions, lots of people wanting to connect. It was supposed to be a throw-away comment, and I certainly had no intention either to boast or elicit sympathy: I am genuinely surprised by all of the facts mentioned – including my age, given that I feel that I’m somewhere between 23 and 31 (both primes, of course).

I remember in my mid- to late-twenties thinking “this business stuff is pretty simple: why don’t the oldies move aside and let talented youngsters[1] take over, or at least provide them some inspired advice?” Even at the time I realised that this was a little naive, and that there is something to be said for breadth of experience and decades of acquired knowledge, but I’m pretty certain that this set of questions has been asked by pretty much every generation since Ogg looked at the failings in his elders’ flint spear-head knapping technique and later got into a huff when his mum wouldn’t let him lead the mammoth hunt that afternoon.

Why expertise matters

Sadly (for young people), there really are benefits associated with praxis (actually doing things), even if you’ve absorbed all of the theory (and you haven’t, which is one of the things you learn with age). Of course, there’s also the Dunning-Kruger effect, which is a cognitive bias (Trust you? I can’t trust myself.) which leads the inexperienced to overestimate their own ability and experts to underestimate theirs.

Given this, there are some interesting and bizarre myths around about software/coding being a “young man’s game”. Leaving aside the glaring gender bias in that statement[2], this is rather odd. I know some extremely talented over-40 and over-50 software engineers, and I’m sure that you can think of quite a few if you try. There are probably a few factors at play here:

  • the lionisation of the “start-ups in the garage” young (mainly white) coders turning their company into “unicorn” trope;
  • the (over-)association of programming with mathematical ability, where a certain set of mathematicians are considered to have done their best work in their twenties;
  • the relative scarcity of roles (particularly in organisations which aren’t tech-specific) of “individual contributor” career tracks with roles where it’s possible to rise in seniority (and pay) without managing other people;
  • a possible tendency (which I’m positing without much evidence) for a sizeable proportion of senior software folks to take a broader view of the discipline and to move into architectural roles which are required by the industry but are difficult to perform without a good grounding in engineering basics.

In my case, I moved away from writing software maybe 15 years ago, and honestly never thought I’d do any serious coding again, only to discover a gap in the project I’m working on (Enarx) which nobody else had the time to fill, but which I felt merited some attention. That, and a continuous desire to learn new things, which had led me to starting to learn Rust, brought me to some serious programming, which I’ve really enjoyed.

We need old coders: people who have been around the block a few times, have made the mistakes and learned from them. People who can look at competing technologies and make reasoned decisions about which is the best fit for a project, rather than just choosing the newest and “coolest”[3].

Why old people should step aside

Having got all of the above out of my system, I’m now going to put forward an extremely important counter-argument. First, some context. I volunteer for the East of England Ambulance Service Trust as a Community First Responder, a role where I attend patients in (possible) emergency situations and work with ambulance staff, paramedics, etc.. I’ve become very interested in some of the theory around patient safety, which it turns out is currently being strongly influenced by lessons learned over the past few decades from transport safety, particularly aviation safety[5].

I need to do more study around this topic, as there are some really interesting lessons that can be applied to our sector (in fact, some are already be learned from our sector, particularly in how DevOps/WebOps respond to incidents), but there are two points that have really hit home for me this week, and which are relevant to the point at hand. They are specifically discussed with relation to high-intensity, stressful situations, but I think there’s broader applicability.

1. With experience comes expectation

While experience is enormously useful – bringing insights and knowledge that others may not have, or will find difficult to synthesise – it can also lead you down paths which are incorrect. If you’ve seen the same thing 99 times, you’re likely to assume that the 100th will be the same: bringing in other voices, including less experienced ones, allows other views to be considered, giving a better chance that the correct conclusion will be met. You increase diversity of opinion and allow alternatives to be brought into the mix. The less experience team members may be wrong, but from time to time, you’ll be wrong, and everyone will benefit from this. By allowing other people a voice, you’re also setting an example that speaking up and offering alternative views is not only acceptable, but valued. You and the team get to learn from each other, whether it’s when you’re wrong, or when you’re right, but you get to discuss with others how you came to your conclusions, and welcome their probing and questions around how you got there.

2. Sometimes you need to step aside to apply yourself elsewhere

Perhaps equally important is that sometimes, tempting as it may be to get your hands dirty and apply your expertise to a particular problem (particularly one which is possibly trivial to you), there are times when it’s best to step aside and let someone less experienced than you do it. Not only because they need the experience themselves, but also because your skills may be better applied at a systems level or dealing with other problems in other contexts (such as funding or resource management). The example sometimes given in healthcare is when a senior clinician arrives on scene at an incident: rather than their taking over the treatment of patients (however skilled the senior clinician may be), their role is to see the larger situation, to prioritise patients for treatment, assess risks to staff on scene, manage transport and the rest. Sometimes they may need to knuckle down and apply their clinical skills directly (much as senior techies may end up coding to meet a demo deadline, for instance), but most of the time, they are best deployed in stepping aside.

Conclusion

With seniority comes perks: getting to do the interesting stuff, taking decisions, having junior folks make the tea and bring the doughnuts in[6]. But it also comes with responsibilities: helping other people learn, seeing the bigger picture, giving less experienced team members the chance to make mistakes, removing barriers imposed by organisational hierarchy and getting the first round in at the pub[7]. Look back at what you were thinking about the beginning of your career, and give your successors (because they will be your successors) the chances that you were so keen for back then. Show them respect, and you (and your organisation) will benefit.

1 – I think that the “like me” is pretty implicit here, yes?

2 – which, sadly, reflects another bias in the market.

3 – there’s an important point here: many of us older folks love new shiny things just as much as the youngsters, and are aware of the problems of the old approaches and languages – but we’re also aware that there are risks and pain points associated with the new, which need to be taken into account[4].

4 – that really made me sound old, didn’t it?

5 – in large part influenced by the work of Martin Bromiley, a civil aviation pilot whose wife Elaine died in a “routine” operation in 2005 and who has worked (and is working) to help the health care sector transition to a no-blame, “just” culture around patient safety.

6 – this is a joke: if you have ever, ever find yourself in an office or team where this is the norm, and hierarchy shows in this sort of way, either get out or change that culture just as soon as you can. It’s toxic.

7 – I’m writing this in the middle of the UK’s second Covid-19 lockdown, and can barely remember what a “pub” or a “round” even is.

Ignorance as a virtue: being proud to say “I don’t know”

“I am the wisest man alive, for I know one thing, and that is that I know nothing.” Socrates

In order to be considered an expert in any field, you have to spend a lot of time learning things.  In fact, I’d argue that one of the distinguishing traits of someone who is – or could become – an expert is their willingness and enthusiasm to learn, and keep learning.  The ability to communicate that knowledge is another of those traits: you can’t really be an expert if you have no way to communicate that knowledge.  Though that doesn’t mean that you need to be a great speaker, or even a great writer: by “communicate” I’m thinking of something much broader.  In the field of security and IT, that communication may be by architecture diagram, by code writing, by firewall rule instantiation, or by GUI, database or kernel module design, to name just a few examples.  These are all ways by which expertise can be communicated, instantiated or realised: the key is that the knowledge that has been gained is not contained, but can be externalised.

There’s another trait that, for me, betrays a true expert, and that’s the ability to say “I don’t know”.  And it’s difficult.  We enjoy and cultivate our expert status and other’s recognition of it: it’s part of our career progression, and it hits the “esteem” block in Maslow’s Hierarchy of Needs[1].  We like people asking our opinion, and we like being able to enlighten them: we take pride in our expertise, and why wouldn’t we?  We’ve earned it, after all, with all that hard graft and studying.  What’s more, we’ve all seen what happens when people get asked a question to which they don’t know the answer to something – they can become flustered, embarrassed, and they can be labelled stupid.*  Why would we want that for ourselves?

The problem, and very particularly in the security field, is that you’ll always get found out if you fake it.  In my experience, you’ll go into a customer meeting, for instance, and there’s either the sandal-wearing grey-beard, the recently-graduated genius or just the subject matter expert who’s been there for fifteen years and knows this specific topic better than … well, possibly anybody else on the planet, but certainly better than you.  They may not be there in the first meeting, but you can bet your bottom dollar*** that they’ll be in the second meeting, or the third – and you’ll get busted.  And when that happens, everything else you’ve said is called into question.  That may not seem fair, but that’s the way it goes.  Your credibility is dented, possibly irreparably.

The alternative to faking it is to accept that awkward question and simply to say, “I don’t know”.  You may want to give the question a moment’s thought – there have been times when I’ve plunged into an response and then stopped myself to admit that I just can’t give a full or knowledgeable answer, and when I could have saved myself bother by just pausing and considering it for a few seconds.  And you may want to follow up that initial acknowledgement of ignorance by saying that you know somebody else who does (if that happens to be true), or “I can find out” (if you think you can) or even “do you have any experts who might be able to help with that?”

This may not impress people who think you should know, but they’re generally either asking because they don’t (in which case they need a real answer) or because they’re trying to trip you up (in which case you don’t want to oblige them).  But it will impress those who are experts, because they know that nobody knows everything, and it’s much better to have that level of self-awareness than to dig yourself an enormous hole from which it’s difficult to recover.  But they’ll also understand, from your follow-up, that you want to find out: you want to learn.  And that is how one expert recognises another.

* it’s always annoyed me when people mock Donald Rumsfeld for pointing out that there are “unknown unknowns”: it’s probably one of the wisest soundbites in recent history**, for my money.

** and for an equivalently wise soundbite in ancient history, how about “I am the wisest man alive, for I know one thing, and that is that I know nothing”, by Socrates
*** other currencies and systems of exchange are available