We’re all fallible. You’re fallible, he’s fallible, she’s fallible, I’m fallible*. We all get things wrong from time to time, and the generally accepted “modern” management approach is that it’s OK to fail – “fail early, fail often” – as long as you learn from your mistakes. In fact, there’s a growing view that if you’d don’t fail, you can’t learn – or that your learning will be slower, and restricted.
The problem with some fields – and IT security is one of them – is that failing can be a very bad thing, with lots of very unexpected consequences. This is particularly true for operational security, but the same can be the case for application, infrastructure or feature security. In fact, one of the few expected consequences is that call to visit your boss once things are over, so that you can find out how many days*** you still have left with your organisation. But if we are to be able to make mistakes**** and learn from them, we need to find ways to allow failure to happen without catastrophic consequences to our organisations (and our careers).
The first thing to be aware of is that we can learn from other people’s mistakes. There’s a famous aphorism, supposedly first said by George Santayana and often translated as “Those who cannot learn from history are doomed to repeat it.” I quite like the alternative: “History repeats itself because no one was listening the first time.” So, let’s listen, and let’s consider how to learn from other people’s mistakes (and our own). The classic way of thinking about this is by following “best practices”, but I have a couple of problems with this phrase. The first is that very rarely can you be certain that the context in which you’re operating is exactly the same as that of those who framed these practices. The other – possibly more important – is that “best” suggests the summit of possibilities: you can’t do better than best. But we all know that many practices can indeed be improved on. For that reason, I rather like the alternative, much-used at Intel Corporation, which is “BKMs”: Best Known Methods. This suggests that there may well be better approaches waiting to be discovered. It also talks about methods, which suggests to me more conscious activities than practices, which may become unconscious or uncritical followings of others.
What other opportunities are open to us to fail? Well, to return to a theme which is dear to my heart, we can – and must – discuss with those within our organisations who run the business what levels of risk are appropriate, and explain that we know that mistakes can occur, so how can we mitigate against them and work around them? And there’s the word “mitigate” – another approach is to consider managed degradation as one way to protect our organisations***** from the full impact of failure.
Another is to embrace methodologies which have failure as a key part of their philosophy. The most obvious is Agile Programming, which can be extended to other disciplines, and, when combined with DevOps, allows not only for fast failure but fast correction of failures. I plan to discuss DevOps – and DevSecOps, the practice of rolling security into DevOps – in more detail in a future post.
One last approach that springs to mind, and which should always be part of our arsenal, is defence in depth. We should be assured that if one element of a system fails, that’s not the end of the whole kit and caboodle******. That only works if we’ve thought about single points of failure, of course.
The approaches above are all well and good, but I’m not entirely convinced that any one of them – or a combination of them – gives us a complete enough picture that we can fully embrace “fail fast, fail often”. There are other pieces, too, including testing, monitoring, and organisational cultural change – an important and often overlooked element – that need to be considered, but it feels to me that we have some way to go, still. I’d be very interested to hear your thoughts and comments.
*my family is very clear on this point**.
**I’m trying to keep it from my manager.
***or if you’re very unlucky, minutes.
****amusingly, I first typed this word as “misteaks”. You’ve got to love those Freudian slips.
*****and hence ourselves.
******no excuse – I just love the phrase.
Alice, Eve and Bob - a security blog 