Tag: open source

Uninterrupted power for the (CI)A

I’d really prefer not to have to restart every time we have a minor power cut.

Regular readers of my blog will know that I love the CIA triad – confidentiality, integrity and availability – as applied to security. Much of the time, I tend to talk about the first two, and spend little time on the third, but today I thought I’d address the balance a little bit by talking about UPSes – Uninterruptible Power Supplies. For anyone who was hoping for a trolling piece on giving extra powers to US Government agencies, it’s time to move along to another blog. And anyone who thinks I’d stoop to a deliberately misleading article title just to draw people into reading the blog … well, you’re here now, so you might as well read on (and you’re welcome to visit others of my articles such as Do I trust this package?, A cybersecurity tip from Hazzard County, and, of course, Defending our homes).

Years ago, when I was young and had more time on my hands, I ran an email server for my own interest (and accounts). This was fairly soon after we moved to ADSL, so I had an “always-on” connection to the Internet for the first time. I kept it on a pretty basic box behind a pretty basic firewall, and it served email pretty well. Except for when it went *thud*. And the reason it went *thud* was usually because of power fluctuations. We live in a village in the East Anglian (UK) countryside where the electricity supply, though usually OK, does go through periods where it just stops from time to time. Usually for under a minute, admittedly, but from the point of view of most computer systems, even a second of interruption is enough to turn them off. Usually, I could reboot the machine and, after thinking to itself for a while, it would come back up – but sometimes it wouldn’t. It was around this time, if I remember correctly, that I started getting into journalling file systems to reduce the chance of unrecoverable file system errors. Today, such file systems are custom-place: in those days, they weren’t.

Even when the box did come back up, if I was out of the house, or in the office for the day, on holiday, or travelling for business, I had a problem, which was that the machine was now down, and somebody needed to go and physically turn it on if I wanted to be able to access my email. What I needed was a way to provide uninterruptible power to the system if the electricity went off, and so I bought a UPS: an Uninterruptible Power System. A UPS is basically a box that sits between your power socket and your computer, and has a big battery in it which will keep your system going for a while in the event of a (short) power failure and the appropriate electronics to provide AC power out from the battery. Most will also have some sort of way to communicate with your system such as a USB port, and software which you can install to talk to it that your system can decide whether or not to shut itself down – when, for instance, the power has been off for long enough that the battery is about to give out. If you’re running a data centre, you’ll typically have lots of UPS boxes keeping your most important servers up while you wait for your back-up generators to kick in, but for my purposes, knowing that my email server would stay up for long enough that it would ride out short power drops, and be able to shut down gracefully if the power was out for longer, was enough: I have no interest in running my own generator.

That old UPS died probably 15 years ago, and I didn’t replace it, as I’d come to my senses and transferred my email accounts to a commercial provider, but over the weekend I bought a new one. I’m running more systems now, some of them are fairly expensive and really don’t like power fluctuations, and there are some which I’d really prefer not to have to restart every time we have a minor power cut. Here’s what I decided I wanted:

  • a product with good open source software support;
  • something which came in under £150;
  • something with enough “juice” (batter power) to tide 2-3 systems over a sub-minute power cut;
  • something with enough juice to keep one low-powered box running for a little longer than that, to allow it to coordinate shutting down the other boxes first, and then take itself down if required;
  • something with enough ports to keep a a couple of network switches up while the previous point happened (I thought ahead!);
  • Lithium Ion rather than Lead battery if possible.

I ended up buying an APC BX950UI, which meets all of my requirements apart from the last one: it turns out that only high-end UPS systems currently seem to have moved to Lithium Ion battery technology. There are two apparently well-maintained open source software suites that support APC UPS systems: apcupsd and nut, both of which are available for my Linux distribution of choice (Fedora). As it happens, they both also have support for Windows and Mac, so you can mix and match if needs be. I chose nut, which doesn’t explicitly list my model of UPS, but which supports most of the product lower priced product line with its usbhid-ups driver – I knew that I could move to apsupsd if this didn’t work out, but nut worked fine.

Set up wasn’t difficult, but required a little research (and borrowing a particular cable/lead from a kind techie friend…), and I plan to provide details of the steps I took in a separate article or articles to make things easier for people wishing do replicate something close to my set-up. However, I’m currently only using it on one system, so haven’t set up the coordinated shutdown configuration. I’ll wait till I’ve got it more set up before trying that.

What has this got to do with security? Well, I’m planning to allow VPN access to at least one of the boxes, and I don’t want it suddenly to disappear, leaving a “hole in the network”. I may well move to a central authentication mechanism for this and other home systems (if you’re interested, check out projects such as yubico-pam): and I want the box that provides that service to stay up. I’m also planning some home automation projects where access to systems from outside the network (to view cameras, for instance) will be a pain if things just go down: IoT devices may well just come back up in the event of a power failure, but the machines which coordinate them are less likely to do so.

Why I’m writing a book about trust

Who spends their holiday in their office? Authors.

Last week, I was on holiday. That is, I took 5 days off work in which I could have relaxed, read novels, watched TV, played lots of games, gone for long walks on the countryside and (in happier, less Covid-19 times) have spend time away from home, wrapped up warm enjoying a view of waves crashing onto a beach. Instead, I spent those 5 days – and fair amount of the 4 weekend days that bracketed them – squirreled away in my office, sitting at my computer. Which is rather similar to what I would have been doing if I hadn’t taken the time off.

The reason I did this is that I’m writing a book at the moment. Last week I managed to write well over 12,500 words of it, taking me to more than 80% of my projected word count (over 82% if you could bibliographic material), so the time felt well-spent. But why am I doing this in the first place?

I’m doing it at one level because I have a contract to do it. Around July-August last year (2019), I sent emails to a few (3-4) publishers pitching the idea for a book. I included a detailed Table of Contents, evidence that I’ve written before (including some links to this blog), and a bit about me, including a link to my LinkedIn profile. One of the replies I had (within 24 hours, to my amazement) was from Jim Minatel, a commissioning editor from Wiley Technology. Over a number of weeks, I talked to him and editors from other publishers, finally signing a contract with Wiley for a book on Trust in Computing and the Cloud, with a planned word count of 125,000 words. I’m not going to provide details of the contract, but I can say that :

  1. I don’t expect it to make me rich (which was never the point anyway);
  2. it has options for another book or books (I must be insane even to be considering the idea, but Jim and I have already have some preliminary conversations);
  3. there are clauses in it about film/movie rights (nobody, but nobody, is ever going to want to make or watch a film about this book: fascinating as it may be, Tom Clancy or John Grisham it is not).

This kind of explains why I spent last week closeted in my office, tapping away on a computer keyboard, but why did I get in touch with these publishers in the first place, pitching the idea of a book that is consuming a fair number of my non-work waking hours?

The basic reason is that I got cross. In fact, I got so annoyed about something that I went to a couple of people – my boss was one, a good friend with a publishing background was another – and announced that I planned to write a book. I was half hoping that they would dissuade me, but they both enthusiastically endorsed the idea, which meant that I had, at least in my head, now committed to doing it. This was in early May 2019, and it took me a couple of months to gather my thoughts, put together some materials and a find few candidate publishers before actually pitching the book to them and ending up with a contract.

But what actually got me to a position where I was cross enough about something to pitch an idea which would take up so much of my time and energy? The answer? I was tipped over the edge by hearing someone speak about trust in a way that made it clear that they had no idea what they were talking about. It was a session at a conference – I can remember the conference, but I can’t remember the session or the speaker – where the subject was security: IT security. This is my field, so that’s good. What wasn’t good was what the speaker said when speaking about trust: it didn’t hold together, it wasn’t consistent with how I felt about trust, and I didn’t feel that it was broadly applicable to computing or the Cloud.

This made me cross. And then I thought: why is there no consensus on what we mean by “trust”? Why do people talk about “zero trust” without really knowing what trust actually means? Why is there no literature on this subject that I’ve been thinking about for nearly 20 years? Why do I keep having conversations with people where they agree that trust is really interesting, but we discover that we don’t have a common starting point as there’s no theoretical underpinning describing exactly we’re discussing? Why are people deploying important workloads and designing business-critical systems without a good framework around what seems like a fundamental concept that everybody is always eager to include in their architectures? Why are we pushing forward with Confidential Computing when people don’t even understand the impact of the trust relationships which underlie it?

And I realised that I could keep asking these questions, and keep having these increasingly frustrating conversations, whilst waiting for somebody to publish some sort of definitive meisterwerk on the subject, or I could just admit that no-one was going to do it, and that I might get on and write something, even if it was never going to be the perfect treatment of what is, after all, a very complex subject.

And so that’s what I decided to do. I thought about what interests me in the field of trust in the realms of computing and the Cloud, about what I’ve heard people talk very badly about, and about what I’d had interesting conversations, decided that I might have something to say about all of those, and then put together a book structure. Wiley liked the idea, and asked me to flesh it out and then write something. It turns out that there’s loads of literature around human-to-human and human-to-organisational trust, and also on human-to-computer system trust, but very little on how computers can trust each other. Given how many organisations run much of their business in the Cloud these days, and the complex trust relationships that exist there, I wanted to write something about how to manage and understand these.

These are topics I’ve thought about (and, increasingly, written about) for around 20 years, since I did some research into the possibility of a PhD (which never materialised) in a related topic. They’ve stayed with me since, and I was involved in some theoretical and standards-based work around trust while involved with the ETSI NFV group nearly 10 years ago. I’m not pretending that I’m perfectly qualified to discuss this topic, but then again, I’m not sure that anybody is, and I feel that putting out some sort of book on this topic makes sense, if only to get the conversation started, and to give people an opportunity to converse with a shared language. The book starts with some theoretical underpinnings, looks at some of the technologies, what their implications are, the place of open source, the commercial and organisational impacts, and then suggests some future and frameworks. I hope to have the manuscript (well, typescript) completed and with Wiley by mid-spring (Northern Hemisphere) 2021: I don’t know when it’s actually likely to appear in print.

I hope people find it interesting, and that it acts as a catalyst for further discussion. I don’t expect it to be the last word on the subject – in fact I hope it’s not – but I do hope that it forces more people to realise that trust is really important in our world of computers, security and risk, and currently ill-understood. And if you happen to be a successful producer of Hollywood blockbusters, then I’m available to talk. Just as soon as I get these last couple of chapters submitted. ..

How open source builds distributed trust

Trust in open source is a positive feedback loop

This is an edited excerpt from my forthcoming book on Trust in Computing and the Cloud for Wiley, and leads on from a previous article I wrote called Trust & choosing open source.

In that article, I asked the question: what are we doing when we say “I trust open source software”? In reply, I suggested that what we are doing is making a determination that enough of the people who have written and tested it have similar requirements to mine, and that their expertise, combined, is such that the risk to my using the software is acceptable. I also introduced the idea of distributed trust.

The concept of distributing trust across a community is an application of the theory of the wisdom of the crowd, posited by Aristotle, where the assumption is that the opinions of many typically show more wisdom than the opinion of one, or a few. While demonstrably false in its simplest form in some situations – the most obvious example being examples of popular support for totalitarian regimes – this principle can provide a very effective mechanism for establishing certain information.

This distillation of collective experience allows what we refer as distributed trust, and is collected through numerous mechanisms on the Internet. Some, like TripAdvisor or Glassdoor, record information about organisations or the services they provide, while others, like UrbanSitter or LinkedIn, allow users to add information about specific people (see, for instance, LinkedIn’s “Recommendations” and “Skills & Endorsements” sections in individual’s profiles). The benefits that can accrue from of these examples are significantly increased by the network effect, as the number of possible connections between members increases exponentially as the number of members increases. Other examples of distributed trust include platforms like Twitter, where the number of followers that an account receives can be seen as a measure of their reputation, and even of their trustworthiness, a calculation which we should view with a strong degree of scepticism: indeed, the company Twitter felt that it had to address the social power of accounts with large numbers of followers and instituted a “verified accounts” mechanism to let people know “that an account of public interest is authentic”. Interestingly, the company had to suspend the service after problems related to users’ expectations of exactly what “verified” meant or implied: a classic case of differing understandings of context between different groups.

Where is the relevance to open source, then? The community aspect of open source is actually a driver towards building distributed trust. This is because once you become a part of the community around an open source project, you assume one or more of the roles that you start trusting once you say that you “trust” an open source project (see my previous article): examples include: architect, designer, developer, reviewer, technical writer, tester, deployer, bug reporter or bug fixer. The more involvement you have in a project, the more one becomes part of the community, which can, in time, become a community of practice. Jean Lave and Etienne Wenger introduced the concept of communities of practice in their book Situated Learning: Legitimate Peripheral Participation, where groups evolve into communities as their members share a passion and participate in shared activities, leading to their improving their skills and knowledge together. The core concept here is that as participants learn around a community of practice, they become members of it at the same time:

“Legitimate peripheral participation refers both to the development of knowledgeably skilled identities in practice and to the reproduction and transformation of communities of practice.”

Wenger further explored the concept of communities of practice, how they form, requirements for their health and how they encourage learning in Communities of Practice: Learning, meaning and identity. He identified negotiability of meaning (“why are we working together, what are we trying to achieve?”) as core to a community of practice, and noted that without engagement, imagination and alignment by individuals, communities of practice will not be robust.

We can align this with our views of how distributed trust is established and built: when you realise that your impact on open source can be equal to that of others, the distributed trust relationships that you hold to members of a community become less transitive (second- or third-hand or even more remote), and more immediate. You understand that the impact that you can have on the creation, maintenance, requirements and quality of the software which you are running can be the same as all of those other, previously anonymous contributors with whom you are now forming a community of practice, or whose existing community of practice you are joining. Then you yourself become part of a network of trust relationships which are distributed, but at less of a remove to that which you experience when buying and operating proprietary software. The process does not stop there, however: a common property of open source projects is cross-pollination, where developers from one project also work on others. This increases as the network effect of multiple open source projects allow reuse and dependencies on other projects to rise, and greater take-up across the entire set of projects.

It is easy to see why many open source contributors become open source enthusiasts or evangelists not just for a single project, but for open source as a whole. In fact, work by Mark Granovetter suggests that too many strong ties within communities can lead to cliques and stagnation, but weak ties provide for movement of ideas and trends around communities. This awareness of other projects and the communities that exist around them, and the flexibility of ideas across projects, leads to distributed trust being able to be extended (albeit with weaker assurances) beyond the direct or short-chain indirect relationships that contributors experience within projects of which they have immediate experience and out towards other projects where external observation or peripheral involvement shows that similar relationships exist between contributors. Put simply, the act of being involved in an open source project and building trust relationships through participation leads to stronger distributed trust towards similar open source projects, or just to other projects which are similarly open source.

What does this mean for each of us? It means that the more we get involved in open source, the more trust we can have in open source, as there will be a corresponding growth in the involvement – and therefore trust – of other people in open source. Trust in open source isn’t just a network effect: it’s a positive feedback loop!

My top 7 Cargo (Rust) commands

I heartily recommend that you spend some time investigating Cargo

This is the third article in my mini-series for Rust newbs like me. I’ve been using Rust for a little over six months now, and I’m far from an expert, but I have stumbled across many, many gotchas and learned many, many things along the way: things which I hope will be of use to record for those who are learning what is easily my favourite language. You can find my other excursions into Rust here:

I plan to write more over time, but this one is about Cargo. I’m ashamed to admit that I don’t use Cargo’s power as widely as a should, but researching this article has given me a better view into the command’s capabilities. In fact, I wasn’t even aware of some of the options available until I started looking in more detail. I’m going to assume a basic familiarity with Cargo – you have it installed, and you can create a package using cargo new <package>, for instance – so let’s work from there. I could have provided more (there are many options!), but here’s your “lucky 7”.

  1. cargo help <command> You can always find out more about a command with the --help option. The same goes for cargo itself: cargo --help will give you a quick intro to what’s out there. For more information on a particular command (more like a man page), you can try using the command “new”: cargo help new will give you extended information about cargo new, for instance. This behaviour is pretty typical for command line tools, particularly in the Linux/Unix world, but it’s very expressively implemented for cargo, and you can gain lots of quick information.
  2. cargo build --bin <target> What happens when you have multiple .rs files in your package, but you only want to build one of them? I have a package called test-and-try which I use for, well, testing and trying functionality, features, commands and crates, and which has around a dozen different files in it. By default, cargo build will try to build all of them, and as they’re often in various states of repair (some of them generating lots of warnings, some of them not even fully compiling), this can be a real pain. Instead, I place a section in my Cargo.toml file for each one thus:

[[bin]]
name = "warp-body"
path = "src/warp-body.rs"

I can then use cargo build --bin warp-body to build just this file (and any dependencies). I can then run it with a similar command: cargo run --bin warp-body

  1. cargo test I have an admission: I am not as assiduous about creating automatic tests in my Rust code as I ought to be. This is because I’m currently mainly writing PoC (Proof of Concept) rather than production code, and also because I’m lazy. Maybe changing this behaviour should be a New Year’s resolution, but when I do get round to writing tests, cargo is there to help me (as it is for you). All you need to do is add a line before the test code in your .rs file thus:

#[cfg(test)]

When you run cargo test, Cargo will automagically find these tests, run them, and tell you if you have problems. As with many of the commands here, you’ll find much more information online, but it’s particularly worth familiarising yourself with the basics of this capability in the relevant Rust By Example section.

  1. cargo search <query> This is one of the commands that I didn’t even know existed until I started researching this article – and which would have saved me so much time over the past few months if I’d known about it. What is does is search crates-io, Rust’s repository of public (and sometimes maintained) packages, and tells you which ones may be relevant (you can specify a different repository if you want, with the intuitively named --registry option). I’ve recently been doing some work on network protocols for non-String data, so I’ve been working with CBOR. Let’s see what happens if I use cargo search:

This is great! I can, of course, also combine with tools like grep to narrow down the search yet further like so: cargo search cbor --limit 70 | grep serde

  1. cargo tree Spoiler alert: this one may scare you. You’ve probably noticed, when you first build a new package, or when you add a new dependency, or just do a cargo clean and then cargo build, that you’ll see long list of crates being printed out as Cargo pulls them down from the relevant repository and then compiles them. How can you tell ahead of time, however, what will be pulled down, and what version it will be? More importantly, how can you know what other dependencies a new crate has pulled into your build? The answer is cargo tree. Just to warn you: for any marginally complex project, you can expect to have a lot of dependencies. I tried cargo tree | wc -l to count the number of dependent crates for a smallish project I’m working on, and got an answer of 350! I tried providing an example, but it didn’t display well, so I recommend that you try it yourself: be prepared for lots of output!
  2. cargo clippy If you tried running this and it didn’t work, that’s because I’ve cheated a little with these last two commands; you may have to install them explicitly (depending on your initial set-up). For this one, run cargo install clippy – you’ll be glad you did. Clippy is Rust’s linter – it goes through your code, looking at ways in which you can reduce and declutter it by removing or changing commands. I try to run cargo clippy before every git commit – partly because the git repositories to which I tend to be committing have automatic actions to reject files which need linting, and partly to keep my code generally more tidy. Let’s see an example:

Let’s face it: this isn’t a major issue (though clippy will also find errors, too, if you run it on non-compiling code), but it’s an easy fix, and so you might as well deal with it – either by removing the code, or prefixing the variable an underscore. As I plan to use this variable later on, but haven’t yet implemented the function to consume it, I plan to perform the latter fix.

  1. cargo readme Not the most earth-shattering of commands, this is another which is very useful (and which you may need to install explicitly, as with cargo clippy, above). If you add the relevant lines to your .rs files, you can output README files from Cargo. For instance, I have the following lines at the beginning of my main.rs file:

I’ll leave the output of the command cargo readme as an exercise to the reader, but it’s interesting to me that the Licence (or “License”, if you must) declaration is added. Use this to create simple documentation for your users, and make them happy with minimal effort (always a good approach!).

I’ve just scratched the surface of Cargo’s capabilities in this article: all of the commands I’ve listed above are actually way more powerful than I’ve described. I heartily recommend that you spend some time investigating Cargo and finding out how it can make your life better.

Do I trust this package?

The area of software supply chain management is growing in importance.

This isn’t one of those police dramas where a suspect parcel arrives at the precinct and someone realises just in time that it may be a bomb – what we’re talking about here is open source software packages (though the impact on your application may be similar if you’re not sufficiently suspicious). Open source software is everywhere these days – which is great – but how can you be sure that you should trust the software you’ve downloaded to do what you want? The area of software supply chain management – of which this discussion forms a part – is fairly newly visible in the industry, but is growing in importance. We’re going to consider a particular example.

There’s a huge conversation to be had here about what trust means (see my article “What is trust?” as a starting point, and I have a forthcoming book on Trust in Computing and the Cloud for Wiley), but let’s assume that you have a need for a library which provides some cryptographic protocol implementation. What do you need to know, and what are you choices? We’ll assume, for now, that you’ve already made what is almost certainly the right choice, and gone with an open source implementation (see many of my articles on this blog for why open source is just best for security), and that you don’t want to be building everything from source all the time: you need something stable and maintained. What should be your source of a new package?

Option 1 – use a vendor

There are many vendors out there now who provide open source software through a variety of mechanisms – typically subscription. Red Hat, my employer (see standard disclosure) is one of them. In this case, the vendor will typically stand behind the fitness for use of a particular package, provide patches, etc.. This is your easiest and best choice in many cases. There may be times, however, when you want to use a package which is not provided by a vendor, or not packaged by your vendor of choice: what do you do then? Equally, what decisions do vendors need to make about how to trust a package?

Option 2 – delve deeper

This is where things get complex. So complex, in fact, that I’m going to be examining them at some length in my book. For the purposes of this article, though, I’ll try to be brief. We’ll start with the assumption that there is a single maintainer of the package, and multiple contributors. The contributors provide code (and tests and documentation, etc.) to the project, and the maintainer provides builds – binaries/libraries – for you to consume, rather than your taking the source code and compiling it yourself (which is actually what a vendor is likely to do, though they still need to consider most of the points below). This is a library to provide cryptographic capabilities, so it’s fairly safe to assume that we care about its security. There are at least five specific areas which we need to consider in detail, all of them relying on the maintainer to a large degree (I’ve used the example of security here, though very similar considerations exist for almost any package): let’s look at the issues.

  1. build – how is the package that you are consuming created? Is the build process performed on a “clean” (that is, non-compromised) machine, with the appropriate compilers and libraries (there’s a turtles problem here!)? If the binary is created with untrusted tools, then how can we trust it at all, so what measures does the maintainer take to ensure the “cleanness” of the build environment? It would be great if the build process is documented as a repeatable build, so that those who want to check it can do so.
  2. integrity – this is related to build, in that we want to be sure that the source code inputs to the build process – the code coming, for instance, from a git repository – are what we expect. If, somehow, compromised code is injected into the build process, then we are in a very bad position. We want to know exactly which version of the source code is being used as the basis for the package we are consuming so that we can track features – and bugs. As above, having a repeatable build is a great bonus here.
  3. responsiveness – this is a measure of how responsive – or not – the maintainer is to changes. Generally, we want stable features, tied to known versions, but a quick response to bug and (in particular) security patches. If the maintainer doesn’t accept patches in a timely manner, then we need to worry about the security of our package. We should also be asking questions like, “is there a well-defined security disclosure of vulnerability management process?” (See my article Security disclosure or vulnerability management?), and, if so, “is it followed”?
  4. provenance – all code is not created equal, and one of the things of which a maintainer should be keeping track is the provenance of contributors. If a large amount of code in a part of the package which provides particularly sensitive features is suddenly submitted by an unknown contributor with a pseudonymous email address and no history of contributions of security functionality, this should raise alarm bells. On the other hand, if there is a group of contributors employed by a company with a history of open source contributions and well-reviewed code who submit a large patch, this is probably less troublesome. This is a difficult issue to manage, and there are typically no definite “OK” or “no-go” signs, but the maintainer’s awareness and management of contributors and their contributions is an important point to consider.
  5. expertise – this is the most tricky. You may have a maintainer who is excellent at managing all of the points above, but is just not an expert in certain aspects of the functionality of the contributed code. As a consumer of the package, however, I need to be sure that it is fit for purpose, and that may include, in the case of the security-related package we’re considering, being assured that the correct cryptographic primitives are used, that bounds-checking is enforced on byte streams, that proper key lengths are used or that constant time implementations are provided for particular primitives. This is very, very hard, and the job of maintainer can easily become a full-time one if they are acting as the expert for a large and/or complex project. Indeed, best practice in such cases is to have a team of trusted, experienced experts who work either as co-maintainers or as a senior advisory group for the project. Alternatively, having external people or organisations (such as industry bodies) perform audits of the project at critical junctures – when a major release is due, or when an important vulnerability is patched, for instance – allows the maintainer to share this responsibility. It’s important to note that the project does not become magically “secure” just because it’s open source (see Disbelieving the many eyes hypothesis), but that the community, when it comes together, can significantly improve the assurance that consumers of a project can have in the packages which is produces.

Once we consider these areas, we then need to work out how we measure and track each of them. Who is in a position to judge the extent to which any particular maintainer is fulfilling each of the areas? How much can we trust them? These are a set of complex issues, and one about which much more needs to be written, but I am passionate about exposing the importance of explicit trust in computing, particularly in open source. There is work going on around open source supply chain management, for instance the new (at time of writing Project Rekor – but there is lots of work still to be done.

Remember, though: when you take a package – whether library or executable – please consider what you’re consuming, what about it you can trust, and on what assurances that trust is founded.

Security is (only) subjective

What aspects of security does it provide?

This article covers ground covered in more detail within (but is not quite an excerpt from) my forthcoming book on Trust in Computing and the Cloud for Wiley.

In 1985, the US Department of Defense [sic] published the “Orange Book”[1], officially named Trusted Computer System Evaluation Criteria. It was a guide to how to create a “trusted system”, and was hugely influential within the IT and security industry as a whole. Eight years later, in 1993, Dorothy Denny published a devastating critique of the Orange Book called A New Paradigm for Trusted Systems[1]. It is a brilliant step-by-step analysis of why the approach taken by the DoD was fundamentally flawed. Denning starts:

“The current paradigm for trusted computer systems holds that trust is a property of a system. It is a property that can be formally modeled, specified, and verified. It can be designed into a system using a rigorous design methodology.”

Later, she explains why this just doesn’t work in the real world:

“The current paradigm of treating trust as a property is inconsistent, with the way trust is actually established in the world. It is not a property, but rather an assessment that is based on experience and shared through networks of people in the world-wide market. It is a declaration made by an observer, rather than a property of the observed.”

Demolishing the idea that trust is an inherent property of a system, and making it relational instead, changed the way that systems designed for security would be considered (and ushered in a new approach by the US Government and associated organisations, known was Common Criteria). Denning was writing about trust, but very similar issues exist around the concept of “security”. Too often, security is considered an inherent or intrinsic property of a system: “it’s secure”, someone will say, or “this fix will secure your computer”. It isn’t, and it won’t.

The first problem with such statements is that it’s not clear what “secure” means. There are a number of properties associated with systems that are relevant to security: three of the ones most often quoted are confidentiality, integrity and availability (which I discuss in more detail in the post The Other CIA: Confidentiality, Integrity and Availability). Specifying which of these you’re interested in removes the temptation just to say that something is “secure”, and if someone says, “it provides security”, we’re now in a position to start asking what that assertion actually means. Which aspects of security does it provide?

I also don’t think it makes sense to say that a system is “confidential” or “available” (there’s no obvious equivalent adjective for integrity – “integral” means something rather different): what we may be able to say is that it exhibits properties associated with confidentiality, integrity and availability, or better, that it has measures associated with it which are designed and intended to provide confidentiality, integrity and availability. These measures can be listed, examined and evaluated, hopefully against well-defined criteria.

This seems like a much better approach: not only have we addressed the suggestion that there is such as thing as “security” that we can apply to a system, but following Denning, we have also challenged the suggestion that it is inherent to – or in – a system. Instead, we have introduced the alternative approach of describing security-related properties which can be subjected to scrutiny by the users of the system. This allows the type of relational understanding of security that Denning was proposing, but it also raises the possibility of differing parties having different views of the security (or not) of a system, depending on who they are, and how it is going to be used.

It turns out, when you think about it, that this makes a lot of sense. A laptop which provides sufficient confidentiality, integrity and availability protection for the computing needs of my retired uncle may not provide sufficient protections for the uses to which an operative of a government security service might put it[3]. Equally, a system which a telecommunications company runs in a physically protected data centre may well be considered to have appropriate security protections, whereas the same system, attached to a pole somewhere on a residential street, might not. The measures applied to provide the protections associated with the properties (e.g. 128 bit AES encryption for the confidentiality) may be objectively specifiable, but the extent to which they provide “security” is not, because they are relative to specific requirements.

One last point, and it’s one which regular readers of my blog will be unsurprised to see: how can you assess the applicability of a system’s security properties to your requirements if it is not open? Open source helps significantly with security. Yes, there are assessment regimes to say that systems meet certain criteria – and sometimes these can be very helpful – but they are generally broad criteria, and difficult to apply to your specific use cases. Equally, most are just a starting point, and many such certified systems will require “exceptions” to be met in order to function in the real world, exceptions which require significant expertise to understand, judge and apply safely (that is, with appropriate levels of risk). If the system you want to use is open, then you, a party who you trust, or the wider community can evaluate the appropriateness of controls and measures, and make an informed decision about whether a system’s security properties are what you need. Without open source, this is impossible.

1 – it had an orange cover.

2 – Denning, Dorothy E. (1993) A New Paradigm for Trusted Systems [online]. Available at: https://www.researchgate.net/publication/234793347_A_New_Paradigm_for_Trusted_Systems [Accessed 3 Apr. 2020]

3 – I’m assuming that my uncle isn’t an operative of a government security service[4].

4 – or at least that his security needs are reduced in retirement[5].

5 – that is, if he has really retired…

Formal verification … or Ken Thompson?

“You can’t trust code that you did not totally create yourself” – Ken Thompson.

This article is an edited excerpt from my forthcoming book on Trust in Computing and the Cloud for Wiley.

How can we be sure that the code we’re running does what we think it does? One of the answers – or partial answers – to that question is “formal verification.” Formal verification is an important field of study, applying mathematics to computing, and it aims to start with proofs – at best, with an equivalent level of assurance to that of formal mathematical proofs – of the correctness of algorithms to be implemented in code to ensure that they perform the operations expected and set forth in a set of requirements. Though implementation of code can often fall down in the actual instructions created by a developer or set of developers – the programming – mistakes are equally possible at the level of the design of the code to be implemented in the first place, and so this must be a minimum step before looking at any actual implementations. What is more, these types of mistakes can be all the more hard to spot, as even if the developer has introduced no bugs in the work they have done, the implementation will be flawed by virtue of it being incorrectly defined in the first place. It is with an acknowledgement of this type of error, and an intention of reducing or eliminating it, that formal verification starts, but some areas go much further, with methods to examine concrete implementations and make statements about their correctness with regards to the algorithms which they are implementing.

Where we can make these work, they are extremely valuable, and the sort of places that they are applied are exactly where we would expect: for systems where security is paramount, and to prove the correctness of cryptographic designs and implementations. Another major focus of formal verification is software for safety systems, where the “correct” operation of the system – by which we mean “as designed and expected” – is vital. Examples might include oil refineries, fire suppression systems, nuclear power station management, aircraft flight systems and electrical grid management – unsurprisingly, given the composition of such systems, formal verification of hardware is also an important field of study. The practical application of formal verification methods to software is, however, more limited than we might like. As Alessandro Abate notes in a paper on formal verification of software:

“Two known shortcomings of standard techniques in formal verification are the limited capability to provide system-level assertions, and the scalability to large, complex models.”

To these shortcomings we can add another, extremely significant one: how sure can you be that what you are running is what you think you are running? Surely knowing what you are running is exactly why we write software, look at the source, and then compile it under our control? That, certainly, is the basic starting point for software that we care about.

The problem is arguably one of layers and dependencies, and was outlined by Ken Thompson, one of the founders or modern computing, in the lecture he gave at his acceptance of the Turing Award in 1983. It is short, stands as one of the establishing artefacts of computing security, and has weathered the tests of time: I have no hesitation in recommending that all readers of this blog read it: Reflections on Trusting Trust. In it, he describes how careful placing of malicious code in the C standard compiler could lead to vulnerabilities (his specific example is in account login code) which are not only undetectable by those without access to the source code, but also not removable. The final section of the paper is entitled “Moral”, and Thompson starts with these words:

“The moral is obvious. You can’t trust code that you did not totally create yourself. (Especially code from companies that employ people like me.) No amount of source-level verification or scrutiny will protect you from using untrusted code.”

However, as he goes on to point out, here is nothing special about the compiler:

“I could have picked on any program-handling program such as an assembler, a loader, or even hardware microcode. As the level of program gets lower, these bugs will be harder and harder to detect. A well-installed microcode bug will be almost impossible to detect.”

It is for this the reasons noted by Thompson that open source software – and hardware – is so vital to the field of computer security, and to our task of defining and understanding what “trust” means in the context of computing. Just relying on the “open source-ness” of your code is not enough: there is more work to be done in understanding your stack, the community and your requirements, but without the ability to look at the source code of all the layers of software and hardware on which you are running code, then you can have only reduced trust that what you are running is what you think you should be running, whether you have performed formal verification on it or not.

Why Enarx is open

It’s not just our coding that we do in the open.

When Nathaniel McCallum and I embarked on the project which is now called Enarx, we made one decision right at the beginning: the code for Enarx would be open source, a stance fully supported by our employer Red Hat (see standard disclaimer). All of it, and for ever. That’s a decision that we’ve not regretted at any point, and it’s something we stand behind. As soon as we had enough code for a demo, and were ready to show it, we created a repository on github and made it public. There’s a very small exception, which is that there are some details of upcoming chip features that are shared with us under NDA[1] where if we write code for them, publishing that code would be a breach of the NDA. But where this applied (which is rarely) we are absolutely clear with the various vendors that we intend to make the code open as soon as possible, and lobby them to release details as early as they can (which may be earlier than they might prefer), so that more experts can look over both their designs and our code.

Auditability and trust

This brings us to possibly the most important reasons for making Enarx open source: auditability and trust. Enarx is a security-related project, and I believe passionately not only that security should be done in the open, but that if anybody is actually going to trust their sensitive data, algorithms and workloads to a piece of software, then they want to be in a position where as many experts as possible have looked at it, scrutinised it, criticised it and improved it: whether that is the people running the software, their employees, contractors or (even better) the wider security community. The more people who check the code, the happier you should be to trust it. This is important for any piece of security software, but vital for software such as Enarx which is designed to protect your more most sensitive workloads.

Bug-catching

There are bugs in Enarx. I know: I’m writing some of the code[2] and I found one yesterday (which I’d put in), just as I was about to give a demo[3]. It is very, very difficult to write perfect code, and we know that if we make our source open, then more people can help us fix issues.

Commonwealth

For Nathaniel and me, open source is an ethical issue, and we make no apologies for that. I think it’s the same for most, if not all, of the team working on Enarx. This include a number of Red Hat employees (see standard disclaimer), so shouldn’t come as a surprise, but we have non-Red Hat contributors from a number of backgrounds, and we feel that Enarx should be a Common Good, and contribute to the commonwealth of intellectual property out there.

More brain power

Making something open source doesn’t just make it easier to fix bugs: it can improve the quality of what you produce in general. The more brain power you have to apply to the problem, but better your chances of making something great – assuming that the brain power is applied efficiently (not always an easy task!). We had a design meeting yesterday where one of the participants said towards the end, “I’m sure I could implement some of this, but don’t know a huge amount about this topic, and I’m worried that I’m not contributing to this discussion.” In fact, they had, by asking questions and clarifying some points, and we assured them that we wanted to include experienced, senior developers for their expertise and knowledge, and to pull out assumptions and to validate the design, and not because we expected everybody to be experts in all parts of the project. Having bright people around, involved in design and coding, spreads expertise and knowledge, and helps keep the work from becoming an insulated, isolated “ivory tower” construction, understood by few, and almost impossible to validate.

Not just code

It’s not just our coding that we do in the open. We manage our architecture in the open, our design meetings, our protocol design, our design methodology[4], our documentation, our bug-tracking, our chat, our CI/CD processes: all of it is open. The one exception is our vulnerability management process, which needs to have the opportunity for confidential exposure for a limited time.

We also take diversity seriously, and the project contributors are subject to the Contributor Covenant Code of Conduct.

In short, Enarx is an open project. I’m sure we could do better, and we’ll strive for that, but our underlying principles are that open is good in general, and vital for security. If you agree, please come and visit!

1 – Non-Disclosure Agreement.

2 – to the surprise of many of the team, including myself. At least it’s not in Perl.

3 – I fixed it. Admittedly after the demo.

4 – we’ve just moved to a Sprint pattern – the details of which we designed and agreed in the open.

Enarx news: WebAssembly + SGX

We can now run a WebAssembly workload in a TEE

Regular readers will know that I’m one of the co-founders for the Enarx project, and write about our progress fairly frequently. You’ll find some more information in these articles (newest first):

I won’t spend time going over what the project is about here, as you can read more in the articles above, and lots more over at our GitHub repository (https://enarx.io), but aim of the project is to allow you run sensitive workloads in Trusted Execution Environments (TEEs) from various silicon vendors. The runtime we’ll be providing is WebAssembly (using the WASI interface), and the TEEs that we’re targetting to start with are AMD’s SEV and Intel’s SGX – though you, the client running the workload, shouldn’t care which is being used at any particular point, as we abstract all of that away.

It’s a complicated architecture, with lots of moving parts and a fairly complex full-stack architecture. Things have been moving really fast recently, partly due to a number of new contributors to the project (which recently reached 150 stars on our main GitHub repository), and one of the important developments is the production of a new component architecture, included below.

I’ve written before about how important it is to have architectural diagrams for projects (and particularly open source projects), and so I’m really pleased that we have this updated – and much more detailed – version of the diagram. I won’t go into it in detail here, but the only trusted components (from the point of view of Enarx and a client using Enarx) are those in green: the Enarx client agent, the Attestation measurement database and the TEE instance containing the Application, Wasm, Shim, etc., which we call the Keep.

It’s only just over a couple of months since we announced our most recent milestone: running binaries within TEEs, including both SEV and SGX. This was a huge deal for us, but then, at the end of last week, one of the team, Daiki Ueno, managed to get a WebAssembly binary loaded using the Wasm/WASI layers and to run it. In other words, we can now run a WebAssembly workload in a TEE. Over the past 10 minutes, I’ve just managed to replicate that on my own machine (for my own interest, not that I doubted the results!). Circled, below, are the components which are involved in this feat.

I should make it clear that none of these components is complete or stable – they are all still very much in development, and, in fact, even the running test yielded around 40 lines for error messages! – but the fact that we’ve got a significant part of the stack working together is an enormous deal. The Keep loader creates an SGX TEE instance, loads the Shim and the Wasm/WASI components into the Keep and starts them running. The App Loader (one of the Wasm components) loads the Application, and it runs.

It’s not just an enormous deal due to the amount of work that it’s taken and the great teamwork that’s been evident throughout – though those are true as well – but also because it means we’re getting much closer to having something that people can try out for themselves. The steps to just replicating the test were lengthy and complex, and my next step is to see if I can make make some changes and create my own test, which may be quite a marathon, but I’d like to stress that my technical expertise lies closer to the application layer than the low-level pieces, which means that if I can make this work now (with a fair amount of guidance), then we shouldn’t be too far off being in a position where pretty much anyone with a decent knowledge of the various pieces can do the same.

We’re still quite a way off being able to provide a simple file with 5-10 steps to creating and running your own workload in a Keep, but that milestone suddenly feels like it’s in sight. In the meantime, there’s loads of design work, documentation, infrastructure automation, testing and good old development to be done: fancy joining us?

8 tips on how to do open source (badly)

Generally, a “build successful” or “build failed” message should be sufficient.

A while ago, I published my wildly popular[1] article How not to make a cup of tea. Casting around for something to write this week, it occurred to me that I might write about something that I believe is almost as important as to world peace, the forward march of progress and brotherly/sisterly love: open source projects. There are so many guides out there around how to create an open source project that it’s become almost too easy to start a new, successful, community-supported one. I think it’s time to redress the balance, and give you some clues about how not to do it.

Throw it over the wall

You know how it is: you’re a large corporation, you’ve had a team of developers working on a project for several years now, and you’re very happy with it. It’s quite expensive to maintain and improve the code, but luckily, it’s occurred to you that other people might want to use it – or bits of it. And recently, some of your customers have complained that it’s difficult to get improvements and new features, and partners complain that your APIs are obscure, ill-defined and subject to undocumented change.

Then you hit on a brilliant idea: why not open source it and tell them their worries are over? All you need to do is take the existing code, create a GitHub or GitLab[2] project (preferably under a Game of Thrones-themed username that happens to belong to one of your developers), make a public repository, upload all of the code, and put out a press release announcing: a) the availability of your project; and b) what a great open source citizen you are.

People will be falling over themselves to contribute to your project, and you’ll suddenly have hundreds of developers basically working for you for free, providing new features and bug fixes!

Keep a tight grip on the project

There’s a danger, however, when you make your project open source, that other people will think that they have a right to make changes to it. The way it’s supposed to work is that your product manager comes up with a bunch of new features that need implementing, and posts them as issues on the repository. Your lucky new contributors then get to write code to satisfy the new features, you get to test them, and then, if they’re OK, you can accept them into the project! Free development! Sometimes, if you’re lucky, customers or partners (the only two parties of any importance in this process, apart from you) will raise issues on the project repository, which, when appropriately subjected to your standard waterfall development process and vetted by the appropriate product managers, can be accepted as “approved” issues and earmarked for inclusion into the project.

There’s a danger that, as you’ve made the project code open source, some people might see this as an excuse to write irrelevant features and fixes for bugs that none of your customers have noticed (and therefore, you can safely assume, don’t care about). Clearly, in this case, you should reject close any issues related to such features or fixes, and reject (or just ignore) any related patched.

Worse yet, you’ll sometimes find developers[2] complaining about how you run the project. They may “fork” the project, making their own version. If they do this, beware setting your legal department on them. There’s a possibility that as your project is open source, they might be able to argue that they have a right to create a new version. Much better is to set your PR department on them, rubbishing the new project, launching ad hominem[4] attacks on them and showing everybody that you hold the moral high ground.

Embrace diversity (in licensing)

There may be some in your organisation who say that an open source project doesn’t need a licence[5]. Do not listen to their siren song: they are wrong. What your open source project needs is lots of licences! Let your developers choose their favourite for each file they touch, or, even better, let the project manager choose. The OSI maintains a useful list, but consider this just a starting point: why not liberally sprinkle different licences through your project? Diversity, we keep hearing, is good, so why not apply it to your open source project code?

Avoid documentation

Some people suggest that documentation can be useful for open source projects, and they are right. What few of them seem to understand is that their expectations for the type of documentation are likely to be skewed by their previous experience. You, on the other hand, have a wealth of internal project documentation and external product marketing material that you accrued before deciding to make your project open source: this is great news. All you need to do is to create a “docs” folder and copy all of the PDF files there. Don’t forget to update them whenever you do a new product version!

Avoid tooling

All you need is code (and docs – see above). Your internal developers have carefully constructed and maintained build environments, and they should therefore have no problems building and testing any parts of the project. Much of this tooling, being internal, could be considered proprietary, and the details must therefore be kept confidential. Any truly useful contributors will be able to work out everything they need for themselves, and shouldn’t need any help, so providing any information about how actually to build the code in the repository is basically redundant: don’t bother.

An alternative, for more “expert” organisations, is to provide build environments which allow contributors to batch builds to see if they compile or not (whilst avoiding giving them access to the tooling themselves, obviously). While this can work, beware providing too much in the way of output for the developer/tester, as this might expose confidential information. Generally, a “build successful” or “build failed” message should be sufficient.

Avoid diagrams

Despite what some people think, diagrams are dangerous. They can give away too much information about your underlying assumptions for the project (and, therefore, the product you’re selling which is based on it), and serious developers should be able to divine all they need from the 1,500 source files that you’ve deposited in the repository anyway.

A few “marketiture” diagrams from earlier iterations of the project may be acceptable, but only if they are somewhat outdated and don’t provide any real insight into the existing structure of the code.

Keep quiet

Sometimes, contributors – or those hoping to become contributors, should you smile upon their requests – will ask questions. In the old days, these questions tended to be sent to email lists[6], where they could be safely ignored (unless they were from an important customer). More recently, there are other channels that developers expect to use to contact members of the project team, such as issues or chat.

It’s important that you remember that you have no responsibility or duty to these external contributors: they are supposed to be helping you. What’s more, your internal developers will be too busy writing code to answer the sort of uninformed queries that are likely to be raised (and as for so-called “vulnerability disclosures“, you can just fix those in your internal version of the product, or at least reassure your customers that you have). Given that most open source projects will come with an issue database, and possibly even a chat channel, what should you do?

The answer is simple: fall back to email. Insist that the only channel which is guaranteed attention[7] is email. Don’t make the mistake of failing to create an email address to which people can send queries: contributors are much more likely to forget that they’re expecting an answer to an email if they get generic auto-response (“Thanks for your email: a member of the team should get back to you shortly”) than if they receive a bounce message. Oh, and close any issues that people create without your permission for “failing to follow project process”[8].

Post huge commits

Nobody[9] wants to have to keep track of lots of tiny changes to code (or, worse, documentation – see above), or have contributors picking holes in it. There’s a useful way to avoid much of this, however, which is to train your developers only to post large commits to the open source project. You need to ensure that your internal developers understand that code should only be posted to the external repository when the project team (or, more specifically, the product team) deems it ready. Don’t be tempted to use the open source repository as your version control system: you should have perfectly good processes internally, and, with a bit of automation, you can set them up to copy batches of updates to the external repository on a regular basis[10].

1 – well, lots of you read it, so I’m assuming you like it.

2 – other public repositories may be available, but you won’t have heard of them, so why should you care?

3 – the canonical term for such people is “whingers”: they are invariably “experts”. According to them (and their 20 years of security experience, etc., etc.).

4 – or ad mulierem – please don’t be sexist in your attacks.

5 – or license, depending on your spelling choice.

6 – where they existed – a wise organisation could carefully avoid creating them.

7 – “attention” can include a “delete all” filter.

8 – you don’t actually need to define what the process is anywhere, obviously.

9 – in your product organisation, at least.

10 – note that “regular” does not equate to “frequent”. Aim for a cadence of once every month or two.