Confidential computing – the new HTTPS?

Security by default hasn’t arrived yet.

Over the past few years, it’s become difficult to find a website which is just “http://…”.  This is because the industry has finally realised that security on the web is “a thing”, and also because it has become easy for both servers and clients to set up and use HTTPS connections.  A similar shift may be on its way in computing across cloud, edge, IoT, blockchain, AI/ML and beyond.  We’ve know for a long time that we should encrypt data at rest (in storage) and in transit (on the network), but encrypting it in use (while processing) has been difficult and expensive.  Confidential computing – providing this type of protection for data and algorithms in use, using hardware capabilities such as Trusted Execution Environments (TEEs) – protects data on hosted system or vulnerable environments.

I’ve written several times about TEEs and, of course, the Enarx project of which I’m a co-founder with Nathaniel McCallum (see Enarx for everyone (a quest) and Enarx goes multi-platform for examples).  Enarx uses TEEs, and provides a platform- and language-independent deployment platform to allow you safely to deploy sensitively applications or components (such as micro-services) onto hosts that you don’t trust.  Enarx is, of course, completely open source (we’re using the Apache 2.0 licence, for those with an interest).  Being able to run workloads on hosts that you don’t trust is the promise of confidential computing, which extends normal practice for sensitive data at rest and in transit to data in use:

  • storage: you encrypt your data at rest because you don’t fully trust the underlying storage infrastructure;
  • networking: you encrypt your data in transit because you don’t fully trust the underlying network infrastructure;
  • compute: you encrypt your data in use because you don’t fully trust the underlying compute infrastructure.

I’ve got a lot to say about trust, and the word “fully” in the statements above is important (I actually added it on re-reading what I’d written).  In each case, you have to trust the underlying infrastructure to some degree, whether it’s to deliver your packets or store your blocks, for instance.  In the case of the compute infrastructure, you’re going to have to trust the CPU and associate firmware, just because you can’t really do computing without trusting them (there are techniques such as homomorphic encryption which are beginning to offer some opportunities here, but they’re limited, and the technology still immature).

Questions sometimes come up about whether you should fully trust CPUs, given some of the security problems that have been found with them and also whether they are fully secure against physical attacks on the host in which they reside.

The answer to both questions is “no”, but this is the best technology we currently have available at scale and at a price point to make it generally deployable.  To address the second question, nobody is pretending that this (or any other technology) is fully secure: what we need to do is consider our threat model and decide whether TEEs (in this case) provide sufficient security for our specific requirements.  In terms of the first question, the model that Enarx adopts is to allow decisions to be made at deployment time as to whether you trust a particular set of CPU.  So, for example, of vendor Q’s generation R chips are found to contain a vulnerability, it will be easy to say “refuse to deploy my workloads to R-type CPUs from Q, but continue to deploy to S-type, T-type and U-type chips from Q and any CPUs from vendors P, M and N.”


5 security tips from Santa

Have you been naughty or nice this year?

If you’re reading this in 2019, it’s less than a month to Christmas (as celebrated according to the Western Christian calendar), or Christmas has just passed.  Let’s assume that it’s the former, and that, like all children and IT professionals, it’s time to write your letter to Santa/St Nick/Father Christmas.  Don’t forget, those who have been good get nice presents, and those who don’t get coal.  Coal is not a clean-burning fuel these days, and with climate change well and truly upon us[1], you don’t want to be going for the latter option.

Think back to all of the good security practices you’ve adopted over the past 11 or so months.  And then think back to all the bad security practices you’ve adopted when you should have been doing the right thing.  Oh, dear.  It’s not looking good for you, is it?

Here’s the good news, though: unless you’re reading this very, very close to Christmas itself[2], then there’s time to make amends.  Here’s a list of useful security tips and practices that Santa follows, and which are therefore bound to put you on his “good” side.

Use a password manager

Santa is very careful with his passwords.  Here’s a little secret: from time to time, rather than have his elves handcraft every little present, he sources his gifts from other parties.  I’m not suggesting that he pays market rates (he’s ordering in bulk, and he has a very, very good credit rating), but he uses lots of different suppliers, and he’s aware that not all of them take security as seriously as he does.  He doesn’t want all of his account logins to be leaked if one of his suppliers is hacked, so he uses separate passwords for each account.  Now, Santa, being Santa, could remember all of these details if he wanted to, and even generate passwords that meet all the relevant complexity requirements for each site, but he uses an open source password manager for safety, and for succession planning[3].

Manage personal information properly

You may work for a large company, organisation or government, and you may think that you have lots of customers and associated data, but consider Santa.  He manages, or has managed, names, dates of birth, addresses, hobby, shoe sizes, colour preferences and other personal data for literally every person on Earth.  That’s an awful lot of sensitive data, and it needs to be protected.  When people grow too old for presents from Santa[4], he needs to delete their data securely.  Santa may well have been the archetypal GDPR Data Controller, and he needs to be very careful who and what can access the data that he holds.  Of course, he encrypts all the data, and is very careful about key management.  He’s also very aware of the dangers associated with Cold Boot Attacks (given the average temperature around his relevance), so he ensures that data is properly wiped before shutdown.

Measure and mitigate risk

Santa knows all about risk.  He has complex systems for ordering, fulfilment, travel planning, logistics and delivery that are the envy of most of the world.  He understands what impact failure in any particular part of the supply chain can have on his customers: mainly children and IT professionals.  He quantifies risk, recalculating on a regular basis to ensure that he is up to date with possible vulnerabilities, and ready with mitigations.

Patch frequently, but carefully

Santa absolutely cannot afford for his systems to go down, particularly around his most busy period.  He has established processes to ensure that the concerns of security are balanced with the needs of the business[5].  He knows that sometimes, business continuity must take priority, and that on other occasions, the impact of a security breach would be so major that patches just have to be applied.  He tells people what he wants, and listens to their views, taking them into account where he can. In other words, he embraces open management, delegating decisions, where possible, to the sets of people who are best positioned to make the call, and only intervenes when asked for an executive decision, or when exceptions arise.  Santa is a very enlightened manager.

Embrace diversity

One of the useful benefits of running a global operation is that Santa values diversity.  Old or young (at heart), male, female or gender-neutral, neuro-typical or neuro-diverse, of whatever culture, sexuality, race, ability, creed or nose-colour, Santa takes into account his stakeholders and their views on what might go wrong.  What a fantastic set of viewpoints Santa has available to him.  And, for an Aging White Guy, he’s surprisingly hip to the opportunities for security practices that a wide and diverse set of opinions and experiences can bring[6].

Summary

Here’s my advice.  Be like Santa, and adopt at least some of his security practices yourself.  You’ll have a much better opportunity of getting onto his good side, and that’s going to go down well not just with Santa, but also your employer, who is just certain to give you a nice bonus, right?  And if not, well, it’s not too late to write that letter directly to Santa himself.


1 – if you have a problem with this statement, then either you need to find another blog, or you’re reading this in the far future, where all our climate problems have been solved. I hope.

2 – or you dwell in one of those cultures where Santa visits quite early in December.

3 – a high-flying goose in the face can do terrible damage to a fast-moving reindeer, and if the sleigh were to crash, what then…?

4 – not me!

5 – Santa doesn’t refer to it as a “business”, but he’s happy for us to call it that so that we can model our own experience on his.  He’s nice like that.

6 – though Santa would never use the phrase “hip to the opportunities”.  He’s way too cool for that.

Of projects, products and (security) community

Not all open source is created (and maintained) equal.

Open source is a  good thing.  Open source is a particularly good thing for security.  I’ve written about this before (notably in Disbelieving the many eyes hypothesis and The commonwealth of Open Source), and I’m going to keep writing about it.  In this article, however, I want to talk a little more about a feature of open source which is arguably both a possible disadvantage and a benefit: the difference between a project and a product.  I’ll come down firmly on one side (spoiler alert: for organisations, it’s “product”), but I’d like to start with a little disclaimer.  I am employed by Red Hat, and we are a company which makes money from supporting open source.  I believe this is a good thing, and I approve of the model that we use, but I wanted to flag any potential bias early in the article.

The main reason that open source is good for security is that you can see what’s going on when there’s a problem, and you have a chance to fix it.  Or, more realistically, unless you’re a security professional with particular expertise in the open source project in which the problem arises, somebody else has a chance to fix it. We hope that there are sufficient security folks with the required expertise to fix security problems and vulnerabilities in software projects about which we care.

It’s a little more complex than that, however.  As an organisation, there are two main ways to consume open source:

  • as a project: you take the code, choose which version to use, compile it yourself, test it and then manage it.
  • as a product: a vendor takes the project, choose which version to package, compiles it, tests it, and then sells support for the package, typically including docs, patching and updates.

Now, there’s no denying that consuming a project “raw” gives you more options.  You can track the latest version, compiling and testing as you go, and you can take security patches more quickly than the product version may supply them, selecting those which seem most appropriate for your business and use cases.  On the whole, this seems like a good thing.  There are, however, downsides which are specific to security.  These include:

  1. some security fixes come with an embargo, to which only a small number of organisations (typically the vendors) have access.  Although you may get access to fixes at the same time as the wider ecosystem, you will need to check and test these (unless you blindly apply them – don’t do that), which will already have been performed by the vendors.
  2. the huge temptation to make changes to the code that don’t necessarily – or immediately – make it into the upstream project means that you are likely to be running a fork of the code.  Even if you do manage to get these upstream in time, during the period that you’re running the changes but they’re not upstream, you run a major risk that any security patches will not be immediately applicable to your version (this is, of course, true for non-security patches, but security patches are typically more urgent).  One option, of course, if you believe that your version is likely to consumed by others, is to make an official fork of project, and try to encourage a community to grow around that, but in the end, you will still have to decide whether to support the new version internally or externally.
  3. unless you ensure that all instances of the software are running the same version in your deployment, any back-porting of security fixes to older versions will require you to invest in security expertise equal or close to equal to that of the people who created the fix in the first place.  In this case, you are giving up the “commonwealth” benefit of open source, as you need to pay experts who duplicate the skills of the community.

What you are basically doing, by choosing to deploy a project rather than a product is taking the decision to do internal productisation of the project.  You lose not only the commonwealth benefit of security fixes, but also the significant economies of scale that are intrinsic to the vendor-supported product model.  There may also be economies of scope that you miss: many vendors will have multiple products that they support, and will be able to apply security expertise across those products in ways which may not be possible for an organisation whose core focus is not on product support.

These economies are reflected in another possible benefit to the commonwealth of using a vendor: the very fact that multiple customers are consuming their products mean that they have an incentive and a revenue stream to spend on security fixes and general features.  There are other types of fixes and improvements on which they may apply resources, but the relative scarcity of skilled security experts means that the principle of comparative advantage suggests that they should be in the best position to apply them for the benefit of the wider community[1].

What if a vendor you use to provide a productised version of an open source project goes bust, or decides to drop support for that product?  Well, this is a problem in the world of proprietary software as well, of course.  But in the case of proprietary software, there are three likely outcomes:

  • you now have no access to the software source, and therefore no way to make improvements;
  • you are provided access to the software source, but it is not available to the wider world, and therefore you are on your own;
  • everyone is provided with the software source, but no existing community exists to improve it, and it either dies or takes significant time for a community to build around it.

In the case of open source, however, if the vendor you have chosen goes out of business, there is always the option to use another vendor, encourage a new vendor to take it on, productise it yourself (and supply it to other organisations) or, if the worst comes to the worst, take the internal productisation route while you search for a scalable long-term solution.

In the modern open source world, we (the community) have got quite good at managing these options, as the growth of open source consortia[2] shows.  In a consortium, groups of organisations and individuals cluster around a software project or set of related projects to encourage community growth, alignment around feature and functionality additions, general security work and productisation for use cases which may as yet be ill-defined, all the while trying to exploit the economies of scale and scope outlined above.  An example of this would be the Linux Foundation’s Confidential Computing Consortium, to which the Enarx project aims to be contributed.

Choosing to consume open source software as a product instead of as a project involves some trade-offs, but from a security point of view at least, the economics for organisations are fairly clear: unless you are in position to employ ample security experts yourself, products are most likely to suit your needs.


1 – note: I’m not an economist, but I believe that this holds in this case.  Happy to have comments explaining why I’m wrong (if I am…).

2 – “consortiums” if you really must.

Humans and (being bad at) trust

Why “signing parties” were never a good idea.

I went to a party recently, and it reminded of quite how bad humans are at trust. It was a work “mixer”, and an attempt to get people who didn’t know each other well to chat and exchange some information. We were each given two cards to hang around our necks: one on which to write our own name, and the other on which we were supposed to collect the initials of those to whom we spoke (in their own hand). At the end of the event, the plan was to hand out rewards whose value was related to the number of initials collected. Pens/markers were provided.

I gamed the system by standing by the entrance, giving out the cards, controlling the markers and ensuring that everybody signed card, hence ending up with easily the largest number of initials of anyone at the party. But that’s not the point. Somebody – a number of people, in fact – pointed out the similarities between this and “key signing parties”, and that got me thinking. For those of you not old enough – or not security-geeky enough – to have come across these, they were events which were popular in the late nineties and early parts of the first decade of the twenty-first century[1] where people would get together, typically at a tech show, and sign each other’s PGP keys. PGP keys are an interesting idea whereby you maintain a public-private key pair which you use to sign emails, assert your identity, etc., in the online world. In order for this to work, however, you need to establish that you are who you say you are, and in order for this to work, you need to convince someone of this fact.

There are two easy ways to do this:

  1. meet someone IRL[2], get them to validate your public key, and sign it with theirs;
  2. have someone who knows the person you met in step 1 agree that they can probably trust you, as the person in step 1 did, and they trust them.

This is a form of trust based on reputation, and it turns out that it is a terrible model for trust. Let’s talk about some of the reasons for it not working. There are four main ones:

  • context
  • decay
  • transitive trust
  • peer pressure.

Let’s evaluate these briefly.

Context

I can’t emphasise this enough: trust is always, always contextual (see “What is trust?” for a quick primer). When people signed other people’s key-pairs, all they should really have been saying was “I believe that the identity of this person is as stated”, but signatures and encryption based on these keys was (and is) frequently misused to make statements about, or claim access to, capabilities that were not necessarily related to identity.

I lay some of the fault of this at the US alcohol consumption policy. Many (US) Americans use their driving licence/license as a form of authorisation: I am over this age, and am therefore entitled to purchase alcohol. It was designed to prove that their were authorised to drive, and nothing more than that, but you can now get a US driving licence to prove your age even if you can’t drive, and it can be used, for instance, as security identification for getting on aircraft at airportsThis is crazy, but partly explains why there is such a confusion between identification, authentication and authorisation.

Decay

Trust, as I’ve noted before in many articles, decays. Just because I trust you now (within a particular context) doesn’t mean that I should trust you in the future (in that or any other context). Mechanisms exist within the PGP framework to expire keys, but it was (I believe) typical for someone to resign a new set of keys just because they’d signed the previous set. If they were only being used for identity, then that’s probably OK – most people rarely change their identity, after all – but, as explained above, these key pairs were often used more widely.

Transitive trust

This is the whole “trusting someone because I trust you” problem. Again, if this were only about identity, then I’d be less worried, but given people’s lack of ability to specify context, and their equal inability to communicate that to others, the “fuzziness” of the trust relationships being expressed was only going to increase with the level of transitiveness, reducing the efficacy of the system as a whole.

Peer pressure

Honestly, this occurred to me due to my gaming of the system, as described in the second paragraph at the top of this article. I remember meeting people at events and agreeing to endorse their key-pairs basically because everybody else was doing it. I didn’t really know them, though (I hope) I had at least heard of them (“oh, you’re Denny’s friend, I think he mentioned you”), and I certainly shouldn’t have been signing their key-pairs. I am certain that I was not the only person to fall into this trap, and it’s a trap because humans are generally social animals[3], and they like to please others. There was ample opportunity for people to game the system much more cynically than I did at the party, and I’d be surprised if this didn’t happen from time to time.

Stepping back a bit

To be fair, it is possible to run a model like this properly. It’s possible to avoid all of these by insisting on proper contextual trust (with multiple keys for different contexts), by re-evaluating trust relationships on a regular basis, by being very careful about trusting people just due to their trusting someone else (or refusing to do so at all), and by refusing just to agree to trust someone because you’ve met them and they “seem nice”. But I’m not aware of anyone – anyone – who kept to these rules, and it’s why I gave up on this trust model over a decade ago. I suspect that I’m going to get some angry comments from people who assert that they used (and use) the system properly, and I’m sure that there are people out there who did and do: but as a widespread system, it was only going to work if the large majority of all users treated it correctly, and given human nature and failings, that never really happened.

I’m also not suggesting that we have many better models – but we really, really need to start looking for some, as this is important, and difficult stuff.


1 – I refuse to refer to these years the “aughts”.

2 – In Real Life – this used to be an actual distinction to online.

3 – even a large enough percentage of IT folks to make this a problem.

Breaking the security chain(s)

Your environment is n-dimensional – your trust must be, too.

One of the security principles by which we[1] live[2] is that security is only as strong as the weakest link in a chain.  That link is variously identified as:

  • your employees
  • external threat actors
  • all humans
  • lack of training
  • cryptography
  • logging
  • anti-virus
  • auditing capabilities
  • the development lifecycle
  • waterfall methodology
  • passwords
  • any other authentication mechanisms
  • electrical wiring
  • hurricanes
  • earthquakes
  • and pestilence.

Actually, I don’t think I’ve ever seen the last one mentioned, but it’s only a matter of time.  However, very rarely does anybody bother to identify exactly what the chain is that it is being broken by the weakest link splintering into a thousand pieces.

There are a number of candidates that spring to mind:

  1. your application flow.  This is rather an old-fashioned way of thinking of applications: that a program is started, goes through a set of actions, and then terminates, but to think more broadly about it, any action which causes an application to behave in unexpected or unintended ways is a possible security flow, whether that is a monolithic application, a set of microservices or an app on  mobile device.
  2. your software stack.  Depending on how you think about your stack, there are likely to be at least 5, maybe a dozen or maybe even scores of layers in your software stack (for an example with just a few simple layers, see Announcing Enarx).  However you think about it, you need to trust all of those layers to do what who expect them to do.  If one of them is compromised, malicious, or just poorly implemented or maintained, then you have a security issue.
  3. your hardware stack.  There was a time, barely five years ago, when most people (excepting us[1], of course), assumed that hardware did what we thought it was supposed to do, all of the time.  In fact, we should all have known better, given the Clipper Chip and the Pentium bug (to name just to famous examples), but with Spectre, Meltdown and a growing realisation that hardware isn’t as trustworthy as was previously thought, everybody needs to decide exactly what security they can trust in which components.
  4. your operational processes.  You can have the best software and hardware in the world, but if you don’t maintain it and operate it properly, it’s going to be full of holes.  Failing to invest in operations, monitoring, logging, auditing and the rest leaves you wide open.
  5. your supply chain. There’s a growing understanding in the industry that our software and hardware supply chains are possible points of failure[3].  Whether your vendor is entirely proprietary (in which case their security is largely opaque) or open source (in which case you’ve got a chance to be able to see what’s going on), errors or maliciousness in the supply chain can scupper any hopes you had of security for your deployment.
  6. your software and hardware lifecycle.  Developing software?  Patching it?  Upgrading hardware (or software)?  Unit testing?  Unit testingg?  We all know that a failure to manage the lifecycle of our environment can lead to security problems.

The point I’m trying to make above is that there’s no single chain.  Your environment is n-dimensional – your trust must be, too.  If you don’t think about all of these contexts – and there will be more beyond the half-dozen that I’ve just noted – then you can’t have a good chance of managing security in your environment.  I honestly don’t think that there’s any single weakest link in the chain, because there are always already multiple chains in play: our job is to think about as many of them as possible, and then manage an mitigate the risks associated with each.


1 – the mythical “IT security community”.

2 – you’re right: “which we live by” would sound much more natural.

3 – and a growing industry to try to provide fixes.

“Unlawful, void, and of no effect”

The news from the UK is amazing today: the Supreme Court has ruled that the Prime Minister has failed to “prorogue” Parliament – the in other words, that the members of the House of Commons and the House of Lords are still in session. The words in the title come from the judgment that they have just handed down.

I’m travelling this week, and wasn’t expecting to write a post today, but this triggered a thought in me: what provisions are in place in your organisation to cope with abuses of power and possible illegal actions by managers and executives?

Having a whistle-blowing policy and an independent appeals process is vital. This is true for all employees, but having specific rules in place for employees who are involved in such areas as compliance and implementations involving regulatory requirements is vital. Robust procedures protect not only an employee who finds themself in a difficult position, but, in the long view, the organisation itself. They a can also act as a deterrent to managers and executives considering actions which might, in the absence of such procedures, likely go unreported.

Such procedures are not enough on their own – they fall into the category of “necessary, but not sufficient” – and a culture of ethical probity also needs to be encouraged. But without such a set of procedures, your organisation is at real risk.

What is DoH, and why should I care?

Firefox is beginning to roll out DoH

DoH is DNS-over-HTTPS.  Let’s break that down.

DNS is Domain Network System, and it’s what allows you to type in the server name (e.g. aliceevebob.com or http://www.redhat.com), which typically makes up the key part of a URL, and then get back the set of numbers which your computer needs actually to contact the machine you want it to talk to.  This is because computers don’t actually use the names, they use the numbers, and the mapping between the two can change, for all sorts of reasons (a server might move to another machine, it might be behind a firewall, it might be behind a load-balancer – those sorts of reasons).   These numbers are called “IP addresses”, and are typically[1] what are called “dotted quads”.  An example would be 127.0.0.1 – in fact, this is a special example, because it maps back to your own machine, so if you ask for “localhost”, then the answer that DNS gives you is “127.0.0.1”.  All IP[1] addresses must be in of the type a.b.c.d, where the a, b, c and d are numbers between 0 and 254 (there are some special rules beyond that, but we won’t go into them here).

Now, your computer doesn’t maintain a list of the millions upon millions of server names and their mappings to specific IP address – that would take too much memory, and ages to download.  Instead, if it needs to find a server (to get email, talk to Facebook, download a webpage, etc.), it will go to a “DNS server”.  Most Internet providers will provide their own DNS servers, and there are a number of special DNS servers to which all others connect from time to time to update their records.  It’s a well-established and generally well-run system across the entire Internet.  Your computer will keep a cache of some of the most recently used mappings, but it’s never going to know all of them across the Internet.

What worries some people about the DNS look-up process, however, is that when you do this look-up, anyone who has access to your network traffic can see where you want to go.  “But isn’t secure browsing supposed to stop that?” you might think.  Well, yes and no.  What secure browsing (websites that start “https://”) means is that nobody with access to your network traffic can see what you download from and transmit to the website itself.  But the initial DNS look-up to find out what server your browser should contact is not encrypted. This might generally  be fine if you’re just checking the BBC news website from the UK, but there are certainly occasions when you don’t want this to be the case.  It turns out although DoH doesn’t completely fix the problem of being able to see where you’re visiting, many organisations (think companies, ISPs, those under the control of countries…) try to block where you can even get to by messing with the responses you get to look-ups.  If your computer can’t even work out where the BBC news server is, then how can it visit it?

DoH – DNS-over-HTTPS – aims to fix this problem.  Rather than your browser asking your computer to do a DNS look-up and give it back the IP address, DoH has the browser itself do the look-up, and do it over a secure connection.  That’s what the HTTPS stands for – “HyperText Transfer Protocol Secure” – it’s what your browser does for all of that other secure traffic (look for the green padlock”).  All someone monitoring your network traffic would see is a connection to a DNS server, but not what you’re asking the DNS server itself.  This is a nice fix, and the system (DoH) is already implemented by the well-known Tor browser.

The reason that I’m writing about it now is that Firefox – a very popular open source browser, used by millions of people across the world – is beginning to roll out DoH by default in a trial of a small percentage of users.  If the trial goes well, it will be available to people worldwide.  This is likely to cause problems in some oppressive regimes, where using this functionality will probably be considered grounds for suspicion on its own, but I generally welcome any move which improves the security of everyday users, and this is definitely an example of one of those.


1 – for IPv4.  I’m not going to start on IPv6: maybe another time.