Who’s saying “hello”? – agency, intent and AI

I don’t yet have one of those Google or Amazon talking speaker thingies in my house or office.  A large part of this is that I’m just not happy about the security side: I know that the respective companies swear that they’re only “listening” when you say the device’s trigger word, but even if that’s the case, I like to pretend[1] that I have at least some semblance of privacy in my life.  Another reason, however, is that I’m not sure that I like what happens to people when they pretend that there’s a person listening to them, but it’s really just a machine.

It’s not just Alexa and the OK, Google persona, however.  When I connect to an automated phone-answering service, I worry when I hear “I’ll direct your call” from a non-human.  Who is “I”?  “We’ll direct your call” is better – “we” could be the organisation with whom I’m interacting.  But “I”?  “I” is the pronoun that people use.  When I hear “I”, I’m expecting sentience: if it’s a machine I’m expecting AI – preferably fully Turing-compliant.

There’s a more important point here, though.  I’m entirely aware that there’s no sentience behind that “I”[2], but there’s an important issue about agency that we should unpack.

What, then, is “agency”?  I’m talking about the ability of an entity to act on its or another’s behalf, and I touched on this this in a previous post, “Wow: autonomous agents!“.  When somebody writes some code, what they’re doing is giving ability to the system that will run that code to do something – that’s the first part.  But the agency doesn’t really occur, I’d say, until that code is run/instantiated/executed.  At this point, I would argue, the software instance has agency.

But whose agency, exactly?  For whom is this software acting?

Here are some answers.  I honestly don’t think that any of them is right.

1. the person who owns the hardware (you own the Alexa hardware, right?  You paid Amazon for it…  Or what about running applications on the cloud?).
2. the person who started the software (you turned on the Alexa hardware, which started the software…  And don’t forget software which is automatically executed in response to triggers or on a time schedule.)
3. the person who gave the software the instructions (what do you mean, “gave it the instructions”?  Wrote its config file?  Spoke to it?  Set up initial settings?  Typed in commands?  And even if you gave it instructions, do you think that your OK Google hardware is implementing your wishes, or Google’s?  For whom is it actually acting?  And what side effects (like recording your search history and deciding what to suggest in your feed) are you happy to believe are “yours”?)
4. the person who installed the software (your phone comes with all sorts of software installed, but surely you are the one who imbues it with agency?  If not, whom are you blaming: Google (for the Android apps) or Samsung (which actually put them on the phone)?)
5. the person who wrote the software (I think we’ve already dealt with this, but even then, is it a single person, or an organisation?  What about open source software, which is typically written, compiled and documented by many different people?  Ascribing “ownership” or “authorship” is a distinctly tricky (and intentionally tricky) issue when you discuss open source)

Another way to think of this problem is to ask: when you write and execute a program, who is saying “hello world?”: you, or the computer?

There are some really interesting questions that come out of this.  Here are a couple that come to mind, which seem to me to be closely connected.

• In the film Wargames[3], is the automatic dialling that Matthew Broderick’s character’s computer carries out an act with agency?  Or is it when it connects to another machine?  Or when it records the details of that machine?  I don’t think anyone would argue that the computer is acting with agency once David Lightman actually gets it to complete a connection and interact with it, but what about before?
• Google used to run automated programs against messages received as part of the Gmail service looking for information and phrases which it could use to serve ads.  They were absolutely adamant that they, Google, weren’t doing the reading: it was just a computer program.  I’m not sure how clear or safe a distinction that is.

Why does this all matter?  Well, one of the more pressing reasons is because of self-driving cars.  Whose fault is it when one goes wrong and injures or kills someone?  What about autonomous defence systems?

And here’s the question that really interests – and vexes – me: is this different when the program which is executing can learn.  I don’t even mean strong AI: just that it can change what it does based on the behaviour it “sees”, “hears” or otherwise senses.  It feels to me that there’s a substantive difference between:

a) actions carried out at the explicit (asynchronous) request of a human operator, or according to sets of rules coded into a program

AND

b) actions carried out in response to rules that have been formed by the operation of the program itself.  There is what I’d called synchronous intent within the program.

You can argue that b) has pretty much always been around, in basic forms, but it seems to me to be different when programs are being created with the expectation that humans will not necessarily be able to decode the rules, and where the intent of the human designers is to allow rulesets to be created in this way.

There is some discussion about at the moment as to how and/or whether rulesets generated by open source projects should be shared.  I think the general feeling is that there’s no requirement for them to be – in the same way that material I write using an open source text editor shouldn’t automatically be considered open source – but open data is valuable, and finding ways to share it is a good idea, IMHO.

In Wargames, that is the key difference between the system as originally planned, and what it ends up doing: Joshua has synchronous intent.

I really don’t think this is all bad: we need these systems, and they’re going to improve our lives significantly.  But I do feel that it’s important that you and I start thinking hard about what is acting for whom, and how.

Now, if you wouldn’t mind opening the Pod bay doors, HAL…[5]

---

1. and yes, I know it’s a pretense.

2. yet…

3. go on – re-watch it: you know you want to[4].

4. and if you’ve never watched it, then stop reading this article and go and watch it NOW.

5. I think you know the problem just as well as I do, Dave.