Engineering Desire: The Problem of What to Want

Engineering Desire: The Problem of What to Want

One of my favourite stories from antiquity is that of King Midas, who famously was granted the wish of turning everything he touched into gold, only to become dissatisfied with life after receiving this wish. The force of desire seems to place human beings in a stunning array of challenging situations. The two that I’ll focus on in this post appear very similar if you look only at the language of their statements, but they’re profoundly different if you consider what it takes to solve each of them. The first problem is knowing that you don’t know what you want. The second is believing you know what you want when you actually don’t know (where ‘believing’ here means being convinced that you know what you want enough to take action toward acquiring/achieving it). In the former condition, you’re in a halted state where you can’t make a decision. In the latter state, you’re about to dive into a decision unstrategically without being aware of the lack of proper strategy.

These problems have reappeared to me in many different guises over the past decade. I’ve also started to hear technical academics start to discuss them. When I see a subject taken up across a range of disciplines, I begin to think that I’ve found a gem of inquiry, and possibly one that’s so ubiquitous as to have hidden from humanity in plain sight. I once told one of my former professors, Dr. Michael Frank (research page at http://ski.cog.brown.edu/), that I was deeply interested in conducting research into goal formation in sentient agents. I recall him nodding slowly and absorbing that, as though it was a reasonable topic but one that he hadn’t thoroughly considered. Dr. Frank studies neural mechanisms of decision making, and yet it didn’t seem to be at the forefront of his mind how agents “choose” what it is they intend to do or whether this is even a well-posed philosophical topic! I certainly don’t fault him if my impression was correct; as an academic, restricting scope is an essential part of research strategy. What I find even more astounding, however, is that it’s only fairly recently that artificial intelligence (AI) experts have come round to thinking about these matters. More on that soon.

I’ll try to summarize other fields and areas of life where I’ve seen these problems appear again and again: engineering, Buddhism, family dynamics, politics, home buying, literature, job searching, cognitive science, et al. When a decision is made by an individual or a group, the reason for the decision at the level of human consciousness (as opposed to the reason(s) at the levels of human biophysics, particle physics, etc.) is to bring about an outcome desired by some subset of the set of agents involved in making the decision. But again and again, we encounter the problems of consciously not knowing which outcome we want to select or of believing that we know which outcome we want to select without adequate information.

I’ll next try to ground this problem in a small sample of what I hope will be experiences shared by many readers. Anyone who has either been a boss or had a boss is likely familiar with the following general scenario: Boss asks boss-ee to do a thing. Boss-ee does exactly what boss asked. Yet boss is not happy! This could repeat a maddening number of times until boss perceives internal satisfaction or gives up and, either way, stops pestering boss-ee. We’ll see this scenario again very soon when I start discussing AI.

Also consider the decision to buy a home. Having just navigated the stressful waters of negotiating a home purchase this past week (felt more like two weeks!), I’m eager to reflect on this experience. I would bet that a lot of people choose a home based on certain desired property attributes only to discover enough pitfalls once they move in that they regret the decision. I might find this to be true for myself in a couple of months! But I tried to be as strategically skeptical and practical as possible before buying in the hope that my case will turn out to be the other way round.

Now back to engineering and AI, which I’m trying to bring together with this post. Engineering is all about producing things to satisfy human requirements. Implicit in the very definition of the field, then, is the problem of choosing requirements. Let’s also look at how we might define AI as a field: the study of intelligent agents. Ah, well then, we had better try to clarify what we mean by ‘intelligent’! This is quite hard, in itself. One definition that I think is useful is one that I’ve seen taken up in the work of eminent AI researcher Stuart Russell (I highly recommend his book Human Compatible: Artificial Intelligence and the Problem of Control.). ‘Intelligence’ for AI purposes might be thought of more like ‘competence’: the ability of an agent to accomplish its objectives, given what it perceives.

[Rest stop: Let’s step back and note that we’re already starting to see the emergence of some very important conceptual distinctions: ‘desires’ vs. ‘objectives’ vs. ‘requirements’, ‘intelligence’ vs. ‘competence’. This is my personal speculation only, but what I think is going on in the language across all these disciplines is an effort to separate the human feelings (e.g. desire) that accompany the selection and achieving of outcomes from the abstract description of those outcomes (denoted by ‘objectives’ or ‘requirements’) and to separate what aspects of human competence that might be uniquely human (i.e. ‘intelligence’) from those aspects of competence that might be had by any type of agent. I do not think that we should forget about or otherwise abandon uniquely human feelings and abilities. I do think it is useful to be able to make those distinctions.]

I have direct personal experience with the meeting of engineering and AI in my job. Any person in the position of doing my job would often have to decide what the requirements for an object or process are going to be. One could choose at random, but the more competent approach seems to be to choose according to what the requirements should be, where this means what the requirements are that optimize among the set of outcomes resulting from implementing the requirements (Discussing the particulars of this decision-making process probably best belongs to a different post that focuses on details of engineering.). How AI enters into my job, and pretty much every job as far as I can tell, is the occurrence of the boss-bossee scenario I introduced above.

As far as people go, I’m a pretty intelligent agent in the AI sense; I have a high success rate at meeting the objectives I set for myself, given what I perceive (i.e. the information available to my senses that my senses are able to process). At the same time, I have made mistakes that have probably seemed moronic to my former boss (He recently retired.), who had access to more information relevant to what I was trying to do — including more information about what he actually wanted me to do! I’ve also striven vainly (apparent only in retrospect) to beat systems that were overwhelmingly stacked against me, but the full truth of that situation wasn’t knowable to me when I was doing the striving.

So very many human beings have been on either or both side(s) of these situations. This is experience that it would hugely benefit us to keep in mind as we integrate AI products into personal and societal life. For just as we humans can be competent but inadequately informed — especially about what is required of us — so too can AI products do exactly what we tell them, with anything from unexpected to unpleasant to catastrophic results.

If you tell your personal home robotic assistant to make you a peanut butter and jelly sandwich, implied in this statement is that the agent is required to make you a sandwich under any circumstances, because, well, you didn’t say otherwise! But of course, there are probably unimaginably many circumstances in which you don’t want the robot to start or proceed making you a sandwich. If the kitchen ceiling falls in, probably not a great time for a sandwich. If you are out of bread and all stores are closed, maybe you don’t want the robot to decide without your consent that it is going to bake bread at 3am to make you this sandwich. We could keep specifying conditions under which you do not want the robot to make this sandwich until we have been specifying so long that you no longer want the sandwich! Thus, simply having to ask precisely for something can be a circumstance under which we actually don’t want it! Mind-bending.

So we find ourselves in a strange frontier of engineering and AI. We’re no longer just trying to design a competent agent, we’re trying to make it and use it to satisfy human needs. This is an engineering problem that we invite to bear down fully on us by creating amazingly competent agents without preparing for the world in which we will find ourselves when they exist. We need to require the ability to control AI products to protect ourselves from the situations of consciously not knowing what we want or believing without proper evidence that we do know what we want. If we don’t know what we want — consciously or unconsciously — we need to be able to control an AI product in such a way that it helps us figure that out without us having to first suffer unpleasant or catastrophic mistakes. Moreover, if we believe falsely that we know what we want, we need an AI product to help us test this belief without us having to ask them to help with this testing; it is exactly the confidence of believing we know what we want that can lead us to not even think of questioning whether or not we are right. If Midas had had an R2D2 to help him out, R2 should have been preprogrammed to ask Midas something like: “Hey, I know you feel great right now thinking about having free money, but have you considered that you might destroy everything worth having in the process?”