Listen to Episode Here
Show Notes
In his Research Agenda v0.9: Synthesizing a human's preferences into a utility function, Stuart Armstrong develops an approach for generating friendly artificial intelligence. His alignment proposal can broadly be understood as a kind of inverse reinforcement learning where most of the task of inferring human preferences is left to the AI itself. It's up to us to build the correct assumptions, definitions, preference learning methodology, and synthesis process into the AI system such that it will be able to meaningfully learn human preferences and synthesize them into an adequate utility function. In order to get this all right, his agenda looks at how to understand and identify human partial preferences, how to ultimately synthesize these learned preferences into an "adequate" utility function, the practicalities of developing and estimating the human utility function, and how this agenda can assist in other methods of AI alignment.
Topics discussed in this episode include:
- The core aspects and ideas of Stuart's research agenda
- Human values being changeable, manipulable, contradictory, and underdefined
- This research agenda in the context of the broader AI alignment landscape
- What the proposed synthesis process looks like
- How to identify human partial preferences
- Why a utility function anyway?
- Idealization and reflective equilibrium
- Open questions and potential problem areas
Last chance to take a short (4 minute) survey to share your feedback about the podcast.
Key points from Stuart:
- "There are two core parts to this research project essentially. The first part is to identify the humans' internal models, figure out what they are, how we use them and how we can get an AI to realize what's going on. So those give us the sort of partial preferences, the pieces from which we build our general preferences. The second part is to then knit all these pieces together into an overall preference for any given individual in a way that works reasonably well and respects as much as possible the person's different preferences, meta-preferences and so on. The second part of the project is the one that people tend to have strong opinions about because they can see how it works and how the building blocks might fit together and how they'd prefer that it would be fit together in different ways and so on but in essence, the first part is the most important because that fundamentally defines the pieces of what human preferences are."
- "So, when I said that human values are contradictory, changeable, manipulable and underdefined, I was saying that the first three are relatively easy to deal with but that the last one is not. Most of the time, people have not considered the whole of the situation that they or the world or whatever is confronted with. No situation is exactly analogous to another, so you have to try and fit it in to different categories. So if someone dubious gets elected in a country and starts doing very authoritarian things, does this fit in the tyranny box which should be resisted or does this fit in the normal process of democracy box in which case it should be endured and dealt with through democratic means. What'll happen is generally that it'll have features of both, so it might not fit comfortably in either box and then there's a wide variety for someone to be hypocritical or to choose one side or the other but the reason that there's such a wide variety of possibilities is because this is a situation that has not been exactly confronted before so people don't actually have preferences here. They don't have a partial preference over this situation because it's not one that they've ever considered... I've actually argued at some point in the research agenda that this is an argument for insuring that we don't go too far from the human baseline normal into exotic things where our preferences are not well-defined because in these areas, the chance that there is a large negative seems higher than the chance that there's a large positive... So, when I say not go too far, I don't mean not embrace a hugely transformative future. I'm saying not embrace a hugely transformative future where our moral categories start breaking down."
- "One of the reasons to look for a utility function is to look for something stable that doesn't change over time and there is evidence that consistency requirements will push any form of preference function towards a utility function and that if you don't have a utility function, you just lose value. So, the desire to put this into a utility function is not out of an admiration for utility functions per se but our desire to get something that won't further change or won't further drift in a direction that we can't control and have no idea about. The other reason is that as we start to control our own preferences better and have a better ability to manipulate our own minds, we are going to be pushing ourselves towards utility functions because of the same pressures of basically not losing value pointlessly."
- "Reflective equilibrium is basically you refine your own preferences, make them more consistent, apply them to yourself until you've reached a moment where your meta-preferences and your preferences are all smoothly aligned with each other. What I'm doing is a much more messy synthesis process and I'm doing it in order to preserve as much as possible of the actual human preferences. It is very easy to reach reflective equilibrium by just, for instance, having completely flat preferences or very simple preferences, these tend to be very reflectively in equilibrium with itself and pushing towards this thing is a push towards, in my view, excessive simplicity and the great risk of losing valuable preferences. The risk of losing valuable preferences seems to me a much higher risk than the gain in terms of simplicity or elegance that you might get. There is no reason that the kludgey human brain and it's mess of preferences should lead to some simple reflective equilibrium. In fact, you could say that this is an argument against reflexive equilibrium because it means that many different starting points, many different minds with very different preferences will lead to similar outcomes which basically means that you're throwing away a lot of the details of your input data."
- "Imagine that we have reached some positive outcome, we have got alignment and we haven't reached it through a single trick and we haven't reached it through the sort of tool AIs or software as a service or those kinds of approaches, we have reached an actual alignment. It, therefore, seems to me all the problems that I've listed or almost all of them will have had to have been solved, therefore, in a sense, much of this research agenda needs to be done directly or indirectly in order to achieve any form of sensible alignment. Now, the term directly or indirectly is doing a lot of the work here but I feel that quite a bit of this will have to be done directly."
Important timestamps:
0:00 Introductions
3:24 A story of evolution (inspiring just-so story)
6:30 How does your “inspiring just-so story” help to inform this research agenda?
8:53 The two core parts to the research agenda
10:00 How this research agenda is contextualized in the AI alignment landscape
12:45 The fundamental ideas behind the research project
15:10 What are partial preferences?
17:50 Why reflexive self-consistency isn’t enough
20:05 How are humans contradictory and how does this affect the difficulty of the agenda?
25:30 Why human values being underdefined presents the greatest challenge
33:55 Expanding on the synthesis process
35:20 How to extract the partial preferences of the person
36:50 Why a utility function?
41:45 Are there alternative goal ordering or action producing methods for agents other than utility functions?
44:40 Extending and normalizing partial preferences and covering the rest of section 2
50:00 Moving into section 3, synthesizing the utility function in practice
52:00 Why this research agenda is helpful for other alignment methodologies
55:50 Limits of the agenda and other problems
58:40 Synthesizing a species wide utility function
1:01:20 Concerns over the alignment methodology containing leaky abstractions
1:06:10 Reflective equilibrium and the agenda not being a philosophical ideal
1:08:10 Can we check the result of the synthesis process?
01:09:55 How did the Mahatma Armstrong idealization process fail?
01:14:40 Any clarifications for the AI alignment community?
Works referenced:
Research Agenda v0.9: Synthesising a human's preferences into a utility function
Some Comments on Stuart Armstrong's "Research Agenda v0.9"
Mahatma Armstrong: CEVed to death
We hope that you will continue to join in the conversations by following us or subscribing to our podcasts on Youtube, Spotify, SoundCloud, iTunes, Google Play, Stitcher, iHeartRadio, or your preferred podcast site/application. You can find all the AI Alignment Podcasts here.
