Advobots and AI Safety

In a 2014 post I argued for the “servant mission” as a solution to AI safety. The basic idea is that there are many AI agents each one of which has the mission of serving a particular person. I gave a concise statement of the mission of my servant as:

Mission: Within the law, fulfill the requests of David McAllester

These days I prefer the term “advobot” — A bot whose mission is to advocate for a particular person. A society in which AGI is restricted to personal advobots has fundamental safety advantages discussed in the 2014 post. However, while the concept of a personal advobot seemed well formed in 2014, today’s large language models (LLMs) raise serious issues that were not apparent at that time. The very concept of a “mission statement” now seems problematic. Much of the discussion of AI safety (or lack thereof) has implicitly assumed that an AI agent can be given a fundamental goal expressed in natural language. For example the servant mission above, or maybe “make as many paper clips as possible”, or “get me to the airport as quickly as possible”. In this post I want to consider the relationship between the details of language model architecture, mission statements, and AI safety.

I will start by speculating that LLM architectures will converge on what I will call belief-memory architectures. Belief-memory architectures will be descendants of current retrieval architectures such as RETRO where text generation and next word prediction take retrieved information into account. I am using RETRO as only one example, there is now a large literature on attempts to construct retrieval architectures. In a belief-memory architecture the retrieved information will be interpretable as beliefs where these beliefs are either explicit text or vector representations that can be rendered as text and where the model responds to retrieved beliefs in a manner roughly corresponding to the effect of beliefs on humans. While there is a significant literature on retrieval models, none of the current models retrieve beliefs in this sense.

Belief-memory models, if they can be constructed, have serious advantages. Most obviously, one can inspect what the model believes. One can also inspect what beliefs were used in generating responses or actions. Furthermore, an explicit belief memory can be manually edited — wrong or harmful beliefs can be removed and the events of the day can be added. Note that a belief might simply be that someone claimed something where claims are not necessarily believed.

I will also assume that a belief-retrieval models will be “belief-generating”. Belief generation is similar to chain-of-thought in that the model generates thoughts before generating an answer. In belief generation the thoughts are beliefs that are added to memory. The LLM will remember “I thought x”. There is an analogy here with a Von Neumann architecture where the CPU is replaced with a transformer and the registers are replaced with the transformer context. The context can be rewritten or extended as in current chain-of-thought processing. But the generated thoughts can be stored in the belief memory. This models human memory of thoughts. Subjectively we are not aware of how our thoughts are generated — they just appear. But we do remember them. By definition, computation which is not remembered is not accessible later.

It is hard to image an AGI that is plotting to turn the world into paperclips that does not form and remember thoughts. A belief-generating architecture has the additional advantage that one can inspect (or search through) the thoughts of the model. An advobot for person x should accept the situation where x can watch its thoughts. After all, the advobot for x has the mission of fulfilling the requests of x.

We must also face the problem of how to give an AI agent a mission stated in text. A good first attempt is the recently introduced notion of constitutional AI. I believe that, as language models get better at understanding language, the constitutional AI approach will become more effective and eventually outperform reinforcement learning with human feedback (RLHF). Constitutional AI has the obvious advantage of being able to explicitly specify the mission.

As of now I am not too concerned about the existential risk (X-risk) from AGI. It seems to me that we will have the common sense to construct a society of personal advobots with servant missions and where the beliefs and thoughts of advobots are inspectable, searchable, and editable. I am very much looking forward to an advobot era which I expect to arrive soon.