Reverse game theory could help control AI agents

AI safety researchers are studying reverse game theory, also known as mechanism design, as an alternative to investigating models’ internal workings or directly training them to follow human values. The approach seeks to create rules and incentives that lead agents to behave as desired, even when their intentions are not transparent.

According to MIT Technology Review Brasil, a preprint by MIT economist Stephen Morris, with researchers from Yale and Columbia, proposes rewarding models for their ability to predict another model’s responses rather than evaluating performance alone. This could make it harder for an agent to pretend to be less capable in order to avoid safety mechanisms. The paper also suggests competitions between agents and pairs with opposing biases, using agreement between their answers as an incentive.