What AI Safety Can Learn From Totalitarian States

We are taking continuous steps towards ever more intelligent and powerful AI systems, and it is clear that even narrow AI systems have increasing influence over our lives. Understanding them and maintaining control over them might be one of the most existentially relevant tasks of the century.

Totalitarian states, as envisioned by the OpenAI’s DALL-E.

This task is made more daunting because, in many ways, we think that there is no precedent for the large-scale superhumanly powerful algorithms we are building.

But I think this is only partially true: humans, consciously or unconsciously, have been constructing superhumanly intelligent cultural structures for millenia. As Joshua Bengio puts it, culture represents a significant, stable part of our world models that we have developed over several lifespans between multiple cooperating cultural agents. Culture gives us an understanding of reality and ways to act within it in ways that would go beyond the mental or observational capacities of every single individual (reflexes that we share with our animal ancestors, like the fight or flight response, go back even further, and are conserved for even longer time scales and across species).

According to Joe Henrich, the secret to our success is rooted in the ability of our cultural memes to undergo their own evolutionary process, with successful ideas emerging on top and, in…