A friend, who works for a large AI research company, posed a question to me the other day. “What is the least capable, that is, least advanced, AI system product demo that you could see where you would be worried that the system you were seeing had the potential to destroy the world?” In the days since, with the FLI call for a six-month pause on AI research, followed by Eliezer Yudkowsky’s essay in Time, this question, and my answer to it have been on my mind a lot. I’ve come to think of this as “The Demo.”
I will worry that we have created a technology capable of destroying the world when we create an AI system that is indistinguishable from a human, except for the fact that it is confined to a computer. That is, as I work with, talk with, and interact with this model, I cannot distinguish it from a human trapped on the other side of a Zoom interface. This is, basically, exactly the Turing Test, except that I emphatically would need to keep feeling like the system was indistinguishable from a human on tasks that didn’t involve verbal communication. The system I’m imagining brings to mind Holden Karnofsky’s Digital People, or Samantha from Her.
Upon receiving a tech demo of such a system, I think I will (and would is the better word choice here, because I think we’re on track to this kind of system) have two reactions. First, a strong sense that I am basically obsolete for the purpose of civilizational technological advance from that point forward. A second, subtler shift is that I think from that point forward I might struggle to be certain that I’m in “base reality”.
The simple reason that I would be struck with a strong feeling that all technological progress from that point forward would be better done by the system itself than by me is because such a system could effortlessly have many copies of itself spun up, to coordinate and research different topics, in a way that I cannot be scaled.
Imagine, in a world with such a system, interacting with a child who dreams of one day being a biologist. Decades will pass before the child is old enough to start studying the basics of biology, and another decade after that may pass before the child masters enough of the field to begin contributing to a specialized discipline, really standing on the shoulders of giants and adding new insights at the forefront of human knowledge. Twenty years hence this child, if they stuck to this goal, will have studied and understood deeply a small sliver of the field of biology, to begin another few decades of beginning to contribute.
A “digital person” would know all of this content instantly. It would sit across from me, right this moment, knowing all of this content, and infinitely more. If it found a research program that looked interesting, a copy of it could be spun up, immediately duplicating the number of capable researchers working on the research. Where the child takes years to develop into a contributing member of society, an instance of such a human-like AI system could be spun up in moments.
Fundamentally, I think this quality of such “digital people” means that it would be unclear why physical flesh-and-bone humans were needed for continued technological advance after such a system is deployed.
The second impact is that it might make me a bit more suspicious that I wasn’t myself a “digital person”. It’s a little trippy to think about, but once I’m actually met face to face with systems that are indistinguishable from humans, I think I will have likely been showed an existence proof that systems like myself can be fully simulated. Knowing this, the odds that I’m in fact a digital replica of myself seem to go up quite a lot.
As another friend pointed out, in a world where there are agent-like systems going around and working alongside flesh-and-bone humans, I expect that religious perspectives on metaphysics will have to adapt. It will become painfully obvious, to everyone, that systems that are in fact more knowledgable and capable than any individual human have become present in every day life, living in virtual realities that we design or that they design for themselves. This will shake many peoples’ sense that their consciousness is special, and that they are embedded in a special layer of reality. This will, potentially, lead many to total solipsism, and many more to new ideologies surrounding metaphysics that are hard for us to anticipate now.
"The second impact is that it might make me a bit more suspicious that I wasn’t myself a “digital person”."
wow imagine an AI propaganda campaign convincing humans that we are all simulations and can "wake up" by killing ourselves