It’s time, readers, that we answer the big question. No, not “but how does the blockchain actually work” — or “what am I actually going to do in the metaverse” — or even “wen Lambo.” I’m talking about the big question: Is artificial intelligence going to kill us all? Okay, that might be a little bit alarmist. But there is, in fact, a dedicated and passionate group of very intelligent people dedicated at the moment to answering this question — determining the risk profile of “artificial general intelligence,” the term for an AI-powered system that matches human cognitive capacity. What happens, these future-minded theorists ask, when it uses that capacity to defy, or thwart, its human creators? This is not exactly one of the big AI policy questions on the agenda right now — when regulators think about AI, they’re mostly thinking about algorithmic fairness, the abuse of personal data, or how the tech might disrupt existing sectors like education or law. But when this does land on the radar, it will be the Big One. “The worry is that by its very nature [a generally intelligent AI]would have a very different set of moral codes (things it should or shouldn’t do) and a vastly increased set of capabilities, which can result in pretty catastrophic outcomes,” wrote the venture capitalist and tech blogger Rohit Krishnan in a blog post last month that explored the idea at length. But there’s one big problem for anyone trying to address this seriously: How likely is it that such a thing is even possible? That question is central when we try to figure out how much to worry about what artificial general intelligence might look like, and how quickly we should act to shape it. (For the record, some very smart people are giving these questions some very serious thought.) Krishnan’s post is getting attention right now because he developed a framework of sorts for answering that question.His formula for predicting existential AI risk, which he calls the “Strange Loop Equation,” is based on the Drake equation, which in the 1960s offered a way to calculate another hard-to-guess number: the number of possible, contactable alien entities in the universe. Krishnan’s version incorporates a number of risk conditions into a prediction of the likelihood that a hostile AI could arise. I spoke with Krishnan about the post today, and he emphasized that he, himself, isn’t freaking out — in fact he’s a skeptic of the idea that runaway AI might be a harbinger of doom. He said that “like most technological progress, we're likely to see incremental progress, and with every step change we have to work a little bit on how we can actually do it smartly and safely.” Based on his own assessments of the likelihood of the various conditions that could lead to a hostile AI — like the speed of its development, or its capability to lie — there is a (drum roll, please) 0.0144 percent chance that power-seeking AI will kill or enslave us all. That makes him much more optimistic than some others who’ve tried their own version of the exercise, like in the Metaculus prediction market (34 percent) or a recent study by the researcher Joseph Carlsmith (roughly five percent), as Krishnan points out. (“Bear in mind chained equations like Drake are great to think through, much less so for precise numbers on probabilities,” he added.) So no sweat, right? Maybe not: despite the serious amount of time and thought put into the problem, Krishnan warns that any current speculation likely bears little resemblance to the form it will ultimately take. “I fundamentally don’t think we can make any credible engineering statements about how to safely align an AI while simultaneously assuming it’s a relatively autonomous, intelligent and capable entity,” he writes in conclusion. “Let's assume that at some point in the future we will be able to create systems that have high levels of agency, and to give them curiosity, the ability to act on the world, and the things that we as autonomous intelligent entities have in this world,” he told me today. “They're not really going to be controllable, because it feels very weird to create something that has the powers of a regular human while at the same time, they can only ever do what you tell them to do. We find it very difficult to do that with anything remotely intelligent in our day-to-day life; as a consequence, the only way out I can see is to try to embed our values in them.” Then how do we determine those values, and what role might non-engineers in government and elsewhere play in preventing an AI apocalypse? Krishnan is equally wary there, saying it’s essentially an engineering problem that will have to be solved iteratively as the problems arise. “I am reasonably skeptical on what governments can actually do here, if only because we're talking about stuff at the very cutting edge of not only technology, but in some ways anthropology — figuring out the life science and behavior of what is effectively a new intelligent entity,” Krishnan said. “I suspect that some things government might do would be to start having treaties with each other similar to how we did with nuclear weapons… [and] ensure that the supply chain remains relatively robust,” the better to keep humanity at the cutting edge of AI development.
| 
