Rethinking AI

some updates

My post “Why I Am Not An AI Doomer”, written in April 2023, was an attempt to lay out why I’m not worried about advanced artificial intelligence killing us all.

On an emotional level, I’m still not worried, and my sympathies are (mostly) more with AI developers than with activists calling for regulation.

But I have to admit that I no longer stand by that post as it’s written.

Mostly my argument rested on basic skepticism that modern machine-learning models are on track to acquire the “special sauce” that makes human intelligence unique (or, even, what makes animal intelligence functional for an animal’s survival.)

I no longer believe that.

I think the way to bet is that the AI/tech industry will, sooner-or-later, create bona fide “minds”, fully intelligent in every sense that we are.

What I’m Still Skeptical About

I still basically think that mastery of the physical world must flow through interaction with the physical world.

No, an LLM trained exclusively on publicly available or purchased digital data is not going to invent molecular nanotechnology.

I am willing to bet that no iteration of AlphaFold is going to replace in vitro experiments in biotech. Specifically, I don’t believe any drug will enter clinical trials that was not tested in some kind of in vitro experiment prior to animal testing.

Experimental science and hardware engineering require, y’know, experiment. I don’t believe enough of the relevant information is captured in papers, manuals, diagrams, etc to allow an AI to invent or discover stuff that works on the first try.

I also still think that there’s something special about “agency” or “goal-seeking”, which I don’t totally understand, but which humans and mammals clearly have, and which may include awareness of one’s own existence, which would be necessary before an AI was able to robustly pursue things like “self-protection” and “accruing resources”. And I still don’t think any current-gen AI models have that.

What I’ve Changed My Mind About

I seriously underrated the AI industry.

I thought, for instance, that AI-trained industrial robots weren’t going to happen — simulation training has a track record of not being very effective, real-world training is expensive, and an AI-trained robot (especially one that contains GPUs for inference) is just not price-competitive with a traditional hard-coded industrial robot.

And even traditional industrial robots are rare — basically the only manufacturing fields that use significant numbers of robots at all are are the electronics and automotive industries. Despite more than a decade of breathless media panics about “automation” taking manufacturing jobs, in the late 20th and early 21st century there just haven’t been enough robots for any jobs to be lost to automation.

So, it seemed to me, that AI-trained robots didn’t make business sense and we wouldn’t see anyone devoting serious resources to them.

Besides, in the LLM era, there’s plenty of money to be made with chatbots and image generators — why bother building robots? Robots are hard.

Well, as it turns out:

• Figure AI raised $675M from investors including Jeff Bezos, Nvidia, Microsoft, and OpenAI

• Skild reportedly raised somewhere close to $300M for a robotics foundation model

• 1X has raised over $125M, including from OpenAI

• Famed Stanford AI researcher Fei-Fei Li is starting a “spatial reasoning” AI company

I feel a little silly about this. As it turns out, they absolutely will build the AI robots and they are willing to pay what it takes to train them.

More broadly: I treated the field as a lot more cliquish and blinkered than it actually is. I assumed it would be “career suicide” for any researcher/engineer to want to do AI differently than the prevailing paradigm, and that nobody would fund such a person.

So, the kind of advance we’re worried about must come from the rare maverick dreamer types who have their sights fixed on a distant vision of “true” AGI and are willing to spend years scribbling in the wilderness to get there.

Yeah. It now seems like those guys can absolutely get funded and appreciated.

In general it’s just become my impression that there’s no hard wall between “working engineers” who talk about “serious” things like learning rates and batch renormalization and “sci-fi dreamers” who talk about AGI. There was a period in the 2010s when it seemed like there was such a division; and I was pretty sympathetic to the sort of Francois Chollet skeptical perspective that “none of the people who actually work on these models entertain these Rapture-of-the-Nerds fantasies.”

But when Richard Sutton, the guy who wrote the standard textbook on reinforcement learning, has cofounded an “AGI” company, and when the CEO of OpenAI leads his employees in rousing chants of “Feel the AGI”¹ it’s clearly long past time to put that stereotype aside. “Serious” machine learning work and “speculative” ideas about the future of machine intelligence clearly interest the same (or at least heavily overlapping) clusters of people.

I was wrong about how the tech industry works and, for that matter, how markets work and how people work.

I had sort of a baseline assumption that “people” are cliquish, blinkered, and lazy, because I had seen lots of examples of tech stagnation and institutional dysfunction. My model was something like:

“Just because something is possible in principle, and would be really valuable, doesn’t mean it’ll get done; if it did, we’d have widespread nuclear power, we’d have a serious effort to build molecular nanotechnology, we wouldn’t have called a decade+ halt on gene therapy, etc — for that matter, you can’t even expect a single company to necessarily do the Sensible Obvious Best Practice for staying in business. If you’re arguing that something will be done because it makes sense for it to be done, you’re confusing Homo sapiens for some idealized ultra-rational Homo economicus.”

The thing my former worldview was completely missing is selection effects, aka taking markets and incentives seriously.

Of course individual people are often blinkered, cliquish, lazy, irrational, etc — but when there’s a giant pile of money going to the group that can create the most powerful technology, there’s consistent selective pressure towards effectiveness.

"It’s too hard” or “it’s too expensive” or “it’s too weird” are not permanent blockers, absent prohibition by force, to a sufficiently lucrative thing happening.

I’m also thinking of recent examples like Tesla’s camera-only full-self-driving beta.

Doing self-driving with imaging alone (instead of supplementing with LIDAR) is the more short-run-expensive, less well-tested, but aesthetically “right” way to do it — we know it’s possible to drive based on vision alone because that’s how humans do it, and LIDAR’s moving parts make it expensive and unreliable. A bet on camera-only autonomous vehicles is a bet that you can get your software to be really good and you won’t have to compensate by “cheating” with special sensors. I’m no longer on top of the industry state of the art, but Tesla currently has the most advanced autonomy level on the market, so that’s a certain degree of vindication.

Sometimes, people do take the harder, more ambitious, more up-front-expensive path, and it works out for them.

If the “right” way to make more powerful AI systems is something that most of the industry isn’t doing yet, and somebody has a halfway credible pitch for doing it “right”, I now believe that they’re going to get an adequate chance to test that hypothesis.

I used to think that basically nobody who was “philosophically” insightful enough to see through the “humans are basically big LLMs” oversimplification², enough so that they could advance AI beyond what “blind” scaling can offer, would actually want to build a machine superintelligence, because surely anyone that insightful would have the sense to see that machine superintelligence is a catastrophically bad idea.

This also seems wrong to me now, in part because I’ve now interacted with people who do seem to be competent/insightful and also support building “true” AGI.

What I Don’t Know

I don’t currently know what I think about “given artificial general intelligence, what should we expect to happen?”

Despite having absorbed a lot of ideas about this by osmosis, I actually haven’t given it a lot of serious thought, or revised my thinking seriously in light of recent advances.

I was, up until very recently, more sympathetic to the perspective that speculating about what a superintelligent machine would do is futile, that it’s (as one commenter put it) an essentially theological activity like debating how many angels could dance on the head of a pin.

I never had a problem with people doing that sort of thing. I’m not the kind of Puritan who gets offended if anyone else is working on something I expect is most likely useless — certainly I do lots of such things myself! But I just found it kind of hard to take AGI theory (of any kind) seriously.

But now it does seem necessarily true that if we can know anything ahead of time (like whether AGI spells doom or not) then we’ll have to know it through a priori reasoning.

I don’t think a priori reasoning is useless in full generality (that would be self-contradictory, to start with! and it would rule out pure math!) but it’s usually less trustworthy than reasoning that draws from a wealth of empirical evidence. There are no guarantees that even something as important as “are we all gonna die” is something we have enough information to do more than guess about.

I kind of want to try to make sense of it anyway. But I’ll have to do a lot of reading and listening before I have anything like a solid opinion.

The relevant kinds of considerations I want to think about are like:

• to what extent is it a priori “likely” that an “alien”/nonhuman mind (such as an AI) would have goals incompatible with human life³?

• to what extent should we expect “wrapper-mind” like considerations to hold, such that the sort of AI that had a rigid utility function incompatible with our survival, would for that very reason be ineffectual at operating in the real world?

• what “space of possible minds” are we selecting from, and what are we selecting for, when we train AIs (and purchase AI products, and so on)? What do the overall incentives push for? If natural selection shaped human brains, then the analogous “shaper” of AIs is the overall human environment, including but not limited to market forces, that builds models and determines which “survive” in use. To what extent is it a valid argument that by default we’re “selecting” AIs to be safe/friendly/etc?

I’m aware that there’s been a lot of discourse about this already, and that these days it’s gotten acrimonious, but I’ve in effect had my head under a rock for a long time because I thought all of this more “speculative” stuff was irrelevant, so think of me as a friendly time traveler from 2013 or so.

I’ll be at LessOnline and Manifest this week so if any of this interests you, I’d love to talk!

1. and when such luminaries as Geoffrey Hinton, Yoshua Bengio, and Stuart Russell take AI risk seriously!

2. I think there’s a fundamentally anti-intellectual impulse in many people to “round down” what human minds do to something much simpler and more routinized than it really is. I’ve argued against the very most extreme versions of this elsewhere several times and I pretty much stand by the overall point, though not all of my specific examples and arguments hold up.

3. or the sort of “posthuman” life that we would intuitively think of as as much of a worthy successor as our own distant and culturally “alien” biological descendants. A Star Trek type of fictional extraterrestrial or android is easily more sympathetic, more “human”, to a 21st century Western-educated person than Genghis Khan.