I've seen some discussion on HN in which people claimed that even really important engineers aren't -too- important and that Ilya is actually replaceable, using Apple's growth after Woz' departure as an example. But I don't think that's the best situation to compare this to. I think a much better one is John Carmack firing Romero from id Software after the release of Quake.
Some background: During a period of about 10 years, Carmack kept making massive graphics advances by pushing cutting-edge technology to the limit in ways nobody else had figured out, starting with smooth horizontal scrolling in Commander Keen, through Doom's pseudo-3D, through Quake's full 3D, to advances in the Quake sequels, Doom 3, etc. It's really no exaggeration to say that every new id game engine from 1991 to 1996 created a new gaming genre, and the engines after that pushed forward the state of the art. I don't think anybody who knows this history could argue that John Carmack was replaceable.
At the time, the rest of id knew this, which gave Carmack a lot of clout and eventually allowed him to fire co-founder John Romero. Romero was considered the kinda flamboyant, and omnipresent, public face of id -- he regularly went to cons, worked the press, played deathmatch tournaments, and so on (to be clear, he was a really talented level designer and programmer, among other things, I only want to point out that he was synonymous with id in the public eye). And what happened after the firing? Romero was given a ton of money and absurd publicity for new games ... and a few years later, it all went up in smoke and his new company folded, as he didn't end up making anything nearly as big as Doom or Quake. Meanwhile, id under Carmack kept cranking out hit after hit for years, essentially shrugging off Romero's firing like nothing happened.
The moral of the story to me is that, when your revenue massively grows for every bit of extra performance you extract from bleeding-edge technology, engineer expertise REALLY matters. In the '90s, every minor improvement in PC graphics quality translated to a giant bump in sales, and the same is true of LLM output quality today. So, just like Carmack ultimately turned out to be the absolute key driver behind id's growth, I think there's a pretty good chance it's going to turn out that Ilya plays the same role at OpenAI.
> Meanwhile, id under Carmack kept cranking out hit after hit for years, essentially shrugging off Romero's firing like nothing happened.
I don't think that is accurate...
The output of id Software after Romero left (post Quake 1) was a clear step down. The technology was fantastic but the games were boring and uninspired, at best "good" but never "great". It took a full 20 years for them to make something interesting again (Doom in 2016).
After Romero left, id Software's biggest success was really as a technology licensing house, but not as a games developer. Powering games like Half Life, Medal of Honor, Call of Duty, ...
Meanwhile Romero's new company (Ion Storm) eventually failed, but at least the creative freedom there led to some interesting games, like Deus Ex and Anachronox. And even Daikatana is a more interesting game than something like Quake 2 or Quake III.
Deus Ex wouldn’t exist if Romero hadn’t created Ion Storm and given creative freedom to Warren Spector to make his dream game
Daikatana had some interesting design ideas but had problems with technology, esp. with AI programming. It was too ambitious for a new team which lacked someone like Carmack to do the heavy technical lifting
Quake 2 and 3 were reviewed less favourably than earlier titles, and they also sold less copies. They were good but not great - basically boring but very pretty to look at.
The comment you’re replying to wasn’t claiming that Romero designed Deus Ex, but that his leaving id led to the game getting made. It absolutely did.
From Wikipedia:
>After Spector and his team were laid off from Looking Glass, John Romero of Ion Storm offered him the chance to make his "dream game" without any restrictions.
A difference in this case is how capital intensive AI research is at the level OpenAI is operating. Someone who can keep the capital rolling in (whether through revenue, investors, or partners) and get access to GPUs and proprietary datasets is essential.
Carmack could make graphics advances on his own with just a computer and his brain. Ilya needs a lot more for OpenAI to keep advancing. His giant brain isn’t enough by itself.
Carmack did not invent that trick; it had been around more than a decade before he used it. I remember reading a Jim Blinn column about that and other dirty tricks like it in an IEEE magazine years before Carmack "invented" it.
Yes, you're right -- I dug around in the Wikipedia article, and it turns out he even confirmed in an email it definitely wasn't him: https://www.beyond3d.com/content/articles/8/
1. I don't think Ilya is equivalent to Carmack in this case — he's been focused on safety and alignment research, not building GPT-[n]. By most accounts Greg Brockman, who quit in disgust over the move, was more impactful than Ilya in recent years, as well as the senior researchers who quit yesterday.
2. I think you are underselling what happened with id: while they didn't blow up as fantastically as Ion Storm (Romero's subsequent company), they slowly faded in prominence, and while graphically advanced, their games no longer represented the pinnacles of innovation that early Carmack+Romero id games represented. They eventually got bought out by Zenimax. Carmack alone was much better than Romero alone, but seemingly not as good as the two combined.
3. I don't think Sam Altman is equivalent to John Romero; Romero's biggest issue at Ion Storm was struggling to ship anything instead of endlessly spinning his wheels chasing perfection — for example, the endless Daikatana delays and rewrites. Ilya's primary issue with Altman was he was shipping too fast, not that he was unable to motivate and push his teams to ship impressive products quickly.
I hope Sam and Greg start a new foundational AI company, and if they do, I am extremely excited to see what they ship. TBH, much more excited than I am currently by OpenAI under a more alignment-and-regulation regime that Ilya and Helen seems to want.
Sutskever has shifted to safety and alignment research this year. Previously he was directly in charge of the development of GPT, from GPT-1 on.
Brockman did an entirely different type of work than Sutskever. Brockman's primary focus was on the infrastructure side of things - by all accounts the software he wrote to manage the pre-training, training, etc., is all world-class and a large part of why they were able to be as efficient as they are, but that is not the same thing as being the brains behind the ML portion.
Until I can trust that when I send an AI agent off to do something that I will be successful without me babysitting and watching over it constantly AI won't truly be transformative (since the human bottleneck will remain).
This is one of the core promises of alignment. Without it how can there be trust? While there are probably short term slow downs with an alignment focus, ultimately it is necessary to avoid throwing darts in the dark.
I wouldn't mind a focus on reliably following tasks with greater intelligence; what I think is negative utility is focusing more compute and research resources on hypothetical superintelligence alignment — the entire focus of Ilya's "Superalignment" project — when GPT-4 is still way, way sub-human-intelligence. For example, I don't think the GPT Store was in any way a dangerous idea, which seems to have been Ilya's claimed safety red line.
I wouldn't call GPT-4 sub-human intelligence. While it's intelligence is less robust aggregate human intelligence, I don't think there is any one person alive who can compete with the breadth of GPT-4 knowledge.
I also think that the potential of what currently is possible with existing models has not been fully realized. Good prompting strategies and reflection may already be able to produce a system that is effectively AGI. Might already exist in several labs.
Wikipedia has broader knowledge, and yet no one calls it intelligent. I'm talking about reasoning capability, and GPT-4 is well below human on complex tasks, especially multi-step tasks, which is why "autonomous agents" like AutoGPT, BabyAGI, etc are not yet very useful.
> Meanwhile, id under Carmack kept cranking out hit after hit for years, essentially shrugging off Romero's firing like nothing happened.
Romero was fired in 1996
Until this point, as you mentioned id had created multiple legendary franchises with unique lore, attributes, and each one groundbreaking tech breakthroughs: Commander Keen, Wolfenstein 3D, Doom, Quake.
* And absolutely nothing else of any value or cultural impact. The only "original" thing was Rage which again had no footprint.
There were a lot of technical achievements, yes, but it turns out that memorable games need more than interesting technology. They were well-reviewed for their graphics at a time when that was the biggest thing people expected from new id games - interesting new advances in graphics. For a while, they were THE ones pushing the industry forward until arguably Crysis.
But the point is for anyone experiencing or interacting with these games today, Quake is Quake. Nobody remembers 1, 2 or 3 - it's just Quake.
Now, was id a successful software company and business? Yes. Would it have become the industry titan and shaped the future all of all videogames based on their post Romero output? Absolutely not.
So, while it is definitely justifiable to claim that Carmack achieved more on his own than Romero did, the truth is at least in the video game domain they needed each other to achieve the real greatness that they will be remembered for.
It remains to be seen what history will say about ALtman and Sutskever.
Quake 3 came out 1 week after Unreal Tournament did in 1999.
Quake 3 had a better engine, but Unreal Touranment had more creative weapons, sound cues, and level design. (Assault mode!)
Quake 3 had better balanced levels for purely deathmatch, which turned out to be the part that was the purest distillment of what people would want to play.
So, yes, I do think you're right that I am underselling Quake 3. I was always a UT fan from day 1, and never understood why Quake 3 took over. But that's personal preference, and I undervalue it's impact to the industry.
It also shows I guess that since Romero previously did all the level designs, Carmack was able to replace him. But Romero was never able to replace Carmack.
Meanwhile, id under Carmack kept cranking out hit after hit for years, essentially shrugging off Romero's firing like nothing happened.
I believe this is absolutely wrong. Quale 2, 3 and Doom 3 were critical success, not commercial ones, which led ID to be bought.
John and John were like Paul and John from the beatles, they never made really great games anymore after their break up.
And to be clear, that's because the role of Romero in the success of ID is often underrated like here. He invented those games (Doom and Quake and Wolf) as much as Carmack did. For example, Romero was the guy who invented percent-based life. He removed the score. This guy invented the modern video game in many ways. Games that werent based on Atari or Nintendo. He invented Wolf, Doom and Quake setups which were considerably more mature than Mario and Bomberman and it was new at the time. Romero invented the deathmatch and its "frag". And on and on.
I think the team that became Looking Glass Studios did a lot of the same things in parallel so it’s a little unfair to say no one else had figured it out
Not at the same level of quality. For example, their game, ultima underworld if my memory doesnt fault me, didnt have sub-pixel precision for texturing. Their texturing was a lot uglier and unpolished compared to Wolf and especially Doom. I remember I checked, they were behind. And their game crashed. Never saw Doom crash, not even once.
Isn't that a massive quality improvement though? How many applications for LLMs are not feasible right now because of the ability for models to be swayed of course by a gentle breeze? If AI is a ship with a sail, data is the wind and alignment is the equivalent of a rudder.
> models to be (easily) swayed is a different problem
No, this is the alignment problem at a high level. You want a model to do X but sometimes it does Y.
Mechanistic interpretability, one area of study in AI alignment, is concerned with being able to reason about how a network "makes decisions" that lead it to an output.
If you wanted an LLM that doesn't succumb to certain prompt injections, it could be very helpful to be able to identity key points in the network that took the AI out of bounds.
Edit: I should add, I'm not referring to AI safety, I'm referring to AI alignment.
Some background: During a period of about 10 years, Carmack kept making massive graphics advances by pushing cutting-edge technology to the limit in ways nobody else had figured out, starting with smooth horizontal scrolling in Commander Keen, through Doom's pseudo-3D, through Quake's full 3D, to advances in the Quake sequels, Doom 3, etc. It's really no exaggeration to say that every new id game engine from 1991 to 1996 created a new gaming genre, and the engines after that pushed forward the state of the art. I don't think anybody who knows this history could argue that John Carmack was replaceable.
At the time, the rest of id knew this, which gave Carmack a lot of clout and eventually allowed him to fire co-founder John Romero. Romero was considered the kinda flamboyant, and omnipresent, public face of id -- he regularly went to cons, worked the press, played deathmatch tournaments, and so on (to be clear, he was a really talented level designer and programmer, among other things, I only want to point out that he was synonymous with id in the public eye). And what happened after the firing? Romero was given a ton of money and absurd publicity for new games ... and a few years later, it all went up in smoke and his new company folded, as he didn't end up making anything nearly as big as Doom or Quake. Meanwhile, id under Carmack kept cranking out hit after hit for years, essentially shrugging off Romero's firing like nothing happened.
The moral of the story to me is that, when your revenue massively grows for every bit of extra performance you extract from bleeding-edge technology, engineer expertise REALLY matters. In the '90s, every minor improvement in PC graphics quality translated to a giant bump in sales, and the same is true of LLM output quality today. So, just like Carmack ultimately turned out to be the absolute key driver behind id's growth, I think there's a pretty good chance it's going to turn out that Ilya plays the same role at OpenAI.