WEBVTT 00:00:01.620 --> 00:00:20.080 Tuesday, November fifth, two thousand nineteen. Nine months after OpenAI said it wouldn't release GPT-2, it releases GPT-2. All of it. The full model, one point five billion parameters, with the code and the weights. 00:00:20.080 --> 00:00:48.360 The post lists what they learned. People found the text convincing. Researchers at Cornell had people rate its credibility, and the full model scored six point nine one out of ten. Researchers at the Middlebury Institute showed it could be fine-tuned to write extremist propaganda. OpenAI built a detector that caught about ninety-five percent of the full model's text, and said that wasn't good enough on its own. 00:00:48.360 --> 00:00:50.540 And then, finding number four. 00:00:50.540 --> 00:00:55.360 "We've seen no strong evidence of misuse so far." 00:00:55.360 --> 00:01:14.660 That's the whole arc of the most famous AI scare of twenty-nineteen. Locked up in February. Called "too dangerous to release" by the press. Let out in pieces. And by November, released in full, with no strong evidence anyone had used it to do harm. 00:01:14.660 --> 00:01:34.980 You can hear that as proof the caution was overblown. Or as proof the caution worked. OpenAI's own post did neither. It said they couldn't know about every threat, and that people who wanted a model like this could build one without them. 00:01:34.980 --> 00:01:47.920 From Will Sigmon Media, this is The Role Models. Season one: OpenAI. Episode six. The Escalator. 00:01:47.920 --> 00:01:59.820 Back in February, when GPT-2 was still locked up, OpenAI's head of policy, Jack Clark, gave the Guardian a phrase. I promised you we'd come back to it. 00:01:59.820 --> 00:02:13.500 "I have a term for this. The escalator from hell. It's always bringing the technology down in cost and down in price. The rules by which you can control technology have fundamentally changed." 00:02:13.500 --> 00:02:35.720 An escalator. You don't climb it. You stand still, and it carries you up anyway. Clark's point was that you can hold back one model, but you can't hold back the falling cost of building the next one. And the GPT-2 release in November said almost the same thing: other people could replicate it without them. 00:02:35.720 --> 00:02:42.080 In the same interview, he said something else. "We're trying to build the road as we travel across it." 00:02:42.080 --> 00:02:55.280 In twenty-twenty, OpenAI stepped onto the escalator on purpose. And they built the next model, and the road for selling it, at the same time. 00:02:55.280 --> 00:03:11.400 January twenty-third, two thousand twenty. Before GPT-3 had a name, ten researchers at OpenAI posted a paper with a title that sounds like a physics lecture: Scaling Laws for Neural Language Models. 00:03:11.400 --> 00:03:27.400 Its claim fits in one sentence. Read it slowly. "The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude." 00:03:27.400 --> 00:03:41.280 Loss is the error, how wrong the model is. And a power law means it's predictable. Make the model bigger, give it more data and more computing, and it gets better, along a line you can draw in advance. 00:03:41.280 --> 00:03:52.680 That's the escalator. If you believe that line, then the question is no longer whether bigger models will be smarter. It's who can afford the next step. 00:03:52.680 --> 00:04:08.680 Look at who signed it. The first author was Jared Kaplan. The second was Sam McCandlish. The last was Dario Amodei. Remember those names. They'll be on a different letterhead soon. 00:04:08.680 --> 00:04:22.480 Thursday, May twenty-eighth, two thousand twenty. The world is in its third month of lockdown. And OpenAI posts a paper with a dry title: Language Models are Few-Shot Learners. 00:04:22.480 --> 00:04:40.500 It introduces GPT-3. One hundred seventy-five billion parameters. GPT-2 had one point five billion. That's more than a hundred times bigger. The paper says it's ten times bigger than any previous model of its kind. 00:04:40.500 --> 00:05:05.020 The title is the discovery. Before this, to teach a language model a new task, you'd retrain it on thousands of examples. GPT-3 didn't need that. You showed it a few examples, in plain text, and it picked up the pattern. Translation. Questions and answers. Unscrambling words. Three-digit arithmetic. 00:05:05.020 --> 00:05:20.380 In one test, the authors gave GPT-3 a made-up word, a farduddle, which means to jump up and down really fast, and asked it to use another made-up word in a sentence. It did, without being trained to. 00:05:20.380 --> 00:05:41.380 And there was a finding the authors flagged as a risk. They had GPT-3 write short news articles and asked about eighty people per test to tell them apart from articles written by humans. For the biggest model, people got it right about fifty-two percent of the time. A coin flip is fifty. 00:05:41.380 --> 00:06:00.380 Two more details from that paper. Where it was trained: on a high-bandwidth cluster provided by Microsoft. And at the very end of the author list, Ilya Sutskever, and then, last of thirty-one names, Dario Amodei. 00:06:00.380 --> 00:06:10.700 Two weeks later. Thursday, June eleventh. OpenAI announces something it had never had before. A product. 00:06:10.700 --> 00:06:16.380 We're releasing an API for accessing new AI models developed by OpenAI. 00:06:16.380 --> 00:06:29.220 An API: a way for other companies' software to send text to OpenAI's model, and get text back. The post describes it the way you'd describe a vending machine. 00:06:29.220 --> 00:06:38.940 The API today provides a general-purpose text in, text out interface, allowing users to try it on virtually any English language task. 00:06:38.940 --> 00:06:50.380 Behind the counter: models from the GPT-3 family. It launched as a private beta, with a waitlist. And the post says plainly what it's for. 00:06:50.380 --> 00:07:00.980 In addition to being a revenue source to help us cover costs in pursuit of our mission, the API has pushed us to sharpen our focus on general-purpose AI technology. 00:07:00.980 --> 00:07:28.680 A revenue source. Eleven months earlier, the Microsoft post had said building a product "would mean changing our focus." Now the product is here, and the post says it sharpened their focus. The post took the question on directly. Why a commercial product? Their answer: "We see developing commercial products as one of the ways to make sure we have enough funding to succeed." 00:07:28.680 --> 00:07:51.600 And why rent the model out instead of releasing it, the way GPT-2 finally was? The post gave three reasons. It pays for research. The models are expensive to run. And, third, renting is safer, because they could cut off customers who misused it. With an open model, you can't take it back. 00:07:51.600 --> 00:08:04.320 That's the GPT-2 lesson, turned into a business model. Don't lock it up, and don't let it go. Keep your hand on the switch, and charge for access. 00:08:04.320 --> 00:08:20.160 That summer, people with early API access started posting what GPT-3 could do. Code from a sentence. Fake poems by real poets. The demos spread fast, and the claims grew faster. 00:08:20.160 --> 00:08:27.440 On July nineteenth, Sam Altman tried to turn the volume down. On Twitter, he wrote: 00:08:27.440 --> 00:08:37.920 The GPT-3 hype is way too much. It's impressive, thanks for the nice compliments, but it still has serious weaknesses and sometimes makes very silly mistakes. 00:08:37.920 --> 00:08:46.080 AI is going to change the world, but GPT-3 is just a very early glimpse. We have a lot still to figure out. 00:08:46.080 --> 00:09:04.380 Seven weeks later, on September eighth, the Guardian ran an opinion piece under a headline built to travel: "A robot wrote this entire article. Are you scared yet, human?" It opened: "I am not a human. I am a robot. A thinking robot." 00:09:04.380 --> 00:09:30.380 Now read the note at the bottom. The Guardian wrote the instructions. A University of California, Berkeley undergraduate named Liam Porr fed them to GPT-3. It produced eight different essays. And the editors, quote, "chose instead to pick the best parts of each." They wrote: "We cut lines and paragraphs, and rearranged the order of them in some places." 00:09:30.380 --> 00:09:53.380 So: a machine wrote the sentences. People chose which ones, and in what order. The headline said entire. The editor's note said otherwise. That gap, between what the model did and what people said it did, is going to follow this company for years. 00:09:53.380 --> 00:10:10.920 Tuesday, September twenty-second, twenty-twenty. Microsoft's chief technology officer, Kevin Scott, posts on the company blog. Microsoft, he writes, is teaming up with OpenAI to "exclusively license GPT-3." 00:10:10.920 --> 00:10:28.960 He also reminds readers where GPT-3 lives. That May, Microsoft had announced a supercomputer built for OpenAI, hosted on its Azure cloud. GPT-3, he writes, "is trained on Azure's AI supercomputer." 00:10:28.960 --> 00:10:55.720 Microsoft's own feature on that machine, back in May, had given the numbers. "More than two hundred eighty-five thousand CPU cores, ten thousand GPUs and four hundred gigabits per second of network connectivity for each GPU server." One of the top five publicly disclosed supercomputers in the world, it said, "built in collaboration with and exclusively for OpenAI." 00:10:55.720 --> 00:11:11.760 And it quoted Sam Altman on how it happened. OpenAI had asked itself, "If we could design our dream system, what would it look like?" Then, Altman said, "Microsoft was able to build it." 00:11:11.760 --> 00:11:35.900 What did exclusive mean? Microsoft could build GPT-3 into its own products. And OpenAI, the post says, "will continue to offer GPT-3 and other powerful models via its own Azure-hosted API." So the public could still rent it, through OpenAI. But the underlying model was licensed to one company. 00:11:35.900 --> 00:11:57.160 Go back to the bet from last episode. A billion dollars, an exclusive cloud, and a line about licensing "pre-AGI technologies" with Microsoft as the preferred partner. Fourteen months later, the first big pre-AGI technology had a name, and it had an exclusive licensee. 00:11:57.160 --> 00:12:25.720 Years later, this is the arrangement Elon Musk's lawsuit would point to. His February twenty-twenty-four complaint alleged that OpenAI had become, in its words, "a closed-source de facto subsidiary of the largest technology company in the world: Microsoft." Those are allegations. OpenAI and Microsoft have disputed them, and none of it has been decided in this telling. 00:12:25.720 --> 00:12:37.800 December twenty-twenty. OpenAI holds an employee meeting. And one of the two people who'd been setting the direction of its research tells his coworkers he's leaving. 00:12:37.800 --> 00:12:57.200 His name has been at the end of the author lists all season. Dario Amodei. Vice president of research. He'd come to OpenAI from Google in twenty-sixteen. He'd helped build GPT-2 and GPT-3, and co-led the research with Ilya Sutskever. 00:12:57.200 --> 00:13:01.760 OpenAI's post quotes what he told his coworkers that day. 00:13:01.760 --> 00:13:14.340 "I want to thank Sam and thank everyone. I'm really proud of the work we've done together. I want to wish everyone the best, and I know that OpenAI will do really great things in the years ahead." 00:13:14.340 --> 00:13:33.260 And he added a line that sounds like a farewell, and also like a warning. "We share the same goal of safe artificial general intelligence to benefit humanity, so it's incumbent on all of us in this space to work together to make sure things go well." 00:13:33.260 --> 00:13:49.720 The announcement went up on December twenty-ninth, between the holidays, under the title "Organizational update from OpenAI." It says he's leaving after nearly five years. And it says he isn't leaving alone. 00:13:49.720 --> 00:13:58.440 We wish him and his co-founders all the best in their new project, and we look forward to a collaborative relationship with them for years to come. 00:13:58.440 --> 00:14:09.740 Co-founders. A new project. The post describes it in one sentence: a project that will "probably focus less on product development and more on research." 00:14:09.740 --> 00:14:16.440 Less product. Six months after OpenAI launched its first one. 00:14:16.440 --> 00:14:32.360 In the same post, OpenAI announced a promotion. Mira Murati, who'd led the API rollout, became senior vice president of Research, Product, and Partnerships. Research, and product, in one title. 00:14:32.360 --> 00:14:46.060 The post doesn't say why Amodei left. Neither does his farewell. He's spoken about it since, and so have others, and the accounts don't all agree. That's next time. 00:14:46.060 --> 00:14:54.060 Next time, on The Role Models. The Quiet Build.