What Is Generative AI — Really?
Generative AI is not a new concept. It's been around for a while." We've been living with it for years — Google Translate, Siri, autocomplete — without calling it that. This lecture strips away the hype to explain what these systems actually do, how they got so big, and what the real risks are.
Watch on YouTubeTL;DR
- Generative AI has been with us for over a decade — Google Translate launched in 2006, Siri in 2011
- ChatGPT hit 100M users in 2 months; Google Translate took 78
- Core technology is deceptively simple: predict the next word from context
- Scaling matters enormously — capabilities emerge only at large parameter counts
- Alignment (helpful, honest, harmless) requires expensive human-in-the-loop fine-tuning
- Hallucinations have real costs — Bard's fake JWST claim wiped $100B from Google's market cap
01 · The past · Generative AI is not new
"We get a computer programme to do the job that a human would otherwise do." And "generative" just means it creates new content the computer hasn't necessarily seen — audio, code, images, text, video.
The speaker's opening premise: generative AI has been invisible in plain sight. Google Translate (2006), Siri (2011), phone autocomplete, Google search prediction — all generative AI. Nobody was "making a fuss" about them.
Yesterday's history, tomorrow is a mystery, today is a gift, and that's why it's called the present.— Mirella Lapata
02 · The 2023 breakthrough · ChatGPT broke the speed record
OpenAI's GPT-4 claimed it could beat 90% of humans on the SAT, get top marks in law and medical exams. But the real story was adoption speed.
03 · How it works · Predict the next word
The core technology is "deceptively simple": given a sequence of words (context), predict what comes next. The training process is self-supervised — take a corpus of text, randomly remove parts, ask the model to predict what was removed.
The model architecture is the transformer (2017) — blocks of neural networks stacked on top of each other. "Nobody's working on new architectures right now... everybody's using transformers."
Training is "dead cheap" in principle: you already have the ground truth (the words you removed). You just keep going for months, adjusting weights until the predictions converge.
When I hear like 'AI will take over the world,' I go like, 'Really?' We have to actually spell it out.— Mirella Lapata
04 · Scaling · The bigger the better
"Size is all that matters, I'm afraid." Since 2018, model sizes have grown exponentially. GPT-4 has one trillion parameters. The human brain has 100 trillion.
But the interesting discovery was that scaling doesn't just make things better — it makes new things possible. At 8B parameters, a model can do a few tasks. At 540B, it can do code completion, reading comprehension, translation, and more. "The tree flourishes."
GPT-4 cost ~$100 million to train. Not everybody has Microsoft behind them.
05 · Alignment · Helpful, honest, harmless
The pre-trained model is general-purpose, but it doesn't behave the way humans want. So you fine-tune it — not just on tasks, but on preferences. Humans rate pairs of outputs and the model learns to prefer the better one.
This is the HHH framework: helpful, honest, harmless. It's the bare minimum. The process is "expensive as it was, now we make it even more expensive because we add a human into the mix."
The live demo showed the gap between capability and alignment. When asked "Is the UK a monarchy?", ChatGPT said Queen Elizabeth III was reigning monarch — at the time of its training data, yes, but obviously wrong. When asked to write a short song about relativity, it produced a 4-verse poem. "It doesn't follow instructions. It is not helpful."
06 · Risks & the future · Climate change is the bigger threat
Hallucinations aren't academic — Google's Bard once claimed JWST took the first image of an exoplanet. An astrophysicist corrected it publicly. "$100 billion out of Google's company Alphabet."
The environmental cost is steep: a single inference query takes 100× more energy than a Google search. Llama 2's training produced 539 metric tonnes of CO₂.
Society-level risks: Goldman Sachs predicted 300M jobs at risk. Fakes are already a problem — a college student wrote a blog that fooled reviewers; unknown artists created a Drake/Weeknd deepfake that sounded "totally believable."
Every time you think about AI coming there to kill you, you should think what is the bigger threat to mankind, AI or climate change? I would personally argue climate change is gonna wipe us all before the AI becomes super intelligent.— Mirella Lapata
Every risky technology has been strongly regulated — nuclear energy, chemicals, aviation
The Australian Research Council tested GPT-4 on autonomous replication, resource acquisition, and self-preservation — it failed on all of them
Tim Berners-Lee's view: some AIs will be good and will help us; some may fall to the hands of those who want to do harm; it seems easier to minimize the harm than to prevent the systems from existing
:::
TL;DR
- Generative AI has been with us for over a decade — Google Translate launched in 2006, Siri in 2011
- ChatGPT hit 100M users in 2 months; Google Translate took 78
- Core technology is deceptively simple: predict the next word from context
- Scaling matters enormously — capabilities emerge only at large parameter counts
- Alignment (helpful, honest, harmless) requires expensive human-in-the-loop fine-tuning
- Hallucinations have real costs — Bard's fake JWST claim wiped $100B from Google's market cap
- Climate change is a far bigger threat than superintelligent AI