Stanford CME295 · Afshine & Shervine Amidi

Agentic LLMs: RAG, Tool Calling, and Agents (Stanford CME295, Lecture 7)

A trained LLM only knows what was in its data and can only talk back at you. This lecture is about closing both gaps — connecting it to current knowledge with RAG, and letting it act through tool calling and agents. It builds up in order, with a running teddy-bear example, from retrieval to the ReAct loop to multi-agent systems and safety.

Creator: Stanford OnlineRuntime: 109 minVideo published: November 18, 202582.5K views when summarized

Watch on YouTube

TL;DR

The lecture in one pass

Two limits of a trained LLM: it only knows its training data (fixed at a cutoff), and on its own it can only produce text. RAG fixes the first; tools and agents fix the second.
RAG = retrieve, augment, generate. The hard part is retrieval: chunk your documents, embed them, then retrieve in two stages — fast candidate retrieval (semantic embeddings, optionally keyword BM25) then a heavier cross-encoder reranking — measured with NDCG, MRR, precision@k, recall@k.
Tool (function) calling is a three-stage loop: the LLM fills a documented function's arguments, you run the function, then the LLM turns the structured result into a natural answer. You can teach it with SFT pairs — or skip SFT and iterate a prompt explanation against an eval set.
Scaling tools: a tool-selector/router narrows a huge tool list to the few relevant ones; MCP (Anthropic) standardizes how tools are exposed so you don't reimplement them per model.
An agent adds reasoning loops on top of tools. ReAct = observe → plan → act, repeated until the goal is met. Multiple agents can coordinate (Google's agent-to-agent protocol) — which raises real safety risks like data exfiltration, handled at training and inference time.

01 · Why connect an LLM to anything

The cutoff problem

A trained model only knows what was in its data, frozen at a knowledge cutoff (GPT-5's, for instance, is September 2024). Retraining to add knowledge is risky and impractical, and you can't just dump everything into the prompt: context windows are finite (~hundreds of thousands of tokens), you pay per token, and performance actually drops when the prompt is stuffed with irrelevant text.

The "needle in a haystack" test: hide a fact in a long prompt and ask for it back. Past a certain length the model struggles to find it — worst when the fact sits in the first half. So even unlimited context wouldn't save the naive approach.00:13:05

02 · RAG

Retrieve, augment, generate

The fix is to put only the relevant information in the prompt. That's RAG: retrieve the relevant documents for a query, augment the prompt with them, then generate the answer. In effect you're handing the model the answer inside the prompt — so the whole technique lives or dies on the retrieval step.

The three steps. "When we talk about RAG, we're mainly focusing on making the retrieval part as good as it can [be]."00:18:40

03 · The knowledge base

Chunks, embeddings, overlap

Before you can retrieve, you build a knowledge base: collect the documents, split them into chunks (a few hundred tokens each), and compute an embedding for each chunk. Three knobs to tune: embedding size (≈ thousands of dimensions; bigger captures nuance but costs space and compute), chunk size (too small loses context, too large blurs meaning), and overlap between chunks (so a chunk keeps the context of the one before it).

Collect → divide → embed. The embedding model can be pre-trained or trained yourself; Sentence-BERT is the recommended read for how those embeddings are learned for similarity search.00:20:45

04 · Retrieval in two stages

Candidate retrieval, then ranking

Retrieval borrows the recommender-systems playbook: two stages. Candidate retrieval quickly narrows millions of chunks to ~100 by embedding the query and doing a cosine-similarity search (using approximate-nearest-neighbor methods to avoid a linear scan). Because query and chunks are embedded separately, this is a bi-encoder setup, and the goal here is recall — cast a wide net.

Stage 1 maximizes recall with semantic embeddings; a second ranking stage (next) tightens it.00:24:30

05 · Semantic vs. keyword

BM25 and hybrid search

Embedding search finds chunks that mean the same thing — but it won't guarantee the exact keyword is present. When you need that (searching for "Cuddly," not just something cuddly-adjacent like "Huggy"), BM25 is a heuristic score based on word overlap between query and document. Many systems run a hybrid of embedding search plus BM25.

"Where is Cuddly?" — semantic search might return the semantically-similar Huggy; BM25 guarantees the keyword match. Pick per use case, or combine.00:35:25

Matching query to documentsA short question and a long document don't embed comparably. HyDE mitigates this: first have the LLM generate a hypothetical answer-document from the query, then embed that to retrieve. (Alternatively, train separate encoders for queries and documents.)

06 · Smarter chunks, cheaper calls

Contextualization + prompt caching

Chunks cut at a fixed token count can lose their meaning out of context. Contextual retrieval prepends a short, LLM-written summary of what you need to know to understand each chunk — based on the whole document. That's a lot of LLM calls, so it leans on prompt caching: reuse the same prefix across prompts and you compute its activations once, then look them up.

Cached input tokens are cheap — on the order of one-tenth the price of regular input — so gather everything repeated across prompts into the prefix.00:43:55

07 · Reranking and metrics

Did retrieval actually work?

The optional second stage is reranking: instead of comparing pre-computed embeddings, feed the query and a candidate chunk together into the model (a cross-encoder) so attention can capture their interaction, and score relevance directly. It's heavier, but you only run it on the ~100 candidates. Then you measure.

NDCG

ranking quality, weighted toward the top, normalized to [0,1]

MRR

inverse rank of the first relevant result

Precision@k / Recall@k

the classic metrics, over your top-k

MTEB

a standard benchmark to compare retrievers

08 · Tool calling

A three-stage loop

Where RAG injects unstructured documents, tool (function) calling lets the LLM act through structured functions. You show the model a documented function — its name, arguments, and docstring, but not the implementation. The loop has three stages: the LLM picks the function and fills its arguments from your query; your code actually runs the function; then the LLM turns the structured result into a natural-language answer.

The running example: `find_teddy_bear(location)` calls an API, returns a structured object, and the model reads it back to you in words. Tools span information (search, weather, stocks), computation (run code), and actions (send an email — and hit send).01:10:20

09 · Teaching a model to use a tool

SFT, or just an explanation

How do you get the model to do this? One way: two sets of SFT pairs — one mapping query → tool call, one mapping the tool's result (plus conversation history) → final answer. But modern models already read code well, so you can often skip SFT and just write a prompt explanation of the tool.

Don't hand-write the explanation end to end. Turn your SFT pairs into an eval set, score a draft explanation against it, and feed the wins/losses to a reasoning model to rewrite it — iterating offline until it generalizes.01:17:50

10 · Too many tools

Selection and MCP

In practice you expose many tools, and a big tool list recreates the needle-in-a-haystack problem — plus context is finite, so you can't fit everyone's tools at once. A tool selector (or router) fixes the first: given the query and a list of tool names, an LLM picks the few relevant ones, and only those go into context. The second problem — every model defining tools its own way — is what MCP addresses.

Without a standard, every LLM × tool pairing is a separate implementation. MCP (the Model Context Protocol, from Anthropic) standardizes how tools are exposed — an MCP *server* serves tools, prompts, and resources to an LLM host.01:29:00

11 · Agents

The ReAct loop

An agent sits one layer above tools: it autonomously pursues a goal, with reasoning loops, not just a single call. The hallmark is ReAct (reason + act), which decomposes a task into repeated stages — here framed as observe → plan → act.

"My teddy bear is cold, please do something." Observe turns that into a temperature problem; plan decides to check the room; act calls `get_current_room_temperature()`; observe reads 65°F as colder than expected; plan/act raise it — then the loop exits and answers the user.01:36:40

12 · Many agents

The agent-to-agent protocol

Once you have one agent, you can have several — a thermostat agent, an energy-management agent, an air-quality agent — and let them coordinate. That need for a common language between agents is why Google released the agent-to-agent (A2A) protocol: each agent exposes a set of skills (with examples), plus a defined way to execute a request, report status, and cancel.

Independent agents, each with its own reasoning loop, communicating through a shared protocol — the analogue of MCP, but between agents rather than between an LLM and its tools.01:39:40

13 · Safety, and how to build

Taste is what's left

New power, new risks: a model that can act can be turned against you — e.g. data exfiltration, where a prompt makes an email tool send out a user's password. Defenses come at two points: training (safety data in the SFT/RL mix) and inference (a safety classifier judging the conversation). The lecture notes Anthropic's just-published report on a large-scale cyberattack that abused tool and agent capabilities — proof the stakes are real.

Risks like data exfiltration, with benchmarks (Agent Safety Bench, ToolSword) to test against. The build advice: start small and start smart (best model first), watch the reasoning chains to debug.01:43:00

Generating code is cheap, but judging whether a code is correct and does the right thing — this is the hard part. Your taste is going to matter most from now on.— Shervine Amidi