Large Language Model (LLM) Fundamentals

Before diving into agents and protocols, it helps to understand what's happening under the hood of a large language model. This page is the conceptual foundation for every other page in these docs.

What Exactly Is an LLM?

A large language model is a neural network trained on a massive volume of text to do one simple but powerful thing: given a sequence of text, predict the most likely "next token." By repeating that prediction, token by token, sentences and paragraphs are built up. The surprising part is that abilities like reasoning, summarization, translation, and even writing code emerge from this seemingly simple task.

Tokenization

Language models don't work directly with letters or words; text is first broken into smaller units called tokens (which can be a whole word, part of a word, or even a single character). For example, "hello world" might be split into two or three tokens. The cost of using most language model APIs is also calculated based on the number of these tokens, not the number of characters or words.

Transformer Architecture

Nearly all of today's LLMs are built on the Transformer architecture, introduced in 2017. Its key mechanism is self-attention: to predict each token, the model looks at every preceding token and assigns a "weight" to how important each one is for that prediction. This mechanism lets the model capture long-range dependencies in text (for example, remembering the subject of a sentence from three lines earlier).

Training Stages

1. Pretraining — the model trains on a massive volume of general text (web, books, code) to learn language and general knowledge.
2. Fine-tuning — the model is further tuned on smaller, more targeted datasets (e.g. conversation or instructions).
3. RLHF / Alignment — using human feedback, the model learns to produce answers that are more helpful, safer, and better aligned with user intent.

The Context Window

On each call, the model only has access to a limited amount of text (measured in tokens) as its "working memory"; this is called the context window. Anything outside this window effectively doesn't exist for the model on that particular call — this limitation is the main reason concepts like RAG and agent memory exist.

Basic Prompting

Most modern language model APIs receive a conversation as a list of messages with a defined role:

[
  { "role": "system", "content": "You are an online store assistant." },
  { "role": "user", "content": "How much is the model X wireless headset?" }
]

The system role sets the model's overall behavior, user is the user's message, and assistant is the model's reply.

Output Sampling Parameters (Decoding)

At each step, the model produces a probability distribution over its entire vocabulary for the next token, not a single definite answer. Which token gets picked from that distribution depends on a few tunable parameters:

• Temperature — how much "risk" is taken in the choice. A value near zero means the most likely token is always picked (more predictable, more repetitive output); a higher value (e.g. above 1) gives less likely tokens a chance too (more varied output, but a higher risk of error).
• Top-p (Nucleus Sampling) — instead of considering the whole vocabulary, the model only picks from the smallest set of tokens whose combined probability reaches p (e.g. 0.9). It's combined with Temperature to control diversity while fully excluding irrelevant, very-low-probability tokens.
• Top-k — similar to Top-p, but instead of a probability threshold, a fixed number of the most likely tokens (e.g. the top 40) is considered.

For use cases that need precision and consistency (like structured data extraction or tool calls in an agent), a lower Temperature is usually chosen; for creative content generation, a higher value.

Limitations

• Hallucination — the model may confidently produce incorrect information.
• Knowledge Cutoff — the model only "knows" what it saw up to its training cutoff.
• Context window limits — it can't process an unlimited amount of text at once.
• Non-determinism — the response to the same input may vary slightly each time.

Why These Fundamentals Matter for Agentic Commerce

When an AI agent is about to make a purchase on a user's behalf, these very limitations directly shape your system design: the knowledge cutoff means the agent needs to fetch real-time information (like stock levels) through a tool — exactly what MCP enables. The context window limit means you can't fit an entire product catalog into a prompt, so you need smart retrieval — the subject of the RAG page.

FAQ