Session 16 – Agentic AI DevOps: Text Generation Models

Agentic AI DevOps: Text Generation Models

Recap of the previous session

• Static embeddings capture the meaning of a token. Contextual embeddings capture the meaning in the sentence.
• The transformer has an encoder and a decoder. Contextual embedding models use only the encoder.
• Attention weighs how much each word matters to the others. Models are trained on text itself: causal (predict the next word, GPT) or masked (predict a hidden word, BERT).
• A sentence transformer gives one vector per sentence, and cosine similarity lets you match, for example, an alert to a runbook.

In this session

• Encoder-only vs. decoder models
• Logits, softmax and probabilities
• GPT-2: predicting the next token step by step
• Generating text: the append-and-repeat loop
• Controlling the output: greedy, temperature and top-k
• What an LLM is, and how models evolved over time

In one sentence: a text generation model predicts one token at a time, adds it to the text, and repeats, and settings such as temperature control how adventurous the choice is.

1. Encoder vs. decoder

• A model with only the encoder gives you embeddings (BERT and the sentence transformers). It cannot write new text.
• A model with the decoder can predict the next token for the text you give it. This is a text generation model.

So you switch from embedding models to text generation models when you want the machine to write, not only to understand.

Inside the decoder, attention works on the previous tokens. Given “Rose loves sweet fresh fruits”, the model works out how much each earlier word matters for predicting the next one, applies those weights, and produces a prediction.

2. Logits and probabilities

Every model starts by applying its equation to the input. In the simplest case, that is m·x + b, which gives a raw number such as 452. This raw output is a logit.

• A logit is not an answer. For a classification, you pass the logits through an activation function (softmax) to get probabilities, then pick the best.
• For a text model, there is one logit for every token in the vocabulary. Softmax turns them into one probability per token, and the highest probability is the usual pick.

3. Demo: GPT-2 predicts the next token

GPT-2 is an open-source text generation model from OpenAI. It is decoder-only. It is a basic, early model, but it is perfect for seeing how it works. It comes from the same transformers library, with its own tokenizer.

from transformers import GPT2LMHeadModel, GPT2Tokenizer
import torch

tok = GPT2Tokenizer.from_pretrained("gpt2")
model = GPT2LMHeadModel.from_pretrained("gpt2")
model.eval()                       # these are PyTorch models

text = "Rose loves sour fresh fruits like strawberries, apples"
inputs = tok(text, return_tensors="pt")   # token IDs for the 9 tokens

The vocabulary of GPT-2 has 50,257 tokens. The next token can be any one of them, so the model returns one logit for each token, for every position in the input:

with torch.no_grad():
    logits = model(**inputs).logits
logits.shape        # (1, 9, 50257) = (sentences, tokens, vocabulary size)

We only care about the last position, because that is where the next token goes:

next_token_logits = logits[:, -1, :]
probs = torch.softmax(next_token_logits, dim=-1)       # 50,257 probabilities
top_k = torch.topk(probs, k=5)                          # the 5 most likely
for idx, score in zip(top_k.indices[0], top_k.values[0]):
    print(repr(tok.decode([idx])), round(float(score), 4))

Out of 50,257 possibilities the comma has the highest probability, so by default the model predicts a comma. The highest probability is not guaranteed to be right. It is just the most likely.

Whether you want the top 1, top 3 or top 5 depends on your use case. In real applications you usually just give the input and get the output, without these steps. You only write them yourself if you want to customise the behaviour.

4. Generating text: append and repeat

The model predicts only one token at a time. To write a sentence, it adds the predicted token to the input and predicts again:

1. Input: “The kids are watching this movie … and see what” gives the prediction happens.
2. Add it to the text, and predict again: .
3. Then They, want, to, see, what, and so on.

... and see what  ->  happens . They want to see what

Each step uses the whole text so far. Even though it makes one token at a time, the sentences make sense. The 50,257 probabilities are always spread over the same vocabulary, which the model learned in training. A different model has a different vocabulary size.

No learning happens here. The weights were fixed in training. At this stage the model only applies them and returns the highest-probability token. It does not know whether the answer is true.

GPT-2 is a small, early model. Its text drifts quickly. Newer models have much better text generation, which is why you move to models such as GPT-3, GPT-4 and GPT-5.

A text generation model can only generate text. To produce an image, you need an image generation model.

5. A bigger example with Flan-T5

The same idea works with models from other vendors. This example uses Google’s Flan-T5 to answer “Can you write a short beautiful love story?”

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

t5_tokenizer = AutoTokenizer.from_pretrained("google/flan-t5-base")
t5_model = AutoModelForSeq2SeqLM.from_pretrained("google/flan-t5-base")

inputs = t5_tokenizer("Can you write a short beautiful love story?", return_tensors="pt")
output = t5_model.generate(**inputs, max_new_tokens=100, no_repeat_ngram_size=3)
print(t5_tokenizer.decode(output[0], skip_special_tokens=True))

• max_new_tokens=100: the model repeats the loop (predict, append, predict) until it has made 100 tokens. Every model has limits on how much it can generate and read, and the next session covers this.
• no_repeat_ngram_size=3: stops the model from repeating the same three-word phrase.
• (Flan-T5 is an encoder-decoder model, not a decoder-only one, but it generates one token at a time in the same way.)

Run it again, and you get the same story every time. By default the model always picks the highest-probability token, which is called greedy decoding, so it is predictable. Always taking the top token is fine for facts, but not for a story generator, where you want variety.

6. Adding randomness: temperature and top-k

Temperature changes the probabilities

To get different output each time, you tell the model not to always take the top token, but to sample (choose randomly according to the probabilities):

output = t5_model.generate(
    **inputs,
    max_new_tokens=100,
    no_repeat_ngram_size=3,
    do_sample=True,        # pick randomly according to the probabilities
    temperature=0.7,
    top_k=50,
)

Top-k and top-p sampling

With sampling on, each run gives a different story. Use a low temperature for facts and a higher one for creativity. For example, a website that tells bedtime stories should add randomness. A tool that answers an operational question should not.

You do not need the equations. You need to know which controls exist and what they do, like the steering wheel and pedals of a car.

7. What is an LLM?

LLM means large language model. In this course:

• An LLM is a decoder-only transformer trained on a very large amount of text (terabytes) to generate the next token.
• By this definition GPT-2, GPT-3, GPT-4 and GPT-5 are all in the LLM family. GPT-2 is an early, small one.
• BERT is not: it is an encoder that produces embeddings but cannot write new text. It understands language, but does not generate it.

How to choose a model for your requirement:

As always, experiment: try two or three models and see which gives the best results. Read the vendor’s documentation for each model’s strengths.

8. How models evolved

[ IMAGE: s16-model-evolution.png ]
How models evolved from scikit-learn to reasoning LLMs

Up to 2019 the models were encoder-based. From 2019 the decoder-based models start the LLM family. The next session starts using LLMs from Python and covers the context window, with Llama (open source), OpenAI and Claude models.

Summary

Key takeaways

• A model with a decoder is a text generation model. It predicts the next token.
• A logit is the raw output. Softmax turns logits into probabilities, one per token in the vocabulary.
• The model writes by predicting one token, appending it and repeating. Nothing is learned at this stage.
• By default it takes the highest-probability token (greedy), so the output is the same every time.
• do_sample, temperature and top_k add controlled randomness. Low temperature for facts, higher for creativity.
• An LLM is a decoder-only text generator trained on a huge amount of text. GPT-2 is an early one, and BERT is not an LLM.
• Models evolved from statistical ML, to embeddings, to text generation, chat models and reasoning LLMs.

Quick revision

Remember: predict a token, append it, repeat.