Session 15 – Agentic AI DevOps: Transformer Architecture

Agentic AI DevOps: Transformer Architecture

Recap of the previous session

• Static embeddings give each word one fixed vector. Contextual embeddings change with the sentence.
• GloVe is a pre-trained static model. Cosine similarity measures how close two vectors are: 1 is the same, 0 is unrelated, negative is opposite.
• With Hugging Face, a tokenizer makes token IDs and a model looks up the vectors. You can inspect the vocabulary and the embedding table.
• Choose an embedding model by its training data, its size, and a test on your own data.

In this session

• Why static embeddings are not enough
• The “Attention Is All You Need” paper and the transformer
• Attention: how a model decides which words matter
• How transformer models are trained: causal and masked language modeling
• Demos: static vs. contextual vs. sentence embeddings
• A DevOps use case: matching an alert to the right runbook
• What happens inside a transformer

In one sentence: a transformer reads a whole sentence, weighs how much each word matters to every other word (attention), and produces embeddings that capture the meaning in context.

Why learn this? You will not build these models yourself, and you will not use tokenization and embedding as separate steps in the agent projects. But the ideas are needed if you ever customise an embedding, and they explain how LLMs work.

1. The goal, and the limit of static embeddings

The goal: let a machine understand human language. The text is split into tokens, each token becomes an embedding, and the machine applies maths to the numbers.

A static embedding model knows the meaning of each token, but not of the sentence. The same word in two sentences gets the same vector. To understand sentences we need contextual embedding models, and these are built on the transformer architecture.

2. The transformer

[ IMAGE: s15-transformer-architecture.png ]
The transformer architecture: encoder and decoder

In 2017, a Google paper called “Attention Is All You Need” introduced the transformer. It is the base of modern LLMs. The architecture has two blocks:

• The encoder turns the input text into embeddings that capture meaning.
• The decoder generates text from them.

Later, people built models using different parts:

Contextual embedding models use only the encoder. Their only job is to produce embeddings. The decoder comes in the next session.

3. Attention: which words matter?

[ IMAGE: s15-attention-fill-blanks.png ]
Attention: filling in the blanks

Try filling the blank: “Rose loves sweet fresh fruits like ______.” You do not guess. You look at the surrounding words and give some more weight than others. Sweet, fresh and fruits matter most, and like matters less. So the answer is a sweet fresh fruit, such as mango.

That is attention: for every word, the model works out how much each other word matters. The model does not pick one answer directly. It scores many candidate tokens (mango 0.62, orange 0.20, kiwi 0.12, lemon 0.06), and a softmax turns the scores into probabilities. The highest probability wins.

4. How a transformer is trained

Training works as before: start with random weights, predict, compare with the truth, calculate the loss, and adjust. But there is a difference. A text has no separate x and y columns. The text supplies its own answers, so no human labelling is needed. This is often called self-supervised learning. There are two main approaches.

Causal vs. masked language modeling

The next word is hidden until the model has predicted itGPT modelsMasked language modeling (MLM)Hide a random word in the sentence and predict it from the words on both sidesBERT models

The model compares each prediction with the real next word (or the hidden word), calculates the loss, and adjusts its weights. This is done over a huge amount of text.

• Training is general, not for one business. The model learns the language, and anyone can then use it.
• Training on terabytes of text takes a very long time and a lot of compute (months for large models). That is why you use pre-trained models. The model keeps millions or billions of learned parameters. The MiniLM model from last session has 22 million.

5. Demo: static vs. contextual

Static (GloVe)

glove.most_similar("bank", topn=15)
# banks, securities, banking, investment, exchange, financial, credit, ...

All the nearest words are about money. So the model knows what a token means, but only one meaning of bank.

The river sense of bank scores low: the model has no idea that bank can also mean a river bank.

Static models also lose the word order: “dog bites man” and “man bites dog” mean different things, but their cosine similarity is 1.0, because they contain the same tokens.

Contextual (DistilBERT): compare the vector of the word bank taken from three sentences:

1. “The fisherman sat on the bank of the river.”
2. “We walked along the muddy riverbank.”
3. “The bank approved my home loan.”

The two river sentences are closest. The same word bank now has a different vector depending on the sentence. That is what contextual embeddings do. (These are scores, not yes or no answers. You choose your own threshold.)

Sentence transformers. The sentence-transformers library does the tokenizing, the embedding and the combining (pooling) for you, so you get one vector per sentence:

from sentence_transformers import SentenceTransformer

st_model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
vectors = st_model.encode(["pods keep restarting after the release",
                           "container crash loop following the deployment",
                           "disk is almost full on the database server",
                           "my favourite pizza has extra cheese"], normalize_embeddings=True)

It understands that pods restarting and crash loop are related without matching words. The numbers are calculated from the embeddings, not from the words.

6. A DevOps use case: find the right runbook

You receive an alert: “containers restart over and over after the new image was deployed”. You have runbooks with titles such as “Fix CrashLoopBackOff: pod keeps restarting” and “Disk full: clean up logs and expand the volume”. How does a model know which one applies?

1. Turn every runbook title into an embedding.
2. Turn the alert into an embedding.
3. Compute the similarity between the alert and each runbook.

scores = runbook_vectors @ st_model.encode(alert, normalize_embeddings=True)

The top score points to the most likely runbook. It is not guaranteed to hold the answer, but it is the best place to look. This idea, finding content by meaning, is the heart of the RAG architecture that follows.

7. What happens inside a transformer

From tokens to contextual embeddings

When text goes in:

1. Tokenization: the text becomes tokens and token IDs.
2. Static embedding: each token ID is looked up in the embedding table.
3. Positional embedding: a vector for each token’s position (1st, 2nd, 3rd…) is added to its static embedding. This lets the model know the word order and process all the tokens in parallel.
4. Multi-head self-attention: the model uses learned weights Q, K and V (query, key and value) to decide how much each token should attend to every other token. Several attention heads do this in parallel, and their results are combined.
5. Feed-forward layers refine each token’s vector.
6. The output is a contextual embedding for every token.

Q, K and V are learned weights, like m and b, but they are matrices with many numbers instead of one value. They are fixed after training and are applied the same way each time.

You do not need to learn the equations. You import a pre-trained model and use it.

Summary

Key takeaways

• Static embeddings capture the meaning of a token. Contextual embeddings capture the meaning in a sentence.
• The transformer (2017) has an encoder and a decoder. Contextual embeddings come from the encoder.
• Attention weighs how much each word matters to the others. A softmax turns the scores into probabilities.
• Transformers are trained on text itself: causal (predict the next word, GPT) or masked (predict a hidden word, BERT). It needs huge compute, so you use pre-trained models.
• A sentence transformer gives one vector per sentence, and cosine similarity shows which sentences are close in meaning.
• This lets you match an alert to a runbook by meaning.

Quick revision

Remember: attention lets a model use context, and context is what gives words their meaning.