Session 19 – Agentic AI DevOps: RAG Architecture

Agentic AI DevOps: RAG Architecture

Recap of the previous session

• Zero-shot, few-shot, chain of thought and constraints and controls are four starting techniques. How you ask decides what you get.
• Prompting is an experiment. Try different prompts, models and settings, and validate the answers.
• Two small helpers, ask() and answer(), send a prompt and pull out the text.
• A larger model (for example 120 billion parameters on Groq) is generally more reliable than a small local model.

In this session

• Constraints in practice: rules in the prompt and a token limit
• The system, user and assistant roles
• Why an LLM has no memory, and how a chatbot gives it one
• The context window, and why long conversations cost more
• The orchestration layer
• Why an LLM cannot answer from your private data
• The RAG architecture: ingestion, query, vector DB, orchestrator and LLM
• Vector database options
• Generative AI vs. statistical AI

In one sentence: an LLM only knows what it learned in training plus what you put in the prompt, so to answer from your own documents you store them in a vector database, search it for the pieces that match the question, and send only those pieces to the LLM.

1. The set-up for the exercises

This is the same Groq client and model as before in a new notebook. One helper changes slightly: ask() now takes a list of messages, because this session is about the roles inside that list.

import getpass
from openai import OpenAI

client = OpenAI(api_key=getpass.getpass("Groq API key: "), base_url="https://api.groq.com/openai/v1")
MODEL = "openai/gpt-oss-120b"


def ask(messages, **settings):
    """Send the messages to the chosen model and return the full response."""
    if "gpt-oss" in MODEL and "max_tokens" in settings:
        settings["max_tokens"] += 1000     # a reasoning model thinks first, and the thinking uses tokens too
    return client.chat.completions.create(model=MODEL, messages=messages, **settings)


def answer(response):
    """Just the text of the answer."""
    return response.choices[0].message.content

• ask() sends the messages and returns the whole response. answer() returns only the text.
• gpt-oss-120b is a reasoning model: it thinks before it answers, and the thinking also uses tokens. The extra 1000 stops a small max_tokens from being used up before the answer starts.

2. Constraints in practice

Here is a small incident, kept in a variable:

incident = ("At 02:10 UTC the checkout service started returning 502 errors right after the v2.3 deploy. "
            "About 30% of requests are failing.")

If you only say “Summarise this incident”, the LLM answers with its own layout: headings such as time of occurrence, affected service and trigger, all taken from the details you put in the prompt. The content is right, but the format is not the one you need.

A constraint is a rule you add to the prompt to change the shape of the answer:

prompt = f"""Summarise this incident.

Rules:
- Exactly 3 bullet points
- Maximum 12 words per bullet
- Bullet 1: what happened. Bullet 2: the impact. Bullet 3: the cause
- Use only facts from the text. If the text does not say, write "unknown"

Incident: {incident}"""

messages = [{"role": "user", "content": prompt}]

print(answer(ask(messages, temperature=0, max_tokens=50)))

- Checkout service began returning 502 errors.
- Approximately 30% of requests failed.
- Cause unknown.

• The answer follows the rules: three bullets, each short, in the order you asked for.
• The cause is unknown because the incident text does not give one. The rule “use only facts from the text” stopped the model from guessing. This is the behaviour you want from an agent that works with real incidents.
• temperature and max_tokens are settings passed with the call. max_tokens is a ceiling on how much the model may generate, not a target.

Choose the constraints that fit what you expect from the LLM. They change how it answers.

3. Roles: telling the LLM who it is

Ask a very simple question: “A pod keeps restarting. What should I do?” The right answer depends on who is asking, and who is answering. An LLM must be told which role to play. A senior engineer and a mentor for beginners answer the same question differently.

You set the LLM’s role with a message whose role is system:

user_prompt = "A pod keeps restarting. What should I do?"

system_prompt = "You are a friendly mentor for people new to Kubernetes. Explain in simple words, in at most 4 sentences."

messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": user_prompt},
]

print(answer(ask(messages, temperature=0, max_tokens=50)))

Check the pod's logs (`kubectl logs <pod>`) to see why it's crashing, then look at the pod's events (`kubectl describe pod <pod>`)
for clues such as failed health checks or OOM kills. If a liveness/readiness probe is failing, adjust its settings or fix the underlying
issue. Make sure the container has enough CPU/memory limits and that required files or environment variables are present. After you
correct the problem, delete the pod so the controller can create a fresh, healthy one.

With the system prompt “You are a senior SRE. Answer in at most 3 short bullet points, each with a kubectl command.” the same question gave a short reply in the session: a kubectl command with a one-line explanation. The question did not change, only the LLM’s role did, so the answers differ, as they would between a senior engineer and a teacher.

Role	Who it is	What it carries
system	You, the developer, instructing the LLM	The LLM’s own role and its rules: how it should behave in every answer
user	The person using the application	The question or task
assistant	The LLM	A reply the LLM gave earlier

Whose role is it? The system prompt is the role of the LLM, not yours. You are asking a question of a senior SRE, or of a mentor.

You do not always need all three. The assistant role is needed only when you build a conversational system, such as a chatbot. The next section shows why.

4. An LLM has no memory

In ChatGPT you ask a question, read the answer, and ask a follow-up. The follow-up works because the application knows what came before. The LLM does not. It takes the prompt, predicts the next tokens, returns the answer and forgets everything. It has no built-in memory of your earlier messages.

So if you send only “It says OOMKilled. What next?”, the LLM has no idea what “it” means. To make it a conversation, your code must send the earlier conversation again with every new question. The model’s previous reply goes back in as an assistant message:

first_answer = answer(ask(messages, temperature=0, max_tokens=50))     # the reply from section 3

messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": "A pod keeps restarting. What should I do?"},
    {"role": "assistant", "content": first_answer},                       # what the LLM said before
    {"role": "user", "content": "It says OOMKilled. What next?"},
]

print(answer(ask(messages, temperature=0, max_tokens=50)))

`OOMKilled` means the container ran out of memory and the kernel stopped it. Increase the memory request/limit for the pod
(e.g., add `resources: { requests: { memory: "256Mi" }, limits: { memory: "512Mi" } }` in the pod spec) or reduce the memory the
application uses. ...

Now “it” has a meaning. The LLM sees that the user asked about a restarting pod, that it answered, and that the user now reports OOMKilled, so it answers the follow-up and does not repeat everything.

For the next follow-up, the list grows again:

#	Role	Content
1	system	You are a friendly mentor for people new to Kubernetes…
2	user	A pod keeps restarting. What should I do?
3	assistant	(the first answer)
4	user	It says OOMKilled. What next?
5	assistant	(the second answer)
6	user	In your previous answer you said “limit”. Can you explain what a limit is?

The last question works because the previous answer is inside the messages. The assistant messages are not a memory in the LLM. They are text that your application stores and sends back each time.

5. The context window and the cost of a conversation

Every new turn sends the whole conversation so far, plus the new question. So:

• The input tokens grow with every turn. (Learner question: “so it uses the previous tokens again?” Yes. You can see the input token count rise in the response’s usage.)
• LLMs bill by tokens, so a long conversation costs more and puts more load on the LLM.
• Every LLM has a context window: a limit on how many tokens it can take in one request. As the conversation grows, it approaches that limit.

When the conversation reaches the limit, your application has to drop the oldest messages and keep the latest ones, so the LLM still knows the recent conversation and forgets the very old part. For example, with a limit of 5,000 tokens you discard the first question and its answer, and keep the newest messages.

There are better ways to manage memory, but this is the core idea behind every chatbot: the context travels with the prompt.

6. The orchestration layer

If the LLM stores nothing, who does? Your application. When you build a ChatGPT-like system, your back end keeps each user message and each reply in a store. When the user asks a follow-up, it:

1. fetches the earlier conversation,
2. adds your new question,
3. builds the final prompt,
4. sends it to the LLM,
5. stores the reply, and shows it to you.

The code that does this between the user and the LLM is called the orchestration layer, or the orchestrator. You build it yourself. It is the difference between an LLM and an application, which was the idea in the previous session.

7. The problem: an LLM does not know your private data

So far the questions were generic: what is Kubernetes, what to do about a restarting pod. An LLM answers those from what it learned in training, because the internet is full of such text.

Now think of an operations team. When an incident arrives, the L1 team looks into it, finds the first cause and tries to fix it. They rely on runbooks: the URLs of dev, staging and production, logins, links, procedures. In a real company this lives in Confluence pages and PDFs, hundreds or thousands of pages, written by many teams for many applications.

The idea of an agent is that the engineer does not dig through a folder of PDFs. They ask: “Give me the production URL”, or “Which runbook applies to this incident?”, and the agent finds it. Try the first question:

system_prompt = "You are the platform assistant agent to help the operations teams"
question = "What is the production Argo CD URL?"

messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": question},
]

print(answer(ask(messages, temperature=0, max_tokens=50)))

In the session the LLM replied that it does not have the production Argo CD URL, and then added some general advice. That is expected. It never saw your organisation’s URLs during training. An LLM cannot help with your private data, such as runbooks and Confluence pages, unless you give that data to it.

8. Giving the data in the prompt

So give it the data. Here is a short sample runbook with the Argo CD and Grafana URLs, the on-call details and the rollback steps:

runbook = """
Platform Runbook: Tools and Contacts

Argo CD
- Production: https://argocd.prod.shopfast.example.com
- Staging: https://argocd.stage.shopfast.example.com
- Login with company SSO. You must be on the VPN.

Grafana
- Production dashboards: https://grafana.prod.shopfast.example.com
- Start with the dashboard named "Platform Overview".

On-call
- The primary on-call changes every week. See the PagerDuty schedule "platform-primary".
- If the primary does not answer in 15 minutes, escalate to the SRE lead.

Rollback
- Open the application in Argo CD, choose History and rollback, and pick the last good version.
"""

Put the question and the runbook into one prompt:

prompt = f"""{question}

runbook: {runbook}"""

messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": prompt},
]

print(answer(ask(messages, temperature=0, max_tokens=50)))

The production Argo CD URL is:

**https://argocd.prod.shopfast.example.com**

(You'll need to be on the VPN and log in via the company SSO.)

The LLM answers with your URL, because the answer was inside the prompt. It read the runbook and found the line.

Why this does not scale

• A real runbook is 100 or 200 pages, and each project has several documents. You cannot paste all of it into the prompt.
• Even if you could, the input tokens would rise, and you would pay for text that has nothing to do with the question.
• Only one page or two of those hundred pages hold the answer.

We need a way to find those pages first. That is what a vector database does, and the architecture that uses it is called RAG.

9. The RAG architecture

RAG stands for retrieval-augmented generation. You retrieve the relevant data, augment the prompt with it, and let the LLM generate the answer.

Step 1: Ingestion (done in advance)

1. Collect your private data: PDFs, Confluence pages and runbooks, stored in S3, a database or a drive. It does not matter where.
2. Pass it through an embedding model, which converts the text into embeddings (vectors of numbers that carry meaning, from the earlier sessions).
3. Store the embeddings in a vector database.

A vector database stores data in the form of embeddings, the way MySQL stores data in tables. Its main job is to store embeddings and to find the ones closest to a given embedding. After this step, the whole runbook sits in the vector DB in embedding form.

Step 2: Query (every time a user asks)

1. The user sends a question, for example “What is the Argo CD URL?”. It does not go straight to the LLM. It goes to the orchestrator.
2. The orchestrator converts the question into an embedding (the query vector) with the same embedding model, and sends it to the vector DB.
3. The vector DB does a semantic search. It compares the query vector with the stored embeddings, using cosine similarity (from the embeddings sessions), and finds the ones that are nearest.
4. It returns the top-k results. If you ask for the top 2, you get the 2 best matches.
5. The orchestrator builds the final prompt: the user’s question plus the top-k results.
6. It sends the prompt to the LLM. The LLM now knows what the user asked and also has the relevant data from your organisation.
7. The LLM writes the answer, and the orchestrator returns it to the user.

Why this is better

You do not send your whole private data to the LLM. All of it stays in the vector DB. Only the pieces that match the question are retrieved, and only those go into the prompt. So:

• the input tokens stay small, and the cost stays low,
• the LLM has less to process, so there is less load and less delay,
• the context window is not filled with irrelevant pages.

The components

1. Embedding Model – Turns documents and questions into vectors
2. Vector DB – Stores the vectors and returns the nearest ones for a query
3. Orchestrator – Takes the question, queries the vector DB, builds the final prompt, calls the LLM, and returns the answer
4. LLM – Reads the question and retrieved data, and writes the answer

10. Questions from this session

Why do we need the LLM at all? Can the orchestrator not just send what the vector DB found?Suppose two runbooks match, and each has 50 lines. You do not want to hand both to the user. The user asked one specific thing and expects a short, specific answer. The LLM reads the retrieved text, picks the information that answers the question, and writes it the way you want. Your system prompt sets the rules: bullets, a paragraph, bold text, a certain tone. The LLM follows them. Dumping the raw vector DB result on the user would not look good.

Do I need to write code for the embedding model and the orchestrator, or is it built in?

You build it. There is no single “RAG” product. You pick an embedding model, parse your data, create the embeddings, store them, and write the orchestrator step by step.

Is RAG a PaaS or a SaaS?

No. RAG is only a name for this architecture, in the same way that “DevOps” is a name for a way of automating many tools into a build and deployment process. Any system that can do a semantic search to retrieve the right chunk of data for a user’s question, and give it to an LLM, is a RAG system. The vector database is one component of it.

11. Which vector database?

Many vector databases exist, and the purpose is the same: store embeddings and return the nearest ones for a query.

Choose by your requirements: data size, query volume, performance, security and budget. It is like choosing between MySQL (open source) and Oracle (enterprise) for a project. Once you know one vector database, the others are easy to pick up. Chroma is a good fit when the amount of data is small, and larger systems may look at others.

12. RAG is generative AI

An application that generates new content is a generative AI application. In RAG the LLM generates new text for the user’s question, so RAG belongs under generative AI.

	Statistical AI	Generative AI
Examples	Regression and classification models from scikit-learn, trained on your own data	Applications built on LLMs
What it does	Learns patterns from your data (weights such as m and b) and predicts or decides	Creates new content: text, images, audio or code
How	A model such as a regression or decision tree	A transformer decoder that predicts the next token

Agentic AI adds the ability to plan, decide and execute on its own. It comes after generative AI in this course.

What comes next

All the pieces are already known: embeddings, the cosine similarity between two vectors (the core of a vector database), and sending a question together with context to an LLM. The next sessions put them together in one project: a RAG system over runbooks, with a chatbot in front of it. In real use, the operations team will not type prompts into a notebook. They will ask the agent in a chat window.

Summary

Key takeaways

• Add constraints to the prompt, such as the number of bullets, the length and “use only facts from the text”, to control the shape of the answer. temperature and max_tokens are settings passed with the call.
• The system message gives the LLM its role and rules, the user message is the question, and the assistant message is an earlier LLM reply.
• An LLM has no memory. A chatbot works because your code resends the conversation, adding each reply as an assistant message.
• A longer conversation means more input tokens, a higher bill, and a risk of reaching the context window limit. Drop the oldest messages when the limit is near.
• The code that stores the history and builds the final prompt is the orchestration layer.
• An LLM cannot answer about your private data unless you put it in the prompt. You cannot paste hundreds of pages every time.
• RAG stores your data as embeddings in a vector DB. For each question the orchestrator retrieves the top-k nearest pieces by semantic search, adds them to the prompt, and the LLM writes the answer.
• The LLM is still needed to turn the retrieved text into a short, well-formed answer.
• RAG is a name for an architecture, not a product. A vector database is one component.
• RAG produces new text, so it is generative AI.

Quick revision

Term	Meaning
Constraint	A rule in the prompt that shapes the answer
System prompt	The message that sets the LLM’s role and rules
User / assistant role	The person’s message / an earlier LLM reply
Context window	The most tokens an LLM can take in one request
Orchestrator	The code between the user and the LLM that builds the final prompt
Private data	Your runbooks, wiki pages and documents, which the LLM never saw in training
RAG	Retrieval-augmented generation: retrieve relevant data, add it to the prompt, generate the answer
Ingestion	Converting documents to embeddings and storing them in the vector DB
Vector DB	A database that stores embeddings and finds the nearest ones
Semantic search	Searching by meaning, with the cosine similarity of vectors
Top-k	The k best matches returned by the vector DB
Chroma, Pinecone	An open-source and a managed vector DB

Remember: the LLM does not remember and does not know your data. Your application remembers the conversation and fetches the right data, and the LLM turns both into an answer.