Session 21 – Agentic AI DevOps: RAG Project – Part 2

Agentic AI DevOps: RAG Project – Part 2 Notes

Recap of the previous session

• A vector database stores your documents as embeddings. For a question, it returns the closest pieces (chunks) by distance, and the LLM writes the final answer from them.
• Long documents are cut into chunks. Each chunk is stored with an id, the text, an embedding and metadata.
• The orchestrator is the code that takes the question, queries the vector DB, builds the prompt, calls the LLM and returns the answer.
• All of that was done in a notebook, as an experiment.

In this session

• Why a RAG project is three separate parts, not one program
• The ingestion pipeline: what it does, who owns it, how it runs in real life, and its best practices
• The orchestrator: a service with an API, and its best practices
• The chatbot UI: who builds it and how it connects
• Which team handles what, and what each team hands to the next
• Using the same RAG service from an agent
• What goes wrong in real life, and how to explain the project in an interview

In one sentence: a real RAG project is three parts owned by three groups, an ingestion pipeline that keeps the vector DB full and current, an orchestrator API that answers questions from it, and a chatbot UI that only calls that API.

How to read these notes. Sections 1 to 6 describe what was discussed in the session. The parts marked Best practice go a little further and describe how teams usually run this in production. Use them as a checklist, and remember that every company builds its own version.

1. The big picture: three parts, three owners

In the notebook, one person did everything: loaded the documents, searched, called the LLM and read the answer. In a real company the same work is split into three parts. They run separately, they are built by different people, and they talk to each other through clear agreements.

This is the same picture as the RAG architecture you already know: step 1 (ingestion) and step 2 (query). What changes is that each box is now a separate program with its own owner, its own deployment and its own monitoring.

What really happens, in two flows

Flow A, continuous (the ingestion pipeline):

1. People write or update documents.
2. The pipeline picks up the new and changed documents.
3. It cleans them, cuts them into chunks and adds metadata.
4. It stores them in the vector DB as embeddings.

Flow B, for every question:

1. The user types a question in the chatbot.
2. The chatbot calls the orchestrator API.
3. The orchestrator asks the vector DB for the closest chunks.
4. The orchestrator builds the prompt and calls the LLM.
5. The orchestrator returns the answer and its sources, and the chatbot shows them.

The two flows only meet in one place, the vector DB. That is why the agreement between the ingestion team and the orchestrator team matters so much (section 2).

2. The agreement between the parts

Each part is independent, so the teams must agree on a few things. If one of them is wrong, the system fails even though every part is fine on its own.

In the demo, the orchestrator did not create the collection. The ingestion pipeline creates it, because it is the part that stores data. The orchestrator only opens the collection that is already there, in the same place and with the same name. That is a good picture of the rule: one part writes, the other part reads.

3. The ingestion pipeline

What it does

The pipeline reads the documents of the project, such as the runbooks and wiki pages, and loads them into the vector DB. It is the same work you did in the notebook, now as a program that anyone can run:

1. Collect the documents from the source.
2. Prepare them: build a title, remove what must not be stored.
3. Chunk them with the chosen strategy.
4. Describe each chunk with an id and metadata.
5. Store the chunks. The vector DB creates the embeddings.

In the demo, the wiki folder had four pages. They gave 18, 10, 27 and 43 chunks, so the vector DB held 98 items. Nobody counts the chunks by hand any more. The pipeline works through every file in the folder, however many there are, and the number of chunks is simply the result.

Who owns it

A separate team takes care of the ingestion pipeline. In general it is a DevOps team or a data engineering team, depending on the company and the requirement. They choose the chunking strategy, experiment with it, clean the data and keep loading new documents. The team that builds the chatbot does not touch it.

Where the documents come from

The source is not always a folder. It can be:

• a shared drive, such as OneDrive,
• Confluence, through its API,
• an S3 bucket,
• any other store where the company keeps its documents.

If the runbooks live in Confluence, you only tell the ingestion pipeline to connect to the Confluence API and fetch the documents. The rest of the pipeline stays the same. Whichever source it is, the process is the company’s choice.

Keeping the data current

Ingestion is never finished. Every day people write new pages and update old ones. If nothing loads them, the chatbot and the agents answer from old documents.

So the company defines a process, for example:

• Engineers upload new process documents to a shared place.
• The ingestion pipeline runs on a schedule, for example every day after office hours, picks up the new files and loads them.
• Or it runs when something changes, or someone starts it by hand.

Which one is used depends on the company. The rule stays the same: some automated process must keep loading the latest content, so that users and agents get accurate, current answers. It should not be an engineer’s daily manual task.

A typical real-life layout

Best practice: ingestion pipeline

1. Automate it and run it on a schedule. Treat it like any other job on the platform, with a log of every run and an alert when it fails. A pipeline that nobody notices has failed becomes a chatbot that quietly gives old answers.
2. Load only what changed. Re-reading everything works for four files. With thousands of pages, process the new and changed documents, and remove the chunks of documents that were deleted.
3. Make a re-run safe. Give every chunk a stable id, built from the document and the position, so that running the pipeline again updates the chunk and does not create a duplicate or fail because the collection already exists.
4. Clean the data before it goes in. The vector DB does not remove sensitive data. It stores what you send. Remove passwords, keys, personal data and anything confidential before ingestion, in the same way you clean data before training a model.
5. Care about the quality of the documents. A document with a clear structure is easy to chunk. A badly structured document gives wrong chunks, poor distance scores and answers full of text nobody asked for. Ask the people who write documents to keep a clear title and headings.
6. Choose the chunking strategy by experiment (fixed size, recursive, semantic, structure based) and keep the settings in version control. Put the document title in every chunk, so a chunk always says where it comes from.
7. Store useful metadata: the source file, the title, when it was last updated, and for example the environment or the team. Metadata is used for filters, and it is what lets the answer name its source.
8. Fix the embedding model and record it. Changing it later means loading everything again.
9. Verify every run. Count the items, and ask a few known questions to see that the right chunk comes back. Fail the job if the check fails.
10. Keep the source as the truth. The vector DB is built from the documents, so you can always rebuild it. Treat it as data that can be produced again, and take backups of it as well if rebuilding is slow.
11. Give least access. Only the pipeline can write. The orchestrator can only read.

4. The orchestrator

What it is

The orchestrator is the service that connects the user, the vector DB and the LLM. It is just an API: a program that keeps running, listens on a port, and answers requests from its clients. There is no special component called an orchestrator, and there is no standard name. One company calls it an orchestrator, another an “ask service”. The name that most people use is orchestrator, because it does a lot of wiring in the background.

In the notebook, you read the answer on your own screen. In real life the answer has to be returned to whoever called the API, so the output goes back as a response, not onto a screen.

What it offers: two endpoints

The contract of ask. The request holds one thing, the question, and the API checks that it is really there. The response holds the answer and the sources: which chunks the answer came from, with the distance score of each one. The sources are extra information. They show where the answer was found and how close each chunk was.

What happens inside ask

1. Receive the question and check that it is valid.
2. Retrieve: ask the vector DB for the closest chunks (the top 4 in the demo).
3. Build the prompt: the system prompt (the LLM’s role and rules) and the user prompt (the retrieved text as context, plus the question).
4. Call the LLM with the final prompt.
5. Return the answer and the sources.

It is the same flow you saw in the notebook. Nothing new is added to RAG.

What it needs to start

Who owns it, and what it means for you

The orchestrator is built by the agentic AI developers. If your role is agentic AI developer, this is your part. Your job is to make sure the orchestrator API is developed, deployed and running fine, and that it accepts questions from its clients and gives back responses. You do not have to build the UI.

A framework such as FastAPI gives the service a documentation page where you can send a test request from the browser. Testing the API from that page or from Postman is good enough.

Best practice: orchestrator

1. Keep it stateless. The service should not keep anything between requests. Then you can run several copies behind a load balancer and restart any of them. (An LLM has no memory. If the bot must remember earlier questions, the history has to be sent with each request.)
2. Run it like any other service. Package it as a container, keep the settings in environment variables, deploy it through a pipeline, and use the health endpoint for the platform’s liveness and readiness checks.
3. Keep secrets out of the code and out of Git. The LLM key belongs in a secrets manager or the platform’s secrets. Only the orchestrator needs it.
4. Return the sources. A user who sees where the answer came from can trust it, and a developer can debug a wrong answer.
5. Write the rules in the system prompt. Answer only from the given pages, say “not found” when the answer is not there, never invent commands or numbers, and mention approvals. The system prompt is also where you add guardrails, for example how to handle a sensitive value typed by a user.
6. Fail clearly. If the LLM or the vector DB is down, return a clear error and do not crash.
7. Protect the API. It uses your LLM key, and it can reveal your internal documents. Put it behind authentication, an API gateway and rate limits. It should not be open to everyone.
8. Watch it. Log the request, the chunks used, the distances, the time taken and the token count. This tells you about slow answers, cost, and questions the documents cannot answer. Handle what you log with care, because questions can hold private data.
9. Test the quality. Keep a set of real questions with the expected answers, and run them after every change to the documents, the chunking or the prompt.
10. In production, the vector DB, the LLM and the orchestrator usually run on separate servers and talk through URLs. For example, ChromaDB on one server and a local LLM such as Ollama on another. In the demo, the program simply reads a database folder next to it.

5. The chatbot UI

The chatbot is the front door for the user: a chat window where the user types a question and reads the answer.

• It is built by the front-end developers, with a front-end framework such as React or Angular.
• In a real application it is only one small part of a large website, often a widget in the corner, next to many other features. The front-end team builds it together with everything else.
• The front-end captures the question the user typed, calls the orchestrator API and shows the answer that comes back.
• It never talks to the vector DB or to the LLM, and it never holds the LLM key. Everything goes through the orchestrator.

The simple chatbot you saw in the previous session was shown only to explain the concept: a chat box, and a service behind it. You are not expected to build the UI in your job.

Best practice: chatbot UI

1. Agree on the API contract early, and keep it stable. A documentation page for the API helps the front-end team a lot.
2. Show the sources with the answer, so users can open the original page.
3. Handle the bad moments: show a message when the service is slow or down, and show “not found” clearly and do not leave the user with a blank.
4. Authenticate the user, and send only what is needed. Do not put secrets in the browser.
5. Collect feedback (for example a thumbs up or down). It tells the ingestion and orchestrator teams where the answers are weak.

6. RAG is not only for chatbots: using it from an agent

A chatbot is for humans. A person asks a question and gets an answer, instead of searching the runbooks by hand. The same orchestrator API can serve an agent.

Example: an auto incident resolver.

1. An agent receives an incident.
2. It calls the orchestrator API: “this is my incident, give me the relevant troubleshooting steps.”
3. The API returns the relevant chunks from the company’s own runbooks.
4. The agent decides which steps to run, logs in to Kubernetes, runs the commands and checks whether the issue is fixed.

Why use RAG here, and not only the LLM? An LLM that has not seen your runbooks gives generic troubleshooting steps from its training. They may not work in your environment. Your environment has its own Argo CD address, its own Kubernetes setup and its own internal details. Only your runbooks hold them. RAG gives the agent the project-specific knowledge, and then the agent can act on it.

A learner asked whether the runbooks can be linked to alerts. This is the way: an alert or an incident becomes the question, and the runbook steps come back from the API. The agentic part of the course will build such an agent.

Remember that RAG only reads. The steps the agent runs are done by tools, not by RAG.

7. Explaining the project in an interview

• Interviewers check whether you understood RAG: what it is, how it works, what problem it solves and where it can be used. They want the fundamentals: embeddings, the distance score, and how chunks are stored and searched.
• The process around it is different in every company: where documents come from, how often ingestion runs, who owns each piece. Nobody expects one fixed process. Be ready to describe your own implementation.
• A short way to tell it: “documents are cleaned, chunked and loaded into a vector DB by a scheduled ingestion pipeline. An orchestrator API takes a question, retrieves the closest chunks, calls the LLM with a strict system prompt and returns the answer with its sources. The chatbot, built by the front-end team, only calls that API. The same API serves our incident agent.”
• As a DevOps, SRE or infrastructure engineer, you do not compete with software developers on how deep you go in coding. Agents can write much of the code. What matters is that you understand the concept and the business use case, and how you approached the problem.
• It does not mean you skip the code. You should be able to read the code an agent gives you and explain what it does.

8. Questions from the session

Can the runbooks be read from Confluence instead of a folder?

Yes. The ingestion pipeline connects to the Confluence API and fetches the documents. When an engineer creates a new page, the pipeline can pick it up and process it, so nobody loads documents by hand.

New documents are created every day. Who loads them?

No engineer should do it by hand every day. Define a process, such as uploading documents to a shared drive, and a scheduled ingestion pipeline that picks up the new files and loads them into the vector DB. Scheduled, triggered or manual depends on the company.

What use cases can we build from this? Can runbooks be linked to alerts?

A chatbot is one use. Another is an agent that gets an incident or an alert, asks the orchestrator API for the relevant steps from your runbooks, and then acts on them (section 8).

Is a vector DB other than ChromaDB the same?

The idea is the same. Every vector DB stores the data as embeddings and finds the closest ones for a query. They differ in performance, accuracy of the semantic results, latency, how much data they hold, and how they handle traffic. The fields can differ a little because each has its own schema. An id, a document, an embedding and metadata are in all of them, and some add extra fields for tuning.

Does the service have to be called “orchestrator”?

No. There is no standard name. It could be an ask service. “Orchestrator” is the most popular term.

Do I need to be strong in coding?

You need to understand the code, not compete with developers. Agents can write the code, so understand what it does and why it is built that way.

What comes next

The RAG part of the course is complete: embeddings, the vector DB, the ingestion pipeline, the orchestrator and the chatbot in front of it. The next lectures move on to the agentic AI concepts. A simple sample chatbot may be shown later for testing, but it is not needed. Testing the orchestrator API from Postman or its documentation page is good enough.

Summary

Key takeaways

• A real RAG project is three parts: the ingestion pipeline, the orchestrator and the chatbot UI. They run separately and are owned by different teams.
• The ingestion pipeline (DevOps or data engineering) collects, cleans, chunks and stores the documents. It is continuous: an automated, scheduled process keeps the vector DB current from a drive, Confluence, S3 or any other source.
• Ingestion best practices: automate and monitor it, load only what changed, make a re-run safe, clean sensitive data first, keep the documents well structured, store useful metadata, fix the embedding model, verify every run, and give only the pipeline write access.
• The orchestrator (agentic AI developers) is an API service. It has a health endpoint and an ask endpoint. Ask takes a question, retrieves the top chunks, builds the system and user prompts, calls the LLM and returns the answer with its sources.
• Orchestrator best practices: stateless, deployed like any service, secrets outside the code, strict system prompt, clear errors, protected access, logs and quality tests.
• The orchestrator opens the collection that the pipeline created. They agree on the vector DB location, the collection name and the embedding model.
• The chatbot UI (front-end developers) is one small part of a bigger website. It captures the question, calls the orchestrator API and shows the answer. It never talks to the vector DB or the LLM.
• Your job as an agentic AI developer is that the orchestrator API is deployed, running and returning good answers.
• When something is wrong, find the part: old answers point to ingestion, odd answers to chunking and the prompt, errors to the orchestrator.
• The same API serves agents as well as chatbots. An incident-resolving agent gets steps specific to your organization instead of generic ones.
• There is no standard name for the orchestrator. Interviewers want the RAG fundamentals, and the process differs in each company.

Quick revision

Remember: ingestion keeps the vector DB full and current, the orchestrator answers from it, and the chatbot and the agents are only clients of the orchestrator API.