Session 17 – Agentic AI DevOps: Calling LLMs from Python

Agentic AI DevOps: Calling LLMs from Python

Recap of the previous session

• A model with a decoder is a text generation model. It predicts one token at a time, appends it and repeats.
• A logit is the raw output, softmax turns logits into probabilities, and by default the model takes the most likely token (greedy). do_sample, temperature and top_k add controlled randomness.
• GPT-2 is a small, early, open-source text generation model. Embedding models such as GloVe and BERT are not LLMs, because they cannot generate new text.
• Models evolved: statistical ML, deep learning, embeddings, text generation, chat models, and reasoning LLMs.

In this session

• What an LLM is, and what it can do
• Generative AI vs. agentic AI
• Why we call LLMs through an API
• Calling OpenAI (GPT-5) and Anthropic (Claude) from Python
• Reading an LLM response
• Free hosted open models, and local vs. hosted LLMs

In one sentence: you call a hosted LLM from Python with an API key, send a prompt, and read the text out of the response.

1. What is an LLM, and what can it do?

An LLM (large language model) is a model trained on a very large amount of natural-language text. It is a decoder-only transformer that generates the next token. Any use case where a machine has to understand human language and answer in it, such as a chat or a question-and-answer tool, uses an LLM.

LLM capabilities: generation, summarization, classification, reasoning

2. Generative AI vs. agentic AI

Statistical AI vs. generative AI vs. agentic AI

All of these use the same LLM. The names describe what you build:

As a DevOps engineer you will mostly use generative AI tools (such as ChatGPT or Claude) and need good prompting skills. For example, you ask for a YAML file or a script. But you will build agentic systems that connect to your environment and do the work. The next sessions cover prompting, then generative AI, then agentic AI, which takes most of the remaining course.

3. Why call an LLM through an API?

GPT-2 could be downloaded and run on a laptop. Newer models are different:

• Closed models (OpenAI since GPT-3, and Anthropic’s Claude) are not released to the public. They run on the vendor’s servers, and you reach them through an SDK and an API key. You pay for what you use.
• Latest models are huge. Even when a model is open, running it needs much more memory and compute than a laptop has.
• Newer models have billions of parameters, against millions for GPT-2 (the small GPT-2 has about 124 million). More parameters and better training give better quality, but the base idea is still next-token prediction.

4. Calling OpenAI (GPT-5)

Four ways to call an LLM

1. Create an account on the OpenAI platform, add billing and credit, and generate an API key.
2. Put the key in an environment variable.
3. Install the SDK and call the model.

import os
from openai import OpenAI              # pip install openai

client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

resp = client.responses.create(
    model="gpt-5",
    input="In two sentences, what is a Kubernetes pod?",   # this is the prompt
    reasoning={"effort": "low"},       # less thinking: faster and cheaper
    max_output_tokens=100,             # the maximum tokens to generate
)
print(resp.output_text)

• The input is the prompt. The model turns it into tokens and predicts the answer one token at a time.
• reasoning effort low means the model does not spend much effort. That is fine for a demo.
• max_output_tokens limits how much it writes. You control it according to your requirement.
• If the account has no credits, the call is rejected (the demo showed exactly this error: the key was valid, but no balance was left).

5. Calling Anthropic (Claude)

Anthropic also hosts its models. You need an account, funds, and an API key from the Anthropic console. The pattern is the same:

import os
import anthropic                       # pip install anthropic

client = anthropic.Anthropic(api_key=os.getenv("ANTHROPIC_API_KEY"))

resp = client.messages.create(
    model="claude-opus-5-5",           # the model name
    max_tokens=100,                    # required
    output_config={"effort": "low"},
    messages=[{"role": "user", "content": "In two sentences, what is a Kubernetes pod?"}],
)

• role: "user" tells the model that this message comes from the person using it.
• Roles are user, assistant (the model) and system. They are covered in the next session on prompting.

Reading the response

If you print resp, you see more than the answer:

• an id and the model name,
• the content, a list of blocks, where the text is inside a text block,
• the role, which is assistant: that is the LLM replying,
• the stop reason, here max_tokens, because the 100-token limit was reached,
• the usage: about 26 input tokens and 100 output tokens.

You normally want only the text, so extract it:

answer = next(block.text for block in resp.content if block.type == "text")
print(answer)

You are charged by the tokens you send and receive, so the usage numbers matter.

6. Free hosted open models: Groq

Not every LLM needs to be paid for. There are many open-source LLMs on Hugging Face. A small one can be downloaded and run like GPT-2. A large one needs too much compute, so you use a hosting provider.

Groq (not xAI’s Grok) is a hosting provider. It runs open models on its own infrastructure and offers them through an API. The free tier has limits, for example about 30 requests per minute and a daily token cap, and it lists only some models (such as Llama from Meta, OpenAI’s open models, and Whisper). If you need more, you pay. Always check the provider’s console to see which models it offers now.

The calling code has the same shape, and often only the endpoint, the key and the model name change:

from openai import OpenAI

client = OpenAI(api_key=os.getenv("GROQ_API_KEY"), base_url="https://api.groq.com/openai/v1")

For practice, use the free open models. The code is the same, and only the answer quality differs. You can spend a few dollars on a paid model to test it once or twice.

7. Local or hosted?

Examples of hosting: an AWS Bedrock-style service calls the models for you and handles the infrastructure. To run an open model like Llama yourself, you could start an EC2 server, download it and serve it as an API. Then you carry the compute cost and the capacity planning. Closed models such as GPT and Claude cannot be run yourself.

Keep your API key safe. The demo put a key in plain text, which is only acceptable for a short test. Never commit a key to GitHub or share it. Use environment variables, and create keys with an expiry (they can be set to expire after 30 days). Better approaches come later in the course.

8. One call is not a conversation

Each call stands alone. The model does not remember the previous question. If a learner asks a follow-up (“and what about a deployment?”), the model has no idea what came before. To build a chatbot, your application must store the earlier messages and send them again with each new question. This conversation memory, and the user, assistant and system roles, are part of prompting, which is next.

Summary

Key takeaways

• An LLM is a decoder-only model that predicts the next token. It can generate, summarise, classify and reason.
• Generative AI produces text. Agentic AI also plans, acts and verifies. Both use LLMs.
• GPT and Claude are hosted and paid. You call them with an SDK and an API key.
• A call has a model name, a prompt, a token limit and optional settings such as reasoning effort. The reply contains the text plus metadata such as the stop reason and token usage.
• Groq and similar providers host open models with a free tier and limits.
• Hosted means your prompts leave your network. Local means you pay for and run the compute yourself.
• Each call is stateless. A chatbot has to resend the history.

Quick revision

Remember: send a prompt, read the text, watch the tokens, and protect the key.