Agentic AI DevOps: ML Training – Linear Regression
Recap of the previous session
- A virtual environment gives each project its own libraries, and
requirements.txtlists them. - Pandas reads a CSV into a DataFrame.
shape,head,infoandcolumnshelp you analyse it. - Split the data into X (features) and Y (target), create
LinearRegression(), train withfit, and predict withpredict. coef_andintercept_are the learnedmandb. The model was saved as a.pklfile with joblib.
In this session
- The usual steps before training: analyse, engineer features, split, train
- Plotting data with Matplotlib to see the relationships
- A dataset with several features: which ones to keep
- Categorical data and one-hot encoding
- Why you must not test on the data you trained on
- Splitting into training and testing data
- Training with many features, and measuring the score with R²
In one sentence: with real data you prepare the features, hold back some records for testing, and judge the model only on data it has never seen.
1. The steps before training
When you receive a dataset, it is never ready for training:
- Analyse the data: how many records and columns, what types.
- Engineer the features: check each feature (each column) and change them if needed.
- Split into features (
X) and target (Y). - Train the model.
2. Seeing the pattern with a plot
Printing numbers is fine for 10 records, but not for hundreds. Plotting shows how x changes y.
- Matplotlib plots the data (Seaborn is another option). Pandas loads the data, Matplotlib draws it and NumPy handles the numbers.
- You do not need to memorise the syntax. Know that this step exists, and copy the pattern from the documentation or ask an AI assistant. Data scientists normally lead this analysis, but you should know how it is done.
!python3 -m pip install matplotlib
from matplotlib import pyplot as plt
plt.scatter(X, Y) # a scatter plot of the points
plt.xlabel("Experience")
plt.ylabel("Salary")
plt.show()
For the experience and salary data, the points rise steadily in a straight line: a linear relationship. Because the data is a perfect line, b is 0 and m is easy to find.
Notebook tip: the notebook is for experiments, not for building the product. You can run everything with Run All Cells, but use the notebook only where you need to try things and go back and forth. Then move the final code to your IDE.
3. A dataset with several features
The next dataset has 100 employees and five columns: experience, projects completed, job title, certificates completed, and salary. Salary is the target and the other four are features.
Which features should you keep? That is a business and domain decision. If your company does not use certificates to decide salary, including that column only adds complexity. Keep the features that have a significant impact and drop the ones that do not. This is part of feature engineering.
The data scientists usually prepare this for you. But you should also challenge it: if a feature looks unnecessary, tell them.
Reading the data with info() and nunique()
data.info()shows the types. Experience, projects, certificates and salary are numbers. Job title is a string.data.nunique()shows how many unique values each column has. Job title has 5, certificates has 6, and salary has 100, so every employee has a different salary.
Why does this matter? Think of a student practising with 100 questions that are all the same. He learns nothing. If all 100 employees had the same salary, the data would teach the model nothing. A column with a healthy spread of values gives the model something to learn from. If the data is not good, go back to the data engineering or data science team.
Which feature matters most?
Plot every feature against the salary:
fig, axes = plt.subplots(1, 3, figsize=(15, 4))
for ax, col in zip(axes, ["Experience_Years", "Projects_Completed", "Certificates_Completed"]):
ax.scatter(data
, data["Salary"])
ax.set_xlabel(col)
ax.set_ylabel("Salary")
plt.show()
When experience changes, the salary changes a lot. When certificates change, the salary barely moves. With 20 features this analysis helps you drop the ones that carry little weight and keep the data lean.
You can also get the numbers with corr():
data[["Experience_Years", "Projects_Completed", "Certificates_Completed", "Salary"]].corr()["Salary"]
| Feature | Correlation with salary |
|---|---|
| Experience_Years | 0.71 |
| Projects_Completed | 0.26 |
| Certificates_Completed | 0.17 |
The closer to 1, the stronger the feature’s influence on the salary. Experience is clearly the main driver.
4. Categorical data and one-hot encoding
A model can only work with numbers. Job title is text, so it is categorical data, and it must be converted. This conversion is called encoding.
One-hot encoding creates one column per category, with 1 for the employee’s category and 0 for all the others:
data_encoded = pd.get_dummies(data, columns=["Job_Title"], dtype=int)
data_encoded.head()
| Job_Title | becomes | Junior | Lead | Manager | Mid | Senior |
|---|---|---|---|---|---|---|
| Mid | 0 | 0 | 0 | 1 | 0 | |
| Lead | 0 | 1 | 0 | 0 | 0 |
The five job titles gave five new columns. Only one is 1 for each employee, so the meaning is kept, but now it is numeric. There are other encoding methods too, and this is the simplest.
From here on, use data_encoded, not the old data.
5. Features and target
X = data_encoded.drop(columns=["Salary"]) # all the features
Y = data_encoded["Salary"] # the target
X has 8 columns (experience, projects, certificates and the five job titles). Y has only the salary.
6. Do not test on the training data
In the first example, the model was tested with the same data it was trained on, so it scored 100%. That is like giving a student ten practice questions and then the same ten in the exam. He passes easily, but it proves nothing. This is called leakage.
To judge a model fairly, you need data it has never seen. So you split the dataset: for example, 67 records for training and 33 for testing.

The ML workflow: training, testing, inference
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.33, random_state=42)
| Parameter | Meaning |
|---|---|
test_size=0.33 |
33% of the records go to testing, the rest to training |
random_state=42 |
A fixed number that controls the random shuffling, so you get the same split every run. Any number works |
The split is random, not “top 80 for training, last 20 for testing”. If the data happened to be sorted by salary, the top rows would all be low salaries, and the model would never see the high ones. A random split mixes them into both sets.
You get four pieces: X_train and y_train (67 rows) for training, and X_test and y_test (33 rows) for testing. Both X parts have 8 columns.
7. Training and predicting
from sklearn.linear_model import LinearRegression
lg_model = LinearRegression()
lg_model.fit(X_train, y_train) # train on the 67 training records
lg_model.coef_ # 8 values
lg_model.intercept_ # 1 value
- One weight per feature: 8 features give 8 coefficients. There is only one intercept.
- The salary for an employee is
m1·x1 + m2·x2 + ... + m8·x8 + b.
predictions = lg_model.predict(X_test) # 33 predictions, one per test employee
8. How good is the model?
y_test holds the real salaries of those 33 employees (the truth), and predictions holds what the model said. Compare them with the R² score:
from sklearn.metrics import r2_score
score = r2_score(y_test, predictions) # about 0.75
The R² score compares the truth with the predictions. 1.0 is perfect. Here it is about 0.75, so the model is roughly 75% accurate. A one-to-one check shows the gap: for one employee the real salary is about 67,000 and the model predicted about 72,000.
This model is not good enough for production, because it predicts too high. To improve the score, you go back and change things: the features, the data, the training, or the algorithm. This is experimentation, and it is the next session. If linear regression stays weak, try three or four other algorithms from scikit-learn. You cannot try all hundreds, so use your judgement.
9. Sizing the server for training
A learner asked how to decide the server capacity (CPU, GPU) for an AI project. The answer is the same as for any application you deploy:
- Start with a reasonable number, such as 2 GB and 1 core.
- Run performance and load tests in a pre-production environment, with the volume the business expects.
- Monitor it, look at the spikes, and tune.
By the time you reach production, you already have a tested capacity number.
Summary
Key takeaways
- Always analyse the data and engineer the features before training.
- Plot features against the target to see which ones matter. Drop those with little influence.
- A model needs numbers. Convert categorical data with one-hot encoding (
pd.get_dummies). - Never test on the training data. Split it with
train_test_split, randomly. - With many features, there is one weight per feature and a single bias.
- Judge the model on the test set with a score such as R². Here it is about 0.75, which is not enough, so you experiment.
Quick revision
| Term | Meaning |
|---|---|
| Feature engineering | Choosing and preparing the input columns |
| Categorical data | Non-numeric data such as a job title |
| One-hot encoding | One 0/1 column per category |
| Leakage | Testing on data the model already saw in training |
train_test_split |
Splits data into training and testing parts |
random_state |
Fixes the random split so that results repeat |
| R² score | Compares predictions with the truth (1.0 is perfect) |
Remember: split first, train on the training set, judge on the test set.
