Every manufacturer knows the frustration: two batches made to the same specification, from the same raw materials, on the same equipment — and the results are different. One passes. One doesn't. Or both pass, but the properties drift just enough to cause downstream problems.

The instinct is to call it random variation and move on. But in most cases, batch-to-batch variation is not random. It has causes. And those causes are usually hiding in data you already have.

Why conventional analysis misses the root cause

The standard approach to batch variation investigation is to look at one variable at a time. Did the temperature change? Did the raw material lot change? Did the mixing time drift?

This works when the cause is simple and obvious. It fails when the cause is an interaction — when the effect of one variable depends on the level of another. A raw material that performs perfectly at one temperature range but causes problems at another. A mixing time that matters only when the feed rate is above a certain threshold.

These interactions are invisible to one-variable-at-a-time analysis. They are exactly what machine learning is designed to find.

What your batch records actually contain

Most manufacturers have years of batch records: raw material lot numbers, process parameters, in-process measurements, and final quality results. This data is collected routinely, often stored in spreadsheets or a LIMS, and rarely analysed systematically.

That data contains the answer to your batch variation problem. The question is whether your analytical approach is capable of extracting it.

A typical batch record might contain 20–50 variables. The number of possible two-way interactions between those variables is in the hundreds. Three-way interactions run into the thousands. No human analyst can evaluate all of them manually. A machine learning model can.

How machine learning approaches the problem

The process starts with your existing data — no new experiments required. We structure the batch records into a format the model can learn from: each batch becomes a row, each measured variable becomes a column, and the quality outcome (pass/fail, or a continuous property measurement) becomes the target.

The model learns which combinations of variables are associated with good outcomes and which with poor ones. It identifies the variables that matter most, the thresholds that separate good batches from bad, and the interactions that conventional analysis would miss.

The output is not a black box. We translate the model's findings into specific, testable hypotheses: "Batches where raw material lot moisture content exceeds X% and mixing temperature is below Y°C show a 3× higher rate of out-of-specification results." That is a hypothesis you can verify, and a process parameter you can control.

How many batches do you need?

A common question. The honest answer is: it depends on how many variables you have and how strong the signal is.

As a rough guide, you need at least 5–10 times as many batches as you have candidate variables. If you have 20 process parameters you want to investigate, you need at least 100–200 batches in your dataset. Most manufacturers with a few years of records have this.

If your dataset is smaller, the analysis is still useful — but the conclusions need to be treated as hypotheses to be confirmed experimentally, rather than definitive findings.

The role of specification limits and control limits

One important distinction before you start: specification limits and control limits are not the same thing, and confusing them leads to poor decisions.

Specification limits define what the customer requires. Control limits define what your process naturally produces. A process can be in statistical control — producing consistent, predictable output — while still producing results that occasionally fall outside specification. Conversely, a process can be out of control (drifting, unstable) while still producing results within specification — for now.

Batch variation analysis should address both. You want to understand whether your process is stable (control limits) and whether it is capable of meeting requirements (specification limits). Machine learning can help with both, but the questions are different and the data requirements are different.

What to do with the findings

The output of a batch variation analysis is a set of prioritised hypotheses about the causes of variation. The next step is to test them — either by running targeted experiments or by implementing process controls and monitoring the results.

In most cases, the highest-priority finding points to a process parameter that can be tightened, a raw material specification that needs to be updated, or a measurement that needs to be added to the in-process monitoring programme.

The goal is not to eliminate all variation — some variation is inherent in any process. The goal is to reduce the variation that matters: the variation that drives out-of-specification results, customer complaints, and rework.

Starting the analysis

If you have batch records going back two or more years, you probably have enough data to start. The first step is to structure the data — ensuring that each batch has a complete record of the variables you want to investigate, and that the quality outcomes are clearly defined.

We can help with that structuring process, and with the analysis that follows. The findings are usually surprising — not because the causes are exotic, but because they were hiding in plain sight in data that was never analysed systematically.