Requirements for AI & Data-Driven Systems: Specifying What You Cannot Predict

September 2, 2026· 9 min read

Classical requirements engineering assumes you can write "the system shall do X" and then test whether it does. Machine learning breaks that assumption: the behavior is not written in code, it emerges from data, and it is never 100% correct. You cannot write "the model shall always be right".

That does not mean ML systems get a free pass on requirements. It means they need a different specification language — one built around measurable performance, evaluation data, and error budgets instead of deterministic guarantees.

Why ML breaks the classic template

A traditional requirement pins down an explicit, enumerable behavior. A spam filter, a translation model, or a defect detector has no single "correct" output for a given input — it has a probability distribution, and correctness is a matter of degree over a population of inputs you have never seen.

So the sentence "the system shall be accurate" is not merely vague — it is category error. Accuracy only means something relative to a specific dataset and a specific definition of what counts as a correct answer.

Specify the metric, not the mechanism

The fix is to state requirements as measurable performance targets on defined data, not as behavioral assertions about the model. The requirement becomes the metric, and the metric becomes testable.

Classification: "the model shall achieve F1 ≥ 0.90 on the held-out evaluation set."
Regression: "the model shall achieve mean absolute error ≤ 5 kWh on the last 12 months of held-out consumption data."
Latency: "inference shall complete in under 100 ms at p95."
Safety: "the model shall never emit <category> on any input in the adversarial test set."

The eval set is part of the requirement

If the metric is the requirement, then the dataset the metric is measured on is as important as the sentence itself. A requirement that says "90% accurate" without specifying the data is meaningless — accuracy on the training set is not accuracy.

So the specification must pin down the evaluation data: what it contains, how it was collected, how it is held out from training, and how it is versioned. The data is not an implementation detail; it is the definition of "correct".

Error budgets and drift

An ML system does not stay correct forever — the world changes, and the model drifts. Requirements must therefore specify what to do when performance degrades:

A performance threshold that triggers retraining or review.
A monitoring signal — a live proxy for the metric, checked on a schedule.
An error budget — the acceptable rate of wrong outputs, with a response when it is exceeded.
A rollback path — the previous model version, kept deployable.

What still works from the classic playbook

Traceability, versioning, and change control do not go away — they change target. You trace a requirement not to a function but to a metric, an evaluation set, and a model version. A "model change" is a requirements change: swapping in a new model means re-verifying against the same held-out data, and recording it.

The discipline is the same. Only the object of the requirement has changed, from a behavior you can enumerate to a performance you must measure.

You cannot specify what an ML system will do for every input. You can specify how good it must be, measured on what data, and what happens when it is not — and that is a requirement.