Why we split ML inference into three services

The obvious way to serve a model is one service: accept a request, run the model, return the prediction. It works until any of three things happens — traffic arrives in bursts, the model gets slower than the request timeout, or a downstream consumer goes down and you start dropping work you have already paid to compute.

The split

• Ingest accepts work, validates it, and writes it to a topic. It is cheap, fast, and stateless, so it can absorb spikes without touching GPU capacity.
• Inference consumes the topic at whatever rate it can sustain. It scales on queue depth rather than on request rate, which is the number that actually correlates with cost.
• Delivery takes results out and gets them where they need to go, with its own retry semantics.

The queue between ingest and inference is doing the real work here. It decouples arrival rate from processing rate, which means a burst becomes latency instead of errors, and a slow model becomes a growing backlog you can see on a dashboard instead of a wave of timeouts.

Testing that it actually recovers

A design that claims fault tolerance and has never been tested is a claim, not a property. The only way to know is to break it on purpose: kill inference pods mid-batch, partition the broker, and assert afterwards that every accepted message was eventually processed exactly once.

# delete a random inference pod while a load test runs
kubectl delete pod -l app=inference \
  --field-selector=status.phase=Running \
  -o name | head -1 | xargs kubectl delete

Two things make the recovery real rather than incidental: consumers commit offsets only after a result is durably written, and handlers are idempotent, keyed on the message id. Without both, a restart either loses work or duplicates it.

When not to do this

This costs you a broker, a schema contract, and three deployables instead of one. If your model responds in tens of milliseconds and traffic is flat, a single service behind an autoscaler is the correct answer and the queue is decoration. The split earns its complexity when arrival is bursty or inference is slow.