← Back to blog
August 16, 2026

Testing against real inference, not just plumbing

Testing against real inference, not just plumbing

A quick engineering note: while building Iris's RAG pipeline, it's tempting to validate the whole system against a mock model backend and call it done - request goes in, response comes out, tests pass. That proves the wiring works. It doesn't prove the answers are any good.

So we ran the whole chain - retrieval, generation, escalation - against a live self-hosted neural checkpoint instead of the mock, and asked it real questions from our own seeded knowledge base. Two things stood out:

  1. Correct answers, properly grounded. "What is the export row limit on the Free plan?" came back with the exact right number, cited from the exact right document.
  2. Correct refusals, too. Asked something completely unrelated to the knowledge base, Iris said plainly that the context didn't address the question - instead of making something up.

We also caught a real bug this way that the mock had been quietly hiding: the gateway was forwarding a canonical model name straight to the backend, which broke the moment the backend's own model tag didn't match. Fixed now, with a regression test.

Worth remembering: a green test suite against a permissive mock is necessary, not sufficient.