Technology
Claude Testing& Development
A model in production is a dependency whose behaviour changes without a version bump on your side. Where a product is built on Claude we treat it that way: a golden dataset, an evaluation harness scoring accuracy, grounding and refusal behaviour, and thresholds that fail a build when quality moves the wrong way after a prompt change or a model update.
Prompt regression suites run in CI, not by hand
Grounding and hallucination checks against source context
Refusal and safety behaviour verified, both directions
Cost and latency tracked alongside quality
- Best fit
- Products where an LLM response reaches a user or triggers an action
- Strength
- Long context, tool use and steerable refusal behaviour
- Typical first work
- A golden dataset and a baseline for what the feature does today
- Honest limits
- Non-deterministic by design, so exact-match assertions are the wrong tool
Self-check
Signs your team needs this
If more than one of these is true, it is usually cheaper to fix now than after the next release.
A prompt change ships because it looked better in a handful of Manual checks
Nobody can say whether last month's model change made the feature better or worse
Hallucinations are found by users rather than by a test
Retrieval quality has never been measured separately from the model
A Senior Engineer replies within one business day. NDA first.
What breaks in a Claude-backed feature
Traditional QA assumes a fixed input produces a fixed output.
Traditional QA assumes a fixed input produces a fixed output. An LLM feature breaks that assumption at the foundation, and the failures that follow are not the ones a test suite is shaped to catch: an answer that is fluent, confident and wrong; a retrieval step that returns the wrong document and a model that summarises it faithfully; a refusal that fires on legitimate requests, or does not fire when it should; a prompt edit that fixes one case and regresses four others nobody re-checked.
None of that shows up as an exception or a failed status code. It ships, and a user finds it.
Measuring instead of eyeballing
The unit of work is a golden dataset: real inputs from your traffic, paired with what a correct response looks like, scored across repeated runs.
The unit of work is a golden dataset: real inputs from your traffic, paired with what a correct response looks like, scored across repeated runs. That turns a non-deterministic system into a measurable one, because you are tracking accuracy, grounding and refusal rates as distributions rather than asserting an exact string.
Once that exists, a prompt change stops being a matter of opinion. You run the harness, you see which rates moved, and the change ships or does not on evidence. The same harness compares models on your actual workload rather than on public benchmarks that may not resemble it, which matters when the frontier moves, and it always does.
Where this sits in the stack
Claude-specific work usually covers prompt regression, grounding and refusal behaviour, tool-use and agent step validation, and evaluation harnesses that gate a release.
Claude-specific work usually covers prompt regression, grounding and refusal behaviour, tool-use and agent step validation, and evaluation harnesses that gate a release. Products that use retrieval need the retrieval layer tested as its own component, because a grounded model answering from the wrong document is a data problem wearing a model costume. Our full approach is on the AI Quality Engineering practice page, and the model-agnostic side lives under LLM integration.
One agent run
Nine steps right, one step wrong, and the task still reports success
This is what “the agent completed nine steps and failed the tenth, silently” looks like. The run finished. The user got an answer. The answer quoted the wrong price.
- 01parse requestintent + entities extracted
- 02search knowledge base8 candidate documents
- 03rank contexttop 3 passed relevance floor
- 04call pricing API200 OK, 42 ms
- 05call inventory API200 OK, 61 ms
- 06reconcile recordsno conflicts
- 07draft responsewithin length budget
- 08policy checkno restricted terms
- 09format for channelvalid markdown
- 10apply customer discounttool returned null — agent answered anyway, quoting list price
The task ran to completion and produced a well-formed answer, so the end-to-end check is green. This is the assertion most teams have.
Illustrative agent run. The failure shape is the one described above, not a specific client system.
Engagement path
How the engagement runs
How a Claude engagement runs from first call to handover.
Build the golden dataset
Real inputs from your traffic, paired with what a correct response looks like, reviewed by someone who knows the domain.
Baseline the current behaviour
Score it as it stands today, so every later claim of improvement has something to be measured against.
Harness the checks
Grounding, refusal behaviour, prompt regression and tool-use validation, run repeatedly rather than once.
Gate the release
Thresholds agreed with you and wired into CI, so a prompt or model change has a measurable effect before it reaches users.
Deliverables
What you get
Artefacts you keep and can run without us. Everything lives in your repositories and your pipelines.
Handover pack
5 artefacts · yours to keep
A golden dataset built from your real traffic, and yours to keep
A baseline score for accuracy, grounding and refusal rates
A repeatable evaluation harness running in your pipeline
Prompt regression suites that catch the fix-one-break-four pattern
Written thresholds and what to do when a run falls below them
Questions
Frequently Asked Questions
Straight answers, written the way we'd say them on a call.
Still curious? Talk to usKeep reading
Where to go next
Teams evaluating Claude usually read these next.
Need Claude Expertise?
Start with a scoping call or a free assessment, Engineers available within days.
- A Senior Engineer replies, not a sales layer
- Within one business day, every time
- NDA available before you share any details
16+
Years QA leadership
The founder's enterprise QA career across OTT, SaaS, e-commerce and regulated utilities. Not a team total.
17
Testing disciplines
Each one has its own page, scope and deliverables. Counted from that list, never typed by hand.
6
Markets served
Availability, not delivery history. Each market's page says plainly where we have clients and where we do not.
1
Business day to reply
A Senior Engineer answers, not an autoresponder or a scheduler.
Tell us where quality hurts
Prefer to talk? Book a 30-minute call