Agents that get better over time, not just agents you can watch
NOMOO tracks performance across runs and detects regressions the moment behavior degrades. Then it goes further: analyzing performance patterns continuously and tuning prompts and sampling parameters toward higher response quality, so improvement stops depending on one engineer's spare afternoon.
You cannot improve what nothing is grading
Optimization needs a signal. Latency and error rate are easy and almost beside the point: an agent can be fast, successful, and wrong. So NOMOO scores the substance of a run, span by span, and keeps the result attached to the trace that earned it.
What the automated reviewer looks at
The reviewer reads a completed run the way a careful engineer would, with the whole hierarchy in front of it: what was asked, what was planned, which tools were chosen, what they returned, and whether the final answer is supported by any of it.
Each dimension produces a finding rather than a vote. Findings carry a severity, attach to a span, and roll up into a score for the run. The score is a summary of the findings, which means it is always possible to ask what it was made of.
| Dimension | The question it answers |
|---|---|
| Grounding | Is every factual claim in the answer supported by something the run actually retrieved? |
| Tool selection | Were the right tools chosen, and was anything called that the task did not need? |
| Redundancy | Did the agent repeat work, calling the same tool with the same arguments twice? |
| Plan quality | Do the sub-tasks map onto available capabilities, at a sensible depth? |
| Completion | Was the user's actual question answered, or only the part that was easy? |
| Efficiency | Did the run spend tokens and latency in proportion to the work required? |
Information
Something worth recording that is not a fault. Plan depth above the trailing median, an unusually long retrieval, a tool used for the first time this week. These accumulate quietly and feed pattern analysis.
Warning
Behavior that costs you something without breaking the answer: duplicate tool calls, retrieval that returned nothing and was ignored, a plan that wandered. These form the working queue.
Error
The answer is wrong or unsupported, or an action was taken that should not have been. These are rare by design, because the lower tiers absorb the noise that would otherwise bury them.
The deploy that announces itself
Scored history plus release tags gives you something teams shipping agents almost never have: the ability to say, with evidence, that quality fell and which change it fell after. NOMOO watches the series and raises the regression before your users write in.
Quality by release
Mean score per release with the trailing baseline behind it. A drop outside normal variance is flagged and attributed to the deploy it followed.
The alert
Delivered to the channel your on-call already watches, with the runs attached.
Mean quality 7.9, baseline 8.7. Detected 41 minutes after rollout across 1,860 runs. Largest contributing finding: plan depth, up from a median of 4 sub-tasks to 6.
The planner template changed in this release. Runs on v1.5 with the previous template still score 8.7.
Detected in minutes, not retrospectives
Detection runs against live traffic, so a bad release is caught in the window when rolling back is still cheap.
Attributed, not just reported
The alert names the release, the findings that moved, and the runs that carry them, so the investigation starts with a suspect.
Confirmed after the fix
The series keeps running after you ship a correction, which is how you learn whether the fix held or only moved the problem.
One trace is an anecdote, ten thousand is a priority list
The findings on a single run tell you what to fix for one user. The same findings counted across a week tell you what to fix for everyone, and in what order. Pattern analysis is how NOMOO turns a stream of critiques into engineering work worth doing.
Redundant tool calls concentrate in one tool. 71 percent of the 4,120 occurrences are repeat calls to lookup_order with identical arguments inside a single run. Estimated cost: 164ms and one database round trip per affected run.
From volume to a concrete change
A pattern is only actionable once it localizes. Counting redundant tool calls is interesting. Discovering that seven out of ten of them are the same tool, called twice with the same arguments, in the same position of the run, is a two-line fix with a measurable payoff.
That is the work pattern analysis does: cluster findings by span, by arguments, by release, and by the shape of the run around them, then rank the clusters by how much they cost.
- Clustered, not just counted. Findings group by the span and arguments that produced them.
- Ranked by cost. Latency and token impact estimated across affected runs, so the biggest wins are visible.
- Sliced by your metadata. If a pattern only affects one locale or tenant, that shows up rather than averaging away.
- Tracked after the fix. A cluster that shrinks to zero is the proof that the change worked.
Prompt changes with the evidence attached
Most prompt engineering is an undocumented guess, tested by hand on a handful of examples and remembered by one person. NOMOO replaces the guess with a proposal: a specific edit, the pattern that motivated it, and the score it is expected to move.
How a proposal is built
NOMOO analyzes performance patterns across runs, identifies the behavior behind the lowest-scoring findings, and generates a prompt edit that addresses it directly. The edit is narrow on purpose: an added instruction, a tightened constraint, a precondition before a tool is allowed to run.
It arrives as a diff against your current template, with the findings and the affected runs linked. You can open any one of them and read the trace that argued for the change.
- Diff, not a rewrite. You see exactly what changes and can judge it in one read.
- Grounded in your traffic. Proposals come from your runs, not from a generic prompt library.
- Reviewable before adoption. Nothing reaches production until you accept it.
- Measured afterwards. The score series continues, so a proposal that did not help is visible as such.
In 2,480 runs the response stated a policy figure that no retrieved passage contained. In 61 percent of those, retrieval had returned nothing above the score threshold and the model answered anyway.
The settings nobody revisits after the first week
Sampling parameters are usually set once, during the first prototype, and then inherited forever. NOMOO treats them as what they are: tunable inputs with a measurable effect on quality, worth re-deriving from the traffic you actually serve.
Tuned against your own scores
Parameter recommendations come from the relationship between settings and scores in your run history, not from a default someone chose for a different workload. A planner that benefits from determinism and a composer that benefits from a little range end up with different values, because the evidence says so.
Per step, not per project
An agent makes several model calls with different jobs. Recommendations are scoped to the span that makes the call, so the planning step and the response step can be tuned independently instead of sharing one compromise setting.
| Span | Temperature | Effect |
|---|---|---|
| plan.generate | 0.2 | Shallower, more repeatable plans |
| respond.compose | 0.3 | Grounding holds, phrasing stays natural |
| summarize.session | 0.5 | Better coverage of long transcripts |
Five steps, running continuously
None of these stages is novel on its own. The value is in their being connected, so that a finding on Monday becomes a measured improvement by Friday without anyone building the pipeline between them.
Score
Every completed run is reviewed and scored, with findings attached to the spans responsible.
Cluster
Findings group by span, arguments, release, and metadata, then rank by their cost across runs.
Propose
The top clusters drive a prompt diff or a parameter recommendation, with the evidence linked.
Adopt
You review, accept or reject, and tag the release so the change is attributable later.
Verify
Scores keep accumulating. The cluster shrinks, holds, or grows, and that answers the question.
The step most teams skip
Verification. A prompt change feels like progress the moment it ships, and almost nobody goes back to check whether the finding it targeted actually went away. Because scoring never stops, the answer arrives whether or not anyone asks for it.
Automation that proposes, engineers that decide
An optimizer with write access to your production prompts is not a feature, it is an outage waiting for a quiet weekend. NOMOO is built the other way around.
Nothing ships without approval
Proposals are artifacts you read and accept. There is no mode in which a prompt in production changes because a score moved overnight.
Every change is attributable
Adopted proposals carry a release tag, so six weeks later the score series can still tell you which edit moved which number.
Reversible by construction
A proposal is a diff against a known template. Reverting is the same operation as adopting, run in the other direction.
Gate releases on the score
Run a scored evaluation in your pipeline and fail the build when quality falls below the threshold you set. See continuous checks.
Alerts where you work
Regressions and high-severity findings are pushed by webhook, so the signal reaches the channel your team already reads.
Your data, your evaluation set
Export scored runs and findings to build the regression suite you own, independent of anything NOMOO keeps.
Find out what your agent scores
We will instrument one agent, score a sample of real runs, and walk you through the findings and the first proposals they produce.