Mend: it diagnoses, fixes and verifies failures on its own, even silent ones.

Results by method

Each fault is injected into a running Docker stack, then each method tries to recover it. Cells show recovered runs over total runs.

Hang runs, before and after the fix

The results file keeps only the latest run per configuration, so the first four runs (before the frozen-process fix) are listed in full here, followed by the runs after it.

How it works

What these results do not show

• The scenarios were written by the same person who wrote the agent, who knew each fix. They are development scenarios, not a blind test. A held-out set of 12 generated faults is locked, and the agent has not been run on it yet.
• Runs use temperature 0, so repeats are close to identical. Repeats show stability, not breadth.
• The hang fix was made after seeing the failures, so the 8 passing hang runs are development results, not a blind test. They also depend on a tool that names the frozen process, and on the 120b model (the 20b model failed one hang run). On the plain frozen backend a blanket restart works just as well and is faster.
• The live Break it panel runs the same agent on a small shared server. It is limited per visitor and per day, and it only injects the five faults in its list.
• The unfixable-credentials check looks for keywords. In one run the agent named the access error but blamed the wrong cause.
• Agent cells pool gpt-oss-120b with a few gpt-oss-20b runs. Early runs used one hosting provider and later runs another, with the same model.
• The queue-loss scenario is not built.

Questions

Why not just a runbook?

A fixed runbook recovers most outages faster and for free. It does nothing about silent data loss or corruption, it never tells anyone about an unfixable fault, and it fails when connections are exhausted.

What happens with a fault it cannot fix?

It escalates with a ticket and takes no repair action. The naive baseline restarts a service on that fault every time.

How is silent data loss caught?

The database accepts writes into a table that stores nothing, so every health check passes. The agent compares an append-only journal with the tables, restores the table, then replays the missing rows.

What did it cost?

One gpt-oss-120b agent run took about 3,287 to 38,697 tokens and 2 to 213 seconds. The slowest runs were the hangs, mostly waiting on the provider's 5 requests a minute limit.