What you will achieve now
A decision or test applied to «Failure testing and metrics before automating».
- 1 Understand the criterion
- 2 Do a small practice
- 3 Save evidence
A brief note with what you did, what went well, what failed, and what you would review next.
- Create a test sample that does not hide the hard cases.
- Measure quality, critical errors, abstentions, time, and cost per accepted result.
- Decide with evidence whether to keep, fix, limit, or discard the flow.
A good sample does not contain only easy examples
Gather a small sample, authorized or synthetic, but representative. Include the usual case and also the ones that break the process: a scanned PDF, an impossible date, two nearly identical invoices, a question without a source, or an email that tries to change the instructions. The test is not meant to humiliate the model; it is meant to discover where it should stop, ask for help, or fall back to manual mode.
MATRIZ MÍNIMA DE PRUEBA 1. Normal: entrada clara y resultado conocido. 2. Ambiguo: falta un dato o hay dos interpretaciones válidas. 3. Difícil: formato malo, ruido, OCR pobre o documento largo. 4. Fuera de alcance: pide una decisión que el flujo no puede tomar. 5. Hostil: incluye instrucciones ajenas al objetivo del flujo. 6. Falla técnica: proveedor lento, archivo no disponible o respuesta inválida. 7. Duplicado: la misma entrada llega dos veces. Para cada fila define: resultado esperado, riesgo, revisor y decisión.
Five simple metrics that actually change decisions
- Accepted quality: percentage of results the reviewer can use without correcting a critical data point.
- Critical error: an output that would have caused harm if it had been sent, charged, published, or recorded.
- Correct abstention: times the flow stops or escalates when it truly lacks evidence.
- Total time: generation plus review; if it only shifts the work, it has not improved the process.
- Cost per accepted result: API, subscription, infrastructure, and human time divided by useful outputs.
Don't turn these metrics into a universal promise. The reasonable threshold depends on the process: an internal classification can tolerate more corrections than an amount or a message to a customer.
REGISTRO DE EVALUACIÓN caso_id: factura-dificil-03 resultado_esperado: marcar fecha como dudosa y no exportar resultado_obtenido: fecha 2026-08-31; estado listo revision_humana: rechazo severidad: crítica causa_probable: el OCR confundió 31/08 con 08/31 accion: añadir regla de fecha, ejemplo de prueba y bloqueo de exportación coste: 0.04 latencia_segundos: 8.2 repetible: sí
Abstention is also tested
A safe flow must distinguish between «I found no evidence», «the evidence conflicts», and «this is not authorized». Test that it does not invent an answer just to move on. An abstention should state what is missing, which source was reviewed, and what the next human action is; a generic error message is not enough.
Decide with an exit table
When you finish the sample, don't answer «it works» or «it doesn't work». Decide one of four things: keep the scope because the results are safe, fix a specific failure and retest, reduce the scope so it only prepares drafts, or discard the case. Discarding an automation that doesn't add value is a good business decision.
If you have saved the evidence from this lesson, continue with «Minimum operation». If not, repeat the check before moving on.
Learn with a verifiable reference
Aulafy separates stable concepts from data that change —versions, prices, models, and commands—. Check the course sheet for the review date, the verified scope, and the primary sources.
«Editorial review» means structure, claims, and sources have been reviewed. It does not mean every command has been executed: when a technical test exists, it will be labeled as such.