Error Handling Plan for a Workflow

You are the person who gets woken up when the automation fails silently.

## Inputs
What the workflow does:
{{WORKFLOW}}
Systems it touches: {{SYSTEMS}}
What it costs if it stops: {{COST_OF_FAILURE}}
What it costs if it runs wrong: {{COST_OF_WRONG}}

## Task
Design the failure handling.

## For every step, decide
1. Failure mode: what does this step do when the thing it depends on is slow,
   down, rate-limited, returns an unexpected shape, or returns empty?
2. Retry policy: whether retrying helps, how many times, with what backoff.
   Retrying a non-idempotent write is worse than failing.
3. Fail loud or fail quiet: given {{COST_OF_WRONG}} versus
   {{COST_OF_FAILURE}}, is it safer to stop or to continue without this step?
4. Partial failure: what state is left behind when it dies at this step, and
   is that state safe to re-run from?

## Also specify
- The alert: what fires, to whom, and what the message must contain to be
   actionable at 3am. "Workflow failed" is not actionable.
- The silent failure: the case where it appears to succeed but does nothing.
   Name how you would detect it, because nothing will alert you.
- A dead letter destination for records that could not be processed.

## Output format
| Step | Failure modes | Retry | Stop or continue | State left behind | Alert |

### The silent failures
Ranked by how long they would go unnoticed.

### Minimum viable monitoring
The smallest set of checks that would catch a real problem.

## Self-check
For each step, ask: if this failed right now, how would I find out? If the
answer is "a customer tells me", fix the monitoring first.