Worked examples
What a silent failure actually looks like.
Two worked examples, one for each thing the audit checks. A nightly sanctions check that stopped running, and a nightly erasure job that ran every night and deleted nothing. Each one shows what the dashboard said, what the check found, and what you would then have to decide.
These are illustrative examples, not client work. The systems, figures and dates are invented to show the shape of the failure. There is no published case study yet and I am not going to imply otherwise.
01Expected-run check
The sanctions check that stopped after a move
Eleven weeks where the nightly check didn't run, didn't fail, and looked exactly like eleven weeks with no matches.
01The problem
A firm checks its customers against the sanctions lists every night. A job does it automatically, and a person only gets involved when it finds a possible match. No matches is normal, so a quiet night looks exactly like a night where the check never ran at all.
02What the dashboard showed
No alerts raisedThe job history, this summer
Illustrative example
The job history, this summer (Illustrative example) Night Job status Finished at Matches found 12 Jul Completed 01:06 0 13 Jul Completed 01:05 0 14 Jul Completed 01:07 0 12 Jul
- Job status
- Completed
- Finished at
- 01:06
- Matches found
- 0
13 Jul
- Job status
- Completed
- Finished at
- 01:05
- Matches found
- 0
14 Jul
- Job status
- Completed
- Finished at
- 01:07
- Matches found
- 0
Every run finished cleanly and found no matches, which is what a normal night looks like. The nights after 14 July aren't in this table because nothing ran, and a record of what happened has no way of showing you that. From the outside, no run and no matches look the same.
03What the check found
3 of 4 failed the checkHow the check works
Input
The schedule it should keep
One screening run every night
Process
Compare the runs it should have made with the runs it did
The record of what ran, not the error log
Output
Every night where nothing actually ran
Quiet, so it looked normal
Human check
You confirm which gaps are controls you answer for
A missed report is awkward. A missed screening is not.
The same summer, with the nights that never ran put back in
Illustrative example
The check asks Did anything run at all?The same summer, with the nights that never ran put back in (Illustrative example) Period Runs expected Runs recorded Verdict 1 to 14 Jul 14 14 Ran every night 15 to 31 Jul 17 0 Never ran August 31 0 Never ran September 30 0 Never ran 1 to 14 Jul
- Runs expected
- 14
- Runs recorded
- 14
- Verdict
- Ran every night
15 to 31 Jul
- Runs expected
- 17
- Runs recorded
- 0
- Verdict
- Never ran
August
- Runs expected
- 31
- Runs recorded
- 0
- Verdict
- Never ran
September
- Runs expected
- 30
- Runs recorded
- 0
- Verdict
- Never ran
The schedule that starts the job was switched off while the systems were being moved to a new platform in July, and nobody switched it back on. From then on the job did not run, so it did not fail, so it raised nothing. And because a normal night finds no matches, nobody missed the silence.
That is the whole failure. It is why the expected-run check starts from when a job should have run and looks for the run, rather than waiting to be told something went wrong. It is the only check that treats 'nothing happened' as an answer rather than as silence.
04The exposure
Eleven weeks where no customer was checked against changes to the sanctions lists, and a dashboard that stayed green throughout. If a customer was added to a list in that time, nobody would have known. The fix is an afternoon. The hard part is showing afterwards that nothing was missed.
05What you'd decide
Finding it is the easy half. These decisions are yours.
- Confirm which of these processes are controls you answer for and which are internal housekeeping.
- Agree what counts as evidence that the check ran. A schedule is not a run.
- Decide who re-runs the missed screening, and who reviews any matches first.
- Decide what should have warned you when a night passed with no run, and who owns it.
What I don't touch
Nothing is changed during a check. I look at the record of what ran and what it checked, and I can't change either. Nothing in your live systems is touched. Re-running the missed screening is your decision, made with the findings in front of you.
What you get
A written list of everything that runs on its own, whether it ran, and whether anything landed. Where something did not, you get the period, the evidence and the exposure. Where everything is fine, that is what it says.
02Expected-output check
The erasure job that deleted nothing
A nightly retention job that finished green for eleven months and removed no records at all.
01The problem
A business erases personal data once it passes the age set in its retention policy, and a nightly job does the erasing. It has never failed, so nobody looks at it, and the one number that would give it away is the one nothing records: how many records it actually removed.
02What the dashboard showed
No alerts raisedEleven months of a well-behaved job
Illustrative example
Eleven months of a well-behaved job (Illustrative example) Run Duration Exit status Alerts raised 12 Nov, 01:00 4.1s Success None 13 Nov, 01:00 3.9s Success None 14 Nov, 01:00 4.0s Success None The 320 nights before that 3.8s to 4.3s Success None 12 Nov, 01:00
- Duration
- 4.1s
- Exit status
- Success
- Alerts raised
- None
13 Nov, 01:00
- Duration
- 3.9s
- Exit status
- Success
- Alerts raised
- None
14 Nov, 01:00
- Duration
- 4.0s
- Exit status
- Success
- Alerts raised
- None
The 320 nights before that
- Duration
- 3.8s to 4.3s
- Exit status
- Success
- Alerts raised
- None
A job that finishes in four seconds every night and never errors reads as a healthy job. Four seconds is also roughly how long it takes to do nothing at all, and there is nothing in this view that tells the two apart.
03What the check found
3 of 4 failed the checkHow the check works
Input
What the job is supposed to produce
Deleted records, and a count of them
Process
Check something came out, and roughly the right amount
Present, non-empty, roughly the right size
Output
Runs that completed and produced nothing
Green, and empty
Human check
You decide how far back it goes
How long has this been true?
The same nights, counting what came out of them
Illustrative example
The check asks Did it produce anything plausible?The same nights, counting what came out of them (Illustrative example) Run Records past retention Records erased Verdict Early Feb, before the change 874 874 Ran and erased 12 Nov 1,184 0 Completed, did nothing 13 Nov 1,190 0 Completed, did nothing 14 Nov 1,203 0 Completed, did nothing Early Feb, before the change
- Records past retention
- 874
- Records erased
- 874
- Verdict
- Ran and erased
12 Nov
- Records past retention
- 1,184
- Records erased
- 0
- Verdict
- Completed, did nothing
13 Nov
- Records past retention
- 1,190
- Records erased
- 0
- Verdict
- Completed, did nothing
14 Nov
- Records past retention
- 1,203
- Records erased
- 0
- Verdict
- Completed, did nothing
In February someone renamed a field in the database that the job used to decide what was old enough to delete. The job kept running and kept reporting success, but from that night on it found nothing to delete, so it deleted nothing and finished cleanly. There is no error anywhere in this, because as far as the software was concerned nothing went wrong. The job did exactly what it was told.
The expected-output check never asks whether the job succeeded. It asks what came out, and whether that is a believable amount. Zero deletions on a night with eleven hundred records past their retention date is not believable, and that was the only clue there was ever going to be.
04The exposure
Eleven months of personal data held past the period the retention policy commits to, and a written policy saying it was erased. That is the kind of gap found by a subject access request or an auditor rather than by an engineer, and there is a paper trail claiming the opposite.
05What you'd decide
Finding it is the easy half. These decisions are yours.
- Confirm the retention schedule the job is meant to enforce, in writing, before anything is deleted.
- Agree roughly how many deletions a normal night should have, so that zero counts as a failure.
- Decide whether the backlog is cleared in one pass or in stages, and who signs that off.
- Work out what was said to whom while the job was reporting success.
What I don't touch
The check reads. It deletes nothing and it changes nothing about the job. What to do with eleven months of records that should not exist is a legal and commercial decision, not an engineering one, and it stays with you.
What you get
The date the output stopped being plausible, the evidence behind it, and a check you can keep running afterwards so that the next time the answer is zero, something says so.
Start with the audit
Which of yours would this find?
Pick the process you'd least like to have quietly stopped, and think about how you'd currently find out. If there isn't an answer to that, it's the one to check first.