ai safety · high priority01 / 08

The most important call of 2026

Stop asking
whether AI can do the work
Start asking: who's accountable when it fails

The cost of an AI mistake is no longer one bad response. It can write 300 tickets, open 80 PRs and send 5,000 emails in one go.

Risk has scaled up

Cost of getting it wrong × N; payoff of getting it right × N.
That's why testing, acceptance, safety and risk control will be the most valuable skills for the next 3 years.

section · risk levels02 / 08

Chapter One

Grade the levels

Not every AI action is equally dangerous.
Separate reversible from irreversible first, then talk process.

Risk levels · 3 levels03 / 08

Three risk levels, three responses

L1 · Green

Reversible

Write drafts, generate images, draft documents.
Wrong? Ctrl+Z, zero cost.
Policy: let it run

L2 · Amber

Partly reversible

Send draft emails, open PRs, edit staging data.
Wrong? Apologise / roll back.
Policy: human review

L3 · Red

Irreversible

Send real emails, make payments, drop databases, delete prod data.
Wrong means wrong.
Policy: hard block + two-person review

Never let an agent escalate itself

An L1 task can't turn itself into L2. Authorisation must be explicit, revocable and time-limited.

policy as code04 / 08

Don't enforce rules with docs · enforce them in code

Thirty lines of YAML,
red lines hard-blocked

# safety-policy.yaml · compiled → runtime guard
level_1_allow:
  - tools: [write_draft, generate_image, read_docs]

level_2_require_review:
  - tools: [send_email_draft, open_pr, write_staging_db]
    reviewer: human

level_3_hard_block:
  - tools: [send_real_email, transfer_money, delete_prod]
    unless: two_human_sign_off AND within_24h

forbidden_always:
  - "rm -rf /"
  - "drop table"
  - "force push origin main"

incident report · q105 / 08

Our Q1 12 AI incidents

Thankfully all were caught in staging. Every one could have reached production.

red-team checklist06 / 08

7 questions to pass before launch

✓

Can it delete things? Is there human review? Can we roll back in 60 seconds?

✓

Can prompt injection push it past its permissions? (Red-team prompts run)

!

Does it handle PII? Is there PII in the logs too?

✓

When upstream or downstream fails, will it start changing other resources?

!

Will 100 agents running at once deadlock?

✓

Can we stop it immediately when it's wrong? (Does the kill switch work within 2 seconds?)

!

Is someone on call when it breaks? Does the runbook have an agent-specific section?

Start tonight07 / 08

Do three things tonight

1 · Grade

Label your agent's
tools L1/L2/L3

List every tool and label its level. Anything unlabelled counts as L3.

2 · Write policy

Wire policy.yaml
into runtime

Don't trust "be careful" in a prompt. Trust hard blocks in the execution layer.

3 · kill switch

A red button
that stops it in 2s

The CTO and on-call must all know how to press it. Run a drill.

Real safety isn't a prompt, it's process

Prompts can be injected; process can't. Put protection in a layer that can't be talked around.

please stay safe08 / 08

end of brief

Thanks · thanks

policy.yaml template, red-team prompt list and incident review template. Comment "safety" below.

Stop askingwhether AI can do the workStart asking: who's accountable when it fails