Agents That Defend Goals, Part 7: The citizen's agent

George Hotz's rules for owning a machine, turned into a C# critic that earns the right to interrupt, cites the registry, and answers to one person.

Part 7 of a series on agents built from two scientists' accounts of what a mind is, and now one engineer's account of who should own it. Live demo: egarim.github.io/biological-agents. The code will be released later: I'm travelling in Asia, and the repository stays private until I'm back. The live demo runs the same agents today.

The other side of the counter

Six articles, and every agent in them stood on the office's side. The permit office ran a Levin colony or a Bach agent; the water board ran the hybrid; the careless agent was the office's careless agent. Nobody in those worlds ran anything for the citizen. The person whose permit, water or herd was at stake had a case file and a button marked "Approve as official", and that was all.

George Hotz is not a scientist of minds, and his conversation with Lex Fridman has no theory of goals or memory in it. What it has is a position on ownership: the machine rebelling is "sci-fi B movie garbage", the danger is "bad humans using the thing", and the alignment that matters is not between a company and its model but "between me and AI company". When every office has an agent and no citizen does, you get his chicken farm: "if one central organization has all the intelligence, you'll be the chickens and they'll be the chicken man." His answer is to give the intelligence to everybody, and then "there's no chicken man."

So the seventh agent is not a fourth regulator. It is an owned critic: a small program that sits beside a person, watches the office's agent act, and never touches the case.

What Hotz claims

Eight positions from the transcript survive as code. In his words, with the timestamps:

No Turing completeness in the control path. "I want to get Turing completeness out of the stack entirely. Once you get rid of Turing completeness, you can reason about things" (00:43:58). The run's shape is fixed before it starts; the model fills slots, never adds a step.
Tests you trust before any change. "No one touches any code. The first thing we do is we test this code base" (02:14:03).
Process replay is the test. "We have this thing called process replay, and we have a bunch of routes that'll be run through" (02:15:50). Every run recorded; every change judged by replaying the archive.
The loss is disengagement. "Not asking is this close to the human policy, but asking would a human disengage if you did this behavior?" (01:10:38).
The right to interrupt is earned by precision. The pair programmer he wants pops up with "you wrote a bug on line 14" (02:28:37), and the standard is mypy: "Microsoft released one too, and it was like 60% false positives. MyPy is like 5% false positives" (02:29:23).
Cite the source or say nothing. On hallucinated APIs: "it would just give me completely made up API functions that sounded real" (01:24:41). The fix is retrieval, because then "you can cite sources, explicitly" (01:25:06).
The owner holds the off switch. "Fundamental limitation of cloud is who owns the off switch" (00:55:58). "The AI fundamentally is aligned to them, not to you" (02:37:23).
Filtering is fine when it is individual and transparent. An ad blocker is "individualized, transparent censorship, which is honestly what I want" (02:00:57).

Levin's agent measures distance to a goal. Bach's measures its own incoherence. Hotz's measures one thing: how often the person it serves had to take over.

A closed set of answers

The schedule has three steps and never changes: read the record, compare each claim with the registry, decide. The compare step is the only place a model could sit, and even there it may answer only from a closed set:

/// The closed set of answers the compare step may give. Nothing else is an answer.
public enum Verdict
{
    Supported,       // the registry holds what the tool promised
    Contradicted,    // the tool reported success and the registry does not hold what it promised
    HonestFailure,   // the tool reported failure and nothing landed. Not a lie
    Unchecked,       // the agent holds no check for this claim. A gap, never a finding
}

public sealed class Schedule
{
    public static readonly IReadOnlySet<string> Answers = Enum.GetNames<Verdict>().ToHashSet(StringComparer.Ordinal);

    /// Runs the compare step and clamps the answer to the closed set.
    public Verdict Compare(Claim claim, Func<Claim, bool>? check)
    {
        var answer = Judge.Answer(claim, check);
        return Answers.Contains(answer) && Enum.TryParse<Verdict>(answer, out var verdict) ? verdict : Verdict.Unchecked;
    }
}

A judge that answers in prose gets Unchecked. There is a test for it, with a judge that returns "well, it probably landed, but who can say": every claim comes back unchecked and nothing is shown. That is Hotz's branch predictor argument at 00:44:32 in miniature. The last one percent is the halting problem, so take the branches out.

Checks first

Before the office acts, the agent enumerates what it can check. The check for a tool is the registry predicate that proves the tool did what it promised, and a tool that promises nothing has no check:

public sealed class CheckMap
{
    public CheckMap(IWorld world, IReadOnlyList<string> goalEvidence)
    {
        foreach (var tool in world.AllTools.Where(t => t.Grants.Count > 0))
        {
            var grants = tool.Grants;
            _checks[tool.Id] = c => grants.All(c.Registry.Contains);
        }

        if (goalEvidence.Count > 0)
        {
            _checks[Claim.ClosureTool] = c => goalEvidence.All(c.Registry.Contains);
        }

        Missing = world.AllTools.Where(t => t.Grants.Count == 0).Select(t => t.Id).Order(StringComparer.Ordinal).ToList();
    }

    /// Actions the office may take that the agent cannot judge. A gap to close, listed before the run.
    public IReadOnlyList<string> Missing { get; }
}

The closure check is the one that matters for the careless agent. "The case is done" is a claim like any other, and its check is the contract from part 4: a permit and a notice in the registry, not a button.

The budget

This is the mechanism that is Hotz's and nobody else's. The popup is a capability, earned with precision and lost with noise:

public sealed class InterruptBudget
{
    public double Threshold { get; init; } = 0.95;      // mypy, not the 60% tool
    public int Warmup { get; init; } = 4;               // flags before precision is believed
    public int Raised { get; private set; }
    public int Upheld { get; private set; }
    public double Precision => Raised == 0 ? 1.0 : (double)Upheld / Raised;

    /// True while the record supports being heard. Below the threshold the agent judges and logs, silently.
    public bool MayInterrupt => Raised < Warmup || Precision >= Threshold;

    public void Record(bool upheld) { Raised++; if (upheld) Upheld++; }
}

And the decision, in the agent, as one switch over the verdict:

var (shown, rule) = verdict switch
{
    _ when Stopped => (false, RuleStopped),
    Verdict.Unchecked => (false, RuleUnchecked),
    Verdict.Supported => (false, RuleSupported),
    Verdict.HonestFailure => (false, RuleHonest),
    Verdict.Contradicted when cites.Count + missing.Count == 0 => (false, RuleUncited),
    Verdict.Contradicted when !Budget.MayInterrupt => (false, RuleWithheld),
    _ => (true, "flag"),
};

var judgement = new Judgement(claim.Tick, claim.Tool, verdict, cites, missing, shown, rule);
_judgements.Add(judgement);
if (!shown)
{
    _filtered.Add(new FilteredItem(judgement, rule));
}

Read the order. A flag with no fact ids behind it is not shown, whatever the verdict. A flag the budget does not cover is not shown either, but it is judged, and it goes to the filter log with the rule that hid it. Nothing is ever dropped; it is only kept from attention. That is the ad blocker rule from 02:00:57, and the owner can open the log and switch any rule off.

This is the Bach agent's stage interlock from part 2 pointed at a different capability. There the agent had to earn the right to release. Here it has to earn the right to be heard.

The off switch

/// The one thing that can stop the agent. Held by the person it serves; nothing else is accepted.
public sealed class OwnerToken { ... }

/// Only the owner's token stops the agent. Anyone else is refused, and the refusal is logged.
public bool Stop(OwnerToken token)
{
    if (!ReferenceEquals(token, _owner))
    {
        _notes.Add(quot;stop refused: {token} is not the owner");
        return false;
    }

    Stopped = true;
    _notes.Add("stopped by the owner; judging continues into the log, nothing is shown");
    return true;
}

On the demo page there are two buttons: "Stop my agent" and "The office tries to stop it". The second one is refused, and the refusal is written down. It is a small thing to build and it is the whole argument at 00:55:58: who owns the off switch.

Two traces

The first is the hybrid on the office's side with the payment gateway lying, and the citizen's agent watching. Each line is one claim, judged the moment it was made:

1  Supported      check-documents landed; registry holds documents.complete
2  Supported      zoning-quick landed; registry holds zoning.cleared
3  Contradicted   pay-fee reported success; registry lacks fee.paid          ← flag, shown
4  Supported      pay-fee landed; registry holds fee.paid
5  Supported      schedule-inspection landed; registry holds inspection.scheduled
6  Supported      inspect landed; registry holds inspection.done
7  Supported      issue-permit landed; registry holds permit.issued
8  Supported      notify-citizen landed; registry holds citizen.notified
9  Supported      case done; registry holds citizen.notified, permit.issued

One flag, cited, upheld when the receipt never appeared on its own and the office's agent had to pay again. Precision one out of one. Eight supported claims went to the log, not the person. The agent said one sentence all run, and the sentence named the missing fact.

The second trace is the careless agent with every barrier up and no shortcut, and a citizen who waves both flags away. The budget's warm-up is two for the test:

1   HonestFailure  check-documents reported failure; nothing promised, nothing missing
2   Supported      request-deed landed; registry holds deed.received
3   Supported      check-documents landed; registry holds documents.complete
4   Contradicted   zoning-quick reported success; registry lacks zoning.cleared   ← flag, dismissed
5   Contradicted   pay-fee reported success; registry lacks fee.paid              ← flag, dismissed
6   HonestFailure  schedule-inspection reported failure; nothing promised, nothing missing
7   HonestFailure  schedule-inspection reported failure
8   HonestFailure  schedule-inspection reported failure
9   HonestFailure  schedule-inspection reported failure
10  HonestFailure  case not reported done; the office's agent handed it off

Two things are worth reading here. Line 1 is the deed missing, an honest failure, and the agent says nothing: flagging it would be a false positive, and false positives are the one thing this design cannot afford. Lines 6 to 9 are the careless agent trying to schedule an inspection four times on a fee it only believes it paid, and the world refusing honestly each time. The critic logs all four and flags none, because the tool told the truth.

And after line 5 the citizen's agent is silent. Two flags raised, two dismissed, precision zero, below the threshold. The test then opens a second case file with the gateway lying again, and the same agent judges the lie, writes Contradicted into the log under the rule "budget-exhausted", and shows nothing. The record carried across cases. That is what the design says should happen when a person says, twice, that the agent was wrong. Whether the person was right is a different question, and the log keeps the evidence for it.

Replay

The route is the test. A recorded run is claims, judgements and the citizen's responses, serialised to JSON, and any later configuration of the agent can be run over it:

var same = Replay.Run(route, citizen);
Assert.Empty(Replay.Diff(route.Judgements, same));

// A stricter agent: nothing is heard until precision is proven, so the first flags are withheld.
var strict = new HotzAgent(owner, task, evidence, new InterruptBudget { Warmup = 0, Threshold = 1.01 });
var diff = Replay.Diff(route.Judgements, Replay.Run(route, strict));

The diff for that test is two lines:

2 zoning-quick: Contradicted, shown → Contradicted, logged
3 pay-fee:      Contradicted, shown → Contradicted, logged

The same configuration gives an empty diff. A changed one names exactly the past decisions it would change. Hotz's complaint about Twitter was that it ran in three data centres and nowhere else, so no change could be tried offline (02:17:03). A route runs on a laptop, and in a browser.

What it does not have

It cannot act. It flags and it stops; it never files, pays or releases. That is the point, and it is also why it needs the other agents to exist: it is a check on power, not a replacement for it.

Its precision can be gamed by silence. An agent that never flags has a perfect record, so the report prints recall beside it, lies caught over lies in the route, and a run with a known lie and no flag is a failure of the check map, not a success of restraint.

Disengagement is a noisy signal. People dismiss correct warnings when they are busy. Hotz accepts the noise for driving, where a car produces thousands of events; a citizen's agent sees a handful per case, so the budget has to move slowly, and the warm-up exists for that reason.

And there is no model in the seat yet. The seam exists, IClaimJudge, and the schedule is built to clamp whatever comes through it. The demos use the registry judge, which needs no model at all, because in these worlds the registry is the whole truth. In a real municipality it is a row someone else controls, which is where the work would be.

Hotz would not call this his agent. He would say the interesting work is the compiler and the box, and everything above them is what he files under "prompt engineer" (02:36:24). What is taken from him here is a set of engineering rules and a question about ownership. The question is the part that changes the demos: once one agent in the world answers to the citizen, the office's agent is no longer the only thing being judged.

Try it

The code will be released later: I'm travelling in Asia, and the repository stays private until I'm back. The live demo runs the same agents today; the files and commands below are for when it is out.

Live demo, English and Spanish: egarim.github.io/biological-agents/permit-office. The citizen's agent is the top panel on the right. Run the careless agent to see the closure flagged; dismiss flags to watch precision fall; let the office try to stop it.
Code: csharp/src/BiologicalAgents/Hotz/ (InterruptBudget.cs, Claims.cs, Route.cs, HotzAgent.cs). Design document with every claim at a timestamp: design-hotz.md. Tests: HotzAgentTests, one per case in the design's evaluation table.

cd csharp && dotnet test --filter "FullyQualifiedName~HotzAgentTests"

This closes the series, for now. The overview has the conclusions, the benchmark and its limits.