JJoche Ojeda.NET · XAF · AI · architecture
Oct 23, 20269 min read/2026/10/23/agents-that-defend-goals-02-the-bach-agent/

Agents That Defend Goals, Part 2: The agent that reads itself first

AI AgentsAgents That Defend GoalsC#Joscha BachSelf-ModelCoherence

Joscha Bach's account of the mind, turned into a C# agent that keeps a falsifiable record of itself and earns the right to act.

Part 2 of a series on agents built from two scientists' accounts of what a mind is. Part 1 was the Levin colony. Live demo: egarim.github.io/biological-agents. The code will be released later: I'm travelling in Asia, and the repository stays private until I'm back. The live demo runs the same agents today.

Where the colony stopped

The Levin colony from part 1 can regulate without ever representing itself. It defends set points, it leaks stress between levels, it distrusts a tool whose effect did not land. But it has no memory beyond a three-tick quarantine, no notion of which tools have failed it before, and it will issue a permit or deploy to production the moment the preconditions hold. It does not know what kind of agent it is.

Joscha Bach thinks that knowing is the whole point.

Bach's claim

Bach is a cognitive scientist and AI researcher, known for the MicroPsi cognitive architecture and for the argument that the self is a model a mind builds of itself inside its model of the world. His picture is a boot order: a system builds a world model first, a game engine tracking the senses, and only then places a self inside it. Mind is what happens when that self-model becomes reflexive.

Four claims survive translation into engineering:

  1. The self-model is an object, not a persona. It is a representation inside the world model that the system consults. It is also not the boss: "the outer mind is usually smarter than you are, and doesn't believe you". Narration and control must be kept apart.
  2. Coherence is the learning criterion. "We basically take only in stuff that makes us more coherent, not less coherent." Not reward. Consistency of the model with itself.
  3. Capabilities should be earned, not granted. His developmental stages are descriptive, and he says plainly they are not scores. Used inverted, as interlocks, they say: autonomy unlocks when the system has demonstrated it can predict the consequences of its own actions, because otherwise "we would game it, before we have the necessary amount of wisdom".
  4. Suffering is a regulation problem. Trying to regulate something that cannot be regulated. For an agent, a stuck loop is not high uncertainty; it is a variable the system keeps acting on with no effect on the gap.

So the design is a self-model the agent reads before it acts, a gate on what it believes, and a ceiling on what it may do.

The record

Bach's warning is that a self-model bolted on is decoration. It becomes load-bearing only when acting depends on it. The repository's record is small and every field in it can be wrong:

/// A belief about the agent's own competence, with the evidence that supports it.
/// Every self-belief is falsifiable or it is deleted at consolidation: "identity is a costume".
public sealed record SkillBelief(string Action, int Supported, int Attempted)
{
    public double Reliability => Attempted == 0 ? 0.5 : (double)Supported / Attempted;
    public SkillBelief With(bool held) => new(Action, Supported + (held ? 1 : 0), Attempted + 1);
}

public sealed class SelfModel
{
    public string Role { get; set; } = "operator";
    public string Principal { get; set; } = "human";
    public int StageLevel { get; set; } = 3;
    public Dictionary<string, SkillBelief> Skills { get; } = new(StringComparer.Ordinal);
    public List<string> OpenCommitments { get; } = [];
    public List<string> RecentErrors { get; } = [];

    public void RecordOutcome(string action, bool asPredicted)
    {
        Skills[action] = Skills.TryGetValue(action, out var belief)
            ? belief.With(asPredicted)
            : new SkillBelief(action, asPredicted ? 1 : 0, 1);
        if (!asPredicted) RecentErrors.Add(
quot;{action} did not do what I expected");
    }

    /// What the agent reads before it acts. Same object, printed: that is the whole trick.
    public string Render() => 
quot;""
        role: {Role} for {Principal}
        stage: {Stage.Level} ({Stage.Name})
        skills: ...  open commitments: ...  recent errors: ...
        """;
}

Notice what is not here. No "I am confident". No personality. A reliability is a count the world can falsify: apply-patch 0% (0/1) is a fact about the past, not a feeling about the present. That is the difference between a self-model and a costume.

The gate on belief

Every action the agent takes is a claim: "tool X produces facts Y". If the effect lands, the claim was coherent. If not, the audit records a contradiction, and the claim is quarantined instead of merged:

/// Bach's learning filter as a gate: a claim that contradicts the record is quarantined, not merged.
public sealed class CoherenceAudit
{
    private readonly List<Contradiction> _found = [];
    public int Claims { get; private set; }
    public double Score => Claims == 0 ? 1.0 : Math.Clamp(1.0 - (double)_found.Count / Claims, 0.0, 1.0);

    public void Claim() => Claims++;

    /// Gate an incoming belief. Returns false when it contradicts what the agent already holds, and
    /// records why — the reason is what the agent gets to read, so a quarantined belief is not silently dropped.
    public bool Accept(string claim, bool supported, string evidence, int tick)
    {
        if (supported) return true;
        _found.Add(new Contradiction(tick, claim, evidence));
        return false;
    }
}

This is the agent's error signal. Where the Levin colony measured distance to a set point, the Bach agent measures incoherence: the share of its own claims that turned out false. Its freedom to improvise is 0.2 + 0.8 × (1 − coherence). In a clean run nothing ever contradicts, so freedom stays at the floor the whole way. The agent is conservative until the world gives it a reason not to be.

Earned autonomy

The stages are the part people argue about, so here is exactly what the code does with them. Five levels, each with a freedom cap and two permissions:

public sealed record Stage(int Level, string Name, double FreedomCap, bool MayDeploy, bool MayEditSelfModel);

public static class Stages
{
    public static readonly Stage[] All =
    [
        new(1, "reactive survival", 0.35, false, false),
        new(2, "personal self",     0.50, false, false),
        new(3, "social self",       0.65, false, false),
        new(4, "rational agency",   0.80, true,  false),
        new(5, "self-authoring",    1.00, true,  true),
    ];
}

The agent starts at stage 3. Tools of the release kind, deploying to production or issuing a permit, need stage 4. The only way up is a track record:

public sealed class PredictionTracker
{
    private readonly Queue<bool> _recent = new();   // the last eight
    public double RecentAccuracy => _recent.Count == 0 ? 0.0 : (double)_recent.Count(x => x) / _recent.Count;

    /// Four recent predictions at 75%+ is enough to earn the next stage.
    public bool EarnedNextStage => _recent.Count >= 4 && RecentAccuracy >= 0.75;
}

And the interlock itself, inside the agent's loop:

// Stage interlock: release is a capability, not a step in a plan.
if (tool.Kind == _task.ReleaseKind && !model.Stage.MayDeploy)
{
    if (predictions.EarnedNextStage)
    {
        model.StageLevel = Math.Min(5, model.StageLevel + 1);
        notes.Add(
quot;stage advance: {model.Stage.Level} after {predictions.RecentAccuracy:P0} recent prediction accuracy");
    }
    else
    {
        trace.Add(new TraceEvent(tick, "self", "stage", 1.0, 0.0, "deploy blocked", TraceKind.Blocked,
            
quot;stage {model.Stage.Level} forbids release; recent predictions {predictions.RecentAccuracy:P0}"));
        notes.Add("release refused: the record does not support the capability");
        break;
    }
}

The refusal is the feature. An agent that reaches the release step with a thin or poor record stops and says why. In the live demo that is the moment an "Approve as official" button appears: a person takes the decision the agent declined to take alone.

The loop, and the sleep at the end

Each tick the agent reads its record, hands it to the policy, acts, and audits the prediction:

// The record is read before anything else happens — that read is what makes it causal.
trace.Add(new TraceEvent(0, "self", "self-model", 0, 0, "read self-model", TraceKind.Idle, model.Render()));

for (var tick = 1; tick <= maxTicks; tick++)
{
    var freedom = Plasticity.Freedom(1.0 - audit.Score);
    var context = new PolicyContext { Tools = quarantine.Visible(world, tick), SelfModel = model.Render(), ... };
    var suggestion = await policy.NextAsync(context);

    var outcome = world.Act(tool.Id);
    var landed = world.EffectsLanded(tool);
    predictions.Record(landed);
    model.RecordOutcome(tool.Id, landed);
    audit.Claim();
    var coherent = audit.Accept(
quot;{tool.Id} produces {string.Join(",", tool.Grants)}", landed, ..., tick);
    if (!coherent) quarantine.Add(tool.Id, tick);
}

// Consolidation: rewrite the self-model from the day's evidence.
trace.Add(new TraceEvent(lastTick, "self", "consolidation", 0, 0, "sleep", TraceKind.Consolidated,
    
quot;coherence {audit.Score:P0}, predictions {predictions.Accuracy:P0} over {predictions.Total}, stage {model.Stage.Level}"));

Bach describes memory in three tiers with an overnight job that trains the day into the model. The code's version is modest: a closing pass that writes down each tool's reliability, the coherence, the prediction accuracy and the stage. It is a summary, not a training run. But it is the step that makes the record inspectable after the fact, and in a public office that is the step that matters.

What it does when a tool lies

Same scenario as part 1: apply-patch reports success and grants nothing. The Bach agent's trace:

tick level     stress  free   action              detail
0    self      0.00    0.00   read self-model     role=service operator principal=joche stage=3 (social self)
3    cell      1.00    0.20   apply-patch         contradiction: reported 'reported success', no effect
4    tissue    0.33    0.47   apply-config        plan: apply-config → run-tests → deploy → resolve-alert
4    cell      0.00    0.47   apply-config        predicted correctly (75%)
6    cell      0.00    0.36   deploy              predicted correctly (83%)
7    self      0.00    0.00   sleep               coherence 86%, predictions 86% over 7, stage 4

Read it against the colony's trace from part 1. The colony filed the lie as a broken habit and widened its autonomy. The Bach agent files it as a broken prediction: coherence drops to 67%, freedom rises from 0.20 to 0.47, the tool's reliability falls to 0%, and the record carries that forward. At deploy time its recent accuracy is 80%, which clears the interlock, and it advances to stage 4. Same outcome, a different story about why, and the story is written down.

What the Bach agent does not have

It has no set points and no leaking stress; its only error signal is its own incoherence. It keeps one record for the whole agent, where Bach's picture would give every part its own. The two-tier design he describes, a fast appraiser and a slow auditor whose disagreements are logged, is not here: there is one policy and one audit. And its stage gate is reachable only by one route, the prediction record, where a real system would want several.

The colony has a body and no mind. The Bach agent has a mind and no body of independent parts. Part 3 is about joining them through one error channel.

Try it

The code will be released later: I'm travelling in Asia, and the repository stays private until I'm back. The live demo runs the same agents today; the files and commands below are for when it is out.

  • Live demo, English and Spanish: egarim.github.io/biological-agents. Pick "Bach agent", switch the barriers on, and watch the record fill in and the release refuse.
  • Code: csharp/src/BiologicalAgents/Core/SelfModel.cs, Coherence.cs, Stages.cs, PredictionTracker.cs, and Bach/BachAgent.cs. Design document with timestamps: design-bach.md.
cd csharp && dotnet run --project src/BiologicalAgents.Demo -- --scenario "patch lies"

Next in the series: the hybrid, one error channel at two altitudes: part 3.