JJoche Ojeda.NET · XAF · AI · architecture
Aug 19, 20267 min read/2026/08/19/dont-fine-tune-your-xaf-model-yet/

Don't Fine-Tune Your XAF Model Yet: Measure the Free Rung First

XAFDevExpressLoRAFine-TuningSmall ModelsEvals

The last stretch of posts here all ended at the same place: a GPU. Distillation, LoRA, fine-tuning, poisoning — every one of them trained weights. That's the expensive end of a ladder, and I want to spend this post on the rung below it, because for a framework I actually build on, the cheap rung turned out to be enough.

The ladder, quickly

When a small or cheap model isn't good enough at a narrow task, there's a ladder of fixes, and it's ordered cheapest-first. I laid it out in an earlier post:

  1. Measure whether the cheap model is actually worse. Often it isn't.
  2. Route — cheap by default, expensive only where it's needed. A config change.
  3. Teach in-context — put the documentation or worked examples in the prompt. Still no training.
  4. Fine-tune — train weights. Days of work and a GPU bill.

The rule of a ladder is that you stop at the first rung that holds. Rung 4 is where all the interesting posts live, so it gets all the attention. Rung 3 is free and boring, so it gets skipped — and skipping it is how you end up paying for a training run you didn't need.

So I measured rung 3 on XAF, the DevExpress application framework I've built on for years. XAF is a good stress test precisely because it's narrow: it's thinly represented in public code, and it's full of house conventions a general model has to guess at. Base models hallucinate XAF APIs constantly. If in-context documentation fixes that, you've saved a fine-tune.

The setup

Two conditions, one model, same six tasks:

  • bare — the XAF coding question, alone.
  • skills — the same question, with the relevant DevExpress dx-xaf skill file pasted into the prompt ahead of it.

Model: mistral-small-3.2-24b-instruct — 24B, small enough to run locally on a 64 GB Mac, which is the whole point. Six tasks across the corners of XAF that trip models up: a ViewController with a SimpleAction, creating and committing an object through an IObjectSpace, filtering a CollectionSource, opening a DetailView in a popup, a validation rule, a role check.

Grading against reality, not against my opinion

Here's the part I care about, because a grader you haven't validated is a random number generator with a spreadsheet attached.

I did not judge whether the generated code was good. I counted whether the DevExpress methods it called actually exist — and "exist" is defined by your own codebases, not by me. I scanned 9,797 real XAF files on my machine and extracted every Receiver.Member call into an allowlist: 26,244 distinct pairs of "this object, this member." A call in the generated code that appears in zero of ten thousand real XAF files is a suspected hallucination.

That's an upper bound, not a precise count — a rare-but-valid API would show up as a false positive. But both conditions inherit the same bias, so the difference between them is the real measurement.

The result

Condition XAF API calls Suspected hallucinations Rate
bare 32 6 18.8%
skills 19 0 0.0%

Nearly one in five XAF calls from the bare model referenced something that appears in none of my real XAF code. With the documentation in context: none.

The honest confound, because the number is too clean to leave unqualified: the skills condition also wrote 21% less code — 19 API calls against 32. Fewer calls is fewer chances to hallucinate, so some of the improvement is just concision. It does not explain 18.8% → 0% — a lower rate on more calls would be the damning result, and that's not what happened — but it isn't nothing, and I'd rather name it than hope you don't notice.

The miss that matters

Aggregate rates are easy to wave away. The specific failure is what makes this real, and it's a beautiful one. The bare model wrote:

Application.ShowViewInPopupWindow(detailView, "Detail View");

Now — ShowViewInPopupWindow is a real method. I checked it against the official DevExpress documentation. It exists. It just doesn't live on Application; it lives on ShowViewStrategyBase, so the correct call is:

Application.ShowViewStrategy.ShowViewInPopupWindow(view);

The model took a real method and attached it to the wrong object, skipping .ShowViewStrategy.

Sit with why that's worse than an invented method. A hallucinated Application.DoTheThing() fails the moment you compile — loud, caught, gone. But Application.ShowViewInPopupWindow(...) reads correctly to anyone who half-remembers XAF. It's the right method and a plausible receiver. It survives a skim. It survives a code review by a tired human. It is the quiet kind of wrong I keep coming back to — the failure that doesn't announce itself.

With the skill file loaded, same task, correct receiver. The documentation didn't make the model smarter. It reminded it which object owns the method.

I didn't take the compiler's word for granted — I asked it

Documentation can be stale, and my allowlist is only as complete as the code I mined it from. So I settled it the one way that can't be argued with: I compiled both versions against the real DevExpress assemblies (XAF 26.1.4, .NET 10).

The skill-guided call:

Application.ShowViewStrategy.ShowViewInPopupWindow(view);
// Build succeeded. 0 Error(s)

The bare model's call:

Application.ShowViewInPopupWindow(view, "Detail View");
// error CS1061: 'XafApplication' does not contain a definition
// for 'ShowViewInPopupWindow'

That's the whole argument in two build logs. CS1061 is the compiler saying the exact thing the allowlist said and the docs said: the method is real, the receiver is wrong. The bare model produced code that a human reviewer might wave through and that the C# compiler rejects on the first pass — which, if you're going to be wrong, is at least the good way to be wrong. It's loud. The quiet version is the one that compiles and misbehaves at runtime, and a bigger model that had memorised the method name but not its owner is exactly how you'd get there.

What this means for XAF specifically

If you're pointing a local model at XAF and getting invented APIs, your first move is not a dataset and it is not a GPU. It's rung 3: put the framework's documentation in the context window and measure again. In my run that took the measurable hallucination rate to zero, and it's a config change — you can try it this afternoon and undo it before dinner.

Fine-tuning XAF into a model's weights is a real option, and I'd reach for it if in-context docs plateaued — if the failures were about judgment rather than lookup, the way I've argued elsewhere. But "which object owns this method" is a lookup problem, and lookup problems are what a document in the context window fixes for free. You don't distil a phone book into someone's memory when you can hand them the phone book.

Caveats

One model, six tasks, n=1. The allowlist is mined from real code but isn't the complete XAF surface, so the absolute rate is an upper bound. And the 21% length difference is a genuine partial confound. This is a signal worth acting on, not a benchmark to quote to three significant figures.

So

I spent a week writing about the expensive rung. Then I measured the free one on my own framework and it was enough.

The whole ladder exists so you climb only as high as the problem requires. XAF hallucinations looked like a capability gap that might justify training. They were a lookup gap that a paragraph of documentation closed. Measure the free rung before you rent the GPU — because the most expensive way to fix a lookup problem is to mistake it for an intelligence one.