Most projects I get to work on lately are in one way or another related to LLMs, and the same request keeps coming back. A company wants to know whether their internal policies are ISO 27001 compliant. There’s a pile of requirements on one side and a pile of policy documents on the other, and someone has to say yes or no for each pair.
A naive solution, to try out, would be to make a vector database, where the rules are embedded. At runtime you pass the input documents, it queries the vector database, follows the prompt and outputs a decision. This system can be improved by choosing better chunking, better retrieval or a bigger model. It’s all context engineering.
But, look at what didn’t change. The model still makes a call. You are still the one improving it, reading failures and adding them to the prompt. You could put a metric on this, tune against it, and get it better. But you’d still be shipping a model, and a prompt nobody can read.
In my case, I couldn’t even write a better prompt, because that would mean that I would need to know the rules myself.
So, I took a good look at the prompt, the rules in documents and figured that most of them are blatant if else statements. The rules being written down doesn’t help either: knowing a rule exists isn’t the same as knowing which rule applies to a given input.
But if it’s if else, then it’s code. And code I can test.
So I stopped asking the model to make the decision, and asked it to write the program that makes the decision. Andrej Karpathy coined a term for this loop, autoresearch.
For this you need a set of inputs, the answers you know are correct, and a function that scores results and indicates improvement. Then the agent writes some code, runs it, sees what failed, and writes the next one.
I left mine running for an evening. And what it wrote had an interesting shape, but kinda predictable in hindsight. First a parser, then the logic on top.
And take a look at the split it made. For ISO 27001, the parser pulls fields like control ID, owner, and review date out of the policy text, and the logic on top checks whether every control has an owner and was reviewed in time. The only reason I needed a LLM model here at all was that the input is text. If the inputs were already structured, code or ML would have done just fine. Sometimes the documents are messy, so you could consider having an LLM as the parser (but it’s pulling out fields, not deciding anything).
And now I can score the two halves separately. Did it read the document right, and do the rules hold.
What I get out of this is code. I can read it. I can reason through it. It caught rules that neither I nor the domain expert had spotted. It runs in milliseconds with no API calls. New regulation, something missing? I add the examples, add them to the metric, and run it again.
To wrap up, for this to work you need inputs, labels, and a metric. That’s the entry condition, and it’s less rare than it looks. You probably already have the eval set. Any process that has to justify its decisions leaves a record of it: what was decided, and why. The what is your label. The why tells you which fields the parser has to get right, the same split you already scored separately. Nobody collected it as training data, but that’s what it is. Same shape for support triage, code review, claims processing, underwriting, moderation. If that log genuinely doesn’t exist, label two hundred examples by hand. It takes an afternoon, and it’s worth it.
Your LLM shouldn’t be in production making the decision. Use it to write the thing that does. It turns the model into an optimizer that can reason about which direction to steer, and then you ship code, not an LLM.