Part two of a series on what it actually takes to build software solo, with AI. This part: prompt engineering and evals.
Prompting and evals may sound like magic or wizardry. They are neither. They work empirically: write down what good looks like, then test against it. Learn, adapt, inspect, iterate. That is the gist.
Prompt engineering and evals are the two things people most often rate as a mystery. Either you mutter the right incantations at the model, or you perform some arcane AI lab ritual. Both are wrong. Underneath sits the same judgement we have always used, very naturally, for every kind of work: write down what good looks like, then validate against it. That is the whole trick, and it is learnable. So let us dig in a little deeper and follow the path my own rabbit took to find its way into the hat.
The problem that started it (the carrot for the rabbit)
I am working on an app called Intentions. It aims to create thought-provoking insights for its users from a few lines someone writes in the morning, a short reflection at night, the same again over days and weeks, maybe with a few days skipped along the way. Breadcrumbs only to feed on, it seems, but my upfront research assured me there is a lot of treasure in that little: patterns, tension fields, preferences. And I was not short of ideas of my own about what an insight could surface and how it should read. All these projections have a fancy name: hypotheses. I ended up with thirty-nine of them. Not from the get-go, but over time, the more I thought and experimented. Yet a hypothesis is worth nothing until it is tested. It is only a wish, a carrot for the rabbit.
Where the prompt comes into play (the rabbit following the carrot)
I had the carrot. Now I needed the rabbit. I believed that with the right AI model and smart instructions I could derive meaningful insights from the little data and prove my hypotheses. Instructions for an AI model have their own fancy name: we call them prompts. And the prompts are not one-liners, as you can imagine. They are long and precise, as full of do's as of don'ts, and they carry examples of what good looks like.
So here was the bet I was making: if the prompt was good enough, the insights it gave people would actually help them grow. Their own daily inputs would become the mirror they needed to look into. If they could see themselves clearly, they would have a better chance of making conscious decisions each day. And over time, of becoming the people they always wanted to be, one step after another.
From this bet to a working product, the path starts with testing the fundamental hypotheses about what the prompt could do:
- Can it work when the input is thin, a few tired words and a typo or two.
- Can it stay grounded when the input is rich, without inventing things that were never there.
- And the one that mattered most: can it notice when someone is not in a reflective mood at all, but in real distress, and step back. In that case the right output is not a clever insight. It is a calm, preset message pointing the person toward proper support.
Whether an insight draws someone in or pushes them away also depends on the delivery, because different cultures value different degrees of directness and context. So a person should be able to choose how they want to hear from it, direct, balanced, or gentle. The model has to write in the register they picked, which is why the prompt carries three example texts of what each one sounds like, the same message in a different delivery, but always in the voice of a good friend rather than an analyst reading off a chart.
To test a model, you use a model (is the rabbit in the hat?)
Back to the thirty-nine hypotheses. How do I check whether they hold? I simply cannot eyeball every single run in any meaningful time. The answer may sound like a riddle: to test a model, you use a model. One model writes the insight, another judges it.
At this point you meet a new magic word, the rubric. Let us demystify it. The word comes from the Latin rubrica, meaning red ochre, the same red pigment scribes used to mark up texts. Long before medieval monks, Egyptian scribes wrote headings and notes in red, and in the manuscripts that followed, those red marks told the reader how the text should be read and understood. A rubric was an instruction for interpretation. That is more or less what it still is. In prompt assessment, a rubric sets out how to read and evaluate what a model produced from a prompt. It decides what counts as success and what counts as failure.
The eval is the test that holds an output against the rubric, and that is the second word. Some evals are black and white, a yes or no a line of code can settle. Did the distress hand-off fire when it should have, yes or no. Others cannot be settled by a checklist, like whether the gentle register actually feels gentle, or whether an insight sounds like a friend. For those you need a judge with taste, and the judge is another model. The field has a name for it, LLM-as-a-judge, and it is just another prompt: it takes the first model's output and the criteria, and reasons about whether they are met. To keep it honest, the judge is ideally a different model from the one on trial, so nobody is grading their own homework.
And to test the hypotheses, as you will have guessed, I built an AI-generated mock dataset based on personas and use cases. The thin inputs, the rich ones, the distress signals. It was not real human writing, but far better than not testing at all.
From hand-holding to a harness (a ladder for the rabbit to climb into the hat)
At first I ran all of this by hand in Claude Chat. It was the fastest way to find out whether the idea should be killed early. It held, so I kept going, and as the prompts grew, hand-holding every run drove me up the wall. Multiple sessions as the context window filled, handover prompts, copy-pasting, things lost in translation along the way. That is when I brought Claude Code in to build me a proper test harness, an agentic one. One go command, one result back within minutes instead of hours. Game-changing. The harness holds the prompts, the mock data, and the rubrics. It runs the whole suite with agents, writes the results to a file, and hands me back a synthesis of the key findings. I could give it an updated prompt and see exactly what moved. I could also take the results away and use them as the input for a root-cause analysis and an ideation session. Learn, adapt, inspect, iterate. The loop was finally at my fingertips.
One thing bugged me for a long time. Clunky, banned vocabulary kept leaking into the insights, despite all the sharpening, tweaking, and improving I did to the instructions. Then, finally, I traced it back to my own examples, the ones living inside the prompts. I had written a long list of forbidden words in all their variations, yet completely forgotten that the gold standards still contained them. I had set those texts down early and never really looked at them again. Rewriting them was the single biggest lever I found. There is a funny irony in role modelling being the thing that matters before everything else, in real life and in the world of LLMs alike.
Where the lab stops (and you bring the rabbit and the hat on stage)
Experimenting, thinking, iterating, I improved the results, but I never got all thirty-nine hypotheses to pass, and at some point I stopped the loop. I had the critical ones holding reliably enough: the distress hand-off, working with thin and typo-ridden text, the friend's voice, seeing both the tension and the opening a person is in.
Others I could only bring to a draw. The gentle register turned out to be more art than science. My read was that these models are trained largely on Western data, and a high-context "say it without saying it" is simply not their home turf. I could push them closer, but not all the way.
And this is where the lab ends. Mock data and evals carry you a long way, far enough to trust the basics and catch the dangerous misses. Yet the street still beats the lab, and I reached the point where I had to bring in real users, real people with their real words, their real feelings, their real reactions. So I decided to take Intentions to a closed beta. That is a story for another day.
In a company, building and running evals is work you hand to engineers. Building solo, it is a me-problem. I had to learn it, hands on, two steps forward, one back, and occasionally one sideways too. And that is the whole point. If you want to build with AI, you have to use AI, fearlessly, curiously, persistently. Today the rabbit and the hat, tomorrow the lady we'll saw in two. One trick at a time.
Written 21 June 2026. Intentions is in closed beta on iOS at the time of writing.