Recently Kevin sent me a video from The AI Breakdown that summarized the community’s analysis of karpathy/autoresearch. Autoresearch’s idea was to use the coding agent loop with a prompt to tune the parameters for training an LLM. The community’s analysis was that the approach demonstrated a fundamental leap in that the agentic loop was now a primitive. Primitive here means that, much like the function call in imperative programming, invoking the loop sets off a process to autonomously get what the user desires. The AI Breakdown assumes that as long as you have the guidelines (the prompt) and the scoring (the goal/tests), the agentic loop gets you to the goal by continually grading the result against the prompt. It sounds reasonable, as you would expect the loop to smoothly ascend towards the goal.
In my opinion, almost everyone who writes or talks about this has dismissed decades of learnings from nonlinear and non-convex optimization.
The parallels are stark between this “agentic loop” and gradient descent, especially to anyone who’s worked on optimization:
- loop = step: single iteration in optimization
- prompt = loss function: the criteria evaluated against the current state (the code output)
- score/feedback = gradient: what comes from loss function to tell loop where to move next
- LLM = stochastic optimizer: the mechanism using the gradient to guess where to move
- goal = global optimum: target of what user is trying to create
The community is hellbent on believing that if you blindly go through the loop enough times, you will march steadily towards the true optimum. It’s an elegant but unfortunately naive solution. Anyone who’s worked on complex optimization knows that this approach is fraught with peril. Usually, you actually end up deeply stuck in a local minimum.
Agentic coding already has this local minimum concept: being stuck. The “forgetting factor” or reflection mechanisms in these loops try to act like more complex derivatives for descent (like momentum), which can help slightly but can’t truly get you out of deep local minima. Global optimization over a nonconvex landscape is NP-hard. Sure, techniques like annealing can help but those techniques require an enormous number of evaluations, which in the agentic loop is prohibitively expensive LLM inference. The approach is too blunt with too few priors or constraints to solve such highly complex optimization problems. The concept of a smoothly ascending “good score” does not actually exist for most problems, because the loss landscape of software engineering is fundamentally jagged and discontinuous.
Sometimes I think the real issue is that the artificial intelligence community is selectively forgetting decades-old institutional knowledge in statistics. We are rediscovering the wheel … and shouldn’t be surprised when the wheel comes out square. As a result, these agentic loops, left unattended, will pass the narrow harness tests but end up vomiting out brittle, local-minimum products that nobody wants, ultimately flooding and hurting users. These agentic loops are essentially automating a violation of Goodhart’s law: when the score is the target, the score - and the code - ceases to be good.