How I Learned to Stop Worrying and Love the AI Genie

As generative models grew more capable and it became clear the token subsidies couldn't last, I leapt from AI chat and code completions to having nine tenths of my code written by AI. This article captures why I'm worried and optimistic about AI. The need to write this also comes from the hyperbole and dramatic thoughts shared by others.

Are we talking about the same thing?

At its core a Large Language Model (LLM) is a deterministic formula: feed it the same prompt and it produces the exact same scores every time, one score for every possible next word. An LLM is probabilistic because the scores it produces are a spread of candidates with a probability ranking for each. Any randomness comes from a separate sampling step we wrap around the formula to draw one word from the spread; turn that step off (temperature 0) and the formula alone will, in principle, repeat itself. So it is a probabilistic model run by a deterministic formula, made non-deterministic in practice by the sampling we add.

The LLM agent is a pattern-matching machine and not a rational agent. A rational agent has stable preferences and beliefs, and the discipline to choose the action with maximal expected utility even when it is boring, non-linguistic, or unlike the examples in its training data. We should not confuse the simulation of deliberation with deliberation itself. The practical question is not whether the genie thinks, but where its imitation of thinking fails.

The nondeterminism is a feature and the simulation of deliberation is great, until it hallucinates and hands us slop. Kent Beck got it right when he likened the LLM agent to a slot machine and a "genie": you ask, and it delivers, but what it delivers is never quite what you meant.

The whole is greater than the sum of its parts

Access to the coding genie has raised your customers' expectations and amplified your competitors' velocity. The coding genie can also amplify your velocity, but what good will it do you to output subpar quality software faster? Software reaches customers through socio-technical systems: collaborative, iterative, and very much improvable. Improving requires treating the system as a whole, because a system is not the sum of its parts but a product of their interactions.

The AI genie is an amplifier that reveals existing dysfunction and rewards existing rigor. Here are some cases where, unless you address the system as a whole, the coding genie will fail to help and likely worsen the results:

• You don't know what features to build: the coding genie does not know your users and their needs. It is not customer- or product-obsessed, it has no understanding of your business principles and it will happily gloss over requirement ambiguities.
• You don't know how to build the feature: the coding genie is likely to output code of mediocre or subpar quality if you lack taste and experience to guide it.
• Lack of clear ownership: If your team is moving slow due to handovers and having to wait and coordinate with other teams, then the coding genie will increase the queues at the handover points.
• Messy models and tightly coupled domains: the coding genie will hallucinate domain meaning and patterns. Lacking architectural direction, it will amplify the mess and tangled dependencies.

I'm excited that rising token costs, demanding customers, and faster competitors will force organizations and software engineers to improve their software quality process. I'm hopeful to see investments in what Birgitta Böckeler calls Feedforward and feedback controls: guides on engineering principles, testing strategies, linters, static code analysis tools, logs, review agents, etc.

How I'm using the coding genie today

I've found value in Dr. Annegret Junkers' identification of three roles that AI can play well.

• Drafter: "produces a first-draft artifact. Domain Story, OpenAPI spec, service skeleton. Concrete enough to be wrong. The team reacts to it rather than starting from scratch." I consider this role to include the tutor role AI can play when it's explaining existing artifacts and concepts.
• Validator: "checks artifacts for internal consistency. Does the term in the new EventStorming match the Visual Glossary? Does the API enforce the Ubiquitous Language? Comparison work, at a scale where manual review is too expensive."
• Provocateur: "asks questions, surfaces assumptions, challenges decisions while they're still soft. Doesn't draft alternatives. Doesn't check consistency. Just makes the team implicitly defend choices they were about to make."

I put these three roles to work under the principles of the Agile Manifesto.

• Individuals and interactions over processes and tools:

  Software is delivered iteratively and collaboratively. An ADR should facilitate team comprehension and team consensus. No good can come from being alone with the machine for prolonged periods of time. The coding genie can augment the interactions with teammates and domain experts through its roles of drafter, validator or provocateur.

• Working software over comprehensive documentation:

  Before the coding genie, I was already doing what people now call Spec-driven development: minimal documentation of the situation, a few solution sketches, then TDD to discover and build the design incrementally. The coding genie, through its three roles, now runs alongside me: drafting the ADR with ASCII diagrams, pushing back on my reasoning, proposing a test for my approval, and writing the code to satisfy it. When I lack the theory to write the ADR, I set the genie aside and use TDD by hand to discover it, then update the ADR to reflect what I learned. I keep waterfall dead by working with small batches and short cycles.

• Customer collaboration over contract negotiation:

  A backlog story is a placeholder for a conversation. The coding genie is not customer-obsessed and it will happily gloss over requirement ambiguities, so I stay the customer proxy: I decide what is right, the genie never gets to define the need. What it does well is make collaboration cheaper by creating throwaway prototypes, challenging my assumptions, and surfacing ambiguities.

• Responding to change over following a plan:

  The coding genie has made it cheap to draft ADRs, tests, and code, and cheaper still to redraft them when reality proves those artifacts wrong or insufficient.

Optimistic and worried

I'm worried, because the powerful will use the genie to amplify their hold over us: to sell covert advertising, to choose who to bomb, who to track, where to patrol, and who to suspect. I'm optimistic, because communities can answer with counter-measures, and laws. We shouldn't allow anyone to shrug off accountability by hiding behind a Limited Liability Company or a Large Language Model.

I'm worried people surrender their thinking to the genie without caring where its simulation of intelligence fails. The coding genie cannot tackle essential complexity, and it reintroduces accidental complexity with every turn after. I'm optimistic this changes as we learn its limits, improve the harness with feedforward and feedback controls, and make human comprehension and training a first-class concern. Writing code without it recently, I felt my own skills atrophying, so I'm going to try out refactoring drills and coding katas.

I'm worried we are blurring the boundaries between roles, accepting more work,, context switching more often, and outputting more than we can review and comprehend. I'm optimistic that once the novelty wears off, organizations and individuals will resist the amplification of work with deliberate pauses and human interaction. I'm also optimistic because the coding genie has lowered the cost to writing contracts no one had time to read, tests no one had hours to write, and documentation that stayed in someone's head.

The genie is no silver bullet for the flow of value from backlog to customer, but it is a useful drafter, validator and provocateur. Our customers and our competitors are already asking how they can use the coding genie. The genie isn't going back in the bottle. Now is the time to experiment, to invest in the harness around it, and to improve the system as a whole for coding genie and humans alike.