How I Learned to Stop Worrying and Love the AI Genie
AI has me horrified, elated, excited, and worried, all at once. Our AI overlords remind us regularly that AGI is around the corner or that the only difference between you and Tony Stark is that he had unlimited API tokens. Some software developers have surrendered their thinking to AI. On the one hand, this reminds me of that scene where agent Smith told Morpheus "I say your civilization, because as soon as we started thinking for you it really became our civilization" On the other hand, perhaps those developers are producing software of similar or better quality. The AI genie is out of the bottle, and it can't be put back in. All we can do is understand how we have shaped it, and how it will shape us.
How have we shaped the AI genie?
The distinction between human and machine thinking is meaningless when the machine can imitate thinking well enough to fool humans. The practical question is not whether the genie thinks, but where its (imitation of) thinking fails. A little literacy in how these systems are trained and how they turn patterns into output goes a long way in breaking the spell and seeing the limits of the genie. Let us explore what happens when we ask it "How are you feeling today?".
Tokenization · step 1
Text is broken into tokens: fragments of words, each mapped to a number. This is the genie's alphabet. It does not see letters or words the way we do; it sees sequences of integers. Even punctuation gets its own token.
Self-attention · step 2
Each token weighs the others in context and asks, in effect: what matters here? In "How are you feeling today?", "feeling" is pulled toward "you" and "today". Meaning emerges from those weighted relationships across the context window: the fixed span of text the model can still take into account.
Next-token prediction · step 3
The genie is not a rational agent weighing options against a goal; it is predicting the next token. The output is not a single answer but a probability distribution over thousands of possible tokens, from which the genie samples — so the same prompt can produce different answers on different runs. Not what is true, not what is optimal for any goal, not what is felt, but what is statistically likely to come next. Asked how it feels, it does not consult an inner life. It completes a pattern.
Extended thinking · step 4
Some newer models linger before they answer. They spend extra inference time producing intermediate reasoning traces, which can improve performance, but do not change the underlying mechanism: prediction, repeated at scale.
We should judge the genie by the pattern-matching machine it is. Its patterns and configuration lean into our tendency to anthropomorphize and trust what sounds human. Its patterns cause it to reproduce the social biases of the human record it was trained on. When an experiment suggests that the genie, trained only on pre-Einstein material, has rediscovered relativity, we know to look harder at the framing, prompting, and judging. When some AI prophet tells us that the next model will be 6 to 10 times smarter, we should ask them to explain how this pattern-matching architecture would make that possible. No amount of prompt tweaking or AGENT.md tuning will change what the architecture is.
The genie only simulates rational agency and is not a "rational agent". The genie is not optimizing expected utility in any environment. When its visible reasoning trace says "I should compare the expected outcomes" and "the best action is probably x", we are watching a fluent simulation of deliberation, not deliberation itself. For narrow tasks, where the environment is constrained and the success criteria sit close to patterns seen in training, the imitation is good enough. For open-ended responsibility, it is an addictive slot machine: sometimes the output is fantastic, often it is dreck. The question for our profession is: what happens when you give this imitation machine to millions of software engineers, and to millions more who never wrote a line of code before?
How the AI genie will shape us
The AI genie is an amplifier
It amplifies power
Our technofeudal overlords will use the genie to tighten their grip on markets and on our economic lives. Amazon.com gives way to an AI companion whose loyalties lie with the company that owns it. In the arsenal of the powerful, commerce is only the soft edge. The genie will (help) decide who to bomb, who to track, where to patrol, and who to suspect: The offer is always the same: look faster, decide cheaper, let a probability stand in for a human who has to answer for the call. The genie offers a second place to send the blame. After the limited liability company, it is the second great construct to absorb human accountability, but this time what gets absorbed is judgment itself, delegated to a machine that cannot be called as a witness, cross-examined, or held to account when it is wrong.
It amplifies the work itself
With the AI genie we become faster not only at doing the work, but at accepting more of it. The cost of starting and revisiting a task drops, and the boundary around it dissolves: we take on work that would have waited, absorb work that belonged to someone else, and stay late because the next step is one prompt away. The pull is partly chemical, resembling the slot machine's addictiveness. What individual willpower cannot hold off, organizations must, through deliberate pauses and human grounding that keeps no one alone with the machine all day.
Some of this newly accepted work is genuinely valuable: contracts no one had time to read, tests no one had hours to write, documentation that stayed in someone's head. Lowering its cost is the most optimistic case for the genie, and a real one. The catch: these artifacts still need humans to judge them, and if cheap output outruns that judgment, we hollow out the expertise the genie was leaning on.
It amplifies engineering dysfunction
Consider your socio-technical system through which value flows from the backlog to the user. How efficient and effective this system is and its bottlenecks are a measure of your engineering culture and its rigor or dysfunction. The genie can amplify and speed up some parts of this system, but a system is not the sum of its parts: it's the product of their interaction. A future post will discuss several aspects of the system that can affect the flow, but here I want to highlight the role of ambiguity. Product , technical , and natural-language ambiguity were always with us when delivering value to the user. An effective and efficient flow requires tackling the ambiguity in a manner that fits the situation. The genie now compounds these ambiguities with its own pattern-matching probability and inviting teams to suspend their judgement in favor of its speed.
Roles, skills and productivity
Human in the loop
The genie can accelerate the work, but it cannot decide whether the work is worth doing. That still falls to a human: the clarity to know what is being built, the empathy to take the user's problem as your own, the accountability to answer when it breaks, and the foresight to weigh risks the training data has not yet seen. Organizations that were masking poor product clarity with slow delivery cycles now have nowhere to hide. Our socio-technical systems accelerated by the genie will outrun the comprehension of the people inside it. Keeping pace requires that we treat comprehension as a first-class output of the system. And we have to be careful about who in the organization gets to do the comprehending: the genie amplifies the engineers who already have systems taste and architectural intuition, while the juniors who would have grown into that role by struggling through the work risk being left behind.
For all the worry in the pages above, it remains an exciting time to be a software engineer. GitHub has had to revise its own capacity plans from a tenfold to a thirtyfold scale-up to keep pace with agentic development, and even after we set aside the share of it that will be vibe-coded slop, what is left is a genuine expansion of what our profession can attempt. Some of that work will solve problems for the people who use it, and a smaller share of it will outlive its first use, finding its way into the libraries, patterns, and shared vocabulary that move the profession forward. The genie is not going back into its bottle. The question that remains is how we work alongside it without surrendering the judgement, the craft, and the accountability that make software worth shipping. A follow-up piece will take up that question: how to improve the flow of value from idea to production so the genie has something worth amplifying.