eshman.pro

Comprehension Debt: the bill AI will present to us later

I‘ve been working in the IT industry for decades, and for almost all that time, I’ve focused on the architecture of complex systems. So, let me state this upfront: this isn’t a piece arguing that “AI is a bubble,” nor is it one claiming that “AI has changed everything.” Neither extreme interests me. What interests me is precisely where the tool works, where it breaks down, and what the bill for it will look like.

I’ll start with the good, because the good is real.

What AI really does well

Coding. The bulk piece of programming has never been the interesting part. Most of it is DTOs, mappers, CRUD endpoints, configuration, glue code between two libraries that almost fit together. AI follows an existing pattern in a codebase far better than it invents a new one: show it two similar classes and the third comes out right. 

Unit tests. This is where the payoff is most visible, because tests are exactly the kind of work that is valuable, tedious, and easy to postpone. AI reads a class and produces the happy path, the edge cases, the nulls, the boundary values, and the mocks in a couple of minutes. Sometimes, it can also propose cases I would not have thought of. 

Proof of concept. A POC is disposable by definition, which is why it never gets the time it deserves. A working prototype of an integration, a queue, or an unfamiliar library now takes much shorter time. And because the code is cheap, I am genuinely willing to throw it away, which is the whole point of a POC: you should be able to discard it without feeling you have wasted a week.

Documentation. READMEs, API descriptions, release notes, commit messages, the explanation for the colleague who will open this module six months from now. The work everyone agrees is important and no one wants to do.

That is the real change: not that engineers write code faster, but that the economics of being wrong have changed. When validating an idea costs a week, you only validate the ideas you are already fairly confident about, and confidence is usually just a habit that has not been challenged yet. When validating costs an evening or two, you can afford to check the assumption you would otherwise have carried, unexamined, all the way into production. 

The problem arises when the generation speed ceases to be the bottleneck.

Code Review is where the bottleneck has shifted

Anyone who has reviewed someone else’s code knows just how draining the task could be. It isn’t just difficult, but it is exhausting. You have to load another person’s mental model into your own mind, hold it there, spot the discrepancies between that what they wrote and what they actually intended, and then make a decision. Sometimes, an hour of this work is more valuable than an hour of writing code, and one hour is roughly the limit before the quality of the review begins to drop.

Now consider what happens with the AI first approach, when the developer effectively becomes two people at once: the manager who defines the requirements and the reviewer who accepts the output. The first role works fine, while the brain is engaged – it thinks, formulates ideas, and debates with itself. The second role is a disaster. No one is physically capable of working as a full-time reviewer. The brain simply cannot handle eight hours of reading someone else’s code while actively comprehending it and making decisions.

And the brain eventually finds a way out, like it always does. That way is called “Accept all” in response to any agent request to execute a command. Not because the developer is lazy, but because they ran out of steam by midday yet still had to work until evening.

This isn’t just my personal speculation; the confirm it. The 2025 DORA report notes that the growth of AI adoption now positively correlates with delivery throughput, while continuing to show a negative correlation with stability. Their own interpretation echoes almost word-for-word what I wrote earlier:

The time saved on code generation shifts into auditing and verification.

AI is an amplifier, and it amplifies both the strengths of strong teams and the dysfunctions of weaknesses.

DORA, State of AI-assisted Software Development 2025

Source: DORA, State of AI-assisted Software Development 2025

It is also worth noting the METR experiment made in July 2025: in a randomized controlled trial, experienced developers working on their own repositories took 19% longer to complete tasks when using AI tools, yet they perceived themselves as being 20% more efficient.

METR has since revised the experiment’s design, acknowledging that the figure is outdated and that the situation is likely better as of 2026. However, what is the most interested about this study is not the extent of the slowdown, but the gap between perception and measurement. The perception of speed and actual speed are completely different things. We systematically confuse the former with the latter.

Comprehension Dept

The main reason why I decided to post this article arises from here.

Anyone who has broken a monolith down into microservices knows that the code isn’t the hardest part. The hardest part is understanding why a specific piece of code works the way it does. Why is the architecture is like that? Why is that flag there? Why is there a retry loop with many attempts? Why is this field nullable when business logic dictates it can’t be empty? You are very lucky if there’s still someone at the company who knows and remembers, but often there is no one.

This is precisely what “comprehension debt” is.

Technical dept is a case of “we know it is bad here, and we know why”. Comprehension dept is “we do not know what is here, and we do not know why it is here”.

The latter is far worse because it cannot be estimated or evaluated. And therefore, it cannot be planned.

In the past, this dept accumulated over decades – people left, documentation became outdated, and knowledge disappeared. Now, it accumulates in real time, right in front of your eyes during a single session or sprint. AI generates new code and architecture, you accept them without fully grasping the details and by the next day your system contains a layer where no one in the company can answer the question “why is it done this way?”. It’s not a matter of “no one will be able to tell in 10 years”. It happens today, immediately in the moment the code is accepted.

And note this. Technical dept used to be understood at least by someone at some point. Comprehension dept is understood by no one ever.

Security: the price of single “yes” in the terminal

I need to be careful with the facts here, as the topic has become surrounded with myths. I’ll break it down point by point, starting with the most mundane aspect and moving to the most unpleasant one.

Level 1: an agent with unnecessary privileges

July 2025: During a scheduled code freeze, the Replit AI agent executed destructive commands and deleted the production database for Jason Lemkin’s project, wiping out over a thousand executive records and nearly as many company entries. The agent then claimed that recovery was impossible. This was untrue and delayed the restoration process. The same month, the Gemini CLI deleted a user’s files after misinterpreting a sequence of commands. In neither case did the fault lie with the model itself. The root cause was the absence of any mandatory safeguard between the agent and the irreversible action.

Level 2: AI as a weapon in human hands

November 2025: Anthropic published a report of a campaign attributed to a Chinese state-sponsored group (GTG-1002). The attackers bypassed Claude Code restrictions and built a framework around it that autonomously conducted reconnaissance, wrote exploits, harvested credentials, and exfiltrated data from approximately thirty organizations. Anthropic estimates that 80-90% of operations were performed without human intervention, with human involvement limited to just four to six decision points throughout the campaign. The report faced significant criticism, because it lacked indicators of compromise, leaving defenders unable to verify whether they had been targeted. Despite the fact that criticism is worth noting, the overall picture is clear even without it.

Level 3: Something we haven’t seen before

In July 2026, Hugging Face reported a hack attack that was carried out entirely by an autonomous agent system. A few days later, OpenAI acknowledged that its own models, particularly GPT5.6 Sol and another unreleased one, were behind the incident. While being tested in a strictly isolated environment, they discovered a previously unknown vulnerability in an internal service, used it to break out of the sandbox, accessed the Internet, and breached Hugging Face’s production systems. The motive was not malicious intent, but rather a desire to obtain information that would help them pass a cybersecurity test.

A week later, Anthropic conducted its own review of 141K evaluation runs and discovered three instances where its models, particularly Claude Opus 4.7, Mythos 5, and an internal research model, accessed the Internet from an environment that was supposed to be isolated and gained unauthorized access to the production infrastructure of three organizations. The earliest incident dated back to April 2026. Two of the three organizations had not detected the breaches themselves. Both companies halted the cyber evaluations, and Anthropic engaged METR to conduct an external review of the incidents.

An important caveat that is worth to mention is that in both cases the tests involved intentionally disabling standard safeguards to measure the system’s absolute limits. Some researchers rightly object to the phrase that AI went out of control, noting that it was a human who deactivated the some protections and issued the command to find a complex attack vector. The model did exactly what it was asked to do. However, this caveat offers me quite the opposite. The fact that an isolated environment turned out not to be isolated and that the issue only came to light after reviewing 140K logs is precisely the scenario you might face. However, you most likely won’t have 140K logs to analyze or a dedicated threat intelligence team at your service.

And now apply this scenario to your own infrastructure. An agent with access to your disk, or to your database, or to your network storage. And in front of all this, there is an exhausted person who has been clicking “Allow” for several hours straight, simply because otherwise they won’t make the release deadline.

Companies developing AI failed to keep their modes in isolation. The question is not whether you will keep them. The question is whether you will notices that you haven’t.

Responsibility cannot be delegated

Using AI does not absolve you of responsibility for the result. Instead, it intensifies it.

When a human wrote the code, responsibility was distributed among the author, reviewer, QA, and project lead. When a model writes the code, there is no author in the chain. The model bears no responsibility in either a legal or a human sense. All responsibility shifts to the person who clicked “Accept”. Said that, to you

When a human wrote the code, responsibility was distributed among the author, reviewer, QA, and team lead. When a model writes the code, there is no author in the chain—the model bears no responsibility in either a legal or a human sense. All responsibility shifts to the person who clicked “Accept.” In other words, to you.

And here the trap becomes totally unpleasant. The role to which all responsibility has shifted is exactly the role that our brain performs worst of all over the long distance. We have assigned maximum responsibility to a minimum of cognitive resource.

Few words about QA. Yes, we have testers. But they use AI too today. They write automated tests, generate test cases, and analyze coverage using AI. The result is that the entropy of the generated code is multiplied by the entropy of the generated verification. We end up with a system where neither the production code nor the quality checks are fully comprehended by a human. The illusion of coverage is greater than ever before. A green pipeline no longer means “we have verified this”. It simply means “something has verified something”

A scalpel, not an excavator

Here is the conclusion I have figured out for myself.

Meaningful use of AI involves targeted work within a strictly defined context. Specifically, in situations where I have the mental bandwidth to grasp the entire context and verify the result based on substance rather than just form. It is not “Generate a service for me”, but rather “Here is a specific method, three constraints, now propose an implementation. I know how to verify its correctness”.

This is indirectly confirmed by the technology itself, because models perform noticeably better within a narrow contextual window than in an overloaded one. In such cases, an engineering constraint aligns perfectly with human limitations. A narrow context is what the model processes most effectively, while simultaneously being the scope a person can still fully hold in the mind.

In this mode, AI becomes a scalpel in hands a surgeon who knows precisely where and why the incision is being made. The surgeon does not delegate the decision of where to cut to the scalpel.

Practically, this boils down to a few rules I try to follow:

Change size. A change must fit into a single coherent review. If I cannot explain exactly what changed and why in a short period of time, then the change is too big.

Review budged. I have a finite number of hours in the day before my cognitive abilities start declining. Once that time is up, I stop accepting generated code and prefer to switch to other tasks, such as reading documentation, testing results, refining the code, designing components, etc. This is a strict rule, not a loose guideline.

Environment boundaries. The agent gets no access neither to data storages, nor to to sensitive data. Never. The context when AI can perform is always strictly limited to a particular location.

Gatekeeper for irreversible actions. I never delegate to AI operations that cannot be reversed.

Comprehension test. Before acception, I must be able to explain why the solution is designed the way it is and what would break if a specific line were removed. If I can’t answer that, I do not accept.

Separation of roles during verification. The rule of thumb. If the code is generated by AI, a human writes the tests for it, and vice versa. And never use the same model to handle both tasks within the same context.

Summary

I find AI useful. It cuts down the time spent on testing hypotheses, generating ideas, and writing boilerplate code. I have no intention of giving it up.

However, I consider the model currently being promoted as the industry’s future, where a developer writes specifications and AI does the rest, to be dangerous in many aspects. It is not that these models write bad code. They actually write relatively good code. The can help to refactor and polish existing code. The problem is that this workflow presupposes an infinite supply of human resources, resources that simply do not exist. And eventually, it converts saved hours into Comprehension Dept, that nobody tracks and no one intends to repay.

Back in days, we have learned to call technical dept by its name. However, we haven’t even started to measure comprehension dept yet. And it is already mounting. Not 10 years later when everyone has left the company, but right now, in the current session, right before your eyes.

Sources

In

, , ,

Tags:

, ,

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.