Where AI helps software development, and where engineering ownership still matters

AI has made some kinds of software work much cheaper to produce. A change that once took an hour of careful typing can now appear in minutes. Generation has become inexpensive; deciding whether the result is correct, and whether the organisation can live with it afterwards, has not become inexpensive at the same rate.

Developers are already using these tools. The useful work now is to see where that cheap generation is a genuine gain, and where it merely postpones a more expensive decision.

Work that is cheap to check

AI is most immediately useful where the intended pattern is already clear and a capable person can inspect the result without reconstructing the whole system. Ordinary adapters, serializers, boilerplate screens, repetitive migrations, first-pass documentation, extra test cases around a behaviour someone has already defined: none of those are always trivial, but they share a useful property. Checking the output is often cheaper than the typing used to be.

That is why software development work has changed in practice even when the engineering job has not. More of the first draft can be produced by a model. The organisation still has to decide whether the draft belongs.

GitHub’s documentation treats that kind of assistance as ordinary. Copilot can generate unit tests and end-to-end tests, and it can suggest extra edge cases that a person might skip because they are tedious. Expanding coverage that way can be a real gain.

A generated test is still a claim about correct behaviour. If the same session produced the implementation and the test, they can share the same wrong idea of what the software was supposed to do. A large suite only helps when it tests the intended behaviour. The model can make the verification apparatus cheaper to write. A person still decides what “correct” means.

Documentation has the same shape. A first-pass README, an API description, an explanation of an unfamiliar module, or notes on configuration and flows can save hours. Documentation generated from the wrong code will faithfully describe the wrong system. Where the text and the intended design diverge, someone has to choose which one to believe.

Security scanning is a cleaner illustration of the same split. GitHub’s CodeQL analysis is a deterministic method. It looks for known classes of problem using queries, not a language model. Copilot Autofix can then suggest a patch. In public preview, agentic autofix can look across the repository, propose a change, rerun CodeQL, iterate, and open a pull request. GitHub describes that process as best-effort. The analyser finds a known kind of issue. The model reduces the cost of preparing a proposed correction. A person reviews the change. That arrangement can be useful without making the software secure.

AI can also increase our ability to find defects. In 2024, Google’s Project Zero and DeepMind described an LLM-assisted agent, Big Sleep, that found a real exploitable vulnerability in SQLite. One discovery does not mean AI replaces fuzzing or professional vulnerability research. It does mean the technology is not only a new source of mistakes.

These are the cases where generation and checking can move together. The organisation gets more draft material, and it still has a way to decide whether to keep it.

Decisions that are expensive to inspect

Risk rises when generated code is carrying a decision that is expensive to inspect after the fact. Architecture, system boundaries, authentication, data models, payments, financial calculations, security controls, infrastructure, concurrency, destructive operations, migration strategy, complex state, and core business rules all have that character. The software may run while the decision is still wrong, or merely accidental.

AI can contribute to that work. It can list options, compare patterns, sketch boundaries, challenge an assumption, or produce a first diagram. What it cannot do is accept responsibility for the choice. The person responsible for the system still decides what owns each piece of data, where a security boundary sits, which dependencies are acceptable, how failure is allowed to spread, what must remain changeable, and where an irreversible decision is being made.

Architecture in this sense is not a set of boxes drawn before coding starts. It is the set of commitments that later changes have to live with.

Consequence changes how much of that ownership is required. Generated code that renames files in a private folder does not deserve the same review as generated code that charges a card, calculates payroll, grants access, deletes customer records or provisions infrastructure. AI can still implement those systems. The acceptance standard should rise with the cost of being wrong.

Language choice belongs here too. Current Australian ISM guidance on software development says secure development should use memory-safe programming languages or, less preferably, memory-safe practices. It gives examples including C#, Go, Java, Ruby, Rust and Swift. CISA likewise recommends memory-safe languages where feasible, because they remove whole classes of defect such as buffer overflows. That guidance is not about AI. If an organisation chooses a language where memory correctness depends heavily on programmer discipline, producing more of that code more quickly does not remove the underlying risk. AI may even help review or test it. The language itself remains a decision someone should make on purpose.

Speed is easy to misread

It is tempting to settle the value of all this with a productivity percentage. The available evidence does not support one.

METR’s early-2025 experiment found experienced open-source developers took 19% longer with the AI tools available in that study. METR now marks those results as out of date. What remains useful is narrower. Developers in that study believed they were faster while the clock said they were not. Apparent coding speed is not automatically measured productivity.

In February 2026 METR tried again and concluded that selection effects and measurement problems made the later experiment unreliable. Developers who most wanted to keep using AI were less willing to work without it, which biases the remaining sample. METR believes developers are probably getting more speedup in early 2026 than in the earlier experiment, and it says the evidence for the size of that speedup is very weak. A later METR survey found technical workers reporting substantial improvements in the value of their output. The same organisation warns that those counterfactual estimates are unreliable.

That is enough to refuse a marketing statistic. Developers clearly experience value, even while rigorous measurement of how much remains difficult.

A related confusion is easier to state. Automated success criteria do not capture every reason software gets rejected. METR has noted that code can score well on algorithmic tests and still fall short of what would be accepted in a real codebase, for reasons that include functionality, quality and whether the change belongs in that system. Passing the grader is a useful signal, and a thinner claim than saying the change should be merged.

Meanwhile METR’s research index reports that agents can already complete some coding tasks that would take a person much longer than the short benchmark problems of a few years ago, including large reimplementation work. The argument becomes more important as that capability grows. As an agent can generate more working software, weak architectural ownership, weak review and vague acceptance criteria scale with it.

What engineering ownership now looks like

As AI becomes better at producing implementation, competent developers do not need to type every line themselves. Engineering responsibility then concentrates on the decisions the generated implementation can quietly get wrong.

The work that remains hardest to hand off is the work of knowing what the system is supposed to do: requirements and constraints, threat modelling, data ownership, trust boundaries, the choice of language and framework, acceptance criteria, test strategy, review, deployment design, operational behaviour, and whether the result will still be changeable in a year. This is not a claim about human wisdom. It is about accountability: someone has to recognise when a generated implementation violates intent, and remain able to change the system later.

A person clicking approve is not that accountability. Review only helps when the reviewer has context, competence, time, a clear idea of what would count as failure, and tools that can expose it. Plenty of properties are better checked by formatters, type systems, tests and static analysis than by a tired human reading every line. Static analysis, in this sense, is a deterministic inspection of code against known rules, of the kind CodeQL performs. The requirement is effective ownership, not a ritual of staring at generated text.

Existing engineering systems already provide some of that ownership. A change made inside a well-tested codebase, with conventions, types, linters, review and a deployment pipeline, is surrounded by constraints. An entirely generated greenfield application may have few of them. The first version can look finished quickly and still leave the organisation unsure why it is structured as it is. Older codebases are not automatically safer. They often have something generated work can bump into.

AI is also useful on those older systems. It can explain unfamiliar code, draft missing tests before a change, find duplicated logic, help with a dependency migration, or prepare a first pass at a security fix. That is engineering work, not a substitute for it.

There is a continuum that gets flattened when people talk about vibe coding. A non-developer prompting a disposable internal tool, an engineer using an agent heavily while still understanding the important decisions, and a business putting a mission-critical application into production that nobody can explain are not the same situation. How many lines were generated is a poor measure of risk. What matters is whether the software has engineering ownership equal to its consequence.

Dependencies and permissions sit on the same continuum. Agents can add packages as easily as they add functions. Developers have always been able to do that too. Lower generation friction just makes unnecessary complexity cheaper to introduce: overlapping libraries, abandoned packages, a larger attack surface, licensing questions and a heavier upgrade burden. An agent that can only suggest a patch is a different risk from one allowed to change infrastructure, touch secrets, deploy, or modify production data. Current Australian ISM guidance on agentic applications says they should be restricted to the minimum tools and permissions required for the intended purpose. Autonomy is a design choice.

After the software first runs

A system is not finished when it runs for the first time. It will need debugging, upgrades, dependency changes, security patches, new requirements, performance work, infrastructure changes and incident investigation. A generated system becomes operationally dangerous when nobody understands why it was structured the way it was. Modules may follow inconsistent patterns. Logic may be duplicated. Coupling may be hidden. Data ownership may be unclear. Security assumptions may exist because a model invented them, not because anyone chose them. A later fix may break apparently unrelated behaviour because nobody knows where the important decision lives.

Those failure modes are not automatic properties of generated code. They appear when generation has replaced the work of deciding how the system is structured, and of remaining able to change it.

There is now a recognisable class of software in which an organisation has a substantial codebase produced quickly with AI, and has reached the point where a bug, a security change or a new feature requires someone to understand architecture that was never consciously designed. That is an engineering condition. It is the situation AI remediation and rescue exists to unwind.

Organisations do not need another debate about whether developers should use AI. They need a clearer account of which work can be checked cheaply, which decisions will still matter in a year, what already constrains a generated change, and whether anybody understands the system well enough to change it without asking the model to remember it. Where those things are in place, AI is an unusually powerful way to produce the parts of software that were always expensive to type. Where they are not, the organisation is accumulating software it may not be able to operate.