AI makes code cheap but not quality, so quality now depends on the practices around the code: reviews, UAT on real data, security checks and clear requirements. At Ralabs, we measure those practices across 50+ client projects with SQM, our platform of ten AI agents. In Ralabs’ data since 2021, practices predicted quality better than the tech stack, test count or AI usage.
Key takeaways
- Coverage, green builds and merged PRs can rise while quality falls; under high AI adoption, Faros found 2.4 times more incidents per pull request.
- Ralabs’ SQM platform scores each project on yes/no practice questions across 14 areas, such as “Is the main branch protected?”
- Ten AI agents collect and validate answers, then turn gaps into Jira tickets ranked by impact; agents suggest, humans approve.
- Good practices with a first AI tool beat weak practices with more AI tools.
- QA is moving from counting tests to measuring practices, and from “Is it green?” to “Is it ready?”
I presented this approach at TestCon Europe 2026 in Vilnius on October 23, in a talk called “AI-Driven Software Quality Measurement in Practice”. This article goes further than the talk: it adds the 2026 industry data, the names of our ten agents and a FAQ.
At TestCon Europe 2026 in Vilnius, with the session board for the day
Slides from the TestCon Europe 2026 talk
Here is the full deck from my TestCon talk: all 51 slides, from the four cases and the 2026 industry numbers to every wave of SQM and what it all means for QA. Use the arrows, your arrow keys or a swipe to move through them. The recording will appear here when TestCon publishes it.
Why AI moved quality risk from code to engineering practices
At TestCon, I started with a question: who in the room expects average software quality to improve over the next 12 months? The answer is not obvious, because AI pushes quality both ways. On one side, agents run regressions overnight, repair broken UI tests and produce test data in seconds. On the other, AI brings vibe-coded features, pull requests twice as big and tests that touch every line without asserting anything.
Before AI, quality sat as a separate stage between development and UAT. When a client wanted speed, it was the first stage to go, and the business paid the real price later, in production. A green build never proved that a release was ready. With AI, the question has changed from “Can we release faster?” to “Can we release faster, with confidence?”
Four cases where a practice was missing, not a test
None of these teams lacked tests. Each one lacked a practice.
Case 1: “Everything was tested.” A small team shipped v1 of an offline-first mobile app after autotests, manual QA and pre-production all came back green. The release lost offline uploads for more than 1,000 users and had to be rolled back. The pre-production data did not resemble production data, and nobody ran UAT.
Case 2: “Just a tiny sub-product.” The company behind it passed a SOC 2 audit every year. To move faster, a new team used a lot of AI to build a small sub-product and connected it to the primary database. The sub-product contained an SQL injection, which led to a data leak, fines and penalties. The audit passed; the practice was missing.
Case 3: “Programming is hard.” On October 14, 2026, Epic Games released a preview of a launcher it said was vibe coded. Tests commissioned by Epic showed a cold start of 5.47 seconds against 10.07 seconds for Steam (Dexerto). Within days, players reported crashes, freezes, login failures, blank store pages, high memory use and missing Unreal projects (Windows Report, VGC). Epic replied that it was a preview and asked users to send feedback. Epic measured start-up time; users measured everything else.
Case 4: “Up to 30% of our code is written by AI.” At LlamaCon in April 2025, Microsoft CEO Satya Nadella said that 20% to 30% of the code in Microsoft’s repositories is written by AI (TechCrunch). Microsoft later documented that, after cumulative updates released from July 2025, Explorer, the Start menu and the taskbar could fail to start on some enterprise and managed Windows 11 devices (KB5072911). Microsoft traced the issue to XAML packages that did not register in time, not to AI, and fixed it in June 2026. That is fair: correlation is not proof. But when AI writes up to a third of your code, the code alone is not where to look. The practices around it are.
What the 2026 numbers show
The industry data points the same way. Faros analysed two years of telemetry from 22,000 developers and found 2.4 times more incidents per pull request under high AI adoption (Faros). In the same data, the wait for a PR review grew 4.4 times, and 31% more pull requests were merged with no review at all. New Relic found that 62% of teams ship AI-generated code without line-by-line review (New Relic).
BCG’s AI Radar 2026 explains why: 70% of the value from AI comes from people and process, 20% from technology and data, and only 10% from the algorithms (BCG). If most of the value sits in people and process, so does most of the risk. We covered the engineering side of this shift in The Five Engineering Challenges That Emerged After the AI Rush.
Why classic software quality metrics miss it
Code coverage, green builds, merged pull requests, merge speed, lines of code and approvals can all rise while quality falls. This is Goodhart’s law: once a measure becomes a target, it stops being a good measure. AI did not invent the problem; it made it happen in weeks instead of years.
Speakers at QCon AI Boston 2026 and TestMu Conf 2026 described the same shift. Teams from Red Hat AI, Gallup, Alkami, CITY Furniture and Paramount stopped counting output and now track escaped defects, the bugs that reach production. Escaped defects show that something broke. Practices show that something is about to break.
What software quality means beyond code
SQM is the internal Software Quality Management platform we built at Ralabs to measure and control software quality. It scores each client project on yes/no practice questions across 14 areas, from product and testing to security, compliance, AI and team happiness. Where clients allow it, SQM adds data from GitHub, Jira and CI to the team’s answers.
Before we built anything, my team and I had to agree on a definition. Standards such as ISO/IEC 25010 describe the quality of the software product itself: reliability, security, maintainability and similar characteristics. Our definition is wider. Software quality, as we use the term, is the sum of three pillars:
- Project excellence: risk management, budget and cost control, scope, stakeholders, and software quality management itself, which is where QA and test automation live.
- People excellence: growth and training, knowledge sharing, onboarding efficiency, and no single person the project cannot do without.
- The client’s voice: regular business syncs, delivery reviews, CSAT and NPS, adoption support, and R&D.
Ralabs builds 50+ products at the same time, on very different stacks, so no existing model fit. Code-level tools only see the code. Process frameworks such as CMMI or COBIT are heavy and expensive. Delivery metrics leave out the practices around delivery.
That is why every practice in SQM is a single yes/no question, such as “Is the main branch protected?” or “Is UAT mandatory?” Every incident like the four cases above adds the missing practices to the list.
How we built SQM: five waves from spreadsheets to AI agents
We did not get it right the first time. Our approach went through five waves, and each one fixed what the previous one got wrong. I described the platform’s architecture in more detail in How to run software quality at scale: A CTO Perspective.
Wave 1: Survey and spreadsheets
Every team answered the same 500+ yes/no questions. Each practice had a weight, and the results became a radar chart and a single score. We learned four things: self-reports are overoptimistic, a quarterly snapshot is already history when someone reads it, teams rely on a few senior people rather than on processes, and simply asking about a practice improves it.
Wave 2: The SQM web app
The first version of SQM replaced one file per project with one view of the whole portfolio, including a quality trend for every project since 2021. Teams filled it in every three months. The lessons: weights matter, trends beat absolute scores, and you score projects, never people. But people still had to answer every question and find the red flags themselves.
Wave 3: The web app plus an MCP server
An MCP server let our managers ask about quality from Claude or ChatGPT, and let teams answer in Slack instead of forms. Where clients agreed, SQM pulled data from GitHub, Jira and CI, so a question like “Is the main branch protected?” got its answer from the repository rather than from memory. Permissions for each client turned out to be harder than MCP itself, and search over project documentation exposed gaps in the docs.
Wave 4: An agentic platform with ten AI agents
Today ten AI agents work in one loop across five stages:
1Collect: the Survey Router and the Evidence Agent
2Validate: the Validation Agent and the Metrics Calculator
3Explain: the Code Quality agent and the Security & Compliance agent
4Act: the Release Readiness agent, the Incident → Practice agent and the Insights Agent
5Learn: the Knowledge Agent, so every answer improves the next survey
Every three months, the Survey Router sends each role only the practices that apply to it, as a question in Slack. The person answers first, with no AI hint to anchor on. Only then does the Validation Agent compare the answer with the docs and, where approved, the repositories. If they disagree, the disagreement starts a conversation, not an override.
From there, a score turns into a to-do list ranked by impact and AI-estimated effort, and the tickets land in the team’s Jira. Code still counts, as one signal among many: the Code Quality agent tracks how much production code was written with AI and what AI reviews found. AI writes the formula, never the number.
Our rule is that agents suggest and humans approve. We test the agents like any other system: by acceptance rate, edit rate, false positives and time saved. Wave 4 taught us three things: AI removed the optimism bias, teams wanted control more than speed, and without evidence there is no trust.
Wave 5: Telemetry and skills with AI Adoption Score
AI Adoption Score measures three things together: what people say about AI, what they know and what they actually do with it. Usage is not skill, and skill is not quality. Heavy users are not always skilled, blind spots can be measured, and skill without practices still fails. The most useful number is how many people rate themselves as advanced and then fail the quiz. I talked about measuring AI adoption in more depth at DEVWorld Amsterdam.
What tied it to the business
After the first two waves, one lesson connected the work to revenue: clients do not buy test coverage, they buy releases without surprises. Better practices lead to fewer incidents, more trust and more revenue.
What more than four years of quality data across 50+ projects showed
About AI and quality
- In Ralabs’ data, practices were the strongest predictor of quality, stronger than the tech stack, the test count or AI usage.
- Small, self-contained teams get more value from AI.
- A senior engineer with AI is a multiplier.
- Good practices with a first AI tool beat weak practices with more AI tools, every time.
About running a quality measurement system
Asking already improves quality. A team that answers “no” to “Do you have a rollback plan?” often writes one before anyone asks.
Visibility makes teams improve faster. Once we shared the company-wide view at Ralabs, teams started competing on it.
Managers and engineers see quality differently. Managers look at deadlines, incidents and client happiness; engineers look at code, tests and tech debt.
People and habits limit quality more than tools do.
The trend matters more than the score.
Measure projects and teams, never individuals. Never tie the score to bonuses, or people stop answering honestly.
How the QA role changes when AI writes code and tests
AI in software testing can write code and cover a feature with tests. What it cannot do, or will not do, is push back on a client’s requirements, understand what the business really needs, keep the system maintainable or make the key decisions without us. It lacks the context of your system, and, like many developers, it is happy to own only the part it wrote. That gap is exactly where QA comes in.
Experienced teams already work this way. The less experienced the team, the stronger the urge to automate everything and treat coverage as the goal. Senior engineers automate what is risky and then check the practices around it. That does not make automation less important: the more features AI ships, the more testing they need.
PractiTest’s State of Testing 2026 shows how far many teams still have to go: 56.4% of testing teams judge themselves by test coverage, and only 8.6% by business impact (PractiTest). AI in quality assurance moves the job in six directions:
| From | To |
|---|---|
| Counting tests | Measuring practices |
| A gate at the end | Signals all the time |
| Testing the code | Testing the agents too |
| Coverage reports | Business impact |
| “Is the whole system correct?” | “What did this change break?” |
| “Is it green?” | “Is it ready?” |
Other talks in the IMAX hall at TestCon Europe 2026
My closing line at TestCon was: quality moved into the practices, and measuring quality practices is your job now.
TestCon Europe 2026 in Vilnius
TestCon Europe is a conference for software testing and QA professionals. This year was its 11th edition, with more than 800 attendees and over 60 talks. I spoke on the morning of October 23 on the main stage, the IMAX hall of Forum Cinemas Vingis in Vilnius.
More from my talks
Measure your team's practices with AI Adoption Score
SQM runs in our own delivery at Ralabs. AI Adoption Score, the product we built from it, brings two parts of this approach to any team:
The practices. The Software Quality Management survey reviews one project against 248 practices in 14 areas. It asks each person only the questions for their role and scores every area out of 10, listing what is missing in order of importance.
AI skills and usage. The 360° AI Maturity Assessment, the Claude Skills Assessment and AI Coding Usage for Claude Code and Codex show what people say, know and do with AI, each with a live report. An MCP server lets you ask your AI assistant about the results.
Want to see it on your own team? Book a demo, and we will show you AI Adoption Score on your own questions.
It is free to start and live in two clicks.
The report fills in as people answer.
at Ralabs
FAQ
How do you measure software quality in AI-assisted development?
Measure the practices around the code as well as the code. At Ralabs, we score each project on yes/no questions across 14 areas, such as “Is the main branch protected?” and “Is UAT mandatory?”. Where clients allow it, we check the answers against GitHub, Jira and CI data.
Why do coverage and green builds miss AI-generated code quality problems?
They measure output. Coverage, green builds, merged pull requests and approvals can all rise while quality falls. Under Goodhart’s law, a measure that becomes a target stops being a good measure, and AI makes that happen in weeks. Escaped defects show what broke; practices show what is about to break.
What is SQM?
SQM is the Software Quality Management platform we built at Ralabs. It checks practices in 14 areas across 50+ client projects. Ten AI agents collect answers, validate them, explain gaps, turn them into ranked Jira tickets and learn from the results.
What predicts software quality best when teams use AI?
In Ralabs’ data since 2021, engineering practices were the strongest predictor, stronger than the tech stack, the test count or AI usage. Good practices with a first AI tool beat weak practices with more AI tools.
Should quality scores be tied to individuals or bonuses?
No. At Ralabs, we measure projects and teams, never individuals, and never tie the score to bonuses, because people stop answering honestly once it is.
How do AI agents check quality answers without biasing them?
The person answers first, with no AI hint. An agent then checks the answer against the docs and repositories, and any disagreement becomes a conversation, not an override. Agents suggest and humans approve. AI writes the formula, never the number.
How does the QA role change when AI writes code and tests?
It moves from counting tests to measuring practices, from a gate at the end to continuous signals, from testing code to testing agents too, and from “Is it green?” to “Is it ready?”
What is AI Adoption Score?
AI Adoption Score is a product that brings SQM’s approach to any team. It combines a Software Quality Management survey of 248 practices with assessments of what people say, know and do with AI, each with a live report.
