AI code review: what to delegate and how to verify it
Use AI in pull request reviews as an additional reviewer that can explain changes, remind the team about missed checks and flag suspicious areas. It does not replace static security analysis, dependency checks, secret scanning, tests or the decision of the responsible developer. Treat every AI comment as a hypothesis until it has been reproduced. A comment does not prove that a defect exists, and silence does not prove that code is safe.
A safer start is a limited pilot: AI suggests findings but cannot approve or merge changes; the existing required checks stay in place; and the team agrees in advance on how to measure value and noise. This article helps you choose review tasks for AI, fit it into your existing process, validate its findings and decide when to expand or stop the pilot.
What to delegate to AI
An AI reviewer is most useful for work that helps focus a human reviewer without making the final decision. First, it can summarize a change. When a pull request touches twenty files, a short explanation of what changed and why can help a reviewer get oriented. Second, it can flag missed checks: an unhandled error, missing input validation, an empty value or a test that does not cover a new branch. Third, it can remind the team about conventions for names, module structure, prohibited libraries and logging. Finally, it can point to areas worth inspecting more closely, such as a manually assembled SQL query, a new external call without a timeout or a change in access permissions.
You can provide team conventions explicitly. For example, Copilot code review can use repository instruction files, both project-wide and scoped to particular paths. Clearer rules tend to produce fewer generic suggestions.
Separate two types of output. A useful prompt directs attention: “permissions may not be checked here.” A confirmed defect is a reproducible problem with known inputs, an action and an incorrect result. AI produces the first. The second comes only after a person or an automated test validates it.
Check the settings of the specific tool. In Copilot, a “Comment” review does not count as a required approval by default. An administrator can separately enable Copilot approvals, which may satisfy a required-approval rule when configured. Keep approvals with a person during the pilot. Copilot may miss a problem or make a mistake, so validate every finding.
What a security review includes
Code security relies on several methods, each designed to catch a different class of problem. Conversational AI review is only one of them.
| Check | What it does | What it does not do |
|---|---|---|
| SAST, static application security testing | Looks for known vulnerability patterns, such as unvalidated data flowing into a database query | Does not know business rules or who is allowed to delete an order |
| Dependency scanning | Compares library versions with known vulnerability databases | Does not assess your own code |
| Secret scanning | Looks for keys, tokens and passwords committed to a repository | Cannot guarantee that every secret is found |
| Tests | Check behavior against specified scenarios | Cover only the cases someone thought to describe |
| Manual security review | A person assesses threat models, permissions and sensitive data | Expensive and selective; difficult to do for every PR |
| AI review | Explains changes, flags suspicious areas and reminds reviewers about omissions | Proves neither that a defect exists nor that code is safe |
GitHub code scanning performs static analysis for vulnerabilities and errors and creates alerts. It can use CodeQL or third-party analyzers that produce SARIF results. Availability and terms for private repositories depend on the plan and organization settings. Copilot code review also has plan and administrative requirements; some usage consumes AI credits, and agentic features can use GitHub Actions minutes. Check the current terms for your organization and repository before starting a pilot.
The practical takeaway: keep SAST if it already exists. If it does not, AI review does not close that gap; configure static analysis separately.
How to fit review into a PR
Connecting an AI reviewer is technically straightforward. The harder part is making it useful instead of noisy. Decide the following before you start.
When to run it. Copilot code review is usually requested manually, but automatic reviews can be enabled in repository or organization settings. Reviewing every new commit also requires a separate setting; otherwise, you can request a repeat review manually. Start the pilot with manual requests so a developer chooses when a change is ready for a first pass. Reviewing every commit creates more comments, and a repeat review may surface earlier findings even after they were closed or rejected.
What the reviewer can access. Apply least privilege: give AI access only to the repositories and tools it needs. Settings may allow Copilot to use MCP tools, which connect to external services. Check which tools are configured and what data they can access.
Where instructions come from. Copilot reads instruction files from the pull request branch, not the base branch. An author can therefore change review rules in the same PR. Review instruction-file changes as carefully as code and assign an owner to those rules.
Who wrote the comment. Clearly label AI findings as automated so human reviewers can tell a colleague’s opinion from a generated suggestion.
How to avoid duplicates. If SAST has already created an alert for a line and AI reports the same issue, the developer gets two tasks. Agree to track the finding in one static-analysis alert and close the duplicate AI comment with a reference to it. Validate the alert too: SAST can be wrong.
How to assess findings
Comment count is not a measure of review quality, much less security. Track four measures instead.
- Confirmed defects: findings that were reproduced and fixed.
- False positives: findings that did not hold up during validation.
- Misses: defects later found by tests, QA or users in changes AI reviewed but did not flag. Measuring these requires a labeled sample of PRs with known defects.
- Developer time: minutes spent reading, checking and closing findings, including false positives.
You can calculate the share of comments that identified a confirmed defect by dividing confirmed findings by all findings for the period. For example, 18 confirmed findings out of 200 equals 9%. These are teaching numbers, not a BotLabs pilot result. The ratio alone does not settle the decision: 18 findings may be valuable if one is an authorization flaw, and low-value if they are stylistic. A person should assess severity after validation, not take the model’s word for it.
A learning matrix: how to verify an AI finding
This is an illustrative example, not a real incident. A developer adds an order-management page. The “Delete” button is visible only to administrators, but the server-side delete handler does not check the user’s role.
| Diff location | AI claim | Reproducible scenario | Severity after validation | Reviewer decision |
|---|---|---|---|---|
| Server-side order-delete handler | “Permissions are checked only in the UI; the server does not check again” | Send a delete request with an ordinary user’s token without using the button. The server returns success: defect confirmed | Critical: any authenticated user can delete someone else’s orders | Block the merge; add a server-side role check and a test for access denial |
| Discount calculation function | “The amount field may be empty” | The type and input validation reject an empty value; a test receives a validation error before the function runs | Not confirmed | Close with an explanation |
| New database query | “The query string is assembled manually and may allow injection” | Static analysis already created an alert for that line | High, but duplicated | Close the AI comment and work from the SAST alert |
| Variable names | “Consider renaming for consistency” | Nothing to reproduce; this is a style preference | Low | Author’s choice; does not block merging |
The first row is the key one. A convincing explanation is not proof. Proof is a request from a user without permission and the server’s response. If the server rejects it, the finding is false; if it permits deletion, the defect is confirmed and the test stays in the suite.
Decide in advance how to track time spent on false positives. For example, developers can label each closed finding “confirmed” or “not confirmed” and record approximate validation time. Otherwise, a month later you will have only impressions.
How to handle fixes
Some tools do more than comment: they suggest ready-made fixes. Treat a suggested fix like any other code change and run the same checks.
An automated fix may contain a syntax error or target the wrong place. It may be syntactically valid but change program behavior because the tool does not know the author’s intent. It may leave a vulnerability unresolved, introduce another one or address only part of the problem. It may even suggest a nonexistent dependency with a plausible name. Review every suggestion and edit it when needed before accepting it.
A practical workflow:
- put the fix in a separate, understandable diff rather than silently folding it into someone else’s commit;
- add a test that fails before the fix and passes afterward;
- rerun all required checks: build, tests, static analysis and dependency scanning;
- verify any new dependency manually: whether it exists, who maintains it and whether it has known vulnerabilities;
- have the module owner review changes to sensitive areas such as authorization, payments, cryptography and data migrations. If the project uses CODEOWNERS, involve the assigned reviewer.
Pilot and expansion
Start with one repository and a limited period, such as several weeks of active development. Choose a repository with regular, medium-sized pull requests: AI may add little to tiny changes, while findings on very large changes are hard to validate.
Before starting, record a baseline: typical time from PR opening to first feedback and to merge, plus the number of defects found after merge. During the pilot, compare similar PRs by type, size and complexity. Comparing a large refactor before the pilot with a small fix afterward will not tell you much.
Set a threshold in advance for when noise outweighs value. Warning signs include:
- developers stop reading AI findings and close them in batches;
- few findings are confirmed, and none of them matter;
- checking false positives takes more time than summaries and reminders save;
- AI repeatedly duplicates static-analysis alerts.
These signals do not always mean you need to abandon the tool immediately. First narrow its job: refine instructions, turn off review on every commit or limit review to certain file types. If value still does not exceed cost, stop the pilot; that is a useful result too.
Summary and first steps
An AI reviewer is useful when it complements existing checks rather than replacing them. A finding is a reason to investigate, not a verdict. Decide whether to expand based on confirmed defects, team time and noise, not comment volume.
You can start by:
- Listing required checks and documenting that AI review replaces none of them.
- Checking the plan and repository settings to see which review and static-analysis features are actually available.
- Choosing one repository, preventing AI from approving or merging changes, and checking connected tools.
- Recording team rules in instruction files and assigning an owner.
- Agreeing on “confirmed” / “not confirmed” labels and a way to track investigation time.
- Building a labeled sample of PRs with known defects so you can measure misses.
- Comparing similar PRs at the end of the pilot and deciding whether to expand, narrow or stop.
If you need to define the scope of a technical pilot — which repositories to include, what validation criteria to use and who will maintain the integration — BotLabs can review your process and help frame a realistic first step.
Frequently asked questions
No. One AI approval is not enough. No findings means only that the model did not flag anything; it does not mean the code is correct or safe. Merging still requires your team’s checks: build, tests, static analysis, dependency scanning and approval by an authorized person. Some tools do not count AI approval toward required reviews by default, though an administrator may change that setting. Keep that permission disabled during a pilot.
Yes. Static analysis and conversational review solve different problems. SAST checks code systematically against rules and creates trackable alerts. AI provides contextual suggestions but cannot guarantee complete coverage or a reproducible set of rules. Use them together: SAST for systematic findings, AI as an additional perspective on logic and team conventions.
Measure outcomes, not comments: how many findings were confirmed, how serious the defects were and how much time validation took. Track false positives and misses separately — defects AI did not flag but that were found later. If useful findings are rare and investigation takes more time than the tool saves, narrow or stop the pilot.
AI code review pilot consultation
In 15 minutes with a BotLabs technical specialist, discuss one repository, how to verify AI comments and the time your team spends assessing them. Outline the pilot scope and the data needed to evaluate confirmed defects and noise.
AI comments
Verify defects
15 minutes with a specialist to outline a pilot and how to track confirmed defects and verification time.
