AI-generated unit and end-to-end tests: choosing tools and measuring value

For unit tests, the most useful assistants work directly with your code and existing test runner, so they can see the function, its dependencies and current tests. End-to-end tests follow a user journey from start to finish, so they need tools with access to the browser and application states that can be checked. There is no universally best product: without the same comparison test on your code, a ranking is an opinion, not a measurement.

The key requirement is that tests find real defects. The number of generated files, higher coverage and a green run do not prove that: a test may pass while repeating the same mistake as the code. This article helps you compare tools, define acceptance criteria for generated tests and set rules for automated “healing” that do not hide bugs.

Unit and end-to-end tests solve different problems

A unit test checks one function or class: given these inputs, expect this result. It usually runs quickly and does not need a browser; external dependencies can be replaced with stubs when necessary. If a test uses a real database or network, that is a different and more costly scenario to assess separately. The value of a unit test is to quickly show that a specific piece of logic broke and where.

An end-to-end test checks a user journey: open a page, sign in, add a product to a cart, place an order and see confirmation. It needs a deployed application, test data, accounts and a browser. It is slower and more brittle, but checks what the customer experiences, including integration between system components.

That leads to different requirements for AI tools. Unit-test generation needs access to code, types, existing tests and a way to run them. End-to-end testing needs access to a working application, the ability to find page elements, and a way to check state outside the screen: did the order really save with the right total?

Which tool categories to compare

Rather than rank products, understand the tool categories and compare them using the same criteria.

IDE assistant. Suggests tests for an open file or selected function. Useful for quick scaffolding, but may have limited context.

Repository agent. Can read several files, run tests, analyze the result and try again. Useful for a series of module tests, but needs limited permissions and careful review of its changes.

Browser agent. Works with the live interface: explores pages, writes scenarios and updates tests. Playwright includes three agent roles: the planner explores the application and creates a Markdown plan; the generator turns the plan into executable tests, checking locators and assertions while running the scenario; and the healer runs a failing test and suggests changes to the locator, state wait or test data. A change to an expected business result still needs a separate human check.

Managed service. An external platform stores and runs tests. It reduces some infrastructure work but adds vendor dependency and access questions for the test environment.

Compare the same things across categories: what context the tool can see; where tests run; who defines the expected result; how much generated tests cost to maintain; and whether they fail when you introduce a defect.

A learning comparison table

Fill this in for your own stack; it is not an evaluation of specific products.

How to prepare context

AI does not know how your business should work. If you provide only code, it may infer expected behavior from the code and check that the code does what it already does. Before generation, prepare:

• Business contract: what the function or scenario must guarantee, such as “discount is no more than 30%,” “an order with a zero total is not created,” or “after payment, the status is paid.”
• Fixtures: prepared test data for products, customers and prices. Keep it stable, known and free of real personal data.
• Allowed side effects: what a test may change. Writing to a test database is fine; emailing real customers is not.
• Test accounts with different roles: administrator, manager and regular customer.
• Existing conventions: test structure, naming, helper functions and use of stubs. Generated tests should look like the team wrote them.

In Playwright, for example, the planner can receive a request plus a seed test that sets up the environment and, optionally, a product requirements document. Better input means less cleanup after generation.

How to accept unit tests

Review a generated unit test by the same standards as a human-written one, with a few additional checks.

Boundaries. Are edge values covered: zero, negative numbers, an empty list, maximum length and invalid formats?

Independent expected result. The expected value should come from the contract or a manual calculation, not from calling the same function. A test that compares “function result” with “function result” checks nothing.

A teaching example shows the difference. The discount function’s contract is: “orders of 10,000 hryvnias or more receive 5%, capped at 1,000 hryvnias.” The code has a bug: it does not apply the 1,000-hryvnia cap. A model that sees only the code may write the test “30,000-hryvnia order → 1,500-hryvnia discount” because that is what the code currently calculates. The test passes and locks in the defect. A contract-based test expects 1,000 hryvnias and fails, catching the issue before a customer sees it. Give the model the contract separately from the code and explicitly require expected values to come from that contract. In review, ask not only “does the test pass?” but also “where did the expected number come from?” If the answer is “from the function’s output,” rewrite the test.

Another useful prompt is to ask the model to list important edge cases before writing tests. The list is easy to scan: does it include exactly 10,000 hryvnias, an amount just below the threshold, an amount where 5% equals the cap, and negative or zero values? A missing case becomes visible immediately, instead of after reading hundreds of lines of test code.

Do not test your own stubs. If a test replaces a dependency with a stub and then checks only that the stub was called with specific arguments, the test is testing itself. It will pass even if the real logic is broken.

Check defect detection. Introduce a deliberate bug — reverse a comparison, remove a check or change a constant — and make sure the test fails. Mutation testing automates this idea by introducing such changes and counting how many the test suite catches.

A useful example of automated filtering is an internal test-generation system described in 2024. It ran proposed tests through successive filters: whether they compiled, passed reliably and increased coverage. In evaluations on two specific products, 75% of generated tests compiled, 57% passed reliably and 25% increased coverage. In internal testing sessions, the system improved tests for 11.5% of the classes it was applied to, and engineers accepted 73% of its recommendations into production code. It was an internal tool used on particular codebases; those numbers do not promise the same result for your repository. The practical lesson is simpler: filter generated tests and review the filtered output rather than accepting every suggestion.

How to accept end-to-end tests

End-to-end tests add stability concerns.

Stable locators. Find elements by role, label or a dedicated test attribute, not by a long path through the page structure that breaks after any layout change.

Controlled data. A test creates the data it needs or uses prepared data; it does not depend on leftovers from other tests.

Wait for state, not a fixed pause. Wait for an element to appear or a status to change instead of sleeping for five seconds. Fixed pauses make tests slower and less reliable.

Order independence. Each test should pass on its own and in any order with the others.

Check system state, not just the screen. A “Thank you for your order” message does not prove that the order was saved with the correct total.

An end-to-end learning scenario: placing an order

Scenario: a customer adds two products to the cart, applies a promo code and places an order. The test checks that the confirmation screen shows the right total and separately — through a test API or database — that the order was saved with the same total and a pending-payment status.

Consider two outcomes after an application update. First, the “Place order” button is renamed “Confirm order,” so the test fails to find it. Updating the locator is a reasonable fix: the interface changed, not the behavior. Second, after the update the order total is 50 hryvnias higher because the promo code no longer applies. The test fails on the total. Do not change the expected total just to make the test pass; it did its job and found a defect.

Coverage deserves a separate caution. Even if line coverage increases after generation, that does not prove defects are absent. Coverage shows which lines ran, not whether the assertions were correct.

How to assess automated fixes

The most sensitive part of test generation is automated “healing.” A tool sees a failing test and changes it so it passes. Sometimes that is right; sometimes it hides a bug.

In Playwright, the healer replays the failing steps, looks for equivalent elements or flows in the interface and suggests a fix, such as updating a locator, adjusting a state wait or correcting test data. It reruns the test until it passes or a guardrail stops the loop. If the agent believes the functionality itself is broken, it may mark the test as skipped. That can be sensible, but a skipped test is still a signal someone must investigate, or the defect may disappear from view.

Set clear rules:

• Allowed without extra approval: update a locator when only the interface changed; replace a fixed pause with a state wait; correct stale test data.
• Requires an explanation and approval: any change to an expected result — amount, status, quantity or message text. The author must explain why the requirement changed and refer to the decision that changed it.
• Prohibited: silently changing an assertion to match the current result; skipping many tests without investigation; deleting a failing test without a task to find the cause.

AI-suggested fixes need human review; existing tests help check whether behavior changed. The same rule applies to tests: changing an assertion changes the contract and must go through review.

Summary and first steps

AI can speed up test scaffolding, edge-case discovery and end-to-end scenario maintenance. But a test’s value depends on whether it finds defects, not who wrote it or whether the run is green.

On your project, you can:

1. Write the business contract for one module: what its functions guarantee and what they must not do.
2. Choose two or three tools from different categories and give them the same task on the same code.
3. Introduce several known defects for each generated test set and count how many the tests catch.
4. Choose one critical end-to-end scenario, such as placing an order, and check system state as well as the screen.
5. Write rules for automated fixes: what is allowed, what needs approval and what is prohibited.
6. Set up notifications for tests marked skipped and investigate each one.

The easiest way to compare testing tools is to start with one module and one critical scenario. If you need to choose tools, define acceptance criteria and decide who will maintain generated tests in CI, BotLabs can review your testing process and help set the pilot boundaries.

Frequently asked questions