AI Wrote the Code. Can Your Tests Catch the Mistakes?

The expected result must come from somewhere other than the code

AI-assisted development can produce an implementation and a matching test suite in the same conversation. The result may look reassuring because every test passes. The important question is whether those tests check the behavior the business actually requires. If the implementation and its tests share the same mistaken assumption, agreement between them can hide the problem. A useful testing process gives the expected result an independent foundation.

Consider a fictional agency portal that lets customers accept project milestones. A generated feature allows an approval to be recorded and sends a notification. The team needs to know which customer may approve which milestone, what happens when the action is repeated, and whether a changed milestone can still be approved using an earlier screen. This article develops a verification plan for that example. It does not report results from a deployed Work Vultures system.

Write the business rule as an observable outcome

Describe approval without referring to the implementation. A possible rule is that an authorized customer can accept the current version of a milestone once, and the accepted version remains identifiable afterward. This statement creates several questions to resolve with the business: who is authorized, what makes a version current, what acceptance changes, and how a correction is handled. Resolve those meanings before asking a tool to generate more tests.

Use examples that include starting state, action, and expected final state. A customer viewing milestone version three should not silently approve version four if it changed in the meantime. An unauthorized account should not be able to approve either version. A repeated request should not create two acceptance events with conflicting meanings. Each example provides a testable outcome that can be reviewed independently of the code's current structure.

Keep the rules close to the people who own them. A product representative can clarify the meaning of approval, while an engineer can explain the enforcement and a designer can review the visible feedback. A test author should not have to infer every requirement from a function name or database field. That inference can reproduce the same ambiguity that entered the implementation during generation.

Choose evidence at the layer where the risk exists

A small calculation can be checked close to the function that performs it. An account boundary needs verification where identity and authorization are enforced. A customer journey needs a check that connects the interface to the accepted result. These tests provide different evidence. A large number of quick function tests cannot substitute for a missing check on who may approve a milestone, even if the overall coverage percentage looks impressive.

OWASP's Application Security Verification Standard offers requirements for reviewing application security. Select the requirements relevant to the portal and attach evidence to the chosen scope. Referencing the standard does not certify the feature. The practical task is to establish whether the application's controls behave as intended under the conditions being tested and to identify where specialist review is needed for risks the team cannot evaluate adequately.

For the approval example, inspect both the trusted response and the visible outcome. A browser test that sees an approved label may miss a server rejection hidden behind optimistic interface behavior. A service test that records acceptance may miss a screen that continues to invite another submission. Define which layer establishes the authoritative state and which layer communicates it to the customer. The test plan should connect those responsibilities.

Design synthetic data that exposes boundaries

Prepare at least two customer accounts, several milestone versions, and recognizable sample projects. Give the records clear identifiers so that a failed test is easy to investigate. The data should include a current milestone, an earlier version, an already accepted milestone, and a record belonging to another account. Avoid real customer information in an ordinary automated test environment when synthetic records can exercise the same rule.

Control how each test starts and ends. A test that depends on an earlier test creating a particular project can fail for reasons unrelated to the behavior being reviewed. Shared state can also allow an unexpected success because another test left the system in a favorable condition. Playwright's best-practice guidance emphasizes isolated tests and checking user-visible behavior. Apply those principles to the portal's actual account and data boundaries.

Keep time-dependent behavior explicit. An invitation may expire, a version may become outdated, or a notification may be processed later. Use a controlled way to establish the relevant state instead of waiting an arbitrary period and hoping the environment changes as expected. Record which parts of the workflow are simulated and which use the actual integration. That distinction defines what a passing test can legitimately establish.

Include rejection, repetition, and interruption

Create cases for an unauthorized account, an expired session, a stale milestone version, a repeated request, a slow response, and a failed notification. Determine the expected outcome for each before implementing the test. If notification failure does not undo a valid acceptance, the interface and operating process need to reflect that rule. If the business expects a different result, the implementation must support that meaning consistently.

For an illustrative plan, divide twenty cases into eight ordinary scenarios, six authorization and state checks, and six interruption or recovery checks. Suppose eighteen pass while an account boundary case and a repeated-submission case fail. The ninety percent pass rate does not make the release acceptable. Report the effect of each failure and the decision it blocks. Counts help summarize work, but they cannot assign business significance automatically.

Observe repeated actions at the trusted service as well as the interface. Disabling an approval button may reduce accidental clicks, while a repeated request can still arrive through a retry or another browser. The system needs a defined response to that situation. A successful test should demonstrate that the accepted milestone state remains correct and that the customer receives an understandable result without creating a contradictory history.

Review generated tests for accidental agreement

Read the assertions and ask what evidence would make them fail. A test that checks only that a function returns a value can pass while the value is wrong. A browser test that waits for an element created by the implementation may simply confirm that the same implementation ran. The useful question is whether the assertion corresponds to the independent rule and whether a realistic defect would be detected.

Try a controlled mutation of the implementation in a disposable test environment. For example, temporarily remove a version check and see whether the relevant test fails. This is a way to investigate the strength of the test, not a change to ship. Restore the implementation afterward and preserve the observation. If the suite still passes, inspect whether the test data ever exercised the condition the check was meant to protect.

Avoid making the assistant the sole judge of its own output. It can propose cases, explain a failure, or draft a test, while a person reviews the rule, evidence, and consequences. Another model's agreement can provide additional questions but does not establish independent operational correctness. Keep the review anchored in requirements and observable behavior so the result remains useful even when the model or prompt changes.

Treat unreliable tests as unresolved information

A test that sometimes passes and sometimes fails needs investigation. Repeating it until it passes may conceal a timing assumption, shared data problem, or an actual race in the application. Record the conditions and diagnostic evidence from failures. A retry policy can help an automated pipeline continue collecting information, but it should not erase the fact that the same configuration produced inconsistent outcomes.

Prefer synchronization tied to an observable event over a fixed delay where the testing tool supports it. For the portal, wait for the accepted milestone state or the relevant response rather than assuming that a notification has finished after an arbitrary number of seconds. Keep the assertion aligned with the layer being tested. A visible confirmation and a background delivery event may require different checks and different time expectations.

Maintain a useful failure record. The starting data, important request and response, configuration version, and visible state can help another engineer reproduce the problem. Avoid retaining complete private records or credentials in ordinary test reports. Diagnostics should be designed for the question being investigated. A large collection of screenshots and logs is less useful when nobody can identify which evidence explains the failed rule.

Use the result to make a scoped release decision

Summarize which behaviors were checked, what failed, and what remains outside the evaluation. Distinguish a mocked notification from a verified delivery integration and a synthetic account from production access configuration. The release decision should account for those limits. A passing suite supports the tested claims; it does not establish that the application is free of defects or that every environment behaves identically.

After a relevant change, rerun the checks that protect the affected behavior and any required broader release checks. Expand testing when new failures or unresolved concerns justify it. Repeating every available experiment without a reason can consume time without improving confidence. The objective is a proportionate body of evidence that helps the team understand the important risks and operate the feature responsibly.

AI-written code benefits from the same disciplined questions as any other implementation, with extra attention to assumptions shared by generated code and generated tests. Define the expected outcome independently, choose the right layer, use boundary-focused data, and inspect what a failure means. A smaller collection of well-founded checks can support a stronger decision than a larger collection that merely agrees with the code it was created to test.