Generating Python Tests with AI

AI-generated tests carry a risk beyond ordinary AI-generated code's general need for review: a test generated by *running* the existing implementation and asserting on whatever output it happens to produce doesn't verify correctness at all — it verifies *consistency with current behavior*, which is a fundamentally different and much weaker guarantee. If calculate_discount(price, percent) has a genuine bug (adding the percent instead of subtracting it, an inversion this curriculum's Custom Exceptions and testing lessons would immediately flag as exactly the kind of error a real test should catch), a test asserting calculate_discount(100, 10) == 110.0 doesn't catch that bug — it *codifies* it, actively locking in the wrong behavior as the 'expected' result.

This is genuinely more dangerous than having no test at all for that function: a missing test is a known, visible gap — anyone looking at the test suite can see this function isn't covered. A test asserting the *wrong* expected value creates false confidence — the test suite reports 100% passing, the function *appears* verified, and a future developer who correctly fixes the actual bug (changing + to -) will find their correct fix now *fails* this incorrect test, creating exactly the kind of confusing, misleading signal that erodes trust in a test suite generally.

This is the single most important risk to understand about AI-generated tests specifically: the test 'passing' tells you nothing whatsoever about whether the assertion inside it is actually correct — only that it currently matches whatever the code happens to do, bug included. Every generated assertion requires independent verification against your own understanding of what the *correct* behavior should genuinely be, entirely separate from whether the test happens to pass against the current implementation.

Given this specific risk, the review discipline for AI-generated tests requires one additional, non-negotiable step beyond the general 'review AI-generated code' discipline from earlier lessons in this section: for every generated assertion, independently determine — using your own understanding of the requirement, not by trusting the test's own claimed expected value — what the *actually correct* result should be, and verify the assertion matches that independently-derived correct value, not merely that the test happens to pass.

For calculate_discount(100, 10), independent reasoning ('a 10% discount on $100 should reduce the price by $10, yielding $90') directly reveals that == 110.0 is wrong, regardless of whether the test 'passes' against the current, buggy implementation. This independent verification step is precisely what a test suite exists to provide in the first place — a check against an *external*, independently-determined standard of correctness, not a check that merely confirms internal self-consistency between a function and a test that was itself derived from that same function's current (possibly incorrect) output.

This connects directly to the Python Testing section's coverage of what a test's assertion should actually represent: a genuine, independently-reasoned specification of correct behavior, never merely 'whatever the code currently happens to produce.' AI-generated tests make this discipline more urgent to apply deliberately, precisely because the generation process (often, though not always, involving the assistant reasoning about or even executing the current implementation) can silently produce exactly this trap if the specific verification step isn't consciously, deliberately applied to every single generated assertion.

Beyond the correctness-verification risk, there's a separate, complementary opportunity: a generic 'write a test for this function' request frequently produces only a single, obvious happy-path test case — exactly the scenario the original developer, writing the function in the first place, almost certainly already considered carefully and got right, making it the case *least* likely to ever actually reveal a bug.

Explicitly prompting for specific categories of edge cases — boundary values (0% discount, 100% discount), unexpected or invalid inputs (a negative percent — should this raise an exception, following the Custom Exceptions lesson's guidance, or is it silently accepted?), and extreme values (a percent over 100, which might produce a negative price) — directs generation toward exactly the categories of input disproportionately likely to reveal genuine bugs, mirroring precisely the boundary-condition testing philosophy the Parameterized Tests lesson established as a natural, high-value fit for exactly this kind of systematic coverage.

Combining this prompt-for-edge-cases technique with the independent-verification discipline from the previous section produces genuinely valuable AI-assisted test generation: broader, more deliberately-targeted coverage of the cases most likely to matter, with every single resulting assertion independently verified against actual correct behavior rather than merely accepted because the test happens to pass — the complete, professional discipline for making AI test generation a genuine asset rather than a source of dangerous false confidence.

A test suite full of tests that pass is only reassuring if those tests are actually verifying correct behavior — an AI-generated test asserting the WRONG expected value is a trap, not a safety net.

The DEFAULT risk: an AI-generated test can assert on whatever the CURRENT implementation happens to return -- even if that's actually WRONG.

Checkpoint: Why is a passing AI-generated test that asserts calculate_discount(100, 10) == 110.0 dangerous, specifically?

• →If the underlying function has a bug (adding the percent instead of subtracting it), this test asserts on that BUGGY output, permanently locking in and 'protecting' the wrong behavior
• →This is invalid pytest syntax and would fail to run at all

Verify EVERY assertion against your own independent understanding of correct behavior -- never accept an AI-generated assertion just because the test 'passes'.

Explicitly prompting for EDGE CASES and FAILURE MODES (not just the happy path) produces genuinely more valuable test coverage.

Checkpoint: Why does explicitly prompting for edge cases (0%, 100%, negative values) produce more valuable coverage than a generic "write a test" request?

• →Edge cases and boundary conditions are disproportionately likely to reveal real bugs, compared to a single happy-path case that rarely fails
• →It simply produces a larger raw NUMBER of tests, which is inherently better regardless of what they cover

Test generation verifies code behaves correctly; AI Code Reviews is next, using AI assistance to catch issues in code before it's merged at all.

Catch a Real Codified Bug. Finish test_matches_correct_formula(): verify the asserted value against the correct formula independently.