[SWIFT-TESTING] Feature: Add test verification lab#3
Conversation
| func disabledFlakyExample() { | ||
| let id = RandomIDGenerator().makeID() | ||
| #expect(id == "user-1234") | ||
| } | ||
| ``` | ||
|
|
||
| It is disabled so the normal verification run remains stable, while the failing test body is still visible. | ||
|
|
||
| The improved design injects randomness: | ||
|
|
||
| ```swift | ||
| let generator = RandomIDGenerator(randomizer: FixedRandomizer(1234)) | ||
| #expect(generator.makeID() == "user-1234") | ||
| ``` |
| @Test("A weak assertion can pass while missing the exact behavior") | ||
| func weakAssertionExample() { | ||
| // 1. Run the game with a number that should produce two claps. | ||
| let result = ThreeSixNineGame().play(33) |
There was a problem hiding this comment.
Since the same code is used in the test methods below, could you move this variable to a suite property? Swift Testing creates a new suite instance for each test method, so the property won’t be shared between tests.
There was a problem hiding this comment.
Updated. I moved this setup into a suite property.
| @Test("Mutation: returning one clap for every match would be caught") | ||
| func mutationReturningOnlyOneClapIsCaught() { | ||
| // 1. Create the system under test. | ||
| let game = ThreeSixNineGame() |
There was a problem hiding this comment.
Updated here as well. This is now a suite property.
|
|
||
| Tests should be independent. A suite groups tests by topic, but tests inside one suite should not depend on being run in the order they appear in the file. | ||
|
|
||
| ## Experiments |
x-0o0
left a comment
x-0o0
left a comment
There was a problem hiding this comment.
Aside from some comments I left, this looks good to me. Once you’ve resolved all the comments, feel free to merge this PR.
Welcome to the world of Testing Guardians
|
Thanks for the comments. I applied the same suite-property cleanup in both places. Swift Testing creates a fresh suite instance for each test, so these immutable properties keep the setup reusable without sharing state across tests. |
2 participants
Summary
This PR adds
VerificationTestingLab, a Swift Package focused on verifying whether tests actually catch bugs when using Swift Testing.The lab explores common sources of false confidence in tests, including weak assertions, false positives, flaky randomness, shared mutable state, insufficient coverage, and mutation validation.
What Changed
VerificationTestingLab/testing-lab-verificationThreeSixNineGameLoginValidatorRandomIDGeneratorSharedCounterVerificationTestingLab/README.mdDocs/verification-tests.md.gitignoreto exclude SwiftPM/Xcode build artifactsMethodology
The lab is organized around test verification risks rather than file structure.
Each experiment asks:
The documentation includes a Mermaid diagram that maps production subjects to verification risks and stronger testing strategies.
Verification
Ran the Swift Package test suite:
Result:
The disabled tests are intentional examples for flaky and order-dependent behavior.
Notes for Reviewers