ci: faster, legible conformance tests (shipped-binary PR check + scheduled against-main monitor)#169
Draft
amiecorso wants to merge 1 commit into
Draft
ci: faster, legible conformance tests (shipped-binary PR check + scheduled against-main monitor)#169amiecorso wants to merge 1 commit into
amiecorso wants to merge 1 commit into
Conversation
…ainst-main monitor Replace the per-PR from-source base-anvil build (slow) with two advisory, never-blocking conformance jobs: - conformance-beryl.yml (PR/merge_group): downloads the published base-anvil binary we ship and runs the suite against it — fast, no Rust build. Reports divergences via a PR comment, never a red check. - conformance-main.yml (scheduled): builds base-anvil against base/base main from source as an early-warning monitor; off the PR path. Adds a 'Reading CI checks' section to the README documenting blocking (base-std correctness) vs advisory (conformance) checks. Removes the old base-std-fork-tests.yml.
Interface Coverage✅ All interface functions have test coverage. |
✅ Conformance (shipped base/base): all 616 passedThe shipped base-anvil build (base/base |
📊 Forge Coverage (
|
| File | Lines | Stmts | Branches | Funcs |
|---|---|---|---|---|
| 🟢 B20FactoryLib.sol | 100.00% | 100.00% | 100.00% | 100.00% |
| 🔴 test/lib/ForceFeeder.sol | 0.00% | 0.00% | 100.00% | 0.00% |
| 🔴 test/lib/PrecompileProbe.sol | 0.00% | 0.00% | 0.00% | 0.00% |
| 🟢 MockActivationRegistry.sol | 100.00% | 100.00% | 100.00% | 100.00% |
| 🟢 MockActivationRegistryStorage.sol | 100.00% | 100.00% | 100.00% | 100.00% |
| 🟢 MockB20.sol | 100.00% | 100.00% | 100.00% | 100.00% |
| 🟢 MockB20Asset.sol | 100.00% | 100.00% | 100.00% | 100.00% |
| 🟡 MockB20Factory.sol | 98.96% | 99.10% | 100.00% | 100.00% |
| 🟢 MockB20Stablecoin.sol | 100.00% | 100.00% | 100.00% | 100.00% |
| 🟢 MockB20Storage.sol | 100.00% | 100.00% | 100.00% | 100.00% |
| 🟢 MockPolicyRegistry.sol | 100.00% | 100.00% | 100.00% | 100.00% |
| 🟢 MockPolicyRegistryStorage.sol | 100.00% | 100.00% | 100.00% | 100.00% |
| Total | 96.69% | 97.23% | 98.39% | 96.89% |
Full report: download artifact. To browse locally: make coverage (runs forge coverage + genhtml + opens the HTML report).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft / proposal — independent of #167 and #168, for us to consider landing later.
Problem
The per-PR
Base Std Fork Testsjob builds base-anvil (forge+anvil) from source against base/basemainon every PR — that compile is the slow part (120-min timeout). And a red "fork tests" check reads as "your PR is broken" when it really means base/base drifted, which is confusing about what blocks merge.Change
Split the one slow, ambiguous job into two advisory, never-blocking conformance jobs, using our published base-anvil binaries so PRs no longer compile anything:
conformance-beryl.yml(pull_request/merge_group): downloads the published base-anvil build we ship (rollingnightly= Beryl at launch), runsmake fork-testsagainst it. No Rust toolchain, no build — minutes instead of tens of minutes. Result is posted as a PR comment; the check is never red.conformance-main.yml(schedule+workflow_dispatch): keeps the from-source build against base/basemainas an early-warning monitor, off the per-PR path.Removes the old
base-std-fork-tests.yml.Legibility (blocking vs advisory)
A new "Reading CI checks" section in the README states the contract, and the check names carry it:
Build,Test,Format, coverage, CodeQL) — reference suite vs the mocks; required; red = fix it.Conformance: shipped base/base,Conformance: base/base main) — same suite vs the live Rust precompiles; base/base diverging is base/base's job to fix; reported as a PR comment, never a red check.Rule of thumb the docs establish: a red check is always something to fix; conformance divergences arrive as a comment, never a red check.
Note
Do not mark the conformance jobs as required in branch protection — they are intentionally advisory. (The old job should be removed from any required-checks list when this lands.)