H-6664: Python monorepo infrastructure (uv workspace + turbo integration) — experimental spike#8994
H-6664: Python monorepo infrastructure (uv workspace + turbo integration) — experimental spike#8994claude[bot] wants to merge 5 commits into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
| [tool.uv] | ||
| package = false | ||
|
|
There was a problem hiding this comment.
This pyproject.toml configures uv but does not set a dependency cooldown. Newly published packages can be malicious or unstable. Add exclude-newer = "7 days" under [tool.uv] to wait 7 days before resolving newly published package versions. Added in: 0.9.17 Reference: https://docs.astral.sh/uv/concepts/resolution/#dependency-cooldowns
🍰 Fixed in commit d55400d 🍰
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #8994 +/- ##
=======================================
Coverage 59.73% 59.73%
=======================================
Files 1371 1371
Lines 135461 135461
Branches 6066 6066
=======================================
Hits 80911 80911
Misses 53618 53618
Partials 932 932
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Merging this PR will degrade performance by 15.38%
|
| Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|
| ❌ | bit_matrix/dense/iter_row[64] |
140.8 ns | 170 ns | -17.16% |
| ❌ | bit_matrix/dense/iter_row[200] |
185.8 ns | 215 ns | -13.57% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing claude/h-6664-python-monorepo-spike (22a66ed) with main (b6fb59e)
…o integration) - root pyproject.toml: uv workspace with shared ruff/basedpyright/pytest config and a single dev dependency group; committed root uv.lock - libs/@local/python-example: minimal example package (hatchling, typed, tested) whose package.json scripts wrap uv so turbo runs lint:ruff, lint:basedpyright and test:unit like any other workspace package - turbo.json: lint:ruff, lint:basedpyright and fix:ruff tasks - lint.yml: detect and run ruff/basedpyright per package, mirroring the existing ESLint/TSC steps; test.yml picks up test:unit automatically - prune-repository action: keep root pyproject.toml and uv.lock - check-license-in-workspaces: ignore .venv
- requires-python >=3.14 across the workspace (matching internal-infra) and ruff target-version py314 - [tool.uv] exclude-newer = "7 days" per the uv-missing-dependency-cooldown Semgrep policy; re-locked (no resolved versions changed)
uv resolves the whole workspace, so a pruned tree missing any member listed in the root pyproject.toml fails uv sync --locked / uv run --frozen. prune.py now adds every uv workspace member to the turbo prune scopes (keeping their pruned yarn.lock entries intact) and falls back to copying member directories that have no package.json.
d55400d to
43e81b5
Compare
…the prune path
libs/@local/python-example-two depends on hash-python-example via
[tool.uv.sources] { workspace = true } (mirrored as workspace:* in its
package.json for turbo's graph) and imports it in module and tests, so
cross-package resolution and the prune exemption are validated on every
CI run instead of only in a local reproduction. Both example packages
are temporary scaffolding, marked for deletion in their READMEs once
the first real Python package joins the workspace.
oxfmt (yarn lint:format) normalizes *emphasis* to _emphasis_ in markdown; the Global lint job caught the discrepancy.
Benchmark results
|
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| resolve_policies_for_actor | user: empty, selectivity: high, policies: 2002 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: medium, policies: 1002 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: high, policies: 3314 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: medium, policies: 1527 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: high, policies: 2078 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: medium, policies: 1033 | Flame Graph |
policy_resolution_medium
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| resolve_policies_for_actor | user: empty, selectivity: high, policies: 102 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: medium, policies: 52 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: high, policies: 269 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: medium, policies: 108 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: high, policies: 133 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: medium, policies: 63 | Flame Graph |
policy_resolution_none
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| resolve_policies_for_actor | user: empty, selectivity: high, policies: 2 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: medium, policies: 2 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: high, policies: 8 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: medium, policies: 3 | Flame Graph |
policy_resolution_small
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| resolve_policies_for_actor | user: empty, selectivity: high, policies: 52 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: medium, policies: 26 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: high, policies: 94 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: medium, policies: 27 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: high, policies: 66 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: medium, policies: 29 | Flame Graph |
read_scaling_complete
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| entity_by_id;one_depth | 1 entities | Flame Graph | |
| entity_by_id;one_depth | 10 entities | Flame Graph | |
| entity_by_id;one_depth | 25 entities | Flame Graph | |
| entity_by_id;one_depth | 5 entities | Flame Graph | |
| entity_by_id;one_depth | 50 entities | Flame Graph | |
| entity_by_id;two_depth | 1 entities | Flame Graph | |
| entity_by_id;two_depth | 10 entities | Flame Graph | |
| entity_by_id;two_depth | 25 entities | Flame Graph | |
| entity_by_id;two_depth | 5 entities | Flame Graph | |
| entity_by_id;two_depth | 50 entities | Flame Graph | |
| entity_by_id;zero_depth | 1 entities | Flame Graph | |
| entity_by_id;zero_depth | 10 entities | Flame Graph | |
| entity_by_id;zero_depth | 25 entities | Flame Graph | |
| entity_by_id;zero_depth | 5 entities | Flame Graph | |
| entity_by_id;zero_depth | 50 entities | Flame Graph |
read_scaling_linkless
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| entity_by_id | 1 entities | Flame Graph | |
| entity_by_id | 10 entities | Flame Graph | |
| entity_by_id | 100 entities | Flame Graph | |
| entity_by_id | 1000 entities | Flame Graph | |
| entity_by_id | 10000 entities | Flame Graph |
representative_read_entity
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/block/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/book/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/building/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/organization/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/page/v/2
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/person/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/playlist/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/song/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/uk-address/v/1
|
Flame Graph |
representative_read_entity_type
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| get_entity_type_by_id | Account ID: bf5a9ef5-dc3b-43cf-a291-6210c0321eba
|
Flame Graph |
representative_read_multiple_entities
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| entity_by_property | traversal_paths=0 | 0 | |
| entity_by_property | traversal_paths=255 | 1,resolve_depths=inherit:1;values:255;properties:255;links:127;link_dests:126;type:true | |
| entity_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:0;links:0;link_dests:0;type:false | |
| entity_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:0;links:1;link_dests:0;type:true | |
| entity_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:2;links:1;link_dests:0;type:true | |
| entity_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:2;properties:2;links:1;link_dests:0;type:true | |
| link_by_source_by_property | traversal_paths=0 | 0 | |
| link_by_source_by_property | traversal_paths=255 | 1,resolve_depths=inherit:1;values:255;properties:255;links:127;link_dests:126;type:true | |
| link_by_source_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:0;links:0;link_dests:0;type:false | |
| link_by_source_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:0;links:1;link_dests:0;type:true | |
| link_by_source_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:2;links:1;link_dests:0;type:true | |
| link_by_source_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:2;properties:2;links:1;link_dests:0;type:true |
scenarios
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| full_test | query-limited | Flame Graph | |
| full_test | query-unlimited | Flame Graph | |
| linked_queries | query-limited | Flame Graph | |
| linked_queries | query-unlimited | Flame Graph |
Requested by Tim Diekmann · Slack thread
Warning
Experimental spike for H-6664 "Create Python monorepo infrastructure" (internal), opened so the approach can be picked apart before the real implementation. Totally fine to close this and start over.
🌟 What is the purpose of this PR?
Demonstrate proper Python monorepo infrastructure, borrowing internal-infra's uv-workspace pattern and wiring it into turborepo.
Before: Python packages are invisible to turbo and CI. There is no shared lint/type-check/test configuration and no lockfile policy — e.g. #8989 adds a uv package with no
package.json(so turbo never sees it), no CI, and a gitignoreduv.lock.After:
turbo run lint:ruff lint:basedpyright test:unit --filter '@local/python-example'runs ruff, basedpyright, and pytest for Python packages via uv — with shared config and one committeduv.lockat the repo root — and the existing Lint/Test CI pipelines pick Python packages up like any other workspace package.How it works
pyproject.tomlis a virtual workspace root ([tool.uv] package = false) listing Python packages as members, withrequires-python = ">=3.14". A singleuv.lockis committed at the repo root (do not gitignore it). Shared tooling lives in one rootdevdependency group:ruff,basedpyright,pytest,pytest-asyncio. Per org Semgrep policy (uv-missing-dependency-cooldown),[tool.uv] exclude-newer = "7 days"applies a dependency cooldown — releases younger than a week are ignored during resolution (recorded inuv.lockasexclude-newer-span = "P7D"; adding it changed no resolved versions).pyproject.toml:select = ["ALL"]with internal-infra's short, documented ignore list (COM812,D203,D213,FIX002,TD003) plus per-file ignores for tests;line-length = 100(matches the Rust width used here);target-version = "py314".typeCheckingMode = "all".--import-mode=importlib,asyncio_mode = "auto".package.jsonshim (the key part): each Python package carries apackage.jsonwhose scripts wrap uv, e.g."test:unit": "uv run --frozen --all-packages pytest". Turbo then treats the package like any other workspace member — task graph, caching,--filter, remote cache all work. Two subtleties:--all-packagesis required:uv runfrom a member directory only syncs that member's dependencies, so without it the rootdevgroup tools would be missing from the shared venv (verified with a fresh.venv).--project ../../..because pyright, unlike ruff and pytest, does not search parent directories for configuration.libs/@local/python-example(hatchling build backend, src layout,py.typed, fully typed module, pytest suite incl. an async test for the pytest-asyncio wiring) andlibs/@local/python-example-two, which depends on the first via[tool.uv.sources] hash-python-example = { workspace = true }(mirrored as aworkspace:*dependency in itspackage.jsonso turbo's graph matches) and imports from it in module + tests — so cross-package uv workspace dependencies and the prune path are exercised on every CI run, not just coexistence. Both are marked as temporary in their READMEs: they validate the infra and should be deleted once the first real Python package (e.g. petrinaut-opt) joins the workspace.CI (mostly the zero-new-workflow path)
test.yml: zero changes needed. It discovers packages viaturbo queryonTASK_NAME == "test:unit"; verified locally that@local/python-exampleshows up in exactly that query. Runners get uv 0.11.18 via mise (install-tools), anduv run --frozensyncs the venv on first use.lint.yml: small addition. It enumerates tools per package (lint:eslint,lint:tsc, clippy, …) rather than running a genericlinttask, so it genuinely can't pick Python up unaided. Addedlint:ruff/lint:basedpyrightdetection + run steps mirroring the existing ESLint/TSC pattern.prune-repositoryaction: the action keeps the rootpyproject.toml+uv.lock(they aren't part ofturbo prune's output), andprune.pytreats every uv workspace member as an always-included prune scope (directory-copy fallback for members without apackage.json) — Python is effectively exempt from pruning, since uv resolves the whole workspace (see open question 4).check-license-in-workspacesnow ignores.venv/(uv creates it at the repo root).turbo.jsonalready listedpyproject.tomland*.lockinglobalDependencies, so Python task caching invalidates correctly with no changes.🔗 Related links
libs/@hashintel/petrinaut-opt(unwired uv package, base branchcf-petrinaut-python)🔍 What does this change?
pyproject.toml(uv workspace + shared ruff/basedpyright/pytest config) and committed rootuv.locklibs/@local/python-exampleandlibs/@local/python-example-two(pyproject.toml, package.json shim, LICENSE.md, typed module, tests each; the second depends on the first through the uv workspace)turbo.json: newlint:ruff,lint:basedpyright,fix:rufftaskspackage.json: matchinglint:ruff/lint:basedpyright/fix:ruffscripts, soyarn lintincludes Python.github/workflows/lint.yml: ruff/basedpyright detection + run steps.github/actions/prune-repository/:action.ymlkeepspyproject.toml+uv.lock;prune.pyalways preserves uv workspace memberslibs/@local/repo-choreslicense check: ignore.venv/yarn.lock: registers the new workspace❓ Open questions for Bilal
Python version floor — resolved:
requires-python = ">=3.14"per Bilal, matching internal-infra (initially shipped as>=3.13). Note a workspace resolves on the intersection of all members, so FE-1162: Python optimisation script (Optuna) driving the Petrinaut CLI #8989's>=3.10package could only join by raising its own floor.Placement/naming:
libs/@local/python-examplewith dist namehash-python-example/ modulehash_python_example. Should Python packages live under a different convention (e.g. alongside@hashintel), and should the npm-scope name mirror the dist name?petrinaut-opt (FE-1162: Python optimisation script (Optuna) driving the Petrinaut CLI #8989): should it join the workspace (member entry + package.json shim + un-gitignore its lock into the root
uv.lock) whencf-petrinaut-pythonlands?CI pruning — reproduced; interim fix shipped (the remove-prune-entirely discussion stays open). With a temporary second workspace member (
libs/@local/python-example-two, not committed) and the prune-repository action's steps run locally scoped to@local/python-example, the pruned tree lacks the sibling member and:uv sync --all-packages --lockedexits 1 —The lockfile at 'uv.lock' needs to be updated, but '--locked' was provided.(uv re-discovers the workspace frommembersand resolves 12 packages against a 13-package lock)uv run --frozen --all-packages pytestexits 2 —error: Failed to determine installation plan/Caused by: Distribution not found at: file:///…/out/libs/@local/python-example-twoi.e. the first additional Python package would have broken every pruned Python CI job.
Interim fix (in this PR):
prune.pyresolves the uv workspace members from the rootpyproject.tomland adds them to theturbo prunescopes, so every pruned tree keeps the full Python workspace — including the members' entries in the prunedyarn.lock(a plain directory-copy after pruning is not enough: yarn then fails withThis package doesn't seem to be present in your lockfile). Members without apackage.jsonfall back to a directory copy. The two-package validation is now committed (python-example-two), so every pruned CI job carries both members and this path is exercised continuously instead of only in a local reproduction. Verified on the committed tree: pruned to a Python scope and to an unrelated scope (@local/advanced-types), both pruned trees passuv sync --all-packages --lockedandturbo run lint:ruff lint:basedpyright test:unitfor both packages (6/6 tasks).For the wider debate on dropping
turbo prunefrom CI: theprune-repositoryaction (customprune.pywrappingturbo prune) is a CI-only checkout optimization —lint.yml(×1),test.yml(×2),bench.yml(×4),codspeed.yml(×1),deploy.yml(×1, sourcemaps job) — while the five docker image builds (hash-graph,hash-frontend,hash-api,hash-ai-worker-ts,hash-integration-worker) call turbo's nativeturbo prune --scope=… --dockerdirectly in their Dockerfiles and never useprune.py, so dropping the CI action would not leave the docker builds' prune path unexercised — they never shared code with it. Separately: no uv download cache in CI yet; worth adding?Line length: picked 100 (Rust width); repo conventions are split (JS 80, SQL 120, ruff default 88).
🚫 Blocked by
Nothing — but this is a spike; the real implementation is Bilal's roadmap item.
Pre-Merge Checklist 🚀
🚢 Has this modified a publishable library?
This PR:
📜 Does this require a change to the docs?
The changes in this PR:
🕸️ Does this require a change to the Turbo Graph?
The changes in this PR:
turbo.json's have been updated to reflect thisuv run --frozenwithout--all-packagessilently falls back to aruff/pytestfound onPATHif the tool isn't in the venv — the scripts always pass--all-packagesto avoid this.🛡 What tests cover this?
libs/@local/python-example/tests/test_greeting.py(3 tests, incl. async) andlibs/@local/python-example-two/tests/test_team_greetings.py(2 tests, imports across the workspace dependency) — run byturbo run test:unit.Verified locally (uv 0.11.18, turbo 2.6.3, yarn 4.16.0, CPython 3.14.6):
uv lock,uv sync --all-packages --locked, and for both packagesuv run --frozen --all-packages ruff check .,… ruff format --check .,… basedpyright --project ../../.. .(strict mode confirmed to really analyze files via an intentional error),… pytest(3 + 2 passed),turbo run lint:ruff lint:basedpyright test:unitfiltered to each package and to both (6/6 tasks,FULL TURBOcache hits on re-run), the pruned-tree validation described in open question 4,yarn constraints,yarn lint:license-in-workspaces, taplo/oxfmt/markdownlint on the changed files.Not verified in this environment: the actual GitHub Actions runs (incl. mise-provisioned uv on runners) and the Rust
sort-package-jsoncheck (mise run lint:package-jsonneeds a cargo build) — watch the checks on this PR.❓ How to test this?
uv sync --all-packagesturbo run lint:ruff lint:basedpyright test:unit --filter '@local/python-example'