From 89071d244c639defb4c81d74cbbd3207d0b20353 Mon Sep 17 00:00:00 2001 From: Tokarzewski Date: Wed, 1 Jul 2026 16:02:31 +0200 Subject: [PATCH 1/2] =?UTF-8?q?Add=20Mojo=20language=20support=20(wiring?= =?UTF-8?q?=20only=20=E2=80=94=20grammar=20vendoring=20deferred)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Mojo (Modular) is a Python-superset systems language. Wire the standard language path — enum, extraction spec, language entry, test cases, and registration — so the hook is ready for the grammar to be re-vendored after provenance audit. Tracking: #737 Grammar: lsh/tree-sitter-mojo @ 33193a99afe6, MIT, ABI 15, C scanner (community grammar — not in nvim-treesitter/Helix registries) The grammar's node types mirror Python's, so the spec reuses the py_* arrays and overrides only class types: - "struct"/"class" → class_definition - "trait"/"__extension" → trait_definition / extension_definition (Interface) - "fn"/"def" → function_definition - "alias NAME = value" → assignment (no dedicated node in upstream grammar) Signed-off-by: Tokarzewski --- README.md | 14 +++++++------- THIRD_PARTY.md | 2 +- internal/cbm/cbm.h | 1 + internal/cbm/lang_specs.c | 19 +++++++++++++++++++ internal/cbm/vendored/grammars/MANIFEST.md | 1 + scripts/new-languages.json | 15 +++++++++++++++ src/discover/language.c | 5 +++++ tests/test_grammar_labels.c | 1 + tests/test_grammar_regression.c | 8 ++++++++ 9 files changed, 58 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index b48a297f0..bc91a44ed 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ [![License](https://img.shields.io/badge/license-MIT-green)](LICENSE) [![CI](https://img.shields.io/github/actions/workflow/status/DeusData/codebase-memory-mcp/dry-run.yml?label=CI)](https://github.com/DeusData/codebase-memory-mcp/actions/workflows/dry-run.yml) [![Tests](https://img.shields.io/badge/tests-5604_passing-brightgreen)](https://github.com/DeusData/codebase-memory-mcp) -[![Languages](https://img.shields.io/badge/languages-158-orange)](https://github.com/DeusData/codebase-memory-mcp) +[![Languages](https://img.shields.io/badge/languages-159-orange)](https://github.com/DeusData/codebase-memory-mcp) [![Hybrid LSP](https://img.shields.io/badge/Hybrid_LSP-9_languages-blue)](#hybrid-lsp) [![Agents](https://img.shields.io/badge/agents-11-purple)](https://github.com/DeusData/codebase-memory-mcp) [![Pure C](https://img.shields.io/badge/pure_C-zero_dependencies-blue)](https://github.com/DeusData/codebase-memory-mcp) @@ -16,7 +16,7 @@ **The fastest and most efficient code intelligence engine for AI coding agents.** Full-indexes an average repository in milliseconds, the Linux kernel (28M LOC, 75K files) in 3 minutes. Answers structural queries in under 1ms. Ships as a single static binary for macOS, Linux, and Windows — download, run `install`, done. -High-quality parsing through [tree-sitter](https://tree-sitter.github.io/tree-sitter/) AST analysis across all 158 languages, enhanced with [**Hybrid LSP** semantic type resolution](#hybrid-lsp) for Python, TypeScript / JavaScript / JSX / TSX, PHP, C#, Go, C, C++, Java, Kotlin, and Rust — producing a persistent knowledge graph of functions, classes, call chains, HTTP routes, and cross-service links. 14 MCP tools. Zero dependencies. Plug and play across 11 coding agents. +High-quality parsing through [tree-sitter](https://tree-sitter.github.io/tree-sitter/) AST analysis across all 159 languages, enhanced with [**Hybrid LSP** semantic type resolution](#hybrid-lsp) for Python, TypeScript / JavaScript / JSX / TSX, PHP, C#, Go, C, C++, Java, Kotlin, and Rust — producing a persistent knowledge graph of functions, classes, call chains, HTTP routes, and cross-service links. 14 MCP tools. Zero dependencies. Plug and play across 11 coding agents. > **Research** — The design and benchmarks behind this project are described in the preprint [*Codebase-Memory: Tree-Sitter-Based Knowledge Graphs for LLM Code Exploration via MCP*](https://arxiv.org/abs/2603.27277) (arXiv:2603.27277). Evaluated across 31 real-world repositories: 83% answer quality, 10× fewer tokens, 2.1× fewer tool calls vs. file-by-file exploration. @@ -32,7 +32,7 @@ High-quality parsing through [tree-sitter](https://tree-sitter.github.io/tree-si - **Extreme indexing speed** — Linux kernel (28M LOC, 75K files) in 3 minutes. RAM-first pipeline: LZ4 compression, in-memory SQLite, fused Aho-Corasick pattern matching. Memory released after indexing. - **Plug and play** — single static binary for macOS (arm64/amd64), Linux (arm64/amd64), and Windows (amd64). No Docker, no runtime dependencies, no API keys. Download → `install` → restart agent → done. -- **158 languages** — vendored tree-sitter grammars compiled into the binary. Nothing to install, nothing that breaks. +- **159 languages** — vendored tree-sitter grammars compiled into the binary. Nothing to install, nothing that breaks. - **120x fewer tokens** — 5 structural queries: ~3,400 tokens vs ~412,000 via file-by-file search. One graph query replaces dozens of grep/read cycles. - **11 agents, one command** — `install` auto-detects Claude Code, Codex CLI, Gemini CLI, Zed, OpenCode, Antigravity, Aider, KiloCode, VS Code, OpenClaw, and Kiro — configures MCP entries, instruction files, and pre-tool hooks for each. - **Built-in graph visualization** — 3D interactive UI at `localhost:9749` (optional UI binary variant). @@ -168,7 +168,7 @@ Removes all agent configs, skills, hooks, and instructions. Does not remove the - `SEMANTICALLY_RELATED` (vocabulary-mismatch, same-language, score ≥ 0.80) ### Indexing pipeline -- **158 vendored tree-sitter grammars** compiled into the binary +- **159 vendored tree-sitter grammars** compiled into the binary - **Generic package / module resolution** — bare specifiers like `@myorg/pkg`, `github.com/foo/bar`, `use my_crate::foo` resolved via manifest scanning (`package.json`, `go.mod`, `Cargo.toml`, `pyproject.toml`, `composer.json`, `pubspec.yaml`, `pom.xml`, `build.gradle`, `mix.exs`, `*.gemspec`) - **Infrastructure-as-code indexing** — Dockerfiles, Kubernetes manifests, Kustomize overlays as graph nodes - **[Hybrid LSP semantic type resolution](#hybrid-lsp)** for Python, TypeScript / JavaScript / JSX / TSX, PHP, C#, Go, C, C++, Java, Kotlin, and Rust — a lightweight C implementation of language type-resolution algorithms, structurally inspired by and compatible with major language servers including tsserver / typescript-go, pyright, gopls, Roslyn, Eclipse JDT, and rust-analyzer (parameter binding, return-type inference, generic substitution, JSX component dispatch, JSDoc inference for plain JS files, namespace + trait + late-static-binding resolution for PHP, file-scoped namespaces + records + LINQ method syntax for C#, class-hierarchy + overload + lambda resolution for Java, extension-function + scope-function resolution for Kotlin, trait-method + UFCS resolution for Rust) @@ -508,14 +508,14 @@ codebase-memory-mcp ships a **lightweight C implementation of language type-reso **Two-layer architecture:** -1. **Tree-sitter pass** — fast, syntactic, runs for every one of the 158 languages. Extracts definitions, calls, imports. +1. **Tree-sitter pass** — fast, syntactic, runs for every one of the 159 languages. Extracts definitions, calls, imports. 2. **Hybrid LSP pass** — type-aware, runs above the tree-sitter pass per-language. Refines call edges using the import graph plus a per-file or pre-built cross-file definition registry. Languages without a Hybrid LSP pass yet fall back to textual resolution, so you always get *some* answer. The result is a knowledge graph accurate enough to drive `trace_path` across packages, inheritance hierarchies, and stdlib calls — without paying for a language server process per project. ## Language Support -158 languages, all parsed via vendored tree-sitter grammars compiled into the binary. Benchmarked against 64 real open-source repositories (78 to 49K nodes): +159 languages, all parsed via vendored tree-sitter grammars compiled into the binary. Benchmarked against 64 real open-source repositories (78 to 49K nodes): | Tier | Score | Languages | |------|-------|-----------| @@ -540,7 +540,7 @@ src/ traces/ Runtime trace ingestion ui/ Embedded HTTP server + 3D graph visualization foundation/ Platform abstractions (threads, filesystem, logging, memory) -internal/cbm/ Vendored tree-sitter grammars (158 languages) + AST extraction engine +internal/cbm/ Vendored tree-sitter grammars (159 languages) + AST extraction engine ``` ## Security diff --git a/THIRD_PARTY.md b/THIRD_PARTY.md index e5f7dc977..40f26f732 100644 --- a/THIRD_PARTY.md +++ b/THIRD_PARTY.md @@ -26,7 +26,7 @@ The core runtime headers in `internal/cbm/vendored/common/tree_sitter/` ## Tree-sitter Grammars -158 pre-generated parsers are vendored in `internal/cbm/vendored/grammars//` +159 pre-generated parsers are vendored in `internal/cbm/vendored/grammars//` (generated `parser.c` plus `scanner.c` where applicable, compiled statically). Each grammar is the work of its upstream authors and each grammar directory contains the upstream `LICENSE` file. diff --git a/internal/cbm/cbm.h b/internal/cbm/cbm.h index cc3607ee3..d6d3db44f 100644 --- a/internal/cbm/cbm.h +++ b/internal/cbm/cbm.h @@ -170,6 +170,7 @@ typedef enum { CBM_LANG_QML, // Qt QML (Qt Modeling Language — declarative UI + embedded JS) CBM_LANG_CFSCRIPT, // CFML script dialect (.cfc components — Lucee/ColdFusion) CBM_LANG_CFML, // CFML tag dialect (.cfm templates — Lucee/ColdFusion) + CBM_LANG_MOJO, // Mojo (Modular — Python-superset systems language; .mojo / .🔥) CBM_LANG_COUNT } CBMLanguage; diff --git a/internal/cbm/lang_specs.c b/internal/cbm/lang_specs.c index 68d2afadd..6da5048b6 100644 --- a/internal/cbm/lang_specs.c +++ b/internal/cbm/lang_specs.c @@ -164,6 +164,7 @@ extern const TSLanguage *tree_sitter_apex(void); extern const TSLanguage *tree_sitter_soql(void); extern const TSLanguage *tree_sitter_sosl(void); extern const TSLanguage *tree_sitter_pine(void); +extern const TSLanguage *tree_sitter_mojo(void); // -- Empty sentinel -- static const char *empty_types[] = {NULL}; @@ -205,6 +206,18 @@ static const char *py_var_types[] = {"assignment", "augmented_assignment", NULL} static const char *py_throw_types[] = {"raise_statement", NULL}; static const char *py_decorator_types[] = {"decorator", NULL}; +// ==================== MOJO ==================== +// Mojo (Modular) is a Python superset; the lsh/tree-sitter-mojo grammar is +// forked from tree-sitter-python, so every node type mirrors Python exactly +// EXCEPT the class array — Mojo's "struct"/"class" both parse as +// class_definition, but "trait" and the "__extension" form get their own +// nodes. So the spec reuses the py_* arrays and overrides only the class +// types. ("fn"/"def" both parse as function_definition; compile-time +// "alias NAME = value" has no dedicated node and is recovered as an +// `assignment`, so it falls under py_var_types like ordinary `var` fields.) +static const char *mojo_class_types[] = {"class_definition", "trait_definition", + "extension_definition", NULL}; + // ==================== JAVASCRIPT ==================== static const char *js_func_types[] = {"function_declaration", "generator_function_declaration", "function_expression", "arrow_function", @@ -2005,6 +2018,12 @@ static const CBMLangSpec lang_specs[CBM_LANG_COUNT] = { empty_types, empty_types, NULL, empty_types, NULL, NULL, tree_sitter_cfml, NULL}, + // CBM_LANG_MOJO (Python-derived; reuses py_* arrays, only class types differ) + [CBM_LANG_MOJO] = {CBM_LANG_MOJO, py_func_types, mojo_class_types, empty_types, py_module_types, + py_call_types, py_import_types, py_import_from_types, py_branch_types, + py_var_types, py_var_types, py_throw_types, NULL, py_decorator_types, + py_env_funcs, py_env_members, tree_sitter_mojo, NULL}, + // CBM_LANG_GLEAM [CBM_LANG_GLEAM] = {CBM_LANG_GLEAM, gleam_func_types, gleam_class_types, gleam_field_types, gleam_module_types, gleam_call_types, gleam_import_types, empty_types, diff --git a/internal/cbm/vendored/grammars/MANIFEST.md b/internal/cbm/vendored/grammars/MANIFEST.md index 75c5b4ff8..6d6c9fc18 100644 --- a/internal/cbm/vendored/grammars/MANIFEST.md +++ b/internal/cbm/vendored/grammars/MANIFEST.md @@ -130,6 +130,7 @@ Guarded by the `contract_all_grammars_in_graph` graph-breadth test in | matlab | 15 | acristoffers/tree-sitter-matlab | `c2390a59016f` | VERIFIED-BOTH | ✅ | | mermaid | 14 | monaqa/tree-sitter-mermaid | `90ae195b3193` | VERIFIED-BOTH | ✅ | | meson | 15 | tree-sitter-grammars/tree-sitter-meson | `c84f3540624b` | VERIFIED-BOTH | ✅ | +| mojo | 15 | lsh/tree-sitter-mojo | `33193a99afe6` | PENDING (vendored files removed pending provenance audit — #737) | ⏳ | | nasm | 14 | naclsn/tree-sitter-nasm | `d1b3638d017f` | VERIFIED-BOTH | ✅ | | nickel | 15 | nickel-lang/tree-sitter-nickel | `b5b6cc3bc7b9` | VERIFIED-BOTH | ✅ | | nix | 13 | nix-community/tree-sitter-nix | `eabf96807ea4` | VERIFIED-BOTH | ✅ | diff --git a/scripts/new-languages.json b/scripts/new-languages.json index b03682e8d..2e713fc36 100644 --- a/scripts/new-languages.json +++ b/scripts/new-languages.json @@ -1291,5 +1291,20 @@ "filenames": [], "has_scanner": true, "module_root": "source_file" + }, + { + "name": "mojo", + "enum": "MOJO", + "display": "Mojo", + "ts_func": "tree_sitter_mojo", + "repo": "https://github.com/lsh/tree-sitter-mojo", + "subdir": "", + "extensions": [ + ".mojo", + ".🔥" + ], + "filenames": [], + "has_scanner": true, + "module_root": "module" } ] \ No newline at end of file diff --git a/src/discover/language.c b/src/discover/language.c index a0254306c..397f6c0a5 100644 --- a/src/discover/language.c +++ b/src/discover/language.c @@ -186,6 +186,10 @@ static const ext_entry_t EXT_TABLE[] = { /* Meson */ {".meson", CBM_LANG_MESON}, + /* Mojo — .mojo and the .🔥 (U+1F525) fire-emoji extension */ + {".mojo", CBM_LANG_MOJO}, + {".🔥", CBM_LANG_MOJO}, + /* Nix */ {".nix", CBM_LANG_NIX}, @@ -834,6 +838,7 @@ static const char *LANG_NAMES[CBM_LANG_COUNT] = { [CBM_LANG_APEX] = "Apex", [CBM_LANG_SOQL] = "SOQL", [CBM_LANG_SOSL] = "SOSL", + [CBM_LANG_MOJO] = "Mojo", }; diff --git a/tests/test_grammar_labels.c b/tests/test_grammar_labels.c index 121fc01cd..c6c62ffe6 100644 --- a/tests/test_grammar_labels.c +++ b/tests/test_grammar_labels.c @@ -100,6 +100,7 @@ static const LabelGolden LABEL_GOLDENS[] = { {"swift", "Class:1,Function:1,Module:1"}, {"scala", "Class:1,Function:1,Method:1,Module:1"}, {"gdscript", "Function:1,Module:1"}, + {"mojo", "Class:1,Function:1,Interface:1,Method:1,Module:1,Variable:1"}, {"groovy", "Class:1,Method:1,Module:1"}, {"zig", "Function:2,Module:1"}, {"solidity", "Class:1,Function:1,Method:1,Module:1"}, diff --git a/tests/test_grammar_regression.c b/tests/test_grammar_regression.c index b578fc0b0..52b48f149 100644 --- a/tests/test_grammar_regression.c +++ b/tests/test_grammar_regression.c @@ -100,6 +100,14 @@ const GrammarCase CBM_GRAMMAR_CASES[] = { {"swift", CBM_LANG_SWIFT, "a.swift", "func foo() {}\nclass A {}\n", 2, {"foo", "A", NULL}}, {"scala", CBM_LANG_SCALA, "a.scala", "object A {\n def foo() = 1\n}\n", 1, {"A", NULL}}, {"gdscript", CBM_LANG_GDSCRIPT, "a.gd", "func foo():\n pass\n", 1, {"foo", NULL}}, + /* Mojo: fn/def -> function, struct -> class, trait -> interface */ + {"mojo", + CBM_LANG_MOJO, + "a.mojo", + "fn foo() -> Int:\n return 1\nstruct Bar:\n var x: Int\ntrait Baz:\n fn m(self): " + "...\n", + 3, + {"foo", "Bar", "Baz", NULL}}, {"groovy", CBM_LANG_GROOVY, "a.groovy", "class A {\n def foo() {}\n}\n", 1, {"A", NULL}}, {"zig", CBM_LANG_ZIG, "a.zig", "fn foo() void {}\nfn bar() void {}\n", 1, {"foo", NULL}}, {"solidity", From 0aa1aedda121636a2979e85108f12f8bad96bf91 Mon Sep 17 00:00:00 2001 From: Tokarzewski Date: Wed, 1 Jul 2026 17:19:50 +0200 Subject: [PATCH 2/2] Merge remote-tracking branch 'upstream/main' into feat/mojo-language-support MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Resolved THIRD_PARTY.md grammar count (159→160) and README.md badge to match upstream/main's ObjectScript additions. --- README.md | 4 ++-- THIRD_PARTY.md | 4 ---- 2 files changed, 2 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 77f6d6f7b..d6102dfb4 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ [![License](https://img.shields.io/badge/license-MIT-green)](LICENSE) [![CI](https://img.shields.io/github/actions/workflow/status/DeusData/codebase-memory-mcp/dry-run.yml?label=CI)](https://github.com/DeusData/codebase-memory-mcp/actions/workflows/dry-run.yml) [![Tests](https://img.shields.io/badge/tests-5604_passing-brightgreen)](https://github.com/DeusData/codebase-memory-mcp) -[![Languages](https://img.shields.io/badge/languages-159-orange)](https://github.com/DeusData/codebase-memory-mcp) +[![Languages](https://img.shields.io/badge/languages-160-orange)](https://github.com/DeusData/codebase-memory-mcp) [![Hybrid LSP](https://img.shields.io/badge/Hybrid_LSP-9_languages-blue)](#hybrid-lsp) [![Agents](https://img.shields.io/badge/agents-11-purple)](https://github.com/DeusData/codebase-memory-mcp) [![Pure C](https://img.shields.io/badge/pure_C-zero_dependencies-blue)](https://github.com/DeusData/codebase-memory-mcp) @@ -16,7 +16,7 @@ **The fastest and most efficient code intelligence engine for AI coding agents.** Full-indexes an average repository in milliseconds, the Linux kernel (28M LOC, 75K files) in 3 minutes. Answers structural queries in under 1ms. Ships as a single static binary for macOS, Linux, and Windows — download, run `install`, done. -High-quality parsing through [tree-sitter](https://tree-sitter.github.io/tree-sitter/) AST analysis across all 159 languages, enhanced with [**Hybrid LSP** semantic type resolution](#hybrid-lsp) for Python, TypeScript / JavaScript / JSX / TSX, PHP, C#, Go, C, C++, Java, Kotlin, and Rust — producing a persistent knowledge graph of functions, classes, call chains, HTTP routes, and cross-service links. 14 MCP tools. Zero dependencies. Plug and play across 11 coding agents. +High-quality parsing through [tree-sitter](https://tree-sitter.github.io/tree-sitter/) AST analysis across all 160 languages, enhanced with [**Hybrid LSP** semantic type resolution](#hybrid-lsp) for Python, TypeScript / JavaScript / JSX / TSX, PHP, C#, Go, C, C++, Java, Kotlin, and Rust — producing a persistent knowledge graph of functions, classes, call chains, HTTP routes, and cross-service links. 14 MCP tools. Zero dependencies. Plug and play across 11 coding agents. > **Research** — The design and benchmarks behind this project are described in the preprint [*Codebase-Memory: Tree-Sitter-Based Knowledge Graphs for LLM Code Exploration via MCP*](https://arxiv.org/abs/2603.27277) (arXiv:2603.27277). Evaluated across 31 real-world repositories: 83% answer quality, 10× fewer tokens, 2.1× fewer tool calls vs. file-by-file exploration. diff --git a/THIRD_PARTY.md b/THIRD_PARTY.md index 390810d8b..c6e8cb24c 100644 --- a/THIRD_PARTY.md +++ b/THIRD_PARTY.md @@ -26,11 +26,7 @@ The core runtime headers in `internal/cbm/vendored/common/tree_sitter/` ## Tree-sitter Grammars -<<<<<<< HEAD -159 pre-generated parsers are vendored in `internal/cbm/vendored/grammars//` -======= 160 pre-generated parsers are vendored in `internal/cbm/vendored/grammars//` ->>>>>>> upstream/main (generated `parser.c` plus `scanner.c` where applicable, compiled statically). Each grammar is the work of its upstream authors and each grammar directory contains the upstream `LICENSE` file.