Skip to content

feat(db): consolidate resource content onto a binary-safe /api/db/resource#59

Merged
line-o merged 11 commits into
eXist-db:developfrom
joewiz:feat/resource-api-consolidation
Jul 9, 2026
Merged

feat(db): consolidate resource content onto a binary-safe /api/db/resource#59
line-o merged 11 commits into
eXist-db:developfrom
joewiz:feat/resource-api-consolidation

Conversation

@joewiz

@joewiz joewiz commented Jun 12, 2026

Copy link
Copy Markdown
Member

[This PR was co-authored with Claude Code. -Joe]

Summary

Consolidates all resource-content transport onto a single, binary-safe /api/db/resource, and retires the JSON-envelope shape. The /api/db/resource endpoint, in the /api/db family, is identical in form to its siblings (collection/move/copy/properties/sync). This design supersedes the path-in-URL binary endpoints (#38, #56) and folds in the latest serialization work (#48), extending it to the full parameter vocabulary.

This is the outcome of a design review (with @line-o on #38 and #56, and in coordination with eXide, dashboard-next, and existdb-oxygen-plugin): the path-in-URL /api/resource/{path} broke the query-param convention of the /api/db family (it dangled under "default" in Swagger and doubled db in the URL), and the JSON envelope wasn't relied on by any client. A review of the clients confirmed they could drop it.

Rebased onto current develop — the decoded-UTF-8 names and db-core work it was originally stacked behind have since merged, so the diff here is the consolidation itself.

The consolidated contract

GET /api/db/resource?path=/db/… — returns the resource's raw content:

  • binary → bytes streamed as-is; XML/text → serialized from the stored node tree.
  • Content-Type = the stored mime type.
  • Serialization parameters (XML/text): the full W3C set (method, indent, omit-xml-declaration, encoding, media-type, item-separator, standalone, …) plus eXist's extensions (expand-xincludes, highlight-matches, add-exist-id, process-xsl-pi, jsonp, insert-final-newline). Omitted params fall through to the conf.xml defaults — defaults are a fallback, never a ceiling.
  • download=trueContent-Disposition: attachment.

PUT /api/db/resource?path=/db/…[&mime=…] — stores a raw request body (binary-safe):

  • the stored mime comes from the optional &mime query param, else it is inferred from the resource name; pass mime=application/octet-stream to force raw-byte storage of content that would otherwise be parsed. The request's own Content-Type is transport only (HTTP clients send unpredictable defaults, so &mime keeps it deterministic).
  • 201 Created (new) / 200 OK (overwrite), body { "path": "<canonical decoded path>" } so the caller can reconcile any server-side name normalization.

DELETE /api/db/resource?path=/db/… — unchanged.

Removed

  • The JSON-envelope GET/PUT shape (content-as-a-string-field).
  • meta=full and the X-Resource-* metadata headers.
  • The separate /api/resource/{path} and /api/db/resource/{path} path-in-URL endpoints.
  • runPath from every response (db-core get-resource and store); dbc:get-run-path deleted.

Note: runPath is derivable client-side and documented in the README:

function runPath(dbPath) {
  return dbPath.startsWith('/db/apps/')
    ? '/exist/apps/' + dbPath.slice('/db/apps/'.length)
    : '/exist/rest' + dbPath;
}

Metadata

  • Metadata stays at GET /api/db/properties?path= — clients fetch it separately (a review of the clients confirmed this is preferred; content reads need only mime + binary-ness, derivable from Content-Type).

Encoding

The endpoint speaks decoded UTF-8 on the wire (eXist-db/exist#6463): a client sends and receives café.xml, never caf%C3%A9.xml. The ?path= form aligns better with this than path-in-URL did — a single query token roaster decodes cleanly, with no per-segment encoding ambiguity. db-core's to-stored/to-display/resolve-stored map to/from the canonical stored form (and stay read-compatible with legacy full-encoded names). Known Phase-1 limitation: literal % in a name is ambiguous (eXist-db/exist#6463), to be made bijective later.

Serialization params — no eXist-core dependency

The eXist-specific serialization params (notably expand-xincludes) are emitted as exist:-namespaced children of output:serialization-parameters (the http://exist.sourceforge.net/NS/exist namespace), which eXist already honors — so they work on any eXist, with no dependency on eXist-db/exist#6447. expand-xincludes=no on read (the data-loss-safe round-trip of <xi:include> documents) is reachable everywhere.

Testing

  • Cypress (db.cy.js): raw GET round-trip (binary bytes intact, XQuery source not executed), serialization params (W3C + eXist extensions, including a real expand-xincludes case), raw PUT create/overwrite with canonical-path echo, DELETE, decoded-UTF-8 name round-trips, non-/db 400. Full suite green (200/200) against a clean existdb/existdb:latest container using CI's install → restart → cypress run sequence.
  • Store path: dbc:store decodes the raw octet-stream body to text for XML-class targets before handing it to xmldb:store, so an XML mime never receives a binary value (which errored on some builds and blocked the request on others); non-XML content is stored byte-for-byte.

joewiz added a commit to joewiz/eXide that referenced this pull request Jun 16, 2026
…cutover

Per review: a skipped (it.skip) test asserting existdb-openapi's /api/db/resource
honors expand-xincludes server-side. When the open path cuts over to that endpoint
(see Scope/notes), db:load-document's own serialization handling goes away; this
documents the expected contract and becomes a tripwire when it lands. Skipped until
the consolidated db-core (eXist-db/existdb-openapi#55 and eXist-db/existdb-openapi#59)
is in eXide's test environment.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@joewiz joewiz force-pushed the feat/resource-api-consolidation branch 2 times, most recently from 92ccced to 59a8a35 Compare June 18, 2026 23:36
@joewiz joewiz force-pushed the feat/resource-api-consolidation branch from 874adaa to 7901900 Compare July 8, 2026 03:05
@codacy-production

codacy-production Bot commented Jul 8, 2026

Copy link
Copy Markdown

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics -107 complexity · -2 duplication

Metric Results
Complexity -107
Duplication -2

View in Codacy

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

@joewiz joewiz marked this pull request as ready for review July 8, 2026 03:28

@line-o line-o left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A much wanted change to the API for sure.
Let's work on the implementation details.

Comment thread modules/db.xqm Outdated
Comment thread modules/db-core.xqm Outdated
@joewiz

joewiz commented Jul 8, 2026

Copy link
Copy Markdown
Member Author

[This response was prompted by Joe, drafted by Claude Code, and reviewed by Joe.]

Thanks @line-o — both addressed.

db:get-resource indentation (ca9e6b1). Split into three private helpers, each taking over one former nesting level: db:resolve-and-stream (resolve + 404), db:stream-content (mime, download header, binary-vs-serialize branch), and db:stream-serialized (serialize + the clean-400 on a bad param). The top function is now just the empty-path guard delegating into the chain, and each step reads top to bottom.

Colon instead of a dot on the wire (08d5b8e). Switched the eXist serializer extensions from an exist. prefix to exist:?exist:expand-xincludes=no — so the wire name matches the actual exist: serialization namespace. The original dot was a hedge against colons confusing OpenAPI/SDK codegen, but colons are valid in query-parameter and OpenAPI parameter names, and a generator has to sanitize either form to a legal identifier anyway, so the hedge bought nothing. Verified end to end against a clean existdb/existdb:latest: exist:expand-xincludes=no preserves <xi:include> verbatim, =yes expands it, and the full db.cy.js suite is green (76/76).

@line-o line-o self-requested a review July 9, 2026 09:32

@line-o line-o left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Much appreciated!

joewiz and others added 9 commits July 9, 2026 10:51
…t-db#38)

Adds GET/PUT /api/db/resource/{path} — a binary-safe, roaster-native resource
transport, the clean alternative to the JSON-envelope /api/db/resource (which is
text/base64-only). Closes the binary side of eXist-db#35/eXist-db#38 without a controller
workaround (Juri's concern about an /api/* side door).

- GET streams a binary resource's raw bytes with response:stream-binary (NOT via
  the serializer, which would emit base64 text and corrupt it — the established
  roaster pattern, the same path eXist's REST server uses); an XML/text resource
  is returned as a node so roaster serializes it once with its stored mime
  (returning a pre-serialized string would be XML-escaped a second time).
- PUT stores the raw request body via db-core (binary bodies arrive intact;
  mime inferred from the name), returning { stored, runPath } (201/200).

No exist-core or roaster change needed — binary transport works on stock eXist
via the existing response:stream-binary. (Zero-copy streaming of very large
stored binaries is a separate exist-core optimization, tracked with
exist-strategy.) Self-contained Cypress coverage (raw round-trip not
base64-mangled, 201/200 + stored/runPath, XML serialized with mime, 404); full
db suite stays green (59 + 4).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…in headers, split tests

Per @line-o's review of eXist-db#56:
- Move the path-in-URL raw endpoint from /api/db/resource/{path} to
  /api/resource/{path}: the {path} already begins with /db, so the old form
  produced a redundant /api/db/resource/db/apps/... Handlers unchanged.
- meta=full now returns the resource's metadata as X-Resource-* response
  headers (owner/group/mode/size/created/last-modified; acl JSON-encoded in
  X-Resource-Acl, omitted when empty) instead of flattening it into the JSON
  body. db-core still returns metadata in the result map for in-process
  callers; only the HTTP wrapper lifts it into headers, so the body stays the
  resource content.
- Split the combined serialization it() blocks into one-behavior-each, and
  update the meta=full test to assert headers.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ource

Retire the JSON envelope and the path-in-URL endpoints; all resource content now
flows through the query-param /api/db/resource, identical in shape to its db
siblings.

- GET /api/db/resource?path= → raw content: binary streamed as-is; XML/text
  serialized from the node tree (eXist has no raw byte form for XML). Content-Type
  is the stored mime. download=true → Content-Disposition: attachment.
- PUT /api/db/resource?path= → raw request body (binary-safe); mime from
  Content-Type or inferred from the name; returns { path } (201/200) so the caller
  can reconcile name normalization.
- Serialization params: full W3C vocabulary + eXist extensions via the output:
  namespace (eXist-db/exist#6447 — expand-xincludes, highlight-matches,
  add-exist-id, process-xsl-pi, jsonp, insert-final-newline). Unsupported params
  on a pre-#6447 eXist return a clean 400.
- Removed: JSON envelope, meta=full + X-Resource-* headers, runPath (derivable
  client-side), the /api/resource/{path} and /api/db/resource/{path} endpoints,
  and dbc:get-run-path.
- Metadata stays at GET /api/db/properties.

Supersedes eXist-db#38, eXist-db#56; folds + extends eXist-db#48. Depends on eXist-db#54, eXist-db#55.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- db.cy.js: store via a raw store() helper (path/&mime in the query, raw body);
  GET assertions read the raw body, not a JSON envelope; fold the binary/source/
  download cases in; drop the runPath and meta=full cases (both removed); add the
  missing-path 400 and the eXist-extension-param graceful-degradation cases.
- README: document /api/db/resource as the single binary-safe content endpoint and
  the client-side runPath derivation formula.
(db-binary.cy.js removal folded into the preceding code commit.)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…op #6447 dependency)

db-core's serialization-params emitted the eXist extension params
(expand-xincludes, highlight-matches, add-exist-id, process-xsl-pi, jsonp,
insert-final-newline) in the W3C output: namespace, which only reaches
fn:serialize with eXist-db/exist#6447. eXist already honors these as
exist:-namespaced (http://exist.sourceforge.net/NS/exist) children of
output:serialization-parameters, so emit the extensions there instead;
standard W3C params stay in output:. expand-xincludes=no is now reachable
on any eXist — the data-loss-safe read for round-tripping <xi:include>
documents — without #6447.

- db-core.xqm: choose the namespace per param (extensions -> exist:, standard -> output:)
- api.json: drop the "requires #6447" notes from the six extension params
- db.xqm: reword the serialization-error comment (no longer #6447-specific)
- db.cy.js: replace the weak "honored-or-400" test with a real expand-xincludes
  round-trip (=no preserves <xi:include>, =yes expands it)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…mp to 0.12.0

The W3C serialization parameters (indent, omit-xml-declaration, …) keep their
plain names, but the implementation-defined eXist extensions now carry an
"exist." prefix on the wire (exist.expand-xincludes, exist.highlight-matches,
exist.add-exist-id, exist.process-xsl-pi, exist.jsonp,
exist.insert-final-newline). This makes the W3C-vs-eXist split explicit in the
query contract and mirrors the serialization model (W3C output: namespace vs the
exist: namespace). A dot is used rather than a colon so the names stay clean for
OpenAPI tooling / SDK codegen.

db-core reads each eXist extension from its exist.-prefixed key but still emits
the canonical local-name in the exist: namespace, so the serialized output is
unchanged. api.json renames the six parameters; the resource spec updates the
expand-xincludes case accordingly. info.version -> 0.12.0.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…gacy-name read-compat in GET

Cleanup surfaced once eXist core (eXist-db/exist#6497) fixed the raw binary PUT,
which had been masking these in the cypress suite.

- create-collection: strict by default — a missing parent is now a clean 409
  (was a raw exception). Pass `recursive: true` to create intermediate
  collections (mkdir -p). api.json gains the `recursive` body property and the
  409 response; db-core gains the recursive arity + dbc:ensure-collection-path.
- db:error-response now pins the response media-type to application/json. An
  error mapped to a status the route doesn't declare in api.json otherwise hits
  roaster's application/xml fallback and the error map fails to serialize
  (SENR0001). Hardens every db error path. See eeditiones/roaster#127.
- db:get-resource resolved its existence check / binary streaming with
  dbc:to-stored, so it 404'd on legacy full-encoded names that
  dbc:get-resource / dbc:properties resolve. Made dbc:resolve-stored public and
  use it in the wrapper.
- tests: migrate the remaining store call-sites (permissions setup; query.cy.js,
  query_pool_reuse, query_scope before-hooks) off the pre-consolidation
  {path,content,mime-type} envelope to the raw transport; make the raw-transport
  block self-sufficient; add strict-409 + recursive create coverage.

Full cypress suite green (194/194) against an eXist build carrying #6497.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
PUT /api/db/resource delivers content as bytes (octet-stream transport),
and dbc:store passed those bytes straight to xmldb:store. When the target
mime routes through eXist's XML parser (application/xml, text/html,
image/svg+xml, any +xml, or an XML-class name when mime is inferred),
handing it a binary value fails — it errors on some builds and, on the
current existdb/existdb:latest, blocks the request while holding a broker,
so a run of XML stores exhausts the pool and every later request hangs.

Decode the bytes to a string first for XML-class targets so the parser
sees the actual markup; non-XML targets (json, xquery, images,
octet-stream, …) still store the bytes verbatim. Adds a dbc:xml-parsed-target
helper keyed on the explicit mime, else the resource-name extension.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Addresses Juri's review — db:get-resource nested four levels deep
(try → if/else → if/else → if/else). Extract db:resolve-and-stream,
db:stream-content, and db:stream-serialized, each encapsulating one
level, so the flow reads top to bottom.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
joewiz and others added 2 commits July 9, 2026 10:51
… wire

Addresses Juri's review — the eXist serializer extensions were wire-named
with an "exist." dot prefix; switch to "exist:" (e.g. ?exist:expand-xincludes=no)
so the wire name mirrors the actual exist: serialization namespace. Colons are
valid in query-parameter and OpenAPI parameter names, and SDK codegen sanitizes
either form, so the earlier dot rationale doesn't hold.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
eXist-db#61 merged to develop at 0.13.0, colliding with this branch's 0.13.0;
bump to 0.14.0 so the api-version guard passes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@joewiz joewiz force-pushed the feat/resource-api-consolidation branch from 08d5b8e to 8bd1157 Compare July 9, 2026 14:54
@line-o line-o merged commit e4abc43 into eXist-db:develop Jul 9, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants