Skip to content

[#11985] fix(openapi): correct the OpenLineage translation in the lineage schemas#11989

Open
nevzheng wants to merge 1 commit into
apache:mainfrom
nevzheng:claude/gravitino-11985-lineage-oneof
Open

[#11985] fix(openapi): correct the OpenLineage translation in the lineage schemas#11989
nevzheng wants to merge 1 commit into
apache:mainfrom
nevzheng:claude/gravitino-11985-lineage-oneof

Conversation

@nevzheng

@nevzheng nevzheng commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

A minimal correction to the lineage event schemas in docs/open-api/lineage.yaml — two line edits and two deletions, no structural change:

  • remove the two not: { required: [...] } blocks on DatasetEvent/JobEvent;
  • fix the eventType example — the pipe-joined START|RUNNING|COMPLETE|ABORT|FAIL|OTHER is a single string that is not a member of the enum; replace it with START;
  • change the /lineage request-body union from oneOf to anyOf.

BaseEvent, the allOf composition, and the open schemas all stay exactly as they are.

Why are the changes needed?

These schemas are translated from the OpenLineage spec, whose source of truth is JSON Schema draft 2020-12. Two constructs did not survive the translation into this OpenAPI 3.0.3 document, and this PR is the faithful re-translation.

1. The not blocks are invalid here, and ineffective everywhere. They express OpenLineage's intent that static events carry no run state. But they reference properties (job, run) the enclosing schemas never declare — which is what Redocly's no-required-schema-properties-undefined flags — and not: {required: [job, run]} only rejects a payload carrying both fields, so a DatasetEvent carrying just run validated anyway. Code generators ignore not entirely.

2. oneOf cannot survive their removal. The event schemas are deliberately open (OpenLineage's extensibility model), so a valid RunEvent — which carries job — also matches JobEvent. Under a strict oneOf that is two matches, and the most common event type is rejected. The not blocks were the only thing masking this. Verified with an ajv payload matrix against the bundled spec:

payload old oneOf not removed, oneOf kept this PR (anyOf)
valid RunEvent ACCEPT REJECT (also matches JobEvent) ACCEPT
valid JobEvent ACCEPT ACCEPT ACCEPT
valid DatasetEvent ACCEPT ACCEPT ACCEPT
garbage (no required fields) REJECT REJECT REJECT

Why anyOf is the right encoding. With open, overlapping variants, "matches exactly one" is unsatisfiable by construction; "matches at least one well-formed event" is the contract the endpoint actually offers. The per-event required sets keep malformed payloads out, and exact event discrimination stays where it already lives — the server's typed deserialization. This mirrors how OpenLineage itself handles the gap (its OpenAPI file offloads validation to the JSON Schema layer) and how Marquez, the reference implementation, accepts a single loose LineageEvent.

Clears 4 Redocly recommended-strict findings (3 × no-required-schema-properties-undefined, 1 × no-invalid-schema-examples).

Part of #11985

Does this PR introduce any user-facing change?

No. Spec/docs-only. Every payload the old spec accepted is still accepted; the schemas remain open, so OpenLineage producer extensibility is preserved. No server behavior change.

How was this patch tested?

  • redocly lint --extends=recommended-strict — the 4 lineage findings are cleared (10 → 6 against main), no new findings.
  • redocly bundle — bundles to valid openapi.json.
  • ajv payload matrix (table above) against three bundles — the original, a not-removed/oneOf-kept variant, and this PR — confirming all valid events accept and payloads missing every event's required fields reject.

@nevzheng nevzheng force-pushed the claude/gravitino-11985-lineage-oneof branch from f615ffb to 843e5aa Compare July 11, 2026 06:31
@nevzheng nevzheng marked this pull request as draft July 11, 2026 06:33
@nevzheng nevzheng force-pushed the claude/gravitino-11985-lineage-oneof branch from 843e5aa to 39740fd Compare July 11, 2026 07:19
@nevzheng nevzheng changed the title [#11985] fix(openapi): make the lineage event oneOf sound under OpenAPI 3.0 [#11985] fix(openapi): correct the OpenLineage translation in the lineage schemas Jul 11, 2026
@nevzheng nevzheng marked this pull request as ready for review July 11, 2026 07:20
…he lineage schemas

The lineage event schemas are translated from OpenLineage, whose source of
truth is JSON Schema draft 2020-12. Two constructs did not survive the
translation to OpenAPI 3.0.3:

- not: { required: [...] } on DatasetEvent/JobEvent references properties
  those schemas never declare (a recommended-strict error), and only rejects
  a payload carrying every listed field, so it under-enforces its own intent.
- eventType's example was the pipe-joined pseudo-value
  START|RUNNING|COMPLETE|ABORT|FAIL|OTHER, not a member of the enum.

Remove the not blocks, fix the example, and change the /lineage request-body
union from oneOf to anyOf. anyOf is the faithful 3.0 encoding: the event
schemas are deliberately open (OpenLineage extensibility), so a valid
RunEvent also matches JobEvent and a strict oneOf rejects it once the not
blocks are gone; verified with an ajv payload matrix against the bundled
spec. Exact event discrimination stays where it already lives, in the
server's typed deserialization.

Part of apache#11985
@nevzheng nevzheng force-pushed the claude/gravitino-11985-lineage-oneof branch from 39740fd to cbbbe73 Compare July 11, 2026 07:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant