[#11985] fix(openapi): correct the OpenLineage translation in the lineage schemas#11989
Open
nevzheng wants to merge 1 commit into
Open
[#11985] fix(openapi): correct the OpenLineage translation in the lineage schemas#11989nevzheng wants to merge 1 commit into
nevzheng wants to merge 1 commit into
Conversation
f615ffb to
843e5aa
Compare
843e5aa to
39740fd
Compare
…he lineage schemas
The lineage event schemas are translated from OpenLineage, whose source of
truth is JSON Schema draft 2020-12. Two constructs did not survive the
translation to OpenAPI 3.0.3:
- not: { required: [...] } on DatasetEvent/JobEvent references properties
those schemas never declare (a recommended-strict error), and only rejects
a payload carrying every listed field, so it under-enforces its own intent.
- eventType's example was the pipe-joined pseudo-value
START|RUNNING|COMPLETE|ABORT|FAIL|OTHER, not a member of the enum.
Remove the not blocks, fix the example, and change the /lineage request-body
union from oneOf to anyOf. anyOf is the faithful 3.0 encoding: the event
schemas are deliberately open (OpenLineage extensibility), so a valid
RunEvent also matches JobEvent and a strict oneOf rejects it once the not
blocks are gone; verified with an ajv payload matrix against the bundled
spec. Exact event discrimination stays where it already lives, in the
server's typed deserialization.
Part of apache#11985
39740fd to
cbbbe73
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
A minimal correction to the lineage event schemas in
docs/open-api/lineage.yaml— two line edits and two deletions, no structural change:not: { required: [...] }blocks onDatasetEvent/JobEvent;eventTypeexample — the pipe-joinedSTART|RUNNING|COMPLETE|ABORT|FAIL|OTHERis a single string that is not a member of the enum; replace it withSTART;/lineagerequest-body union fromoneOftoanyOf.BaseEvent, theallOfcomposition, and the open schemas all stay exactly as they are.Why are the changes needed?
These schemas are translated from the OpenLineage spec, whose source of truth is JSON Schema draft 2020-12. Two constructs did not survive the translation into this OpenAPI 3.0.3 document, and this PR is the faithful re-translation.
1. The
notblocks are invalid here, and ineffective everywhere. They express OpenLineage's intent that static events carry no run state. But they reference properties (job,run) the enclosing schemas never declare — which is what Redocly'sno-required-schema-properties-undefinedflags — andnot: {required: [job, run]}only rejects a payload carrying both fields, so aDatasetEventcarrying justrunvalidated anyway. Code generators ignorenotentirely.2.
oneOfcannot survive their removal. The event schemas are deliberately open (OpenLineage's extensibility model), so a validRunEvent— which carriesjob— also matchesJobEvent. Under a strictoneOfthat is two matches, and the most common event type is rejected. Thenotblocks were the only thing masking this. Verified with an ajv payload matrix against the bundled spec:oneOfnotremoved,oneOfkeptanyOf)RunEventJobEvent)JobEventDatasetEventWhy
anyOfis the right encoding. With open, overlapping variants, "matches exactly one" is unsatisfiable by construction; "matches at least one well-formed event" is the contract the endpoint actually offers. The per-eventrequiredsets keep malformed payloads out, and exact event discrimination stays where it already lives — the server's typed deserialization. This mirrors how OpenLineage itself handles the gap (its OpenAPI file offloads validation to the JSON Schema layer) and how Marquez, the reference implementation, accepts a single looseLineageEvent.Clears 4 Redocly
recommended-strictfindings (3 ×no-required-schema-properties-undefined, 1 ×no-invalid-schema-examples).Part of #11985
Does this PR introduce any user-facing change?
No. Spec/docs-only. Every payload the old spec accepted is still accepted; the schemas remain open, so OpenLineage producer extensibility is preserved. No server behavior change.
How was this patch tested?
redocly lint --extends=recommended-strict— the 4 lineage findings are cleared (10 → 6 againstmain), no new findings.redocly bundle— bundles to validopenapi.json.not-removed/oneOf-kept variant, and this PR — confirming all valid events accept and payloads missing every event's required fields reject.