feat(parquet): add experimental VECTOR repetition for Arrow FixedSizeList

[rok]

DO NOT MERGE. At this point this is a proposal meant to support discussion about a change to parquet format.

Rationale for this change

Arrow FixedSizeList<T, N> (embeddings, multidimensional array scientific data, etc) round-trips through Parquet today as a standard 3-level LIST, paying per-element repetition/definition levels for a shape that is fixed and known from the schema. On C++ we showed ~2-10x read improved performance is possible which motivates a denser encoding.

This adds an experimental Parquet VECTOR repetition type - the "Option B" design from the Fixed-size list type for Parquet proposal - that stores fixed-shape data without those inner levels.

Closes #855.

What changes are included in this PR?

• Format: FieldRepetitionType.VECTOR = 3 and SchemaElement.vector_length (field id 11), hand-applied to the generated parquet/internal/gen-go/parquet/parquet.go in the existing Thrift 0.21.0 generator style; parquet/parquet_vector.thrift vendors the IDL fragment as the source of truth.
• schema: a primitive VECTOR leaf node (NewPrimitiveNodeLogicalVector), vector_length plumbing, level computation (VECTOR adds no def/rep level), and NewSchemaChecked, which returns an error instead of panicking on a malformed VECTOR schema.
• file: the column writer counts rows as values / vector_length and keeps every data page on a whole-vector boundary across all write paths (WriteBatch, WriteBatchSpaced, WriteBitmapBatchSpaced, dictionary indices, FLBA); the reader supports row-ordinal seeking by value stride and rejects malformed VECTOR chunks (num_values not a whole multiple of, or inconsistent with, the row count).
• pqarrow: opt-in encoding via WithVectorEncoding(); eligible top-level FixedSizeList columns are written as VECTOR and reconstructed on read without a stored Arrow schema, and ineligible ones fall back to LIST. Works alongside WithStoreSchema (element timezone / field metadata are restored).

A VECTOR column is a single primitive leaf (vector <element-type> <name> [N]), not a nested group — the leaf carries vector_length and adds no definition/repetition level, so a dense vector has no inner levels.

Scope: dense, non-nullable, top-level FixedSizeList with a fixed-width primitive element. Every other FixedSizeList transparently falls back to LIST; nothing that writes today changes unless the flag is set.

Are these changes tested?

Yes. New tests cover:

• write/read round-trips across element types (bool, int32/64, float32/64, float16, fixed_size_binary, decimal128/256, date32, timestamp) and list sizes that don't divide the write batch size;
• whole-vector page boundaries and row accounting for every write path;
• row-ordinal seeking on DataPageV1, DataPageV2, and via offset index;
• compression (Snappy/Zstd) and dictionary-read;
• the LIST fallback for every ineligible case;
• rejection of nulls in non-nullable VECTOR data and of malformed VECTOR files;
• WithStoreSchema round-trip; and the schema/Thrift compact-protocol round-trip.

All new tests pass; the only failing tests in the suite are the pre-existing ones that need the parquet-testing data submodule / PARQUET_TEST_DATA.

Are there any user-facing changes?

Yes:

• New writer option pqarrow.WithVectorEncoding() (default off) and new schema helpers schema.NewPrimitiveNodeLogicalVector / schema.NewSchemaChecked / schema.NewColumnChecked.
• New exported error file.ErrVectorBatchMisaligned, returned by the typed column writers when a VECTOR column is given a partial-vector batch.
• A new parquet.Repetitions.Vector value; parquet.Repetitions.Undefined shifts from 3 to 4.

⚠️ Compatibility: files written with VECTOR are not readable by Parquet implementations that don't understand the VECTOR repetition type. This is the defining trade-off of VECTOR repetition type and the reason it is strictly opt-in and documented experimental, until VECTOR is standardized in apache/parquet-format.

Potential follow-up: nullable vectors (spaced leaf materialization + def-level→validity collapse), struct elements, and nested vectors.

[zeroshade]

Review notes

Rebased locally onto current main (past #852 / #856 / #857) — clean, no conflicts, the generated column_writer_types.gen.go still matches its template, and the full parquet/... test suite passes. For an experimental, opt-in proposal this is in good shape; nothing below blocks it landing as a proposal. The items worth resolving before VECTOR graduates from experimental are the two panic paths and the cross-version read behavior.

Correctness — looks right

• Row accounting (values / vector_length) is consistent across every write path via rowsForLeafValues (file/column_writer.go:664), and the whole-vector-per-page guarantee holds (alignBatchToVector + the per-batch flush in commitWriteAndCheckPageLimit).
• Composes cleanly with fix(parquet): add WriteBatchSpacedWithError to surface spaced-write failures #852: the misaligned-batch panic is caught by defer recover() at each public writer boundary and returned as an error.
• Reader rejection is thorough — file/row_group_reader.go:97-110 (value count must be a whole, consistent multiple of the row count, with an overflow guard), plus the stride/overflow guards in seekToRowWithValueStride and the BuildArray divisibility check.
• The Repetitions.Undefined 3→4 shift is safe: it is an internal-only sentinel (schema/reflection.go), never serialized; on-disk repetition_type was only ever 0–2.

Worth addressing before graduating from experimental

1. Panic-as-validation on the public file writer. vectorLengthForBatch panics on a misaligned batch (file/column_writer.go:677); through the low-level WriteBatch API this surfaces as a recover()'d "unknown error type: …". pqarrow never triggers it, but the public writer should validate n % vector_length == 0 up front and return a clean typed error.
2. NewSchemaChecked can still panic. effectiveVectorLength (schema/column.go:58) panics on int32 overflow, reachable via buildTree→NewColumn under NewSchemaChecked, which is documented to return an error instead of panicking. Unreachable today (groups can't be VECTOR), but a latent contract break once nested vectors exist — better to thread an error out.
3. Cross-version read is a silent misread. A VECTOR (repetition type 3) file opened by a pre-this-change reader has no case for 3 and would be read as a flat required column with N×len values / wrong row count rather than failing loudly. This is inherent to introducing a new enum value with no feature guard — worth calling out in the WithVectorEncoding docs, and ideally a read-side guard before this is non-experimental.
4. Statistics stay element-level (file/column_writer.go:660), so page/column min-max on a VECTOR column are not row-meaningful for predicate pushdown. A format-proposal decision worth stating explicitly.

Minor

• DataPageV1 without an offset index: the seek does a throwaway page seek and then re-seeks to 0 (file/column_reader.go:644-683) — correct result, but two scans.
• The V1 + offset-index seek relies on FirstRowIndex() being in parent-row units; it holds, but a test would lock it in (multi-page VECTOR, V1, page index enabled, seek into later pages).
• No explicit test for the WriteBatchSpacedWithError + VECTOR error path, or for a malformed vector_length.
• Read reconstruction without WithStoreSchema names the element "element" and drops element/list metadata (both restored when the schema is stored) — fine for Phase 1, worth a doc note.
• pqarrow/encode_arrow.go now returns err instead of swallowing it on builder-creation failure — good catch; unrelated to VECTOR but correct.

Format / spec

The Thrift choices (FieldRepetitionType.VECTOR = 3, SchemaElement.vector_length = field id 12) are hand-applied. Worth confirming these match the apache/parquet-format proposal exactly — if the standard settles on different values, files written now become silently incompatible.

[zeroshade]

Follow-up — format/spec conformance

I checked the hand-applied Thrift against the in-flight parquet-format proposal. VECTOR is not in apache/parquet-format master yet (FieldRepetitionType there is still just REQUIRED/OPTIONAL/REPEATED). Comparing against the current Option B draft (Antoine Pitrou's vector-repetition branch):

	FieldRepetitionType.VECTOR	SchemaElement.vector_length field id
This PR (parquet/parquet_vector.thrift)	3	12
parquet-format draft (pitrou:vector-repetition)	3	11

The enum value matches, but the vector_length Thrift field id differs (12 here vs 11 in the draft). A reader built to the draft would skip the unknown field 12, then see repetition_type = VECTOR with no vector_length and reject the file as malformed — so data written now wouldn't interop with a draft-conformant reader. Since cross-implementation compatibility is the whole crux of Option B, it'd be worth aligning the field id with the proposal (or calling out the divergence explicitly) before any files get written.

Secondary, lower confidence: the Parquet C++ Option B prototype (rok/arrow#51) appears to model VECTOR as a three-level group carrying a dedicated Vector logical type, whereas this PR uses the primitive-leaf "reduced Option B" with no logical annotation — another representational difference to reconcile as the proposal converges.

[rok]

Thanks for the very thorough review, @zeroshade — much appreciated. Pushed e667d465 addressing the actionable items.

Format/spec — field-id alignment

• Changed SchemaElement.vector_length from Thrift field id 12 → 11 to match the pitrou:vector-repetition draft, so files interoperate with a draft-conformant reader. Updated both parquet_vector.thrift and the hand-applied gen-go/parquet.go (struct tag, Read/Write field id + dispatch). The thrift round-trip test goes through the real serializer, so write-id == read-id == 11 is enforced end-to-end. FieldRepetitionType.VECTOR = 3 already matched.
• On the C++ prototype's 3-level group + Vector logical type vs this reduced primitive-leaf form: that's the larger representational question for the proposal to converge on. I kept the leaf-only encoding since it's what the pitrou:vector-repetition draft models, but happy to follow whichever the format settles on.

Worth addressing before graduating

1. Panic-as-validation on the public writer — added file.ErrVectorBatchMisaligned and an up-front validateVectorBatch(n) in every public write path (WriteBatch, WriteBatchSpacedWithError, WriteBitmapBatch/...Spaced, WriteDictIndices). A misaligned batch now returns the typed error instead of a recovered "unknown error type". The internal vectorLengthForBatch panic stays as an (now unreachable) invariant guard.
2. NewSchemaChecked can still panic — effectiveVectorLength now returns an error, threaded via a new NewColumnChecked through buildTree, so NewSchemaChecked returns it. NewColumn still panics for direct construction. (Still unreachable until nested vectors exist, but the documented contract now holds.)
3. Cross-version silent misread — documented explicitly on WithVectorEncoding: a pre-VECTOR reader may misread repetition-type-3 as a flat required column with the wrong row count rather than failing loudly. Agree a read-side feature guard is worth adding before this is non-experimental.
4. Statistics stay element-level — kept; the intent is now stated in code. Agree it's a format-proposal decision to make explicit.

Minor

• Element named "element" and element/list metadata dropped without WithStoreSchema — added a doc note (values, element type, list size, and shape always round-trip).
• V1 + offset-index multi-page seek and malformed vector_length rejection are already covered (TestVectorColumnReaderSeekToRowWithOffsetIndex, TestVectorValidationErrors); added explicit typed-error coverage for the misaligned WriteBatch/WriteBatchSpaced paths.
• DataPageV1-without-offset-index double scan: left as-is (correct result; a micro-optimization for later).

Left the statistics semantics and the group-vs-leaf representation as open proposal-convergence questions rather than changing them here. Thanks again!

[zeroshade]

Re-reviewed at ae93a6ad: all the actionable items from above are resolved and verified. In particular vector_length field id is now 11 (matching the pitrou:vector-repetition draft, alongside VECTOR = 3), the 12→11 change is consistent across the struct tag, read/write dispatch, IDL, and the round-trip test, and field 11 is genuinely free in parquet-format master. The draft's primitive-leaf shape matches this PR, so the earlier group-vs-leaf question is moot for now. file / schema / thrift and the full pqarrow suite (with PARQUET_TEST_DATA) pass; the gen file matches its template.

Remaining before this graduates from experimental — none blocking the proposal:

1. Read-side guard for repetition-type-3. Still documentation-only: a pre-VECTOR reader silently misreads a VECTOR column as a flat required column with the wrong row count. This wants a loud rejection / feature gate on read before VECTOR is non-experimental.
2. Fallback round-trip tests. Add write→read assertions that the ineligible cases (nullable, zero-length, nested, non-primitive-element FixedSizeList) transparently fall back to LIST and round-trip, plus multiple VECTOR columns, a VECTOR column mixed with normal columns, and an explicit row-count check across more than one row group.

Everything else (the DataPageV1 no-offset-index double scan, element-level statistics, and leaf-vs-group representation) is fine to leave as documented proposal decisions.

[zeroshade]

Re-reviewed at fcad9330 (rebased onto current main; column_writer_types.gen.go still regenerates byte-identically from its template, and the parquet/schema, parquet/file, parquet/pqarrow, parquet/internal/thrift, and parquet/metadata suites pass). The headline change since my last pass (ae93a6ad) is the pivot from the reduced leaf-only encoding to the canonical 3-level VECTOR group with a LogicalType.VECTOR annotation. This is the right call and resolves the leaf-vs-group question I'd previously flagged as open.

Format/spec conformance — now matches the draft exactly

I diffed the hand-applied Thrift against the current parquet-format draft (GH-430, rok/parquet-vector-logical-type). Everything lines up:

	This PR	parquet-format draft
FieldRepetitionType.VECTOR	3	3
SchemaElement.vector_length field id	11 (optional i32)	11
LogicalType.VECTOR union member	field id 19	19
VectorType	empty struct VectorType {}	empty
Layout	required group <name> (VECTOR) { vector group list [N] { required <elem> } }	identical

The hand-applied LogicalType.VECTOR member in gen-go/parquet/parquet.go is complete and consistent with the 0.21.0 generator output (struct tag VECTOR,19, plus GetVECTOR / IsSetVECTOR / CountSetFieldsLogicalType / ReadField19 / writeField19 / dispatch case 19 / Equals). TestVectorThriftRoundTrip runs the real compact + binary serializers and asserts vector_length rides field id 11 (and explicitly that 12 is absent) and that LogicalType.VECTOR survives the round-trip — so the field-id alignment is enforced end-to-end, not just in the IDL comment.

Schema / level handling — solid

• The structural invariants are tight and fail closed: validateVectorStructure (schema/schema.go:206) requires a top-level (VECTOR) group → a single VECTOR-repeated child named list (no logical/converted type) → a single element child that is required, primitive, and fixed-width; validateVectorColumnLevels (schema/schema.go:170) rejects any VECTOR context with maxDef≠0 / maxRep≠0, and the Repetitions.Vector branch adds no def/rep level (schema/schema.go:113). Net: a VECTOR leaf is guaranteed dense (maxDef = maxRep = 0).
• effectiveVectorLength (schema/column.go:57) now accumulates the product over VECTOR ancestors in int64 and returns an error on int32 overflow instead of panicking — closes the latent NewSchemaChecked panic path from the earlier round.
• Old leaf-only files are now rejected on read (pqarrow/column_readers.go:1171 — "VECTOR repetition on primitive nodes is not supported"), which is the correct, loud behavior for the now-superseded encoding.

Prior items — status

1. Fallback round-trip tests — done, and genuinely thorough. TestVectorFallbackRoundTrips, TestVectorMultipleAndMixedColumns, and TestVectorMultiRowGroupRowCount assert the physical encoding (InVectorColumn() / columnIsVector()), not just data equality: nullable value, nullable element, nested FSL, and variable-width element all fall back to LIST; multiple VECTOR columns and VECTOR-mixed-with-normal round-trip; and row counts are checked per row group across 4 row groups (parent rows ≠ leaf slots). WithStoreSchema is covered both ways.
2. Cross-version silent misread — still documentation-only, as expected. A pre-VECTOR reader hitting repetition_type = 3 on the middle group has no case for it; the new LogicalType.VECTOR marker on the outer group helps a conformant-but-VECTOR-unaware reader notice something is off, but the real incompatibility is the repetition enum, which is inherent to introducing it. Fine to leave documented for experimental; still wants a loud read-side gate before graduation.

Remaining before graduating from experimental (none blocking the proposal)

• Nullable VECTOR. The draft permits an optional outer group and optional element; this PR's reader rejects both (validateVectorElement, schema/schema.go:182; outer group in vectorToSchemaField). That's the right loud behavior for now, but it means arrow-go can't yet read the full set of draft-conformant VECTOR files — the spaced-leaf + def-level→validity work is the main functional gap (already on your follow-up list).
• Size-0 FixedSizeList. Eligibility correctly excludes it (pqarrow/schema.go:265, fsl.Len() > 0) so it falls back to LIST, but there's no explicit test for the size-0 case — worth a one-liner alongside the other fallback subtests.
• Element-level statistics on a VECTOR leaf aren't row-meaningful for predicate pushdown — unchanged, documented; a format-proposal decision.

Everything I flagged previously is either resolved or consciously deferred with a doc note. For an experimental, opt-in proposal this is in good shape, and the move to the annotated 3-level group materially improves cross-implementation interop.