Skip to content

[#11980] fix(core): sort column ordinal positions while fetching from entity store#11979

Merged
yuqi1129 merged 1 commit into
apache:mainfrom
MehulBatra:fix-lance-rest-catalog-positional-ordering
Jul 13, 2026
Merged

[#11980] fix(core): sort column ordinal positions while fetching from entity store#11979
yuqi1129 merged 1 commit into
apache:mainfrom
MehulBatra:fix-lance-rest-catalog-positional-ordering

Conversation

@MehulBatra

@MehulBatra MehulBatra commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Closes: #11980

What changes were proposed in this pull request?

Proposed fix:

Sort by position in toGenericTable():

.withColumns(                                                                                                                                                   
    tableEntity.columns().stream()                                                                                                                              
        .sorted(Comparator.comparingInt(ColumnEntity::position))                                                                                                
        .map(this::toGenericColumn)                                                                                                                             
        .toArray(Column[]::new))    

column_position is already persisted correctly, so this fixes existing tables on read with no migration or re-registration. Comparator and ColumnEntity are already imported in the file.

Why are the changes needed?

ManagedTableOperations.toGenericTable() builds the column list from tableEntity.columns() without sorting by position, so loaded tables can return columns in an arbitrary, unstable order even though column_position is stored correctly. This breaks position-based consumers (e.g. the Lance REST catalog), causing wrong column mapping or crashes. The fix sorts columns by position on load, matching what the alter path already does.

Fix: #11980

Does this PR introduce any user-facing change?

No API or property changes.
Behavior fix only: loadTable/describe now always returns columns in their declared position order (previously they could come back unordered). No user-facing API changes, and no property keys added or removed.

How was this patch tested?

Before Fix:
image

After Fix:
image

Without sorting:

image

With sorting:

image

@MehulBatra MehulBatra changed the title sort column ordinal positions while fetching from entity store [Gravitino/LRC] sort column ordinal positions while fetching from entity store Jul 10, 2026
@github-actions

Copy link
Copy Markdown

Code Coverage Report

Overall Project 67.67% +0.08% 🟢
Files changed 92.25% 🟢

Module Coverage
aliyun 1.72% 🔴
api 46.54% 🟢
authorization-common 85.96% 🟢
aws 42.04% 🟢
azure 2.47% 🔴
catalog-common 9.92% 🔴
catalog-fileset 80.23% 🟢
catalog-glue 66.91% 🟢
catalog-hive 79.42% 🟢
catalog-jdbc-clickhouse 81.16% 🟢
catalog-jdbc-common 44.22% 🟢
catalog-jdbc-doris 81.6% 🟢
catalog-jdbc-hologres 54.03% 🟢
catalog-jdbc-mysql 79.23% 🟢
catalog-jdbc-oceanbase 80.91% 🟢
catalog-jdbc-postgresql 82.29% 🟢
catalog-jdbc-starrocks 78.51% 🟢
catalog-kafka 77.01% 🟢
catalog-lakehouse-generic 59.18% 🟢
catalog-lakehouse-hudi 79.1% 🟢
catalog-lakehouse-iceberg 86.14% 🟢
catalog-lakehouse-paimon 84.25% 🟢
catalog-model 77.72% 🟢
cli 44.49% 🟢
client-java 78.15% 🟢
common 51.45% 🟢
core 82.44% +0.11% 🟢
filesystem-hadoop3 77.3% 🟢
flink 0.0% 🔴
flink-common 47.09% 🟢
flink-runtime 0.0% 🔴
gcp 14.12% 🔴
hadoop-auth 68.0% 🟢
hadoop-common 12.7% 🔴
hive-metastore-common 53.29% 🟢
iceberg-common 63.63% 🟢
iceberg-rest-server 74.3% 🟢
idp-basic 85.71% 🟢
integration-test-common 0.0% 🔴
jobs 66.17% 🟢
lance-common 20.67% 🔴
lance-rest-server 64.84% 🟢
lineage 53.02% 🟢
optimizer 83.24% 🟢
optimizer-api 21.95% 🔴
server 85.97% 🟢
server-common 74.62% 🟢
spark 28.57% 🔴
spark-common 46.01% 🟢
tencent 69.84% 🟢
trino-connector 40.29% 🟢
Files
Module File Coverage
core ManagedTableOperations.java 92.25% 🟢

@MehulBatra

Copy link
Copy Markdown
Contributor Author

Hi @yuqi1129 could you please help me with the review!

@yuqi1129 yuqi1129 changed the title [Gravitino/LRC] sort column ordinal positions while fetching from entity store [#11980] fix(core): sort column ordinal positions while fetching from entity store Jul 13, 2026
@yuqi1129 yuqi1129 added the branch-1.3 Automatically cherry-pick commit to branch-1.3 label Jul 13, 2026
@yuqi1129 yuqi1129 requested a review from Copilot July 13, 2026 08:25

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes an ordering bug in the core table load path by ensuring columns are returned in stable ordinal-position order when converting stored TableEntity metadata into a GenericTable. This aligns loadTable/describe behavior with the existing alter-table path and prevents position-based consumers from misinterpreting column mappings.

Changes:

  • Sort tableEntity.columns() by ColumnEntity::position in ManagedTableOperations.toGenericTable() before mapping to Column[].
  • Add a unit test that persists a TableEntity with an intentionally unordered column list and verifies loadTable() returns columns ordered by position.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
core/src/main/java/org/apache/gravitino/catalog/ManagedTableOperations.java Sort columns by stored ordinal position when loading tables from the entity store.
core/src/test/java/org/apache/gravitino/catalog/TestManagedTableOperations.java Adds coverage ensuring loadTable() returns columns ordered by position even if the store returns them unordered.

@yuqi1129 yuqi1129 merged commit 61ee16a into apache:main Jul 13, 2026
35 checks passed
@yuqi1129

Copy link
Copy Markdown
Contributor

@MehulBatra
Thanks for your contributions!!!

yuqi1129 pushed a commit that referenced this pull request Jul 13, 2026
…ositions while fetching from entity store (#11979) (#12003)

**Cherry-pick Information:**
- Original commit: 61ee16a
- Target branch: `branch-1.3`
- Status: ✅ Clean cherry-pick (no conflicts)

Co-authored-by: MehulBatra <66407733+MehulBatra@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

branch-1.3 Automatically cherry-pick commit to branch-1.3

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug report] Table columns returned in wrong order on load (Gravitino / Lance rest catalog does not sort columns by position)

3 participants