roryqi opened a new pull request, #13277:
URL: https://github.com/apache/gravitino/pull/13277
### What changes were proposed in this pull request?
- Split table column metadata inserts into batches of 1,000 columns.
- Use the batched insertion path for both table creation and table updates.
- Add a regression test that stores and reads back a table with 5,000
columns.
### Why are the changes needed?
The current implementation inserts all column metadata using one multi-value
SQL statement. Each column uses 16 bound parameters, so tables with 4,096 or
more columns exceed PostgreSQL's 65,535-parameter limit.
Executing bounded batches inside the existing table transaction avoids the
parameter limit while preserving atomicity.
Fix: #13273
### Does this PR introduce _any_ user-facing change?
Yes. Tables with more than 4,095 columns can now be created when PostgreSQL
is used as the relational metadata store.
There are no API or configuration changes.
### How was this patch tested?
Ran the table column metadata test suite against H2, MySQL, and PostgreSQL:
./gradlew :core:test --tests
org.apache.gravitino.storage.relational.service.TestTableColumnMetaService
-PskipDockerTests=false
All 24 tests passed, including the new 5,000-column regression test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
