roryqi opened a new issue, #13273:
URL: https://github.com/apache/gravitino/issues/13273
### Version
main branch
### Describe what's wrong
Creating a table with 4,096 or more columns fails when PostgreSQL is used as
the relational metadata store.
`TableColumnBaseSQLProvider#insertColumnPOs` generates one multi-value
INSERT statement for all columns. Each column uses 16 bound parameters, so a
table with 4,096 columns generates 65,536 parameters and exceeds PostgreSQL's
65,535-parameter limit.
Observed results for tables containing integer columns:
| Number of columns | Result |
| --- | --- |
| 2,000 | Success |
| 4,000 | Success |
| 5,000 | Failure |
| 6,400 | Failure |
The table may be created in the underlying Iceberg catalog, but its metadata
is not stored in Gravitino. The subsequent owner operation then reports a
misleading metadata-object-not-found error.
Column metadata should be inserted in bounded batches while retaining the
existing transaction boundary.
### Error message and/or stacktrace
The create request returns an error similar to:
The metadata object of <catalog>.<schema>.<table> isn't found
This response hides the PostgreSQL metadata-store write failure. The
underlying PostgreSQL parameter-limit error is also not visible in the server
log.
### How to reproduce
1. Run Gravitino with PostgreSQL as its relational metadata store.
2. Create a `lakehouse-iceberg` catalog using a JDBC catalog backend.
3. Create a schema.
4. Submit a create-table request containing 5,000 integer columns.
5. Observe that the request fails, while an equivalent table with 4,000
columns succeeds.
6. Check the stores:
- The table exists in the Iceberg catalog store.
- No corresponding row exists in the Gravitino metadata store.
### Additional context
The current SQL provider emits 16 bound parameters per column in one
statement:
https://github.com/apache/gravitino/blob/main/core/src/main/java/org/apache/gravitino/storage/relational/mapper/provider/base/TableColumnBaseSQLProvider.java
The failure boundary is therefore:
floor(65,535 / 16) = 4,095 columns
A possible fix is to split column metadata into bounded INSERT batches and
execute all batches inside the existing table-insert transaction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
