yuqi1129 commented on issue #12431:
URL: https://github.com/apache/gravitino/issues/12431#issuecomment-5364662351
@shaoyu-li
The direction holds up. Gravitino never touches data for Hive, Iceberg or
JDBC — the managed Lance table in the generic lakehouse catalog is the one
exception, and only
because Gravitino is the metastore of record there, so someone has to
write the manifest. "A metadata service should not hold broad write authority
over storage" is
consistent with the rest of the architecture, and the three motivations
are real problems, not hypothetical ones.
But as scoped, the proposal does not deliver two of its three motivations.
That is the part I think is most worth resolving before we go further:
1. The credentials do not actually leave the server. The request objects
are specified to carry "the resolved storage options", and all four call sites
resolve them
server-side: resolveLanceStorageOptions(catalogProperties,
table.properties()) at LanceTableOperations.java:207, :390, :440 and :759. The
server therefore still resolves
and holds the credentials for every table it serves — they just take one
extra hop before being used. To address motivation 1, credential resolution
itself would have to
move into the executor, and that changes the shape of the interface
substantially.
2. The native library stays in the server process. The proposal explicitly
puts LancePartitionStatisticStorage and core's implementation(libs.lance) out
of scope — but
those two are precisely why the Lance native library is on the server's
runtime classpath at all. With them unchanged, a JNI crash or an off-heap
allocator exhaustion can
still take down the metadata service, and the Lance version is still
pinned process-wide. So motivations 2 and 3 are largely unaddressed by this
iteration.
More information is welcome and can help us understand it precisely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
