[
https://issues.apache.org/jira/browse/IMPALA-13832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18104093#comment-18104093
]
Aleksandr Efimov edited comment on IMPALA-13832 at 8/12/26 2:51 PM:
--------------------------------------------------------------------
Hi [~boroknagyz], [~prozsa],
I spent some time on IMPALA-13832 and put together a small end-to-end PoC for
{{INSERT INTO}} against an Iceberg REST catalog without CatalogD. It reuses the
existing Iceberg sink, then commits the transaction from the coordinator-side
Java Frontend through the REST-loaded table. A fresh query sees the new
snapshot and row.
I kept the first cut to {{INSERT INTO}} and the simple standalone topology with
exactly one REST provider. If CatalogD is deployed or several REST providers
are configured, the write is rejected. {{{}INSERT OVERWRITE{}}},
{{{}DELETE{}}}, {{{}UPDATE{}}}, {{{}MERGE{}}}, and {{OPTIMIZE}} are also
rejected during analysis. The focused tests cover a successful insert, cleanup
after a known commit failure, and retaining the file when the commit state is
unknown.
The runtime test is still small: one unpartitioned table, one impalad, one
writer, one data file, and {{{}INSERT VALUES{}}}. I have not tested partitioned
writes, multiple executors/files, zero-row inserts, or concurrent inserts yet.
There are two shortcuts in the PoC that I do not want to turn into a contract
by accident. The finalizer is selected from the global topology, and the target
table is reloaded by name through the single REST provider. The design note
explains both and sketches a path with provider affinity, table UUID
verification, and shared Iceberg commit/cleanup code.
Before I take this further, I would like to check two design choices and one
scope question:
# My current preference is for the owning {{MetaProvider}} to reload and
commit the table, with {{Frontend}} kept as the JNI entry point and the Iceberg
transaction code moved to a shared helper. Does that fit the direction you want
for {{MetaProvider}} / {{{}MultiMetaProvider{}}}, or should mutations stay
outside the provider interface?
# For provider affinity, I propose using a non-empty, unique
{{iceberg.rest-catalog.name}} and carrying it in the Iceberg finalization
parameters. Should the first mergeable change support mixed/multiple providers,
or keep the current single-provider restriction and add affinity in the next
step?
# I plan to carry {{Table.uuid()}} from analysis and verify it before commit.
The CatalogD path also resolves the target by name, so it appears to have the
same drop/recreate gap. Would you prefer that protection in the same change or
as a separate follow-up for the existing path?
The WIP is here: [Gerrit 24682|https://gerrit.cloudera.org/c/24682/]
I also attached [^impala-13832-standalone-rest-insert-design.md] with the code
path, test coverage, known gaps, and the direction I would take after design
agreement.
was (Author: JIRAUSER313444):
Hi Zoltan, Peter,
I spent some time on IMPALA-13832 and put together a small end-to-end PoC for
{{INSERT INTO}} against an Iceberg REST catalog without CatalogD. It reuses the
existing Iceberg sink, then commits the transaction from the coordinator-side
Java Frontend through the REST-loaded table. A fresh query sees the new
snapshot and row.
I kept the first cut to {{INSERT INTO}} and the simple standalone topology with
exactly one REST provider. If CatalogD is deployed or several REST providers
are configured, the write is rejected. {{{}INSERT OVERWRITE{}}},
{{{}DELETE{}}}, {{{}UPDATE{}}}, {{{}MERGE{}}}, and {{OPTIMIZE}} are also
rejected during analysis. The focused tests cover a successful insert, cleanup
after a known commit failure, and retaining the file when the commit state is
unknown.
The runtime test is still small: one unpartitioned table, one impalad, one
writer, one data file, and {{{}INSERT VALUES{}}}. I have not tested partitioned
writes, multiple executors/files, zero-row inserts, or concurrent inserts yet.
There are two shortcuts in the PoC that I do not want to turn into a contract
by accident. The finalizer is selected from the global topology, and the target
table is reloaded by name through the single REST provider. The design note
explains both and sketches a path with provider affinity, table UUID
verification, and shared Iceberg commit/cleanup code.
Before I take this further, I would like to check two design choices and one
scope question:
# My current preference is for the owning {{MetaProvider}} to reload and
commit the table, with {{Frontend}} kept as the JNI entry point and the Iceberg
transaction code moved to a shared helper. Does that fit the direction you want
for {{MetaProvider}} / {{{}MultiMetaProvider{}}}, or should mutations stay
outside the provider interface?
# For provider affinity, I propose using a non-empty, unique
{{iceberg.rest-catalog.name}} and carrying it in the Iceberg finalization
parameters. Should the first mergeable change support mixed/multiple providers,
or keep the current single-provider restriction and add affinity in the next
step?
# I plan to carry {{Table.uuid()}} from analysis and verify it before commit.
The CatalogD path also resolves the target by name, so it appears to have the
same drop/recreate gap. Would you prefer that protection in the same change or
as a separate follow-up for the existing path?
The WIP is here: [Gerrit 24682|https://gerrit.cloudera.org/c/24682/]
I also attached [^impala-13832-standalone-rest-insert-design.md] with the code
path, test coverage, known gaps, and the direction I would take after design
agreement.
> Add DML support for Iceberg tables in Iceberg REST Catalog
> ----------------------------------------------------------
>
> Key: IMPALA-13832
> URL: https://issues.apache.org/jira/browse/IMPALA-13832
> Project: IMPALA
> Issue Type: Improvement
> Components: Catalog, Frontend
> Reporter: Zoltán Borók-Nagy
> Priority: Major
> Labels: impala-iceberg
> Attachments: impala-13832-standalone-rest-insert-design.md
>
>
> Add DML support for Iceberg tables in Iceberg REST Catalog
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
