Hi, Just to double check: while I believe most Iceberg content refers to Iceberg as a format, it's considered as a connector in Flink terms, right? So there is no 'format' option to specify.
And one more question: is there a specific goal or reason why you would want to contribute this to Flink instead of keeping it under the Iceberg umbrella? Best regards, Martijn Op wo 12 okt. 2022 om 20:11 schreef Danny Cranmer <dannycran...@apache.org> > +1 from me, let's open a VOTE thread to make this official. > > On Wed, Oct 12, 2022 at 1:37 PM Hang Ruan <ruanhang1...@gmail.com> wrote: > > > +1, I have used the iceberg connector and catalog before. It is easy to > > use. > > > > Márton Balassi <balassi.mar...@gmail.com> 于2022年10月12日周三 16:38写道： > > > > > +1 from me, thanks for the clarification. > > > > > > On Wed, Oct 12, 2022 at 7:57 AM Péter Váry < > peter.vary.apa...@gmail.com> > > > wrote: > > > > > > > Thanks Abid, > > > > > > > > Count me in, and drop a note, if I can help in any way. > > > > > > > > Thanks, > > > > Peter > > > > > > > > On Tue, Oct 11, 2022, 20:13 <abmo.w...@icloud.com.invalid> wrote: > > > > > > > > > Hi Martijn, > > > > > > > > > > Yes catalog integration exists and catalogs can be created using > > Flink > > > > > SQL. > > > > > > > > > > > > > > > > > > > > https://iceberg.apache.org/docs/latest/flink/#creating-catalogs-and-using-catalogs > > > > > has more details. > > > > > We may need some discussion within Iceberg community but based on > the > > > > > current iceberg-flink code structure we are looking to externalize > > this > > > > as > > > > > well. > > > > > > > > > > Thanks > > > > > Abid > > > > > > > > > > > > > > > On 2022/10/11 08:24:44 Martijn Visser wrote: > > > > > > Hi Abid, > > > > > > > > > > > > Thanks for the FLIP. I have a question about Iceberg's Catalog: > has > > > > that > > > > > > integration between Flink and Iceberg been created already and > are > > > you > > > > > > looking to externalize that as well? > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Martijn > > > > > > > > > > > > On Tue, Oct 11, 2022 at 12:14 AM <ab...@icloud.com.invalid> > wrote: > > > > > > > > > > > > > Hi Marton, > > > > > > > > > > > > > > Yes, we are initiating this as part of the Externalize Flink > > > > Connectors > > > > > > > effort. Plan is to externalize the existing Flink connector > from > > > > > Iceberg > > > > > > > repo into a separate repo under the Flink umbrella. > > > > > > > > > > > > > > Sorry about the doc permissions! I was able to create a > FLIP-267: > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP+267%3A+Iceberg+Connector > > > > > > > < > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP+267%3A+Iceberg+Connector > > > > > > > > > > > > > > > Lets use that to discuss. > > > > > > > > > > > > > > Thanks > > > > > > > Abid > > > > > > > > > > > > > > On 2022/10/10 12:57:32 Márton Balassi wrote: > > > > > > > > Hi Abid, > > > > > > > > > > > > > > > > Just to clarify does your suggestion mean that the Iceberg > > > > community > > > > > > > would > > > > > > > > like to remove the iceberg-flink connector from the Iceberg > > > > codebase > > > > > and > > > > > > > > maintain it under Flink instead? A new separate repo under > the > > > > Flink > > > > > > > > project umbrella given the current existing effort to extract > > > > > connectors > > > > > > > to > > > > > > > > their individual repos (externalize) makes sense to me. > > > > > > > > > > > > > > > > [1] > > > > https://lists.apache.org/thread/mpzzlpob9ymkjfybm96vz2y2m5fjyvfo > > > > > > > > > > > > > > > > Best, > > > > > > > > Marton > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Oct 10, 2022 at 5:31 AM Jingsong Li <ji...@gmail.com > > > > > > wrote: > > > > > > > > > > > > > > > > > Thanks Abid for driving. > > > > > > > > > > > > > > > > > > +1 for this. > > > > > > > > > > > > > > > > > > Can you open the permissions for > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1WC8xkPiVdwtsKL2VSPAUgzm9EjrPs8ZRjEtcwv93ISI/edit?usp=sharing > > > > > > > > > ? > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > Jingsong > > > > > > > > > > > > > > > > > > On Mon, Oct 10, 2022 at 9:22 AM Abid Mohammed > > > > > > > > > <ab...@icloud.com.invalid> wrote: > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > I would like to start a discussion about contributing > > Iceberg > > > > > Flink > > > > > > > > > Connector to Flink. > > > > > > > > > > > > > > > > > > > > I created a doc < > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1WC8xkPiVdwtsKL2VSPAUgzm9EjrPs8ZRjEtcwv93ISI/edit?usp=sharing > > > > > > > > > > > > > > > > > with all the details following the Flink Connector template > > as > > > I > > > > > don’t > > > > > > > have > > > > > > > > > permissions to create a FLIP yet. > > > > > > > > > > High level details are captured below: > > > > > > > > > > > > > > > > > > > > Motivation: > > > > > > > > > > > > > > > > > > > > This FLIP aims to contribute the existing Apache Iceberg > > > Flink > > > > > > > Connector > > > > > > > > > to Flink. > > > > > > > > > > > > > > > > > > > > Apache Iceberg is an open table format for huge analytic > > > > > datasets. > > > > > > > > > Iceberg adds tables to compute engines including Spark, > > Trino, > > > > > > > PrestoDB, > > > > > > > > > Flink, Hive and Impala using a high-performance table > format > > > that > > > > > works > > > > > > > > > just like a SQL table. > > > > > > > > > > Iceberg avoids unpleasant surprises. Schema evolution > works > > > and > > > > > won’t > > > > > > > > > inadvertently un-delete data. Users don’t need to know > about > > > > > > > partitioning > > > > > > > > > to get fast queries. Iceberg was designed to solve > > correctness > > > > > > > problems in > > > > > > > > > eventually-consistent cloud object stores. > > > > > > > > > > > > > > > > > > > > Iceberg supports both Flink’s DataStream API and Table > API. > > > > > Based on > > > > > > > the > > > > > > > > > guideline of the Flink community, only the latest 2 minor > > > > versions > > > > > are > > > > > > > > > actively maintained. See the Multi-Engine > > Support#apache-flink > > > > for > > > > > > > further > > > > > > > > > details. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Iceberg connector supports: > > > > > > > > > > > > > > > > > > > > • Source: detailed Source design < > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1q6xaBxUPFwYsW9aXWxYUh7die6O7rDeAPFQcTAMQ0GM/edit# > > > > > > > >, > > > > > > > > > based on FLIP-27 > > > > > > > > > > • Sink: detailed Sink design and interfaces used > < > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1O-dPaFct59wUWQECXEEYIkl9_MOoG3zTbC2V-fZRwrg/edit# > > > > > > > > > > > > > > > > > > > > • Usable in both DataStream and Table API/SQL > > > > > > > > > > • DataStream read/append/overwrite > > > > > > > > > > • SQL create/alter/drop table, select, insert > into, > > > > > insert > > > > > > > > > overwrite > > > > > > > > > > • Streaming or batch read in Java API > > > > > > > > > > • Support for Flink’s Python API > > > > > > > > > > > > > > > > > > > > See Iceberg Flink < > > > > > > > https://iceberg.apache.org/docs/latest/flink/#flink>for > > > > > > > > > detailed usage instructions. > > > > > > > > > > > > > > > > > > > > Looking forward to the discussion! > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > Abid > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- immerok GmbH - immerok.io - o...@immerok.io - Linienstraße 214, 10119 Berlin, Germany Registered at Amtsgericht Charlottenburg: HRB 241661 B Managing directors: Holger Temme, Konstantin Knauf, Johannes Moser
