Is there enough commonality across Delta, Hudi, Iceberg for this generic solution? I imagined we'd potentially have individual IOs for each. A generic one seems possible, but certainly would like to learn more.
Also, are others in the community working on connectors for ANY of those Delta Lake, Hudi, or Iceberg IOs? Would hope for some form of coordination and/or at least awareness between people addressing complementary/overlapping areas. On Mon, Aug 29, 2022 at 4:15 PM Neil Kolban via dev <dev@beam.apache.org> wrote: > Howdy, > I have a client who would be interested to use this. Is there a link to a > GitHub repo or other place I can read more? > > Neil (kol...@google.com) > > On 2022/08/05 07:23:31 张涛 wrote: > > > > Hi, we developed a new IO connector named DataLakeIO, to connect Beam > and data lake, such as Delta Lake, Apache Hudi, Apache iceberg. Beam can > use DataLakeIO to read data from data lake, and write data to data lake. We > did not find data lake IO on > https://beam.apache.org/documentation/io/built-in/, we want to contribute > this new IO connector to Beam, what should we do next? Thank you very much! >
