Thanks for the detailed report ! One more thing: We now have made a lot of progress in integrating Alibaba Cloud (https://www.aliyun.com/), Please see https://github.com/apache/iceberg/projects/21 (Thanks @xingbowu - https://github.com/xingbowu).
On Thu, Oct 21, 2021 at 11:30 PM Sam Redai <s...@tabular.io> wrote: > Good Morning Everyone, > > Here are the minutes from our Iceberg Sync that took place on October > 20th, 9am-10am PT. Please remember that anyone can join the discussion so > feel free to share the Iceberg-Sync > <https://groups.google.com/g/iceberg-sync> google group with anyone who > is seeking an invite. As usual, the notes and the agenda are posted in the > live > doc > <https://docs.google.com/document/d/1YuGhUdukLP5gGiqCbk0A5_Wifqe2CZWgOd3TbhY3UQg/edit?usp=drive_web> > that's > also attached to the meeting invitation. > > We covered a lot of topics...here we go!: > > Top of the Meeting Highlights > > - > > Sort based compaction - This is finished, reviewed, and merged. When > you compact data files, you can now also have Spark re-sort them, either by > the table’s sort order or the sort order given when you create the > compaction job. > - > > Spark build refactor: Thank you to Jack for getting us started on the > Spark build refactor and also thanks to Anton for reviewing and helping get > these changes in. We’ve gone with a variant of option 3 from our last > discussions where we include all of the spark modules in our build but make > it easy to turn them off. This way we can get the CI to run Spark, Hive, > and Flink tests separately and only if necessary. > - > > Delete files implementation for ORC: Thanks to Peter for adding > builders to store deletes in ORC (previously we could only store deletes in > Parquet or Avro). This means we now have support for all 3 formats for this > feature. > - > > Flink Update: We’ve updated Flink to 1.13 so we’re back on a supported > version. 1.14 is out this week so we can aim to move to that at some point. > > Iceberg 0.12.1 Upcoming Patch Release (milestone > <https://github.com/apache/iceberg/milestone/15?closed=1>) > > - > > Fix for the parquet map projection bug > - > > Fix Flink CDC bug > - > > A few other fixes that we also want to get out to the community so > we’re going to start a release candidate as soon as possible > - > > Kyle will start a thread in the general slack channel so everyone > please feel free to mention any additional fixes that they want to see in > this patch release > > Snapshot Releases > > - > > Eduard will tackle adding snapshot releases > - > > In our deploy.gradle file, it’s setup to deploy to the snapshot > repository > - > > May require certain credentials so it may be required to reach out to > the ASF infrastructure team > > Iceberg 0.13.0 Upcoming Release > > - > > There’s agreement to switch to a time based release schedule so the > next release is roughly mid-November > - > > Jack will cut a branch close to that time and any features that aren’t > in yet will be pushed to the next release > - > > We agree not to hold up releases to squeeze features in and prefer > instead to aim for releasing sooner the next time > > Adding v3.2 to Spark Build Refactoring > > - > > Russell and Anton will coordinate on dropping in a Spark 3.2 module > - > > We currently have 3.1 in the `spark3` module. We’ll move that out to > its own module and mirror what we do with the 3.2 module. (This will enable > cleaning up some mixed 3.0/3.1 code) > > Merge on Read > > - > > Anton has a bunch of PRs ready to queue up to contribute their > internal implementation. (Russell will work with him) > - > > This feature will allow for a much lower write amplification > - > > The expectation is that in Spark 3.3 we can rely on Spark’s internal > merge on read > > Snapshot Tagging (design doc > <https://docs.google.com/document/d/1PvxK_0ebEoX3s7nS6-LOJJZdBYr_olTWH9oepNUfJ-A/edit>) > (PR #3104 <https://github.com/apache/iceberg/pull/3104>) > > - > > We just had a meeting on Monday about that and made some conclusions > and designs, so anyone who is interested please take a look. > - > > Next steps are to add the feature in the stack and Jack already has a > WIP implementation into the table metadata class > > Delete Compaction (design doc > <https://docs.google.com/document/d/1-EyKSfwd_W9iI5jrzAvomVw3w1mb_kayVNT7f2I-SUg> > ) > > - > > Discussion happening at 5pm ET on 10/21 5-6pm PT for anyone interested > (meeting link <https://meet.google.com/nxx-nnvj-omx>) > - > > Some more discussion is needed to hone in on a final design choice. > There are a few options that each have their own pros and cons. > > The New Source Interface for Flink (FLIP-27 > <https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface> > ) > > - > > Eventually everything will move to this new source interface (Kafka is > already using this and it will be the default in Flink 1.14) > - > > A few PR’s for Iceberg are out there and are pending review and merge > (may not make the deadline for the next release but that’s ok) > > Encryption MVP > > - > > Just had a recent sync on this and we're currently waiting on a few > updates to the design > - > > Flesh out how the new pushdown encryption into ORC and Parquet will > work > - > > Need some people to review the stream based encryption, > particularly around splitability > - > > A few offline discussions are currently happening and for the > interface we are expecting a few additional PRs separate from the main > encryption MVP PR > > Python Library Development > > - > > The high level design discussions have been concluded recently > - > > We’ll delay the top level API discussions until some of the core is > implemented > - > > We have a collection of issues created and a handful of engineers > working on it > > Iceberg Docsite Refactoring > > - > > Large refactoring coming for the Iceberg docsite > - > > Versioned docs (In the future need to decide how to represent the > python versions) > - > > Organized more by the persona of the visitor (Data Engineer, > Systems Engineer, etc.) > - > > Searchable > - > > Expect a PR from Sam, ready for review by the end of this week or > early next week > > Row-Level Support in the Vectorized Reader (PR #3141 > <https://github.com/apache/iceberg/issues/3141>) > > - > > Yufei is working on this and it’s part of the effort for merge on read > - > > PR #3287 <https://github.com/apache/iceberg/pull/3287> is only for the > position delete in parquet > - > > We should have something ready to add by next week > > View Spec (PR #3188 <https://github.com/apache/iceberg/pull/3188>) > > - > > There was a discussion on if we should we have just the SQL text > exactly as it was passed to the engine or should we also include the parsed > and analyze plan (includes column resolution). In theory, the resolved SQL > text should be very useful but it’s usefulness may be limited to certain > edge cases. > - > > The broader discussion here is: Should we allow having multiple > dialects (Trino, Spark, etc..) > - > > Adds complexity > - > > Time traveling needs to be considered. What does time traveling a > view mean? If the underlying table is an Iceberg table we may be able > to, > but even that would require “as of” time travel to allow time travel > across > multiple tables. > - > > Time traveling schemas needs to be added > - > > Agreement that we should not try to solve everything at once but break > this into smaller problems. > - > > Let’s keep an eye on upcoming engine features to see if this will be > implicitly solved and let's also refrain from over engineering this. > > That's it! Thanks everyone for the high level of participation and enjoy > the rest of your week! > >
