+1 on releasing structured streaming source. I should be able to do one more review round tomorrow.
- Anton > On 26 Aug 2020, at 17:12, Jungtaek Lim <kabhwan.opensou...@gmail.com> wrote: > > I hope we include Spark structured streaming read as well in the next > release; that was proposed in Feb this year and still around. Quoting my > comment on benefit of the streaming read on Spark; > > This would be the major feature to cover the gap on use case for structured > streaming between Delta Lake and Iceberg. There's a technical limitation on > Spark structured streaming itself (global watermark), which requires > workaround via splitting query into multiple queries & intermediate storage > supporting end-to-end exactly once. Delta Lake covers the case, and I really > would like to see the case also covered by Iceberg. > I see there're lots of works in progress on the milestone (and these are > great features which should be done), but after this we cover both batch and > streaming workloads being done with Spark, which is a huge step forward on > Spark users. > > Thanks, > Jungtaek Lim (HeartSaVioR) > > On Thu, Aug 27, 2020 at 1:13 AM Ryan Blue <rb...@netflix.com.invalid> wrote: > Hi Marton, > > 0.9.0 was released about 6 weeks ago, so I don't think we've planned when the > next release will be yet. I think it's a good idea to release soon, though. > The Flink sink is close to being ready as well and I'd like to get both of > those released so that the contributors can start using them. > > Seems like a good question for the broader community: how about a release in > the next month or so for Hive reads and the Flink sink? > > rb > > On Wed, Aug 26, 2020 at 8:58 AM Marton Bod <marton....@gmail.com > <mailto:marton....@gmail.com>> wrote: > Hi Team, > > I was wondering whether there is a release cadence already in place for > Iceberg, e.g. how often releases will take place approximately? Which > commits/features as release candidates in the near term? > > We're looking to integrate Iceberg into Hive, however, the current 0.9.1 > release does not yet contain the StorageHandler code in iceberg-mr. Knowing > the approximate release timelines would help greatly with our integration > planning. > > Of course, happy to get involved with ongoing dev/stability efforts to help > achieve a new release of this module. > > Thanks a lot, > Marton > > > -- > Ryan Blue > Software Engineer > Netflix
