Re: Question about Iceberg release cadence

2020-08-26 Thread Anton Okolnychyi
+1 on releasing structured streaming source. I should be able to do one more review round tomorrow. - Anton > On 26 Aug 2020, at 17:12, Jungtaek Lim wrote: > > I hope we include Spark structured streaming read as well in the next > release; that was proposed in Feb this year and still around.

Re: Question about Iceberg release cadence

2020-08-26 Thread Jungtaek Lim
I hope we include Spark structured streaming read as well in the next release; that was proposed in Feb this year and still around. Quoting my comment on benefit of the streaming read on Spark; This would be the major feature to cover the gap on use case for structured > streaming between Delta La

Re: Hive Iceberg writes

2020-08-26 Thread Ryan Blue
I think Edgar and Adrien who have been contributing support for ORC and Hive are interested in this as well. On Wed, Aug 26, 2020 at 9:22 AM Peter Vary wrote: > Hi Team, > > We are thinking about implementing HiveOutputFormat, so writes through > Hive can work as well. > Has anybody working on t

Re: Question about logging

2020-08-26 Thread Ryan Blue
Hi Peter, Thanks for thinking about this! Improving logs is a great contribution. My philosophy is to stick to the usual definitions of logging levels. Here’s my quick summary: - FATAL: the event that caused the service to stop (not used in Iceberg, since it’s a library) - ERROR: an eve

Hive Iceberg writes

2020-08-26 Thread Peter Vary
Hi Team, We are thinking about implementing HiveOutputFormat, so writes through Hive can work as well. Has anybody working on this? Do you know any ongoing effort related to Hive writes? Asking because we would like to prevent duplicate effort. Also if anyone has some good pointers to start for

Re: Question about Iceberg release cadence

2020-08-26 Thread Ryan Blue
Hi Marton, 0.9.0 was released about 6 weeks ago, so I don't think we've planned when the next release will be yet. I think it's a good idea to release soon, though. The Flink sink is close to being ready as well and I'd like to get both of those released so that the contributors can start using th

Question about Iceberg release cadence

2020-08-26 Thread Marton Bod
Hi Team, I was wondering whether there is a release cadence already in place for Iceberg, e.g. how often releases will take place approximately? Which commits/features as release candidates in the near term? We're looking to integrate Iceberg into Hive, however, the current 0.9.1 release does not

Question about logging

2020-08-26 Thread Peter Vary
Hi Team, I was wondering if there is a general best practice for using log levels in Iceberg, what is the usual way we do it. I have been playing around with Iceberg/Hive integration and was wondering how I would be able to debug a customer case based on the logs. To be the honest answer based