Re: OutOfMemoryError after loading lots of dynamic partitions

2020-01-09 Thread Suresh Kumar Sethuramaswamy
That's awesome. Thanks Suresh Sethuramaswamy On Thu, Jan 9, 2020 at 2:00 PM Patrick Duin wrote: > Thanks Suresh, changing the heap was our first guess as well actually. I > think we were on the right track there. Weird thing is that our jobs seems > to now run fine (all partitions are added) d

Re: OutOfMemoryError after loading lots of dynamic partitions

2020-01-09 Thread Patrick Duin
Thanks Suresh, changing the heap was our first guess as well actually. I think we were on the right track there. Weird thing is that our jobs seems to now run fine (all partitions are added) despite still giving this error. Weird but it seems to be ok now. Thanks for the help. Op wo 8 jan. 2020 o

Re: Apache Iceberg integration

2020-01-09 Thread Feng Lu
For someone like me who is new to the hive community, is there a (semi-)formal process on contributing a large-scale feature like Hive-Iceberg integration? For example, hive improvement proposal, community voting, development and code review, release, etc. Thank you and sorry for derailing this co

Re: Apache Iceberg integration

2020-01-09 Thread Peter Vary
Hi Elliot, I think would be really worthwhile to have Iceberg integration with Hive. Minimally for reading through the available interfaces, then handling schema evolution / schema synchronization etc. Later having the possibility to write to an Iceberg table would be good as well, but integrati