Re: hadoop.fs.Path impacts on Impala

2018-08-01 Thread Philip Zeyliger
Hi Barnabas, If I may suggest a way to approach this sort of question, I'd take a heapdump of an impalad and a catalogd (using "jmap") and then use Eclipse MAT or http://www.jxray.com/ to see if we're using Path. You'll want to load some tables and partitions ahead of time. Based on a little quick

Re: hadoop.fs.Path impacts on Impala

2018-08-01 Thread Barnabás Maidics
On Wed, Aug 1, 2018 at 11:17 AM Barnabás Maidics < barnabas.maid...@cloudera.com> wrote: > Hi Everyone! > > I'm an intern at Cloudera and analysing where the memory goes in Hive. I > was looking at a heapdump with many partitions, and found a memory waste, > that comes from HDFS. > > We store path