It is already fixed as part of PIG-3017 Sent from my iPad
On Nov 5, 2012, at 8:06 PM, Haitao Yao <yao.e...@gmail.com> wrote: > hi, all > I think we need to optimize the > org.apache.pig.impl.util.ObjectSerializer, because it uses java object > serialization, which wastes a lot of space, so that it causes the tasktracker > to OOME. here's the analyze result of tasktracker heap dump: > <aa.jpg> > This illustrates that the heap is retained by the JobConf objects, and we > known jobconf contains a lot of Key-value strings. > > > > So here's the statistics of heap retention: > <bb.jpg> > > > And dive into the object histogram, here it is: > <cc.jpg> > > > And here's the source code: > <dd.jpg> > > > So I think we need to compress the output of object serializer. I'm > submitting my patch. > > > > > > Haitao Yao > yao.e...@gmail.com > weibo: @haitao_yao > Skype: haitao.yao.final >
