15K mappers on a 4 node system will definitely crash it unless you have tuned yarn (RM, NM) well. That many mappers reading data off few disks in parallel can create disk storm and disk can also turn out to be your bottle neck. Pig creates 1 map per 128MB ( pig.maxCombinedSplitSize default value) of data. 15K mappers means you are reading 1.9 TB of data. Based on the number of memory capacity you have, you can reduce the number of mappers. For eg: If you have 44 G heap per node for tasks (Assuming 48G RAM and some memory taken for node manager, data node, etc) and you are running mappers with 1G heap (mapreduce.map.java.opts) and 1.5G (mapreduce.map.memory.mb) container sizes, you can run 117 containers in parallel.
set pig.maxCombinedSplitSize 10737418240 Above setting will make each map process 10G of data which will create about ~190 maps and you should be able to run without bringing down your cluster. Regards, Rohini On Wed, May 11, 2016 at 10:17 AM, Olaf Collider <olaf.colli...@gmail.com> wrote: > I am using a hadoop Cloudera system with 4 nodes only, but a lot of disk > space (*200TB*). > > In my pig script, I load several monthly files that are about *200Gb* in > size each. > > I load my data like this > > data = LOAD 'mypath/data_2015*' > > USING com.twitter.elephanbird.pig.load.JsonLoader('-nestedload') > > then I FILTER the data and probably remove 80% of the data after the filter > step. > > I noticed that if I load in my pig script about one year of data, Pig > creates about 15k mappers and the whole process takes approximately 3 hours > (including the reduce step). > > Instead, if I load 2 years of data, then Pig creates about 30k mappers and > basically all the nodes become unhealthy after processing for more than 15 > hours. > > Am I hitting some kind of bottleneck here? Or there is some default options > I should play with? > > Many thanks! >