Hello Harpreet, It seems that your job is going beyond the limits established.
What are the values for yarn.scheduler.minimum-allocation-mb and yarn.scheduler.maximum-allocation-mb on your cluster? Some background on the meaning of these configurations can be found here: https://discuss.pivotal.io/hc/en-us/articles/201462036-MapReduce-YARN-Memory-Parameters Regards, Douglas On Wed, Aug 2, 2017 at 8:00 AM, Harpreet Singh <[email protected]> wrote: > Hi All, > I have a sqoop job which is running in production and fails sometimes. > Restart of job executes successfully . > Logs show that failure happens with error that container is running beyond > physical memory limits. Current usage 2.3 GB of 2GB physical memory used. > 4.0 GB of 4.2 GB virtual memory used. Killing container. > Environment is > Cdh5.8.3 > Sqoop 1 client > Mapreduce.map.Java.opts=-Djava.net.preferIPv4Stack=true -Xmx1717986918 > Mapreduce.map.memory.MB= 2GB > > Sqoop job details. Pulling data from netezza using 6 mappers and putting > into parquet format on hdfs. Data processed is 14 GB. Splits seem to be > even. > Please provide your insights. > > Regards > Harpreet Singh >
