Reduce yarn.nodemanager.vmem-pmem-ratio to 2.1 and lower. On Tue, Oct 13, 2015 at 2:32 PM, hadoop hive <hadooph...@gmail.com> wrote:
> <property> > > <name>mapreduce.reduce.memory.mb</name> > > <value>4096</value> > </property> > > > change this to 8 G > > > On Wed, Oct 14, 2015 at 12:52 AM, Ranjana Rajendran < > ranjana.rajend...@gmail.com> wrote: > >> Here is Altiscale's documentation about the topic. Do let me know if you >> have any more questions. >> >> http://documentation.altiscale.com/heapsize-for-mappers-and-reducers >> >> On Tue, Oct 13, 2015 at 9:31 AM, Mich Talebzadeh <m...@peridale.co.uk> >> wrote: >> >>> Hi, >>> >>> >>> >>> I have been having some issues with loading data into hive from one >>> table to another for 1,767,886 rows. I was getting the following error >>> >>> >>> >>> Task with the most failures(4): >>> >>> ----- >>> >>> Task ID: >>> >>> task_1444731612741_0001_r_000000 >>> >>> >>> >>> URL: >>> >>> >>> http://0.0.0.0:8088/taskdetails.jsp?jobid=job_1444731612741_0001&tipid=task_1444731612741_0001_r_000000 >>> >>> ----- >>> >>> Diagnostic Messages for this Task: >>> >>> Container >>> [pid=16238,containerID=container_1444731612741_0001_01_000019] is >>> running beyond physical memory limits. Current usage: 2.0 GB of 2 GB >>> physical memory used; 6.6 GB of 8 GB virtual memory used. Killing >>> container. >>> >>> >>> >>> >>> >>> Changed parameters in yarn-site.xml and mapred-site.xml files few times >>> but no joy. >>> >>> >>> >>> Finally the following changes in mapred-site.xml worked for me >>> >>> >>> >>> <property> >>> >>> <name>mapreduce.job.tracker.reserved.physicalmemory.mb</name> >>> >>> <value>1024</value> >>> >>> </property> >>> >>> >>> >>> <property> >>> >>> <name>mapreduce.map.memory.mb</name> >>> >>> <value>4096</value> >>> >>> </property> >>> >>> >>> >>> <property> >>> >>> <name>mapreduce.reduce.memory.mb</name> >>> >>> <value>4096</value> >>> >>> </property> >>> >>> >>> >>> <property> >>> >>> <name>mapreduce.map.java.opts</name> >>> >>> <value>-Xmx3072m</value> >>> >>> </property> >>> >>> >>> >>> <property> >>> >>> <name>mapreduce.reduce.java.opts</name> >>> >>> <value>-Xmx6144m</value> >>> >>> </property> >>> >>> >>> >>> And the following changes to yarn-site.xml >>> >>> >>> >>> <property> >>> >>> <name>yarn.nodemanager.vmem-check-enabled</name> >>> >>> <value>false</value> >>> >>> </property> >>> >>> <property> >>> >>> <name>yarn.nodemanager.resource.memory-mb</name> >>> >>> <value>8192</value> >>> >>> <description>Amount of physical memory, in MB, that can be allocated >>> for containers.</description> >>> >>> </property> >>> >>> <property> >>> >>> <name>yarn.nodemanager.vmem-pmem-ratio</name> >>> >>> <value>4</value> >>> >>> <description>Ratio between virtual memory to physical memory when >>> setting memory limits for containers</description> >>> >>> </property> >>> >>> >>> >>> I did a lot of web search but most resolution to this issue seems to be >>> cryptic or anecdotal. Anyone has better explanation I would be interested. >>> >>> >>> >>> Mich Talebzadeh >>> >>> >>> >>> *Sybase ASE 15 Gold Medal Award 2008* >>> >>> A Winning Strategy: Running the most Critical Financial Data on ASE 15 >>> >>> >>> http://login.sybase.com/files/Product_Overviews/ASE-Winning-Strategy-091908.pdf >>> >>> Author of the books* "A Practitioner’s Guide to Upgrading to Sybase ASE >>> 15", ISBN 978-0-9563693-0-7*. >>> >>> co-author *"Sybase Transact SQL Guidelines Best Practices", ISBN >>> 978-0-9759693-0-4* >>> >>> *Publications due shortly:* >>> >>> *Complex Event Processing in Heterogeneous Environments*, ISBN: >>> 978-0-9563693-3-8 >>> >>> *Oracle and Sybase, Concepts and Contrasts*, ISBN: 978-0-9563693-1-4, volume >>> one out shortly >>> >>> >>> >>> http://talebzadehmich.wordpress.com >>> >>> >>> >>> NOTE: The information in this email is proprietary and confidential. >>> This message is for the designated recipient only, if you are not the >>> intended recipient, you should destroy it immediately. Any information in >>> this message shall not be understood as given or endorsed by Peridale >>> Technology Ltd, its subsidiaries or their employees, unless expressly so >>> stated. It is the responsibility of the recipient to ensure that this email >>> is virus free, therefore neither Peridale Ltd, its subsidiaries nor their >>> employees accept any responsibility. >>> >>> >>> >> >> >
