Here is Altiscale's documentation about the topic. Do let me know if you have any more questions.
http://documentation.altiscale.com/heapsize-for-mappers-and-reducers On Tue, Oct 13, 2015 at 9:31 AM, Mich Talebzadeh <m...@peridale.co.uk> wrote: > Hi, > > > > I have been having some issues with loading data into hive from one table > to another for 1,767,886 rows. I was getting the following error > > > > Task with the most failures(4): > > ----- > > Task ID: > > task_1444731612741_0001_r_000000 > > > > URL: > > > http://0.0.0.0:8088/taskdetails.jsp?jobid=job_1444731612741_0001&tipid=task_1444731612741_0001_r_000000 > > ----- > > Diagnostic Messages for this Task: > > Container [pid=16238,containerID=container_1444731612741_0001_01_000019] is > running beyond physical memory limits. Current usage: 2.0 GB of 2 GB > physical memory used; 6.6 GB of 8 GB virtual memory used. Killing > container. > > > > > > Changed parameters in yarn-site.xml and mapred-site.xml files few times > but no joy. > > > > Finally the following changes in mapred-site.xml worked for me > > > > <property> > > <name>mapreduce.job.tracker.reserved.physicalmemory.mb</name> > > <value>1024</value> > > </property> > > > > <property> > > <name>mapreduce.map.memory.mb</name> > > <value>4096</value> > > </property> > > > > <property> > > <name>mapreduce.reduce.memory.mb</name> > > <value>4096</value> > > </property> > > > > <property> > > <name>mapreduce.map.java.opts</name> > > <value>-Xmx3072m</value> > > </property> > > > > <property> > > <name>mapreduce.reduce.java.opts</name> > > <value>-Xmx6144m</value> > > </property> > > > > And the following changes to yarn-site.xml > > > > <property> > > <name>yarn.nodemanager.vmem-check-enabled</name> > > <value>false</value> > > </property> > > <property> > > <name>yarn.nodemanager.resource.memory-mb</name> > > <value>8192</value> > > <description>Amount of physical memory, in MB, that can be allocated for > containers.</description> > > </property> > > <property> > > <name>yarn.nodemanager.vmem-pmem-ratio</name> > > <value>4</value> > > <description>Ratio between virtual memory to physical memory when > setting memory limits for containers</description> > > </property> > > > > I did a lot of web search but most resolution to this issue seems to be > cryptic or anecdotal. Anyone has better explanation I would be interested. > > > > Mich Talebzadeh > > > > *Sybase ASE 15 Gold Medal Award 2008* > > A Winning Strategy: Running the most Critical Financial Data on ASE 15 > > > http://login.sybase.com/files/Product_Overviews/ASE-Winning-Strategy-091908.pdf > > Author of the books* "A Practitioner’s Guide to Upgrading to Sybase ASE > 15", ISBN 978-0-9563693-0-7*. > > co-author *"Sybase Transact SQL Guidelines Best Practices", ISBN > 978-0-9759693-0-4* > > *Publications due shortly:* > > *Complex Event Processing in Heterogeneous Environments*, ISBN: > 978-0-9563693-3-8 > > *Oracle and Sybase, Concepts and Contrasts*, ISBN: 978-0-9563693-1-4, volume > one out shortly > > > > http://talebzadehmich.wordpress.com > > > > NOTE: The information in this email is proprietary and confidential. This > message is for the designated recipient only, if you are not the intended > recipient, you should destroy it immediately. Any information in this > message shall not be understood as given or endorsed by Peridale Technology > Ltd, its subsidiaries or their employees, unless expressly so stated. It is > the responsibility of the recipient to ensure that this email is virus > free, therefore neither Peridale Ltd, its subsidiaries nor their employees > accept any responsibility. > > >
