Re: how to prevent JAVA HEAP OOM in shuffle process?
Setting map memory at command line with the *new* api: hadoop jar hadoop-mapreduce-examples-2.2.0.2.0.6.0-76.jar wordcount -Dmapreduce.map.java.opts=-Xmx1024m /user/hdfs/ades /tmp/wordcount Setting map memory at command line with the *old* api: hadoop jar hadoop-mapreduce-examples-2.2.0.2.0.6.0-76.jar wordcount -Dmapred.map.child.java.opts=-Xmx1024m /user/hdfs/ades /tmp/wordcount This is more for the user list, not hdfs-dev. On Mon, Dec 2, 2013 at 7:07 AM, ch huang wrote: > hi,maillist: > i recent get a problem,when i run MR job, it happened OOM in shuffle > process,the options about MR is default,not changed,which option should i > tuning? thanks > -- * Adam Muise * Solutions Engineer -- Phone:416-417-4037 Email: amu...@hortonworks.com Website: http://www.hortonworks.com/ * Follow Us: * <http://facebook.com/hortonworks/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> <http://twitter.com/hortonworks?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> <http://www.linkedin.com/company/hortonworks?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> [image: photo] Latest From Our Blog: SAP HANA + Hadoop: A Perfect Match <http://hortonworks.com/blog/sap-hana-hadoop-a-perfect-match/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: HDFS read/write data throttling
See https://issues.apache.org/jira/browse/HDFS-3475 Please note that this has met with many unexpected impacts on workload. Be careful and be mindful of your Datanode memory and network capacity. On Mon, Nov 11, 2013 at 1:59 PM, lohit wrote: > Hello Devs, > > Wanted to reach out and see if anyone has thought about ability to throttle > data transfer within HDFS. One option we have been thinking is to throttle > on a per FileSystem basis, similar to Statistics in FileSystem. This would > mean anyone with handle to HDFS/Hftp will be throttled globally within JVM. > Right value to come up for this would be based on type of hardware we use > and how many tasks/clients we allow. > > On the other hand doing something like this at FileSystem layer would mean > many other tasks such as Job jar copy, DistributedCache copy and any hidden > data movement would also be throttled. We wanted to know if anyone has had > such requirement on their clusters in the past and what was the thinking > around it. Appreciate your inputs/comments > > -- > Have a Nice Day! > Lohit > -- * Adam Muise * Solutions Engineer -- Phone:416-417-4037 Email: amu...@hortonworks.com Website: http://www.hortonworks.com/ * Follow Us: * <http://facebook.com/hortonworks/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> <http://twitter.com/hortonworks?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> <http://www.linkedin.com/company/hortonworks?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> [image: photo] Latest From Our Blog: How to use R and other non-Java languages in MapReduce and Hive <http://hortonworks.com/blog/using-r-and-other-non-java-languages-in-mapreduce-and-hive/?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature> -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[jira] [Created] (HDFS-3971) Add a resume feature to the copyFromLocal and put commands
Adam Muise created HDFS-3971: Summary: Add a resume feature to the copyFromLocal and put commands Key: HDFS-3971 URL: https://issues.apache.org/jira/browse/HDFS-3971 Project: Hadoop HDFS Issue Type: New Feature Components: tools Affects Versions: 2.0.1-alpha Reporter: Adam Muise Priority: Minor Fix For: 2.0.1-alpha Add a resume feature to the copyFromLocal command. Failures in large transfers result in a great deal of wasted time. For large files, it would be good to be able to continue from the last good block onwards. The file would have to be unavailable to other clients for reads or regular writes until the "resume" process was completed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
