Job with only maptasks and map output still in local disk?

Hi, I noticed that when we have a mapreduce job with no reduce tasks, YARN saves the map output is the HDFS. I want that the job still save the map output in the local disk. In YARN, is it possible to have a mapreduce job that only executes map tasks (no reduce tasks to execute), and that

Is there a way to submit a job using the YARN REST API?

Hi, Is there a way to submit a job using the YARN REST API? Thanks,

Submit mapreduce job in remote YARN

Hi, I would like to submit a mapreduce job in a remote YARN cluster. Can I do this in java, or using a REST API? Thanks,

Steps for container release

Hi everyone, I was trying to understand the process that makes the resources of a container available again to the ResourceManager. As far as I can guess from the logs, the AM: - sends a stop request to the NodeManager for the specific container - suddenly tells the RM about the release of the

Re: Is there a way to submit a job using the YARN REST API?

Please take a look at https://issues.apache.org/jira/browse/MAPREDUCE-5874 Cheers On Feb 20, 2015, at 3:11 AM, xeonmailinglist xeonmailingl...@gmail.com wrote: Hi, Is there a way to submit a job using the YARN REST API? Thanks,

BLOCK and Split size question

Hello Every one, I have couple of doubts can any one please point me in right direction. 1What exactly happen when I want to copy 1TB file to Hadoop Cluster using copyfromlocal command 1 what will be the split size? will it be same as the block size? 2 What is a block and split? If we have

How to Tune Hadoop Cluster from Administrator prospective

Hi, How to Tune Hadoop Cluster from Administrator prospective ? What parameters we should consider etc? What to look for performance tuning ? Thanks Krish

Scheduling in YARN according to available resources

I have 7 nodes in my Hadoop cluster [8GB RAM and 4VCPUs to each nodes], 1 Namenode + 6 datanodes. I followed the link from Hortonwroks [ http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.6.0/bk_installing_manually_book/content/rpm-chap1-11.html ] and made calculation according to the hardware

Get method in Writable

Am I able to get the values from writable of a previous job. ie I have 2 MR jobs *MR 1:* I need to pass 3 element as values from reducer and the key is NullWritable. So I created a custom writable class to achieve this. * public class TreeInfoWritable implements Writable{* * DoubleWritable

secure checksum in HDFS

Hi, Is it possible to use SHA-256, or MD5 as a checksum in a file in HDFS? Thanks,

YARN container lauch failed exception and mapred-site.xml configuration

I have 7 nodes in my Hadoop cluster [8GB RAM and 4VCPUs to each nodes], 1 Namenode + 6 datanodes. **EDIT-1@ARNON:** I followed the link, mad calculation according to the hardware configruation on my nodes and have added the update mapred-site and yarn-site.xml files in my question. Still my

Re: Scheduling in YARN according to available resources

I had an issue very similar, I changed and used Oracle JDK. There is nothing I see wrong with your configuration in my first look, thanks Regards, Nair On Sat, Feb 21, 2015 at 1:42 AM, tesm...@gmail.com tesm...@gmail.com wrote: I have 7 nodes in my Hadoop cluster [8GB RAM and 4VCPUs to each

Re: Encryption At Rest Question

In case of SSL enabled cluster, the DEK will be encrypted on the wire by the SSL layer. In case of non-SSL enabled cluster, it is not. But the intercepter only gets the DEK and not the encrypted data, so the data is still safe. Only if the intercepter also manages to gain access to the encrypted

Re: Scheduling in YARN according to available resources

Thanks for your answer Nair, Is installing Oracle JDK on Ubuntu is that complicated as described in this link http://askubuntu.com/questions/56104/how-can-i-install-sun-oracles-proprietary-java-jdk-6-7-8-or-jre Is there an alternate? Regards On Sat, Feb 21, 2015 at 6:50 AM, R Nair

Fwd: YARN container lauch failed exception and mapred-site.xml configuration

I have 7 nodes in my Hadoop cluster [8GB RAM and 4VCPUs to each nodes], 1 Namenode + 6 datanodes. I followed the link o horton works [ http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.6.0/bk_installing_manually_book/content/rpm-chap1-11.html] and made calculation according to the hardware

Re: secure checksum in HDFS

There seem to be some work done on this here: https://issues.apache.org/jira/browse/HADOOP-9209 3rd party tool: https://github.com/rdsr/hdfs-checksum Regards, Shahab On Fri, Feb 20, 2015 at 12:39 PM, xeonmailinglist xeonmailingl...@gmail.com wrote: Hi, Is it possible to use SHA-256, or MD5

Re: How to get Hadoop's Generic Options value

Here is an example: https://adhoop.wordpress.com/2012/02/16/generate-a-list-of-anagrams-round-3/ -Rajesh On Thu, Feb 19, 2015 at 9:32 PM, Haoming Zhang haoming.zh...@outlook.com wrote: Thanks guys, I will try your solutions later and update the result! -- From:

Re: Submit mapreduce job in remote YARN

yes, you can do this in java, if these conditions are satisfied 1. your client is in the same network with the hadoop cluster 2. add the hadoop configuration to your java classpath, then the jvm will load the hadoop configuration but the suggesttiong way is hadoop jar 2015-02-20

Re: suspend and resume a job in execution?

I am not aware of an API that would let you do this. You may be able to move an application to a queue with 0 resources to achieve the desired behavior but I'm not entirely sure. On Wednesday, February 18, 2015 9:24 AM, xeonmailinglist xeonmailingl...@gmail.com wrote: By job, I