RE: Can you help me to install HDFS Federation and test?
No its not appearing from other name node. Here is the procedure I followed:In NameNode1 I ran following commandsbin/hdfs dfs -mkdir testbin/hdfs dfs -put dummy.txt test When ran bin/hdfs -ls test command from NameNode1 its listing file fin hdfs but if I ran same command from NameNode2 out put is ls: test : No such file or directory Thanks,Sandeep. Date: Wed, 18 Sep 2013 16:58:50 +0530 Subject: Re: Can you help me to install HDFS Federation and test? From: visioner.sa...@gmail.com To: user@hadoop.apache.org It shud be visible from every namenode machine have you tried this commmand bin/hdfs dfs -ls /yourdirectoryname/ On Wed, Sep 18, 2013 at 9:23 AM, Sandeep L sandeepvre...@outlook.com wrote: Hi, I resolved the issue.There is some problem with /etc/hosts file. One more question I would like to ask is: I created a directory in HDFS of NameNode1 and copied a file into it. My question is did it visible when I ran hadoop fs -ls PathToDirectory from NameNode2 machine? For me its not visible, can you explain with bit detailed. Thanks,Sandeep. Date: Tue, 17 Sep 2013 17:56:00 +0530 Subject: Re: Can you help me to install HDFS Federation and test? From: visioner.sa...@gmail.com To: user@hadoop.apache.org 1. make sure to check hadoop logs once u start u r datanode at /home/hadoop/hadoop-version(your)/logs 2. make sure all the datanodes are mentioned in slaves file and slaves file is placed on all machines 3. check which datanode is not available check log file of that machine are both the machines able to do a passwordleh with each other4. check your etc/hosts file make sure all your node machines ip is mentioned there 5. make sure you have datanode folder created as mentioned in config file.. let me know if u have any problem.. On Tue, Sep 17, 2013 at 2:44 PM, Sandeep L sandeepvre...@outlook.com wrote: Hi, I tried to install HDFS federation with the help of document given by you. I have small issue.I used 2 slave in setup, both will act as namenode and datanode. Now the issue is when I am looking at home pages of both namenodes only one datanode is appearing.As per my understanding 2 datanodes should appear in both namenodes home pages. Can you please let me if am missing any thing? Thanks,Sandeep. Date: Wed, 11 Sep 2013 15:34:38 +0530 Subject: Re: Can you help me to install HDFS Federation and test? From: visioner.sa...@gmail.com To: user@hadoop.apache.org may be this can help you On Wed, Sep 11, 2013 at 3:07 PM, Oh Seok Keun ohs...@gmail.com wrote: Hello~ I am Rho working in korea I am trying to install HDFS Federation( with 2.1.0 beta version ) and to test After 2.1.0 ( for federation test ) install I have a big trouble when file putting test I command to hadoopCan you help me to install HDFS Federation and test?./bin/hadoop fs -put test.txt /NN1/ there is error message put: Renames across FileSystems not supported But ./bin/hadoop fs -put test.txt hdfs://namnode:8020/NN1/ is ok Why this is happen? This is very sad to me ^^ Can you explain why this is happend and give me solution? Additionally Namenode1 is access to own Namespace( named NN1 ) and Namenode2 is access to own Namespace( named NN2 ) When making directory in namenode1 server ./bin/hadoop fs -mkdir /NN1/nn1_org is ok but ./bin/hadoop fs -mkdir /NN2/nn1_org is error Error message is /NN2/nn1_org': No such file or directory I think this is very right But in namenode2 server./bin/hadoop fs -mkdir /NN1/nn2_org is ok but ./bin/hadoop fs -mkdir /NN2/nn2_org is errorError message is mkdir: `/NN2/nn2_org': No such file or directory I think when making directory in NN1 is error and making directory in NN2 is ok Why this is happen and can you give solution?
Re: Issue: Max block location exceeded for split error when running hive
We are using the default replication factor of 3. When new files are put on HDFS we never override the replication factor. When there is more data involved it fails on a larger split size. On Wed, Sep 18, 2013 at 6:34 PM, Harsh J ha...@cloudera.com wrote: Do your input files carry a replication factor of 10+? That could be one cause behind this. On Thu, Sep 19, 2013 at 6:20 AM, Murtaza Doctor murtazadoc...@gmail.com wrote: Folks, Any one run into this issue before: java.io.IOException: Max block location exceeded for split: Paths: /foo/bar InputFormatClass: org.apache.hadoop.mapred.TextInputFormat splitsize: 15 maxsize: 10 at org.apache.hadoop.mapreduce.split.JobSplitWriter.writeOldSplits(JobSplitWriter.java:162) at org.apache.hadoop.mapreduce.split.JobSplitWriter.createSplitFiles(JobSplitWriter.java:87) at org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:501) at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:471) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:366) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1269) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1266) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1266) at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:606) at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:601) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:601) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:586) at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:447) When we set the property to something higher as suggested like: mapreduce.job.max.split.locations = more than on what it failed then the job runs successfully. I am trying to dig up additional documentation on this since the default seems to be 10, not sure how that limit was set. Additionally what is the recommended value and what factors does it depend on? We are running YARN, the actual query is Hive on CDH 4.3, with Hive version 0.10 Any pointers in this direction will be helpful. Regards, md -- Harsh J
Stable version of Hadoop
Hi Folks, I want to use hbase for my data storage on the top of HDFS, Please help me to find out the best version which i should used , like CDH4 I data size would be around 500gb - 5Tb. My operations would be write intensive Thanks
HDFs file-create performance
Are there any rough numbers one can give me regarding the latency of creating, writing, and closing a small HDFS-based file? Does replication have a big impact? I am trying to decide whether to communicate some modestly-sized (~200KB) information via HDFS files or go to the trouble of creating a protocol. Thanks John
How to make hadoop use all nodes?
Hi all, I am working with Hadoop 2.0.5 (I plan to migrate to 2.1.0 soon). When I am starting a Job, I notice that some nodes are not used or partially used. For example, if my nodes can hold 2 containers, I notice that some nodes are not running any or just 1 while others are running 2. All my nodes are configured the same way. Is this an expected behavior (maybe in case others jobs are started) ? Is there a configuration to change this behavior? Thanks, Antoine
Re: Issue: Max block location exceeded for split error when running hive
We have this job submit property buried in hive that defaults to 10. We should make that configurable. On Wed, Sep 18, 2013 at 9:34 PM, Harsh J ha...@cloudera.com wrote: Do your input files carry a replication factor of 10+? That could be one cause behind this. On Thu, Sep 19, 2013 at 6:20 AM, Murtaza Doctor murtazadoc...@gmail.com wrote: Folks, Any one run into this issue before: java.io.IOException: Max block location exceeded for split: Paths: /foo/bar InputFormatClass: org.apache.hadoop.mapred.TextInputFormat splitsize: 15 maxsize: 10 at org.apache.hadoop.mapreduce.split.JobSplitWriter.writeOldSplits(JobSplitWriter.java:162) at org.apache.hadoop.mapreduce.split.JobSplitWriter.createSplitFiles(JobSplitWriter.java:87) at org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:501) at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:471) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:366) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1269) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1266) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1266) at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:606) at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:601) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:601) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:586) at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:447) When we set the property to something higher as suggested like: mapreduce.job.max.split.locations = more than on what it failed then the job runs successfully. I am trying to dig up additional documentation on this since the default seems to be 10, not sure how that limit was set. Additionally what is the recommended value and what factors does it depend on? We are running YARN, the actual query is Hive on CDH 4.3, with Hive version 0.10 Any pointers in this direction will be helpful. Regards, md -- Harsh J
Re: Issue: Max block location exceeded for split error when running hive
It used to throw a warning in 1.03 and now has become an IOException. I was more trying to figure out why it is exceeding the limit even though the replication factor is 3. Also Hive may use CombineInputSplit or some version of it, are we saying it will always exceed the limit of 10? On Thu, Sep 19, 2013 at 10:05 AM, Edward Capriolo edlinuxg...@gmail.comwrote: We have this job submit property buried in hive that defaults to 10. We should make that configurable. On Wed, Sep 18, 2013 at 9:34 PM, Harsh J ha...@cloudera.com wrote: Do your input files carry a replication factor of 10+? That could be one cause behind this. On Thu, Sep 19, 2013 at 6:20 AM, Murtaza Doctor murtazadoc...@gmail.com wrote: Folks, Any one run into this issue before: java.io.IOException: Max block location exceeded for split: Paths: /foo/bar InputFormatClass: org.apache.hadoop.mapred.TextInputFormat splitsize: 15 maxsize: 10 at org.apache.hadoop.mapreduce.split.JobSplitWriter.writeOldSplits(JobSplitWriter.java:162) at org.apache.hadoop.mapreduce.split.JobSplitWriter.createSplitFiles(JobSplitWriter.java:87) at org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:501) at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:471) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:366) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1269) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1266) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1266) at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:606) at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:601) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:601) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:586) at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:447) When we set the property to something higher as suggested like: mapreduce.job.max.split.locations = more than on what it failed then the job runs successfully. I am trying to dig up additional documentation on this since the default seems to be 10, not sure how that limit was set. Additionally what is the recommended value and what factors does it depend on? We are running YARN, the actual query is Hive on CDH 4.3, with Hive version 0.10 Any pointers in this direction will be helpful. Regards, md -- Harsh J
YARN MapReduce 2 concepts
I am going through the concepts of resource manager, application master and node manager. As I undersand resource manager receives the job submission and launches application master. It also launches node manager to monitor application master. My questions are: 1. Is Node manager long lived and that one node manager monitors all the containers launed on the data nodes? 2. How is resource negotiation done between the application master and the resource manager? In other words what happens during this step? Does resource manager looks at the active and pending tasks and resources consumed by those before giving containers to the application master? 3. As it happens in old map reduce cluster that task trackers sends periodic heartbeats to the job tracker nodes. How does this compare to YARN? It looks like application master is a task tracker? Little confused here. 4. It looks like client polls application master to get the progress of the job but initially client connects to the resource manager. How does client gets reference to the application master? Does it mean that client gets the node ip/port from resource manager where application master was launced by the resource manager?
Re: HDFS federation Configuration
Have you looked at - http://hadoop.apache.org/docs/r2.1.0-beta/hadoop-project-dist/hadoop-hdfs/Federation.html Let me know if the document is not clear or needs improvements. Regards, Suresh On Thu, Sep 19, 2013 at 12:01 PM, Manickam P manicka...@outlook.com wrote: Guys, I need some tutorials to configure fedration. Can you pls suggest me some? Thanks, Manickam P -- http://hortonworks.com/download/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
HDFS federation Configuration
Guys, I need some tutorials to configure fedration. Can you pls suggest me some? Thanks, Manickam P
[no subject]
-- This e-mail and any attachments transmitted with it are for the sole use of the intended recipient(s) and may contain confidential , proprietary or privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this e-mail or any action taken in reliance on this e-mail is strictly prohibited and may be unlawful.
Task status query
How does a YARN application master typically query ongoing status (like percentage completion) of its tasks? I would like to be able to ultimately relay information to the user like: 100 tasks are scheduled 10 tasks are complete 4 tasks are running and they are (4%, 10%, 50%, 70%) complete But, given that YARN tasks are simply executables, how can the AM even get at this information? Can the AM get access to stdout/stderr? Thanks John
Re: Name node High Availability in Cloudera 4.1.1
Please do not cross-post these emails to hdfs-user. The relevant email list is only cdh-user. On Thu, Sep 19, 2013 at 1:44 AM, Pavan Kumar Polineni smartsunny...@gmail.com wrote: Hi all, *Name Node High Availability Job tracker high availability* is there in Cloudera 4.1.1 ? If not, Then what are the properties need to change in Cloudera 4.1.1 to make the cluster as High availability. please help on this.. Thanks in Advance -- Pavan Kumar Polineni -- http://hortonworks.com/download/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Issue: Max block location exceeded for split error when running hive
I am assuming you have looked at this already: https://issues.apache.org/jira/browse/MAPREDUCE-5186 You do have a workaround here to increase *mapreduce.job.max.split.locations *value in hive configuration, or do we need more than that here ? -Rahul On Thu, Sep 19, 2013 at 11:00 AM, Murtaza Doctor murtazadoc...@gmail.comwrote: It used to throw a warning in 1.03 and now has become an IOException. I was more trying to figure out why it is exceeding the limit even though the replication factor is 3. Also Hive may use CombineInputSplit or some version of it, are we saying it will always exceed the limit of 10? On Thu, Sep 19, 2013 at 10:05 AM, Edward Capriolo edlinuxg...@gmail.comwrote: We have this job submit property buried in hive that defaults to 10. We should make that configurable. On Wed, Sep 18, 2013 at 9:34 PM, Harsh J ha...@cloudera.com wrote: Do your input files carry a replication factor of 10+? That could be one cause behind this. On Thu, Sep 19, 2013 at 6:20 AM, Murtaza Doctor murtazadoc...@gmail.com wrote: Folks, Any one run into this issue before: java.io.IOException: Max block location exceeded for split: Paths: /foo/bar InputFormatClass: org.apache.hadoop.mapred.TextInputFormat splitsize: 15 maxsize: 10 at org.apache.hadoop.mapreduce.split.JobSplitWriter.writeOldSplits(JobSplitWriter.java:162) at org.apache.hadoop.mapreduce.split.JobSplitWriter.createSplitFiles(JobSplitWriter.java:87) at org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:501) at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:471) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:366) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1269) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1266) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1266) at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:606) at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:601) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:601) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:586) at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:447) When we set the property to something higher as suggested like: mapreduce.job.max.split.locations = more than on what it failed then the job runs successfully. I am trying to dig up additional documentation on this since the default seems to be 10, not sure how that limit was set. Additionally what is the recommended value and what factors does it depend on? We are running YARN, the actual query is Hive on CDH 4.3, with Hive version 0.10 Any pointers in this direction will be helpful. Regards, md -- Harsh J
Re: Yarn Exception while getting JobStatus
Hey Harsh, Here is the more complete stacktrace. I had truncated it earlier since it was application specific. Let me know if this helps. Thrown: java.lang.RuntimeException java.io.IOException Thrown-StackTrace: at HadoopJobUpdaterProcess.execute(HadoopJobUpdaterProcess.java:37) at HadoopJobUpdaterProcess.execute(HadoopJobUpdaterProcess.java:19) at org.quartz.core.JobRunShell.run(JobRunShell.java:202) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:525) Cause0: java.lang.RuntimeException java.io.IOException Cause0-StackTrace: at hadoop.client.RemoteHadoopJobClientImpl.getJobStatus(RemoteHadoopJobClientImpl.java:45) at HadoopJobUtil.updateRunningHadoopJobs(HadoopJobUtil.java:260) at HadoopJobUpdaterProcess.executeInner(HadoopJobUpdaterProcess.java:54) at HadoopJobUpdaterProcess.execute(HadoopJobUpdaterProcess.java:35) ... 6 shared with parent Cause1: java.io.IOException Cause1-StackTrace: at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:317) at org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:385) at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:487) at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:184) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:635) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:633) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) at org.apache.hadoop.mapred.JobClient.getJobUsingCluster(JobClient.java:633) at org.apache.hadoop.mapred.JobClient.getJob(JobClient.java:651) at hadoop.client.RemoteHadoopJobClientImpl.getJobStatus(RemoteHadoopJobClientImpl.java:39) ... 16 shared with parent On Wed, Sep 18, 2013 at 6:36 PM, Harsh J ha...@cloudera.com wrote: Please post your complete stacktrace - the provided one appears incomplete/truncated. On Thu, Sep 19, 2013 at 1:07 AM, Siddhi Mehta smehtau...@gmail.com wrote: Hey All, I am running a Hadoop 2.0(cdh4.2.1) cluster. From the job client whenever we try to get the job status from the cluster we run into the following IOException. The exception has no message on the cause of the IOException. Any ideas on what could be causing this issue. Looking at the code it seems like it fails while talking to the jobhistoryserver. The application where the job client runs has most of the ports blocked. We opened a the following ports for the job client to talk to the cluster(8030,8031,8032 8033,19888, 10020) but still see the exception. Is there a defined list of ports on which a mapreduce client communicates with the Hadoop Cluster Cause1: java.io.IOException Cause1-StackTrace: at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:317) at org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:385) at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:487) at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:184) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:635) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:633) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) at org.apache.hadoop.mapred.JobClient.getJobUsingCluster(JobClient.java:633) at org.apache.hadoop.mapred.JobClient.getJob(JobClient.java:651) Thanks, Siddhi -- Harsh J
Re: Issue: Max block location exceeded for split error when running hive
What are the ramifications of setting a hard coded value in our scripts and then changing parameters which influence the input data size. I.e. I want to run across 1 day worth of data, then a different day I want to run against 30 days? On Thu, Sep 19, 2013 at 3:11 PM, Rahul Jain rja...@gmail.com wrote: I am assuming you have looked at this already: https://issues.apache.org/jira/browse/MAPREDUCE-5186 You do have a workaround here to increase *mapreduce.job.max.split.locations *value in hive configuration, or do we need more than that here ? -Rahul On Thu, Sep 19, 2013 at 11:00 AM, Murtaza Doctor murtazadoc...@gmail.comwrote: It used to throw a warning in 1.03 and now has become an IOException. I was more trying to figure out why it is exceeding the limit even though the replication factor is 3. Also Hive may use CombineInputSplit or some version of it, are we saying it will always exceed the limit of 10? On Thu, Sep 19, 2013 at 10:05 AM, Edward Capriolo edlinuxg...@gmail.comwrote: We have this job submit property buried in hive that defaults to 10. We should make that configurable. On Wed, Sep 18, 2013 at 9:34 PM, Harsh J ha...@cloudera.com wrote: Do your input files carry a replication factor of 10+? That could be one cause behind this. On Thu, Sep 19, 2013 at 6:20 AM, Murtaza Doctor murtazadoc...@gmail.com wrote: Folks, Any one run into this issue before: java.io.IOException: Max block location exceeded for split: Paths: /foo/bar InputFormatClass: org.apache.hadoop.mapred.TextInputFormat splitsize: 15 maxsize: 10 at org.apache.hadoop.mapreduce.split.JobSplitWriter.writeOldSplits(JobSplitWriter.java:162) at org.apache.hadoop.mapreduce.split.JobSplitWriter.createSplitFiles(JobSplitWriter.java:87) at org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:501) at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:471) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:366) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1269) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1266) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1266) at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:606) at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:601) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:601) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:586) at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:447) When we set the property to something higher as suggested like: mapreduce.job.max.split.locations = more than on what it failed then the job runs successfully. I am trying to dig up additional documentation on this since the default seems to be 10, not sure how that limit was set. Additionally what is the recommended value and what factors does it depend on? We are running YARN, the actual query is Hive on CDH 4.3, with Hive version 0.10 Any pointers in this direction will be helpful. Regards, md -- Harsh J
Re: Issue: Max block location exceeded for split error when running hive
Matt, It would be better for you to do an global config update: set *mapreduce.job.max.split.locations *to at least the number of datanodes in your cluster, either in hive-site.xml or mapred-site.xml. Either case, this is a sensible configuration update if you are going to use CombineFileInputFormat to read input data in hive. -Rahul On Thu, Sep 19, 2013 at 3:31 PM, Matt Davies m...@mattdavies.net wrote: What are the ramifications of setting a hard coded value in our scripts and then changing parameters which influence the input data size. I.e. I want to run across 1 day worth of data, then a different day I want to run against 30 days? On Thu, Sep 19, 2013 at 3:11 PM, Rahul Jain rja...@gmail.com wrote: I am assuming you have looked at this already: https://issues.apache.org/jira/browse/MAPREDUCE-5186 You do have a workaround here to increase *mapreduce.job.max.split.locations *value in hive configuration, or do we need more than that here ? -Rahul On Thu, Sep 19, 2013 at 11:00 AM, Murtaza Doctor murtazadoc...@gmail.com wrote: It used to throw a warning in 1.03 and now has become an IOException. I was more trying to figure out why it is exceeding the limit even though the replication factor is 3. Also Hive may use CombineInputSplit or some version of it, are we saying it will always exceed the limit of 10? On Thu, Sep 19, 2013 at 10:05 AM, Edward Capriolo edlinuxg...@gmail.com wrote: We have this job submit property buried in hive that defaults to 10. We should make that configurable. On Wed, Sep 18, 2013 at 9:34 PM, Harsh J ha...@cloudera.com wrote: Do your input files carry a replication factor of 10+? That could be one cause behind this. On Thu, Sep 19, 2013 at 6:20 AM, Murtaza Doctor murtazadoc...@gmail.com wrote: Folks, Any one run into this issue before: java.io.IOException: Max block location exceeded for split: Paths: /foo/bar InputFormatClass: org.apache.hadoop.mapred.TextInputFormat splitsize: 15 maxsize: 10 at org.apache.hadoop.mapreduce.split.JobSplitWriter.writeOldSplits(JobSplitWriter.java:162) at org.apache.hadoop.mapreduce.split.JobSplitWriter.createSplitFiles(JobSplitWriter.java:87) at org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:501) at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:471) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:366) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1269) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1266) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1266) at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:606) at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:601) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:601) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:586) at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:447) When we set the property to something higher as suggested like: mapreduce.job.max.split.locations = more than on what it failed then the job runs successfully. I am trying to dig up additional documentation on this since the default seems to be 10, not sure how that limit was set. Additionally what is the recommended value and what factors does it depend on? We are running YARN, the actual query is Hive on CDH 4.3, with Hive version 0.10 Any pointers in this direction will be helpful. Regards, md -- Harsh J
Re: How to make hadoop use all nodes?
Hi, Let me clarify few things. 1) you are making container requests which are not explicitly looking for certain nodes. (No white listing). 2) All nodes are identical in terms of resources (memory/cores) and every container requires same amount of resources. 3) All nodes have capacity to run say 2 containers. 4) You have 20 nodes. Now if an application is running and is requesting 20 containers then you can not say that you will get all on different nodes (uniformly distributed). It more depends on which node heartbeated to the Resource manager at what time and how much memory is available with it and also how many applications are present in queue and how much they are requesting at what request priorities. If it has say sufficient memory to run 2 containers then they will get allocated (This allocation is quite complex ..I am assuming very simple * reuqest). So you may see few running 2, few running 1 where as few with 0 containers. I hope it clarifies your doubt. Thanks, Omkar Joshi *Hortonworks Inc.* http://www.hortonworks.com On Thu, Sep 19, 2013 at 7:19 AM, Vandecreme, Antoine antoine.vandecr...@nist.gov wrote: Hi all, I am working with Hadoop 2.0.5 (I plan to migrate to 2.1.0 soon). When I am starting a Job, I notice that some nodes are not used or partially used. For example, if my nodes can hold 2 containers, I notice that some nodes are not running any or just 1 while others are running 2. All my nodes are configured the same way. Is this an expected behavior (maybe in case others jobs are started) ? Is there a configuration to change this behavior? Thanks, Antoine -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
RE: HDFS performance with an without replication
Thanks, that makes sense. john -Original Message- From: Harsh J [mailto:ha...@cloudera.com] Sent: Sunday, September 15, 2013 12:39 PM To: user@hadoop.apache.org Subject: Re: HDFS performance with an without replication Write performance improves with lesser replicas (as a result of synchronous and sequenced write pipelines in HDFS). Reads would be the same, unless you're unable to schedule a rack-local read (at worst case) due to only one (busy) rack holding it. On Sun, Sep 15, 2013 at 10:38 PM, John Lilley john.lil...@redpoint.net wrote: In our YARN application, we are considering whether to store temporary data with replication=1 or replication=3 (or give the user an option). Obviously there is a tradeoff between reliability and performance, but on smaller clusters I'd expect this to be less of an issue. What is the difference in write performance using replication=1 vs 3? For reading I'd expect the performance to be roughly requivalent. john -- Harsh J
Re: YARN MapReduce 2 concepts
Hi Mohit, answers inline On Fri, Sep 20, 2013 at 1:33 AM, Mohit Anchlia mohitanch...@gmail.comwrote: I am going through the concepts of resource manager, application master and node manager. As I undersand resource manager receives the job submission and launches application master. It also launches node manager to monitor application master. My questions are: 1. Is Node manager long lived and that one node manager monitors all the containers launed on the data nodes? Correct 2. How is resource negotiation done between the application master and the resource manager? In other words what happens during this step? Does resource manager looks at the active and pending tasks and resources consumed by those before giving containers to the application master? The ResourceManager contains a pluggable scheduler that is responsible for deciding which applications to give resources to when they become available. When a NodeManager heartbeats to the ResourceManager, the scheduler will decide whether there are any containers it should place on that node for an application, and will let the Application Master know about its decision on the next AM-RM heartbeat. Here's documentation for the two recommended schedulers: http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html 3. As it happens in old map reduce cluster that task trackers sends periodic heartbeats to the job tracker nodes. How does this compare to YARN? It looks like application master is a task tracker? Little confused here. The analog to this is the NodeManager sending periodic heartbeats to the ResourceManager. The Application Master also sends periodic heartbeats to the NodeManagers that its containers are running on to check on their status. 4. It looks like client polls application master to get the progress of the job but initially client connects to the resource manager. How does client gets reference to the application master? Does it mean that client gets the node ip/port from resource manager where application master was launced by the resource manager? Correct
Re: Issue: Max block location exceeded for split error when running hive
Thanks Rahul. Our ops people have implemented the config change. On Thursday, September 19, 2013, Rahul Jain wrote: Matt, It would be better for you to do an global config update: set *mapreduce.job.max.split.locations *to at least the number of datanodes in your cluster, either in hive-site.xml or mapred-site.xml. Either case, this is a sensible configuration update if you are going to use CombineFileInputFormat to read input data in hive. -Rahul On Thu, Sep 19, 2013 at 3:31 PM, Matt Davies m...@mattdavies.net wrote: What are the ramifications of setting a hard coded value in our scripts and then changing parameters which influence the input data size. I.e. I want to run across 1 day worth of data, then a different day I want to run against 30 days? On Thu, Sep 19, 2013 at 3:11 PM, Rahul Jain rja...@gmail.com wrote: I am assuming you have looked at this already: https://issues.apache.org/jira/browse/MAPREDUCE-5186 You do have a workaround here to increase *mapreduce.job.max.split.locations *value in hive configuration, or do we need more than that here ? -Rahul On Thu, Sep 19, 2013 at 11:00 AM, Murtaza Doctor murtazadoc...@gmail.comwrote: It used to throw a warning in 1.03 and now has become an IOException. I was more trying to figure out why it is exceeding the limit even though the replication factor is 3. Also Hive may use CombineInputSplit or some version of it, are we saying it will always exceed the limit of 10? On Thu, Sep 19, 2013 at 10:05 AM, Edward Capriolo edlinuxg...@gmail.comwrote: We have this job submit property buried in hive that defaults to 10. We should make that configurable. On Wed, Sep 18, 2013 at 9:34 PM, Harsh J ha...@cloudera.com wrote: Do your input files carry a replication factor of 10+? That could be one cause behind this. On Thu, Sep 19, 2013 at 6:20 AM, Murtaza Doctor murtazadoc...@gmail.com wrote: Folks, Any one run into this issue before: java.io.IOException: Max block location exceeded for split: Paths: /foo/bar InputFormatClass: org.apache.hadoop.mapred.TextInputFormat splitsize: 15 maxsize: 10 at org.apache.hadoop.mapreduce.split.JobSplitWriter.writeOldSplits(JobSplitWriter.java:162) at org.apache.hadoop.mapreduce.split.JobSplitWriter.createSplitFiles(JobSplitWriter.java:87) at org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:501) at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:471) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:366) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1269) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1266) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) at org.apac
Re: Task status query
Hi John, YARN tasks can be more than simple executables. In case of MR, for example, tasks talk to the AM and report their individual progress and counters back to it, via a specific protocol (over the network), giving the AM more data to compute an near-accurate global progress. On Fri, Sep 20, 2013 at 12:18 AM, John Lilley john.lil...@redpoint.net wrote: How does a YARN application master typically query ongoing status (like percentage completion) of its tasks? I would like to be able to ultimately relay information to the user like: 100 tasks are scheduled 10 tasks are complete 4 tasks are running and they are (4%, 10%, 50%, 70%) complete But, given that YARN tasks are simply executables, how can the AM even get at this information? Can the AM get access to stdout/stderr? Thanks John -- Harsh J
