Wondering if you have used the REST API to submit jobs:
http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Applications_APISubmit_Application
There are some issues that I have come across, but it does seem to work.
Also the message
java.io.IOException: File
/tmp/hadoop-yarn/staging/xeon/.staging/job_1432045089375_0001/job.split
could only be replicated to 0 nodes instead of minReplication (=1).
There are 1 datanode(s) running and 1 node(s) are excluded in this
operation.
One node excluded from the operation (?) do you know why it is excluded ?
-Rajesh
On Tue, May 19, 2015 at 7:34 AM, xeonmailinglist-gmail <
xeonmailingl...@gmail.com> wrote:
> This has been a real struggle to launch a remote MapReduce job. I know
> that there is the Netflix genie to submit the job, but for the purpose of
> this application (small and personal), I want to code it from scratch.
>
> I am debugging my code to see what is going on during the submission of a
> remote job, and now I have the error [1]. This error happens during the
> submission of the job, more precisely when it is writing in the remote
> HDFS. I have put the Hadoop code [2] where I get the error. The error [1]
> happens in the instruction out.close() of [2].
>
> The Namenode and the datanodes are working properly. I have 1 Namenode and
> 1 datanode. The replication factor is set to 1.
> Despite everything is running ok I get this error. Any hint so that I can
> see what is going on?
>
> [1]
>
> 2015-05-19 10:21:03,147 WARN
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Failed to
> place enough replicas, still in need of 1 to reach 1 (unavailab
> leStorages=[DISK], storagePolicy=BlockStoragePolicy{HOT:7,
> storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]},
> newBlock=true) All required storage
> types are unavailable: unavailableStorages=[DISK],
> storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK],
> creationFallbacks=[], replicationFallbacks=[ARCHIVE]}
> 2015-05-19 10:21:03,147 DEBUG org.apache.hadoop.ipc.Server: Served: addBlock
> queueTime= 1 procesingTime= 1 exception= IOException
> 2015-05-19 10:21:03,147 DEBUG
> org.apache.hadoop.security.UserGroupInformation: PrivilegedActionException
> as:xeon (auth:SIMPLE) cause:java.io.IOException: File /tmp/hadoop
> -yarn/staging/xeon/.staging/job_1432045089375_0001/job.split could only be
> replicated to 0 nodes instead of minReplication (=1). There are 1
> datanode(s) running and 1 no
> de(s) are excluded in this operation.
> 2015-05-19 10:21:03,148 INFO org.apache.hadoop.ipc.Server: IPC Server handler
> 6 on 9000, call org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from
> 194.117.18.101:39006 Call#18 Retry#0
> java.io.IOException: File
> /tmp/hadoop-yarn/staging/xeon/.staging/job_1432045089375_0001/job.split could
> only be replicated to 0 nodes instead of minReplication (=1). There are 1
> datanode(s) running and 1 node(s) are excluded in this operation.
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1549)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3200)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:641)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:482)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)
> 2015-05-19 10:21:03,148 DEBUG org.apache.hadoop.ipc.Server: IPC Server
> handler 6 on 9000: responding to
> org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from
> 194.117.18.101:39006 Call#18 Retry#0
>
> [2]
>
> public static <T extends InputSplit> void createSplitFiles(Path jobSubmitDir,
> Configuration conf, FileSystem fs, T[] splits)
> throws IOException, InterruptedException {
> FSDataOutputStream out = createFile(fs,
> JobSubmissionFiles.getJobSplitFile(jobSubmitDir), conf);
> SplitMetaInfo[] info = writeNewSplits(conf, splits, out);
> out.close();
>
> writeJobSplitMetaInfo(fs,JobSubmissionFiles.getJobSplitMetaFile(jobSubmitDir),
> new FsPermission(JobSubmissionFiles.JOB_FILE_PERMISSION),
> splitVersion,
> info);
> }
>
> Thanks,
>
> On 05/18/2015 03:56 PM, xeonmailinglist-gmail wrote:
>
> Shabab, I think so, but the Hadoop’s site says The user@ mailing list
> is the preferred mailing list for end-user questions and discussion.So I
> am using the right mailing list.
>
> Back to my problem, I think that this is a problem about HDFS security.
> But the strangest thing is that I have disabled it in hdfs-site.xml [1].
>
> I think that this error happens when MapReduce is trying to write the job
> configuration files in HDFS.
>
> I have set the username of the remote client in the mapreduce using the
> commands in [2].
>
> Now, I am looking to the Netflix Geni to figure it out how they do it, but
> right now I still haven’t found a solution to submit a remote job using
> Java. If anyone have a hint, or advice, please tell me. I really don’t
> understand why I get this error.
>
> [1]
>
> $ cat etc/hadoop/hdfs-site.xml
>
> <property> <name>dfs.permissions</name> <value>false</value> </property>
>
> [2]
>
> ```
> In Namenode host.
>
> $ sudo adduser xeon
> $ sudo adduser xeon ubuntu
>
> ```
>
> On 05/18/2015 02:46 PM, Shahab Yunus wrote:
>
> I think that poster wanted to unsubscribe from the mailing list?
>
> Gopy, if that is the case then please see this for that:
> https://hadoop.apache.org/mailing_lists.html
>
> Regards,
> Shahab
>
> On Mon, May 18, 2015 at 9:42 AM, xeonmailinglist-gmail <
> xeonmailingl...@gmail.com> wrote:
>
>> Why "Remove"?
>>
>>
>> On 05/18/2015 02:25 PM, Gopy Krishna wrote:
>>
>> REMOVE
>>
>> On Mon, May 18, 2015 at 6:54 AM, xeonmailinglist-gmail <
>> xeonmailingl...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I am trying to submit a remote job in Yarn MapReduce, but I can’t
>>> because I get the error [1]. I don’t have more exceptions in the other logs.
>>>
>>> My Mapreduce runtime have 1 *ResourceManager* and 3 *NodeManagers*, and
>>> the HDFS is running properly (all nodes are alive).
>>>
>>> I have looked to all logs, and I still don’t understand why I get this
>>> error. Any help to fix this? Is it a problem of the remote job that I am
>>> submitting?
>>>
>>> [1]
>>>
>>> $ less logs/hadoop-ubuntu-namenode-ip-172-31-17-45.log
>>>
>>> 2015-05-18 10:42:16,570 DEBUG org.apache.hadoop.hdfs.StateChange: *BLOCK*
>>> NameNode.addBlock: file
>>> /tmp/hadoop-yarn/staging/xeon/.staging/job_1431945660897_0001/job.split
>>> fileId=16394 for DFSClient_NONMAPREDUCE_-1923902075_14
>>> 2015-05-18 10:42:16,570 DEBUG org.apache.hadoop.hdfs.StateChange: BLOCK*
>>> NameSystem.getAdditionalBlock:
>>> /tmp/hadoop-yarn/staging/xeon/.staging/job_1431945660897_0001/job.
>>> split inodeId 16394 for DFSClient_NONMAPREDUCE_-1923902075_14
>>> 2015-05-18 10:42:16,571 DEBUG
>>> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Failed
>>> to choose remote rack (location = ~/default-rack), fallback to lo
>>> cal rack
>>> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy$NotEnoughReplicasException:
>>> at
>>> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:691)
>>> at
>>> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRemoteRack(BlockPlacementPolicyDefault.java:580)
>>> at
>>> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:348)
>>> at
>>> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:214)
>>> at
>>> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:111)
>>> at
>>> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:126)
>>> at
>>> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1545)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3200)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:641)
>>>
>>>
>>>
>>> --
>>> --
>>>
>>>
>>
>>
>> --
>> Thanks & Regards
>> Gopy
>> Rapisource LLC
>> Direct: 732-419-9663
>> Fax: 512-287-4047
>> Email: g...@rapisource.com
>> www.rapisource.com
>> http://www.linkedin.com/in/gopykrishna
>>
>> According to Bill S.1618 Title III passed by the 105th US Congress,this
>> message is not considered as "Spam" as we have included the contact
>> information.If you wish to be removed from our mailing list, please respond
>> with "remove" in the subject field.We apologize for any inconvenience
>> caused.
>>
>>
>> --
>> --
>>
>>
>
>
> --
> --
>
>
>
> --
> --
>
>
