Re: QueueMetrics.AppsKilled/Failed metrics and failure reasons
Using hadoop 2.4.0. #of Applications running on average is small ~ 40 -60. The metrics in Ganglia shows around around 10-30 apps killed every 5 mins which is very high wrt to the apps running at any given time(40-60). The RM logs though show 0 failed apps in audit logs during that hour. The RM UI also doesnt show any apps in Applications-Failed tab . The logs are getting rolled over at a slower rate ..every 1-2 hours. Am searching for Application Finished - Failed to find the apps failed. Please let me know if I am missing something here. Thanks Suma On Wed, Feb 4, 2015 at 10:03 AM, Rohith Sharma K S rohithsharm...@huawei.com wrote: Hi Could you give more information, which version of hadoop are you using? QueueMetrics.AppsKilled/Failed metrics shows much higher nos i.e ~100. However RMAuditLogger shows 1 or 2 Apps as Killed/Failed in the logs. May be I suspect that Logs might be rolled out. Does more applications are running? All the applications history will be displayed on RM web UI (provided RM is not restarted or RM recovery enabled). May be you can check these applications lists. For finding reasons for application killed/failed, one way is you can check in NodeManager logs also. Here you need to check using container_id for corresponding application. Thanks Regards Rohith Sharma K S *From:* Suma Shivaprasad [mailto:sumasai.shivapra...@gmail.com] *Sent:* 03 February 2015 21:35 *To:* user@hadoop.apache.org; yarn-...@hadoop.apache.org *Subject:* QueueMetrics.AppsKilled/Failed metrics and failure reasons Hello, Was trying to debug reasons for Killed/Failed apps and was checking for the applications that were killed/failed in RM logs - from RMAuditLogger. QueueMetrics.AppsKilled/Failed metrics shows much higher nos i.e ~100. However RMAuditLogger shows 1 or 2 Apps as Killed/Failed in the logs. Is it possible that some logs are missed by AuditLogger or is it the other way round and metrics are being reported higher ? Thanks Suma
Can I configure multiple M/Rs and normal processes to one workflow?
Hello all. We're periodically scan HBase tables to aggregate statistic information, and store it to MySQL. We have 3 kinds of CP (kind of data source), each has one Channel and one Article table. (Channel : Article is 1:N relation.) All CPs table schema are different a bit, so in order to aggregate we should apply different logics, with joining Channel and Article. I've thought about workflow like this, but I wonder it can make sense. 1. run single process which initializes MySQL by creating table, deleting row, etc. 2. run 3 M/Rs simultaneously to aggregate statistic information for each CP, and insert rows per Channel to MySQL. 3. run single process which finalizes whole aggregation - runs aggregation query from MySQL to insert new row to MySQL, rolling table, etc. Definitely 1,2,3 should be run in a row. Any helps are really appreciated! Thanks. Regards. Jungtaek Lim (HeartSaVioR)
RE: QueueMetrics.AppsKilled/Failed metrics and failure reasons
Hi Could you give more information, which version of hadoop are you using? QueueMetrics.AppsKilled/Failed metrics shows much higher nos i.e ~100. However RMAuditLogger shows 1 or 2 Apps as Killed/Failed in the logs. May be I suspect that Logs might be rolled out. Does more applications are running? All the applications history will be displayed on RM web UI (provided RM is not restarted or RM recovery enabled). May be you can check these applications lists. For finding reasons for application killed/failed, one way is you can check in NodeManager logs also. Here you need to check using container_id for corresponding application. Thanks Regards Rohith Sharma K S From: Suma Shivaprasad [mailto:sumasai.shivapra...@gmail.com] Sent: 03 February 2015 21:35 To: user@hadoop.apache.org; yarn-...@hadoop.apache.org Subject: QueueMetrics.AppsKilled/Failed metrics and failure reasons Hello, Was trying to debug reasons for Killed/Failed apps and was checking for the applications that were killed/failed in RM logs - from RMAuditLogger. QueueMetrics.AppsKilled/Failed metrics shows much higher nos i.e ~100. However RMAuditLogger shows 1 or 2 Apps as Killed/Failed in the logs. Is it possible that some logs are missed by AuditLogger or is it the other way round and metrics are being reported higher ? Thanks Suma
RE: QueueMetrics.AppsKilled/Failed metrics and failure reasons
There are several ways to confirm from YARN that total number of Killed/Failed applications in cluster 1. Get from RM web UI lists OR 2. From admin try using this to get numbers of failed and killed applications: ./yarn application -list -appStates FAILED,KILLED 3. Using client API's Since metrics values are displayed in ganglia is incorrect, I get doubt that 1. does ganglia is pointing out to correct RM cluster? Or 2. what is the method ganglia uses to retrieve QueueMetrics? 3. Any client program calculates you have written retrieve apps and calculate it? Thanks Regards Rohith Sharma K S -Original Message- From: Suma Shivaprasad [mailto:sumasai.shivapra...@gmail.com] Sent: 04 February 2015 11:03 To: user@hadoop.apache.org Cc: yarn-...@hadoop.apache.org Subject: Re: QueueMetrics.AppsKilled/Failed metrics and failure reasons Using hadoop 2.4.0. #of Applications running on average is small ~ 40 -60. The metrics in Ganglia shows around around 10-30 apps killed every 5 mins which is very high wrt to the apps running at any given time(40-60). The RM logs though show 0 failed apps in audit logs during that hour. The RM UI also doesnt show any apps in Applications-Failed tab . The logs are getting rolled over at a slower rate ..every 1-2 hours. Am searching for Application Finished - Failed to find the apps failed. Please let me know if I am missing something here. Thanks Suma On Wed, Feb 4, 2015 at 10:03 AM, Rohith Sharma K S rohithsharm...@huawei.com wrote: Hi Could you give more information, which version of hadoop are you using? QueueMetrics.AppsKilled/Failed metrics shows much higher nos i.e ~100. However RMAuditLogger shows 1 or 2 Apps as Killed/Failed in the logs. May be I suspect that Logs might be rolled out. Does more applications are running? All the applications history will be displayed on RM web UI (provided RM is not restarted or RM recovery enabled). May be you can check these applications lists. For finding reasons for application killed/failed, one way is you can check in NodeManager logs also. Here you need to check using container_id for corresponding application. Thanks Regards Rohith Sharma K S *From:* Suma Shivaprasad [mailto:sumasai.shivapra...@gmail.com] *Sent:* 03 February 2015 21:35 *To:* user@hadoop.apache.org; yarn-...@hadoop.apache.org *Subject:* QueueMetrics.AppsKilled/Failed metrics and failure reasons Hello, Was trying to debug reasons for Killed/Failed apps and was checking for the applications that were killed/failed in RM logs - from RMAuditLogger. QueueMetrics.AppsKilled/Failed metrics shows much higher nos i.e ~100. However RMAuditLogger shows 1 or 2 Apps as Killed/Failed in the logs. Is it possible that some logs are missed by AuditLogger or is it the other way round and metrics are being reported higher ? Thanks Suma
Pass lib jars when invoking an hadoop program
Hi,
I am trying to run |distcp| using a java class, but I get the error of
class not found |DistCpOptions|. I have used the argument |-libjars
./share/hadoop/tools/lib/hadoop-distcp-2.6.0.jar| to pass the jar file,
but it seems that is not right. How I pass the lib properly?
Output:
|vagrant@hadoop-coc-1:~/Programs/hadoop$ hadoop jar wordcount.jar -libjars
./share/hadoop/tools/lib/hadoop-distcp-2.6.0.jar /input1 /outputmp /output1
Configuration: core-default.xml, core-site.xml, mapred-default.xml,
mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml,
hdfs-site.xml
-libjars
./share/hadoop/tools/lib/hadoop-distcp-2.6.0.jar
/input1
/outputmp
/output1
Exception in thread main java.lang.NoClassDefFoundError:
org/apache/hadoop/tools/DistCpOptions
at org.apache.hadoop.mapred.examples.WordCount.main(WordCount.java:101)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.tools.DistCpOptions
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
|
My class:
|public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
String[] otherArgs = new GenericOptionsParser(conf,
args).getRemainingArgs();
if (otherArgs.length 2) {
System.err.println(Usage: wordcount in [in...] out);
System.exit(2);
}
System.out.println(conf.toString());
for (int i = 0; i args.length; i++) {
System.out.println(args[i]);
}
// distcp
String proto = webhdfs://;
String src = hadoop-coc-1/input1;
String dest = hadoop-coc-2/input1;
ListPath lsrc = new ArrayListPath();
lsrc.add(new Path(src));
DistCpOptions options = new DistCpOptions(lsrc, new Path(dest));
DistCp distcp = new DistCp(new Configuration(), options);
distcp.execute();
}
|
Re: tools.DistCp: Invalid arguments
I have found the problem. I started to use `webhdfs` and everything is ok. On 03-02-2015 10:40, xeonmailinglist wrote: What do you mean by no path is given? Even if I launch this command, I get the same error…. What path should I put here? |$ hadoop distcp hdfs://hadoop-coc-1:50070/input1 hdfs://hadoop-coc-2:50070/input1| Thanks, On 02-02-2015 19:59, Alexander Alten-Lorenz wrote: Have a closer look: |hdfs://hadoop-coc-2:50070/| No Path is given. On 02 Feb 2015, at 20:52, xeonmailinglist xeonmailingl...@gmail.com mailto:xeonmailingl...@gmail.com wrote: Hi, I am trying to copy data using |distcp| but I get this error. Both hadoop runtime are working properly. Why is this happening? | vagrant@hadoop-coc-1:~/Programs/hadoop$ hadoop distcphdfs://hadoop-coc-1:50070/input1 hdfs://hadoop-coc-2:50070/ 15/02/02 19:46:37 ERROR tools.DistCp: Invalid arguments: java.io.IOException: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: hadoop-coc-1/127.0.1.1; destination host is: hadoop-coc-2:50070; at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772) at org.apache.hadoop.ipc.Client.call(Client.java:1472) at org.apache.hadoop.ipc.Client.call(Client.java:1399) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1988) at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1118) at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1114) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1114) at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400) at org.apache.hadoop.tools.DistCp.setTargetPathExists(DistCp.java:188) at org.apache.hadoop.tools.DistCp.run(DistCp.java:111) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.tools.DistCp.main(DistCp.java:401) Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag. at com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:94) at com.google.protobuf.CodedInputStream.checkLastTagWas(CodedInputStream.java:124) at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:202) at com.google.protobuf.AbstractParser.parsePartialDelimitedFrom(AbstractParser.java:241) at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:253) at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:259) at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:49) at org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcHeaderProtos.java:3167) at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1072) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:966) Invalid arguments: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: hadoop-coc-1/127.0.1.1; destination host is: hadoop-coc-2:50070; usage: distcp OPTIONS [source_path...] target_path | Thanks,
Re: tools.DistCp: Invalid arguments
Hi, Can you please try webhdfs instead hdfs? - Alexander On 03 Feb 2015, at 12:05, xeonmailinglist xeonmailingl...@gmail.com wrote: Maybe this has to do with this error… I can’t do ls to my own machine using the command below. Can this be related to the other problem? Shouldn't I list the files with this command? vagrant@hadoop-coc-1:~$ hdfs dfs -ls hdfs://192.168.56.100/ ls: Call From hadoop-coc-1/192.168.56.100 to hadoop-coc-1:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused http://wiki.apache.org/hadoop/ConnectionRefused On 02-02-2015 19:59, Alexander Alten-Lorenz wrote: Have a closer look: hdfs://hadoop-coc-2:50070/ hdfs://hadoop-coc-2:50070/ No Path is given. On 02 Feb 2015, at 20:52, xeonmailinglist xeonmailingl...@gmail.com mailto:xeonmailingl...@gmail.com wrote: Hi, I am trying to copy data using distcp but I get this error. Both hadoop runtime are working properly. Why is this happening? vagrant@hadoop-coc-1:~/Programs/hadoop$ hadoop distcp hdfs://hadoop-coc-1:50070/input1 hdfs://hadoop-coc-1:50070/input1 hdfs://hadoop-coc-2:50070/ hdfs://hadoop-coc-2:50070/ 15/02/02 19:46:37 ERROR tools.DistCp: Invalid arguments: java.io.IOException: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: hadoop-coc-1/127.0.1.1; destination host is: hadoop-coc-2:50070; at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772) at org.apache.hadoop.ipc.Client.call(Client.java:1472) at org.apache.hadoop.ipc.Client.call(Client.java:1399) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1988) at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1118) at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1114) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1114) at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400) at org.apache.hadoop.tools.DistCp.setTargetPathExists(DistCp.java:188) at org.apache.hadoop.tools.DistCp.run(DistCp.java:111) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.tools.DistCp.main(DistCp.java:401) Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag. at com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:94) at com.google.protobuf.CodedInputStream.checkLastTagWas(CodedInputStream.java:124) at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:202) at com.google.protobuf.AbstractParser.parsePartialDelimitedFrom(AbstractParser.java:241) at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:253) at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:259) at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:49) at org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcHeaderProtos.java:3167) at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1072) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:966) Invalid arguments: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: hadoop-coc-1/127.0.1.1; destination host is: hadoop-coc-2:50070; usage: distcp OPTIONS [source_path...] target_path Thanks,
Re: tools.DistCp: Invalid arguments
Ah, good. Cross-posting :) BR, Alex On 03 Feb 2015, at 12:41, xeonmailinglist xeonmailingl...@gmail.com wrote: I have found the problem. I started to use `webhdfs` and everything is ok. On 03-02-2015 10:40, xeonmailinglist wrote: What do you mean by no path is given? Even if I launch this command, I get the same error…. What path should I put here? $ hadoop distcp hdfs://hadoop-coc-1:50070/input1 hdfs://hadoop-coc-2:50070/input1 Thanks, On 02-02-2015 19:59, Alexander Alten-Lorenz wrote: Have a closer look: hdfs://hadoop-coc-2:50070/ hdfs://hadoop-coc-2:50070/ No Path is given. On 02 Feb 2015, at 20:52, xeonmailinglist xeonmailingl...@gmail.com mailto:xeonmailingl...@gmail.com wrote: Hi, I am trying to copy data using distcp but I get this error. Both hadoop runtime are working properly. Why is this happening? vagrant@hadoop-coc-1:~/Programs/hadoop$ hadoop distcp hdfs://hadoop-coc-1:50070/input1 hdfs://hadoop-coc-1:50070/input1 hdfs://hadoop-coc-2:50070/ hdfs://hadoop-coc-2:50070/ 15/02/02 19:46:37 ERROR tools.DistCp: Invalid arguments: java.io.IOException: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: hadoop-coc-1/127.0.1.1; destination host is: hadoop-coc-2:50070; at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772) at org.apache.hadoop.ipc.Client.call(Client.java:1472) at org.apache.hadoop.ipc.Client.call(Client.java:1399) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1988) at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1118) at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1114) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1114) at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400) at org.apache.hadoop.tools.DistCp.setTargetPathExists(DistCp.java:188) at org.apache.hadoop.tools.DistCp.run(DistCp.java:111) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.tools.DistCp.main(DistCp.java:401) Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag. at com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:94) at com.google.protobuf.CodedInputStream.checkLastTagWas(CodedInputStream.java:124) at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:202) at com.google.protobuf.AbstractParser.parsePartialDelimitedFrom(AbstractParser.java:241) at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:253) at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:259) at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:49) at org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcHeaderProtos.java:3167) at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1072) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:966) Invalid arguments: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: hadoop-coc-1/127.0.1.1; destination host is: hadoop-coc-2:50070; usage: distcp OPTIONS [source_path...] target_path Thanks,
Re: tools.DistCp: Invalid arguments
Another good option is hftp. Artem Ervits On Feb 3, 2015 6:42 AM, xeonmailinglist xeonmailingl...@gmail.com wrote: I have found the problem. I started to use `webhdfs` and everything is ok. On 03-02-2015 10:40, xeonmailinglist wrote: What do you mean by no path is given? Even if I launch this command, I get the same error…. What path should I put here? $ hadoop distcp hdfs://hadoop-coc-1:50070/input1 hdfs://hadoop-coc-2:50070/input1 Thanks, On 02-02-2015 19:59, Alexander Alten-Lorenz wrote: Have a closer look: hdfs://hadoop-coc-2:50070/ No Path is given. On 02 Feb 2015, at 20:52, xeonmailinglist xeonmailingl...@gmail.com wrote: Hi, I am trying to copy data using distcp but I get this error. Both hadoop runtime are working properly. Why is this happening? vagrant@hadoop-coc-1:~/Programs/hadoop$ hadoop distcp hdfs://hadoop-coc-1:50070/input1 hdfs://hadoop-coc-2:50070/ 15/02/02 19:46:37 ERROR tools.DistCp: Invalid arguments: java.io.IOException: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: hadoop-coc-1/127.0.1.1; destination host is: hadoop-coc-2:50070; at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772) at org.apache.hadoop.ipc.Client.call(Client.java:1472) at org.apache.hadoop.ipc.Client.call(Client.java:1399) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1988) at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1118) at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1114) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1114) at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400) at org.apache.hadoop.tools.DistCp.setTargetPathExists(DistCp.java:188) at org.apache.hadoop.tools.DistCp.run(DistCp.java:111) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.tools.DistCp.main(DistCp.java:401) Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag. at com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:94) at com.google.protobuf.CodedInputStream.checkLastTagWas(CodedInputStream.java:124) at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:202) at com.google.protobuf.AbstractParser.parsePartialDelimitedFrom(AbstractParser.java:241) at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:253) at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:259) at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:49) at org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcHeaderProtos.java:3167) at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1072) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:966) Invalid arguments: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: hadoop-coc-1/127.0.1.1; destination host is: hadoop-coc-2:50070; usage: distcp OPTIONS [source_path...] target_path Thanks,
Can not start HA namenode with security enabled
I am converting a secure non-HA cluster into a secure HA cluster. After the configuration and started all the journalnodes, I executed the following commands on the original NameNode: 1. hdfs name -initializeSharedEdits #this step succeeded 2. hadoop-daemon.sh start namenode # this step failed. The namenode did not start successfully. I verified that my principals are right. And I checked the DNS is configured correctly so that I could use the nslookup command to lookup and reverse-lookup the Namenode and JournalNodes. I also checked the logs. The JournalNodes did not report any ERROR. The Namenode Log report some ERRORs, but I still could not find the reason according to these ERRORS. In the following I listed the main part of my hdfs-site.xml and the error log from my Namenode. Could anyone help me to figure it out? Many Thanks! **The main part of my hdfs-site.xml* property namedfs.nameservices/name valuebgdt-dev-hrb/value /property property namedfs.ha.namenodes.bgdt-dev-hrb/name valuenn1,nn2/value /property property namedfs.namenode.rpc-address.bgdt-dev-hrb.nn1/name valuebgdt01.dev.hrb:9000/value /property property namedfs.namenode.rpc-address.bgdt-dev-hrb.nn2/name valuebgdt02.dev.hrb:9000/value /property property namedfs.namenode.shared.edits.dir/name valueqjournal://bgdt01.dev.hrb:8485;bgdt03.dev.hrb:8485;bgdt04.dev.hrb:8485/bgdt-dev-hrb/value /property property namedfs.client.failover.proxy.provider.bgdt-dev-hrb/name valueorg.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider/value /property property namedfs.ha.fencing.methods/name valuesshfence shell(/bin/true) /value /property property namedfs.ha.fencing.ssh.private-key-files/name value/home/hadoop/.ssh/id_rsa/value /property property namedfs.journalnode.edits.dir/name value/bgdt/hadoop/hdfs/jn/value /property property namedfs.permissions.enabled/name valuetrue/value /property property namedfs.namenode.name.dir/name valuefile:///bgdt/hadoop/hdfs/nn/value finaltrue/final /property property namedfs.datanode.name.dir/name valuefile:///bgdt/hadoop/hdfs/dn/value /property property namedfs.namenode.http-address.bgdt-dev-hrb.nn1/name valuebgdt01.dev.hrb:50070/value /property property namedfs.namenode.http-address.bgdt-dev-hrb.nn2/name valuebgdt02.dev.hrb:50070/value /property property namedfs.permissions.superusergroup/name valuebgdtgrp/value /property property namedfs.block.access.token.enable/name valuetrue/value /property property namedfs.http.policy/name valueHTTP_ONLY/value /property property namedfs.namenode.https-address.bgdt-dev-hrb.nn1/name valuebgdt01.dev.hrb:50470/value /property property namedfs.namenode.https-address.bgdt-dev-hrb.nn2/name valuebgdt02.dev.hrb:50470/value /property property namedfs.namenode.keytab.file/name value/etc/hadoop/keytab/hadoop.service.keytab/value /property property namedfs.namenode.kerberos.principal/name valuehdfs/_h...@bgdt.dev.hrb/value /property property namedfs.namenode.kerberos.https.principal/name valuehost/_h...@bgdt.dev.hrb/value /property property namedfs.webhdfs.enabled/name valuetrue/value /property property namedfs.web.authentication.kerberos.principal/name valuehttp/_h...@bgdt.dev.hrb/value /property property namedfs.web.authentication.kerberos.keytab/name value/etc/hadoop/keytab/hadoop.service.keytab/value /property property namedfs.journalnode.kerberos.principal/name valuehdfs/_h...@bgdt.dev.hrb/value /property property namedfs.journalnode.kerberos.https.principal/name valuehost/_h...@bgdt.dev.hrb/value /property property namedfs.journalnode.kerberos.internal.spnego.principal/name valuehttp/_h...@bgdt.dev.hrb/value /property property namedfs.journalnode.keytab.file/name value/etc/hadoop/keytab/hadoop.service.keytab/value /property *The Error Log from the Namenode** 2015-02-03 17:42:06,020 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Start loading edits file http://bgdt04.dev.hrb:8480/getJournal?jid=bgdt-dev-hrbsegmentTxId=68994storageInfo=-57%3A876630880%3A0%3ACID-ea4c77aa-882d-4adf-a347-42f1344421f3, http://bgdt01.dev.hrb:8480/getJournal?jid=bgdt-dev-hrbsegmentTxId=68994storageInfo=-57%3A876630880%3A0%3ACID-ea4c77aa-882d-4adf-a347-42f1344421f3 2015-02-03 17:42:06,024 INFO org.apache.hadoop.hdfs.server.namenode.EditLogInputStream: Fast-forwarding stream 'http://bgdt04.dev.hrb:8480/getJournal?jid=bgdt-dev-hrbsegmentTxId=68994storageInfo=-57%3A876630880%3A0%3ACID-ea4c77aa-882d-4adf-a347-42f1344421f3, http://bgdt01.dev.hrb:8480/getJournal?jid=bgdt-dev-hrbsegmentTxId=68994storageInfo=-57%3A876630880%3A0%3ACID-ea4c77aa-882d-4adf-a347-42f1344421f3' to transaction ID 68994 2015-02-03 17:42:06,024 INFO org.apache.hadoop.hdfs.server.namenode.EditLogInputStream: Fast-forwarding stream 'http://bgdt04.dev.hrb:8480/getJournal?jid=bgdt-dev-hrbsegmentTxId=68994storageInfo=-57%3A876630880%3A0%3ACID-ea4c77aa-882d-4adf-a347-42f1344421f3' to transaction
QueueMetrics.AppsKilled/Failed metrics and failure reasons
Hello, Was trying to debug reasons for Killed/Failed apps and was checking for the applications that were killed/failed in RM logs - from RMAuditLogger. QueueMetrics.AppsKilled/Failed metrics shows much higher nos i.e ~100. However RMAuditLogger shows 1 or 2 Apps as Killed/Failed in the logs. Is it possible that some logs are missed by AuditLogger or is it the other way round and metrics are being reported higher ? Thanks Suma
IBM JAVA and KerberosTestUtils
Hi All, This is with respect to the JIRA defect: HADOOP-10774 related to kerberose authentication using IBM JAVA. Looks like there are lot of changes haven been done to properly handle the kerbserose authentication using the JIRA defect: HADOOP-9446 for IBM JAVA. But their are still some failures can been seen in Apache Hadoop Common tests in case of IBM JAVA. Available patch for HADOOP-10774 will solve the authentication issues plus the path issues. Please let me know how to proceed on this issue or let me know the right contact or mailing list. Thanks Regards, -Sangamesh AIX BOS Development Bangalore, EGL D Block, 6th floor Phone: +91 (80) 417-76965
Re: Can not start HA namenode with security enabled
Have you added all host specific principals in kerberos database ? Thanks, On Tue, Feb 3, 2015 at 7:59 AM, 郝东 donhof...@163.com wrote: I am converting a secure non-HA cluster into a secure HA cluster. After the configuration and started all the journalnodes, I executed the following commands on the original NameNode: 1. hdfs name -initializeSharedEdits #this step succeeded 2. hadoop-daemon.sh start namenode # this step failed. The namenode did not start successfully. I verified that my principals are right. And I checked the DNS is configured correctly so that I could use the nslookup command to lookup and reverse-lookup the Namenode and JournalNodes. I also checked the logs. The JournalNodes did not report any ERROR. The Namenode Log report some ERRORs, but I still could not find the reason according to these ERRORS. In the following I listed the main part of my hdfs-site.xml and the error log from my Namenode. Could anyone help me to figure it out? Many Thanks! **The main part of my hdfs-site.xml* property namedfs.nameservices/name valuebgdt-dev-hrb/value /property property namedfs.ha.namenodes.bgdt-dev-hrb/name valuenn1,nn2/value /property property namedfs.namenode.rpc-address.bgdt-dev-hrb.nn1/name valuebgdt01.dev.hrb:9000/value /property property namedfs.namenode.rpc-address.bgdt-dev-hrb.nn2/name valuebgdt02.dev.hrb:9000/value /property property namedfs.namenode.shared.edits.dir/name valueqjournal://bgdt01.dev.hrb:8485;bgdt03.dev.hrb:8485;bgdt04.dev.hrb:8485/bgdt-dev-hrb/value /property property namedfs.client.failover.proxy.provider.bgdt-dev-hrb/name valueorg.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider/value /property property namedfs.ha.fencing.methods/name valuesshfence shell(/bin/true) /value /property property namedfs.ha.fencing.ssh.private-key-files/name value/home/hadoop/.ssh/id_rsa/value /property property namedfs.journalnode.edits.dir/name value/bgdt/hadoop/hdfs/jn/value /property property namedfs.permissions.enabled/name valuetrue/value /property property namedfs.namenode.name.dir/name valuefile:///bgdt/hadoop/hdfs/nn/value finaltrue/final /property property namedfs.datanode.name.dir/name valuefile:///bgdt/hadoop/hdfs/dn/value /property property namedfs.namenode.http-address.bgdt-dev-hrb.nn1/name valuebgdt01.dev.hrb:50070/value /property property namedfs.namenode.http-address.bgdt-dev-hrb.nn2/name valuebgdt02.dev.hrb:50070/value /property property namedfs.permissions.superusergroup/name valuebgdtgrp/value /property property namedfs.block.access.token.enable/name valuetrue/value /property property namedfs.http.policy/name valueHTTP_ONLY/value /property property namedfs.namenode.https-address.bgdt-dev-hrb.nn1/name valuebgdt01.dev.hrb:50470/value /property property namedfs.namenode.https-address.bgdt-dev-hrb.nn2/name valuebgdt02.dev.hrb:50470/value /property property namedfs.namenode.keytab.file/name value/etc/hadoop/keytab/hadoop.service.keytab/value /property property namedfs.namenode.kerberos.principal/name valuehdfs/_h...@bgdt.dev.hrb/value /property property namedfs.namenode.kerberos.https.principal/name valuehost/_h...@bgdt.dev.hrb/value /property property namedfs.webhdfs.enabled/name valuetrue/value /property property namedfs.web.authentication.kerberos.principal/name valuehttp/_h...@bgdt.dev.hrb/value /property property namedfs.web.authentication.kerberos.keytab/name value/etc/hadoop/keytab/hadoop.service.keytab/value /property property namedfs.journalnode.kerberos.principal/name valuehdfs/_h...@bgdt.dev.hrb/value /property property namedfs.journalnode.kerberos.https.principal/name valuehost/_h...@bgdt.dev.hrb/value /property property namedfs.journalnode.kerberos.internal.spnego.principal/name valuehttp/_h...@bgdt.dev.hrb/value /property property namedfs.journalnode.keytab.file/name value/etc/hadoop/keytab/hadoop.service.keytab/value /property *The Error Log from the Namenode** 2015-02-03 17:42:06,020 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Start loading edits file http://bgdt04.dev.hrb:8480/getJournal?jid=bgdt-dev-hrbsegmentTxId=68994storageInfo=-57%3A876630880%3A0%3ACID-ea4c77aa-882d-4adf-a347-42f1344421f3, http://bgdt01.dev.hrb:8480/getJournal?jid=bgdt-dev-hrbsegmentTxId=68994storageInfo=-57%3A876630880%3A0%3ACID-ea4c77aa-882d-4adf-a347-42f1344421f3 2015-02-03 17:42:06,024 INFO org.apache.hadoop.hdfs.server.namenode.EditLogInputStream: Fast-forwarding stream ' http://bgdt04.dev.hrb:8480/getJournal?jid=bgdt-dev-hrbsegmentTxId=68994storageInfo=-57%3A876630880%3A0%3ACID-ea4c77aa-882d-4adf-a347-42f1344421f3, http://bgdt01.dev.hrb:8480/getJournal?jid=bgdt-dev-hrbsegmentTxId=68994storageInfo=-57%3A876630880%3A0%3ACID-ea4c77aa-882d-4adf-a347-42f1344421f3' to transaction ID 68994
Re: Pass lib jars when invoking an hadoop program
Got it. Here's the solution:
```
vagrant@hadoop-coc-1:~/Programs/hadoop$ export
HADOOP_CLASSPATH=share/hadoop/tools/lib/hadoop-distcp-2.6.0.jar; hadoop
jar wordcount.jar -libjars
$HADOOP_HOME/share/hadoop/tools/lib/hadoop-distcp-2.6.0.jar /input1
/outputmp /output1
```
On 03-02-2015 14:58, xeonmailinglist wrote:
Hi,
I am trying to run |distcp| using a java class, but I get the error of
class not found |DistCpOptions|. I have used the argument |-libjars
./share/hadoop/tools/lib/hadoop-distcp-2.6.0.jar| to pass the jar
file, but it seems that is not right. How I pass the lib properly?
Output:
|vagrant@hadoop-coc-1:~/Programs/hadoop$ hadoop jar wordcount.jar -libjars
./share/hadoop/tools/lib/hadoop-distcp-2.6.0.jar /input1 /outputmp /output1
Configuration: core-default.xml, core-site.xml, mapred-default.xml,
mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml,
hdfs-site.xml
-libjars
./share/hadoop/tools/lib/hadoop-distcp-2.6.0.jar
/input1
/outputmp
/output1
Exception in thread main java.lang.NoClassDefFoundError:
org/apache/hadoop/tools/DistCpOptions
at org.apache.hadoop.mapred.examples.WordCount.main(WordCount.java:101)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.tools.DistCpOptions
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
|
My class:
|public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
String[] otherArgs = new GenericOptionsParser(conf,
args).getRemainingArgs();
if (otherArgs.length 2) {
System.err.println(Usage: wordcount in [in...] out);
System.exit(2);
}
System.out.println(conf.toString());
for (int i = 0; i args.length; i++) {
System.out.println(args[i]);
}
// distcp
String proto = webhdfs://;
String src = hadoop-coc-1/input1;
String dest = hadoop-coc-2/input1;
ListPath lsrc = new ArrayListPath();
lsrc.add(new Path(src));
DistCpOptions options = new DistCpOptions(lsrc, new Path(dest));
DistCp distcp = new DistCp(new Configuration(), options);
distcp.execute();
}
|
Re: Pass lib jars when invoking an hadoop program
Got it. Here’s the solution:
|vagrant@hadoop-coc-1:~/Programs/hadoop$ export
HADOOP_CLASSPATH=share/hadoop/tools/lib/hadoop-distcp-2.6.0.jar; hadoop jar
wordcount.jar -libjars
$HADOOP_HOME/share/hadoop/tools/lib/hadoop-distcp-2.6.0.jar /input1 /outputmp
/output1
|
On 03-02-2015 14:58, xeonmailinglist wrote:
Hi,
I am trying to run |distcp| using a java class, but I get the error of
class not found |DistCpOptions|. I have used the argument |-libjars
./share/hadoop/tools/lib/hadoop-distcp-2.6.0.jar| to pass the jar
file, but it seems that is not right. How I pass the lib properly?
Output:
|vagrant@hadoop-coc-1:~/Programs/hadoop$ hadoop jar wordcount.jar -libjars
./share/hadoop/tools/lib/hadoop-distcp-2.6.0.jar /input1 /outputmp /output1
Configuration: core-default.xml, core-site.xml, mapred-default.xml,
mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml,
hdfs-site.xml
-libjars
./share/hadoop/tools/lib/hadoop-distcp-2.6.0.jar
/input1
/outputmp
/output1
Exception in thread main java.lang.NoClassDefFoundError:
org/apache/hadoop/tools/DistCpOptions
at org.apache.hadoop.mapred.examples.WordCount.main(WordCount.java:101)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.tools.DistCpOptions
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
|
My class:
|public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
String[] otherArgs = new GenericOptionsParser(conf,
args).getRemainingArgs();
if (otherArgs.length 2) {
System.err.println(Usage: wordcount in [in...] out);
System.exit(2);
}
System.out.println(conf.toString());
for (int i = 0; i args.length; i++) {
System.out.println(args[i]);
}
// distcp
String proto = webhdfs://;
String src = hadoop-coc-1/input1;
String dest = hadoop-coc-2/input1;
ListPath lsrc = new ArrayListPath();
lsrc.add(new Path(src));
DistCpOptions options = new DistCpOptions(lsrc, new Path(dest));
DistCp distcp = new DistCp(new Configuration(), options);
distcp.execute();
}
|
create Job with java code.
Hi, I want this because I want to create depency between 2 jobs. The first job execute the wordcount example, and the second job copy the output of the wordcount to another HDFS. Therefore, I want to create a job (job 2) that includes the code to copy data to another HDFS. The code is below. Is there a way to do it with the API? | // Put inside a job() String proto = webhdfs://; String src = hadoop-coc-1/input1; String dest = hadoop-coc-2/input1; ListPath lsrc = new ArrayListPath(); lsrc.add(new Path(proto+src)); DistCpOptions options = new DistCpOptions(lsrc, new Path(proto+dest)); DistCp distcp = new DistCp(conf, options); distcp.execute(); |
Re: tools.DistCp: Invalid arguments
Maybe this has to do with this error… I can’t do |ls| to my own machine using the command below. Can this be related to the other problem? Shouldn't I list the files with this command? |vagrant@hadoop-coc-1:~$ hdfs dfs -ls hdfs://192.168.56.100/ ls: Call From hadoop-coc-1/192.168.56.100 to hadoop-coc-1:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused | On 02-02-2015 19:59, Alexander Alten-Lorenz wrote: Have a closer look: |hdfs://hadoop-coc-2:50070/| No Path is given. On 02 Feb 2015, at 20:52, xeonmailinglist xeonmailingl...@gmail.com mailto:xeonmailingl...@gmail.com wrote: Hi, I am trying to copy data using |distcp| but I get this error. Both hadoop runtime are working properly. Why is this happening? | vagrant@hadoop-coc-1:~/Programs/hadoop$ hadoop distcphdfs://hadoop-coc-1:50070/input1 hdfs://hadoop-coc-2:50070/ 15/02/02 19:46:37 ERROR tools.DistCp: Invalid arguments: java.io.IOException: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: hadoop-coc-1/127.0.1.1; destination host is: hadoop-coc-2:50070; at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772) at org.apache.hadoop.ipc.Client.call(Client.java:1472) at org.apache.hadoop.ipc.Client.call(Client.java:1399) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1988) at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1118) at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1114) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1114) at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400) at org.apache.hadoop.tools.DistCp.setTargetPathExists(DistCp.java:188) at org.apache.hadoop.tools.DistCp.run(DistCp.java:111) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.tools.DistCp.main(DistCp.java:401) Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag. at com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:94) at com.google.protobuf.CodedInputStream.checkLastTagWas(CodedInputStream.java:124) at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:202) at com.google.protobuf.AbstractParser.parsePartialDelimitedFrom(AbstractParser.java:241) at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:253) at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:259) at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:49) at org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcHeaderProtos.java:3167) at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1072) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:966) Invalid arguments: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: hadoop-coc-1/127.0.1.1; destination host is: hadoop-coc-2:50070; usage: distcp OPTIONS [source_path...] target_path | Thanks,
Re: tools.DistCp: Invalid arguments
What do you mean by no path is given? Even if I launch this command, I get the same error…. What path should I put here? |$ hadoop distcp hdfs://hadoop-coc-1:50070/input1 hdfs://hadoop-coc-2:50070/input1| Thanks, On 02-02-2015 19:59, Alexander Alten-Lorenz wrote: Have a closer look: |hdfs://hadoop-coc-2:50070/| No Path is given. On 02 Feb 2015, at 20:52, xeonmailinglist xeonmailingl...@gmail.com mailto:xeonmailingl...@gmail.com wrote: Hi, I am trying to copy data using |distcp| but I get this error. Both hadoop runtime are working properly. Why is this happening? | vagrant@hadoop-coc-1:~/Programs/hadoop$ hadoop distcphdfs://hadoop-coc-1:50070/input1 hdfs://hadoop-coc-2:50070/ 15/02/02 19:46:37 ERROR tools.DistCp: Invalid arguments: java.io.IOException: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: hadoop-coc-1/127.0.1.1; destination host is: hadoop-coc-2:50070; at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772) at org.apache.hadoop.ipc.Client.call(Client.java:1472) at org.apache.hadoop.ipc.Client.call(Client.java:1399) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1988) at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1118) at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1114) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1114) at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400) at org.apache.hadoop.tools.DistCp.setTargetPathExists(DistCp.java:188) at org.apache.hadoop.tools.DistCp.run(DistCp.java:111) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.tools.DistCp.main(DistCp.java:401) Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag. at com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:94) at com.google.protobuf.CodedInputStream.checkLastTagWas(CodedInputStream.java:124) at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:202) at com.google.protobuf.AbstractParser.parsePartialDelimitedFrom(AbstractParser.java:241) at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:253) at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:259) at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:49) at org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcHeaderProtos.java:3167) at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1072) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:966) Invalid arguments: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: hadoop-coc-1/127.0.1.1; destination host is: hadoop-coc-2:50070; usage: distcp OPTIONS [source_path...] target_path | Thanks,
Re: unsubscribe
Check http://hadoop.apache.org/mailing_lists.html#User Regards, Ramkumar Bashyam On Wed, Jan 7, 2015 at 7:01 PM, Kiran Prasad Gorigay kiranprasa...@imimobile.com wrote: unsubscribe
How to rolling upgrade??
my cluster A, and cluster B. To upgrade to version 2.6 In what order should I upgrade? Journalnode 1 gt;gt; Journalnode 2 gt; Journalnode 3 gt;gt; Namenode Std gt;gt; Namenode Act gt;gt; Datanode ?? Do I also need to upgrade the zookeeper? hadoop-2.4.1 : JournalNode, Namenode, Datanode zookeeper-3.4.6 : zookeeper http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
unsubscribe
unsubscribe
Re:Re: Can not start HA namenode with security enabled
Hi, I have checked my kerberos database. All the principals are there. By the way, if I did not enable HA, just enable the secure-mode, the Namenode can be started correctly. At 2015-02-04 01:24:21, Manoj Samel manojsamelt...@gmail.com wrote: Have you added all host specific principals in kerberos database ? Thanks, On Tue, Feb 3, 2015 at 7:59 AM, 郝东 donhof...@163.com wrote: I am converting a secure non-HA cluster into a secure HA cluster. After the configuration and started all the journalnodes, I executed the following commands on the original NameNode: 1. hdfs name -initializeSharedEdits #this step succeeded 2. hadoop-daemon.sh start namenode # this step failed. The namenode did not start successfully. I verified that my principals are right. And I checked the DNS is configured correctly so that I could use the nslookup command to lookup and reverse-lookup the Namenode and JournalNodes. I also checked the logs. The JournalNodes did not report any ERROR. The Namenode Log report some ERRORs, but I still could not find the reason according to these ERRORS. In the following I listed the main part of my hdfs-site.xml and the error log from my Namenode. Could anyone help me to figure it out? Many Thanks! **The main part of my hdfs-site.xml* property namedfs.nameservices/name valuebgdt-dev-hrb/value /property property namedfs.ha.namenodes.bgdt-dev-hrb/name valuenn1,nn2/value /property property namedfs.namenode.rpc-address.bgdt-dev-hrb.nn1/name valuebgdt01.dev.hrb:9000/value /property property namedfs.namenode.rpc-address.bgdt-dev-hrb.nn2/name valuebgdt02.dev.hrb:9000/value /property property namedfs.namenode.shared.edits.dir/name valueqjournal://bgdt01.dev.hrb:8485;bgdt03.dev.hrb:8485;bgdt04.dev.hrb:8485/bgdt-dev-hrb/value /property property namedfs.client.failover.proxy.provider.bgdt-dev-hrb/name valueorg.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider/value /property property namedfs.ha.fencing.methods/name valuesshfence shell(/bin/true) /value /property property namedfs.ha.fencing.ssh.private-key-files/name value/home/hadoop/.ssh/id_rsa/value /property property namedfs.journalnode.edits.dir/name value/bgdt/hadoop/hdfs/jn/value /property property namedfs.permissions.enabled/name valuetrue/value /property property namedfs.namenode.name.dir/name valuefile:///bgdt/hadoop/hdfs/nn/value finaltrue/final /property property namedfs.datanode.name.dir/name valuefile:///bgdt/hadoop/hdfs/dn/value /property property namedfs.namenode.http-address.bgdt-dev-hrb.nn1/name valuebgdt01.dev.hrb:50070/value /property property namedfs.namenode.http-address.bgdt-dev-hrb.nn2/name valuebgdt02.dev.hrb:50070/value /property property namedfs.permissions.superusergroup/name valuebgdtgrp/value /property property namedfs.block.access.token.enable/name valuetrue/value /property property namedfs.http.policy/name valueHTTP_ONLY/value /property property namedfs.namenode.https-address.bgdt-dev-hrb.nn1/name valuebgdt01.dev.hrb:50470/value /property property namedfs.namenode.https-address.bgdt-dev-hrb.nn2/name valuebgdt02.dev.hrb:50470/value /property property namedfs.namenode.keytab.file/name value/etc/hadoop/keytab/hadoop.service.keytab/value /property property namedfs.namenode.kerberos.principal/name valuehdfs/_h...@bgdt.dev.hrb/value /property property namedfs.namenode.kerberos.https.principal/name valuehost/_h...@bgdt.dev.hrb/value /property property namedfs.webhdfs.enabled/name valuetrue/value /property property namedfs.web.authentication.kerberos.principal/name valuehttp/_h...@bgdt.dev.hrb/value /property property namedfs.web.authentication.kerberos.keytab/name value/etc/hadoop/keytab/hadoop.service.keytab/value /property property namedfs.journalnode.kerberos.principal/name valuehdfs/_h...@bgdt.dev.hrb/value /property property namedfs.journalnode.kerberos.https.principal/name valuehost/_h...@bgdt.dev.hrb/value /property property namedfs.journalnode.kerberos.internal.spnego.principal/name valuehttp/_h...@bgdt.dev.hrb/value /property property namedfs.journalnode.keytab.file/name value/etc/hadoop/keytab/hadoop.service.keytab/value /property *The Error Log from the Namenode** 2015-02-03 17:42:06,020 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Start loading edits file http://bgdt04.dev.hrb:8480/getJournal?jid=bgdt-dev-hrbsegmentTxId=68994storageInfo=-57%3A876630880%3A0%3ACID-ea4c77aa-882d-4adf-a347-42f1344421f3, http://bgdt01.dev.hrb:8480/getJournal?jid=bgdt-dev-hrbsegmentTxId=68994storageInfo=-57%3A876630880%3A0%3ACID-ea4c77aa-882d-4adf-a347-42f1344421f3 2015-02-03 17:42:06,024 INFO org.apache.hadoop.hdfs.server.namenode.EditLogInputStream: Fast-forwarding stream 'http://bgdt04.dev.hrb:8480/getJournal?jid=bgdt-dev-hrbsegmentTxId=68994storageInfo=-57%3A876630880%3A0%3ACID-ea4c77aa-882d-4adf-a347-42f1344421f3,
