Re: Large number of application attempts
But I’ve noticed that there are ~4000 znodes in /hadoop-ha/testhbase1/rmstore/ZKRMStateRoot/RMAppRoot in Zookeeper. Can I manually remove this znodes, or do I need to do this through yarn command line? Thank you. On 07 Jul 2015, at 10:23, Akmal Abbasov akmal.abba...@icloud.com wrote: Hi all, I have HBase cluster. I run a cron job each hour to create a snapshot of tables in HBase and export these snapshots to another HBase cluster for backup. But I’ve noticed that there are ~4000 znodes in /hadoop-ha/testhbase1/rmstore/ZKRMStateRoot/RMAppRoot in Zookeeper. What are these znodes? Are they failed yarn applications? How I need to handle them? Thank you. Akmal
Re: Large number of application attempts
Hi, Are you running a lots of apps ? I have 5 tables in HBase, and I run a cron job each hour which will create a snapshot of each table, and tries to export it to a remote HBase cluster. Or could you, maybe, share some of file names of those znodes, please ? they are all have the name format as follows ‘application_x_’ Thank you. On 07 Jul 2015, at 19:00, Xuan Gong xg...@hortonworks.com wrote: Hey, Akmal: Are you running a lots of apps ? Or could you, maybe, share some of file names of those znodes, please ? Thanks Xuan Gong On 7/7/15, 1:23 AM, Akmal Abbasov akmal.abba...@icloud.com wrote: Hi all, I have HBase cluster. I run a cron job each hour to create a snapshot of tables in HBase and export these snapshots to another HBase cluster for backup. But I¹ve noticed that there are ~4000 znodes in /hadoop-ha/testhbase1/rmstore/ZKRMStateRoot/RMAppRoot in Zookeeper. What are these znodes? Are they failed yarn applications? How I need to handle them? Thank you. Akmal
Re: Large number of application attempts
bq. Can I manually remove this znodes, or do I need to do this through yarn command line? Which version of hadoop you are using ? If the version is 2.6 or after 2.6, you could do yarn resourcemanager -format-state-store which will format all the rmstore (basically delete path ha/testhbase1/rmstore/ZKRMStateRoot and all the subdir/files) Thanks Xuan Gong On 7/8/15, 5:45 AM, Akmal Abbasov akmal.abba...@icloud.com wrote: But I¹ve noticed that there are ~4000 znodes in /hadoop-ha/testhbase1/rmstore/ZKRMStateRoot/RMAppRoot in Zookeeper. Can I manually remove this znodes, or do I need to do this through yarn command line? Thank you. On 07 Jul 2015, at 10:23, Akmal Abbasov akmal.abba...@icloud.com wrote: Hi all, I have HBase cluster. I run a cron job each hour to create a snapshot of tables in HBase and export these snapshots to another HBase cluster for backup. But I¹ve noticed that there are ~4000 znodes in /hadoop-ha/testhbase1/rmstore/ZKRMStateRoot/RMAppRoot in Zookeeper. What are these znodes? Are they failed yarn applications? How I need to handle them? Thank you. Akmal
Different outputformats in avro map reduce job
Hi All, I have a map reduce job which reads a binary file and needs to output multiple avro files and a textformat file.I was able to output multiplle avro files using (Avromultipleouts).How would I modify the job to output textformat as well along with these avro files.Is it possible. Thanks, Nishanth
Re: tools.DistCp: Invalid arguments
will distcp do checksum after it copied the data to target ? On Tue, Feb 3, 2015 at 4:15 AM, Artem Ervits artemerv...@gmail.com wrote: Another good option is hftp. Artem Ervits On Feb 3, 2015 6:42 AM, xeonmailinglist xeonmailingl...@gmail.com wrote: I have found the problem. I started to use `webhdfs` and everything is ok. On 03-02-2015 10:40, xeonmailinglist wrote: What do you mean by no path is given? Even if I launch this command, I get the same error…. What path should I put here? $ hadoop distcp hdfs://hadoop-coc-1:50070/input1 hdfs://hadoop-coc-2:50070/input1 Thanks, On 02-02-2015 19:59, Alexander Alten-Lorenz wrote: Have a closer look: hdfs://hadoop-coc-2:50070/ No Path is given. On 02 Feb 2015, at 20:52, xeonmailinglist xeonmailingl...@gmail.com wrote: Hi, I am trying to copy data using distcp but I get this error. Both hadoop runtime are working properly. Why is this happening? vagrant@hadoop-coc-1:~/Programs/hadoop$ hadoop distcp hdfs://hadoop-coc-1:50070/input1 hdfs://hadoop-coc-2:50070/ 15/02/02 19:46:37 ERROR tools.DistCp: Invalid arguments: java.io.IOException: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: hadoop-coc-1/127.0.1.1; destination host is: hadoop-coc-2:50070; at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772) at org.apache.hadoop.ipc.Client.call(Client.java:1472) at org.apache.hadoop.ipc.Client.call(Client.java:1399) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1988) at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1118) at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1114) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1114) at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400) at org.apache.hadoop.tools.DistCp.setTargetPathExists(DistCp.java:188) at org.apache.hadoop.tools.DistCp.run(DistCp.java:111) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.tools.DistCp.main(DistCp.java:401) Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag. at com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:94) at com.google.protobuf.CodedInputStream.checkLastTagWas(CodedInputStream.java:124) at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:202) at com.google.protobuf.AbstractParser.parsePartialDelimitedFrom(AbstractParser.java:241) at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:253) at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:259) at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:49) at org.apache.hadoop.ipc.protobuf.RpcHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcHeaderProtos.java:3167) at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1072) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:966) Invalid arguments: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.; Host Details : local host is: hadoop-coc-1/127.0.1.1; destination host is: hadoop-coc-2:50070; usage: distcp OPTIONS [source_path...] target_path Thanks,
Re: Different outputformats in avro map reduce job
You can write out the text file by using a direct HDFS file writer. Your only concern in doing this approach would be the use of proper target directories, which is documented at http://wiki.apache.org/hadoop/FAQ#Can_I_write_create.2Fwrite-to_hdfs_files_directly_from_map.2Freduce_tasks.3F. On Thu, Jul 9, 2015 at 2:40 AM, Nishanth S chinchu2...@gmail.com wrote: Hi All, I have a map reduce job which reads a binary file and needs to output multiple avro files and a textformat file.I was able to output multiplle avro files using (Avromultipleouts).How would I modify the job to output textformat as well along with these avro files.Is it possible. Thanks, Nishanth -- Harsh J
Re: share the same namespace in 2 YARN instances?
Xeon I believe there is a YARN federation project going on and another project Apache Myriad that you may want to look at. YARN Federation https://issues.apache.org/jira/browse/YARN-2915 Apache Myriad http://incubator.apache.org/projects/myriad.html -- Thanks, Anoop Dawar Senior Director, Product Management m: 408 464 2273 skypeid: anooprdawar www.mapr.com On Mon, Mar 2, 2015 at 7:12 AM, xeonmailinglist xeonmailingl...@gmail.com wrote: Hi, I was reading about Federation of HDFS, which is possible in YARN ( http://www.devx.com/opensource/enhance-existing-hdfs-architecture-with-hadoop-federation.html), and I started to wonder if is it possible to have 2 YARN runtimes that share the same HDFS namespace? Thanks,
