It's rather looks like a problem with MR/Yarn client or configuration. When the MR job starts, it creates copies of all jars in staging directory and ClientDistributedCacheManager is supposed to just check whether all files exist in the cache. For better understanding why it works incorrectly, could you please add to hbase log4.properties: log4j.logger.org.apache.hadoop=DEBUG
and collect the execution log. Thanks, Sergey On Mon, Apr 24, 2017 at 8:58 AM, Vasco Pinho <va...@hotjar.com> wrote: > For additional information, CopyTable also fails with the same missing > file: > > Exception in thread "main" java.io.FileNotFoundException: File does not > exist: > hdfs://hjcluster/home/ubuntu/hbase-1.2.4/hbase-prefix-tree/ > target/hbase-prefix-tree-1.2.4.jar > > > On Wed, Apr 19, 2017 at 10:27 AM, Vasco Pinho <va...@hotjar.com> wrote: > > > Anyone have any ideas what might be causing this? Wrong classpath? Some > > package missing? > > > > On Sun, Apr 16, 2017 at 9:25 AM, Vasco Pinho <va...@hotjar.com> wrote: > > > >> Running Hadoop 2.7.2, Hbase 1.2.4. > >> > >> On Fri, Apr 14, 2017 at 6:47 PM, Ted Yu <yuzhih...@gmail.com> wrote: > >> > >>> I can debug this next week when I have access to an HA cluster. > >>> > >>> Which release of Hadoop are you using ? > >>> > >>> Cheers > >>> > >>> > On Apr 14, 2017, at 9:21 AM, Vasco Pinho <va...@hotjar.com> wrote: > >>> > > >>> > I'm not sure I follow. This file already exists under the install dir > >>> at: > >>> > > >>> > /home/ubuntu/hbase-1.2.4/hbase-prefix-tree/target/hbase-pref > >>> ix-tree-1.2.4.jar > >>> > > >>> > The class is also present at: > >>> > > >>> > /home/ubuntu/hbase-1.2.4/hbase-prefix-tree/target/classes/or > >>> g/apache/hadoop/hbase/codec/prefixtree/PrefixTreeCodec.class > >>> > > >>> > Why would I rename it? Is my classpath wrong for some reason? In any > >>> case, > >>> > renaming it to something like > >>> > "hbase-prefix-tree/target/hbase-prefix-tree-1.2.4.jar.bak " yields > >>> another > >>> > exception of the same sort (full trace: > https://pastebin.com/mzcPthZR) > >>> : > >>> > > >>> > java.io.FileNotFoundException: File does not exist: > >>> > hdfs://clusterID/root/.m2/repository/org/apache/htrace/htrac > >>> e-core/3.1.0-incubating/htrace-core-3.1.0-incubating.jar > >>> > > >>> > So it looks like something else is wrong with the installation or > >>> there is > >>> > a bug that is making these searches attempt to look in "hdfs://" > >>> instead of > >>> > the filesystem? > >>> > > >>> > Thanks, > >>> > Vasco Pinho > >>> > > >>> >> On Fri, Apr 14, 2017 at 4:32 PM, Ted Yu <yuzhih...@gmail.com> > wrote: > >>> >> > >>> >> Here is how hbase-prefix-tree dependency is detected: > >>> >> > >>> >> try { > >>> >> > >>> >> prefixTreeCodecClass = > >>> >> > >>> >> Class.forName( > >>> >> "org.apache.hadoop.hbase.codec.prefixtree.PrefixTreeCodec"); > >>> >> > >>> >> } catch (ClassNotFoundException e) { > >>> >> > >>> >> // this will show up in unit tests but should not show in real > >>> >> deployments > >>> >> > >>> >> LOG.warn("The hbase-prefix-tree module jar containing > >>> PrefixTreeCodec > >>> >> is not present." + > >>> >> > >>> >> " Continuing without it."); > >>> >> > >>> >> } > >>> >> > >>> >> As a workaround, consider temporarily renaming hbase-prefix-tree jar > >>> on the > >>> >> node where ExportSnapshot is run. > >>> >> > >>> >> Remember to rename hbase-prefix-tree jar back after the job. > >>> >> > >>> >> FYI > >>> >> > >>> >>> On Fri, Apr 14, 2017 at 5:02 AM, Vasco Pinho <va...@hotjar.com> > >>> wrote: > >>> >>> > >>> >>> Sure Ted, > >>> >>> > >>> >>> Here's the full trace: https://pastebin.com/AGimZ8wZ > >>> >>> And here's the hbase-site.xml: https://pastebin.com/yhRWCsaU > >>> >>> > >>> >>> This is a working Hbase HA setup which has been happily chugging > >>> along. 3 > >>> >>> node HDFS JN, 3 node zk, 2 of them NN+backup NN. Also two of them > >>> >>> HMaster+HMaster backup. Then several data nodes. > >>> >>> > >>> >>> Thanks for taking a look! > >>> >>> > >>> >>> Vasco > >>> >>> > >>> >>>> On Fri, Apr 14, 2017 at 1:41 PM, Ted Yu <yuzhih...@gmail.com> > >>> wrote: > >>> >>>> > >>> >>>> Can you show the complete stack trace ? > >>> >>>> > >>> >>>> Please pastebin contents of hbase/site.xml > >>> >>>> > >>> >>>> Thanks > >>> >>>> > >>> >>>>> On Apr 14, 2017, at 3:51 AM, Vasco Pinho <va...@hotjar.com> > wrote: > >>> >>>>> > >>> >>>>> When running: > >>> >>>>> > >>> >>>>> bin/hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot > >>> >>>>> -Dfs.s3a.buffer.dir=/tmp/hbase_snap_tmp -snapshot > >>> >>>> TestTable-20170413-143020 > >>> >>>>> -copy-to s3a://bucket-backup/hbase/snapshots/ > >>> >>>>> > >>> >>>>> > >>> >>>>> The operation fails with: > >>> >>>>> > >>> >>>>> > >>> >>>>> 2017-04-13 15:03:24,947 ERROR [main] snapshot.ExportSnapshot: > >>> >> Snapshot > >>> >>>>> export failed > >>> >>>>> java.io.FileNotFoundException: File does not exist: > >>> >>>>> hdfs://clusterID/home/ubuntu/hbase-1.2.4/hbase-prefix-tree/ > >>> >>>> target/hbase-prefix-tree-1.2.4.jar > >>> >>>>> at > >>> >>>>> org.apache.hadoop.hdfs.DistributedFileSystem$17. > >>> >>>> doCall(DistributedFileSystem.java:1072) > >>> >>>>> > >>> >>>>> > >>> >>>>> I assume it's looking for the > >>> >>>>> "/home/ubuntu/hbase-1.2.4/hbase-prefix-tree/target/ > >>> >>>> hbase-prefix-tree-1.2.4.jar" > >>> >>>>> which does exist but obviously not under hdfs://. I don't know > why > >>> >> this > >>> >>>>> file is needed or why it's being looked for in the hdfs:// > instead > >>> of > >>> >>> the > >>> >>>>> local filesystem, where it does exist. I confirmed that it looks > >>> >> under > >>> >>>>> "fs.defaultFS" + <hbase install path> + <relative path to > >>> >>>> hbase-prefix-tree > >>> >>>>> jar", although I have no idea how to fix this. Any ideas? > >>> >>>>> > >>> >>>>> > >>> >>>>> Thanks, > >>> >>>>> Vasco Pinho > >>> >> > >>> > >> > >> > >> > >> -- > >> Vasco Pinho > >> DevOps Engineer > >> www.hotjar.com > >> Connect with me on LinkedIn > >> <https://pt.linkedin.com/in/vasco-pinho-58770534> > >> IMPORTANT CONFIDENTIALITY NOTICE: This message is confidential and > >> intended for the use only of the person to whom this message is > addressed. > >> If you are not the intended recipient you are strictly prohibited from > >> reading, disseminating, copying or using this message, or its contents > in > >> any way and must contact the sender immediately. > >> > > > > > > > > -- > > Vasco Pinho > > DevOps Engineer > > www.hotjar.com > > Connect with me on LinkedIn > > <https://pt.linkedin.com/in/vasco-pinho-58770534> > > IMPORTANT CONFIDENTIALITY NOTICE: This message is confidential and > > intended for the use only of the person to whom this message is > addressed. > > If you are not the intended recipient you are strictly prohibited from > > reading, disseminating, copying or using this message, or its contents in > > any way and must contact the sender immediately. > > > > > > -- > Vasco Pinho > DevOps Engineer > www.hotjar.com > Connect with me on LinkedIn > <https://pt.linkedin.com/in/vasco-pinho-58770534> > IMPORTANT CONFIDENTIALITY NOTICE: This message is confidential and intended > for the use only of the person to whom this message is addressed. If you > are not the intended recipient you are strictly prohibited from reading, > disseminating, copying or using this message, or its contents in any way > and must contact the sender immediately. >
