Hi Sergey, Thanks for the tip, I've added the DEBUG line to the log properties and this is what I got: https://pastebin.com/0k1tgw3n Unfortunately it looks like other than lots of S3 chatter there's not much more info regarding the actual issue. As for the MR/Yarn, there was no care to tune any specific settings as we're just running HBase on top of Hadoop. Are there any specific things related to that I might check first?
Thanks, Vasco On Tue, Apr 25, 2017 at 1:56 AM, Sergey Soldatov <sergeysolda...@gmail.com> wrote: > It's rather looks like a problem with MR/Yarn client or configuration. When > the MR job starts, it creates copies of all jars in staging directory and > ClientDistributedCacheManager is supposed to just check whether all files > exist in the cache. For better understanding why it works incorrectly, > could you please add to hbase log4.properties: > log4j.logger.org.apache.hadoop=DEBUG > > and collect the execution log. > > Thanks, > Sergey > > On Mon, Apr 24, 2017 at 8:58 AM, Vasco Pinho <va...@hotjar.com> wrote: > > > For additional information, CopyTable also fails with the same missing > > file: > > > > Exception in thread "main" java.io.FileNotFoundException: File does not > > exist: > > hdfs://hjcluster/home/ubuntu/hbase-1.2.4/hbase-prefix-tree/ > > target/hbase-prefix-tree-1.2.4.jar > > > > > > On Wed, Apr 19, 2017 at 10:27 AM, Vasco Pinho <va...@hotjar.com> wrote: > > > > > Anyone have any ideas what might be causing this? Wrong classpath? Some > > > package missing? > > > > > > On Sun, Apr 16, 2017 at 9:25 AM, Vasco Pinho <va...@hotjar.com> wrote: > > > > > >> Running Hadoop 2.7.2, Hbase 1.2.4. > > >> > > >> On Fri, Apr 14, 2017 at 6:47 PM, Ted Yu <yuzhih...@gmail.com> wrote: > > >> > > >>> I can debug this next week when I have access to an HA cluster. > > >>> > > >>> Which release of Hadoop are you using ? > > >>> > > >>> Cheers > > >>> > > >>> > On Apr 14, 2017, at 9:21 AM, Vasco Pinho <va...@hotjar.com> wrote: > > >>> > > > >>> > I'm not sure I follow. This file already exists under the install > dir > > >>> at: > > >>> > > > >>> > /home/ubuntu/hbase-1.2.4/hbase-prefix-tree/target/hbase-pref > > >>> ix-tree-1.2.4.jar > > >>> > > > >>> > The class is also present at: > > >>> > > > >>> > /home/ubuntu/hbase-1.2.4/hbase-prefix-tree/target/classes/or > > >>> g/apache/hadoop/hbase/codec/prefixtree/PrefixTreeCodec.class > > >>> > > > >>> > Why would I rename it? Is my classpath wrong for some reason? In > any > > >>> case, > > >>> > renaming it to something like > > >>> > "hbase-prefix-tree/target/hbase-prefix-tree-1.2.4.jar.bak " yields > > >>> another > > >>> > exception of the same sort (full trace: > > https://pastebin.com/mzcPthZR) > > >>> : > > >>> > > > >>> > java.io.FileNotFoundException: File does not exist: > > >>> > hdfs://clusterID/root/.m2/repository/org/apache/htrace/htrac > > >>> e-core/3.1.0-incubating/htrace-core-3.1.0-incubating.jar > > >>> > > > >>> > So it looks like something else is wrong with the installation or > > >>> there is > > >>> > a bug that is making these searches attempt to look in "hdfs://" > > >>> instead of > > >>> > the filesystem? > > >>> > > > >>> > Thanks, > > >>> > Vasco Pinho > > >>> > > > >>> >> On Fri, Apr 14, 2017 at 4:32 PM, Ted Yu <yuzhih...@gmail.com> > > wrote: > > >>> >> > > >>> >> Here is how hbase-prefix-tree dependency is detected: > > >>> >> > > >>> >> try { > > >>> >> > > >>> >> prefixTreeCodecClass = > > >>> >> > > >>> >> Class.forName( > > >>> >> "org.apache.hadoop.hbase.codec.prefixtree.PrefixTreeCodec"); > > >>> >> > > >>> >> } catch (ClassNotFoundException e) { > > >>> >> > > >>> >> // this will show up in unit tests but should not show in > real > > >>> >> deployments > > >>> >> > > >>> >> LOG.warn("The hbase-prefix-tree module jar containing > > >>> PrefixTreeCodec > > >>> >> is not present." + > > >>> >> > > >>> >> " Continuing without it."); > > >>> >> > > >>> >> } > > >>> >> > > >>> >> As a workaround, consider temporarily renaming hbase-prefix-tree > jar > > >>> on the > > >>> >> node where ExportSnapshot is run. > > >>> >> > > >>> >> Remember to rename hbase-prefix-tree jar back after the job. > > >>> >> > > >>> >> FYI > > >>> >> > > >>> >>> On Fri, Apr 14, 2017 at 5:02 AM, Vasco Pinho <va...@hotjar.com> > > >>> wrote: > > >>> >>> > > >>> >>> Sure Ted, > > >>> >>> > > >>> >>> Here's the full trace: https://pastebin.com/AGimZ8wZ > > >>> >>> And here's the hbase-site.xml: https://pastebin.com/yhRWCsaU > > >>> >>> > > >>> >>> This is a working Hbase HA setup which has been happily chugging > > >>> along. 3 > > >>> >>> node HDFS JN, 3 node zk, 2 of them NN+backup NN. Also two of them > > >>> >>> HMaster+HMaster backup. Then several data nodes. > > >>> >>> > > >>> >>> Thanks for taking a look! > > >>> >>> > > >>> >>> Vasco > > >>> >>> > > >>> >>>> On Fri, Apr 14, 2017 at 1:41 PM, Ted Yu <yuzhih...@gmail.com> > > >>> wrote: > > >>> >>>> > > >>> >>>> Can you show the complete stack trace ? > > >>> >>>> > > >>> >>>> Please pastebin contents of hbase/site.xml > > >>> >>>> > > >>> >>>> Thanks > > >>> >>>> > > >>> >>>>> On Apr 14, 2017, at 3:51 AM, Vasco Pinho <va...@hotjar.com> > > wrote: > > >>> >>>>> > > >>> >>>>> When running: > > >>> >>>>> > > >>> >>>>> bin/hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot > > >>> >>>>> -Dfs.s3a.buffer.dir=/tmp/hbase_snap_tmp -snapshot > > >>> >>>> TestTable-20170413-143020 > > >>> >>>>> -copy-to s3a://bucket-backup/hbase/snapshots/ > > >>> >>>>> > > >>> >>>>> > > >>> >>>>> The operation fails with: > > >>> >>>>> > > >>> >>>>> > > >>> >>>>> 2017-04-13 15:03:24,947 ERROR [main] snapshot.ExportSnapshot: > > >>> >> Snapshot > > >>> >>>>> export failed > > >>> >>>>> java.io.FileNotFoundException: File does not exist: > > >>> >>>>> hdfs://clusterID/home/ubuntu/hbase-1.2.4/hbase-prefix-tree/ > > >>> >>>> target/hbase-prefix-tree-1.2.4.jar > > >>> >>>>> at > > >>> >>>>> org.apache.hadoop.hdfs.DistributedFileSystem$17. > > >>> >>>> doCall(DistributedFileSystem.java:1072) > > >>> >>>>> > > >>> >>>>> > > >>> >>>>> I assume it's looking for the > > >>> >>>>> "/home/ubuntu/hbase-1.2.4/hbase-prefix-tree/target/ > > >>> >>>> hbase-prefix-tree-1.2.4.jar" > > >>> >>>>> which does exist but obviously not under hdfs://. I don't know > > why > > >>> >> this > > >>> >>>>> file is needed or why it's being looked for in the hdfs:// > > instead > > >>> of > > >>> >>> the > > >>> >>>>> local filesystem, where it does exist. I confirmed that it > looks > > >>> >> under > > >>> >>>>> "fs.defaultFS" + <hbase install path> + <relative path to > > >>> >>>> hbase-prefix-tree > > >>> >>>>> jar", although I have no idea how to fix this. Any ideas? > > >>> >>>>> > > >>> >>>>> > > >>> >>>>> Thanks, > > >>> >>>>> Vasco Pinho > > >>> >> > > >>> > > >> > > >> > > >> > > >> -- > > >> Vasco Pinho > > >> DevOps Engineer > > >> www.hotjar.com > > >> Connect with me on LinkedIn > > >> <https://pt.linkedin.com/in/vasco-pinho-58770534> > > >> IMPORTANT CONFIDENTIALITY NOTICE: This message is confidential and > > >> intended for the use only of the person to whom this message is > > addressed. > > >> If you are not the intended recipient you are strictly prohibited from > > >> reading, disseminating, copying or using this message, or its contents > > in > > >> any way and must contact the sender immediately. > > >> > > > > > > > > > > > > -- > > > Vasco Pinho > > > DevOps Engineer > > > www.hotjar.com > > > Connect with me on LinkedIn > > > <https://pt.linkedin.com/in/vasco-pinho-58770534> > > > IMPORTANT CONFIDENTIALITY NOTICE: This message is confidential and > > > intended for the use only of the person to whom this message is > > addressed. > > > If you are not the intended recipient you are strictly prohibited from > > > reading, disseminating, copying or using this message, or its contents > in > > > any way and must contact the sender immediately. > > > > > > > > > > > -- > > Vasco Pinho > > DevOps Engineer > > www.hotjar.com > > Connect with me on LinkedIn > > <https://pt.linkedin.com/in/vasco-pinho-58770534> > > IMPORTANT CONFIDENTIALITY NOTICE: This message is confidential and > intended > > for the use only of the person to whom this message is addressed. If you > > are not the intended recipient you are strictly prohibited from reading, > > disseminating, copying or using this message, or its contents in any way > > and must contact the sender immediately. > > >
