Hi Sean, Many thanks for the clarification. I read some notes on GitHub and JIRAs for Hbase and Hadoop 3 integration.
So my decision was to revert back to an earlier stable version of Hbase as I did not have the bandwidth trying to make Hbase work with Hadoop 3+ In fairness to Ted, he has always been very knowledgeable and helpful in the forum and being an engineer myself, I would not think Ted's suggestion was far off. Kind Regards, Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On Mon, 2 Jul 2018 at 14:27, Sean Busbey <[email protected]> wrote: > Hi Mich, > > Please check out the section of our reference guide on Hadoop versions: > > http://hbase.apache.org/book.html#hadoop > > the short version is that there is not yet a Hadoop 3 version that the > HBase community considers appropriate for running HBase. if you'd like > to get into details and work arounds, please join the dev@hbase > mailing list and bring it up there. > > Ted, please stop suggesting folks on the user list use anything other > than PMC sanctioned releases of HBase. > > On Sun, Jul 1, 2018 at 1:09 AM, Mich Talebzadeh > <[email protected]> wrote: > > Hi, > > > > What is the ETA with version of Hbase that will work with Hadoop 3.1 and > > may not require HA setup for HDFS? > > > > Thanks > > > > Dr Mich Talebzadeh > > > > > > > > LinkedIn * > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > > < > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > >* > > > > > > > > http://talebzadehmich.wordpress.com > > > > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > > loss, damage or destruction of data or any other property which may arise > > from relying on this email's technical content is explicitly disclaimed. > > The author will in no case be liable for any monetary damages arising > from > > such loss, damage or destruction. > > > > > > > > > > On Sun, 1 Jul 2018 at 00:26, Mich Talebzadeh <[email protected]> > > wrote: > > > >> Thanks Ted. > >> > >> Went back to hbase-1.2.6 that works OK with Hadoop 3.1 > >> > >> Dr Mich Talebzadeh > >> > >> > >> > >> LinkedIn * > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > >> < > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > >* > >> > >> > >> > >> http://talebzadehmich.wordpress.com > >> > >> > >> *Disclaimer:* Use it at your own risk. Any and all responsibility for > any > >> loss, damage or destruction of data or any other property which may > arise > >> from relying on this email's technical content is explicitly disclaimed. > >> The author will in no case be liable for any monetary damages arising > from > >> such loss, damage or destruction. > >> > >> > >> > >> > >> On Sun, 1 Jul 2018 at 00:15, Ted Yu <[email protected]> wrote: > >> > >>> Have you tried setting the value for the config to filesystem ? > >>> > >>> Cheers > >>> > >>> On Sat, Jun 30, 2018 at 4:07 PM, Mich Talebzadeh < > >>> [email protected]> > >>> wrote: > >>> > >>> > One way would be to set WAL outside of Hadoop environment. Will that > >>> work? > >>> > > >>> > The following did not work > >>> > > >>> > <property> > >>> > <name>hbase.wal.provider</name> > >>> > <value>multiwal</value> > >>> > </property> > >>> > > >>> > > >>> > Dr Mich Talebzadeh > >>> > > >>> > > >>> > > >>> > LinkedIn * https://www.linkedin.com/profile/view?id= > >>> > AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > >>> > < > >>> > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCd > >>> > OABUrV8Pw>* > >>> > > >>> > > >>> > > >>> > http://talebzadehmich.wordpress.com > >>> > > >>> > > >>> > *Disclaimer:* Use it at your own risk. Any and all responsibility for > >>> any > >>> > loss, damage or destruction of data or any other property which may > >>> arise > >>> > from relying on this email's technical content is explicitly > disclaimed. > >>> > The author will in no case be liable for any monetary damages arising > >>> from > >>> > such loss, damage or destruction. > >>> > > >>> > > >>> > > >>> > > >>> > On Sat, 30 Jun 2018 at 23:36, Ted Yu <[email protected]> wrote: > >>> > > >>> > > Please read : > >>> > > > >>> > > http://hbase.apache.org/book.html#wal.providers > >>> > > > >>> > > On Sat, Jun 30, 2018 at 3:31 PM, Mich Talebzadeh < > >>> > > [email protected]> > >>> > > wrote: > >>> > > > >>> > > > Thanks > >>> > > > > >>> > > > In your point below > >>> > > > > >>> > > > …. or you can change default WAL to FSHLog. > >>> > > > > >>> > > > is there any configuration parameter to allow me to do so in > >>> > > > hbase-site.xml? > >>> > > > > >>> > > > Dr Mich Talebzadeh > >>> > > > > >>> > > > > >>> > > > > >>> > > > LinkedIn * https://www.linkedin.com/profile/view?id= > >>> > > > AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > >>> > > > <https://www.linkedin.com/profile/view?id= > >>> > AAEAAAAWh2gBxianrbJd6zP6AcPCCd > >>> > > > OABUrV8Pw>* > >>> > > > > >>> > > > > >>> > > > > >>> > > > http://talebzadehmich.wordpress.com > >>> > > > > >>> > > > > >>> > > > *Disclaimer:* Use it at your own risk. Any and all responsibility > >>> for > >>> > any > >>> > > > loss, damage or destruction of data or any other property which > may > >>> > arise > >>> > > > from relying on this email's technical content is explicitly > >>> > disclaimed. > >>> > > > The author will in no case be liable for any monetary damages > >>> arising > >>> > > from > >>> > > > such loss, damage or destruction. > >>> > > > > >>> > > > > >>> > > > > >>> > > > > >>> > > > On Sat, 30 Jun 2018 at 23:25, Ted Yu <[email protected]> > wrote: > >>> > > > > >>> > > > > Do you plan to deploy onto hadoop 3.1.x ? > >>> > > > > > >>> > > > > If so, you'd better build against hadoop 3.1.x yourself. > >>> > > > > You can either patch in HBASE-20244 and use asyncfswal. > >>> > > > > Or you can change default WAL to FSHLog. > >>> > > > > > >>> > > > > If you don't have to deploy onto hadoop 3.1.x, you can use > hbase > >>> > 2.0.1 > >>> > > > > > >>> > > > > FYI > >>> > > > > > >>> > > > > On Sat, Jun 30, 2018 at 3:21 PM, Mich Talebzadeh < > >>> > > > > [email protected]> > >>> > > > > wrote: > >>> > > > > > >>> > > > > > so what options do I have her?. Is there any conf parameter I > >>> can > >>> > set > >>> > > > in > >>> > > > > > hbase-site,xml to make this work? or shall I go back to a > more > >>> > stable > >>> > > > > > version of Hbase? > >>> > > > > > > >>> > > > > > cheers > >>> > > > > > > >>> > > > > > Dr Mich Talebzadeh > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > LinkedIn * https://www.linkedin.com/profile/view?id= > >>> > > > > > AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > >>> > > > > > <https://www.linkedin.com/profile/view?id= > >>> > > > AAEAAAAWh2gBxianrbJd6zP6AcPCCd > >>> > > > > > OABUrV8Pw>* > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > http://talebzadehmich.wordpress.com > >>> > > > > > > >>> > > > > > > >>> > > > > > *Disclaimer:* Use it at your own risk. Any and all > >>> responsibility > >>> > for > >>> > > > any > >>> > > > > > loss, damage or destruction of data or any other property > which > >>> may > >>> > > > arise > >>> > > > > > from relying on this email's technical content is explicitly > >>> > > > disclaimed. > >>> > > > > > The author will in no case be liable for any monetary damages > >>> > arising > >>> > > > > from > >>> > > > > > such loss, damage or destruction. > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > On Sat, 30 Jun 2018 at 23:15, Ted Yu <[email protected]> > >>> wrote: > >>> > > > > > > >>> > > > > > > trunk version would correspond to hbase 3.0 which has lot > more > >>> > > > changes > >>> > > > > > > compared to hbase 2. > >>> > > > > > > The trunk build wouldn't serve you if your goal is to run > >>> hbase > >>> > on > >>> > > > > hadoop > >>> > > > > > > 3.1 (see HBASE-20244) > >>> > > > > > > > >>> > > > > > > FYI > >>> > > > > > > > >>> > > > > > > On Sat, Jun 30, 2018 at 3:11 PM, Mich Talebzadeh < > >>> > > > > > > [email protected]> > >>> > > > > > > wrote: > >>> > > > > > > > >>> > > > > > > > Thanks Ted. > >>> > > > > > > > > >>> > > > > > > > I downloaded the latest Hbase binary which is 2.0.1 > >>> 2018/06/19 > >>> > > > > > > > > >>> > > > > > > > Is there any trunc version build for Hadoop 3.1 please > and > >>> if > >>> > so > >>> > > > > where > >>> > > > > > > can > >>> > > > > > > > I download it? > >>> > > > > > > > > >>> > > > > > > > Regards, > >>> > > > > > > > > >>> > > > > > > > Dr Mich Talebzadeh > >>> > > > > > > > > >>> > > > > > > > > >>> > > > > > > > > >>> > > > > > > > LinkedIn * https://www.linkedin.com/profile/view?id= > >>> > > > > > > > AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > >>> > > > > > > > <https://www.linkedin.com/profile/view?id= > >>> > > > > > AAEAAAAWh2gBxianrbJd6zP6AcPCCd > >>> > > > > > > > OABUrV8Pw>* > >>> > > > > > > > > >>> > > > > > > > > >>> > > > > > > > > >>> > > > > > > > http://talebzadehmich.wordpress.com > >>> > > > > > > > > >>> > > > > > > > > >>> > > > > > > > *Disclaimer:* Use it at your own risk. Any and all > >>> > responsibility > >>> > > > for > >>> > > > > > any > >>> > > > > > > > loss, damage or destruction of data or any other property > >>> which > >>> > > may > >>> > > > > > arise > >>> > > > > > > > from relying on this email's technical content is > explicitly > >>> > > > > > disclaimed. > >>> > > > > > > > The author will in no case be liable for any monetary > >>> damages > >>> > > > arising > >>> > > > > > > from > >>> > > > > > > > such loss, damage or destruction. > >>> > > > > > > > > >>> > > > > > > > > >>> > > > > > > > > >>> > > > > > > > > >>> > > > > > > > On Sat, 30 Jun 2018 at 22:52, Ted Yu < > [email protected]> > >>> > > wrote: > >>> > > > > > > > > >>> > > > > > > > > Which hadoop release was the 2.0.1 built against ? > >>> > > > > > > > > > >>> > > > > > > > > In order to build hbase 2 against hadoop 3.0.1+ / > 3.1.0+, > >>> you > >>> > > > will > >>> > > > > > need > >>> > > > > > > > > HBASE-20244. > >>> > > > > > > > > > >>> > > > > > > > > FYI > >>> > > > > > > > > > >>> > > > > > > > > On Sat, Jun 30, 2018 at 2:34 PM, Mich Talebzadeh < > >>> > > > > > > > > [email protected]> > >>> > > > > > > > > wrote: > >>> > > > > > > > > > >>> > > > > > > > > > I am using the following hbase-site.xml > >>> > > > > > > > > > > >>> > > > > > > > > > <configuration> > >>> > > > > > > > > > <property> > >>> > > > > > > > > > <name>hbase.rootdir</name> > >>> > > > > > > > > > <value>hdfs://rhes75:9000/hbase</value> > >>> > > > > > > > > > </property> > >>> > > > > > > > > > <property> > >>> > > > > > > > > > <name>hbase.zookeeper.property.dataDir</name> > >>> > > > > > > > > > <value>/home/hduser/zookeeper-3.4.6</value> > >>> > > > > > > > > > </property> > >>> > > > > > > > > > <property> > >>> > > > > > > > > > <name>hbase.master</name> > >>> > > > > > > > > > <value>localhost:60000</value> > >>> > > > > > > > > > </property> > >>> > > > > > > > > > <property> > >>> > > > > > > > > > > <name>hbase.zookeeper.property.clientPort</name> > >>> > > > > > > > > > <value>2181</value> > >>> > > > > > > > > > </property> > >>> > > > > > > > > > <property> > >>> > > > > > > > > > <name>hbase.cluster.distributed</name> > >>> > > > > > > > > > <value>true</value> > >>> > > > > > > > > > </property> > >>> > > > > > > > > > <property> > >>> > > > > > > > > > <name>hbase.defaults.for.version.skip</name> > >>> > > > > > > > > > <value>true</value> > >>> > > > > > > > > > </property> > >>> > > > > > > > > > <property> > >>> > > > > > > > > > <name>phoenix.query.dateFormatTimeZone</name> > >>> > > > > > > > > > <value>Europe/London</value> > >>> > > > > > > > > > </property> > >>> > > > > > > > > > <property> > >>> > > > > > > > > > <name>hbase.procedure.store.wal.use.hsync</name> > >>> > > > > > > > > > <value>false</value> > >>> > > > > > > > > > </property>` > >>> > > > > > > > > > <property> > >>> > > > > > > > > > <name>hbase.unsafe.stream.capability.enforce</name> > >>> > > > > > > > > > <value>false</value> > >>> > > > > > > > > > </property> > >>> > > > > > > > > > </configuration> > >>> > > > > > > > > > > >>> > > > > > > > > > master starts OK but region server throws some errors > >>> > > > > > > > > > > >>> > > > > > > > > > 2018-06-30 22:23:56,607 INFO > >>> [regionserver/rhes75:16020] > >>> > > > > > > > > > wal.AbstractFSWAL: WAL configuration: blocksize=256 > MB, > >>> > > > > > rollsize=128 > >>> > > > > > > > MB, > >>> > > > > > > > > > prefix=rhes75%2C16020%2C1530393832024, suffix=, > >>> > > > > > > > > > logDir=hdfs://rhes75:9000/hbase/WALs/rhes75,16020,153 > >>> > > > > > > > > > 0393832024, > archiveDir=hdfs://rhes75:9000/hbase/oldWALs > >>> > > > > > > > > > 2018-06-30 22:23:56,629 ERROR > >>> [regionserver/rhes75:16020] > >>> > > > > > > > > > regionserver.HRegionServer: ason: > >>> > > > > > > > > > Type 'org/apache/hadoop/fs/ContentSummary' > (current > >>> > > frame, > >>> > > > > > > > stack[1]) > >>> > > > > > > > > > is > >>> > > > > > > > > > not assignable to 'org/apache/hadoop/fs/QuotaUsage' > >>> > > > > > > > > > Current Frame: > >>> > > > > > > > > > bci: @105 > >>> > > > > > > > > > flags: { } > >>> > > > > > > > > > locals: { 'org/apache/hadoop/fs/ContentSummary', > >>> > > > > > > > > > 'org/apache/hadoop/hdfs/protocol/proto/HdfsProtos$ > >>> > > > > > > > > > ContentSummaryProto$Builder' > >>> > > > > > > > > > } > >>> > > > > > > > > > stack: { > >>> > > > > > > > > > 'org/apache/hadoop/hdfs/protocol/proto/HdfsProtos$ > >>> > > > > > > > > > ContentSummaryProto$Builder', > >>> > > > > > > > > > 'org/apache/hadoop/fs/ContentSummary' } > >>> > > > > > > > > > Bytecode: > >>> > > > > > > > > > 0x0000000: 2ac7 0005 01b0 b805 984c 2b2a b605 > 99b6 > >>> > > > > > > > > > 0x0000010: 059a 2ab6 059b b605 9c2a b605 9db6 > 059e > >>> > > > > > > > > > 0x0000020: 2ab6 059f b605 a02a b605 a1b6 05a2 > 2ab6 > >>> > > > > > > > > > 0x0000030: 05a3 b605 a42a b605 a5b6 05a6 2ab6 > 05a7 > >>> > > > > > > > > > 0x0000040: b605 a82a b605 a9b6 05aa 2ab6 05ab > b605 > >>> > > > > > > > > > 0x0000050: ac2a b605 adb6 05ae 572a b605 af9a > 000a > >>> > > > > > > > > > 0x0000060: 2ab6 05b0 9900 0c2b 2ab8 0410 b605 > b157 > >>> > > > > > > > > > 0x0000070: 2bb6 05b2 b0 > >>> > > > > > > > > > Stackmap Table: > >>> > > > > > > > > > same_frame(@6) > >>> > > > > > > > > > append_frame(@103,Object[#2940]) > >>> > > > > > > > > > same_frame(@112) > >>> > > > > > > > > > ***** > >>> > > > > > > > > > java.lang.VerifyError: Bad type on operand stack > >>> > > > > > > > > > Exception Details: > >>> > > > > > > > > > Location: > >>> > > > > > > > > > > >>> > > > > > > > > > org/apache/hadoop/hdfs/protocolPB/PBHelperClient. > >>> > > > > > > > > > > >>> convert(Lorg/apache/hadoop/fs/ContentSummary;)Lorg/apache/ > >>> > > > > > > > > > > >>> hadoop/hdfs/protocol/proto/HdfsProtos$ContentSummaryProto; > >>> > > > > > > > > > @105: invokestatic > >>> > > > > > > > > > Reason: > >>> > > > > > > > > > Type 'org/apache/hadoop/fs/ContentSummary' > (current > >>> > > frame, > >>> > > > > > > > stack[1]) > >>> > > > > > > > > > is > >>> > > > > > > > > > not assignable to 'org/apache/hadoop/fs/QuotaUsage' > >>> > > > > > > > > > Current Frame: > >>> > > > > > > > > > bci: @105 > >>> > > > > > > > > > flags: { } > >>> > > > > > > > > > locals: { 'org/apache/hadoop/fs/ContentSummary', > >>> > > > > > > > > > 'org/apache/hadoop/hdfs/protocol/proto/HdfsProtos$ > >>> > > > > > > > > > ContentSummaryProto$Builder' > >>> > > > > > > > > > } > >>> > > > > > > > > > stack: { > >>> > > > > > > > > > 'org/apache/hadoop/hdfs/protocol/proto/HdfsProtos$ > >>> > > > > > > > > > ContentSummaryProto$Builder', > >>> > > > > > > > > > 'org/apache/hadoop/fs/ContentSummary' } > >>> > > > > > > > > > Bytecode: > >>> > > > > > > > > > 0x0000000: 2ac7 0005 01b0 b805 984c 2b2a b605 > 99b6 > >>> > > > > > > > > > 0x0000010: 059a 2ab6 059b b605 9c2a b605 9db6 > 059e > >>> > > > > > > > > > 0x0000020: 2ab6 059f b605 a02a b605 a1b6 05a2 > 2ab6 > >>> > > > > > > > > > 0x0000030: 05a3 b605 a42a b605 a5b6 05a6 2ab6 > 05a7 > >>> > > > > > > > > > 0x0000040: b605 a82a b605 a9b6 05aa 2ab6 05ab > b605 > >>> > > > > > > > > > 0x0000050: ac2a b605 adb6 05ae 572a b605 af9a > 000a > >>> > > > > > > > > > 0x0000060: 2ab6 05b0 9900 0c2b 2ab8 0410 b605 > b157 > >>> > > > > > > > > > 0x0000070: 2bb6 05b2 b0 > >>> > > > > > > > > > Stackmap Table: > >>> > > > > > > > > > same_frame(@6) > >>> > > > > > > > > > append_frame(@103,Object[#2940]) > >>> > > > > > > > > > same_frame(@112) > >>> > > > > > > > > > > >>> > > > > > > > > > any ideas? > >>> > > > > > > > > > > >>> > > > > > > > > > thanks > >>> > > > > > > > > > > >>> > > > > > > > > > Dr Mich Talebzadeh > >>> > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > LinkedIn * https://www.linkedin.com/profile/view?id= > >>> > > > > > > > > > AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > >>> > > > > > > > > > <https://www.linkedin.com/profile/view?id= > >>> > > > > > > > AAEAAAAWh2gBxianrbJd6zP6AcPCCd > >>> > > > > > > > > > OABUrV8Pw>* > >>> > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > http://talebzadehmich.wordpress.com > >>> > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > *Disclaimer:* Use it at your own risk. Any and all > >>> > > > responsibility > >>> > > > > > for > >>> > > > > > > > any > >>> > > > > > > > > > loss, damage or destruction of data or any other > >>> property > >>> > > which > >>> > > > > may > >>> > > > > > > > arise > >>> > > > > > > > > > from relying on this email's technical content is > >>> > explicitly > >>> > > > > > > > disclaimed. > >>> > > > > > > > > > The author will in no case be liable for any monetary > >>> > damages > >>> > > > > > arising > >>> > > > > > > > > from > >>> > > > > > > > > > such loss, damage or destruction. > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > >>> > >> >
