> Good questions. If you disable snappy does it work?
See below. I don't think it's related to snappy anymore.
> If you start over fresh does it work?
DistCP in progress. Will let you know in 4 hours...
> It should be picking up native libs. Make an issue please jms. Thanks
for giving it a go.
Native was my bad. So no issue here, except maybe on documentation ;)
Ok. Some progress here. I'm able to get Snappy working fine in both HDFS
and HBase side.
hbase@node2:~/hbase-2.0.0-beta-1$ bin/hbase
org.apache.hadoop.hbase.util.CompressionTest hdfs://node2/tmp/empty.txt
snappy
Linux-amd64-64
2017-12-31 02:36:51,745 INFO [main] metrics.MetricRegistries: Loaded
MetricRegistries class
org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2017-12-31 02:36:51,874 INFO [main] hfile.CacheConfig: Created
cacheConfig: CacheConfig:disabled
2017-12-31 02:36:52,122 INFO [main] compress.CodecPool: Got brand-new
compressor [.snappy]
2017-12-31 02:36:52,142 INFO [main] compress.CodecPool: Got brand-new
compressor [.snappy]
2017-12-31 02:36:52,647 INFO [main] hfile.CacheConfig: Created
cacheConfig: CacheConfig:disabled
2017-12-31 02:36:52,758 INFO [main] compress.CodecPool: Got brand-new
decompressor [.snappy]
SUCCESS
But my regions are still not able to open, with still not any information
on the RS side. So I don't think anymore if it's because of Snappy... I
kept it running over night and still same state this morning. All my snappy
table are not deployed, but also some of my non-snappy tables are not. And
some small tables are. All single region tables are deployed correctly. All
multi-region tables are stuck. Interesting, but I don't really think there
is a pattern here. I tried running the disable command on tables with
regions in transition but the command never returns.
Last, when looking at the master web UI while HBase is starting, I got the
error below.
I will continue to play with that today to try to get it work. I will try
to open JIRAs for what ever I think is failing.
HTTP ERROR 500
Problem accessing /master-status. Reason:
Server Error
Caused by:
java.lang.NullPointerException
at
org.apache.hadoop.hbase.master.HMaster.isInMaintenanceMode(HMaster.java:2738)
at
org.apache.hadoop.hbase.master.HMaster.isBalancerOn(HMaster.java:3257)
at
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmplImpl.renderNoFlush(MasterStatusTmplImpl.java:249)
at
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmpl.renderNoFlush(MasterStatusTmpl.java:387)
at
org.apache.hadoop.hbase.tmpl.master.MasterStatusTmpl.render(MasterStatusTmpl.java:378)
at
org.apache.hadoop.hbase.master.MasterStatusServlet.doGet(MasterStatusServlet.java:81)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
at
org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:112)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
at
org.apache.hadoop.hbase.http.ClickjackingPreventionFilter.doFilter(ClickjackingPreventionFilter.java:48)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
at
org.apache.hadoop.hbase.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:1371)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
at
org.apache.hadoop.hbase.http.NoCacheFilter.doFilter(NoCacheFilter.java:49)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
at
org.apache.hadoop.hbase.http.NoCacheFilter.doFilter(NoCacheFilter.java:49)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:534)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
at
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
2017-12-31 9:37 GMT-05:00 stack <saint....@gmail.com>:
> Good questions. If you disable snappy does it work? If you start over
> fresh does it work? It should be picking up native libs. Make an issue
> please jms. Thanks for giving it a go.
>
> S
>
> On Dec 30, 2017 11:49 PM, "Jean-Marc Spaggiari" <jean-m...@spaggiari.org>
> wrote:
>
> > Hi Stack,
> >
> > I just tried to give it a try... Wipe out all HDFS content and code, all
> > HBase content and code, and all ZK. Re-build a brand new cluster with 7
> > physical worker nodes. I'm able to get HBase start, how-ever I'm not able
> > to get my regions online.
> >
> > 2017-12-31 00:42:03,187 WARN [ProcExecTimeout]
> > assignment.AssignmentManager: TODO Handle stuck in transition:
> rit=OPENING,
> > location=node8.16020,1514693333206, table=pageMini,
> > region=a778eb67898dfd378e426f2e7700faea
> > 2017-12-31 00:42:03,187 WARN [ProcExecTimeout]
> > assignment.AssignmentManager: TODO Handle stuck in transition:
> rit=OPENING,
> > location=node6.16020,1514693336563, table=work_proposed,
> > region=4a1d86197ace3f4c8b1c8de28dbe1d34
> > 2017-12-31 00:42:03,187 WARN [ProcExecTimeout]
> > assignment.AssignmentManager: TODO Handle stuck in transition:
> rit=OPENING,
> > location=node1.16020,1514693336898, table=page_crc,
> > region=86b3912a09a5676b6851636ed22c2abc
> > 2017-12-31 00:42:03,187 WARN [ProcExecTimeout]
> > assignment.AssignmentManager: TODO Handle stuck in transition:
> rit=OPENING,
> > location=node7.16020,1514693337406, table=pageAvro,
> > region=391784c43c87bdea6df05f96accad0ff
> > 2017-12-31 00:42:03,187 WARN [ProcExecTimeout]
> > assignment.AssignmentManager: TODO Handle stuck in transition:
> rit=OPENING,
> > location=node8.16020,1514693333206, table=page,
> > region=5850d782a3beea18872769bf8fd70fc7
> > 2017-12-31 00:42:03,187 WARN [ProcExecTimeout]
> > assignment.AssignmentManager: TODO Handle stuck in transition:
> rit=OPENING,
> > location=node5.16020,1514693330961, table=work_proposed,
> > region=1d892c9b54b66f802b82c2f9fe847f1f
> > 2017-12-31 00:42:03,187 WARN [ProcExecTimeout]
> > assignment.AssignmentManager: TODO Handle stuck in transition:
> rit=OPENING,
> > location=node5.16020,1514693330961, table=pageAvro,
> > region=e9de2c68cc01883e959d7953a4251687
> > 2017-12-31 00:42:03,187 WARN [ProcExecTimeout]
> > assignment.AssignmentManager: TODO Handle stuck in transition:
> rit=OPENING,
> > location=node3.16020,1514693337210, table=page,
> > region=e2e5fc1c262273893f10e92f24817d1b
> > 2017-12-31 00:42:03,187 WARN [ProcExecTimeout]
> > assignment.AssignmentManager: TODO Handle stuck in transition:
> rit=OPENING,
> > location=node3.16020,1514693337210, table=page,
> > region=89c443c09f10bd1584b1bb86a637e1a8
> > 2017-12-31 00:42:03,188 WARN [ProcExecTimeout]
> > assignment.AssignmentManager: TODO Handle stuck in transition:
> rit=OPENING,
> > location=node5.16020,1514693330961, table=page,
> > region=8ca93e9285233ca7b31992f194056bc1
> > 2017-12-31 00:42:03,188 WARN [ProcExecTimeout]
> > assignment.AssignmentManager: TODO Handle stuck in transition:
> rit=OPENING,
> > location=node4.16020,1514693339685, table=work_proposed,
> > region=9afcf06c4d0d21d7e04b0223edcfc40a
> > 2017-12-31 00:42:03,188 WARN [ProcExecTimeout]
> > assignment.AssignmentManager: TODO Handle stuck in transition:
> rit=OPENING,
> > location=node6.16020,1514693336563, table=page,
> > region=3457b3237c576eecd550eccee3f584cd
> > 2017-12-31 00:42:03,188 WARN [ProcExecTimeout]
> > assignment.AssignmentManager: TODO Handle stuck in transition:
> rit=OPENING,
> > location=node1.16020,1514693336898, table=page,
> > region=dd5fb1dbd41945a9ccbc110b8d4a51b5
> > 2017-12-31 00:42:03,188 WARN [ProcExecTimeout]
> > assignment.AssignmentManager: TODO Handle stuck in transition:
> rit=OPENING,
> > location=node7.16020,1514693337406, table=work_proposed,
> > region=480bb37af54d9fa57c727da9e8a33578
> > 2017-12-31 00:42:03,188 WARN [ProcExecTimeout]
> > assignment.AssignmentManager: TODO Handle stuck in transition:
> rit=OPENING,
> > location=node8.16020,1514693333206, table=page_crc,
> > region=56b18d470a569c5474ea084f0d995726
> > 2017-12-31 00:42:03,188 WARN [ProcExecTimeout]
> > assignment.AssignmentManager: TODO Handle stuck in transition:
> rit=OPENING,
> > location=node6.16020,1514693336563, table=page_duplicate,
> > region=e744a9af161de965c70c7d1a08b07660
> > 2017-12-31 00:42:03,188 WARN [ProcExecTimeout]
> > assignment.AssignmentManager: TODO Handle stuck in transition:
> rit=OPENING,
> > location=node1.16020,1514693336898, table=page_proposed,
> > region=1c75e53308acac6313db4be63c2b48fe
> > 2017-12-31 00:42:03,188 WARN [ProcExecTimeout]
> > assignment.AssignmentManager: TODO Handle stuck in transition:
> rit=OPENING,
> > location=node8.16020,1514693333206, table=work_proposed,
> > region=45a25ba85f6341a177db7b15554259f9
> > 2017-12-31 00:42:03,188 WARN [ProcExecTimeout]
> > assignment.AssignmentManager: TODO Handle stuck in transition:
> rit=OPENING,
> > location=node3.16020,1514693337210, table=work_proposed,
> > region=d0a58b76ad9376b12b3e763660049d3d
> > 2017-12-31 00:42:03,188 WARN [ProcExecTimeout]
> > assignment.AssignmentManager: TODO Handle stuck in transition:
> rit=OPENING,
> > location=node3.16020,1514693337210, table=page,
> > region=599a4b7b21b1d93fa232ebbbef37a31b
> > 2017-12-31 00:42:03,188 WARN [ProcExecTimeout]
> > assignment.AssignmentManager: TODO Handle stuck in transition:
> rit=OPENING,
> > location=node1.16020,1514693336898, table=page_proposed,
> > region=55c07269cc907b8e8875c2a1c4ec27d5
> > 2017-12-31 00:42:03,188 WARN [ProcExecTimeout]
> > assignment.AssignmentManager: TODO Handle stuck in transition:
> rit=OPENING,
> > location=node5.,16020,1514693330961, table=page_crc,
> > region=fa3a3d7ebc64ce2a5494cae01477d8d8
> >
> > I'm 99% confident this is because of SNAPPY. I'm fighting to get it
> working
> > but it's such a pain! My concern here is I don't see any exception
> anywhere
> > on any logs. Nothing on the RS side, nothing on the master side (Except
> > extract above).
> >
> > I suspect it's snappy because of this:
> >
> > hbase@node2:~/hbase-2.0.0-beta-1$ bin/hbase
> > org.apache.hadoop.hbase.util.CompressionTest hdfs://node2/tmp/snappy
> > snappy
> > 2017-12-31 00:45:31,006 WARN [main] util.NativeCodeLoader: Unable to
> load
> > native-hadoop library for your platform... using builtin-java classes
> where
> > applicable
> > 2017-12-31 00:45:33,283 INFO [main] metrics.MetricRegistries: Loaded
> > MetricRegistries class
> > org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
> > 2017-12-31 00:45:33,366 INFO [main] hfile.CacheConfig: Created
> > cacheConfig: CacheConfig:disabled
> > Exception in thread "main" java.lang.RuntimeException: native snappy
> > library not available: this version of libhadoop was built without snappy
> > support.
> > at
> > org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(
> > SnappyCodec.java:65)
> > at
> > org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(
> > SnappyCodec.java:134)
> > at
> > org.apache.hadoop.io.compress.CodecPool.getCompressor(
> CodecPool.java:150)
> > at
> > org.apache.hadoop.io.compress.CodecPool.getCompressor(
> CodecPool.java:168)
> > at
> > org.apache.hadoop.hbase.io.compress.Compression$Algorithm.getCompressor(
> > Compression.java:355)
> > at
> > org.apache.hadoop.hbase.io.encoding.HFileBlockDefaultEncodingConte
> > xt.<init>(HFileBlockDefaultEncodingContext.java:90)
> > at
> > org.apache.hadoop.hbase.io.hfile.NoOpDataBlockEncoder.
> > newDataBlockEncodingContext(NoOpDataBlockEncoder.java:85)
> > at
> > org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.<init>
> > (HFileBlock.java:923)
> > at
> > org.apache.hadoop.hbase.io.hfile.HFileWriterImpl.
> > finishInit(HFileWriterImpl.java:296)
> > at
> > org.apache.hadoop.hbase.io.hfile.HFileWriterImpl.<init>(
> > HFileWriterImpl.java:186)
> > at
> > org.apache.hadoop.hbase.io.hfile.HFile$WriterFactory.
> > create(HFile.java:339)
> > at
> > org.apache.hadoop.hbase.util.CompressionTest.doSmokeTest(
> > CompressionTest.java:129)
> > at
> > org.apache.hadoop.hbase.util.CompressionTest.main(
> > CompressionTest.java:167)
> >
> > But I think my installation is fine:
> > hbase@node2:~/hbase-2.0.0-beta-1$ ll native-build/
> > total 308
> > lrwxrwxrwx 1 hbase hbase 24 déc 31 00:29 libhadoopsnappy.so ->
> > libhadoopsnappy.so.0.0.1
> > lrwxrwxrwx 1 hbase hbase 24 déc 31 00:29 libhadoopsnappy.so.0 ->
> > libhadoopsnappy.so.0.0.1
> > -rwxr-xr-x 1 hbase hbase 120144 déc 31 00:29 libhadoopsnappy.so.0.0.1
> > lrwxrwxrwx 1 hbase hbase 18 déc 1 2012 libsnappy.so ->
> > libsnappy.so.1.1.3
> > lrwxrwxrwx 1 hbase hbase 18 déc 1 2012 libsnappy.so.1 ->
> > libsnappy.so.1.1.3
> > -rwxr-xr-x 1 hbase hbase 178210 déc 1 2012 libsnappy.so.1.1.3
> > drwxr-xr-x 3 hbase hbase 4096 déc 30 15:44 python2.6
> > drwxr-xr-x 4 hbase hbase 4096 déc 30 23:35 python2.7
> > drwxr-xr-x 3 hbase hbase 4096 déc 30 23:29 python3.5
> >
> > an in hbase-env.sh:
> > export JAVA_HOME=/usr/local/jdk1.8.0_151
> > export HBASE_LIBRARY_PATH=/home/hbase/hbase-2.0.0-beta-1/native-build
> >
> >
> > So there is 2 things here.
> > 1) Why are the region servers not reporting any error when they are not
> > able to open a region because of the compression codec not being loaded?
> > 2) Why is HBase not picking up the Snappy codec.
> >
> > Thanks,
> >
> > JMS
> >
> >
> > 2017-12-29 13:15 GMT-05:00 Stack <st...@duboce.net>:
> >
> > > The first release candidate for HBase 2.0.0-beta-1 is up at:
> > >
> > > https://dist.apache.org/repos/dist/dev/hbase/hbase-2.0.0-beta-1-RC0/
> > >
> > > Maven artifacts are available from a staging directory here:
> > >
> > > https://repository.apache.org/content/repositories/
> orgapachehbase-1188
> > >
> > > All was signed with my key at 8ACC93D2 [1]
> > >
> > > I tagged the RC as 2.0.0-beta-1-RC0
> > > (0907563eb72697b394b8b960fe54887d6ff304fd)
> > >
> > > hbase-2.0.0-beta-1 is our first beta release. It includes all that was
> in
> > > previous alphas (new assignment manager, offheap read/write path,
> > in-memory
> > > compactions, etc.). The APIs and feature-set are sealed.
> > >
> > > hbase-2.0.0-beta-1 is a not-for-production preview of hbase-2.0.0. It
> is
> > > meant for devs and downstreamers to test drive and flag us if we messed
> > up
> > > on anything ahead of our rolling GAs. We are particular interested in
> > > hearing from Coprocessor developers.
> > >
> > > The list of features addressed in 2.0.0 so far can be found here [3].
> > There
> > > are thousands. The list of ~2k+ fixes in 2.0.0 exclusively can be found
> > > here [4] (My JIRA JQL foo is a bit dodgy -- forgive me if mistakes).
> > >
> > > I've updated our overview doc. on the state of 2.0.0 [6]. We'll do one
> > more
> > > beta before we put up our first 2.0.0 Release Candidate by the end of
> > > January, 2.0.0-beta-2. Its focus will be making it so users can do a
> > > rolling upgrade on to hbase-2.x from hbase-1.x (and any bug fixes found
> > > running beta-1). Here is the list of what we have targeted so far for
> > > beta-2 [5]. Check it out.
> > >
> > > One knownissue is that the User API has not been properly filtered so
> it
> > > shows more than just InterfaceAudience Public content (HBASE-19663, to
> be
> > > fixed by beta-2).
> > >
> > > Please take this beta for a spin. Please vote on whether it ok to put
> out
> > > this RC as our first beta (Note CHANGES has not yet been updated). Let
> > the
> > > VOTE be open for 72 hours (Monday)
> > >
> > > Thanks,
> > > Your 2.0.0 Release Manager
> > >
> > > 1. http://pgp.mit.edu/pks/lookup?op=get&search=0x9816C7FC8ACC93D2
> > > 3. https://goo.gl/scYjJr
> > > 4. https://goo.gl/dFFT8b
> > > 5. https://issues.apache.org/jira/projects/HBASE/versions/12340862
> > > 6. https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_
> > > ktczrlKHK8N4SZzs/
> > >
> >
>
