Apologies again. Here is the full error message.
2021-01-08 11:34:15,831 ERROR org.apache.hadoop.hbase.ScheduledChore: Caught
error
java.lang.IllegalStateException: Expected only one of namespace and tablename
to be null
at
org.apache.hadoop.hbase.quotas.SnapshotQuotaObserverChore.getSnapshotsToComputeSize(SnapshotQuotaObserverChore.java:198)
at
org.apache.hadoop.hbase.quotas.SnapshotQuotaObserverChore._chore(SnapshotQuotaObserverChore.java:126)
at
org.apache.hadoop.hbase.quotas.SnapshotQuotaObserverChore.chore(SnapshotQuotaObserverChore.java:113)
at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:186)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:111)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
-----Original Message-----
From: Marc Hoppins <[email protected]>
Sent: Friday, January 8, 2021 10:57 AM
To: [email protected]
Subject: RE: Region server idle
EXTERNAL
So, I tried decommission that RS and recommission it. No change. Server still
idle.
Tried decommission another server and see if HBASE sets itself right. Now I
have two RS that are idle.
ba-hbase18.jumbo.hq.com,16020,1604413480001 Tue Nov 03 15:24:40 CET 2020
1 s 2.1.0-cdh6.3.2 13 471
ba-hbase19.jumbo.hq.com,16020,1610095488001 Fri Jan 08 09:44:48 CET 2021
0 s 2.1.0-cdh6.3.2 0 6
ba-hbase20.jumbo.hq.com,16020,1610096850259 Fri Jan 08 10:07:30 CET 2021
0 s 2.1.0-cdh6.3.2 0 0
ba-hbase21.jumbo.hq.com,16020,1604414101652 Tue Nov 03 15:35:01 CET 2020
1 s 2.1.0-cdh6.3.2 15 447
From the logs:
2021-01-08 10:25:36,875 ERROR org.apache.hadoop.hbase.ScheduledChore: Caught
error java.lang.IllegalStateException: Expected only one of namespace and
tablename to be null
This is reappearing in hbase master log
M
-----Original Message-----
From: Sean Busbey <[email protected]>
Sent: Thursday, January 7, 2021 7:30 PM
To: Hbase-User <[email protected]>
Subject: Re: Region server idle
EXTERNAL
Sounds like https://issues.apache.org/jira/browse/HBASE-24139
The description of that jira has a workaround.
On Thu, Jan 7, 2021, 05:23 Marc Hoppins <[email protected]> wrote:
> Hi all,
>
> I have a setup with 67 region servers. 29 Dec one system had to be
> shut down to have EMM module swapped out - which took one work day.
> Host was back online 30 Dec.
>
> My HBASE is very basic so I appreciate your patience.
>
> My understanding of the defaults that are setup is that a major
> compaction should occur every 7 days. Moreover, do I assume that more
> extensive balancing may occur after this happens?
>
> When I check (via hbase master UI) the status of HBASE, I see the
> following:
>
> ServerName
>
> Start time
>
> Last contact
>
> Version
>
> Requests Per Second
>
> Num. Regions
>
> ba-hbase16.jumbo.hq.com,16020,1604413068640<
> http://ba-hbase16.jumbo.hq.eset.com:16030/rs-status>
>
> Tue Nov 03 15:17:48 CET 2020
>
> 3 s
>
> 2.1.0-cdh6.3.2
>
> 46
>
> 462
>
> ba-hbase17.jumbo.hq.com,16020,1604413274393<
> http://ba-hbase17.jumbo.hq.eset.com:16030/rs-status>
>
> Tue Nov 03 15:21:14 CET 2020
>
> 1 s
>
> 2.1.0-cdh6.3.2
>
> 19
>
> 462
>
> ba-hbase18.jumbo.hq.com,16020,1604413480001<
> http://ba-hbase18.jumbo.hq.eset.com:16030/rs-status>
>
> Tue Nov 03 15:24:40 CET 2020
>
> 2 s
>
> 2.1.0-cdh6.3.2
>
> 62
>
> 461
>
> ba-hbase19.jumbo.hq.com,16020,1609326754985<
> http://ba-hbase19.jumbo.hq.eset.com:16030/rs-status>
>
> Wed Dec 30 12:12:34 CET 2020
>
> 2 s
>
> 2.1.0-cdh6.3.2
>
> 0
>
> 0
>
> ba-hbase20.jumbo.hq.com,16020,1604413895967<
> http://ba-hbase20.jumbo.hq.eset.com:16030/rs-status>
>
> Tue Nov 03 15:31:35 CET 2020
>
> 2 s
>
> 2.1.0-cdh6.3.2
>
> 62
>
> 503
>
> ba-hbase21.jumbo.hq.com,16020,1604414101652<
> http://ba-hbase21.jumbo.hq.eset.com:16030/rs-status>
>
> Tue Nov 03 15:35:01 CET 2020
>
> 3 s
>
> 2.1.0-cdh6.3.2
>
> 59
>
> 442
>
> ba-hbase22.jumbo.hq.com,16020,1604414308289<
> http://ba-hbase22.jumbo.hq.eset.com:16030/rs-status>
>
> Tue Nov 03 15:38:28 CET 2020
>
> 0 s
>
> 2.1.0-cdh6.3.2
>
> 40
>
> 438
>
>
> Why, after more than 7 days, is this host not hosting more (any) regions?
> Should I initiate some kind of rebalancing?
>
> Thanks in advance.
>
> M
>
