Hi All,
If we have HBCK messages like:
21/10/11 09:54:32 WARN util.FSUtils: Skipping region because it has no family
dirs:
hdfs://nameservice-hbase-jumbo/hbase/data/urldb/prod_udb_ssl_domain/673bb257d718ac1f86a662af626a740b
and there is no matching HDFS file/dir, does this exist in META or zoo
Hi all,
I would guess that this topic has probably been raised umpteen times in the
past, so I apologise in advance if some of you are miffed. I am not a 'big
data' database person so all this tuning mucky-muck has me a bit confused.
We currently have two clusters:
Cluster 1
76 regionservers,
If you are paying for CDH then just upgrade via cloudera manager. If you are
not paying for it then I think you will find it a huge problem.
Upgade may have to be done using a version 6 then a newer version to get to a
suitable Hbase/Hadoop version.
We are currently on CDH6.3.2 but the Hbase i
of the cluster.
-Original Message-
From: Stack
Sent: Wednesday, April 7, 2021 12:55 AM
To: Hbase-User
Subject: Re: HBASE WALs
EXTERNAL
On Tue, Mar 30, 2021 at 2:52 AM Marc Hoppins wrote:
> Dear HBASE gang,
>
> ...and, as I previously mentioned, we now have a grand bun
Master WALs
-Original Message-
From: Stack
Sent: Wednesday, April 7, 2021 12:55 AM
To: Hbase-User
Subject: Re: HBASE WALs
EXTERNAL
On Tue, Mar 30, 2021 at 2:52 AM Marc Hoppins wrote:
> Dear HBASE gang,
>
> ...and, as I previously mentioned, we now have a grand bunch of OL
g that was pending in masterWALs for (possibly) all
tables?
Is there any sane way to deal with the information in masterWALs? Or is that
only a Java API thing?
Thanks for all the help/info thus far.
-Original Message-----
From: Marc Hoppins
Sent: Friday, March 26, 2021 10:49 A
in, messing
> with master WALs has me on edge.
From what I understand, you already have the tables disabled, and no unfinished
procs apart from the rogue one, so just clean out masterProcWALs and restart
master.
Em ter., 23 de mar. de 2021 às 11:13, Marc Hoppins
escreveu:
> I am still not
once again, messing
> with master WALs has me on edge.
From what I understand, you already have the tables disabled, and no unfinished
procs apart from the rogue one, so just clean out masterProcWALs and restart
master.
Em ter., 23 de mar. de 2021 às 11:13, Marc Hoppins
escreveu:
> I a
-
From: Marc Hoppins
Sent: Tuesday, March 23, 2021 12:13 PM
To: user@hbase.apache.org
Subject: RE: HBASE WALs
EXTERNAL
I am still not certain what will happen. masterProcWALs contain info for all
(running) tables, yes?
If all tables are disabled and I remove the master wals, how will that affect
ean your tables are disabled, then yeah, all
related data would already have been flushed into hfiles and wouldn't be on
your wals. But please be aware that what you really need here to get rid of the
rogue proc is to remove master proc wals, not normal wals.
Em ter., 16 de mar. de 2021
operations...or have I misinterpreted entirely?
-Original Message-
From: Stack
Sent: Tuesday, March 16, 2021 7:16 AM
To: Hbase-User
Subject: Re: HBASE WALs
EXTERNAL
On Fri, Mar 12, 2021 at 2:17 AM Marc Hoppins wrote:
> Hi, all,
>
> For our stuck region, this exists in met
then start it again.
>
>
>> I suppose it means I am asking, the 73587 DisableTableProcedure, does
>> it mean that the table is waiting to be disabled? HBASE master
>> declares that table is NOT enabled.
>>
> The table state may have been already updated to disabled
TableProcedure, does
>> it mean that the table is waiting to be disabled? HBASE master
>> declares that table is NOT enabled.
>>
> The table state may have been already updated to disabled, most of its
> regions may already be offline, but the 73587 DisableTable
ng to be disabled? HBASE master
>> declares that table is NOT enabled.
>>
> The table state may have been already updated to disabled, most of its
> regions may already be offline, but the 73587 DisableTableProcedure
> cannot be considered "done" until all its sub
more tooling supported by the community.
Em seg., 8 de mar. de 2021 às 07:50, Marc Hoppins
escreveu:
> Hi, Wellington,
>
> I was on 'vacation' (no road trip or overseas anything) for a week.
>
> All fails are waiting on the same PID (73587), a DISABLE TABLE procedure.
>
Hi all,
I have been looking for more info on this matter but cannot seem to find
anything.
What (if any) of hbck2 operator tools can be used on a disabled table?
Thanks
M
help fix a
similar problem with an offline table and RITs. Both HBASE versions are the
same.
I attach a 'sheet' with the current procs/locks.
-Original Message-
From: Marc Hoppins
Sent: Wednesday, March 3, 2021 9:51 AM
To: user@hbase.apache.org
Cc: Martin Oravec
Subject: RE:
isruptive intervention similar to what you had described in your first
email, but instead of normal wal folder, master proc wals is what you really
would need to clean out here, as that is where procedures state is persisted,
and you wouldn't want the rogue procedure to be resumed.
Em seg.,
e. A similar
> problem (but on a test cluster) involved me clearing znode info,
> deleting HDFS data for the table and deleting WALs/MasterProcWAL
> files, finally restarting HBASE service.
>
Which hbase version are you on?
Em qui., 25 de fev. de 2021 às 11:51, Marc Hoppins
escreveu:
a for the table and deleting WALs/MasterProcWAL
> files, finally restarting HBASE service.
>
Which hbase version are you on?
Em qui., 25 de fev. de 2021 às 11:51, Marc Hoppins
escreveu:
> Hi all,
>
> Do WAL files contain information for multiple regions per WAL or is
> one WAL
Hi all,
Do WAL files contain information for multiple regions per WAL or is one WAL
associated with one region?
I am trying to find a way to clear a RIT for a disabled table. A similar
problem (but on a test cluster) involved me clearing znode info, deleting HDFS
data for the table and deletin
Sorry, it seems my cut/paste omitted the PID for most of these. The PID is the
initial disabletableprocedure (73587).
Is there a sane method to kill child procedure IDs, then parent ID?
-Original Message-
From: Marc Hoppins
Sent: Monday, February 22, 2021 12:16 PM
To: user
SUCCESS hbase UnassignProcedure table=hds2_md5
I am informed that the table can remain disabled after.
What is the method to fix these issues? I have built a 'operator-tools1.0.0'
jar and dropped it in place on the active master.
Thanks again
Marc
-Original Message-
From:
Hi all,
The RIT message shows the following:
Owner procedure: { ID => '73827', PARENT_ID => '73587', STATE =>
'WAITING_TIMEOUT', OWNER => 'hbase', TYPE => 'UnassignProcedure table=hds2_md5,
region=f25fe93e24b34cb2f7fffddee1d89eec, server=ba-hbase25.jumbo.hq.
com,16020,1604475904456', START_TIM
t have had no compactions for many months, is it possible that adding new
resources (nodes) into a cluster may have no visible benefit for days, weeks,
or months?
-Original Message-----
From: Marc Hoppins
Sent: Wednesday, January 27, 2021 8:52 AM
To: user@hbase.apache.org
Subject: RE: Reg
On Tue, Jan 26, 2021 at 9:46 AM Marc Hoppins wrote:
>
> I did change it on the master. The log filled up with a bunch of reads and
> writes, so I discontinued it after about an hour. Certainly, this was
> because another IT-er had kicked off HDFS balance. Obviously, I didn't real
s balancer functions?
On Tue, Jan 26, 2021 at 8:47 AM Marc Hoppins wrote:
>
> Region counts have crept up (very slowly) to
>
> Hbase19 - 15 regions
> Hbase20 - 8 regions
>
> So, would either continual reads or, (more likely) continual writes have such
> an effect on re
.
On 1/18/21 7:23 AM, Marc Hoppins wrote:
> I have been checking for days and there are no outstanding RITs. Region
> servers do not have their own WAL files, do they?
>
> What gives me pause is that, although the affected servers (hbase19 &
> hbase20) have 11 and 3 region
g, that may be your smoking gun.
It may also be helpful for you to reach out to your vendor for support if you
have not done so already.
On 1/12/21 6:11 AM, Marc Hoppins wrote:
> I read that HDFS balancing doesn't sit well with HBASE balancing. A
> colleague rebalanced HDFS on F
hbase-site.xml)
If DEBUG doesn't help, I'd set TRACE level for
org.apache.hadoop.hbase.master.balancer. Granted, it might not be obvious to
the untrained eye, but if you can share that DEBUG/TRACE after you manuall
invoke the balancer again via hbase shell, it should be enough for
> 100req/sec, SCOPE => MACHINE
1 row(s)
Not sure what to make of it but it doesn't seem like it is enough to prevent
balancing. There are other tables and (probably) other users.
-Original Message-
From: Marc Hoppins
Sent: Monday, January 11, 2021 9:52 AM
To: user@hbase.apache.
about balancing or why
we are skipping assign to these servers? Try a balance run in the shell and
then check master log to see why no work done?
S
On Fri, Jan 8, 2021 at 2:51 AM Marc Hoppins wrote:
> Apologies again. Here is the full error message.
>
&g
-
From: Marc Hoppins
Sent: Friday, January 8, 2021 10:57 AM
To: user@hbase.apache.org
Subject: RE: Region server idle
EXTERNAL
So, I tried decommission that RS and recommission it. No change. Server still
idle.
Tried decommission another server and see if HBASE sets itself right. Now I
have two
-Original Message-
From: Sean Busbey
Sent: Thursday, January 7, 2021 7:30 PM
To: Hbase-User
Subject: Re: Region server idle
EXTERNAL
Sounds like https://issues.apache.org/jira/browse/HBASE-24139
The description of that jira has a workaround.
On Thu, Jan 7, 2021, 05:23 Marc Hoppins wrote
Sorry, I forgot to point out that MEMSTORE and STOREFILES are also 0, and that
USED HEAP is only <200MB, whereas all other hosts heap used is >16GB and <23GB.
-Original Message-----
From: Marc Hoppins
Sent: Thursday, January 7, 2021 12:23 PM
To: user@hbase.apache.org
Subjec
Hi all,
I have a setup with 67 region servers. 29 Dec one system had to be shut down to
have EMM module swapped out - which took one work day. Host was back online 30
Dec.
My HBASE is very basic so I appreciate your patience.
My understanding of the defaults that are setup is that a major comp
> Marc, what type of data are loading into hbase? curious..
> > On 12/17/2020 8:46 AM Marc Hoppins wrote:
> >
> >
> > I did restart 'instances' on master(s) so I was surprised to see a
> lingering row.
> >
> > -Original Message-
&
but not the table state rows.
Can you confirm if you still see those regions in Master UI even after
restarting masters?
Em qui., 17 de dez. de 2020 às 15:19, escreveu:
> Marc, what type of data are loading into hbase? curious..
> > On 12/17/2020 8:46 AM Marc Hoppins wrote:
> >
uld need a master restart to effectively
clean master cache.
On Thu, 17 Dec 2020, 10:12 Marc Hoppins, wrote:
> OK. I had a meandering circuit to get a version of operator-tools
> built and running.
>
> After running it on the meta table
>
> sudo hbase hbck -j /tmp/hb
he build itself is needed for Ubuntu16.
I would stick to the same java major version. OS wise, this module doesn't rely
on any OS native call, AFAIK, so should work fine between these different
versions.
Em qua., 16 de dez. de 2020 às 13:30, Marc Hoppins
escreveu:
> Thanks. Once again,
rom the main module:
https://github.com/apache/hbase-operator-tools.git
Em qua., 16 de dez. de 2020 às 08:02, Marc Hoppins
escreveu:
> Hbase is 2.1.0 (via Cloudera CDH 6.3.2)
>
> The UI page for the master shows NO report links/tabs.
>
> Hbase:meta has info for one table region which
ez. de 2020 às 17:23, Stack escreveu:
> On Tue, Dec 15, 2020 at 7:46 AM Marc Hoppins wrote:
>
> > Hi all,
> >
> > I am an HBASE newbie so I apologise if I am being repetitious.
> >
> > Apologies also if this is not the right group. Am not sure if this
> &
Hi all,
I am an HBASE newbie so I apologise if I am being repetitious.
Apologies also if this is not the right group. Am not sure if this may be more
suited to 'dev' list. However,
A problem question and a technical/wishlist question.
Problem:
I have inherited a problem with an HBASE table.
43 matches
Mail list logo
