I tend to lean towards what Andrew is saying here, but I will also admit
that this is in part from not having a good user-experience about
getting up an HMaster in maintenance mode to do surgical stuff (feels
like two steps instead of just one).
Naively, rebuilding the SFT meta files from the filesystem doesn't
require the HMaster to be up because there isn't any other "state" to
consider (which was a big reason behind pushing the work that hbck2 was
doing into the active master to avoid split-brain).
Is doing logic in HBCK2 that doesn't talk to the HMaster a -1 from you,
Duo? Similarly, is a utility in hbase-operator-tools (not a part of the
hbck2 wrapper command) also a -1?
Either are feasible, but I do think trying to build this SFT
rebuilding/recovery into a maintenance-mode HMaster will be more work.
On 2/21/22 12:27 PM, Andrew Purtell wrote:
There are some recovery cases where the cluster cannot be expected to be up
and running. What happens if we have no tooling for those? The user has a
dead cluster. So I don't think a requirement that the cluster be up and
running always is sufficient. For this type of recovery operator-tools must
be able to parse and write on disk formats. On the other hand hopefully the
cases for which that is not true are rare. In HBase 1, we had
OffineMetaRebuild. For my operations occasionally it has been necessary, in
test environments especially where users are not always clueful, and it has
shortened incident time from many hours to less than one hour. The
alternative would have been rebuild from scratch with total data loss,
which is a totally unsatisfying user experience.
On Sun, Feb 20, 2022 at 4:29 AM 张铎(Duo Zhang) <palomino...@gmail.com> wrote:
Sorry a bit late...
IIRC, the design of HBCK2 is that, most of the actual fix logic should be
done inside hbase(usually as a procedure), and the hbase-operator-tools is
just a facade for calling these methods. It will query the cluster to find
out which features are supportted. So in general, the design here is to
always have the cluster up when fixing. We have a maintenance mode where we
will just bring up HMaster and make meta table online, without loading any
other regions.
So I prefer we just use snapshot dependencies of hbase in HBCK2. It is not
a big deal for end users as if we have not make the release yet, the new
fixing options can never be actually used against a production cluster.
Anyway, this means we need to publish nightly builds then.
Thanks.
Peter Somogyi <psomo...@apache.org> 于2022年2月18日周五 06:40写道：
Makes sense. Thanks Andrew for clarifying!
On Thu, Feb 17, 2022, 21:28 Andrew Purtell <apurt...@apache.org> wrote:
On Thu, Feb 17, 2022 at 12:19 PM Peter Somogyi <psomo...@apache.org>
wrote:
I like the idea of including the store file tracking in 2.5.0 to
unblock
the HBCK development efforts.
Unfortunately, I was not following its development that much. Can it
cause
any issues if 2.5.0 has the feature but later an incompatible change
is
needed for SFT? Can it be marked as a beta feature where we are free
to
modify interfaces?
Yes, this is what I meant when I suggested we could mark it as
'experimental'. We have done this in the past. The word 'experimental'
is
prominently included adjacent to any discussion of the feature in
documentation and release notes. When we feel for sure it is stable
that
word is removed. We can do something different this time of course but
that
has been our past practice when introducing new functionality into
releasing code lines. And I presume we would use the Evolving interface
annotation everywhere.
Peter
On Tue, Feb 15, 2022 at 11:07 PM Andrew Purtell <
andrew.purt...@gmail.com>
wrote:
Another option which I do not see mentioned yet is to extract the
relevant
common proto and source files from the ‘hbase’ repository into a
new
repository (‘hbase-storage’?), from which we would release
artifacts
to
be
consumed by both hbase and hbase-operator-tools. This maintains
D.R.Y.
through refactoring although it may down the road cause some
complexity
in
coordinating evolution among the three (if not more) repositories
and
releases produced from them. This is like Josh’s Option 1 but
without
duplication.
Regarding the option 2 issue… If it would help we can drop SFT into
branch-2.5 along with the log4j2 changes and release 2.5.0
afterward.
We
are taking the opportunity of this minor increment to accelerate
log4j1
retirement, which is why it’s still waiting (but not for long). We
can
use
the same opportunity to release SFT even if we designate it as an
experimental feature if that would simplify some other logistics.
For
what
it’s worth.
On Feb 15, 2022, at 7:44 AM, Josh Elser <els...@apache.org>
wrote:
I was talking with Szabolcs prior to him sending this one, and
it's
a
tricky issue for sure.
To date, we've solved any HBase API issues by copying code into
HBCK2
e.g. HBCKMetaTableAccessor which copies parts of MetaTableAccessor,
or
we
push the logic down server-side to the HBase Master and invoke it
over
the
Hbck RPC interface.
I definitely want to avoid HBase version specific builds of the
operator-tools, so that is not an option in my mind for 2.x. The
discussions we had (that I remember) around HBCK2 were limited in
scope
to
HBase 2.x.
Option 1: we copy the necessary proto files from HBase into the
operator-tools and try to remember that, if we make any change to
the
serialization of the storefile list files, we have to copy that
change
to
HBCK2. Brittle on the surface but effective.
Option 2: We bump HBCK2 to hbase-2.6.0-SNAPSHOT. Problematic
until
we
make an HBase 2.6.0[-alpha] release. We should already have wire
compat
between all of HBase 2.x which makes that a non-issue.
Option 3: We create an HBCK3 targeted for HBase 3.x. I'm not
convinced
we need to do that (hbck for hbase 3.x would be just like hbck for
hbase
2.x). This would also not solve the problem for the SFT feature in
hbase
2.6.
I think option 3 is a no-go. I am leaning towards option 1 at
this
point. Hopefully my thought process is helpful for others to weigh
in.
On 2/14/22 11:31 AM, Szabolcs Bukros wrote:
Hi Folks!
While working on adding tools to handle potential FileBased
StoreFileTracker issues to HBCK2 (HBASE-26624
<https://issues.apache.org/jira/browse/HBASE-26624>) I ran into
multiple
problems I'm unsure how to solve.
First of all the tools would rely on files not yet available in
any
of
the
released hbase artifacts. I tried to solve this without changing
the
hbase
dependency version to keep HBCK2 as hbase version independent as
possible,
but none of the solutions I have found looked acceptable:
- Pushing the logic to the hbase side (as far as I can tell) is
not
feasible because it has to be able to repair meta which is
easier
when
hbase is down and the tool should be able to run without a
working
hbase.
- The files tracking the store content are serialized proto
objects
so
while replicating those files in the operator tools is possible,
it
would
not be pretty.
Bumping operator tools to use hbase 2.6.0-SNAPSHOT (branch-2 has
the
SFT
changes) would mean that now we need that or a newer version to
build
the
project and a version check to avoid runtime problems with the
new
tools,
but otherwise this looks rather painless and backwards
compatible. I
know
operator tools tries to avoid having a hbase-specific release,
but
having
2.6 as a min version to build against might be acceptable.
While looking into this I also checked what needs to be done to
make
operator tools work with hbase 3.0.0-alpha-3-SNAPSHOT. Most of
the
changes
are backwards compatible but not all of them and the ones that
aren't
would
make a big chunk of Fsck unusable with older hbases. For me that
looks
acceptable since this is a major version change, but that would
mean I
can
not rely on a potential HBCK3 to fix SFT issues, I would also
need a
solution for HBCK2.
I tried to look for plans/direction regarding the new 1.3
operator
tools
but could not find any.
Do you think it would be possible to bump the hbase version it
uses
to
2.6.0-SNAPSHOT?
Do you think it would make sense to start working on a hbase3
compatible
branch or is it too early?
NOTE:
I'm aware hbase does not publish SNAPSHOT builds for years, but
I
do
not
know how the internal build system works and if these artifacts
would
be
available for internal builds or not. I also do not know if
necessary
could
they be made available.
--
Best regards,
Andrew
Unrest, ignorance distilled, nihilistic imbeciles -
It's what we’ve earned
Welcome, apocalypse, what’s taken you so long?
Bring us the fitting end that we’ve been counting on
- A23, Welcome, Apocalypse
