Makes sense. Thanks Andrew for clarifying! On Thu, Feb 17, 2022, 21:28 Andrew Purtell <apurt...@apache.org> wrote:
> On Thu, Feb 17, 2022 at 12:19 PM Peter Somogyi <psomo...@apache.org> > wrote: > > > I like the idea of including the store file tracking in 2.5.0 to unblock > > the HBCK development efforts. > > > > Unfortunately, I was not following its development that much. Can it > cause > > any issues if 2.5.0 has the feature but later an incompatible change is > > needed for SFT? Can it be marked as a beta feature where we are free to > > modify interfaces? > > > > Yes, this is what I meant when I suggested we could mark it as > 'experimental'. We have done this in the past. The word 'experimental' is > prominently included adjacent to any discussion of the feature in > documentation and release notes. When we feel for sure it is stable that > word is removed. We can do something different this time of course but that > has been our past practice when introducing new functionality into > releasing code lines. And I presume we would use the Evolving interface > annotation everywhere. > > Peter > > > > On Tue, Feb 15, 2022 at 11:07 PM Andrew Purtell < > andrew.purt...@gmail.com> > > wrote: > > > > > Another option which I do not see mentioned yet is to extract the > > relevant > > > common proto and source files from the ‘hbase’ repository into a new > > > repository (‘hbase-storage’?), from which we would release artifacts to > > be > > > consumed by both hbase and hbase-operator-tools. This maintains D.R.Y. > > > through refactoring although it may down the road cause some complexity > > in > > > coordinating evolution among the three (if not more) repositories and > > > releases produced from them. This is like Josh’s Option 1 but without > > > duplication. > > > > > > Regarding the option 2 issue… If it would help we can drop SFT into > > > branch-2.5 along with the log4j2 changes and release 2.5.0 afterward. > We > > > are taking the opportunity of this minor increment to accelerate log4j1 > > > retirement, which is why it’s still waiting (but not for long). We can > > use > > > the same opportunity to release SFT even if we designate it as an > > > experimental feature if that would simplify some other logistics. For > > what > > > it’s worth. > > > > > > > On Feb 15, 2022, at 7:44 AM, Josh Elser <els...@apache.org> wrote: > > > > > > > > I was talking with Szabolcs prior to him sending this one, and it's > a > > > tricky issue for sure. > > > > > > > > To date, we've solved any HBase API issues by copying code into HBCK2 > > > e.g. HBCKMetaTableAccessor which copies parts of MetaTableAccessor, or > we > > > push the logic down server-side to the HBase Master and invoke it over > > the > > > Hbck RPC interface. > > > > > > > > I definitely want to avoid HBase version specific builds of the > > > operator-tools, so that is not an option in my mind for 2.x. The > > > discussions we had (that I remember) around HBCK2 were limited in scope > > to > > > HBase 2.x. > > > > > > > > Option 1: we copy the necessary proto files from HBase into the > > > operator-tools and try to remember that, if we make any change to the > > > serialization of the storefile list files, we have to copy that change > to > > > HBCK2. Brittle on the surface but effective. > > > > > > > > Option 2: We bump HBCK2 to hbase-2.6.0-SNAPSHOT. Problematic until we > > > make an HBase 2.6.0[-alpha] release. We should already have wire compat > > > between all of HBase 2.x which makes that a non-issue. > > > > > > > > Option 3: We create an HBCK3 targeted for HBase 3.x. I'm not > convinced > > > we need to do that (hbck for hbase 3.x would be just like hbck for > hbase > > > 2.x). This would also not solve the problem for the SFT feature in > hbase > > > 2.6. > > > > > > > > I think option 3 is a no-go. I am leaning towards option 1 at this > > > point. Hopefully my thought process is helpful for others to weigh in. > > > > > > > > > > > >> On 2/14/22 11:31 AM, Szabolcs Bukros wrote: > > > >> Hi Folks! > > > >> While working on adding tools to handle potential FileBased > > > >> StoreFileTracker issues to HBCK2 (HBASE-26624 > > > >> <https://issues.apache.org/jira/browse/HBASE-26624>) I ran into > > > multiple > > > >> problems I'm unsure how to solve. > > > >> First of all the tools would rely on files not yet available in any > of > > > the > > > >> released hbase artifacts. I tried to solve this without changing the > > > hbase > > > >> dependency version to keep HBCK2 as hbase version independent as > > > possible, > > > >> but none of the solutions I have found looked acceptable: > > > >> - Pushing the logic to the hbase side (as far as I can tell) is not > > > >> feasible because it has to be able to repair meta which is easier > when > > > >> hbase is down and the tool should be able to run without a working > > > hbase. > > > >> - The files tracking the store content are serialized proto objects > > so > > > >> while replicating those files in the operator tools is possible, it > > > would > > > >> not be pretty. > > > >> Bumping operator tools to use hbase 2.6.0-SNAPSHOT (branch-2 has the > > SFT > > > >> changes) would mean that now we need that or a newer version to > build > > > the > > > >> project and a version check to avoid runtime problems with the new > > > tools, > > > >> but otherwise this looks rather painless and backwards compatible. I > > > know > > > >> operator tools tries to avoid having a hbase-specific release, but > > > having > > > >> 2.6 as a min version to build against might be acceptable. > > > >> While looking into this I also checked what needs to be done to make > > > >> operator tools work with hbase 3.0.0-alpha-3-SNAPSHOT. Most of the > > > changes > > > >> are backwards compatible but not all of them and the ones that > aren't > > > would > > > >> make a big chunk of Fsck unusable with older hbases. For me that > looks > > > >> acceptable since this is a major version change, but that would > mean I > > > can > > > >> not rely on a potential HBCK3 to fix SFT issues, I would also need a > > > >> solution for HBCK2. > > > >> I tried to look for plans/direction regarding the new 1.3 operator > > tools > > > >> but could not find any. > > > >> Do you think it would be possible to bump the hbase version it uses > to > > > >> 2.6.0-SNAPSHOT? > > > >> Do you think it would make sense to start working on a hbase3 > > compatible > > > >> branch or is it too early? > > > >> NOTE: > > > >> I'm aware hbase does not publish SNAPSHOT builds for years, but I do > > not > > > >> know how the internal build system works and if these artifacts > would > > be > > > >> available for internal builds or not. I also do not know if > necessary > > > could > > > >> they be made available. > > > > > > > > -- > Best regards, > Andrew > > Unrest, ignorance distilled, nihilistic imbeciles - > It's what we’ve earned > Welcome, apocalypse, what’s taken you so long? > Bring us the fitting end that we’ve been counting on > - A23, Welcome, Apocalypse >
