Istvan, What makes the case tricky is that the broken class is IA.Private. HBase has set out compatibility guidelines about what is and isn't safe for external projects to depend upon, and in this particular case it looks like we were trying to do something using an IA.Private API that isn't safe to use externally. HBase is allowed to break Phoenix on a patch level when Phoenix depends on private APIs.
One thing to follow-up on would be to work with the HBase community to allow us to replace our IA.Private calls with a new IA.LimitedPrivate or IA.Public API that does have compatibility guarantees. From looking at the Phoenix code, it looks like we're just trying to create StoreFiles as the output of a MapReduce job, which seems a reasonable thing for there to be HBase support for, in addition to HBase's own internal MapReduce formats. However, that new public or limited private API would only be usable with HBase 2.5 and up (since interface changes need a minor release.) I agree with either Andrew's kind offer to bring back temporary support of the old HStore methods in 2.4.2, or alternatively to only support 2.4.1 and up in Phoenix 5.1.1 (since 5.1 exists with 2.4.0 support and there's a workaround for PHOENIX-6377 for anyone who needs it). Would be good to avoid the extra compatibility profile this time, even if we eventually need more in the future for other reasons. Geoffrey On Tue, Feb 16, 2021 at 9:53 PM Istvan Toth <[email protected]> wrote: > Putting back the methods for 2.4.2 and blacklisting 2.4.1 could be a > solution for this specific case. (thanks for the offer) > > However similar past changes were more involved, where this wouldn't be an > option. The last two that I remember: > - multi-join support in 2.0, 2.1, 2.2 > - raw filter support in 2.2 > > I think that we should expect that these api breakages happen, and handle > these cases as the norm, > and have a policy for them. > > (Not that we shouldn't ask HBase not to break us in a patch level, if it is > just for the sake of refactoring) > > Istvan > > On Wed, Feb 17, 2021 at 5:49 AM Andrew Purtell <[email protected]> > wrote: > > > If you opened a PR that puts the HStore methods back, marks them > > @Deprecated, and implements them by calling the new StoreUtil methods, I > > would approve it. Just for branch-2.4. Call it a courtesy for Phoenix. > > > > Assuming this gets released in 2.4.2, around the end of the month, then > > you could opt to blacklist 2.4.1 in your build rather than deal with it. > > Then this becomes an issue for a future 2.5 compat module. > > > > > > > On Feb 16, 2021, at 8:38 PM, Viraj Jasani <[email protected]> wrote: > > > > > > Thanks Andrew! That’s it, HBASE-25249 is the only refactoring that > broke > > > compat and since HStore is IA.Private, I could not argue from HBase > side > > to > > > bring methods back in HStore. If we can do it in 2.4.2, that would be > > great > > > and we can skip 2.4.1 support i think. > > > WDYT Istvan? > > > > > > > > >> On Wed, 17 Feb 2021 at 3:28 AM, Andrew Purtell <[email protected]> > > wrote: > > >> > > >> Hmm... I see. This thread is probably about PHOENIX-6359, so the > change > > was > > >> HBASE-25249. > > >> > > >> The methods moved from HStore to StoreUtils can be added back to > HStore > > as > > >> compatibility methods in branch-2.4. > > >> > > >> Is there more? > > >> > > >> > > >> On Tue, Feb 16, 2021 at 1:34 PM Andrew Purtell <[email protected]> > > >> wrote: > > >> > > >>>> While supporting a new HBase patch release version (e.g 2.4.1), if > it > > >>> turns out to be incompatible with existing HBase minor release > profile > > >> (e.g > > >>> profile 2.4), we might have to consider some extra steps. > > >>> > > >>> What happened? If we broke something in a public or limited private > > >>> interface between 2.4.0 and 2.4.1, we can fix it, and you'll be good > > >> again > > >>> in 2.4.2 and forward. You could blacklist 2.4.1 in your build of the > > 2.4 > > >>> compat module using the enforcer plugin if you like. > > >>> > > >>> If the break was a private interface, but it is a simple issue, like > > >>> removal of a constant field, or removal of a method that's easy to > put > > >> back > > >>> for sake of compatibility, we can probably just put it back. By > > >> 'probably' > > >>> I mean I would not be opposed to it but there's always the chance > that > > >>> someone would object. > > >>> > > >>> > > >>>> On Tue, Feb 16, 2021 at 4:07 AM Viraj Jasani <[email protected]> > > wrote: > > >>> > > >>>> Hi, > > >>>> > > >>>> While supporting a new HBase patch release version (e.g 2.4.1), if > it > > >>>> turns > > >>>> out to be incompatible with existing HBase minor release profile > (e.g > > >>>> profile 2.4), we might have to consider some extra steps. > > >>>> > > >>>> Proposals: > > >>>> 1. Add a new profile for each compat module > > >>>> 2. Profile with HBase minor version always support latest supported > > >> HBase > > >>>> compat module and HBase patch release (e.g in our case, profile 2.4 > > uses > > >>>> compat-module 2.4.1, and profile 2.4.0 uses compat module 2.4.0) > > >>>> 3. We run jenkins tests only for latest minor release profiles (e.g > > with > > >>>> profiles 2.4 and 2.4.0 in place, we run tests for profile 2.4 only, > > >> which > > >>>> points to latest version 2.4.1) > > >>>> > > >>>> HBase profile to build compatibility examples: > > >>>> > > >>>> *Profile for HBase minor version:* > > >>>> 2.1 (supports all 2.1.x releases), > > >>>> 2.3 (supports all 2.3.x releases), > > >>>> 2.4 (if we have profile 2.4.0, 2.4 profile supports 2.4.1+ releases > > and > > >>>> not > > >>>> 2.4.0) > > >>>> > > >>>> *Profile for HBase patch version:* > > >>>> 2.4.0 (supports only 2.4.0 release) > > >>>> > > >>>> Thoughts? > > >>>> > > >>> > > >>> > > >>> -- > > >>> Best regards, > > >>> Andrew > > >>> > > >>> Words like orphans lost among the crosstalk, meaning torn from > truth's > > >>> decrepit hands > > >>> - A23, Crosstalk > > >>> > > >> > > >> > > >> -- > > >> Best regards, > > >> Andrew > > >> > > >> Words like orphans lost among the crosstalk, meaning torn from truth's > > >> decrepit hands > > >> - A23, Crosstalk > > >> > > > > > -- > *István Tóth* | Staff Software Engineer > [email protected] <https://www.cloudera.com> > [image: Cloudera] <https://www.cloudera.com/> > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera > on LinkedIn] <https://www.linkedin.com/company/cloudera> > <https://www.cloudera.com/> > ------------------------------ >
