As Viraj wrote, those are just plans. If HBase 3 won't be released by the time the other features are ready, then it won't make it into 5.3. If other major features are ready by that time, then they will be included. (though we are not aware of any now)
As for the new major version, in the past Phoenix didn't have a compatibility module system, so a new branch was required, which didn't support older HBases. Also, the API changes between HBase 1.x and 2.x were much larger, The HBase 2 and 3 API are pretty similar, apart from the removal of deprecated 1.x APIs. (and the protobuf/protocol thing, which requires a rather ugly hack). I will start the discussion on how we can add HBase 3 support as soon as I have a working POC patch. We could call 5.3 6.0 instead, after all Phoenix isn't using a strict semantic versioning, but then 6.0 would also support HBase 2. If we do not come to a consensus on the version name, we can always have a vote on it. I think that the main motivation is that the community wants to maintain as few branches as possible. Istvan On Mon, Jan 29, 2024 at 10:02 PM Stephen Jiang <syuanjiang...@gmail.com> wrote: > I am not sure how close HBase 3.0 is. Even if it is only less than one > year away, the adoption would be low at the beginning. I don't think 5.3 > should wait for that. And traditionally, Phoenix would have a major > release to support the HBase major release (4.x for HBase 1.x and 5.x for > HBase 2.x), in this case, we are talking about Phoenix 6.0 for HBase 3.0. > > Maybe we should adopt the HBase release model: master branch for next major > release (6.0) and branch-5.x branch for next 5.x minor release and > branch-5.2 for 5.2 minor release. > > Thanks > Stephen > > > > On Thu, Jan 25, 2024 at 11:50 PM Viraj Jasani <vjas...@apache.org> wrote: > > > Sounds good. > > > > Planned major changes for 5.3.0: > > > > 1. JSON support. > > 2. HBase 3.0 support. > > 3. CDC feature (leveraging uncovered global index framework and JSON > > support). > > > > > > On Wed, Jan 24, 2024 at 8:22 PM Kadir Ozdemir > > <kozde...@salesforce.com.invalid> wrote: > > > > > I suggest including another major change for 5.3, Phoenix CDC, > > > https://issues.apache.org/jira/browse/PHOENIX-7001. The PR for it will > > be > > > posted soon. > > > > > > On Thu, Jan 25, 2024 at 5:35 AM Viraj Jasani <vjas...@apache.org> > wrote: > > > > > > > Thanks Istvan! I agree with your points. It’s really been a while we > > are > > > > talking about releasing 5.2.0 and yet due to bandwidth issues, unable > > to > > > do > > > > so. > > > > > > > > I agree to getting these fixes out, cut 5.2 and start the release > work > > > and > > > > in the meantime I also need to prepare 5.1 backport. > > > > > > > > For 5.3, let's plan HBase 3.0 support and JSON as major changes. > > > > > > > > > > > > On Mon, Jan 22, 2024 at 9:17 PM Istvan Toth <st...@apache.org> > wrote: > > > > > > > > > Hi! > > > > > > > > > > In my opinion cutting 5.2 now only makes sense IF we DO NOT plan to > > > > release > > > > > the outstanding big features (like JSON) in 5.2. , otherwise it's > > just > > > > > extra work to maintain more branches. > > > > > > > > > > Having said that, releasing a 5.2 and 5.1.4 with the data integrity > > > fixes > > > > > real soon, and then releasing 5.3 in a few months with JSON, and > any > > > > other > > > > > outstanding big features > > > > > that are close to being finished (and HBase 3.0 support, if it's > > ready > > > by > > > > > then) would not be a bad idea. > > > > > > > > > > On the CLDR side the only outstanding big feature which could > impact > > > > > Viraj's integrity work is HBase 3.0 support, and even that is only > > > > because > > > > > it may require some larger refactors of existing code, not because > it > > > > would > > > > > change the actual behaviour or algorithms. > > > > > > > > > > Phoenix used to have several minor releases per year, the current > > state > > > > of > > > > > extreme longevity of 5.1 and several big new features being added > to > > it > > > > > (like uncovered indexes) is not ideal. > > > > > Releasing 5.2 and 5.3 relatively close together could be a return > to > > a > > > > > quicker cadence for minor releases, which could also help with the > > > public > > > > > image and adoption of Phoenix. > > > > > > > > > > We were talking about releasing 5.2 at least a year ago, and I have > > > > started > > > > > working on that then, but then emergencies have come up at $dayjob, > > > and I > > > > > could not see that through. > > > > > (So I am in part responsible for the lack of minor releases) > > > > > > > > > > regards > > > > > Istvan > > > > > > > > > > > > > > > On Mon, Jan 22, 2024 at 11:35 PM Viraj Jasani <vjas...@apache.org> > > > > wrote: > > > > > > > > > > > Sorry for the late reply. > > > > > > > > > > > > > Do you think cutting 5.2.0 now makes sense? > > > > > > > > > > > > No problem with that. I can cut 5.2 branch by the end of this > week > > or > > > > at > > > > > > the start of next week. > > > > > > > > > > > > If there is any very big change or feature ready for merge to > > master > > > > > branch > > > > > > with PR approvals already in place, please do let me know so > that I > > > can > > > > > > help collaborate on how best we can get it merged without > impacting > > > 5.2 > > > > > > release if required. My main motivation was for any big change to > > go > > > > > > through newly introduced tests so that we know that anything > > > additional > > > > > is > > > > > > not broken, and also to prioritize for upcoming 5.2.0 and 5.1.4 > > > > releases. > > > > > > Moreover, there are several PRs getting merged on the master > > branch, > > > we > > > > > can > > > > > > continue that as long as they are not very big changes, which > might > > > > > require > > > > > > significant time to understand any correlation with data > integrity > > > > > issues. > > > > > > > > > > > > The PR is also ready for review with some additional cases fixed > > last > > > > > week: > > > > > > > > > > > > > > > > https://urldefense.com/v3/__https://github.com/apache/phoenix/pull/1736__;!!DCbAVzZNrAf4!D4OVjUp2EWW2BqhGnBxsapDX_AHsibRphIpoFBWfgRsd3dsAikrFLo6PGxdTzGbSXJJ2fJ0j9mcz3asXMXo$ > > > > > > > > > > > > Depending on the review bandwidth, I am hopeful we should be good > > to > > > > land > > > > > > them sooner. > > > > > > > > > > > > > > > > > > On Wed, Jan 17, 2024 at 11:31 AM Rushabh Shah > > > > > > <rushabh.s...@salesforce.com.invalid> wrote: > > > > > > > > > > > > > Thank you Viraj for initiating this thread. > > > > > > > > > > > > > > > Given the critical nature of these issues, I would like to > > > propose > > > > > that > > > > > > > we > > > > > > > treat this as a high priority for the upcoming 5.2.0 release, > and > > > not > > > > > > > include any other feature or big change to master branch until > we > > > > merge > > > > > > > this. > > > > > > > > > > > > > > Do you think cutting 5.2.0 now makes sense? This will enable > > other > > > > > > > developers to merge features into master branch (5.3.0) and you > > can > > > > > take > > > > > > > some more time to make sure we cover all the corner cases for > the > > > > data > > > > > > > integrity issues that you uncovered. > > > > > > > > > > > > > > > > > > > > > On Fri, Jan 5, 2024 at 6:38 PM Viraj Jasani < > vjas...@apache.org> > > > > > wrote: > > > > > > > > > > > > > > > Sounds good, thanks Rajeshbabu. I will try to get the first > PR > > > out > > > > > next > > > > > > > > week and while reviews happen in parallel, will try to get > 5.1 > > PR > > > > > soon. > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Jan 3, 2024 at 8:49 PM rajeshb...@apache.org < > > > > > > > > chrajeshbab...@gmail.com> wrote: > > > > > > > > > > > > > > > > > Hi Viraj, > > > > > > > > > > > > > > > > > > Would be better to include the changes in 5.1.4 as in any > > way > > > it > > > > > > will > > > > > > > > take > > > > > > > > > at least 3-4 days to complete the omid release. > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Rajeshbabu. > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Jan 4, 2024 at 5:06 AM Viraj Jasani < > > > vjas...@apache.org> > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Thank you Kadir and Geoffrey for your replies!! > > > > > > > > > > > > > > > > > > > > > How does this affect 5.1.4, which is also listed as a > Fix > > > > > Version > > > > > > > for > > > > > > > > > > > PHOENIX-7106? > > > > > > > > > > > > > > > > > > > > Yes, it also needs to be ported to 5.1. Once the master > PR > > is > > > > up > > > > > > for > > > > > > > > > final > > > > > > > > > > review, I would start working on the backport PR. > > > > > > > > > > We just need some more additional testing to ensure old > > > client > > > > > > (e.g. > > > > > > > > > 5.1.3) > > > > > > > > > > is compatible with the new server with the changes. > > > > > > > > > > > > > > > > > > > > Hence, yes it is now a blocker for upcoming 5.1.4 as well > > > since > > > > > > 5.1.4 > > > > > > > > RC > > > > > > > > > > preparation is still pending (while Omid release is in > > > > progress). > > > > > > > > > > Otherwise if 5.1.4 was ready for release, I would have > > > proposed > > > > > > > > immediate > > > > > > > > > > 5.1.5 release to include the changes proposed with > > > > PHOENIX-7106. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Jan 3, 2024 at 3:08 PM Geoffrey Jacoby < > > > > > gjac...@apache.org > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > I agree that data integrity issues are a higher > priority > > > than > > > > > > > feature > > > > > > > > > > > development, so I also support the decision. The fact > > that > > > > > > several > > > > > > > of > > > > > > > > > the > > > > > > > > > > > major remaining 5.2 features are currently being > > developed > > > in > > > > > > > > > > long-running > > > > > > > > > > > feature branches also helps, as work can continue there > > at > > > > the > > > > > > cost > > > > > > > > of > > > > > > > > > a > > > > > > > > > > > rebase later. > > > > > > > > > > > > > > > > > > > > > > How does this affect 5.1.4, which is also listed as a > Fix > > > > > Version > > > > > > > for > > > > > > > > > > > PHOENIX-7106? From the bug description it also sounds > > like > > > > > 5.1.3 > > > > > > > and > > > > > > > > > the > > > > > > > > > > > forthcoming .4 are affected, since we have server-side > > > paging > > > > > in > > > > > > > 5.1. > > > > > > > > > > (Feel > > > > > > > > > > > free to move that to a separate thread if you feel it > > > should > > > > > be a > > > > > > > > > > separate > > > > > > > > > > > discussion.) Should this be a blocker for releasing > > 5.1.4? > > > > > > > > > > > > > > > > > > > > > > Geoffrey > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Jan 3, 2024 at 5:06 PM Kadir Ozdemir < > > > > > > > > > > > ka...@gsuite.cloud.apache.org> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Being a database, Phoenix has to make sure that the > > data > > > > > stays > > > > > > on > > > > > > > > > disk > > > > > > > > > > > > intact and its queries return correct data. In this > > case, > > > > > > Phoenix > > > > > > > > > fails > > > > > > > > > > > to > > > > > > > > > > > > return correct data for some queries if their scans > > > > > experience > > > > > > > > region > > > > > > > > > > > > movement. Now that we know these data integrity > issues > > > and > > > > > how > > > > > > to > > > > > > > > > > > reproduce > > > > > > > > > > > > them, fixing them should be our first priority. So, I > > > fully > > > > > > > support > > > > > > > > > > this > > > > > > > > > > > > proposal. > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Jan 3, 2024 at 10:58 PM Viraj Jasani < > > > > > > vjas...@apache.org > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Hello, > > > > > > > > > > > > > > > > > > > > > > > > > > I would like to bring PHOENIX-7106 > > > > > > > > > > > > > < > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://urldefense.com/v3/__https://issues.apache.org/jira/browse/PHOENIX-7106__;!!DCbAVzZNrAf4!FZG5sv55IC1NqItQLY7GKWgUG2Do0gSta01gOiSdd36Dx3XHGtQx4M3c9visVXIt9DctPQzS-orob9vhzrCfVA$ > > > > > > > > > to everyone's > > > > > > > > > > > > > attention here and brief about the data integrity > > > issues > > > > > that > > > > > > > we > > > > > > > > > have > > > > > > > > > > > in > > > > > > > > > > > > > various coprocessors. Majority of the issues are > > > related > > > > to > > > > > > the > > > > > > > > > fact > > > > > > > > > > > that > > > > > > > > > > > > > we do not return valid rowkey for certain queries. > If > > > any > > > > > > > region > > > > > > > > > > moves > > > > > > > > > > > in > > > > > > > > > > > > > the middle of the scan, the HBase client relies on > > the > > > > last > > > > > > > > > returned > > > > > > > > > > > > rowkey > > > > > > > > > > > > > and accordingly changes the scan boundaries while > the > > > > > scanner > > > > > > > is > > > > > > > > > > > getting > > > > > > > > > > > > > reset to continue the scan operation. If the region > > > does > > > > > not > > > > > > > > move, > > > > > > > > > > scan > > > > > > > > > > > > is > > > > > > > > > > > > > not expected to return invalid data, however if the > > > > region > > > > > > > moves > > > > > > > > in > > > > > > > > > > the > > > > > > > > > > > > > middle of ongoing scan operation, scan would return > > > > > > > > > invalid/incorrect > > > > > > > > > > > > data > > > > > > > > > > > > > causing data integrity issues. > > > > > > > > > > > > > > > > > > > > > > > > > > Given the critical nature of these issues, I would > > like > > > > to > > > > > > > > propose > > > > > > > > > > that > > > > > > > > > > > > we > > > > > > > > > > > > > treat this as a high priority for the upcoming > 5.2.0 > > > > > release, > > > > > > > and > > > > > > > > > not > > > > > > > > > > > > > include any other feature or big change to master > > > branch > > > > > > until > > > > > > > we > > > > > > > > > > merge > > > > > > > > > > > > > this. The PR is still not ready as additional > changes > > > are > > > > > > still > > > > > > > > in > > > > > > > > > my > > > > > > > > > > > > > local, requiring rebase with the current master. > > > > > > > > > > > > > > > > > > > > > > > > > > I would get back to this discuss thread as soon as > > the > > > PR > > > > > and > > > > > > > the > > > > > > > > > doc > > > > > > > > > > > are > > > > > > > > > > > > > updated with the latest findings so far. The > changes > > > > > include > > > > > > > many > > > > > > > > > of > > > > > > > > > > > our > > > > > > > > > > > > > coproc scanner implementations and hence it would > > > require > > > > > > > > > significant > > > > > > > > > > > > > review as well. > > > > > > > > > > > > > It would be great if we can hold on to merging any > > > > feature > > > > > or > > > > > > > big > > > > > > > > > > > change > > > > > > > > > > > > to > > > > > > > > > > > > > master branch until this gets in so as to not > > > complicate > > > > > > > > > > > > merging/rebasing. > > > > > > > > > > > > > Once this is merged to the master branch, I would > > like > > > to > > > > > cut > > > > > > > 5.2 > > > > > > > > > > > branch > > > > > > > > > > > > > from master and we can move forward with 5.2.0 > > release. > > > > > > > > > > > > > > > > > > > > > > > > > > Please let me know if this looks good or if you > have > > > any > > > > > > other > > > > > > > > high > > > > > > > > > > > > > priority work for 5.2.0. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- *István Tóth* | Sr. Staff Software Engineer *Email*: st...@cloudera.com cloudera.com <https://www.cloudera.com> [image: Cloudera] <https://www.cloudera.com/> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera> ------------------------------ ------------------------------
