Re: [Discuss] Releasing Phoenix 4.16
The new scripts also automatically generate the CHANGES and README files. (Of course it's also possible to run those Yetus scripts as part of the make-rc release process) On Sat, Jan 16, 2021 at 6:38 AM Istvan Toth wrote: > Hi! > > I can see that Xinyi has filed PHOENX-6319 and PHOENIX-6320 for the "old" > make_rc script. > While of course every release manager is free to choose to the tools they > use, I ask you to consider using > the HBase-derived scripts at dev/create-release in the master branch. They > are not copied to 4.x, > because they operate directly on the ASF git repository, but they are > capable of handling a 4.x release, > > Using them has several advantages: > * They handle the profiles in a single step, meaning less manual work and > place for errors > * They handle more step of the release process automatically, again less > work and more consistency > > * I also intend to use them for 5.0, and as they use the HBase convention > for the files names and git tags, > using them would mean consistency for the 4.16 and 5.1 release artifacts. > > They are documented in the README file, and in the patch attached to > PHOENIX-6315. > There is a wall of text in the README about using the scripts with GKE, > but don't let that put you off, > that is completely optional. (I have never even tested that) > I am also happy to assist if you run into trouble with them. > > Istvan > > On Wed, Jan 13, 2021 at 4:12 AM Xinyi Yan wrote: > >> Thank you for the update and great work, Istvan. I feel the same way, this >> is the best approach so far for 4.16 and 5.1. >> >> On Tue, Jan 12, 2021 at 7:01 PM Josh Elser wrote: >> >> > IMO, a slow build is better than forcing users to run their own dev >> > build of a release. >> > >> > Very proud of all of the work y'all have been doing on 4.16. Keep >> > pushing on the release :) >> > >> > On 1/12/21 12:35 PM, Istvan Toth wrote: >> > > I've been working this, and came up with the following: >> > > >> > > * We no longer generate phoenix-client.jar and phoenix-server.jar, we >> > call >> > > them phoenix-client-hbase-X.Y.jar and phoenix-server-hbase-X.Y.jar >> > instead. >> > > * These file names are used in the binary assemblies, as well as as >> maven >> > > coordinates for consistency. >> > > * We generate four release binaries, for each HBase profile >> > > * We build and publish to maven the phoenix-client-hbase-X.Y and >> > > phoenix-server-hbase-X.Y artifacts for each profile (Actually, we >> deploy >> > > four times, but the rest of the artifacts are identical) >> > > >> > > * On the downside, running the release script on a Macbook takes >> several >> > > hours, as we build HBase 4 and Phoenix 8 times on the slow and kludgy >> Mac >> > > Docker filesystem. >> > > >> > > The almost finished patch to the build system is linked to >> > > https://issues.apache.org/jira/browse/PHOENIX-6307 >> > > >> > > 4.x would have the same build system changes once this is approved. >> > > >> > > Please check it out, and let me know if this solution is >> satisfactory, or >> > > if you have a better idea. >> > > >> > > regards >> > > Istvan >> > > >> > > >> > > On Fri, Jan 8, 2021 at 4:22 PM Istvan Toth >> wrote: >> > > >> > >> As I started to work on this I realized that while providing binary >> > >> tarballs for each HBase profile is fine, >> > >> this does not solve the maven artifact issue. >> > >> >> > >> Are we OK with publishing a single phoenix-client maven artifact (for >> > the >> > >> oldest HBase), >> > >> or do we want to publish a separate one for each HBase version ? >> > >> >> > >> I looked at publishing multiple client versions, but none of them are >> > >> particularly easy or attractive. >> > >> >> > >> The best I could come up with is adding a separate maven module for >> > (each >> > >> version x embedded) (i.e 6 for 4.x, 8 for master), >> > >> and activating them according to hbase.profile. >> > >> >> > >> This would also mean that we need to add the hbase version to the >> > artifact >> > >> id. i.e.: phoenix-client-hbase-2.1 >> > >> >> > >> Once we publish separate binaries for the HBase profiles, we can undo >> > the >> > >> change that excludes compat-module from phoenix-server, >> > >> and shade it in again for the binary assemblies. >> > >> >> > >> In this case we'd also have to do something about the phoenix-server >> > maven >> > >> artifacts. Either they get the same treatment as phoenix-client, >> > >> or we simply skip publishing them. I personally do not see anyone >> > getting >> > >> phoenix-server.jar from maven. >> > >> >> > >> The easiest version is >> > >> * publish the oldest profile phoenix-client to maven >> > >> * do not publish phoenix-server to maven >> > >> * add compat-module to phoenix-server for the binary artifact >> > >> >> > >> >> > >> regards >> > >> Istvan >> > >> >> > >> On Fri, Jan 8, 2021 at 6:56 AM Istvan Toth >> wrote: >> > >> >> > >>> On the release binaries: >> > >>> >> > >>> The current solution (which
Re: [Discuss] Releasing Phoenix 4.16
Hi! I can see that Xinyi has filed PHOENX-6319 and PHOENIX-6320 for the "old" make_rc script. While of course every release manager is free to choose to the tools they use, I ask you to consider using the HBase-derived scripts at dev/create-release in the master branch. They are not copied to 4.x, because they operate directly on the ASF git repository, but they are capable of handling a 4.x release, Using them has several advantages: * They handle the profiles in a single step, meaning less manual work and place for errors * They handle more step of the release process automatically, again less work and more consistency * I also intend to use them for 5.0, and as they use the HBase convention for the files names and git tags, using them would mean consistency for the 4.16 and 5.1 release artifacts. They are documented in the README file, and in the patch attached to PHOENIX-6315. There is a wall of text in the README about using the scripts with GKE, but don't let that put you off, that is completely optional. (I have never even tested that) I am also happy to assist if you run into trouble with them. Istvan On Wed, Jan 13, 2021 at 4:12 AM Xinyi Yan wrote: > Thank you for the update and great work, Istvan. I feel the same way, this > is the best approach so far for 4.16 and 5.1. > > On Tue, Jan 12, 2021 at 7:01 PM Josh Elser wrote: > > > IMO, a slow build is better than forcing users to run their own dev > > build of a release. > > > > Very proud of all of the work y'all have been doing on 4.16. Keep > > pushing on the release :) > > > > On 1/12/21 12:35 PM, Istvan Toth wrote: > > > I've been working this, and came up with the following: > > > > > > * We no longer generate phoenix-client.jar and phoenix-server.jar, we > > call > > > them phoenix-client-hbase-X.Y.jar and phoenix-server-hbase-X.Y.jar > > instead. > > > * These file names are used in the binary assemblies, as well as as > maven > > > coordinates for consistency. > > > * We generate four release binaries, for each HBase profile > > > * We build and publish to maven the phoenix-client-hbase-X.Y and > > > phoenix-server-hbase-X.Y artifacts for each profile (Actually, we > deploy > > > four times, but the rest of the artifacts are identical) > > > > > > * On the downside, running the release script on a Macbook takes > several > > > hours, as we build HBase 4 and Phoenix 8 times on the slow and kludgy > Mac > > > Docker filesystem. > > > > > > The almost finished patch to the build system is linked to > > > https://issues.apache.org/jira/browse/PHOENIX-6307 > > > > > > 4.x would have the same build system changes once this is approved. > > > > > > Please check it out, and let me know if this solution is satisfactory, > or > > > if you have a better idea. > > > > > > regards > > > Istvan > > > > > > > > > On Fri, Jan 8, 2021 at 4:22 PM Istvan Toth wrote: > > > > > >> As I started to work on this I realized that while providing binary > > >> tarballs for each HBase profile is fine, > > >> this does not solve the maven artifact issue. > > >> > > >> Are we OK with publishing a single phoenix-client maven artifact (for > > the > > >> oldest HBase), > > >> or do we want to publish a separate one for each HBase version ? > > >> > > >> I looked at publishing multiple client versions, but none of them are > > >> particularly easy or attractive. > > >> > > >> The best I could come up with is adding a separate maven module for > > (each > > >> version x embedded) (i.e 6 for 4.x, 8 for master), > > >> and activating them according to hbase.profile. > > >> > > >> This would also mean that we need to add the hbase version to the > > artifact > > >> id. i.e.: phoenix-client-hbase-2.1 > > >> > > >> Once we publish separate binaries for the HBase profiles, we can undo > > the > > >> change that excludes compat-module from phoenix-server, > > >> and shade it in again for the binary assemblies. > > >> > > >> In this case we'd also have to do something about the phoenix-server > > maven > > >> artifacts. Either they get the same treatment as phoenix-client, > > >> or we simply skip publishing them. I personally do not see anyone > > getting > > >> phoenix-server.jar from maven. > > >> > > >> The easiest version is > > >> * publish the oldest profile phoenix-client to maven > > >> * do not publish phoenix-server to maven > > >> * add compat-module to phoenix-server for the binary artifact > > >> > > >> > > >> regards > > >> Istvan > > >> > > >> On Fri, Jan 8, 2021 at 6:56 AM Istvan Toth > wrote: > > >> > > >>> On the release binaries: > > >>> > > >>> The current solution (which the default profile change has broken) > was > > >>> based on Lars's idea at > > >>> > > >>> > > > https://issues.apache.org/jira/browse/PHOENIX-5902?focusedCommentId=17125122&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17125122 > > >>> However, I agree that providing separate assemblies for each HBase > > >>> profile is better for our users, as
Re: [Discuss] Releasing Phoenix 4.16
Thank you for the update and great work, Istvan. I feel the same way, this is the best approach so far for 4.16 and 5.1. On Tue, Jan 12, 2021 at 7:01 PM Josh Elser wrote: > IMO, a slow build is better than forcing users to run their own dev > build of a release. > > Very proud of all of the work y'all have been doing on 4.16. Keep > pushing on the release :) > > On 1/12/21 12:35 PM, Istvan Toth wrote: > > I've been working this, and came up with the following: > > > > * We no longer generate phoenix-client.jar and phoenix-server.jar, we > call > > them phoenix-client-hbase-X.Y.jar and phoenix-server-hbase-X.Y.jar > instead. > > * These file names are used in the binary assemblies, as well as as maven > > coordinates for consistency. > > * We generate four release binaries, for each HBase profile > > * We build and publish to maven the phoenix-client-hbase-X.Y and > > phoenix-server-hbase-X.Y artifacts for each profile (Actually, we deploy > > four times, but the rest of the artifacts are identical) > > > > * On the downside, running the release script on a Macbook takes several > > hours, as we build HBase 4 and Phoenix 8 times on the slow and kludgy Mac > > Docker filesystem. > > > > The almost finished patch to the build system is linked to > > https://issues.apache.org/jira/browse/PHOENIX-6307 > > > > 4.x would have the same build system changes once this is approved. > > > > Please check it out, and let me know if this solution is satisfactory, or > > if you have a better idea. > > > > regards > > Istvan > > > > > > On Fri, Jan 8, 2021 at 4:22 PM Istvan Toth wrote: > > > >> As I started to work on this I realized that while providing binary > >> tarballs for each HBase profile is fine, > >> this does not solve the maven artifact issue. > >> > >> Are we OK with publishing a single phoenix-client maven artifact (for > the > >> oldest HBase), > >> or do we want to publish a separate one for each HBase version ? > >> > >> I looked at publishing multiple client versions, but none of them are > >> particularly easy or attractive. > >> > >> The best I could come up with is adding a separate maven module for > (each > >> version x embedded) (i.e 6 for 4.x, 8 for master), > >> and activating them according to hbase.profile. > >> > >> This would also mean that we need to add the hbase version to the > artifact > >> id. i.e.: phoenix-client-hbase-2.1 > >> > >> Once we publish separate binaries for the HBase profiles, we can undo > the > >> change that excludes compat-module from phoenix-server, > >> and shade it in again for the binary assemblies. > >> > >> In this case we'd also have to do something about the phoenix-server > maven > >> artifacts. Either they get the same treatment as phoenix-client, > >> or we simply skip publishing them. I personally do not see anyone > getting > >> phoenix-server.jar from maven. > >> > >> The easiest version is > >> * publish the oldest profile phoenix-client to maven > >> * do not publish phoenix-server to maven > >> * add compat-module to phoenix-server for the binary artifact > >> > >> > >> regards > >> Istvan > >> > >> On Fri, Jan 8, 2021 at 6:56 AM Istvan Toth wrote: > >> > >>> On the release binaries: > >>> > >>> The current solution (which the default profile change has broken) was > >>> based on Lars's idea at > >>> > >>> > https://issues.apache.org/jira/browse/PHOENIX-5902?focusedCommentId=17125122&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17125122 > >>> However, I agree that providing separate assemblies for each HBase > >>> profile is better for our users, as they won't have to rebuild Phoenix > to > >>> take advantage of any new features, and to get the general > improvements in > >>> later HBase releases. > >>> I have opened https://issues.apache.org/jira/browse/PHOENIX-6307 to > >>> track this. > >>> > >>> On the 5.1 release: > >>> > >>> Yes, I do want to release 5.1 shortly. In fact $dayjob desperately > needs > >>> it ASAP. > >>> I have a few older PRs waiting for review that fell between the cracks, > >>> but as soon as those are merged, I want to cut the first RC. > >>> It would be nice to have 4.16 and 5.1 as close as possible, and > >>> PHOENIX-6211 seems to be ready, so I hope to include it in 5.1 too. > >>> I will start an official 5.1 release thread and volunteer to be the > >>> release manager soon. (unless you want to take that up too, Xinyi). > >>> > >>> > >>> > >>> On Fri, Jan 8, 2021 at 1:08 AM Xinyi Yan wrote: > >>> > If we can modify the dev/create-release scripts and make them work for > the > 4.16 release with this hbase.profile option, it would make our life > much > easier to release multiple HBase profiles from the single branch in > the > future too(the master branch will have a release shortly right?). > Geoffrey > and Istvan, what do you think? > > Thanks, > Xinyi > > On Thu, Jan 7, 2021 at 11:28 AM Geoffrey Jacoby > wrote: >
Re: [Discuss] Releasing Phoenix 4.16
IMO, a slow build is better than forcing users to run their own dev build of a release. Very proud of all of the work y'all have been doing on 4.16. Keep pushing on the release :) On 1/12/21 12:35 PM, Istvan Toth wrote: I've been working this, and came up with the following: * We no longer generate phoenix-client.jar and phoenix-server.jar, we call them phoenix-client-hbase-X.Y.jar and phoenix-server-hbase-X.Y.jar instead. * These file names are used in the binary assemblies, as well as as maven coordinates for consistency. * We generate four release binaries, for each HBase profile * We build and publish to maven the phoenix-client-hbase-X.Y and phoenix-server-hbase-X.Y artifacts for each profile (Actually, we deploy four times, but the rest of the artifacts are identical) * On the downside, running the release script on a Macbook takes several hours, as we build HBase 4 and Phoenix 8 times on the slow and kludgy Mac Docker filesystem. The almost finished patch to the build system is linked to https://issues.apache.org/jira/browse/PHOENIX-6307 4.x would have the same build system changes once this is approved. Please check it out, and let me know if this solution is satisfactory, or if you have a better idea. regards Istvan On Fri, Jan 8, 2021 at 4:22 PM Istvan Toth wrote: As I started to work on this I realized that while providing binary tarballs for each HBase profile is fine, this does not solve the maven artifact issue. Are we OK with publishing a single phoenix-client maven artifact (for the oldest HBase), or do we want to publish a separate one for each HBase version ? I looked at publishing multiple client versions, but none of them are particularly easy or attractive. The best I could come up with is adding a separate maven module for (each version x embedded) (i.e 6 for 4.x, 8 for master), and activating them according to hbase.profile. This would also mean that we need to add the hbase version to the artifact id. i.e.: phoenix-client-hbase-2.1 Once we publish separate binaries for the HBase profiles, we can undo the change that excludes compat-module from phoenix-server, and shade it in again for the binary assemblies. In this case we'd also have to do something about the phoenix-server maven artifacts. Either they get the same treatment as phoenix-client, or we simply skip publishing them. I personally do not see anyone getting phoenix-server.jar from maven. The easiest version is * publish the oldest profile phoenix-client to maven * do not publish phoenix-server to maven * add compat-module to phoenix-server for the binary artifact regards Istvan On Fri, Jan 8, 2021 at 6:56 AM Istvan Toth wrote: On the release binaries: The current solution (which the default profile change has broken) was based on Lars's idea at https://issues.apache.org/jira/browse/PHOENIX-5902?focusedCommentId=17125122&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17125122 However, I agree that providing separate assemblies for each HBase profile is better for our users, as they won't have to rebuild Phoenix to take advantage of any new features, and to get the general improvements in later HBase releases. I have opened https://issues.apache.org/jira/browse/PHOENIX-6307 to track this. On the 5.1 release: Yes, I do want to release 5.1 shortly. In fact $dayjob desperately needs it ASAP. I have a few older PRs waiting for review that fell between the cracks, but as soon as those are merged, I want to cut the first RC. It would be nice to have 4.16 and 5.1 as close as possible, and PHOENIX-6211 seems to be ready, so I hope to include it in 5.1 too. I will start an official 5.1 release thread and volunteer to be the release manager soon. (unless you want to take that up too, Xinyi). On Fri, Jan 8, 2021 at 1:08 AM Xinyi Yan wrote: If we can modify the dev/create-release scripts and make them work for the 4.16 release with this hbase.profile option, it would make our life much easier to release multiple HBase profiles from the single branch in the future too(the master branch will have a release shortly right?). Geoffrey and Istvan, what do you think? Thanks, Xinyi On Thu, Jan 7, 2021 at 11:28 AM Geoffrey Jacoby wrote: Thanks for bringing up the default branch issue, Istvan, I've been meaning to start a conversation about it on this list. As part of PHOENIX-5435, I changed the default 4.x HBase release to 1.5 and the default 5.x HBase to 2.3 because the WAL annotation feature introduced by 5435 only works with HBase 1.5+ or 2.3+. (It depends on a coproc hook introduced in HBASE-22623). That means that all tests of that feature must no-op when run against an earlier HBase, which means that it would never be tested in our CI pipelines if the default was 1.3 or 2.1. This has come up quite a few times recently. In particular, the new indexing framework runs in a degraded state against HBase 2.1 and 2.2 (still better than the old indexing fr
Re: [Discuss] Releasing Phoenix 4.16
I've been working this, and came up with the following: * We no longer generate phoenix-client.jar and phoenix-server.jar, we call them phoenix-client-hbase-X.Y.jar and phoenix-server-hbase-X.Y.jar instead. * These file names are used in the binary assemblies, as well as as maven coordinates for consistency. * We generate four release binaries, for each HBase profile * We build and publish to maven the phoenix-client-hbase-X.Y and phoenix-server-hbase-X.Y artifacts for each profile (Actually, we deploy four times, but the rest of the artifacts are identical) * On the downside, running the release script on a Macbook takes several hours, as we build HBase 4 and Phoenix 8 times on the slow and kludgy Mac Docker filesystem. The almost finished patch to the build system is linked to https://issues.apache.org/jira/browse/PHOENIX-6307 4.x would have the same build system changes once this is approved. Please check it out, and let me know if this solution is satisfactory, or if you have a better idea. regards Istvan On Fri, Jan 8, 2021 at 4:22 PM Istvan Toth wrote: > As I started to work on this I realized that while providing binary > tarballs for each HBase profile is fine, > this does not solve the maven artifact issue. > > Are we OK with publishing a single phoenix-client maven artifact (for the > oldest HBase), > or do we want to publish a separate one for each HBase version ? > > I looked at publishing multiple client versions, but none of them are > particularly easy or attractive. > > The best I could come up with is adding a separate maven module for (each > version x embedded) (i.e 6 for 4.x, 8 for master), > and activating them according to hbase.profile. > > This would also mean that we need to add the hbase version to the artifact > id. i.e.: phoenix-client-hbase-2.1 > > Once we publish separate binaries for the HBase profiles, we can undo the > change that excludes compat-module from phoenix-server, > and shade it in again for the binary assemblies. > > In this case we'd also have to do something about the phoenix-server maven > artifacts. Either they get the same treatment as phoenix-client, > or we simply skip publishing them. I personally do not see anyone getting > phoenix-server.jar from maven. > > The easiest version is > * publish the oldest profile phoenix-client to maven > * do not publish phoenix-server to maven > * add compat-module to phoenix-server for the binary artifact > > > regards > Istvan > > On Fri, Jan 8, 2021 at 6:56 AM Istvan Toth wrote: > >> On the release binaries: >> >> The current solution (which the default profile change has broken) was >> based on Lars's idea at >> >> https://issues.apache.org/jira/browse/PHOENIX-5902?focusedCommentId=17125122&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17125122 >> However, I agree that providing separate assemblies for each HBase >> profile is better for our users, as they won't have to rebuild Phoenix to >> take advantage of any new features, and to get the general improvements in >> later HBase releases. >> I have opened https://issues.apache.org/jira/browse/PHOENIX-6307 to >> track this. >> >> On the 5.1 release: >> >> Yes, I do want to release 5.1 shortly. In fact $dayjob desperately needs >> it ASAP. >> I have a few older PRs waiting for review that fell between the cracks, >> but as soon as those are merged, I want to cut the first RC. >> It would be nice to have 4.16 and 5.1 as close as possible, and >> PHOENIX-6211 seems to be ready, so I hope to include it in 5.1 too. >> I will start an official 5.1 release thread and volunteer to be the >> release manager soon. (unless you want to take that up too, Xinyi). >> >> >> >> On Fri, Jan 8, 2021 at 1:08 AM Xinyi Yan wrote: >> >>> If we can modify the dev/create-release scripts and make them work for >>> the >>> 4.16 release with this hbase.profile option, it would make our life much >>> easier to release multiple HBase profiles from the single branch in the >>> future too(the master branch will have a release shortly right?). >>> Geoffrey >>> and Istvan, what do you think? >>> >>> Thanks, >>> Xinyi >>> >>> On Thu, Jan 7, 2021 at 11:28 AM Geoffrey Jacoby >>> wrote: >>> >>> > Thanks for bringing up the default branch issue, Istvan, I've been >>> meaning >>> > to start a conversation about it on this list. >>> > >>> > As part of PHOENIX-5435, I changed the default 4.x HBase release to >>> 1.5 and >>> > the default 5.x HBase to 2.3 because the WAL annotation feature >>> introduced >>> > by 5435 only works with HBase 1.5+ or 2.3+. (It depends on a coproc >>> hook >>> > introduced in HBASE-22623). That means that all tests of that feature >>> must >>> > no-op when run against an earlier HBase, which means that it would >>> never be >>> > tested in our CI pipelines if the default was 1.3 or 2.1. >>> > >>> > This has come up quite a few times recently. In particular, the new >>> > indexing framework runs in a degraded state against HBase 2.1 a
Re: [Discuss] Releasing Phoenix 4.16
As I started to work on this I realized that while providing binary tarballs for each HBase profile is fine, this does not solve the maven artifact issue. Are we OK with publishing a single phoenix-client maven artifact (for the oldest HBase), or do we want to publish a separate one for each HBase version ? I looked at publishing multiple client versions, but none of them are particularly easy or attractive. The best I could come up with is adding a separate maven module for (each version x embedded) (i.e 6 for 4.x, 8 for master), and activating them according to hbase.profile. This would also mean that we need to add the hbase version to the artifact id. i.e.: phoenix-client-hbase-2.1 Once we publish separate binaries for the HBase profiles, we can undo the change that excludes compat-module from phoenix-server, and shade it in again for the binary assemblies. In this case we'd also have to do something about the phoenix-server maven artifacts. Either they get the same treatment as phoenix-client, or we simply skip publishing them. I personally do not see anyone getting phoenix-server.jar from maven. The easiest version is * publish the oldest profile phoenix-client to maven * do not publish phoenix-server to maven * add compat-module to phoenix-server for the binary artifact regards Istvan On Fri, Jan 8, 2021 at 6:56 AM Istvan Toth wrote: > On the release binaries: > > The current solution (which the default profile change has broken) was > based on Lars's idea at > > https://issues.apache.org/jira/browse/PHOENIX-5902?focusedCommentId=17125122&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17125122 > However, I agree that providing separate assemblies for each HBase profile > is better for our users, as they won't have to rebuild Phoenix to take > advantage of any new features, and to get the general improvements in > later HBase releases. > I have opened https://issues.apache.org/jira/browse/PHOENIX-6307 to track > this. > > On the 5.1 release: > > Yes, I do want to release 5.1 shortly. In fact $dayjob desperately needs > it ASAP. > I have a few older PRs waiting for review that fell between the cracks, > but as soon as those are merged, I want to cut the first RC. > It would be nice to have 4.16 and 5.1 as close as possible, and > PHOENIX-6211 seems to be ready, so I hope to include it in 5.1 too. > I will start an official 5.1 release thread and volunteer to be the > release manager soon. (unless you want to take that up too, Xinyi). > > > > On Fri, Jan 8, 2021 at 1:08 AM Xinyi Yan wrote: > >> If we can modify the dev/create-release scripts and make them work for the >> 4.16 release with this hbase.profile option, it would make our life much >> easier to release multiple HBase profiles from the single branch in the >> future too(the master branch will have a release shortly right?). Geoffrey >> and Istvan, what do you think? >> >> Thanks, >> Xinyi >> >> On Thu, Jan 7, 2021 at 11:28 AM Geoffrey Jacoby >> wrote: >> >> > Thanks for bringing up the default branch issue, Istvan, I've been >> meaning >> > to start a conversation about it on this list. >> > >> > As part of PHOENIX-5435, I changed the default 4.x HBase release to 1.5 >> and >> > the default 5.x HBase to 2.3 because the WAL annotation feature >> introduced >> > by 5435 only works with HBase 1.5+ or 2.3+. (It depends on a coproc hook >> > introduced in HBASE-22623). That means that all tests of that feature >> must >> > no-op when run against an earlier HBase, which means that it would >> never be >> > tested in our CI pipelines if the default was 1.3 or 2.1. >> > >> > This has come up quite a few times recently. In particular, the new >> > indexing framework runs in a degraded state against HBase 2.1 and 2.2 >> > (still better than the old indexing framework though!), because we lack >> > support in 2.1 for Filters on raw scans and can't use the "max lookback >> > age" feature with HBase 2.1 or 2.2, which keeps major compactions from >> > messing with index rebuilds. That's why lots of indexing tests no-op or >> > have to make different assertions when run against 2.1 or 2.2, so we >> only >> > properly test the indexing framework if CI and local developer >> > tests run against HBase 2.3 or up. >> > >> > As for the release, don't we usually release separate binaries that are >> > suitable for all the HBase versions we support? Back before we had a >> > unified 4.x branch, I believe we used to release from 3 different >> > 4.x-HBase-* branches each time. I've been assuming that part of the >> release >> > process would be generating binaries for 4.x-HBase-1.3, 4.x-HBase-1.4, >> > 4.x-HBase-1.5, and 4.x-HBase-1.6 (if that was different from 1.5). >> > >> > Geoffrey >> > >> > On Thu, Jan 7, 2021 at 11:19 AM Istvan Toth wrote: >> > >> > > Two more issues came to my mind: >> > > >> > > In PHOENIX-5435 the default profile was changed to HBase 1.5.0 >> > > This causes two conflicts: >> > > - Our docume
Re: [Discuss] Releasing Phoenix 4.16
On the release binaries: The current solution (which the default profile change has broken) was based on Lars's idea at https://issues.apache.org/jira/browse/PHOENIX-5902?focusedCommentId=17125122&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17125122 However, I agree that providing separate assemblies for each HBase profile is better for our users, as they won't have to rebuild Phoenix to take advantage of any new features, and to get the general improvements in later HBase releases. I have opened https://issues.apache.org/jira/browse/PHOENIX-6307 to track this. On the 5.1 release: Yes, I do want to release 5.1 shortly. In fact $dayjob desperately needs it ASAP. I have a few older PRs waiting for review that fell between the cracks, but as soon as those are merged, I want to cut the first RC. It would be nice to have 4.16 and 5.1 as close as possible, and PHOENIX-6211 seems to be ready, so I hope to include it in 5.1 too. I will start an official 5.1 release thread and volunteer to be the release manager soon. (unless you want to take that up too, Xinyi). On Fri, Jan 8, 2021 at 1:08 AM Xinyi Yan wrote: > If we can modify the dev/create-release scripts and make them work for the > 4.16 release with this hbase.profile option, it would make our life much > easier to release multiple HBase profiles from the single branch in the > future too(the master branch will have a release shortly right?). Geoffrey > and Istvan, what do you think? > > Thanks, > Xinyi > > On Thu, Jan 7, 2021 at 11:28 AM Geoffrey Jacoby > wrote: > > > Thanks for bringing up the default branch issue, Istvan, I've been > meaning > > to start a conversation about it on this list. > > > > As part of PHOENIX-5435, I changed the default 4.x HBase release to 1.5 > and > > the default 5.x HBase to 2.3 because the WAL annotation feature > introduced > > by 5435 only works with HBase 1.5+ or 2.3+. (It depends on a coproc hook > > introduced in HBASE-22623). That means that all tests of that feature > must > > no-op when run against an earlier HBase, which means that it would never > be > > tested in our CI pipelines if the default was 1.3 or 2.1. > > > > This has come up quite a few times recently. In particular, the new > > indexing framework runs in a degraded state against HBase 2.1 and 2.2 > > (still better than the old indexing framework though!), because we lack > > support in 2.1 for Filters on raw scans and can't use the "max lookback > > age" feature with HBase 2.1 or 2.2, which keeps major compactions from > > messing with index rebuilds. That's why lots of indexing tests no-op or > > have to make different assertions when run against 2.1 or 2.2, so we only > > properly test the indexing framework if CI and local developer > > tests run against HBase 2.3 or up. > > > > As for the release, don't we usually release separate binaries that are > > suitable for all the HBase versions we support? Back before we had a > > unified 4.x branch, I believe we used to release from 3 different > > 4.x-HBase-* branches each time. I've been assuming that part of the > release > > process would be generating binaries for 4.x-HBase-1.3, 4.x-HBase-1.4, > > 4.x-HBase-1.5, and 4.x-HBase-1.6 (if that was different from 1.5). > > > > Geoffrey > > > > On Thu, Jan 7, 2021 at 11:19 AM Istvan Toth wrote: > > > > > Two more issues came to my mind: > > > > > > In PHOENIX-5435 the default profile was changed to HBase 1.5.0 > > > This causes two conflicts: > > > - Our documentation says that the default profile is the lowest > > supported, > > > which is no longer true > > > - Unless we change our binary release process, the published maven > > > artifacts and binaries will also be built with HBase 1.5, thus they > will > > be > > > incompatible with Hbase 1.3 and 1.4 > > > > > > This should be addressed before release. > > > One possible solution: > > > * Add an "oldest" hbase.profile option which always chooses the oldest > > > supported version > > > * Update the release scripts to specify this profile > > > * Update the docs. > > > > > > I have also adopted the hbase release scripts to Phoenix, they are > > present > > > in dev/create-release in the master branch, but > > > should be able to perform 4.16 release as well. I have used this for > the > > > phoenix-thirdparty, omid, and tephra releases, but no live phoenix > > releases > > > have been done with them yet, obviously. > > > > > > regards > > > Istvan > > > > > > On Thu, Jan 7, 2021 at 8:19 AM Istvan Toth wrote: > > > > > > > Hi! > > > > > > > > A question of process: Should we backport fixes to the 4.16 branch at > > our > > > > discretion, or is backporting those handled by the release manager > > > (Xinyi) ? > > > > > > > > On Thu, Jan 7, 2021 at 5:42 AM Istvan Toth > wrote: > > > > > > > >> Hi! > > > >> > > > >> Thanks for everyone's effort on fixing the flakey tests. > > > >> However, our ASF Jenkins runs are still very unstable, and I am > afraid > > > >> t
Re: [Discuss] Releasing Phoenix 4.16
If we can modify the dev/create-release scripts and make them work for the 4.16 release with this hbase.profile option, it would make our life much easier to release multiple HBase profiles from the single branch in the future too(the master branch will have a release shortly right?). Geoffrey and Istvan, what do you think? Thanks, Xinyi On Thu, Jan 7, 2021 at 11:28 AM Geoffrey Jacoby wrote: > Thanks for bringing up the default branch issue, Istvan, I've been meaning > to start a conversation about it on this list. > > As part of PHOENIX-5435, I changed the default 4.x HBase release to 1.5 and > the default 5.x HBase to 2.3 because the WAL annotation feature introduced > by 5435 only works with HBase 1.5+ or 2.3+. (It depends on a coproc hook > introduced in HBASE-22623). That means that all tests of that feature must > no-op when run against an earlier HBase, which means that it would never be > tested in our CI pipelines if the default was 1.3 or 2.1. > > This has come up quite a few times recently. In particular, the new > indexing framework runs in a degraded state against HBase 2.1 and 2.2 > (still better than the old indexing framework though!), because we lack > support in 2.1 for Filters on raw scans and can't use the "max lookback > age" feature with HBase 2.1 or 2.2, which keeps major compactions from > messing with index rebuilds. That's why lots of indexing tests no-op or > have to make different assertions when run against 2.1 or 2.2, so we only > properly test the indexing framework if CI and local developer > tests run against HBase 2.3 or up. > > As for the release, don't we usually release separate binaries that are > suitable for all the HBase versions we support? Back before we had a > unified 4.x branch, I believe we used to release from 3 different > 4.x-HBase-* branches each time. I've been assuming that part of the release > process would be generating binaries for 4.x-HBase-1.3, 4.x-HBase-1.4, > 4.x-HBase-1.5, and 4.x-HBase-1.6 (if that was different from 1.5). > > Geoffrey > > On Thu, Jan 7, 2021 at 11:19 AM Istvan Toth wrote: > > > Two more issues came to my mind: > > > > In PHOENIX-5435 the default profile was changed to HBase 1.5.0 > > This causes two conflicts: > > - Our documentation says that the default profile is the lowest > supported, > > which is no longer true > > - Unless we change our binary release process, the published maven > > artifacts and binaries will also be built with HBase 1.5, thus they will > be > > incompatible with Hbase 1.3 and 1.4 > > > > This should be addressed before release. > > One possible solution: > > * Add an "oldest" hbase.profile option which always chooses the oldest > > supported version > > * Update the release scripts to specify this profile > > * Update the docs. > > > > I have also adopted the hbase release scripts to Phoenix, they are > present > > in dev/create-release in the master branch, but > > should be able to perform 4.16 release as well. I have used this for the > > phoenix-thirdparty, omid, and tephra releases, but no live phoenix > releases > > have been done with them yet, obviously. > > > > regards > > Istvan > > > > On Thu, Jan 7, 2021 at 8:19 AM Istvan Toth wrote: > > > > > Hi! > > > > > > A question of process: Should we backport fixes to the 4.16 branch at > our > > > discretion, or is backporting those handled by the release manager > > (Xinyi) ? > > > > > > On Thu, Jan 7, 2021 at 5:42 AM Istvan Toth wrote: > > > > > >> Hi! > > >> > > >> Thanks for everyone's effort on fixing the flakey tests. > > >> However, our ASF Jenkins runs are still very unstable, and I am afraid > > >> that the single clean 4.x run was down more to luck than to fixing the > > >> underlying problem. > > >> While I do not consider this a blocker, fixing this would make the > pre- > > >> and post commit tests far more useful, and let us start with a clean > > slate > > >> for the next release. > > >> I have created https://issues.apache.org/jira/browse/PHOENIX-6288 to > > >> track the generic cluster setup issue, but my attempts to fix this, or > > at > > >> least figure out the root cause have been unsuccessful so far. > > >> > > >> I ask for your help in fixing this issue. I have added some > inconclusive > > >> analysis to the ticket, and the full failsafe output directory is > > >> downloadable as artifacts from the failed multibranch runs (stored > for a > > >> few days), which should hold the answer to this riddle. > > >> > > >> regards, > > >> Istvan > > >> > > >> On Thu, Jan 7, 2021 at 4:58 AM Xinyi Yan wrote: > > >> > > >>> Hi, > > >>> > > >>> We finally have a stable 4.x branch build after many flapper test > > fixing > > >>> contributions from the community. I'm going to fork a new > branch(4.16) > > >>> from > > >>> the current 4.x branch for the 4.16 release. As a cutoff, I will not > > >>> include any new features other than PHOENIX-6211 and bug fix. Please > > let > > >>> me > > >>> know if you have any questions or conce
Re: [Discuss] Releasing Phoenix 4.16
Thanks for bringing up the default branch issue, Istvan, I've been meaning to start a conversation about it on this list. As part of PHOENIX-5435, I changed the default 4.x HBase release to 1.5 and the default 5.x HBase to 2.3 because the WAL annotation feature introduced by 5435 only works with HBase 1.5+ or 2.3+. (It depends on a coproc hook introduced in HBASE-22623). That means that all tests of that feature must no-op when run against an earlier HBase, which means that it would never be tested in our CI pipelines if the default was 1.3 or 2.1. This has come up quite a few times recently. In particular, the new indexing framework runs in a degraded state against HBase 2.1 and 2.2 (still better than the old indexing framework though!), because we lack support in 2.1 for Filters on raw scans and can't use the "max lookback age" feature with HBase 2.1 or 2.2, which keeps major compactions from messing with index rebuilds. That's why lots of indexing tests no-op or have to make different assertions when run against 2.1 or 2.2, so we only properly test the indexing framework if CI and local developer tests run against HBase 2.3 or up. As for the release, don't we usually release separate binaries that are suitable for all the HBase versions we support? Back before we had a unified 4.x branch, I believe we used to release from 3 different 4.x-HBase-* branches each time. I've been assuming that part of the release process would be generating binaries for 4.x-HBase-1.3, 4.x-HBase-1.4, 4.x-HBase-1.5, and 4.x-HBase-1.6 (if that was different from 1.5). Geoffrey On Thu, Jan 7, 2021 at 11:19 AM Istvan Toth wrote: > Two more issues came to my mind: > > In PHOENIX-5435 the default profile was changed to HBase 1.5.0 > This causes two conflicts: > - Our documentation says that the default profile is the lowest supported, > which is no longer true > - Unless we change our binary release process, the published maven > artifacts and binaries will also be built with HBase 1.5, thus they will be > incompatible with Hbase 1.3 and 1.4 > > This should be addressed before release. > One possible solution: > * Add an "oldest" hbase.profile option which always chooses the oldest > supported version > * Update the release scripts to specify this profile > * Update the docs. > > I have also adopted the hbase release scripts to Phoenix, they are present > in dev/create-release in the master branch, but > should be able to perform 4.16 release as well. I have used this for the > phoenix-thirdparty, omid, and tephra releases, but no live phoenix releases > have been done with them yet, obviously. > > regards > Istvan > > On Thu, Jan 7, 2021 at 8:19 AM Istvan Toth wrote: > > > Hi! > > > > A question of process: Should we backport fixes to the 4.16 branch at our > > discretion, or is backporting those handled by the release manager > (Xinyi) ? > > > > On Thu, Jan 7, 2021 at 5:42 AM Istvan Toth wrote: > > > >> Hi! > >> > >> Thanks for everyone's effort on fixing the flakey tests. > >> However, our ASF Jenkins runs are still very unstable, and I am afraid > >> that the single clean 4.x run was down more to luck than to fixing the > >> underlying problem. > >> While I do not consider this a blocker, fixing this would make the pre- > >> and post commit tests far more useful, and let us start with a clean > slate > >> for the next release. > >> I have created https://issues.apache.org/jira/browse/PHOENIX-6288 to > >> track the generic cluster setup issue, but my attempts to fix this, or > at > >> least figure out the root cause have been unsuccessful so far. > >> > >> I ask for your help in fixing this issue. I have added some inconclusive > >> analysis to the ticket, and the full failsafe output directory is > >> downloadable as artifacts from the failed multibranch runs (stored for a > >> few days), which should hold the answer to this riddle. > >> > >> regards, > >> Istvan > >> > >> On Thu, Jan 7, 2021 at 4:58 AM Xinyi Yan wrote: > >> > >>> Hi, > >>> > >>> We finally have a stable 4.x branch build after many flapper test > fixing > >>> contributions from the community. I'm going to fork a new branch(4.16) > >>> from > >>> the current 4.x branch for the 4.16 release. As a cutoff, I will not > >>> include any new features other than PHOENIX-6211 and bug fix. Please > let > >>> me > >>> know if you have any questions or concerns. > >>> > >>> Thanks, > >>> Xinyi > >>> > >>> On Wed, Dec 23, 2020 at 6:07 AM Viraj Jasani > wrote: > >>> > >>> > An update on test stability: > >>> > > >>> > As per recent 4.x build results, we are left with very few flappers > and > >>> > specifically > >>> > with HBase profile 1.6, I can see recent 7 build (multibranch + PR > >>> > precommit results) > >>> > results without any test failure. > >>> > > >>> > > >>> > On Tue, Dec 15, 2020 at 5:52 AM Xinyi Yan > wrote: > >>> > > >>> > > Yes, we are currently working on fixing the 4.x branch test > flappers > >>> > while > >>> > > waiting for the PHOENIX-54
Re: [Discuss] Releasing Phoenix 4.16
Two more issues came to my mind: In PHOENIX-5435 the default profile was changed to HBase 1.5.0 This causes two conflicts: - Our documentation says that the default profile is the lowest supported, which is no longer true - Unless we change our binary release process, the published maven artifacts and binaries will also be built with HBase 1.5, thus they will be incompatible with Hbase 1.3 and 1.4 This should be addressed before release. One possible solution: * Add an "oldest" hbase.profile option which always chooses the oldest supported version * Update the release scripts to specify this profile * Update the docs. I have also adopted the hbase release scripts to Phoenix, they are present in dev/create-release in the master branch, but should be able to perform 4.16 release as well. I have used this for the phoenix-thirdparty, omid, and tephra releases, but no live phoenix releases have been done with them yet, obviously. regards Istvan On Thu, Jan 7, 2021 at 8:19 AM Istvan Toth wrote: > Hi! > > A question of process: Should we backport fixes to the 4.16 branch at our > discretion, or is backporting those handled by the release manager (Xinyi) ? > > On Thu, Jan 7, 2021 at 5:42 AM Istvan Toth wrote: > >> Hi! >> >> Thanks for everyone's effort on fixing the flakey tests. >> However, our ASF Jenkins runs are still very unstable, and I am afraid >> that the single clean 4.x run was down more to luck than to fixing the >> underlying problem. >> While I do not consider this a blocker, fixing this would make the pre- >> and post commit tests far more useful, and let us start with a clean slate >> for the next release. >> I have created https://issues.apache.org/jira/browse/PHOENIX-6288 to >> track the generic cluster setup issue, but my attempts to fix this, or at >> least figure out the root cause have been unsuccessful so far. >> >> I ask for your help in fixing this issue. I have added some inconclusive >> analysis to the ticket, and the full failsafe output directory is >> downloadable as artifacts from the failed multibranch runs (stored for a >> few days), which should hold the answer to this riddle. >> >> regards, >> Istvan >> >> On Thu, Jan 7, 2021 at 4:58 AM Xinyi Yan wrote: >> >>> Hi, >>> >>> We finally have a stable 4.x branch build after many flapper test fixing >>> contributions from the community. I'm going to fork a new branch(4.16) >>> from >>> the current 4.x branch for the 4.16 release. As a cutoff, I will not >>> include any new features other than PHOENIX-6211 and bug fix. Please let >>> me >>> know if you have any questions or concerns. >>> >>> Thanks, >>> Xinyi >>> >>> On Wed, Dec 23, 2020 at 6:07 AM Viraj Jasani wrote: >>> >>> > An update on test stability: >>> > >>> > As per recent 4.x build results, we are left with very few flappers and >>> > specifically >>> > with HBase profile 1.6, I can see recent 7 build (multibranch + PR >>> > precommit results) >>> > results without any test failure. >>> > >>> > >>> > On Tue, Dec 15, 2020 at 5:52 AM Xinyi Yan wrote: >>> > >>> > > Yes, we are currently working on fixing the 4.x branch test flappers >>> > while >>> > > waiting for the PHOENIX-5435. After that, I will try to have an RC >>> ASAP. >>> > > >>> > > On Sun, Dec 13, 2020 at 3:29 PM Ankit Singhal < >>> ankitsingha...@gmail.com> >>> > > wrote: >>> > > >>> > > > I see that both the blockers listed here PHOENIX-5712 and >>> PHOENIX-6241 >>> > > > have been resolved(Thanks to Xinyi), and also as per the JIRA query >>> > there >>> > > > is no Jira marked as a blocker for 4.16 except the one related to >>> > > > documentation >>> > > > for "splittable catalog table". >>> > > > >>> > > > Xinyi, so are we good to start the release process now? >>> > > > >>> > > > On Wed, Dec 2, 2020 at 9:32 PM Xinyi Yan >>> wrote: >>> > > > >>> > > > > Thanks for replying and providing suggestions. I looked at the >>> wrong >>> > > > result >>> > > > > Jira list that Daniel provided and did some local testing, and >>> here >>> > is >>> > > > the >>> > > > > result: >>> > > > > [resolved] PHOENIX-4116, PHOENIX-4419, and PHOENIX-4642: cannot >>> > > reproduce >>> > > > > it. >>> > > > > [More information is required] PHOENIX-4504 cannot reproduce it >>> but >>> > > > someone >>> > > > > claimed that he had a similar issue. >>> > > > > [unusual query] PHOENIX-4540 and PHOENIX-6217 >>> > > > > >>> > > > > Based on my finding, I think it's better to have more frequent >>> > > > housekeeping >>> > > > > and resolve unreproducible bugs especially since many of them are >>> > > > > considering out of date (phoenix-4.11 or even phoenix-4.6). >>> Since I >>> > > still >>> > > > > need time to work on the blocker Jira(PHOENIX-5712, >>> PHOENIX-6241) and >>> > > fix >>> > > > > test flappers, if you want to fix "unusual query" bugs, feel >>> free to >>> > do >>> > > > so. >>> > > > > >>> > > > > >>> > > > > Sincerely, >>> > > > > Xinyi >>> > > > > >>> > > > > On Wed, Dec 2, 2020 at 12:41 AM Ankit Singhal >>> >
Re: [Discuss] Releasing Phoenix 4.16
Hi! A question of process: Should we backport fixes to the 4.16 branch at our discretion, or is backporting those handled by the release manager (Xinyi) ? On Thu, Jan 7, 2021 at 5:42 AM Istvan Toth wrote: > Hi! > > Thanks for everyone's effort on fixing the flakey tests. > However, our ASF Jenkins runs are still very unstable, and I am afraid > that the single clean 4.x run was down more to luck than to fixing the > underlying problem. > While I do not consider this a blocker, fixing this would make the pre- > and post commit tests far more useful, and let us start with a clean slate > for the next release. > I have created https://issues.apache.org/jira/browse/PHOENIX-6288 to > track the generic cluster setup issue, but my attempts to fix this, or at > least figure out the root cause have been unsuccessful so far. > > I ask for your help in fixing this issue. I have added some inconclusive > analysis to the ticket, and the full failsafe output directory is > downloadable as artifacts from the failed multibranch runs (stored for a > few days), which should hold the answer to this riddle. > > regards, > Istvan > > On Thu, Jan 7, 2021 at 4:58 AM Xinyi Yan wrote: > >> Hi, >> >> We finally have a stable 4.x branch build after many flapper test fixing >> contributions from the community. I'm going to fork a new branch(4.16) >> from >> the current 4.x branch for the 4.16 release. As a cutoff, I will not >> include any new features other than PHOENIX-6211 and bug fix. Please let >> me >> know if you have any questions or concerns. >> >> Thanks, >> Xinyi >> >> On Wed, Dec 23, 2020 at 6:07 AM Viraj Jasani wrote: >> >> > An update on test stability: >> > >> > As per recent 4.x build results, we are left with very few flappers and >> > specifically >> > with HBase profile 1.6, I can see recent 7 build (multibranch + PR >> > precommit results) >> > results without any test failure. >> > >> > >> > On Tue, Dec 15, 2020 at 5:52 AM Xinyi Yan wrote: >> > >> > > Yes, we are currently working on fixing the 4.x branch test flappers >> > while >> > > waiting for the PHOENIX-5435. After that, I will try to have an RC >> ASAP. >> > > >> > > On Sun, Dec 13, 2020 at 3:29 PM Ankit Singhal < >> ankitsingha...@gmail.com> >> > > wrote: >> > > >> > > > I see that both the blockers listed here PHOENIX-5712 and >> PHOENIX-6241 >> > > > have been resolved(Thanks to Xinyi), and also as per the JIRA query >> > there >> > > > is no Jira marked as a blocker for 4.16 except the one related to >> > > > documentation >> > > > for "splittable catalog table". >> > > > >> > > > Xinyi, so are we good to start the release process now? >> > > > >> > > > On Wed, Dec 2, 2020 at 9:32 PM Xinyi Yan >> wrote: >> > > > >> > > > > Thanks for replying and providing suggestions. I looked at the >> wrong >> > > > result >> > > > > Jira list that Daniel provided and did some local testing, and >> here >> > is >> > > > the >> > > > > result: >> > > > > [resolved] PHOENIX-4116, PHOENIX-4419, and PHOENIX-4642: cannot >> > > reproduce >> > > > > it. >> > > > > [More information is required] PHOENIX-4504 cannot reproduce it >> but >> > > > someone >> > > > > claimed that he had a similar issue. >> > > > > [unusual query] PHOENIX-4540 and PHOENIX-6217 >> > > > > >> > > > > Based on my finding, I think it's better to have more frequent >> > > > housekeeping >> > > > > and resolve unreproducible bugs especially since many of them are >> > > > > considering out of date (phoenix-4.11 or even phoenix-4.6). Since >> I >> > > still >> > > > > need time to work on the blocker Jira(PHOENIX-5712, PHOENIX-6241) >> and >> > > fix >> > > > > test flappers, if you want to fix "unusual query" bugs, feel free >> to >> > do >> > > > so. >> > > > > >> > > > > >> > > > > Sincerely, >> > > > > Xinyi >> > > > > >> > > > > On Wed, Dec 2, 2020 at 12:41 AM Ankit Singhal >> > > wrote: >> > > > > >> > > > > > Thanks Daniel and appreciate the effort you put in getting the >> list >> > > > ready >> > > > > > for bugs producing wrong results >> > > > > > but none of them seems to be a blocker to me for 4.16 as they >> are >> > not >> > > > the >> > > > > > regression and doesn't break the general functionality >> > > > > > except for specific features, RVC/desc as Chenglei also pointed >> out >> > > > > (though >> > > > > > I'll defer the assessment to RM "Xinyi"). >> > > > > > Probably these can be a part of 4.16.1 or we can do 4.17.0 soon >> > maybe >> > > > > after >> > > > > > a few weeks/month? >> > > > > > >> > > > > > Considering that we have already fixed 137 bugs and done 85+ >> > > > > > improvements/features in 4.16, >> > > > > > it will not be a good idea to deprive the user from such fixes. >> > > > > > It's been a year since our last 4.15 release, having no release >> > > brings >> > > > > more >> > > > > > questions on the project >> > > > > > rather than the bugs which affect a certain % of feature/users, >> > would >> > > > the >> > > > > > release notes >> > > > > > expl
Re: [Discuss] Releasing Phoenix 4.16
Hi! Thanks for everyone's effort on fixing the flakey tests. However, our ASF Jenkins runs are still very unstable, and I am afraid that the single clean 4.x run was down more to luck than to fixing the underlying problem. While I do not consider this a blocker, fixing this would make the pre- and post commit tests far more useful, and let us start with a clean slate for the next release. I have created https://issues.apache.org/jira/browse/PHOENIX-6288 to track the generic cluster setup issue, but my attempts to fix this, or at least figure out the root cause have been unsuccessful so far. I ask for your help in fixing this issue. I have added some inconclusive analysis to the ticket, and the full failsafe output directory is downloadable as artifacts from the failed multibranch runs (stored for a few days), which should hold the answer to this riddle. regards, Istvan On Thu, Jan 7, 2021 at 4:58 AM Xinyi Yan wrote: > Hi, > > We finally have a stable 4.x branch build after many flapper test fixing > contributions from the community. I'm going to fork a new branch(4.16) from > the current 4.x branch for the 4.16 release. As a cutoff, I will not > include any new features other than PHOENIX-6211 and bug fix. Please let me > know if you have any questions or concerns. > > Thanks, > Xinyi > > On Wed, Dec 23, 2020 at 6:07 AM Viraj Jasani wrote: > > > An update on test stability: > > > > As per recent 4.x build results, we are left with very few flappers and > > specifically > > with HBase profile 1.6, I can see recent 7 build (multibranch + PR > > precommit results) > > results without any test failure. > > > > > > On Tue, Dec 15, 2020 at 5:52 AM Xinyi Yan wrote: > > > > > Yes, we are currently working on fixing the 4.x branch test flappers > > while > > > waiting for the PHOENIX-5435. After that, I will try to have an RC > ASAP. > > > > > > On Sun, Dec 13, 2020 at 3:29 PM Ankit Singhal < > ankitsingha...@gmail.com> > > > wrote: > > > > > > > I see that both the blockers listed here PHOENIX-5712 and > PHOENIX-6241 > > > > have been resolved(Thanks to Xinyi), and also as per the JIRA query > > there > > > > is no Jira marked as a blocker for 4.16 except the one related to > > > > documentation > > > > for "splittable catalog table". > > > > > > > > Xinyi, so are we good to start the release process now? > > > > > > > > On Wed, Dec 2, 2020 at 9:32 PM Xinyi Yan > wrote: > > > > > > > > > Thanks for replying and providing suggestions. I looked at the > wrong > > > > result > > > > > Jira list that Daniel provided and did some local testing, and here > > is > > > > the > > > > > result: > > > > > [resolved] PHOENIX-4116, PHOENIX-4419, and PHOENIX-4642: cannot > > > reproduce > > > > > it. > > > > > [More information is required] PHOENIX-4504 cannot reproduce it but > > > > someone > > > > > claimed that he had a similar issue. > > > > > [unusual query] PHOENIX-4540 and PHOENIX-6217 > > > > > > > > > > Based on my finding, I think it's better to have more frequent > > > > housekeeping > > > > > and resolve unreproducible bugs especially since many of them are > > > > > considering out of date (phoenix-4.11 or even phoenix-4.6). Since I > > > still > > > > > need time to work on the blocker Jira(PHOENIX-5712, PHOENIX-6241) > and > > > fix > > > > > test flappers, if you want to fix "unusual query" bugs, feel free > to > > do > > > > so. > > > > > > > > > > > > > > > Sincerely, > > > > > Xinyi > > > > > > > > > > On Wed, Dec 2, 2020 at 12:41 AM Ankit Singhal > > > wrote: > > > > > > > > > > > Thanks Daniel and appreciate the effort you put in getting the > list > > > > ready > > > > > > for bugs producing wrong results > > > > > > but none of them seems to be a blocker to me for 4.16 as they are > > not > > > > the > > > > > > regression and doesn't break the general functionality > > > > > > except for specific features, RVC/desc as Chenglei also pointed > out > > > > > (though > > > > > > I'll defer the assessment to RM "Xinyi"). > > > > > > Probably these can be a part of 4.16.1 or we can do 4.17.0 soon > > maybe > > > > > after > > > > > > a few weeks/month? > > > > > > > > > > > > Considering that we have already fixed 137 bugs and done 85+ > > > > > > improvements/features in 4.16, > > > > > > it will not be a good idea to deprive the user from such fixes. > > > > > > It's been a year since our last 4.15 release, having no release > > > brings > > > > > more > > > > > > questions on the project > > > > > > rather than the bugs which affect a certain % of feature/users, > > would > > > > the > > > > > > release notes > > > > > > explaining the stability of certain features set the right > > > expectation > > > > > for > > > > > > those users who rely on these features to wait for a future > > release? > > > > > > > > > > > > Regards, > > > > > > Ankit Singhal > > > > > > > > > > > > On Tue, Dec 1, 2020 at 8:21 PM cheng...@apache.org < > > > > cheng...@apache.org> > > > > > > wrote: > > > > > > > > > > >
Re: [Discuss] Releasing Phoenix 4.16
Hi, We finally have a stable 4.x branch build after many flapper test fixing contributions from the community. I'm going to fork a new branch(4.16) from the current 4.x branch for the 4.16 release. As a cutoff, I will not include any new features other than PHOENIX-6211 and bug fix. Please let me know if you have any questions or concerns. Thanks, Xinyi On Wed, Dec 23, 2020 at 6:07 AM Viraj Jasani wrote: > An update on test stability: > > As per recent 4.x build results, we are left with very few flappers and > specifically > with HBase profile 1.6, I can see recent 7 build (multibranch + PR > precommit results) > results without any test failure. > > > On Tue, Dec 15, 2020 at 5:52 AM Xinyi Yan wrote: > > > Yes, we are currently working on fixing the 4.x branch test flappers > while > > waiting for the PHOENIX-5435. After that, I will try to have an RC ASAP. > > > > On Sun, Dec 13, 2020 at 3:29 PM Ankit Singhal > > wrote: > > > > > I see that both the blockers listed here PHOENIX-5712 and PHOENIX-6241 > > > have been resolved(Thanks to Xinyi), and also as per the JIRA query > there > > > is no Jira marked as a blocker for 4.16 except the one related to > > > documentation > > > for "splittable catalog table". > > > > > > Xinyi, so are we good to start the release process now? > > > > > > On Wed, Dec 2, 2020 at 9:32 PM Xinyi Yan wrote: > > > > > > > Thanks for replying and providing suggestions. I looked at the wrong > > > result > > > > Jira list that Daniel provided and did some local testing, and here > is > > > the > > > > result: > > > > [resolved] PHOENIX-4116, PHOENIX-4419, and PHOENIX-4642: cannot > > reproduce > > > > it. > > > > [More information is required] PHOENIX-4504 cannot reproduce it but > > > someone > > > > claimed that he had a similar issue. > > > > [unusual query] PHOENIX-4540 and PHOENIX-6217 > > > > > > > > Based on my finding, I think it's better to have more frequent > > > housekeeping > > > > and resolve unreproducible bugs especially since many of them are > > > > considering out of date (phoenix-4.11 or even phoenix-4.6). Since I > > still > > > > need time to work on the blocker Jira(PHOENIX-5712, PHOENIX-6241) and > > fix > > > > test flappers, if you want to fix "unusual query" bugs, feel free to > do > > > so. > > > > > > > > > > > > Sincerely, > > > > Xinyi > > > > > > > > On Wed, Dec 2, 2020 at 12:41 AM Ankit Singhal > > wrote: > > > > > > > > > Thanks Daniel and appreciate the effort you put in getting the list > > > ready > > > > > for bugs producing wrong results > > > > > but none of them seems to be a blocker to me for 4.16 as they are > not > > > the > > > > > regression and doesn't break the general functionality > > > > > except for specific features, RVC/desc as Chenglei also pointed out > > > > (though > > > > > I'll defer the assessment to RM "Xinyi"). > > > > > Probably these can be a part of 4.16.1 or we can do 4.17.0 soon > maybe > > > > after > > > > > a few weeks/month? > > > > > > > > > > Considering that we have already fixed 137 bugs and done 85+ > > > > > improvements/features in 4.16, > > > > > it will not be a good idea to deprive the user from such fixes. > > > > > It's been a year since our last 4.15 release, having no release > > brings > > > > more > > > > > questions on the project > > > > > rather than the bugs which affect a certain % of feature/users, > would > > > the > > > > > release notes > > > > > explaining the stability of certain features set the right > > expectation > > > > for > > > > > those users who rely on these features to wait for a future > release? > > > > > > > > > > Regards, > > > > > Ankit Singhal > > > > > > > > > > On Tue, Dec 1, 2020 at 8:21 PM cheng...@apache.org < > > > cheng...@apache.org> > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In my opinion, we should keep releases light and frequent, and > for > > > > some > > > > > > unusual query bugs like RVC and DESC > > > > > > we could delay fix to next release . I think we should release > > 4.16.0 > > > > and > > > > > > 5.1.0 as quickly as possible. In China, many users > > > > > > in HBase&Phoenix User Group thought that Phoenix was dead > because > > > our > > > > > too > > > > > > long interval release and stopped using > > > > > > Phoenix. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > At 2020-12-02 08:45:46, "Chinmay Kulkarni" < > > > chinmayskulka...@gmail.com > > > > > > > > > > > wrote: > > > > > > >I agree. These are all major bugs and we should aim at solving > > them > > > > > after > > > > > > >checking that they are still issues. I am +1 on 5833 and I think > > > 5484 > > > > > > would > > > > > > >be a great addition to 4.16 as well. We should aim at resolving > > high > > > > > > >priority bugs like this in every release. > > > > > > > > > > > > > >Sometimes we let these bugs sli
Re: [Discuss] Releasing Phoenix 4.16
An update on test stability: As per recent 4.x build results, we are left with very few flappers and specifically with HBase profile 1.6, I can see recent 7 build (multibranch + PR precommit results) results without any test failure. On Tue, Dec 15, 2020 at 5:52 AM Xinyi Yan wrote: > Yes, we are currently working on fixing the 4.x branch test flappers while > waiting for the PHOENIX-5435. After that, I will try to have an RC ASAP. > > On Sun, Dec 13, 2020 at 3:29 PM Ankit Singhal > wrote: > > > I see that both the blockers listed here PHOENIX-5712 and PHOENIX-6241 > > have been resolved(Thanks to Xinyi), and also as per the JIRA query there > > is no Jira marked as a blocker for 4.16 except the one related to > > documentation > > for "splittable catalog table". > > > > Xinyi, so are we good to start the release process now? > > > > On Wed, Dec 2, 2020 at 9:32 PM Xinyi Yan wrote: > > > > > Thanks for replying and providing suggestions. I looked at the wrong > > result > > > Jira list that Daniel provided and did some local testing, and here is > > the > > > result: > > > [resolved] PHOENIX-4116, PHOENIX-4419, and PHOENIX-4642: cannot > reproduce > > > it. > > > [More information is required] PHOENIX-4504 cannot reproduce it but > > someone > > > claimed that he had a similar issue. > > > [unusual query] PHOENIX-4540 and PHOENIX-6217 > > > > > > Based on my finding, I think it's better to have more frequent > > housekeeping > > > and resolve unreproducible bugs especially since many of them are > > > considering out of date (phoenix-4.11 or even phoenix-4.6). Since I > still > > > need time to work on the blocker Jira(PHOENIX-5712, PHOENIX-6241) and > fix > > > test flappers, if you want to fix "unusual query" bugs, feel free to do > > so. > > > > > > > > > Sincerely, > > > Xinyi > > > > > > On Wed, Dec 2, 2020 at 12:41 AM Ankit Singhal > wrote: > > > > > > > Thanks Daniel and appreciate the effort you put in getting the list > > ready > > > > for bugs producing wrong results > > > > but none of them seems to be a blocker to me for 4.16 as they are not > > the > > > > regression and doesn't break the general functionality > > > > except for specific features, RVC/desc as Chenglei also pointed out > > > (though > > > > I'll defer the assessment to RM "Xinyi"). > > > > Probably these can be a part of 4.16.1 or we can do 4.17.0 soon maybe > > > after > > > > a few weeks/month? > > > > > > > > Considering that we have already fixed 137 bugs and done 85+ > > > > improvements/features in 4.16, > > > > it will not be a good idea to deprive the user from such fixes. > > > > It's been a year since our last 4.15 release, having no release > brings > > > more > > > > questions on the project > > > > rather than the bugs which affect a certain % of feature/users, would > > the > > > > release notes > > > > explaining the stability of certain features set the right > expectation > > > for > > > > those users who rely on these features to wait for a future release? > > > > > > > > Regards, > > > > Ankit Singhal > > > > > > > > On Tue, Dec 1, 2020 at 8:21 PM cheng...@apache.org < > > cheng...@apache.org> > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > In my opinion, we should keep releases light and frequent, and for > > > some > > > > > unusual query bugs like RVC and DESC > > > > > we could delay fix to next release . I think we should release > 4.16.0 > > > and > > > > > 5.1.0 as quickly as possible. In China, many users > > > > > in HBase&Phoenix User Group thought that Phoenix was dead because > > our > > > > too > > > > > long interval release and stopped using > > > > > Phoenix. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > At 2020-12-02 08:45:46, "Chinmay Kulkarni" < > > chinmayskulka...@gmail.com > > > > > > > > > wrote: > > > > > >I agree. These are all major bugs and we should aim at solving > them > > > > after > > > > > >checking that they are still issues. I am +1 on 5833 and I think > > 5484 > > > > > would > > > > > >be a great addition to 4.16 as well. We should aim at resolving > high > > > > > >priority bugs like this in every release. > > > > > > > > > > > >Sometimes we let these bugs slip without a resolution before a > > > release, > > > > > >citing that these are "known issues" or "not regressions from the > > last > > > > > >release". In some cases this may be fine since we want to keep > > > releases > > > > > >light and frequent, but perhaps we can track such issues and aim > at > > > > > >reducing the number of bugs by x% in each release? This will also > > keep > > > > old > > > > > >Jiras alive since we will potentially periodically review them. > > > > > > > > > > > > > > > > > >On Tue, Dec 1, 2020 at 4:01 PM Geoffrey Jacoby < > gjac...@apache.org> > > > > > wrote: > > > > > > > > > > > >> I've got PHOENIX-5435 in review right now, and would like to get > > it > > >
Re: [Discuss] Releasing Phoenix 4.16
Yes, we are currently working on fixing the 4.x branch test flappers while waiting for the PHOENIX-5435. After that, I will try to have an RC ASAP. On Sun, Dec 13, 2020 at 3:29 PM Ankit Singhal wrote: > I see that both the blockers listed here PHOENIX-5712 and PHOENIX-6241 > have been resolved(Thanks to Xinyi), and also as per the JIRA query there > is no Jira marked as a blocker for 4.16 except the one related to > documentation > for "splittable catalog table". > > Xinyi, so are we good to start the release process now? > > On Wed, Dec 2, 2020 at 9:32 PM Xinyi Yan wrote: > > > Thanks for replying and providing suggestions. I looked at the wrong > result > > Jira list that Daniel provided and did some local testing, and here is > the > > result: > > [resolved] PHOENIX-4116, PHOENIX-4419, and PHOENIX-4642: cannot reproduce > > it. > > [More information is required] PHOENIX-4504 cannot reproduce it but > someone > > claimed that he had a similar issue. > > [unusual query] PHOENIX-4540 and PHOENIX-6217 > > > > Based on my finding, I think it's better to have more frequent > housekeeping > > and resolve unreproducible bugs especially since many of them are > > considering out of date (phoenix-4.11 or even phoenix-4.6). Since I still > > need time to work on the blocker Jira(PHOENIX-5712, PHOENIX-6241) and fix > > test flappers, if you want to fix "unusual query" bugs, feel free to do > so. > > > > > > Sincerely, > > Xinyi > > > > On Wed, Dec 2, 2020 at 12:41 AM Ankit Singhal wrote: > > > > > Thanks Daniel and appreciate the effort you put in getting the list > ready > > > for bugs producing wrong results > > > but none of them seems to be a blocker to me for 4.16 as they are not > the > > > regression and doesn't break the general functionality > > > except for specific features, RVC/desc as Chenglei also pointed out > > (though > > > I'll defer the assessment to RM "Xinyi"). > > > Probably these can be a part of 4.16.1 or we can do 4.17.0 soon maybe > > after > > > a few weeks/month? > > > > > > Considering that we have already fixed 137 bugs and done 85+ > > > improvements/features in 4.16, > > > it will not be a good idea to deprive the user from such fixes. > > > It's been a year since our last 4.15 release, having no release brings > > more > > > questions on the project > > > rather than the bugs which affect a certain % of feature/users, would > the > > > release notes > > > explaining the stability of certain features set the right expectation > > for > > > those users who rely on these features to wait for a future release? > > > > > > Regards, > > > Ankit Singhal > > > > > > On Tue, Dec 1, 2020 at 8:21 PM cheng...@apache.org < > cheng...@apache.org> > > > wrote: > > > > > > > > > > > > > > > > > > > In my opinion, we should keep releases light and frequent, and for > > some > > > > unusual query bugs like RVC and DESC > > > > we could delay fix to next release . I think we should release 4.16.0 > > and > > > > 5.1.0 as quickly as possible. In China, many users > > > > in HBase&Phoenix User Group thought that Phoenix was dead because > our > > > too > > > > long interval release and stopped using > > > > Phoenix. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > At 2020-12-02 08:45:46, "Chinmay Kulkarni" < > chinmayskulka...@gmail.com > > > > > > > wrote: > > > > >I agree. These are all major bugs and we should aim at solving them > > > after > > > > >checking that they are still issues. I am +1 on 5833 and I think > 5484 > > > > would > > > > >be a great addition to 4.16 as well. We should aim at resolving high > > > > >priority bugs like this in every release. > > > > > > > > > >Sometimes we let these bugs slip without a resolution before a > > release, > > > > >citing that these are "known issues" or "not regressions from the > last > > > > >release". In some cases this may be fine since we want to keep > > releases > > > > >light and frequent, but perhaps we can track such issues and aim at > > > > >reducing the number of bugs by x% in each release? This will also > keep > > > old > > > > >Jiras alive since we will potentially periodically review them. > > > > > > > > > > > > > > >On Tue, Dec 1, 2020 at 4:01 PM Geoffrey Jacoby > > > > wrote: > > > > > > > > > >> I've got PHOENIX-5435 in review right now, and would like to get > it > > in > > > > 4.16 > > > > >> / 5.1. > > > > >> > > > > >> It's allowing the annotation of Phoenix metadata into HBase WALs > as > > a > > > > >> pre-req for the Phoenix Change Detection Capture framework > > > > (PHOENIX-5442). > > > > >> Since it has both client/server logic, and adds a field to > > > > System.Catalog, > > > > >> it can't go in a patch release. > > > > >> > > > > >> Depending on timing, I'd _like_ to get PHOENIX-6227, which is the > > last > > > > part > > > > >> of CDC that will go into core Phoenix, into 4.16, but since that > > _can_ > > > > go > > > > >> in a patch rele
Re: [Discuss] Releasing Phoenix 4.16
I see that both the blockers listed here PHOENIX-5712 and PHOENIX-6241 have been resolved(Thanks to Xinyi), and also as per the JIRA query there is no Jira marked as a blocker for 4.16 except the one related to documentation for "splittable catalog table". Xinyi, so are we good to start the release process now? On Wed, Dec 2, 2020 at 9:32 PM Xinyi Yan wrote: > Thanks for replying and providing suggestions. I looked at the wrong result > Jira list that Daniel provided and did some local testing, and here is the > result: > [resolved] PHOENIX-4116, PHOENIX-4419, and PHOENIX-4642: cannot reproduce > it. > [More information is required] PHOENIX-4504 cannot reproduce it but someone > claimed that he had a similar issue. > [unusual query] PHOENIX-4540 and PHOENIX-6217 > > Based on my finding, I think it's better to have more frequent housekeeping > and resolve unreproducible bugs especially since many of them are > considering out of date (phoenix-4.11 or even phoenix-4.6). Since I still > need time to work on the blocker Jira(PHOENIX-5712, PHOENIX-6241) and fix > test flappers, if you want to fix "unusual query" bugs, feel free to do so. > > > Sincerely, > Xinyi > > On Wed, Dec 2, 2020 at 12:41 AM Ankit Singhal wrote: > > > Thanks Daniel and appreciate the effort you put in getting the list ready > > for bugs producing wrong results > > but none of them seems to be a blocker to me for 4.16 as they are not the > > regression and doesn't break the general functionality > > except for specific features, RVC/desc as Chenglei also pointed out > (though > > I'll defer the assessment to RM "Xinyi"). > > Probably these can be a part of 4.16.1 or we can do 4.17.0 soon maybe > after > > a few weeks/month? > > > > Considering that we have already fixed 137 bugs and done 85+ > > improvements/features in 4.16, > > it will not be a good idea to deprive the user from such fixes. > > It's been a year since our last 4.15 release, having no release brings > more > > questions on the project > > rather than the bugs which affect a certain % of feature/users, would the > > release notes > > explaining the stability of certain features set the right expectation > for > > those users who rely on these features to wait for a future release? > > > > Regards, > > Ankit Singhal > > > > On Tue, Dec 1, 2020 at 8:21 PM cheng...@apache.org > > wrote: > > > > > > > > > > > > > > In my opinion, we should keep releases light and frequent, and for > some > > > unusual query bugs like RVC and DESC > > > we could delay fix to next release . I think we should release 4.16.0 > and > > > 5.1.0 as quickly as possible. In China, many users > > > in HBase&Phoenix User Group thought that Phoenix was dead because our > > too > > > long interval release and stopped using > > > Phoenix. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > At 2020-12-02 08:45:46, "Chinmay Kulkarni" > > > > wrote: > > > >I agree. These are all major bugs and we should aim at solving them > > after > > > >checking that they are still issues. I am +1 on 5833 and I think 5484 > > > would > > > >be a great addition to 4.16 as well. We should aim at resolving high > > > >priority bugs like this in every release. > > > > > > > >Sometimes we let these bugs slip without a resolution before a > release, > > > >citing that these are "known issues" or "not regressions from the last > > > >release". In some cases this may be fine since we want to keep > releases > > > >light and frequent, but perhaps we can track such issues and aim at > > > >reducing the number of bugs by x% in each release? This will also keep > > old > > > >Jiras alive since we will potentially periodically review them. > > > > > > > > > > > >On Tue, Dec 1, 2020 at 4:01 PM Geoffrey Jacoby > > > wrote: > > > > > > > >> I've got PHOENIX-5435 in review right now, and would like to get it > in > > > 4.16 > > > >> / 5.1. > > > >> > > > >> It's allowing the annotation of Phoenix metadata into HBase WALs as > a > > > >> pre-req for the Phoenix Change Detection Capture framework > > > (PHOENIX-5442). > > > >> Since it has both client/server logic, and adds a field to > > > System.Catalog, > > > >> it can't go in a patch release. > > > >> > > > >> Depending on timing, I'd _like_ to get PHOENIX-6227, which is the > last > > > part > > > >> of CDC that will go into core Phoenix, into 4.16, but since that > _can_ > > > go > > > >> in a patch release and I haven't started it yet, if the release gets > > cut > > > >> before it's ready, no big deal. (The rest of CDC will go into > > > >> phoenix-connectors for a future release of that project.) > > > >> > > > >> As for the correctness problems that Daniel points out, I think we > > > should > > > >> fix the ones that were detected with a recent version (4.14 or > 4.15?), > > > and > > > >> test to see which of the older ones can still be reproduced. Once we > > > know > > > >> which bugs are real and which are just historical, we can b
Re: [Discuss] Releasing Phoenix 4.16
Thanks for replying and providing suggestions. I looked at the wrong result Jira list that Daniel provided and did some local testing, and here is the result: [resolved] PHOENIX-4116, PHOENIX-4419, and PHOENIX-4642: cannot reproduce it. [More information is required] PHOENIX-4504 cannot reproduce it but someone claimed that he had a similar issue. [unusual query] PHOENIX-4540 and PHOENIX-6217 Based on my finding, I think it's better to have more frequent housekeeping and resolve unreproducible bugs especially since many of them are considering out of date (phoenix-4.11 or even phoenix-4.6). Since I still need time to work on the blocker Jira(PHOENIX-5712, PHOENIX-6241) and fix test flappers, if you want to fix "unusual query" bugs, feel free to do so. Sincerely, Xinyi On Wed, Dec 2, 2020 at 12:41 AM Ankit Singhal wrote: > Thanks Daniel and appreciate the effort you put in getting the list ready > for bugs producing wrong results > but none of them seems to be a blocker to me for 4.16 as they are not the > regression and doesn't break the general functionality > except for specific features, RVC/desc as Chenglei also pointed out (though > I'll defer the assessment to RM "Xinyi"). > Probably these can be a part of 4.16.1 or we can do 4.17.0 soon maybe after > a few weeks/month? > > Considering that we have already fixed 137 bugs and done 85+ > improvements/features in 4.16, > it will not be a good idea to deprive the user from such fixes. > It's been a year since our last 4.15 release, having no release brings more > questions on the project > rather than the bugs which affect a certain % of feature/users, would the > release notes > explaining the stability of certain features set the right expectation for > those users who rely on these features to wait for a future release? > > Regards, > Ankit Singhal > > On Tue, Dec 1, 2020 at 8:21 PM cheng...@apache.org > wrote: > > > > > > > > > In my opinion, we should keep releases light and frequent, and for some > > unusual query bugs like RVC and DESC > > we could delay fix to next release . I think we should release 4.16.0 and > > 5.1.0 as quickly as possible. In China, many users > > in HBase&Phoenix User Group thought that Phoenix was dead because our > too > > long interval release and stopped using > > Phoenix. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > At 2020-12-02 08:45:46, "Chinmay Kulkarni" > > wrote: > > >I agree. These are all major bugs and we should aim at solving them > after > > >checking that they are still issues. I am +1 on 5833 and I think 5484 > > would > > >be a great addition to 4.16 as well. We should aim at resolving high > > >priority bugs like this in every release. > > > > > >Sometimes we let these bugs slip without a resolution before a release, > > >citing that these are "known issues" or "not regressions from the last > > >release". In some cases this may be fine since we want to keep releases > > >light and frequent, but perhaps we can track such issues and aim at > > >reducing the number of bugs by x% in each release? This will also keep > old > > >Jiras alive since we will potentially periodically review them. > > > > > > > > >On Tue, Dec 1, 2020 at 4:01 PM Geoffrey Jacoby > > wrote: > > > > > >> I've got PHOENIX-5435 in review right now, and would like to get it in > > 4.16 > > >> / 5.1. > > >> > > >> It's allowing the annotation of Phoenix metadata into HBase WALs as a > > >> pre-req for the Phoenix Change Detection Capture framework > > (PHOENIX-5442). > > >> Since it has both client/server logic, and adds a field to > > System.Catalog, > > >> it can't go in a patch release. > > >> > > >> Depending on timing, I'd _like_ to get PHOENIX-6227, which is the last > > part > > >> of CDC that will go into core Phoenix, into 4.16, but since that _can_ > > go > > >> in a patch release and I haven't started it yet, if the release gets > cut > > >> before it's ready, no big deal. (The rest of CDC will go into > > >> phoenix-connectors for a future release of that project.) > > >> > > >> As for the correctness problems that Daniel points out, I think we > > should > > >> fix the ones that were detected with a recent version (4.14 or 4.15?), > > and > > >> test to see which of the older ones can still be reproduced. Once we > > know > > >> which bugs are real and which are just historical, we can better judge > > >> scope. And hopefully close a bunch of obsolete bugs. (Thanks, Daniel, > > for > > >> collecting that list!) > > >> > > >> Geoffrey > > >> > > >> > > >> > > >> On Tue, Dec 1, 2020 at 1:33 PM Daniel Wong > > >> wrote: > > >> > > >> > Hi, I wanted to bring up wrong results in Phoenix and some JIRAs > > around > > >> > them that I think we should fix as the wrong result lessens the end > > >> user's > > >> > trust in Phoenix. Releasing a new version without addressing these > > in a > > >> > minor release hurts our visibility in that these critical issues are > > not > > >> > addressed. > > >> > > > >>
Re: [Discuss] Releasing Phoenix 4.16
Thanks Daniel and appreciate the effort you put in getting the list ready for bugs producing wrong results but none of them seems to be a blocker to me for 4.16 as they are not the regression and doesn't break the general functionality except for specific features, RVC/desc as Chenglei also pointed out (though I'll defer the assessment to RM "Xinyi"). Probably these can be a part of 4.16.1 or we can do 4.17.0 soon maybe after a few weeks/month? Considering that we have already fixed 137 bugs and done 85+ improvements/features in 4.16, it will not be a good idea to deprive the user from such fixes. It's been a year since our last 4.15 release, having no release brings more questions on the project rather than the bugs which affect a certain % of feature/users, would the release notes explaining the stability of certain features set the right expectation for those users who rely on these features to wait for a future release? Regards, Ankit Singhal On Tue, Dec 1, 2020 at 8:21 PM cheng...@apache.org wrote: > > > > In my opinion, we should keep releases light and frequent, and for some > unusual query bugs like RVC and DESC > we could delay fix to next release . I think we should release 4.16.0 and > 5.1.0 as quickly as possible. In China, many users > in HBase&Phoenix User Group thought that Phoenix was dead because our too > long interval release and stopped using > Phoenix. > > > > > > > > > > > > > > > At 2020-12-02 08:45:46, "Chinmay Kulkarni" > wrote: > >I agree. These are all major bugs and we should aim at solving them after > >checking that they are still issues. I am +1 on 5833 and I think 5484 > would > >be a great addition to 4.16 as well. We should aim at resolving high > >priority bugs like this in every release. > > > >Sometimes we let these bugs slip without a resolution before a release, > >citing that these are "known issues" or "not regressions from the last > >release". In some cases this may be fine since we want to keep releases > >light and frequent, but perhaps we can track such issues and aim at > >reducing the number of bugs by x% in each release? This will also keep old > >Jiras alive since we will potentially periodically review them. > > > > > >On Tue, Dec 1, 2020 at 4:01 PM Geoffrey Jacoby > wrote: > > > >> I've got PHOENIX-5435 in review right now, and would like to get it in > 4.16 > >> / 5.1. > >> > >> It's allowing the annotation of Phoenix metadata into HBase WALs as a > >> pre-req for the Phoenix Change Detection Capture framework > (PHOENIX-5442). > >> Since it has both client/server logic, and adds a field to > System.Catalog, > >> it can't go in a patch release. > >> > >> Depending on timing, I'd _like_ to get PHOENIX-6227, which is the last > part > >> of CDC that will go into core Phoenix, into 4.16, but since that _can_ > go > >> in a patch release and I haven't started it yet, if the release gets cut > >> before it's ready, no big deal. (The rest of CDC will go into > >> phoenix-connectors for a future release of that project.) > >> > >> As for the correctness problems that Daniel points out, I think we > should > >> fix the ones that were detected with a recent version (4.14 or 4.15?), > and > >> test to see which of the older ones can still be reproduced. Once we > know > >> which bugs are real and which are just historical, we can better judge > >> scope. And hopefully close a bunch of obsolete bugs. (Thanks, Daniel, > for > >> collecting that list!) > >> > >> Geoffrey > >> > >> > >> > >> On Tue, Dec 1, 2020 at 1:33 PM Daniel Wong > >> wrote: > >> > >> > Hi, I wanted to bring up wrong results in Phoenix and some JIRAs > around > >> > them that I think we should fix as the wrong result lessens the end > >> user's > >> > trust in Phoenix. Releasing a new version without addressing these > in a > >> > minor release hurts our visibility in that these critical issues are > not > >> > addressed. > >> > > >> > Jira's that I'm involved with for example: I've already given a patch > >> > several months ago for 5833 and there is a chance it may fix 5484. > >> > https://issues.apache.org/jira/browse/PHOENIX-5833 > >> > https://issues.apache.org/jira/browse/PHOENIX-5484 > >> > > >> > In addition, inspecting apache JIRA i see several other wrong result > >> JIRAs > >> > from the community. Some of these certainly are probably old issues > or > >> > incorrect understanding but some of these are opened by our own dev > >> > community and are likely real problems. > >> > https://issues.apache.org/jira/browse/PHOENIX-6217 > >> > https://issues.apache.org/jira/browse/PHOENIX-5571 > >> > https://issues.apache.org/jira/browse/PHOENIX-4642 > >> > https://issues.apache.org/jira/browse/PHOENIX-4540 > >> > https://issues.apache.org/jira/browse/PHOENIX-4504 > >> > https://issues.apache.org/jira/browse/PHOENIX-4419 > >> > https://issues.apache.org/jira/browse/PHOENIX-4116 > >> > > >> > What is our stance on this type of issue? Are we going to say these > were
Re:[Discuss] Releasing Phoenix 4.16
In my opinion, we should keep releases light and frequent, and for some unusual query bugs like RVC and DESC we could delay fix to next release . I think we should release 4.16.0 and 5.1.0 as quickly as possible. In China, many users in HBase&Phoenix User Group thought that Phoenix was dead because our too long interval release and stopped using Phoenix. At 2020-12-02 08:45:46, "Chinmay Kulkarni" wrote: >I agree. These are all major bugs and we should aim at solving them after >checking that they are still issues. I am +1 on 5833 and I think 5484 would >be a great addition to 4.16 as well. We should aim at resolving high >priority bugs like this in every release. > >Sometimes we let these bugs slip without a resolution before a release, >citing that these are "known issues" or "not regressions from the last >release". In some cases this may be fine since we want to keep releases >light and frequent, but perhaps we can track such issues and aim at >reducing the number of bugs by x% in each release? This will also keep old >Jiras alive since we will potentially periodically review them. > > >On Tue, Dec 1, 2020 at 4:01 PM Geoffrey Jacoby wrote: > >> I've got PHOENIX-5435 in review right now, and would like to get it in 4.16 >> / 5.1. >> >> It's allowing the annotation of Phoenix metadata into HBase WALs as a >> pre-req for the Phoenix Change Detection Capture framework (PHOENIX-5442). >> Since it has both client/server logic, and adds a field to System.Catalog, >> it can't go in a patch release. >> >> Depending on timing, I'd _like_ to get PHOENIX-6227, which is the last part >> of CDC that will go into core Phoenix, into 4.16, but since that _can_ go >> in a patch release and I haven't started it yet, if the release gets cut >> before it's ready, no big deal. (The rest of CDC will go into >> phoenix-connectors for a future release of that project.) >> >> As for the correctness problems that Daniel points out, I think we should >> fix the ones that were detected with a recent version (4.14 or 4.15?), and >> test to see which of the older ones can still be reproduced. Once we know >> which bugs are real and which are just historical, we can better judge >> scope. And hopefully close a bunch of obsolete bugs. (Thanks, Daniel, for >> collecting that list!) >> >> Geoffrey >> >> >> >> On Tue, Dec 1, 2020 at 1:33 PM Daniel Wong >> wrote: >> >> > Hi, I wanted to bring up wrong results in Phoenix and some JIRAs around >> > them that I think we should fix as the wrong result lessens the end >> user's >> > trust in Phoenix. Releasing a new version without addressing these in a >> > minor release hurts our visibility in that these critical issues are not >> > addressed. >> > >> > Jira's that I'm involved with for example: I've already given a patch >> > several months ago for 5833 and there is a chance it may fix 5484. >> > https://issues.apache.org/jira/browse/PHOENIX-5833 >> > https://issues.apache.org/jira/browse/PHOENIX-5484 >> > >> > In addition, inspecting apache JIRA i see several other wrong result >> JIRAs >> > from the community. Some of these certainly are probably old issues or >> > incorrect understanding but some of these are opened by our own dev >> > community and are likely real problems. >> > https://issues.apache.org/jira/browse/PHOENIX-6217 >> > https://issues.apache.org/jira/browse/PHOENIX-5571 >> > https://issues.apache.org/jira/browse/PHOENIX-4642 >> > https://issues.apache.org/jira/browse/PHOENIX-4540 >> > https://issues.apache.org/jira/browse/PHOENIX-4504 >> > https://issues.apache.org/jira/browse/PHOENIX-4419 >> > https://issues.apache.org/jira/browse/PHOENIX-4116 >> > >> > What is our stance on this type of issue? Are we going to say these were >> > issues prior to 4.15 and not address them? Should we have requirements >> for >> > our releases to fix wrong results? >> > >> > Daniel Wong >> > >> > On Mon, Nov 30, 2020 at 7:30 PM Xinyi Yan wrote: >> > >> > > Hi all, >> > > >> > > It's time to discuss the Phoenix 4.16 release. After many people's >> > > contributions on the bug fixes, new features, and other works in the >> past >> > > few months, we are kind of close to the point to have a RC (still need >> to >> > > fix test flappers). Please let me know if you think any JIRA must be >> part >> > > of the Phoenix 4.16 release other than major blocker PHOENIX-5712. >> > > >> > > If no surprise comes up, I will not wait for any new major features and >> > > focus on the RC as soon as possible. >> > > >> > > Sincerely, >> > > Xinyi >> > > >> > >> > >> > -- >> > Daniel Wong >> > Salesforce >> > Mobile: 628.217.1808 >> > >> > > >-- >Chinmay Kulkarni
Re: [Discuss] Releasing Phoenix 4.16
I agree. These are all major bugs and we should aim at solving them after checking that they are still issues. I am +1 on 5833 and I think 5484 would be a great addition to 4.16 as well. We should aim at resolving high priority bugs like this in every release. Sometimes we let these bugs slip without a resolution before a release, citing that these are "known issues" or "not regressions from the last release". In some cases this may be fine since we want to keep releases light and frequent, but perhaps we can track such issues and aim at reducing the number of bugs by x% in each release? This will also keep old Jiras alive since we will potentially periodically review them. On Tue, Dec 1, 2020 at 4:01 PM Geoffrey Jacoby wrote: > I've got PHOENIX-5435 in review right now, and would like to get it in 4.16 > / 5.1. > > It's allowing the annotation of Phoenix metadata into HBase WALs as a > pre-req for the Phoenix Change Detection Capture framework (PHOENIX-5442). > Since it has both client/server logic, and adds a field to System.Catalog, > it can't go in a patch release. > > Depending on timing, I'd _like_ to get PHOENIX-6227, which is the last part > of CDC that will go into core Phoenix, into 4.16, but since that _can_ go > in a patch release and I haven't started it yet, if the release gets cut > before it's ready, no big deal. (The rest of CDC will go into > phoenix-connectors for a future release of that project.) > > As for the correctness problems that Daniel points out, I think we should > fix the ones that were detected with a recent version (4.14 or 4.15?), and > test to see which of the older ones can still be reproduced. Once we know > which bugs are real and which are just historical, we can better judge > scope. And hopefully close a bunch of obsolete bugs. (Thanks, Daniel, for > collecting that list!) > > Geoffrey > > > > On Tue, Dec 1, 2020 at 1:33 PM Daniel Wong > wrote: > > > Hi, I wanted to bring up wrong results in Phoenix and some JIRAs around > > them that I think we should fix as the wrong result lessens the end > user's > > trust in Phoenix. Releasing a new version without addressing these in a > > minor release hurts our visibility in that these critical issues are not > > addressed. > > > > Jira's that I'm involved with for example: I've already given a patch > > several months ago for 5833 and there is a chance it may fix 5484. > > https://issues.apache.org/jira/browse/PHOENIX-5833 > > https://issues.apache.org/jira/browse/PHOENIX-5484 > > > > In addition, inspecting apache JIRA i see several other wrong result > JIRAs > > from the community. Some of these certainly are probably old issues or > > incorrect understanding but some of these are opened by our own dev > > community and are likely real problems. > > https://issues.apache.org/jira/browse/PHOENIX-6217 > > https://issues.apache.org/jira/browse/PHOENIX-5571 > > https://issues.apache.org/jira/browse/PHOENIX-4642 > > https://issues.apache.org/jira/browse/PHOENIX-4540 > > https://issues.apache.org/jira/browse/PHOENIX-4504 > > https://issues.apache.org/jira/browse/PHOENIX-4419 > > https://issues.apache.org/jira/browse/PHOENIX-4116 > > > > What is our stance on this type of issue? Are we going to say these were > > issues prior to 4.15 and not address them? Should we have requirements > for > > our releases to fix wrong results? > > > > Daniel Wong > > > > On Mon, Nov 30, 2020 at 7:30 PM Xinyi Yan wrote: > > > > > Hi all, > > > > > > It's time to discuss the Phoenix 4.16 release. After many people's > > > contributions on the bug fixes, new features, and other works in the > past > > > few months, we are kind of close to the point to have a RC (still need > to > > > fix test flappers). Please let me know if you think any JIRA must be > part > > > of the Phoenix 4.16 release other than major blocker PHOENIX-5712. > > > > > > If no surprise comes up, I will not wait for any new major features and > > > focus on the RC as soon as possible. > > > > > > Sincerely, > > > Xinyi > > > > > > > > > -- > > Daniel Wong > > Salesforce > > Mobile: 628.217.1808 > > > -- Chinmay Kulkarni
Re: [Discuss] Releasing Phoenix 4.16
I've got PHOENIX-5435 in review right now, and would like to get it in 4.16 / 5.1. It's allowing the annotation of Phoenix metadata into HBase WALs as a pre-req for the Phoenix Change Detection Capture framework (PHOENIX-5442). Since it has both client/server logic, and adds a field to System.Catalog, it can't go in a patch release. Depending on timing, I'd _like_ to get PHOENIX-6227, which is the last part of CDC that will go into core Phoenix, into 4.16, but since that _can_ go in a patch release and I haven't started it yet, if the release gets cut before it's ready, no big deal. (The rest of CDC will go into phoenix-connectors for a future release of that project.) As for the correctness problems that Daniel points out, I think we should fix the ones that were detected with a recent version (4.14 or 4.15?), and test to see which of the older ones can still be reproduced. Once we know which bugs are real and which are just historical, we can better judge scope. And hopefully close a bunch of obsolete bugs. (Thanks, Daniel, for collecting that list!) Geoffrey On Tue, Dec 1, 2020 at 1:33 PM Daniel Wong wrote: > Hi, I wanted to bring up wrong results in Phoenix and some JIRAs around > them that I think we should fix as the wrong result lessens the end user's > trust in Phoenix. Releasing a new version without addressing these in a > minor release hurts our visibility in that these critical issues are not > addressed. > > Jira's that I'm involved with for example: I've already given a patch > several months ago for 5833 and there is a chance it may fix 5484. > https://issues.apache.org/jira/browse/PHOENIX-5833 > https://issues.apache.org/jira/browse/PHOENIX-5484 > > In addition, inspecting apache JIRA i see several other wrong result JIRAs > from the community. Some of these certainly are probably old issues or > incorrect understanding but some of these are opened by our own dev > community and are likely real problems. > https://issues.apache.org/jira/browse/PHOENIX-6217 > https://issues.apache.org/jira/browse/PHOENIX-5571 > https://issues.apache.org/jira/browse/PHOENIX-4642 > https://issues.apache.org/jira/browse/PHOENIX-4540 > https://issues.apache.org/jira/browse/PHOENIX-4504 > https://issues.apache.org/jira/browse/PHOENIX-4419 > https://issues.apache.org/jira/browse/PHOENIX-4116 > > What is our stance on this type of issue? Are we going to say these were > issues prior to 4.15 and not address them? Should we have requirements for > our releases to fix wrong results? > > Daniel Wong > > On Mon, Nov 30, 2020 at 7:30 PM Xinyi Yan wrote: > > > Hi all, > > > > It's time to discuss the Phoenix 4.16 release. After many people's > > contributions on the bug fixes, new features, and other works in the past > > few months, we are kind of close to the point to have a RC (still need to > > fix test flappers). Please let me know if you think any JIRA must be part > > of the Phoenix 4.16 release other than major blocker PHOENIX-5712. > > > > If no surprise comes up, I will not wait for any new major features and > > focus on the RC as soon as possible. > > > > Sincerely, > > Xinyi > > > > > -- > Daniel Wong > Salesforce > Mobile: 628.217.1808 >
Re: [Discuss] Releasing Phoenix 4.16
Hi, I wanted to bring up wrong results in Phoenix and some JIRAs around them that I think we should fix as the wrong result lessens the end user's trust in Phoenix. Releasing a new version without addressing these in a minor release hurts our visibility in that these critical issues are not addressed. Jira's that I'm involved with for example: I've already given a patch several months ago for 5833 and there is a chance it may fix 5484. https://issues.apache.org/jira/browse/PHOENIX-5833 https://issues.apache.org/jira/browse/PHOENIX-5484 In addition, inspecting apache JIRA i see several other wrong result JIRAs from the community. Some of these certainly are probably old issues or incorrect understanding but some of these are opened by our own dev community and are likely real problems. https://issues.apache.org/jira/browse/PHOENIX-6217 https://issues.apache.org/jira/browse/PHOENIX-5571 https://issues.apache.org/jira/browse/PHOENIX-4642 https://issues.apache.org/jira/browse/PHOENIX-4540 https://issues.apache.org/jira/browse/PHOENIX-4504 https://issues.apache.org/jira/browse/PHOENIX-4419 https://issues.apache.org/jira/browse/PHOENIX-4116 What is our stance on this type of issue? Are we going to say these were issues prior to 4.15 and not address them? Should we have requirements for our releases to fix wrong results? Daniel Wong On Mon, Nov 30, 2020 at 7:30 PM Xinyi Yan wrote: > Hi all, > > It's time to discuss the Phoenix 4.16 release. After many people's > contributions on the bug fixes, new features, and other works in the past > few months, we are kind of close to the point to have a RC (still need to > fix test flappers). Please let me know if you think any JIRA must be part > of the Phoenix 4.16 release other than major blocker PHOENIX-5712. > > If no surprise comes up, I will not wait for any new major features and > focus on the RC as soon as possible. > > Sincerely, > Xinyi > -- Daniel Wong Salesforce Mobile: 628.217.1808
[Discuss] Releasing Phoenix 4.16
Hi all, It's time to discuss the Phoenix 4.16 release. After many people's contributions on the bug fixes, new features, and other works in the past few months, we are kind of close to the point to have a RC (still need to fix test flappers). Please let me know if you think any JIRA must be part of the Phoenix 4.16 release other than major blocker PHOENIX-5712. If no surprise comes up, I will not wait for any new major features and focus on the RC as soon as possible. Sincerely, Xinyi
