Re: [DISUCSS] Fix and republish the pulsar-all image for Pulsar 3.0.0
I agree with Zike that the root cause of this specific case is that 3.0 changes the docker images artifacts and we don't update the process. Before we correct the process to ensure that human can follow, automation only automatically make mistakes. tison 于2023年6月2日 周五00:53写道： > The ASF only counts the source releases the official releases. All > binaries including docker images should be for convenience only. > > Even the image needs to go through a vote, you can of course build and > stage the image with any method in hand. The problem is where we can share > a build environment - Since that building pulsar-all is a bit resource > consuming, it can requires extra platform resource. > > Zixuan Liu 于2023年6月1日 周四23:47写道： > >> > The best solution is to make the build repeatable. IIUC, that will let >> us build all of the artifacts using CI instead of personal machines, >> which removes this class of errors. >> >> I agree! But I don't know if Apache rules allow this operation, and if >> so, I recommend using the CI to publish our project. >> >> Thanks, >> Zixuan >> >> >> Zike Yang 于2023年6月1日周四 18:45写道： >> > >> > > The best solution is to make the build repeatable. IIUC, that will let >> > us build all of the artifacts using CI instead of personal machines, >> > which removes this class of errors. >> > >> > I don't think this issue is related to where the docker image was >> > built. It has nothing to do with caching and nothing to do with the >> > machine on which it is built. It's just because the build uses the >> > `latest` tag, which points to an older version. >> > I'm + 0.5 for this solution. I am currently wondering how much reward >> > it will bring. >> > >> > > If we are able to prepare the docker image from the RELEASED tarbal >> > and connectors then I am fine with it. >> > >> > We currently build our images using compiled tarball rather than >> > rebuilding from sources. You can check the code here: >> > >> https://github.com/apache/pulsar/blob/5e6e6cebcdbeec32ed49729f658f2d5cd0d98347/docker/pulsar/Dockerfile#L25-L26 >> > >> > BR, >> > Zike Yang >> > >> > On Thu, Jun 1, 2023 at 4:52 PM Asaf Mesika >> wrote: >> > > >> > > I understand that you can't build a release process due to Apache >> > > foundation rules, that makes it mandatory to release something you >> built on >> > > your own machine. >> > > >> > > >> > > On Wed, May 31, 2023 at 5:54 PM Enrico Olivelli >> wrote: >> > > >> > > > Il giorno mer 31 mag 2023 alle ore 16:50 Michael Marshall >> > > > ha scritto: >> > > > > >> > > > > The best solution is to make the build repeatable. IIUC, that >> will let >> > > > > us build all of the artifacts using CI instead of personal >> machines, >> > > > > which removes this class of errors. >> > > > > >> > > > > That being said, I don't know how much effort it would be to >> achieve. >> > > > > >> > > > > Thanks, >> > > > > Michael >> > > > > >> > > > > On Tue, May 30, 2023 at 2:24 AM Zike Yang >> wrote: >> > > > > > >> > > > > > Hi, Enrico >> > > > > > >> > > > > > > When we ran the VOTE and we provided the docker images, were >> they >> > > > > > already broken ? >> > > > > > >> > > > > > Actually, they are not broken unless we use the new features of >> Pulsar >> > > > 3.0.0. >> > > > >> > > > It is not Pulsar 3 but Pulsar 2.11. >> > > > So for users this is a problem >> > > > >> > > > If we are able to prepare the docker image from the RELEASED tarbal >> > > > and connectors then I am fine with it. >> > > > If you have to rebuild from the sources I strongly believe that we >> > > > cannot do it without a proper release process >> > > > >> > > > Enrico >> > > > >> > > > >> > > > >> > > > > > >> > > > > > I think we need something like verification test scripts, to >> verify >> > > > > > the release candidate. For example, we use the image provided >> in the >> > > > > > RC to run the integration tests. And we need to make sure that >> we have >> > > > > > tested the newly added feature for the RC docker image. >> > > > > > >> > > > > > What do you think? >> > > > > > >> > > > > > BR, >> > > > > > Zike Yang >> > > > > > >> > > > > > On Tue, May 30, 2023 at 3:14 PM Zike Yang >> wrote: >> > > > > > > >> > > > > > > Hi, Asaf >> > > > > > > >> > > > > > > > How do you suggest we prevent it from happening next time? >> > > > > > > >> > > > > > > I have pushed a PR to fix it: >> > > > https://github.com/apache/pulsar/pull/20435 >> > > > > > > This PR specifies the correct image name for `pulsar` image >> to build >> > > > pulsar-all. >> > > > > > > >> > > > > > > Note that, in the release of Pulsar 3.0, we build the docker >> image by >> > > > > > > executing the following command instead of the >> `docker/build.sh`: >> > > > > > > ``` >> > > > > > > mvn install -DUBUNTU_MIRROR= >> http://azure.archive.ubuntu.com/ubuntu/ >> > > > \ >> > > > > > > -DskipTests \ >> > > > > > > -Pdocker -Pdocker-push \ >> > > > > > > -Ddocker.platforms=linux/amd64,linux/arm64 \ >> > > > > >
Re: [DISUCSS] Fix and republish the pulsar-all image for Pulsar 3.0.0
The ASF only counts the source releases the official releases. All binaries including docker images should be for convenience only. Even the image needs to go through a vote, you can of course build and stage the image with any method in hand. The problem is where we can share a build environment - Since that building pulsar-all is a bit resource consuming, it can requires extra platform resource. Zixuan Liu 于2023年6月1日 周四23:47写道： > > The best solution is to make the build repeatable. IIUC, that will let > us build all of the artifacts using CI instead of personal machines, > which removes this class of errors. > > I agree! But I don't know if Apache rules allow this operation, and if > so, I recommend using the CI to publish our project. > > Thanks, > Zixuan > > > Zike Yang 于2023年6月1日周四 18:45写道： > > > > > The best solution is to make the build repeatable. IIUC, that will let > > us build all of the artifacts using CI instead of personal machines, > > which removes this class of errors. > > > > I don't think this issue is related to where the docker image was > > built. It has nothing to do with caching and nothing to do with the > > machine on which it is built. It's just because the build uses the > > `latest` tag, which points to an older version. > > I'm + 0.5 for this solution. I am currently wondering how much reward > > it will bring. > > > > > If we are able to prepare the docker image from the RELEASED tarbal > > and connectors then I am fine with it. > > > > We currently build our images using compiled tarball rather than > > rebuilding from sources. You can check the code here: > > > https://github.com/apache/pulsar/blob/5e6e6cebcdbeec32ed49729f658f2d5cd0d98347/docker/pulsar/Dockerfile#L25-L26 > > > > BR, > > Zike Yang > > > > On Thu, Jun 1, 2023 at 4:52 PM Asaf Mesika > wrote: > > > > > > I understand that you can't build a release process due to Apache > > > foundation rules, that makes it mandatory to release something you > built on > > > your own machine. > > > > > > > > > On Wed, May 31, 2023 at 5:54 PM Enrico Olivelli > wrote: > > > > > > > Il giorno mer 31 mag 2023 alle ore 16:50 Michael Marshall > > > > ha scritto: > > > > > > > > > > The best solution is to make the build repeatable. IIUC, that will > let > > > > > us build all of the artifacts using CI instead of personal > machines, > > > > > which removes this class of errors. > > > > > > > > > > That being said, I don't know how much effort it would be to > achieve. > > > > > > > > > > Thanks, > > > > > Michael > > > > > > > > > > On Tue, May 30, 2023 at 2:24 AM Zike Yang wrote: > > > > > > > > > > > > Hi, Enrico > > > > > > > > > > > > > When we ran the VOTE and we provided the docker images, were > they > > > > > > already broken ? > > > > > > > > > > > > Actually, they are not broken unless we use the new features of > Pulsar > > > > 3.0.0. > > > > > > > > It is not Pulsar 3 but Pulsar 2.11. > > > > So for users this is a problem > > > > > > > > If we are able to prepare the docker image from the RELEASED tarbal > > > > and connectors then I am fine with it. > > > > If you have to rebuild from the sources I strongly believe that we > > > > cannot do it without a proper release process > > > > > > > > Enrico > > > > > > > > > > > > > > > > > > > > > > > > I think we need something like verification test scripts, to > verify > > > > > > the release candidate. For example, we use the image provided in > the > > > > > > RC to run the integration tests. And we need to make sure that > we have > > > > > > tested the newly added feature for the RC docker image. > > > > > > > > > > > > What do you think? > > > > > > > > > > > > BR, > > > > > > Zike Yang > > > > > > > > > > > > On Tue, May 30, 2023 at 3:14 PM Zike Yang > wrote: > > > > > > > > > > > > > > Hi, Asaf > > > > > > > > > > > > > > > How do you suggest we prevent it from happening next time? > > > > > > > > > > > > > > I have pushed a PR to fix it: > > > > https://github.com/apache/pulsar/pull/20435 > > > > > > > This PR specifies the correct image name for `pulsar` image to > build > > > > pulsar-all. > > > > > > > > > > > > > > Note that, in the release of Pulsar 3.0, we build the docker > image by > > > > > > > executing the following command instead of the > `docker/build.sh`: > > > > > > > ``` > > > > > > > mvn install -DUBUNTU_MIRROR= > http://azure.archive.ubuntu.com/ubuntu/ > > > > \ > > > > > > > -DskipTests \ > > > > > > > -Pdocker -Pdocker-push \ > > > > > > > -Ddocker.platforms=linux/amd64,linux/arm64 \ > > > > > > > -Ddocker.organization=snzkyang \ > > > > > > > -pl docker/pulsar,docker/pulsar-all > > > > > > > ``` > > > > > > > I think to take it a step further, we could fix these > > > > scripts(build.sh > > > > > > > and publish.sh) and use the shell scripts to build the image. > > > > > > > > > > > > > > I have verified the PR, and it works well. Please see more > detail in > > > > > > > the PR description. > >
Re: [DISUCSS] Fix and republish the pulsar-all image for Pulsar 3.0.0
> The best solution is to make the build repeatable. IIUC, that will let us build all of the artifacts using CI instead of personal machines, which removes this class of errors. I agree! But I don't know if Apache rules allow this operation, and if so, I recommend using the CI to publish our project. Thanks, Zixuan Zike Yang 于2023年6月1日周四 18:45写道： > > > The best solution is to make the build repeatable. IIUC, that will let > us build all of the artifacts using CI instead of personal machines, > which removes this class of errors. > > I don't think this issue is related to where the docker image was > built. It has nothing to do with caching and nothing to do with the > machine on which it is built. It's just because the build uses the > `latest` tag, which points to an older version. > I'm + 0.5 for this solution. I am currently wondering how much reward > it will bring. > > > If we are able to prepare the docker image from the RELEASED tarbal > and connectors then I am fine with it. > > We currently build our images using compiled tarball rather than > rebuilding from sources. You can check the code here: > https://github.com/apache/pulsar/blob/5e6e6cebcdbeec32ed49729f658f2d5cd0d98347/docker/pulsar/Dockerfile#L25-L26 > > BR, > Zike Yang > > On Thu, Jun 1, 2023 at 4:52 PM Asaf Mesika wrote: > > > > I understand that you can't build a release process due to Apache > > foundation rules, that makes it mandatory to release something you built on > > your own machine. > > > > > > On Wed, May 31, 2023 at 5:54 PM Enrico Olivelli wrote: > > > > > Il giorno mer 31 mag 2023 alle ore 16:50 Michael Marshall > > > ha scritto: > > > > > > > > The best solution is to make the build repeatable. IIUC, that will let > > > > us build all of the artifacts using CI instead of personal machines, > > > > which removes this class of errors. > > > > > > > > That being said, I don't know how much effort it would be to achieve. > > > > > > > > Thanks, > > > > Michael > > > > > > > > On Tue, May 30, 2023 at 2:24 AM Zike Yang wrote: > > > > > > > > > > Hi, Enrico > > > > > > > > > > > When we ran the VOTE and we provided the docker images, were they > > > > > already broken ? > > > > > > > > > > Actually, they are not broken unless we use the new features of Pulsar > > > 3.0.0. > > > > > > It is not Pulsar 3 but Pulsar 2.11. > > > So for users this is a problem > > > > > > If we are able to prepare the docker image from the RELEASED tarbal > > > and connectors then I am fine with it. > > > If you have to rebuild from the sources I strongly believe that we > > > cannot do it without a proper release process > > > > > > Enrico > > > > > > > > > > > > > > > > > > > I think we need something like verification test scripts, to verify > > > > > the release candidate. For example, we use the image provided in the > > > > > RC to run the integration tests. And we need to make sure that we have > > > > > tested the newly added feature for the RC docker image. > > > > > > > > > > What do you think? > > > > > > > > > > BR, > > > > > Zike Yang > > > > > > > > > > On Tue, May 30, 2023 at 3:14 PM Zike Yang wrote: > > > > > > > > > > > > Hi, Asaf > > > > > > > > > > > > > How do you suggest we prevent it from happening next time? > > > > > > > > > > > > I have pushed a PR to fix it: > > > https://github.com/apache/pulsar/pull/20435 > > > > > > This PR specifies the correct image name for `pulsar` image to build > > > pulsar-all. > > > > > > > > > > > > Note that, in the release of Pulsar 3.0, we build the docker image > > > > > > by > > > > > > executing the following command instead of the `docker/build.sh`: > > > > > > ``` > > > > > > mvn install -DUBUNTU_MIRROR=http://azure.archive.ubuntu.com/ubuntu/ > > > \ > > > > > > -DskipTests \ > > > > > > -Pdocker -Pdocker-push \ > > > > > > -Ddocker.platforms=linux/amd64,linux/arm64 \ > > > > > > -Ddocker.organization=snzkyang \ > > > > > > -pl docker/pulsar,docker/pulsar-all > > > > > > ``` > > > > > > I think to take it a step further, we could fix these > > > scripts(build.sh > > > > > > and publish.sh) and use the shell scripts to build the image. > > > > > > > > > > > > I have verified the PR, and it works well. Please see more detail in > > > > > > the PR description. > > > > > > > > > > > > Thanks, > > > > > > Zike Yang > > > > > > > > > > > > On Mon, May 29, 2023 at 9:50 PM Enrico Olivelli > > > > > > > > > wrote: > > > > > > > > > > > > > > I am really worried about the process. > > > > > > > > > > > > > > When we ran the VOTE and we provided the docker images, were they > > > > > > > already broken ? > > > > > > > > > > > > > > In any case we cannot overwrite those images, they have been > > > > > > > cached > > > > > > > all over the world now. > > > > > > > > > > > > > > It is safer to cut a new 3.0.1 release and run a VOTE. > > > > > > > > > > > > > > Maybe we can remove the old images, forever > > > > > > > > > > > > > > Enrico > > > >
Re: [DISUCSS] Fix and republish the pulsar-all image for Pulsar 3.0.0
> The best solution is to make the build repeatable. IIUC, that will let us build all of the artifacts using CI instead of personal machines, which removes this class of errors. I don't think this issue is related to where the docker image was built. It has nothing to do with caching and nothing to do with the machine on which it is built. It's just because the build uses the `latest` tag, which points to an older version. I'm + 0.5 for this solution. I am currently wondering how much reward it will bring. > If we are able to prepare the docker image from the RELEASED tarbal and connectors then I am fine with it. We currently build our images using compiled tarball rather than rebuilding from sources. You can check the code here: https://github.com/apache/pulsar/blob/5e6e6cebcdbeec32ed49729f658f2d5cd0d98347/docker/pulsar/Dockerfile#L25-L26 BR, Zike Yang On Thu, Jun 1, 2023 at 4:52 PM Asaf Mesika wrote: > > I understand that you can't build a release process due to Apache > foundation rules, that makes it mandatory to release something you built on > your own machine. > > > On Wed, May 31, 2023 at 5:54 PM Enrico Olivelli wrote: > > > Il giorno mer 31 mag 2023 alle ore 16:50 Michael Marshall > > ha scritto: > > > > > > The best solution is to make the build repeatable. IIUC, that will let > > > us build all of the artifacts using CI instead of personal machines, > > > which removes this class of errors. > > > > > > That being said, I don't know how much effort it would be to achieve. > > > > > > Thanks, > > > Michael > > > > > > On Tue, May 30, 2023 at 2:24 AM Zike Yang wrote: > > > > > > > > Hi, Enrico > > > > > > > > > When we ran the VOTE and we provided the docker images, were they > > > > already broken ? > > > > > > > > Actually, they are not broken unless we use the new features of Pulsar > > 3.0.0. > > > > It is not Pulsar 3 but Pulsar 2.11. > > So for users this is a problem > > > > If we are able to prepare the docker image from the RELEASED tarbal > > and connectors then I am fine with it. > > If you have to rebuild from the sources I strongly believe that we > > cannot do it without a proper release process > > > > Enrico > > > > > > > > > > > > > > I think we need something like verification test scripts, to verify > > > > the release candidate. For example, we use the image provided in the > > > > RC to run the integration tests. And we need to make sure that we have > > > > tested the newly added feature for the RC docker image. > > > > > > > > What do you think? > > > > > > > > BR, > > > > Zike Yang > > > > > > > > On Tue, May 30, 2023 at 3:14 PM Zike Yang wrote: > > > > > > > > > > Hi, Asaf > > > > > > > > > > > How do you suggest we prevent it from happening next time? > > > > > > > > > > I have pushed a PR to fix it: > > https://github.com/apache/pulsar/pull/20435 > > > > > This PR specifies the correct image name for `pulsar` image to build > > pulsar-all. > > > > > > > > > > Note that, in the release of Pulsar 3.0, we build the docker image by > > > > > executing the following command instead of the `docker/build.sh`: > > > > > ``` > > > > > mvn install -DUBUNTU_MIRROR=http://azure.archive.ubuntu.com/ubuntu/ > > \ > > > > > -DskipTests \ > > > > > -Pdocker -Pdocker-push \ > > > > > -Ddocker.platforms=linux/amd64,linux/arm64 \ > > > > > -Ddocker.organization=snzkyang \ > > > > > -pl docker/pulsar,docker/pulsar-all > > > > > ``` > > > > > I think to take it a step further, we could fix these > > scripts(build.sh > > > > > and publish.sh) and use the shell scripts to build the image. > > > > > > > > > > I have verified the PR, and it works well. Please see more detail in > > > > > the PR description. > > > > > > > > > > Thanks, > > > > > Zike Yang > > > > > > > > > > On Mon, May 29, 2023 at 9:50 PM Enrico Olivelli > > wrote: > > > > > > > > > > > > I am really worried about the process. > > > > > > > > > > > > When we ran the VOTE and we provided the docker images, were they > > > > > > already broken ? > > > > > > > > > > > > In any case we cannot overwrite those images, they have been cached > > > > > > all over the world now. > > > > > > > > > > > > It is safer to cut a new 3.0.1 release and run a VOTE. > > > > > > > > > > > > Maybe we can remove the old images, forever > > > > > > > > > > > > Enrico > > > > > > > > > > > > Il giorno lun 29 mag 2023 alle ore 13:55 Asaf Mesika > > > > > > ha scritto: > > > > > > > > > > > > > > Good catch! > > > > > > > > > > > > > > How do you suggest we prevent it from happening next time? > > > > > > > > > > > > > > On Mon, May 29, 2023 at 1:34 PM Zike Yang > > wrote: > > > > > > > > > > > > > > > Hi, all > > > > > > > > > > > > > > > > Recently, we found an issue with the `pulsar-all:3.0.0` image. > > The > > > > > > > > pulsar library included in `pulsar-all:3.0.0` is the version of > > > > > > > > 2.11.0: > > > > > > > > > > > > > > > > ``` > > > > > > > > docker run
Re: [DISUCSS] Fix and republish the pulsar-all image for Pulsar 3.0.0
I understand that you can't build a release process due to Apache foundation rules, that makes it mandatory to release something you built on your own machine. On Wed, May 31, 2023 at 5:54 PM Enrico Olivelli wrote: > Il giorno mer 31 mag 2023 alle ore 16:50 Michael Marshall > ha scritto: > > > > The best solution is to make the build repeatable. IIUC, that will let > > us build all of the artifacts using CI instead of personal machines, > > which removes this class of errors. > > > > That being said, I don't know how much effort it would be to achieve. > > > > Thanks, > > Michael > > > > On Tue, May 30, 2023 at 2:24 AM Zike Yang wrote: > > > > > > Hi, Enrico > > > > > > > When we ran the VOTE and we provided the docker images, were they > > > already broken ? > > > > > > Actually, they are not broken unless we use the new features of Pulsar > 3.0.0. > > It is not Pulsar 3 but Pulsar 2.11. > So for users this is a problem > > If we are able to prepare the docker image from the RELEASED tarbal > and connectors then I am fine with it. > If you have to rebuild from the sources I strongly believe that we > cannot do it without a proper release process > > Enrico > > > > > > > > > I think we need something like verification test scripts, to verify > > > the release candidate. For example, we use the image provided in the > > > RC to run the integration tests. And we need to make sure that we have > > > tested the newly added feature for the RC docker image. > > > > > > What do you think? > > > > > > BR, > > > Zike Yang > > > > > > On Tue, May 30, 2023 at 3:14 PM Zike Yang wrote: > > > > > > > > Hi, Asaf > > > > > > > > > How do you suggest we prevent it from happening next time? > > > > > > > > I have pushed a PR to fix it: > https://github.com/apache/pulsar/pull/20435 > > > > This PR specifies the correct image name for `pulsar` image to build > pulsar-all. > > > > > > > > Note that, in the release of Pulsar 3.0, we build the docker image by > > > > executing the following command instead of the `docker/build.sh`: > > > > ``` > > > > mvn install -DUBUNTU_MIRROR=http://azure.archive.ubuntu.com/ubuntu/ > \ > > > > -DskipTests \ > > > > -Pdocker -Pdocker-push \ > > > > -Ddocker.platforms=linux/amd64,linux/arm64 \ > > > > -Ddocker.organization=snzkyang \ > > > > -pl docker/pulsar,docker/pulsar-all > > > > ``` > > > > I think to take it a step further, we could fix these > scripts(build.sh > > > > and publish.sh) and use the shell scripts to build the image. > > > > > > > > I have verified the PR, and it works well. Please see more detail in > > > > the PR description. > > > > > > > > Thanks, > > > > Zike Yang > > > > > > > > On Mon, May 29, 2023 at 9:50 PM Enrico Olivelli > wrote: > > > > > > > > > > I am really worried about the process. > > > > > > > > > > When we ran the VOTE and we provided the docker images, were they > > > > > already broken ? > > > > > > > > > > In any case we cannot overwrite those images, they have been cached > > > > > all over the world now. > > > > > > > > > > It is safer to cut a new 3.0.1 release and run a VOTE. > > > > > > > > > > Maybe we can remove the old images, forever > > > > > > > > > > Enrico > > > > > > > > > > Il giorno lun 29 mag 2023 alle ore 13:55 Asaf Mesika > > > > > ha scritto: > > > > > > > > > > > > Good catch! > > > > > > > > > > > > How do you suggest we prevent it from happening next time? > > > > > > > > > > > > On Mon, May 29, 2023 at 1:34 PM Zike Yang > wrote: > > > > > > > > > > > > > Hi, all > > > > > > > > > > > > > > Recently, we found an issue with the `pulsar-all:3.0.0` image. > The > > > > > > > pulsar library included in `pulsar-all:3.0.0` is the version of > > > > > > > 2.11.0: > > > > > > > > > > > > > > ``` > > > > > > > docker run apachepulsar/pulsar-all:3.0.0 ls lib/ | grep > pulsar-broker > > > > > > > > > > > > > > org.apache.pulsar-pulsar-broker-2.11.0.jar > > > > > > > org.apache.pulsar-pulsar-broker-auth-sasl-2.11.0.jar > > > > > > > org.apache.pulsar-pulsar-broker-common-2.11.0.jar > > > > > > > ``` > > > > > > > > > > > > > > The root cause is that we use `apachepulsar/pulsar:latest` to > build > > > > > > > the `pulsar-all` image. But at the time of building Pulsar > 3.0.0, > > > > > > > `apachepulsar/pulsar:latest` was pointing to version 2.11.0. > > > > > > > > > > > > > > Therefore, the `pulsar-all:3.0.0` is actually a version 2.11.0 > of > > > > > > > Pulsar but with 3.0.0 connectors and offloaders. > > > > > > > > > > > > > > Please see more detail in this issue: > > > > > > > https://github.com/apache/pulsar/issues/20420 > > > > > > > > > > > > > > I have rebuilt the `pulsar-all:3.0.0` image: > > > > > > > > > > > > > > > https://hub.docker.com/layers/snzkyang/pulsar-all/3.0.0/images/sha256-833ea988bce8c704b179cc4c9c38fac8980e108b0bc67454e06c22927990b169?context=explore > > > > > > > > > > > > > > Please help and verify it. And check if there are any other > problems > > > > > > >
Re: [DISUCSS] Fix and republish the pulsar-all image for Pulsar 3.0.0
Il giorno mer 31 mag 2023 alle ore 16:50 Michael Marshall ha scritto: > > The best solution is to make the build repeatable. IIUC, that will let > us build all of the artifacts using CI instead of personal machines, > which removes this class of errors. > > That being said, I don't know how much effort it would be to achieve. > > Thanks, > Michael > > On Tue, May 30, 2023 at 2:24 AM Zike Yang wrote: > > > > Hi, Enrico > > > > > When we ran the VOTE and we provided the docker images, were they > > already broken ? > > > > Actually, they are not broken unless we use the new features of Pulsar > > 3.0.0. It is not Pulsar 3 but Pulsar 2.11. So for users this is a problem If we are able to prepare the docker image from the RELEASED tarbal and connectors then I am fine with it. If you have to rebuild from the sources I strongly believe that we cannot do it without a proper release process Enrico > > > > I think we need something like verification test scripts, to verify > > the release candidate. For example, we use the image provided in the > > RC to run the integration tests. And we need to make sure that we have > > tested the newly added feature for the RC docker image. > > > > What do you think? > > > > BR, > > Zike Yang > > > > On Tue, May 30, 2023 at 3:14 PM Zike Yang wrote: > > > > > > Hi, Asaf > > > > > > > How do you suggest we prevent it from happening next time? > > > > > > I have pushed a PR to fix it: https://github.com/apache/pulsar/pull/20435 > > > This PR specifies the correct image name for `pulsar` image to build > > > pulsar-all. > > > > > > Note that, in the release of Pulsar 3.0, we build the docker image by > > > executing the following command instead of the `docker/build.sh`: > > > ``` > > > mvn install -DUBUNTU_MIRROR=http://azure.archive.ubuntu.com/ubuntu/ \ > > > -DskipTests \ > > > -Pdocker -Pdocker-push \ > > > -Ddocker.platforms=linux/amd64,linux/arm64 \ > > > -Ddocker.organization=snzkyang \ > > > -pl docker/pulsar,docker/pulsar-all > > > ``` > > > I think to take it a step further, we could fix these scripts(build.sh > > > and publish.sh) and use the shell scripts to build the image. > > > > > > I have verified the PR, and it works well. Please see more detail in > > > the PR description. > > > > > > Thanks, > > > Zike Yang > > > > > > On Mon, May 29, 2023 at 9:50 PM Enrico Olivelli > > > wrote: > > > > > > > > I am really worried about the process. > > > > > > > > When we ran the VOTE and we provided the docker images, were they > > > > already broken ? > > > > > > > > In any case we cannot overwrite those images, they have been cached > > > > all over the world now. > > > > > > > > It is safer to cut a new 3.0.1 release and run a VOTE. > > > > > > > > Maybe we can remove the old images, forever > > > > > > > > Enrico > > > > > > > > Il giorno lun 29 mag 2023 alle ore 13:55 Asaf Mesika > > > > ha scritto: > > > > > > > > > > Good catch! > > > > > > > > > > How do you suggest we prevent it from happening next time? > > > > > > > > > > On Mon, May 29, 2023 at 1:34 PM Zike Yang wrote: > > > > > > > > > > > Hi, all > > > > > > > > > > > > Recently, we found an issue with the `pulsar-all:3.0.0` image. The > > > > > > pulsar library included in `pulsar-all:3.0.0` is the version of > > > > > > 2.11.0: > > > > > > > > > > > > ``` > > > > > > docker run apachepulsar/pulsar-all:3.0.0 ls lib/ | grep > > > > > > pulsar-broker > > > > > > > > > > > > org.apache.pulsar-pulsar-broker-2.11.0.jar > > > > > > org.apache.pulsar-pulsar-broker-auth-sasl-2.11.0.jar > > > > > > org.apache.pulsar-pulsar-broker-common-2.11.0.jar > > > > > > ``` > > > > > > > > > > > > The root cause is that we use `apachepulsar/pulsar:latest` to build > > > > > > the `pulsar-all` image. But at the time of building Pulsar 3.0.0, > > > > > > `apachepulsar/pulsar:latest` was pointing to version 2.11.0. > > > > > > > > > > > > Therefore, the `pulsar-all:3.0.0` is actually a version 2.11.0 of > > > > > > Pulsar but with 3.0.0 connectors and offloaders. > > > > > > > > > > > > Please see more detail in this issue: > > > > > > https://github.com/apache/pulsar/issues/20420 > > > > > > > > > > > > I have rebuilt the `pulsar-all:3.0.0` image: > > > > > > > > > > > > https://hub.docker.com/layers/snzkyang/pulsar-all/3.0.0/images/sha256-833ea988bce8c704b179cc4c9c38fac8980e108b0bc67454e06c22927990b169?context=explore > > > > > > > > > > > > Please help and verify it. And check if there are any other problems > > > > > > with the image. > > > > > > > > > > > > I'm going to publish the image to the `apachepulsar` organization to > > > > > > replace the old one. But before we do that, do we need a Vote or > > > > > > other > > > > > > ways to reach a consensus? Is there any problem if we replace the > > > > > > old > > > > > > image? > > > > > > > > > > > > Besides, I will also fix the docker build script to avoid similar > > > > > > issues. > > > > > > > > > > > > Thanks, >
Re: [DISUCSS] Fix and republish the pulsar-all image for Pulsar 3.0.0
The best solution is to make the build repeatable. IIUC, that will let us build all of the artifacts using CI instead of personal machines, which removes this class of errors. That being said, I don't know how much effort it would be to achieve. Thanks, Michael On Tue, May 30, 2023 at 2:24 AM Zike Yang wrote: > > Hi, Enrico > > > When we ran the VOTE and we provided the docker images, were they > already broken ? > > Actually, they are not broken unless we use the new features of Pulsar 3.0.0. > > I think we need something like verification test scripts, to verify > the release candidate. For example, we use the image provided in the > RC to run the integration tests. And we need to make sure that we have > tested the newly added feature for the RC docker image. > > What do you think? > > BR, > Zike Yang > > On Tue, May 30, 2023 at 3:14 PM Zike Yang wrote: > > > > Hi, Asaf > > > > > How do you suggest we prevent it from happening next time? > > > > I have pushed a PR to fix it: https://github.com/apache/pulsar/pull/20435 > > This PR specifies the correct image name for `pulsar` image to build > > pulsar-all. > > > > Note that, in the release of Pulsar 3.0, we build the docker image by > > executing the following command instead of the `docker/build.sh`: > > ``` > > mvn install -DUBUNTU_MIRROR=http://azure.archive.ubuntu.com/ubuntu/ \ > > -DskipTests \ > > -Pdocker -Pdocker-push \ > > -Ddocker.platforms=linux/amd64,linux/arm64 \ > > -Ddocker.organization=snzkyang \ > > -pl docker/pulsar,docker/pulsar-all > > ``` > > I think to take it a step further, we could fix these scripts(build.sh > > and publish.sh) and use the shell scripts to build the image. > > > > I have verified the PR, and it works well. Please see more detail in > > the PR description. > > > > Thanks, > > Zike Yang > > > > On Mon, May 29, 2023 at 9:50 PM Enrico Olivelli wrote: > > > > > > I am really worried about the process. > > > > > > When we ran the VOTE and we provided the docker images, were they > > > already broken ? > > > > > > In any case we cannot overwrite those images, they have been cached > > > all over the world now. > > > > > > It is safer to cut a new 3.0.1 release and run a VOTE. > > > > > > Maybe we can remove the old images, forever > > > > > > Enrico > > > > > > Il giorno lun 29 mag 2023 alle ore 13:55 Asaf Mesika > > > ha scritto: > > > > > > > > Good catch! > > > > > > > > How do you suggest we prevent it from happening next time? > > > > > > > > On Mon, May 29, 2023 at 1:34 PM Zike Yang wrote: > > > > > > > > > Hi, all > > > > > > > > > > Recently, we found an issue with the `pulsar-all:3.0.0` image. The > > > > > pulsar library included in `pulsar-all:3.0.0` is the version of > > > > > 2.11.0: > > > > > > > > > > ``` > > > > > docker run apachepulsar/pulsar-all:3.0.0 ls lib/ | grep pulsar-broker > > > > > > > > > > org.apache.pulsar-pulsar-broker-2.11.0.jar > > > > > org.apache.pulsar-pulsar-broker-auth-sasl-2.11.0.jar > > > > > org.apache.pulsar-pulsar-broker-common-2.11.0.jar > > > > > ``` > > > > > > > > > > The root cause is that we use `apachepulsar/pulsar:latest` to build > > > > > the `pulsar-all` image. But at the time of building Pulsar 3.0.0, > > > > > `apachepulsar/pulsar:latest` was pointing to version 2.11.0. > > > > > > > > > > Therefore, the `pulsar-all:3.0.0` is actually a version 2.11.0 of > > > > > Pulsar but with 3.0.0 connectors and offloaders. > > > > > > > > > > Please see more detail in this issue: > > > > > https://github.com/apache/pulsar/issues/20420 > > > > > > > > > > I have rebuilt the `pulsar-all:3.0.0` image: > > > > > > > > > > https://hub.docker.com/layers/snzkyang/pulsar-all/3.0.0/images/sha256-833ea988bce8c704b179cc4c9c38fac8980e108b0bc67454e06c22927990b169?context=explore > > > > > > > > > > Please help and verify it. And check if there are any other problems > > > > > with the image. > > > > > > > > > > I'm going to publish the image to the `apachepulsar` organization to > > > > > replace the old one. But before we do that, do we need a Vote or other > > > > > ways to reach a consensus? Is there any problem if we replace the old > > > > > image? > > > > > > > > > > Besides, I will also fix the docker build script to avoid similar > > > > > issues. > > > > > > > > > > Thanks, > > > > > Zike Yang > > > > >
Re: [DISUCSS] Fix and republish the pulsar-all image for Pulsar 3.0.0
Hi, Enrico > When we ran the VOTE and we provided the docker images, were they already broken ? Actually, they are not broken unless we use the new features of Pulsar 3.0.0. I think we need something like verification test scripts, to verify the release candidate. For example, we use the image provided in the RC to run the integration tests. And we need to make sure that we have tested the newly added feature for the RC docker image. What do you think? BR, Zike Yang On Tue, May 30, 2023 at 3:14 PM Zike Yang wrote: > > Hi, Asaf > > > How do you suggest we prevent it from happening next time? > > I have pushed a PR to fix it: https://github.com/apache/pulsar/pull/20435 > This PR specifies the correct image name for `pulsar` image to build > pulsar-all. > > Note that, in the release of Pulsar 3.0, we build the docker image by > executing the following command instead of the `docker/build.sh`: > ``` > mvn install -DUBUNTU_MIRROR=http://azure.archive.ubuntu.com/ubuntu/ \ > -DskipTests \ > -Pdocker -Pdocker-push \ > -Ddocker.platforms=linux/amd64,linux/arm64 \ > -Ddocker.organization=snzkyang \ > -pl docker/pulsar,docker/pulsar-all > ``` > I think to take it a step further, we could fix these scripts(build.sh > and publish.sh) and use the shell scripts to build the image. > > I have verified the PR, and it works well. Please see more detail in > the PR description. > > Thanks, > Zike Yang > > On Mon, May 29, 2023 at 9:50 PM Enrico Olivelli wrote: > > > > I am really worried about the process. > > > > When we ran the VOTE and we provided the docker images, were they > > already broken ? > > > > In any case we cannot overwrite those images, they have been cached > > all over the world now. > > > > It is safer to cut a new 3.0.1 release and run a VOTE. > > > > Maybe we can remove the old images, forever > > > > Enrico > > > > Il giorno lun 29 mag 2023 alle ore 13:55 Asaf Mesika > > ha scritto: > > > > > > Good catch! > > > > > > How do you suggest we prevent it from happening next time? > > > > > > On Mon, May 29, 2023 at 1:34 PM Zike Yang wrote: > > > > > > > Hi, all > > > > > > > > Recently, we found an issue with the `pulsar-all:3.0.0` image. The > > > > pulsar library included in `pulsar-all:3.0.0` is the version of > > > > 2.11.0: > > > > > > > > ``` > > > > docker run apachepulsar/pulsar-all:3.0.0 ls lib/ | grep pulsar-broker > > > > > > > > org.apache.pulsar-pulsar-broker-2.11.0.jar > > > > org.apache.pulsar-pulsar-broker-auth-sasl-2.11.0.jar > > > > org.apache.pulsar-pulsar-broker-common-2.11.0.jar > > > > ``` > > > > > > > > The root cause is that we use `apachepulsar/pulsar:latest` to build > > > > the `pulsar-all` image. But at the time of building Pulsar 3.0.0, > > > > `apachepulsar/pulsar:latest` was pointing to version 2.11.0. > > > > > > > > Therefore, the `pulsar-all:3.0.0` is actually a version 2.11.0 of > > > > Pulsar but with 3.0.0 connectors and offloaders. > > > > > > > > Please see more detail in this issue: > > > > https://github.com/apache/pulsar/issues/20420 > > > > > > > > I have rebuilt the `pulsar-all:3.0.0` image: > > > > > > > > https://hub.docker.com/layers/snzkyang/pulsar-all/3.0.0/images/sha256-833ea988bce8c704b179cc4c9c38fac8980e108b0bc67454e06c22927990b169?context=explore > > > > > > > > Please help and verify it. And check if there are any other problems > > > > with the image. > > > > > > > > I'm going to publish the image to the `apachepulsar` organization to > > > > replace the old one. But before we do that, do we need a Vote or other > > > > ways to reach a consensus? Is there any problem if we replace the old > > > > image? > > > > > > > > Besides, I will also fix the docker build script to avoid similar > > > > issues. > > > > > > > > Thanks, > > > > Zike Yang > > > >
Re: [DISUCSS] Fix and republish the pulsar-all image for Pulsar 3.0.0
Hi, Asaf > How do you suggest we prevent it from happening next time? I have pushed a PR to fix it: https://github.com/apache/pulsar/pull/20435 This PR specifies the correct image name for `pulsar` image to build pulsar-all. Note that, in the release of Pulsar 3.0, we build the docker image by executing the following command instead of the `docker/build.sh`: ``` mvn install -DUBUNTU_MIRROR=http://azure.archive.ubuntu.com/ubuntu/ \ -DskipTests \ -Pdocker -Pdocker-push \ -Ddocker.platforms=linux/amd64,linux/arm64 \ -Ddocker.organization=snzkyang \ -pl docker/pulsar,docker/pulsar-all ``` I think to take it a step further, we could fix these scripts(build.sh and publish.sh) and use the shell scripts to build the image. I have verified the PR, and it works well. Please see more detail in the PR description. Thanks, Zike Yang On Mon, May 29, 2023 at 9:50 PM Enrico Olivelli wrote: > > I am really worried about the process. > > When we ran the VOTE and we provided the docker images, were they > already broken ? > > In any case we cannot overwrite those images, they have been cached > all over the world now. > > It is safer to cut a new 3.0.1 release and run a VOTE. > > Maybe we can remove the old images, forever > > Enrico > > Il giorno lun 29 mag 2023 alle ore 13:55 Asaf Mesika > ha scritto: > > > > Good catch! > > > > How do you suggest we prevent it from happening next time? > > > > On Mon, May 29, 2023 at 1:34 PM Zike Yang wrote: > > > > > Hi, all > > > > > > Recently, we found an issue with the `pulsar-all:3.0.0` image. The > > > pulsar library included in `pulsar-all:3.0.0` is the version of > > > 2.11.0: > > > > > > ``` > > > docker run apachepulsar/pulsar-all:3.0.0 ls lib/ | grep pulsar-broker > > > > > > org.apache.pulsar-pulsar-broker-2.11.0.jar > > > org.apache.pulsar-pulsar-broker-auth-sasl-2.11.0.jar > > > org.apache.pulsar-pulsar-broker-common-2.11.0.jar > > > ``` > > > > > > The root cause is that we use `apachepulsar/pulsar:latest` to build > > > the `pulsar-all` image. But at the time of building Pulsar 3.0.0, > > > `apachepulsar/pulsar:latest` was pointing to version 2.11.0. > > > > > > Therefore, the `pulsar-all:3.0.0` is actually a version 2.11.0 of > > > Pulsar but with 3.0.0 connectors and offloaders. > > > > > > Please see more detail in this issue: > > > https://github.com/apache/pulsar/issues/20420 > > > > > > I have rebuilt the `pulsar-all:3.0.0` image: > > > > > > https://hub.docker.com/layers/snzkyang/pulsar-all/3.0.0/images/sha256-833ea988bce8c704b179cc4c9c38fac8980e108b0bc67454e06c22927990b169?context=explore > > > > > > Please help and verify it. And check if there are any other problems > > > with the image. > > > > > > I'm going to publish the image to the `apachepulsar` organization to > > > replace the old one. But before we do that, do we need a Vote or other > > > ways to reach a consensus? Is there any problem if we replace the old > > > image? > > > > > > Besides, I will also fix the docker build script to avoid similar issues. > > > > > > Thanks, > > > Zike Yang > > >
Re: [DISUCSS] Fix and republish the pulsar-all image for Pulsar 3.0.0
I am really worried about the process. When we ran the VOTE and we provided the docker images, were they already broken ? In any case we cannot overwrite those images, they have been cached all over the world now. It is safer to cut a new 3.0.1 release and run a VOTE. Maybe we can remove the old images, forever Enrico Il giorno lun 29 mag 2023 alle ore 13:55 Asaf Mesika ha scritto: > > Good catch! > > How do you suggest we prevent it from happening next time? > > On Mon, May 29, 2023 at 1:34 PM Zike Yang wrote: > > > Hi, all > > > > Recently, we found an issue with the `pulsar-all:3.0.0` image. The > > pulsar library included in `pulsar-all:3.0.0` is the version of > > 2.11.0: > > > > ``` > > docker run apachepulsar/pulsar-all:3.0.0 ls lib/ | grep pulsar-broker > > > > org.apache.pulsar-pulsar-broker-2.11.0.jar > > org.apache.pulsar-pulsar-broker-auth-sasl-2.11.0.jar > > org.apache.pulsar-pulsar-broker-common-2.11.0.jar > > ``` > > > > The root cause is that we use `apachepulsar/pulsar:latest` to build > > the `pulsar-all` image. But at the time of building Pulsar 3.0.0, > > `apachepulsar/pulsar:latest` was pointing to version 2.11.0. > > > > Therefore, the `pulsar-all:3.0.0` is actually a version 2.11.0 of > > Pulsar but with 3.0.0 connectors and offloaders. > > > > Please see more detail in this issue: > > https://github.com/apache/pulsar/issues/20420 > > > > I have rebuilt the `pulsar-all:3.0.0` image: > > > > https://hub.docker.com/layers/snzkyang/pulsar-all/3.0.0/images/sha256-833ea988bce8c704b179cc4c9c38fac8980e108b0bc67454e06c22927990b169?context=explore > > > > Please help and verify it. And check if there are any other problems > > with the image. > > > > I'm going to publish the image to the `apachepulsar` organization to > > replace the old one. But before we do that, do we need a Vote or other > > ways to reach a consensus? Is there any problem if we replace the old > > image? > > > > Besides, I will also fix the docker build script to avoid similar issues. > > > > Thanks, > > Zike Yang > >
Re: [DISUCSS] Fix and republish the pulsar-all image for Pulsar 3.0.0
Good catch! How do you suggest we prevent it from happening next time? On Mon, May 29, 2023 at 1:34 PM Zike Yang wrote: > Hi, all > > Recently, we found an issue with the `pulsar-all:3.0.0` image. The > pulsar library included in `pulsar-all:3.0.0` is the version of > 2.11.0: > > ``` > docker run apachepulsar/pulsar-all:3.0.0 ls lib/ | grep pulsar-broker > > org.apache.pulsar-pulsar-broker-2.11.0.jar > org.apache.pulsar-pulsar-broker-auth-sasl-2.11.0.jar > org.apache.pulsar-pulsar-broker-common-2.11.0.jar > ``` > > The root cause is that we use `apachepulsar/pulsar:latest` to build > the `pulsar-all` image. But at the time of building Pulsar 3.0.0, > `apachepulsar/pulsar:latest` was pointing to version 2.11.0. > > Therefore, the `pulsar-all:3.0.0` is actually a version 2.11.0 of > Pulsar but with 3.0.0 connectors and offloaders. > > Please see more detail in this issue: > https://github.com/apache/pulsar/issues/20420 > > I have rebuilt the `pulsar-all:3.0.0` image: > > https://hub.docker.com/layers/snzkyang/pulsar-all/3.0.0/images/sha256-833ea988bce8c704b179cc4c9c38fac8980e108b0bc67454e06c22927990b169?context=explore > > Please help and verify it. And check if there are any other problems > with the image. > > I'm going to publish the image to the `apachepulsar` organization to > replace the old one. But before we do that, do we need a Vote or other > ways to reach a consensus? Is there any problem if we replace the old > image? > > Besides, I will also fix the docker build script to avoid similar issues. > > Thanks, > Zike Yang >
[DISUCSS] Fix and republish the pulsar-all image for Pulsar 3.0.0
Hi, all Recently, we found an issue with the `pulsar-all:3.0.0` image. The pulsar library included in `pulsar-all:3.0.0` is the version of 2.11.0: ``` docker run apachepulsar/pulsar-all:3.0.0 ls lib/ | grep pulsar-broker org.apache.pulsar-pulsar-broker-2.11.0.jar org.apache.pulsar-pulsar-broker-auth-sasl-2.11.0.jar org.apache.pulsar-pulsar-broker-common-2.11.0.jar ``` The root cause is that we use `apachepulsar/pulsar:latest` to build the `pulsar-all` image. But at the time of building Pulsar 3.0.0, `apachepulsar/pulsar:latest` was pointing to version 2.11.0. Therefore, the `pulsar-all:3.0.0` is actually a version 2.11.0 of Pulsar but with 3.0.0 connectors and offloaders. Please see more detail in this issue: https://github.com/apache/pulsar/issues/20420 I have rebuilt the `pulsar-all:3.0.0` image: https://hub.docker.com/layers/snzkyang/pulsar-all/3.0.0/images/sha256-833ea988bce8c704b179cc4c9c38fac8980e108b0bc67454e06c22927990b169?context=explore Please help and verify it. And check if there are any other problems with the image. I'm going to publish the image to the `apachepulsar` organization to replace the old one. But before we do that, do we need a Vote or other ways to reach a consensus? Is there any problem if we replace the old image? Besides, I will also fix the docker build script to avoid similar issues. Thanks, Zike Yang
