Re: [NOTICE] Building trunk needs protoc 3.7.1
The new protobuf plugin related issues have all been pushed to trunk(though I think we'd better port them to all active branches). So what's the next step? Shade and relocated protobuf? HBase has already done this before so I do not think it will take too much time. If we all agree on the solution, I think we can finish this in one week. But maybe a problem is that, is it OK to upgrade protobuf in a minor release? Of course if we shade and relocate protobuf it will less hurt to users as they can depend on protobuf 2.5 explicitly if they want, but still a bit uncomfortable. Thanks. Wangda Tan 于2019年9月24日周二 上午2:29写道: > Hi Vinay, > > Thanks for the clarification. > > Do you have a timeline about all you described works w.r.t. the > compatibility will be completed? I'm asking this is because we need to > release 3.3.0 earlier if possible since there're 1k+ of patches in 3.3.0 > already, we should get it out earlier. > > If the PB work will take more time, do you think if we should create a > branch for 3.3, revert PB changes from branch-3.3, and keep on working on > PB for the next minor release? (or major release if we do see some > compatibility issue in the future). > > Just my $0.02 > > Thanks, > Wangda > > On Mon, Sep 23, 2019 at 5:43 AM Steve Loughran > > wrote: > > > aah, that makes sense > > > > On Sun, Sep 22, 2019 at 6:11 PM Vinayakumar B > > wrote: > > > > > Thanks Steve. > > > > > > Idea is not to shade all artifacts. > > > Instead maintain one artifact ( hadoop-thirdparty) which have all such > > > dependencies ( com.google.* may be), add this artifact as dependency > in > > > hadoop modules. Use shaded classes directly in the code of hadoop > modules > > > instead of shading at package phase. > > > > > > Hbase, ozone and ratis already following this way. The artifact ( > > > hadoop-thirdparty) with shaded dependencies can be maintained in a > > separate > > > repo as suggested by stack on HADOOP-13363 or could be maintained as a > > > separate module in Hadoop repo. If maintained in separate repo, need to > > > build this only when there are changes related to shaded dependencies. > > > > > > > > > -Vinay > > > > > > On Sun, 22 Sep 2019, 10:11 pm Steve Loughran, > > wrote: > > > > > > > > > > > > > > > On Sun, Sep 22, 2019 at 3:22 PM Vinayakumar B < > vinayakum...@apache.org > > > > > > > wrote: > > > > > > > >>Protobuf provides Wire compatibility between releases.. but not > > > >> guarantees the source compatibility in generated sources. There will > > be > > > a > > > >> problem in compatibility if anyone uses generated protobuf message > > > outside > > > >> of Hadoop modules. Which ideally shouldn't be as generated sources > are > > > not > > > >> public APIs. > > > >> > > > >>There should not be any compatibility problems between releases > in > > > >> terms > > > >> of communication provided both uses same syntax (proto2) of proto > > > message. > > > >> This I have verified by communication between protobuf 2.5.0 client > > with > > > >> protobuf 3.7.1 server. > > > >> > > > >>To avoid the downstream transitive dependency classpath problem, > > who > > > >> might be using protobuf 2.5.0 classes, planning to shade the 3.7.1 > > > classes > > > >> and its usages in all hadoop modules.. and keep 2.5.0 jar back in > > hadoop > > > >> classpath. > > > >> > > > >> Hope I have answered your question. > > > >> > > > >> -Vinay > > > >> > > > >> > > > > While I support the move and CP isolation, this is going to (finally) > > > > force us to make shaded versions of all artifacts which we publish > with > > > the > > > > intent of them being loaded on the classpath of other applications > > > > > > > > > >
Re: [NOTICE] Building trunk needs protoc 3.7.1
Hi Vinay, Thanks for the clarification. Do you have a timeline about all you described works w.r.t. the compatibility will be completed? I'm asking this is because we need to release 3.3.0 earlier if possible since there're 1k+ of patches in 3.3.0 already, we should get it out earlier. If the PB work will take more time, do you think if we should create a branch for 3.3, revert PB changes from branch-3.3, and keep on working on PB for the next minor release? (or major release if we do see some compatibility issue in the future). Just my $0.02 Thanks, Wangda On Mon, Sep 23, 2019 at 5:43 AM Steve Loughran wrote: > aah, that makes sense > > On Sun, Sep 22, 2019 at 6:11 PM Vinayakumar B > wrote: > > > Thanks Steve. > > > > Idea is not to shade all artifacts. > > Instead maintain one artifact ( hadoop-thirdparty) which have all such > > dependencies ( com.google.* may be), add this artifact as dependency in > > hadoop modules. Use shaded classes directly in the code of hadoop modules > > instead of shading at package phase. > > > > Hbase, ozone and ratis already following this way. The artifact ( > > hadoop-thirdparty) with shaded dependencies can be maintained in a > separate > > repo as suggested by stack on HADOOP-13363 or could be maintained as a > > separate module in Hadoop repo. If maintained in separate repo, need to > > build this only when there are changes related to shaded dependencies. > > > > > > -Vinay > > > > On Sun, 22 Sep 2019, 10:11 pm Steve Loughran, > wrote: > > > > > > > > > > > On Sun, Sep 22, 2019 at 3:22 PM Vinayakumar B > > > > wrote: > > > > > >>Protobuf provides Wire compatibility between releases.. but not > > >> guarantees the source compatibility in generated sources. There will > be > > a > > >> problem in compatibility if anyone uses generated protobuf message > > outside > > >> of Hadoop modules. Which ideally shouldn't be as generated sources are > > not > > >> public APIs. > > >> > > >>There should not be any compatibility problems between releases in > > >> terms > > >> of communication provided both uses same syntax (proto2) of proto > > message. > > >> This I have verified by communication between protobuf 2.5.0 client > with > > >> protobuf 3.7.1 server. > > >> > > >>To avoid the downstream transitive dependency classpath problem, > who > > >> might be using protobuf 2.5.0 classes, planning to shade the 3.7.1 > > classes > > >> and its usages in all hadoop modules.. and keep 2.5.0 jar back in > hadoop > > >> classpath. > > >> > > >> Hope I have answered your question. > > >> > > >> -Vinay > > >> > > >> > > > While I support the move and CP isolation, this is going to (finally) > > > force us to make shaded versions of all artifacts which we publish with > > the > > > intent of them being loaded on the classpath of other applications > > > > > >
Re: [NOTICE] Building trunk needs protoc 3.7.1
aah, that makes sense On Sun, Sep 22, 2019 at 6:11 PM Vinayakumar B wrote: > Thanks Steve. > > Idea is not to shade all artifacts. > Instead maintain one artifact ( hadoop-thirdparty) which have all such > dependencies ( com.google.* may be), add this artifact as dependency in > hadoop modules. Use shaded classes directly in the code of hadoop modules > instead of shading at package phase. > > Hbase, ozone and ratis already following this way. The artifact ( > hadoop-thirdparty) with shaded dependencies can be maintained in a separate > repo as suggested by stack on HADOOP-13363 or could be maintained as a > separate module in Hadoop repo. If maintained in separate repo, need to > build this only when there are changes related to shaded dependencies. > > > -Vinay > > On Sun, 22 Sep 2019, 10:11 pm Steve Loughran, wrote: > > > > > > > On Sun, Sep 22, 2019 at 3:22 PM Vinayakumar B > > wrote: > > > >>Protobuf provides Wire compatibility between releases.. but not > >> guarantees the source compatibility in generated sources. There will be > a > >> problem in compatibility if anyone uses generated protobuf message > outside > >> of Hadoop modules. Which ideally shouldn't be as generated sources are > not > >> public APIs. > >> > >>There should not be any compatibility problems between releases in > >> terms > >> of communication provided both uses same syntax (proto2) of proto > message. > >> This I have verified by communication between protobuf 2.5.0 client with > >> protobuf 3.7.1 server. > >> > >>To avoid the downstream transitive dependency classpath problem, who > >> might be using protobuf 2.5.0 classes, planning to shade the 3.7.1 > classes > >> and its usages in all hadoop modules.. and keep 2.5.0 jar back in hadoop > >> classpath. > >> > >> Hope I have answered your question. > >> > >> -Vinay > >> > >> > > While I support the move and CP isolation, this is going to (finally) > > force us to make shaded versions of all artifacts which we publish with > the > > intent of them being loaded on the classpath of other applications > > >
Re: [NOTICE] Building trunk needs protoc 3.7.1
Thanks Steve. Idea is not to shade all artifacts. Instead maintain one artifact ( hadoop-thirdparty) which have all such dependencies ( com.google.* may be), add this artifact as dependency in hadoop modules. Use shaded classes directly in the code of hadoop modules instead of shading at package phase. Hbase, ozone and ratis already following this way. The artifact ( hadoop-thirdparty) with shaded dependencies can be maintained in a separate repo as suggested by stack on HADOOP-13363 or could be maintained as a separate module in Hadoop repo. If maintained in separate repo, need to build this only when there are changes related to shaded dependencies. -Vinay On Sun, 22 Sep 2019, 10:11 pm Steve Loughran, wrote: > > > On Sun, Sep 22, 2019 at 3:22 PM Vinayakumar B > wrote: > >>Protobuf provides Wire compatibility between releases.. but not >> guarantees the source compatibility in generated sources. There will be a >> problem in compatibility if anyone uses generated protobuf message outside >> of Hadoop modules. Which ideally shouldn't be as generated sources are not >> public APIs. >> >>There should not be any compatibility problems between releases in >> terms >> of communication provided both uses same syntax (proto2) of proto message. >> This I have verified by communication between protobuf 2.5.0 client with >> protobuf 3.7.1 server. >> >>To avoid the downstream transitive dependency classpath problem, who >> might be using protobuf 2.5.0 classes, planning to shade the 3.7.1 classes >> and its usages in all hadoop modules.. and keep 2.5.0 jar back in hadoop >> classpath. >> >> Hope I have answered your question. >> >> -Vinay >> >> > While I support the move and CP isolation, this is going to (finally) > force us to make shaded versions of all artifacts which we publish with the > intent of them being loaded on the classpath of other applications >
Re: [NOTICE] Building trunk needs protoc 3.7.1
On Sun, Sep 22, 2019 at 3:22 PM Vinayakumar B wrote: >Protobuf provides Wire compatibility between releases.. but not > guarantees the source compatibility in generated sources. There will be a > problem in compatibility if anyone uses generated protobuf message outside > of Hadoop modules. Which ideally shouldn't be as generated sources are not > public APIs. > >There should not be any compatibility problems between releases in terms > of communication provided both uses same syntax (proto2) of proto message. > This I have verified by communication between protobuf 2.5.0 client with > protobuf 3.7.1 server. > >To avoid the downstream transitive dependency classpath problem, who > might be using protobuf 2.5.0 classes, planning to shade the 3.7.1 classes > and its usages in all hadoop modules.. and keep 2.5.0 jar back in hadoop > classpath. > > Hope I have answered your question. > > -Vinay > > While I support the move and CP isolation, this is going to (finally) force us to make shaded versions of all artifacts which we publish with the intent of them being loaded on the classpath of other applications
Re: [NOTICE] Building trunk needs protoc 3.7.1
Protobuf provides Wire compatibility between releases.. but not guarantees the source compatibility in generated sources. There will be a problem in compatibility if anyone uses generated protobuf message outside of Hadoop modules. Which ideally shouldn't be as generated sources are not public APIs. There should not be any compatibility problems between releases in terms of communication provided both uses same syntax (proto2) of proto message. This I have verified by communication between protobuf 2.5.0 client with protobuf 3.7.1 server. To avoid the downstream transitive dependency classpath problem, who might be using protobuf 2.5.0 classes, planning to shade the 3.7.1 classes and its usages in all hadoop modules.. and keep 2.5.0 jar back in hadoop classpath. Hope I have answered your question. -Vinay On Sun, 22 Sep 2019, 7:38 pm Vinod Kumar Vavilapalli, wrote: > Quick question, being lazy here, lots of JIRA updates on HADOOP-13363 over > the years not helping either. > > Does anyone know what this upgrade will mean w.r.t compatibility for the > Hadoop releases themselves? Remember that trunk is still 3.x. > > Thanks > +Vinod > > > On Sep 21, 2019, at 9:55 AM, Vinayakumar B > wrote: > > > > @Wangda Tan , > > Sorry for the confusion. HADOOP-13363 is umbrella jira to track multiple > > stages of protobuf upgrade in subtasks. (jar upgrade, Docker update, > plugin > > upgrade, shading, etc). > > Right now, first task of jar upgrade is done. So need to update the > protoc > > executable in the in build environments. > > > > @张铎(Duo Zhang) , > > Sorry for the inconvenience. Yes, indeed plugin update before jar upgrde > > was possible. Sorry I missed it. > > > > Plugin update needed to be done for whole project, for which precommit > > jenkins will need more time complete end-to-end runs. > > So plugin update is planned in stages in further subtasks. It could be > done > > in 2-3 days. > > > > -Vinay > > > > On Sat, 21 Sep 2019, 5:55 am 张铎(Duo Zhang), > wrote: > > > >> I think this one is alread in place so we have to upgrade... > >> > >> https://issues.apache.org/jira/browse/HADOOP-16557 > >> > >> Wangda Tan 于2019年9月21日周六 上午7:19写道: > >> > >>> Hi Vinay, > >>> > >>> A bit confused, I saw the HADOOP-13363 is still pending. Do we need to > >>> upgrade protobuf version to 3.7.1 NOW or once HADOOP-13363 is > completed? > >>> > >>> Thanks, > >>> Wangda > >>> > >>> On Fri, Sep 20, 2019 at 8:11 AM Vinayakumar B > > >>> wrote: > >>> > Hi All, > > A very long pending task, protobuf upgrade is happening in > >> HADOOP-13363. > >>> As > part of that protobuf version is upgraded to 3.7.1. > > Please update your build environments to have 3.7.1 protobuf version. > > BUILIDING.txt has been updated with latest instructions. > > This pre-requisite to update protoc dependecy manually is required > >> until > 'hadoop-maven-plugin' is replaced with 'protobuf-mavem-plugin' to > dynamically resolve required protoc exe. > > Dockerfile is being updated to have latest 3.7.1 as default protoc for > >>> test > environments. > > Thanks, > -Vinay > > >>> > >> > >
Re: [NOTICE] Building trunk needs protoc 3.7.1
Quick question, being lazy here, lots of JIRA updates on HADOOP-13363 over the years not helping either. Does anyone know what this upgrade will mean w.r.t compatibility for the Hadoop releases themselves? Remember that trunk is still 3.x. Thanks +Vinod > On Sep 21, 2019, at 9:55 AM, Vinayakumar B wrote: > > @Wangda Tan , > Sorry for the confusion. HADOOP-13363 is umbrella jira to track multiple > stages of protobuf upgrade in subtasks. (jar upgrade, Docker update, plugin > upgrade, shading, etc). > Right now, first task of jar upgrade is done. So need to update the protoc > executable in the in build environments. > > @张铎(Duo Zhang) , > Sorry for the inconvenience. Yes, indeed plugin update before jar upgrde > was possible. Sorry I missed it. > > Plugin update needed to be done for whole project, for which precommit > jenkins will need more time complete end-to-end runs. > So plugin update is planned in stages in further subtasks. It could be done > in 2-3 days. > > -Vinay > > On Sat, 21 Sep 2019, 5:55 am 张铎(Duo Zhang), wrote: > >> I think this one is alread in place so we have to upgrade... >> >> https://issues.apache.org/jira/browse/HADOOP-16557 >> >> Wangda Tan 于2019年9月21日周六 上午7:19写道: >> >>> Hi Vinay, >>> >>> A bit confused, I saw the HADOOP-13363 is still pending. Do we need to >>> upgrade protobuf version to 3.7.1 NOW or once HADOOP-13363 is completed? >>> >>> Thanks, >>> Wangda >>> >>> On Fri, Sep 20, 2019 at 8:11 AM Vinayakumar B >>> wrote: >>> Hi All, A very long pending task, protobuf upgrade is happening in >> HADOOP-13363. >>> As part of that protobuf version is upgraded to 3.7.1. Please update your build environments to have 3.7.1 protobuf version. BUILIDING.txt has been updated with latest instructions. This pre-requisite to update protoc dependecy manually is required >> until 'hadoop-maven-plugin' is replaced with 'protobuf-mavem-plugin' to dynamically resolve required protoc exe. Dockerfile is being updated to have latest 3.7.1 as default protoc for >>> test environments. Thanks, -Vinay >>> >> - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: [NOTICE] Building trunk needs protoc 3.7.1
@Wangda Tan , Sorry for the confusion. HADOOP-13363 is umbrella jira to track multiple stages of protobuf upgrade in subtasks. (jar upgrade, Docker update, plugin upgrade, shading, etc). Right now, first task of jar upgrade is done. So need to update the protoc executable in the in build environments. @张铎(Duo Zhang) , Sorry for the inconvenience. Yes, indeed plugin update before jar upgrde was possible. Sorry I missed it. Plugin update needed to be done for whole project, for which precommit jenkins will need more time complete end-to-end runs. So plugin update is planned in stages in further subtasks. It could be done in 2-3 days. -Vinay On Sat, 21 Sep 2019, 5:55 am 张铎(Duo Zhang), wrote: > I think this one is alread in place so we have to upgrade... > > https://issues.apache.org/jira/browse/HADOOP-16557 > > Wangda Tan 于2019年9月21日周六 上午7:19写道: > > > Hi Vinay, > > > > A bit confused, I saw the HADOOP-13363 is still pending. Do we need to > > upgrade protobuf version to 3.7.1 NOW or once HADOOP-13363 is completed? > > > > Thanks, > > Wangda > > > > On Fri, Sep 20, 2019 at 8:11 AM Vinayakumar B > > wrote: > > > > > Hi All, > > > > > > A very long pending task, protobuf upgrade is happening in > HADOOP-13363. > > As > > > part of that protobuf version is upgraded to 3.7.1. > > > > > > Please update your build environments to have 3.7.1 protobuf version. > > > > > > BUILIDING.txt has been updated with latest instructions. > > > > > > This pre-requisite to update protoc dependecy manually is required > until > > > 'hadoop-maven-plugin' is replaced with 'protobuf-mavem-plugin' to > > > dynamically resolve required protoc exe. > > > > > > Dockerfile is being updated to have latest 3.7.1 as default protoc for > > test > > > environments. > > > > > > Thanks, > > > -Vinay > > > > > >
Re: [NOTICE] Building trunk needs protoc 3.7.1
I think this one is alread in place so we have to upgrade... https://issues.apache.org/jira/browse/HADOOP-16557 Wangda Tan 于2019年9月21日周六 上午7:19写道: > Hi Vinay, > > A bit confused, I saw the HADOOP-13363 is still pending. Do we need to > upgrade protobuf version to 3.7.1 NOW or once HADOOP-13363 is completed? > > Thanks, > Wangda > > On Fri, Sep 20, 2019 at 8:11 AM Vinayakumar B > wrote: > > > Hi All, > > > > A very long pending task, protobuf upgrade is happening in HADOOP-13363. > As > > part of that protobuf version is upgraded to 3.7.1. > > > > Please update your build environments to have 3.7.1 protobuf version. > > > > BUILIDING.txt has been updated with latest instructions. > > > > This pre-requisite to update protoc dependecy manually is required until > > 'hadoop-maven-plugin' is replaced with 'protobuf-mavem-plugin' to > > dynamically resolve required protoc exe. > > > > Dockerfile is being updated to have latest 3.7.1 as default protoc for > test > > environments. > > > > Thanks, > > -Vinay > > >
Re: [NOTICE] Building trunk needs protoc 3.7.1
As suggested before, please consider switching to maven protobuf plugin first before upgrading the protobuf dependency. https://www.xolstice.org/protobuf-maven-plugin/examples/protoc-artifact.html This plugin will download the protoc binary automatically so we do not need to have developers install protoc manually on their machine. Wangda Tan 于2019年9月21日周六 上午7:19写道: > Hi Vinay, > > A bit confused, I saw the HADOOP-13363 is still pending. Do we need to > upgrade protobuf version to 3.7.1 NOW or once HADOOP-13363 is completed? > > Thanks, > Wangda > > On Fri, Sep 20, 2019 at 8:11 AM Vinayakumar B > wrote: > > > Hi All, > > > > A very long pending task, protobuf upgrade is happening in HADOOP-13363. > As > > part of that protobuf version is upgraded to 3.7.1. > > > > Please update your build environments to have 3.7.1 protobuf version. > > > > BUILIDING.txt has been updated with latest instructions. > > > > This pre-requisite to update protoc dependecy manually is required until > > 'hadoop-maven-plugin' is replaced with 'protobuf-mavem-plugin' to > > dynamically resolve required protoc exe. > > > > Dockerfile is being updated to have latest 3.7.1 as default protoc for > test > > environments. > > > > Thanks, > > -Vinay > > >
Re: [NOTICE] Building trunk needs protoc 3.7.1
Hi Vinay, A bit confused, I saw the HADOOP-13363 is still pending. Do we need to upgrade protobuf version to 3.7.1 NOW or once HADOOP-13363 is completed? Thanks, Wangda On Fri, Sep 20, 2019 at 8:11 AM Vinayakumar B wrote: > Hi All, > > A very long pending task, protobuf upgrade is happening in HADOOP-13363. As > part of that protobuf version is upgraded to 3.7.1. > > Please update your build environments to have 3.7.1 protobuf version. > > BUILIDING.txt has been updated with latest instructions. > > This pre-requisite to update protoc dependecy manually is required until > 'hadoop-maven-plugin' is replaced with 'protobuf-mavem-plugin' to > dynamically resolve required protoc exe. > > Dockerfile is being updated to have latest 3.7.1 as default protoc for test > environments. > > Thanks, > -Vinay >
[NOTICE] Building trunk needs protoc 3.7.1
Hi All, A very long pending task, protobuf upgrade is happening in HADOOP-13363. As part of that protobuf version is upgraded to 3.7.1. Please update your build environments to have 3.7.1 protobuf version. BUILIDING.txt has been updated with latest instructions. This pre-requisite to update protoc dependecy manually is required until 'hadoop-maven-plugin' is replaced with 'protobuf-mavem-plugin' to dynamically resolve required protoc exe. Dockerfile is being updated to have latest 3.7.1 as default protoc for test environments. Thanks, -Vinay