Re: Turn off patch
Hi Matt, We moved away from CDH 5 (hadoop 2.6.x, hive 1.x, etc..) to Bigtop (1.4 initially then 1.5) very recently. My team ended up simply swapping the CDH packages with the Bigtop ones, adapting to some changes (mostly related to services like Hive) and upgrading HDFS in place. It took a long time to prepare for the transition/upgrade (months), but we are very happy about the result. If you need any info please reach out, I'd be very happy to help (we are also prepping a wiki page with suggestions/gotchas/pitfalls about transitioning to Bigtop and it would be really nice if somebody else went through it to validate it). Luca On Thu, Jun 3, 2021 at 2:56 PM Matt Andruff wrote: > > Thank you for the support. > > My idea was (naively) that I could build from Apache source code the same > Hadoop versions as what HDP defined as a release. Specifically the goal is > to **not** use HDP artifacts, so not a true clone but a look-a-like. This > would give a good transition point to a team that was on HDP to move to > BigTop. Then we could help migrate them to a BigTop release in the future. > > My goal is to show that BigTop is the successor to HDP. > > Removing patches from bigtop-packages/src/common// only works > on your first try. (Clean git pull) After that the patches get > sticky.(You can delete them but they still get applied.) If a Bigtop > feature is to build any arbitrary version of a hadoop package (Use git to > build) should we support version specific patches? > > I'm happy to help do the work to make this happen, but I want to make sure > it's in alignment with the project goals. > > On Thu, Jun 3, 2021 at 4:44 AM Jun HE wrote: > > > I'm a little confused. So the idea is to build source code from HDP using > > Bigtop? If that is true, then as Masatake mentioned removing a patch is not > > enough. You may need to change do-component-build, install_, and > > even those svc and configuration files, to make it work with existing HDP > > components. > > > > Masatake Iwasaki 于2021年6月3日周四 下午2:36写道： > > > > > > I want to build an old version of HDP. > > > > > > I think you need some considerations. > > > > > > * Products of HDP x.y.z is expected to be built against products of HDP > > > x.y.z. > > >The versions of products of HDP and Bigtop do not match. > > >You will need to fix scripts of Bigtop for resolving mismatch. > > >(do-component-build and install_*.sh tend to be relevant.) > > > > > > * Even if you succeeded to build Bigtop package from HDP source code, > > >the contents should be quite different from HDP one. > > >(You should just use .srpm of HDP for building .rpm of HDP if > > > available.) > > > > > > * Artifacts of HDP usually depend on HDP specific artifacts > > >published in Hortonworks Maven repositories. > > >(This could be resolved by just tweaking ~/.m2/settings.xml or pom.xml > > > of products.) > > > > > > > > > > Practical example: *In master:* I want to build spark 2.3.0. If I use > > > > bigtop.bom or git to build both fail when they try to apply > > > > patch2-snappy-java-1.1.8.diff. > > > > > > The master branch of Bigtop is expected to build Spark 3 against Hadoop > > 3. > > > > > > > > https://github.com/apache/bigtop/commit/f47d4bcb16e67f50b2403f4fc5071b60ec88c081 > > > > > > Just removing patches should not be enough. > > > > > > > > > On 2021/06/03 10:26, Jun HE wrote: > > > > Hi Matt, > > > > > > > > Are you just trying to disable a certain patch temporarily? If yes, my > > > > understanding is that just delete that patch > > > > in bigtop-packages/src/common// would work. Pls have a try. > > > > > > > > Regards, > > > > > > > > Jun > > > > > > > > Matt Andruff 于2021年6月3日周四 上午4:16写道： > > > > > > > >> Good Day Team, > > > >> > > > >> I want to build an old version of HDP. Theoretically I should be able > > > to > > > >> do that with BigTop, but I'm having issues because patches are running > > > and > > > >> not finding what they're looking for and failing my build. > > > >> > > > >> How do I turn off a patch intended for a different version (say spark > > > >> 2.3.0)? > > > >> > > > >> Practical example: *In master:* I want to build spark 2.3.0. If I > > use > > > >> bigtop.bom or git to build both fail when they try to apply > > > >> patch2-snappy-java-1.1.8.diff. > > > >> Which is totally needed for a fix in some later version of spark, but > > > not > > > >> the version I'm using. As far as I can tell, there isn't a way to turn > > > >> on/off specific version patches, or perhaps this patch isn't following > > > the > > > >> paradigm. > > > >> > > > >> Should I open a ticket to create a new functionality that only applies > > > >> patches to specific versions? Or is this already baked in and I have > > > hit a > > > >> subtle bug? > > > >> Thoughts? > > > >> > > > > > > > > >
Re: Turn off patch
... If a Bigtop feature is to build any arbitrary version of a hadoop package (Use git to build) should we support version specific patches? I think that to build any arbitrary version of a hadoop package is not a goal. Bigtop provides working combination of products. Specific version of Bigtop concerns only specific version of Hadoop and other products. On 2021/06/03 21:56, Matt Andruff wrote: Thank you for the support. My idea was (naively) that I could build from Apache source code the same Hadoop versions as what HDP defined as a release. Specifically the goal is to **not** use HDP artifacts, so not a true clone but a look-a-like. This would give a good transition point to a team that was on HDP to move to BigTop. Then we could help migrate them to a BigTop release in the future. My goal is to show that BigTop is the successor to HDP. Removing patches from bigtop-packages/src/common// only works on your first try. (Clean git pull) After that the patches get sticky.(You can delete them but they still get applied.) If a Bigtop feature is to build any arbitrary version of a hadoop package (Use git to build) should we support version specific patches? I'm happy to help do the work to make this happen, but I want to make sure it's in alignment with the project goals. On Thu, Jun 3, 2021 at 4:44 AM Jun HE wrote: I'm a little confused. So the idea is to build source code from HDP using Bigtop? If that is true, then as Masatake mentioned removing a patch is not enough. You may need to change do-component-build, install_, and even those svc and configuration files, to make it work with existing HDP components. Masatake Iwasaki 于2021年6月3日周四 下午2:36写道： I want to build an old version of HDP. I think you need some considerations. * Products of HDP x.y.z is expected to be built against products of HDP x.y.z. The versions of products of HDP and Bigtop do not match. You will need to fix scripts of Bigtop for resolving mismatch. (do-component-build and install_*.sh tend to be relevant.) * Even if you succeeded to build Bigtop package from HDP source code, the contents should be quite different from HDP one. (You should just use .srpm of HDP for building .rpm of HDP if available.) * Artifacts of HDP usually depend on HDP specific artifacts published in Hortonworks Maven repositories. (This could be resolved by just tweaking ~/.m2/settings.xml or pom.xml of products.) Practical example: *In master:* I want to build spark 2.3.0. If I use bigtop.bom or git to build both fail when they try to apply patch2-snappy-java-1.1.8.diff. The master branch of Bigtop is expected to build Spark 3 against Hadoop 3. https://github.com/apache/bigtop/commit/f47d4bcb16e67f50b2403f4fc5071b60ec88c081 Just removing patches should not be enough. On 2021/06/03 10:26, Jun HE wrote: Hi Matt, Are you just trying to disable a certain patch temporarily? If yes, my understanding is that just delete that patch in bigtop-packages/src/common// would work. Pls have a try. Regards, Jun Matt Andruff 于2021年6月3日周四 上午4:16写道： Good Day Team, I want to build an old version of HDP. Theoretically I should be able to do that with BigTop, but I'm having issues because patches are running and not finding what they're looking for and failing my build. How do I turn off a patch intended for a different version (say spark 2.3.0)? Practical example: *In master:* I want to build spark 2.3.0. If I use bigtop.bom or git to build both fail when they try to apply patch2-snappy-java-1.1.8.diff. Which is totally needed for a fix in some later version of spark, but not the version I'm using. As far as I can tell, there isn't a way to turn on/off specific version patches, or perhaps this patch isn't following the paradigm. Should I open a ticket to create a new functionality that only applies patches to specific versions? Or is this already baked in and I have hit a subtle bug? Thoughts?
Re: Turn off patch
Thank you for the support. My idea was (naively) that I could build from Apache source code the same Hadoop versions as what HDP defined as a release. Specifically the goal is to **not** use HDP artifacts, so not a true clone but a look-a-like. This would give a good transition point to a team that was on HDP to move to BigTop. Then we could help migrate them to a BigTop release in the future. My goal is to show that BigTop is the successor to HDP. Removing patches from bigtop-packages/src/common// only works on your first try. (Clean git pull) After that the patches get sticky.(You can delete them but they still get applied.) If a Bigtop feature is to build any arbitrary version of a hadoop package (Use git to build) should we support version specific patches? I'm happy to help do the work to make this happen, but I want to make sure it's in alignment with the project goals. On Thu, Jun 3, 2021 at 4:44 AM Jun HE wrote: > I'm a little confused. So the idea is to build source code from HDP using > Bigtop? If that is true, then as Masatake mentioned removing a patch is not > enough. You may need to change do-component-build, install_, and > even those svc and configuration files, to make it work with existing HDP > components. > > Masatake Iwasaki 于2021年6月3日周四 下午2:36写道： > > > > I want to build an old version of HDP. > > > > I think you need some considerations. > > > > * Products of HDP x.y.z is expected to be built against products of HDP > > x.y.z. > >The versions of products of HDP and Bigtop do not match. > >You will need to fix scripts of Bigtop for resolving mismatch. > >(do-component-build and install_*.sh tend to be relevant.) > > > > * Even if you succeeded to build Bigtop package from HDP source code, > >the contents should be quite different from HDP one. > >(You should just use .srpm of HDP for building .rpm of HDP if > > available.) > > > > * Artifacts of HDP usually depend on HDP specific artifacts > >published in Hortonworks Maven repositories. > >(This could be resolved by just tweaking ~/.m2/settings.xml or pom.xml > > of products.) > > > > > > > Practical example: *In master:* I want to build spark 2.3.0. If I use > > > bigtop.bom or git to build both fail when they try to apply > > > patch2-snappy-java-1.1.8.diff. > > > > The master branch of Bigtop is expected to build Spark 3 against Hadoop > 3. > > > > > https://github.com/apache/bigtop/commit/f47d4bcb16e67f50b2403f4fc5071b60ec88c081 > > > > Just removing patches should not be enough. > > > > > > On 2021/06/03 10:26, Jun HE wrote: > > > Hi Matt, > > > > > > Are you just trying to disable a certain patch temporarily? If yes, my > > > understanding is that just delete that patch > > > in bigtop-packages/src/common// would work. Pls have a try. > > > > > > Regards, > > > > > > Jun > > > > > > Matt Andruff 于2021年6月3日周四 上午4:16写道： > > > > > >> Good Day Team, > > >> > > >> I want to build an old version of HDP. Theoretically I should be able > > to > > >> do that with BigTop, but I'm having issues because patches are running > > and > > >> not finding what they're looking for and failing my build. > > >> > > >> How do I turn off a patch intended for a different version (say spark > > >> 2.3.0)? > > >> > > >> Practical example: *In master:* I want to build spark 2.3.0. If I > use > > >> bigtop.bom or git to build both fail when they try to apply > > >> patch2-snappy-java-1.1.8.diff. > > >> Which is totally needed for a fix in some later version of spark, but > > not > > >> the version I'm using. As far as I can tell, there isn't a way to turn > > >> on/off specific version patches, or perhaps this patch isn't following > > the > > >> paradigm. > > >> > > >> Should I open a ticket to create a new functionality that only applies > > >> patches to specific versions? Or is this already baked in and I have > > hit a > > >> subtle bug? > > >> Thoughts? > > >> > > > > > >
Re: Turn off patch
I'm a little confused. So the idea is to build source code from HDP using Bigtop? If that is true, then as Masatake mentioned removing a patch is not enough. You may need to change do-component-build, install_, and even those svc and configuration files, to make it work with existing HDP components. Masatake Iwasaki 于2021年6月3日周四 下午2:36写道： > > I want to build an old version of HDP. > > I think you need some considerations. > > * Products of HDP x.y.z is expected to be built against products of HDP > x.y.z. >The versions of products of HDP and Bigtop do not match. >You will need to fix scripts of Bigtop for resolving mismatch. >(do-component-build and install_*.sh tend to be relevant.) > > * Even if you succeeded to build Bigtop package from HDP source code, >the contents should be quite different from HDP one. >(You should just use .srpm of HDP for building .rpm of HDP if > available.) > > * Artifacts of HDP usually depend on HDP specific artifacts >published in Hortonworks Maven repositories. >(This could be resolved by just tweaking ~/.m2/settings.xml or pom.xml > of products.) > > > > Practical example: *In master:* I want to build spark 2.3.0. If I use > > bigtop.bom or git to build both fail when they try to apply > > patch2-snappy-java-1.1.8.diff. > > The master branch of Bigtop is expected to build Spark 3 against Hadoop 3. > > https://github.com/apache/bigtop/commit/f47d4bcb16e67f50b2403f4fc5071b60ec88c081 > > Just removing patches should not be enough. > > > On 2021/06/03 10:26, Jun HE wrote: > > Hi Matt, > > > > Are you just trying to disable a certain patch temporarily? If yes, my > > understanding is that just delete that patch > > in bigtop-packages/src/common// would work. Pls have a try. > > > > Regards, > > > > Jun > > > > Matt Andruff 于2021年6月3日周四 上午4:16写道： > > > >> Good Day Team, > >> > >> I want to build an old version of HDP. Theoretically I should be able > to > >> do that with BigTop, but I'm having issues because patches are running > and > >> not finding what they're looking for and failing my build. > >> > >> How do I turn off a patch intended for a different version (say spark > >> 2.3.0)? > >> > >> Practical example: *In master:* I want to build spark 2.3.0. If I use > >> bigtop.bom or git to build both fail when they try to apply > >> patch2-snappy-java-1.1.8.diff. > >> Which is totally needed for a fix in some later version of spark, but > not > >> the version I'm using. As far as I can tell, there isn't a way to turn > >> on/off specific version patches, or perhaps this patch isn't following > the > >> paradigm. > >> > >> Should I open a ticket to create a new functionality that only applies > >> patches to specific versions? Or is this already baked in and I have > hit a > >> subtle bug? > >> Thoughts? > >> > > >
Re: Turn off patch
I want to build an old version of HDP. I think you need some considerations. * Products of HDP x.y.z is expected to be built against products of HDP x.y.z. The versions of products of HDP and Bigtop do not match. You will need to fix scripts of Bigtop for resolving mismatch. (do-component-build and install_*.sh tend to be relevant.) * Even if you succeeded to build Bigtop package from HDP source code, the contents should be quite different from HDP one. (You should just use .srpm of HDP for building .rpm of HDP if available.) * Artifacts of HDP usually depend on HDP specific artifacts published in Hortonworks Maven repositories. (This could be resolved by just tweaking ~/.m2/settings.xml or pom.xml of products.) Practical example: *In master:* I want to build spark 2.3.0. If I use bigtop.bom or git to build both fail when they try to apply patch2-snappy-java-1.1.8.diff. The master branch of Bigtop is expected to build Spark 3 against Hadoop 3. https://github.com/apache/bigtop/commit/f47d4bcb16e67f50b2403f4fc5071b60ec88c081 Just removing patches should not be enough. On 2021/06/03 10:26, Jun HE wrote: Hi Matt, Are you just trying to disable a certain patch temporarily? If yes, my understanding is that just delete that patch in bigtop-packages/src/common// would work. Pls have a try. Regards, Jun Matt Andruff 于2021年6月3日周四 上午4:16写道： Good Day Team, I want to build an old version of HDP. Theoretically I should be able to do that with BigTop, but I'm having issues because patches are running and not finding what they're looking for and failing my build. How do I turn off a patch intended for a different version (say spark 2.3.0)? Practical example: *In master:* I want to build spark 2.3.0. If I use bigtop.bom or git to build both fail when they try to apply patch2-snappy-java-1.1.8.diff. Which is totally needed for a fix in some later version of spark, but not the version I'm using. As far as I can tell, there isn't a way to turn on/off specific version patches, or perhaps this patch isn't following the paradigm. Should I open a ticket to create a new functionality that only applies patches to specific versions? Or is this already baked in and I have hit a subtle bug? Thoughts?
Re: Turn off patch
Hi Matt, Are you just trying to disable a certain patch temporarily? If yes, my understanding is that just delete that patch in bigtop-packages/src/common// would work. Pls have a try. Regards, Jun Matt Andruff 于2021年6月3日周四 上午4:16写道： > Good Day Team, > > I want to build an old version of HDP. Theoretically I should be able to > do that with BigTop, but I'm having issues because patches are running and > not finding what they're looking for and failing my build. > > How do I turn off a patch intended for a different version (say spark > 2.3.0)? > > Practical example: *In master:* I want to build spark 2.3.0. If I use > bigtop.bom or git to build both fail when they try to apply > patch2-snappy-java-1.1.8.diff. > Which is totally needed for a fix in some later version of spark, but not > the version I'm using. As far as I can tell, there isn't a way to turn > on/off specific version patches, or perhaps this patch isn't following the > paradigm. > > Should I open a ticket to create a new functionality that only applies > patches to specific versions? Or is this already baked in and I have hit a > subtle bug? > Thoughts? >
