Hi Matt, We moved away from CDH 5 (hadoop 2.6.x, hive 1.x, etc..) to Bigtop (1.4 initially then 1.5) very recently. My team ended up simply swapping the CDH packages with the Bigtop ones, adapting to some changes (mostly related to services like Hive) and upgrading HDFS in place. It took a long time to prepare for the transition/upgrade (months), but we are very happy about the result. If you need any info please reach out, I'd be very happy to help (we are also prepping a wiki page with suggestions/gotchas/pitfalls about transitioning to Bigtop and it would be really nice if somebody else went through it to validate it).
Luca On Thu, Jun 3, 2021 at 2:56 PM Matt Andruff <m...@andruffsolutions.com> wrote: > > Thank you for the support. > > My idea was (naively) that I could build from Apache source code the same > Hadoop versions as what HDP defined as a release. Specifically the goal is > to **not** use HDP artifacts, so not a true clone but a look-a-like. This > would give a good transition point to a team that was on HDP to move to > BigTop. Then we could help migrate them to a BigTop release in the future. > > My goal is to show that BigTop is the successor to HDP. > > Removing patches from bigtop-packages/src/common/<component>/ only works > on your first try. (Clean git pull) After that the patches get > sticky.(You can delete them but they still get applied.) If a Bigtop > feature is to build any arbitrary version of a hadoop package (Use git to > build) should we support version specific patches? > > I'm happy to help do the work to make this happen, but I want to make sure > it's in alignment with the project goals. > > On Thu, Jun 3, 2021 at 4:44 AM Jun HE <ju...@apache.org> wrote: > > > I'm a little confused. So the idea is to build source code from HDP using > > Bigtop? If that is true, then as Masatake mentioned removing a patch is not > > enough. You may need to change do-component-build, install_<component>, and > > even those svc and configuration files, to make it work with existing HDP > > components. > > > > Masatake Iwasaki <iwasak...@oss.nttdata.co.jp> 于2021年6月3日周四 下午2:36写道： > > > > > > I want to build an old version of HDP. > > > > > > I think you need some considerations. > > > > > > * Products of HDP x.y.z is expected to be built against products of HDP > > > x.y.z. > > > The versions of products of HDP and Bigtop do not match. > > > You will need to fix scripts of Bigtop for resolving mismatch. > > > (do-component-build and install_*.sh tend to be relevant.) > > > > > > * Even if you succeeded to build Bigtop package from HDP source code, > > > the contents should be quite different from HDP one. > > > (You should just use .srpm of HDP for building .rpm of HDP if > > > available.) > > > > > > * Artifacts of HDP usually depend on HDP specific artifacts > > > published in Hortonworks Maven repositories. > > > (This could be resolved by just tweaking ~/.m2/settings.xml or pom.xml > > > of products.) > > > > > > > > > > Practical example: *In master:* I want to build spark 2.3.0. If I use > > > > bigtop.bom or git to build both fail when they try to apply > > > > patch2-snappy-java-1.1.8.diff. > > > > > > The master branch of Bigtop is expected to build Spark 3 against Hadoop > > 3. > > > > > > > > https://github.com/apache/bigtop/commit/f47d4bcb16e67f50b2403f4fc5071b60ec88c081 > > > > > > Just removing patches should not be enough. > > > > > > > > > On 2021/06/03 10:26, Jun HE wrote: > > > > Hi Matt, > > > > > > > > Are you just trying to disable a certain patch temporarily? If yes, my > > > > understanding is that just delete that patch > > > > in bigtop-packages/src/common/<component>/ would work. Pls have a try. > > > > > > > > Regards, > > > > > > > > Jun > > > > > > > > Matt Andruff <m...@andruffsolutions.com> 于2021年6月3日周四 上午4:16写道： > > > > > > > >> Good Day Team, > > > >> > > > >> I want to build an old version of HDP. Theoretically I should be able > > > to > > > >> do that with BigTop, but I'm having issues because patches are running > > > and > > > >> not finding what they're looking for and failing my build. > > > >> > > > >> How do I turn off a patch intended for a different version (say spark > > > >> 2.3.0)? > > > >> > > > >> Practical example: *In master:* I want to build spark 2.3.0. If I > > use > > > >> bigtop.bom or git to build both fail when they try to apply > > > >> patch2-snappy-java-1.1.8.diff. > > > >> Which is totally needed for a fix in some later version of spark, but > > > not > > > >> the version I'm using. As far as I can tell, there isn't a way to turn > > > >> on/off specific version patches, or perhaps this patch isn't following > > > the > > > >> paradigm. > > > >> > > > >> Should I open a ticket to create a new functionality that only applies > > > >> patches to specific versions? Or is this already baked in and I have > > > hit a > > > >> subtle bug? > > > >> Thoughts? > > > >> > > > > > > > > >
