... If a Bigtop
feature is to build any arbitrary version of a hadoop package (Use git to
build) should we support version specific patches?
I think that to build any arbitrary version of a hadoop package is not a goal.
Bigtop provides working combination of products.
Specific version of Bigtop concerns only specific version of Hadoop and other
products.
On 2021/06/03 21:56, Matt Andruff wrote:
Thank you for the support.
My idea was (naively) that I could build from Apache source code the same
Hadoop versions as what HDP defined as a release. Specifically the goal is
to **not** use HDP artifacts, so not a true clone but a look-a-like. This
would give a good transition point to a team that was on HDP to move to
BigTop. Then we could help migrate them to a BigTop release in the future.
My goal is to show that BigTop is the successor to HDP.
Removing patches from bigtop-packages/src/common/<component>/ only works
on your first try. (Clean git pull) After that the patches get
sticky.(You can delete them but they still get applied.) If a Bigtop
feature is to build any arbitrary version of a hadoop package (Use git to
build) should we support version specific patches?
I'm happy to help do the work to make this happen, but I want to make sure
it's in alignment with the project goals.
On Thu, Jun 3, 2021 at 4:44 AM Jun HE <ju...@apache.org> wrote:
I'm a little confused. So the idea is to build source code from HDP using
Bigtop? If that is true, then as Masatake mentioned removing a patch is not
enough. You may need to change do-component-build, install_<component>, and
even those svc and configuration files, to make it work with existing HDP
components.
Masatake Iwasaki <iwasak...@oss.nttdata.co.jp> 于2021年6月3日周四 下午2:36写道：
I want to build an old version of HDP.
I think you need some considerations.
* Products of HDP x.y.z is expected to be built against products of HDP
x.y.z.
The versions of products of HDP and Bigtop do not match.
You will need to fix scripts of Bigtop for resolving mismatch.
(do-component-build and install_*.sh tend to be relevant.)
* Even if you succeeded to build Bigtop package from HDP source code,
the contents should be quite different from HDP one.
(You should just use .srpm of HDP for building .rpm of HDP if
available.)
* Artifacts of HDP usually depend on HDP specific artifacts
published in Hortonworks Maven repositories.
(This could be resolved by just tweaking ~/.m2/settings.xml or pom.xml
of products.)
Practical example: *In master:* I want to build spark 2.3.0. If I use
bigtop.bom or git to build both fail when they try to apply
patch2-snappy-java-1.1.8.diff.
The master branch of Bigtop is expected to build Spark 3 against Hadoop
3.
https://github.com/apache/bigtop/commit/f47d4bcb16e67f50b2403f4fc5071b60ec88c081
Just removing patches should not be enough.
On 2021/06/03 10:26, Jun HE wrote:
Hi Matt,
Are you just trying to disable a certain patch temporarily? If yes, my
understanding is that just delete that patch
in bigtop-packages/src/common/<component>/ would work. Pls have a try.
Regards,
Jun
Matt Andruff <m...@andruffsolutions.com> 于2021年6月3日周四 上午4:16写道：
Good Day Team,
I want to build an old version of HDP. Theoretically I should be able
to
do that with BigTop, but I'm having issues because patches are running
and
not finding what they're looking for and failing my build.
How do I turn off a patch intended for a different version (say spark
2.3.0)?
Practical example: *In master:* I want to build spark 2.3.0. If I
use
bigtop.bom or git to build both fail when they try to apply
patch2-snappy-java-1.1.8.diff.
Which is totally needed for a fix in some later version of spark, but
not
the version I'm using. As far as I can tell, there isn't a way to turn
on/off specific version patches, or perhaps this patch isn't following
the
paradigm.
Should I open a ticket to create a new functionality that only applies
patches to specific versions? Or is this already baked in and I have
hit a
subtle bug?
Thoughts?
