Firstly we'll not multiple select's right..? We can have distro-select
and conf-select which can be included as part of the bigtop code. User's
can adopt it and even they can write their own. And user will not deploy
with multiple selects right, as they can get the conflicts..
If it is the tool to switch distribution to Bigtop and
does not care about other distributions,
hosting it in Bigtop as bigtop-select would make sense.
If the tool need to care about distribution other than Bigtop,
it should be under Ambari.
On 2022/07/04 23:39, Battula, Brahma Reddy wrote:
Could you elaborate the reason why we need the *-select if so?
Firstly we'll not multiple select's right..? We can have distro-select
and conf-select which can be included as part of the bigtop code. User's
can adopt it and even they can write their own. And user will not deploy
with multiple selects right, as they can get the conflicts..
It does not make sense.
While Hadoop rpm/deb are built with Bigtop, Hadoop is independent
project.
As of today rpm's built from bigtop which are used to deploy with ambari
right which is single source...Even projects are independent, Might not be
the best way to create rpm's from project's like Hadoop right?
In the past, we used to get hdp-select rpm
> | downloaded and installed from HDP repositories
As HDP repo's will not available now, what's your suggestion for any new
user who want to adopt their own stack..?
On 04/07/22, 4:37 PM, "Masatake Iwasaki" <[email protected]>
wrote:
> > Multiple versions of the same package (deb/rpm) can not be
installed at the same time.
> I dn't think, we'll required to install multiple versions same
time. Even this can be handled.
Could you elaborate the reason why we need the *-select if so?
> IMO, as components rpm's are build with bigtop, having it in
bigtop will be perfect choice. As we need to change the spec's.
It does not make sense.
While Hadoop rpm/deb are built with Bigtop, Hadoop is independent
project.
The following description implies that the *-select is updated
along with Ambari
and Ambari is the perfect place to host it.
| These scripts are a basic necessity in the Ambari framework for
the
| installation of various Bigdata packages. The only major changes
they would
| receive is when we onboard new services and components to Ambari,
else they
| usually do not receive updates. In the past, we used to get
hdp-select rpm
| downloaded and installed from HDP repositories.
On 2022/07/04 19:35, Battula, Brahma Reddy wrote:
>> Multiple versions of the same package (deb/rpm) can not be
installed at the same time.
> I dn't think, we'll required to install multiple versions same
time. Even this can be handled.
> Anyway Bigtop will be used to create the RPM's not for deploying
the cluster with ambari now.
>
>> I feel hdp-select/bdp-select should belongs to Ambari
>> since it is assuming the convention specific to HDP/BDP/Ambari.
> > We can easily add packaging for the code under Ambari
> > (as ambari-bdp-select for example) based on the existing
stuff under
> > bigtop-packages/*/ambari.
>
> IMO, as components rpm's are build with bigtop, having it in
bigtop will be perfect choice. As we need to change the spec's.
>
>
>
> On 03/07/22, 9:56 AM, "Masatake Iwasaki" <
[email protected]> wrote:
>
> > For example, assuming that the following stack is
installed:
> >
> > /usr/hdp/SOME_VERSION/hadoop/...
> > /spark/...
> > /zookeeper/...
>
> Multiple versions of the same package (deb/rpm) can not be
installed at the same time.
> IIRC, HDP uses convention in which the product version is
part of package name
> for addressing this. like::
>
> $ rpm -qi hadoop_2_6_1_0_129
> Name : hadoop_2_6_1_0_129
> Version : 2.7.3.2.6.1.0
> ...
>
>
> > So if we accept bdp-select, we also have to develop a new
feature to
> > change install directories of the packages,
> > keeping the current default for avoiding user-facing
incompatibility.
> > Otherwise bdp-select itself is useless.
>
> I don't like to bring the awkward package name convention to
Bigtop
> at least by default.
>
> I feel hdp-select/bdp-select should belongs to Ambari
> since it is assuming the convention specific to
HDP/BDP/Ambari.
> We can easily add packaging for the code under Ambari
> (as ambari-bdp-select for example) based on the existing
stuff under
> bigtop-packages/*/ambari.
>
>
> Masatake Iwasaki
>
> On 2022/07/03 7:17, Kengo Seki wrote:
> > Thank you for proposing this, Viraj and Brahma.
> > This proposal may feel sudden to someone, so let me
explain some context,
> > mainly to the Bigtop community.
> >
> > Ambari went to Attic once due to inactivity,
> > but some of its users and developers are reviving it now.
> > From the Bigtop community, Roman and Evans undertook the
role of
> > initial PMC members,
> > and Jun, Yuqi, Ganesh, Masatake, and I are also going to
support it.
> >
> > The default stack of Ambari was HDP, but it has gone
behind paywall,
> > so we have to add an alternative stack in the next release
of Ambari.
> > Viraj and Brahma are planning to leverage Bigtop packages
for that purpose,
> > and that stack is tentatively named BDP (BigData Platform).
> >
> > We would also like to take over a feature called
hdp-select,
> > which enables users to switch the current stack easily
with symlink.
> > For example, assuming that the following stack is
installed:
> >
> > /usr/hdp/SOME_VERSION/hadoop/...
> > /spark/...
> > /zookeeper/...
> >
> > /etc/hadoop, /var/lib/hadoop, /usr/lib/hadoop are symlinks
of
> > /usr/hdp/SOME_VERSION/hadoop/etc,
/usr/hdp/SOME_VERSION/hadoop/var/lib,
> > /usr/hdp/SOME_VERSION/hadoop/usr/lib respectively.
> >
> > Then the new stack is installed under /usr/hdp/NEW_VERSION,
> > users can run hdp-select to switch the symlinks to the new
directories
> > for upgrading the stack.
> > (Is my understanding correct? Correct me if I'm wrong,
Viraj and Brahma)
> >
> > hdp-select was provided as a RPM/DEB package just like
other packages
> > HDP provided,
> > so it is user-friendly to provide bdp-select (the BDP
version of
> > hdp-select) as well.
> > This is the background of Viraj's proposal.
> >
> > But there is a problem here to leverage Bigtop packages in
this manner.
> > Bigtop respects Linux's Filesystem Hierarchy Standard,
> > so it doesn't install files into a single place
(/usr/hdp/VERSION in
> > the example above),
> > but directly into /etc, /usr, /var and so on.
> > So if we accept bdp-select, we also have to develop a new
feature to
> > change install directories of the packages,
> > keeping the current default for avoiding user-facing
incompatibility.
> > Otherwise bdp-select itself is useless.
> > But it will be a tough work, because installation paths
are hard-coded
> > in the RPM specs and DEB packaging scripts now.
> >
> > Based on the above, how about creating a new branch for
adding hdp-select
> > and developing the feature to switch installation paths?
> > After finishing the work on that branch and ensuring that
it works expectedly,
> > we can merge it into master, I think.
> >
> > Kengo Seki <[email protected]>
> >
> > On Thu, Jun 30, 2022 at 4:21 PM Battula, Brahma Reddy
> > <[email protected]> wrote:
> >>
> >> Thanks @Viraj Jasani to start discussing on this.
> >>
> >> IMO, For upcoming ambari release(2.7.7) , we should have
select other than HDP so that people can start using the ambari without HDP.
> >>
> >> I am +1(non-binding) on this approach to have selects in
bigtop code. May be for long term , we can think for tarballs or mpacks
which can go in mabari trunk.
> >>
> >>
> >> On 29/06/22, 10:22 AM, "Viraj Jasani" <[email protected]>
wrote:
> >>
> >> Hi Ambari/Bigtop dev,
> >>
> >> As per the new roadmap of Apache Ambari, we would
like to propose moving
> >> certain scripts (previously known as hdp-select and
conf-select) to Bigtop
> >> so that their rpm installation could be managed
independently.
> >> These scripts are a basic necessity in the Ambari
framework for the
> >> installation of various Bigdata packages. The only
major changes they would
> >> receive is when we onboard new services and
components to Ambari, else they
> >> usually do not receive updates. In the past, we used
to get hdp-select rpm
> >> downloaded and installed from HDP repositories.
> >>
> >> We have still not officially finalized the new
replacement name for
> >> hdp-select, it would likely be bdp-select (bdp:
BigData Platform). However,
> >> the purpose of bdp-select and conf-select remains
the same.
> >>
>
