>> Assuming the understanding above is correct, my new concerns are:
Yes, Hope we'll in same page now. Still we can discuss in weekly call.
>> * Can we still call it as "HDP"? Doesn't it infringe Cloudera's
>> trademark?
>> If so, should we give a new stack name to it, as you mentioned as
>> planA in [3]?
>> (or it makes upgrading HDP cluster difficult? I don't understand it so
>> much)
Yes, it may be. Hence I introduced the planA and preferred it. one more
idea(bad) came in my mind that can we treat "HDP: Hadoop Data Platform", not
sure whether it should be ok or not(and we might not delivering as distro's).
>>* It may be a bit tough work to fix Bigtop's build scripts so that its
>> artifacts match Ambari's requirements.
>> (package versioning, file layouts, included and not included files,
>> file permissions, user/group creation, etc.)
Only paths matter right everything else can be re-used from the hdp-select.
On 20/06/22, 8:13 PM, "Kengo Seki" <sek...@apache.org> wrote:
Thanks for the elaboration Brahma, but I'm a bit confused.
At this point, my understanding is as follows. Are these correct?
* In the 2.7.x line, we'll still use a stack definition based on HDP's one
[1].
* But regarding RPM and DEB packages, we'll build them from source by
leveraging
Bigtop's build scripts and publish them on the ASF distribution site
or somewhere.
repoinfo.xml [2] will also be fixed to point that repository.
Assuming the understanding above is correct, my new concerns are:
* Can we still call it as "HDP"? Doesn't it infringe Cloudera's trademark?
If so, should we give a new stack name to it, as you mentioned as
planA in [3]?
(or it makes upgrading HDP cluster difficult? I don't understand it so
much)
* It may be a bit tough work to fix Bigtop's build scripts so that its
artifacts match Ambari's requirements.
(package versioning, file layouts, included and not included files,
file permissions, user/group creation, etc.)
I'm not sure if it's faster than developing Mpack.
[1]:
https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fambari%2Ftree%2Frelease-2.7.6%2Fambari-server%2Fsrc%2Fmain%2Fresources%2Fstacks%2FHDP&data=05%7C01%7Cbbattula%40visa.com%7C3ad9759845234dc6216008da52cb2e1b%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913329846408523%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=6cm2kZCvgvI2%2Fc%2BJCxhBbVyBlHgxcUvl2brLX3PDz%2Bg%3D&reserved=0
[2]:
https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fambari%2Fblob%2Frelease-2.7.6%2Fambari-server%2Fsrc%2Fmain%2Fresources%2Fstacks%2FHDP%2F2.6%2Frepos%2Frepoinfo.xml&data=05%7C01%7Cbbattula%40visa.com%7C3ad9759845234dc6216008da52cb2e1b%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913329846408523%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=pkYcSo4Emv9%2BUQdP8L9uzgsgApV3IIua%2Bxbc3VNW7fQ%3D&reserved=0
[3]:
https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fm77lhzo0njr2dhoock5twtm0c19j2py8&data=05%7C01%7Cbbattula%40visa.com%7C3ad9759845234dc6216008da52cb2e1b%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913329846408523%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=U1y4%2B%2FeckCFB2IMdgEei65DwI0rrEfCD7bFDHH3O2xQ%3D&reserved=0
Kengo Seki <sek...@gmail.com>
On Mon, Jun 20, 2022 at 8:05 PM Battula, Brahma Reddy
<bbatt...@visa.com.invalid> wrote:
>
> We'll not use the HDP Packages (Only selects we'll use from hdp, which
can be plugged in bigtop). Still we'll use opensource packages and bigtop to
create the RPM's and selects.
> Just we can use the source tar balls opensource components(e.g
Hadoop-3.2.3 source tarballs from apache repo) and build rpm's and publish in
following repo. Or from the apache github repo's.
>
>
https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdlcdn.apache.org%2Fbigtop%2Fbigtop-3.1.0%2Frepos%2F&data=05%7C01%7Cbbattula%40visa.com%7C3ad9759845234dc6216008da52cb2e1b%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913329846408523%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=6kBAOFoW%2BsBHM8b4KX68e0UZ%2B%2BHg7Oxewx%2BJcCEdRXg%3D&reserved=0
>
> Planning to have sync up call on this week, once it's finalize, we can
summarise here the approach. What do you think..?
>
>
> On 20/06/22, 4:06 PM, "Kengo Seki" <sek...@apache.org> wrote:
>
> Thanks for the reply, Brahma!
>
> > We'll use following repo, how other components in Apache are hosted.
>
> Using that site for distribution has no problem. My concern is, do we
> have the source tree for building HDP packages and does it still work
> without the artifacts behind the paywall?
> According to ASF's release policy [1], we must publish the
> corresponding source releases whenever we publish binary artifacts,
> such as RPM and DEB packages for HDP.
>
> [1]:
https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.apache.org%2Flegal%2Frelease-policy.html%23compiled-packages&data=05%7C01%7Cbbattula%40visa.com%7C3ad9759845234dc6216008da52cb2e1b%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913329846408523%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Cz%2BmZEjwjNnq9lICZmRtIDMLge9xZLQ0w3yK75AcHkc%3D&reserved=0
>
> Kengo Seki <sek...@apache.org>
>
> On Mon, Jun 20, 2022 at 6:19 PM Battula, Brahma Reddy
> <bbatt...@visa.com.invalid> wrote:
> >
> > Hi Kengo Seki,
> >
> > Thanks for pitching here.. We'll use following repo, how other
components in Apache are hosted. Any other suggestions..?
> >
> >
https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdlcdn.apache.org%2F&data=05%7C01%7Cbbattula%40visa.com%7C3ad9759845234dc6216008da52cb2e1b%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913329846408523%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=mQLNExpiw6fywDjtEbcUFWJlOuGJ6Ja45LWxjAWL6dY%3D&reserved=0
> >
> >
> >
> > On 20/06/22, 2:32 PM, "Kengo Seki" <sek...@apache.org> wrote:
> >
> > Oh, I've just noticed that I submitted an additional question
into the
> > wrong thread mistakenly.
> >
https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fj6kc2h1lphvjvsblcthcl5yzzmjk3tb5&data=05%7C01%7Cbbattula%40visa.com%7C3ad9759845234dc6216008da52cb2e1b%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913329846408523%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=R61kELeUwqNTvFHwQ5Gu4HuhnWuhrhPl07K40FLSrL0%3D&reserved=0
is
> > originally intended as a reply to this thread. Sorry for the
> > confusion.
> >
> > Kengo Seki <sek...@apache.org>
> >
> > On Mon, Jun 20, 2022 at 11:31 AM Kengo Seki <sek...@apache.org>
wrote:
> > >
> > > Thank you so much for writing the draft and commenting on it,
Brahma and Viraj!
> > > Firstly, a minor comment:
> > >
> > > > Trunk Code : Parallel we can work for the trunk code, this
we might give as 2.8.
> > > > * Working on bigtop mpack
> > >
> > > This means adding a new built-in stack which uses the Bigtop
> > > repository, and not developing Mpack on the Bigtop side,
right?
> > > If so, "bigtop mpack" sounds confusing a bit.
> > >
> > > > hdp-select can stay on branch-2.7 and it can be moved to
bigtop for trunk branch. Thoughts?
> > >
> > > Sounds good to me. During the 2.7.x releases, users can use
the HDP
> > > stack by default. In addition, they can also add the Bigtop
stack
> > > through the Mpack, as Zhiguo mentioned in [1].
> > > Since 2.8 or 3.0, we will drop HDP and users can use the
built-in
> > > Bigtop stack by default. It also allows us to drop the Mpack
on the
> > > Bigtop side.
> > >
> > > Regarding the built-in Bigtop stack developed on trunk, the
current
> > > BIGTOP stack in Ambari [2] may be a good start point to be
upgraded.
> > > I ensured that Ambari 2.7.6 was successfully built with the
> > > `-Dstack.distribution=BIGTOP` option, though Bigtop 0.8 is
too old to
> > > use for now.
> > >
> > > [1]:
https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fljnyymp2g9fg7cq4z7sfxd36bh8x7nlm&data=05%7C01%7Cbbattula%40visa.com%7C3ad9759845234dc6216008da52cb2e1b%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913329846408523%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=FOPQS1G0t6phSKTW8LIVxgJqUEWFOMlehrQgh7kVjv8%3D&reserved=0
> > > [2]:
https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fambari%2Ftree%2Frelease-2.7.6%2Fambari-server%2Fsrc%2Fmain%2Fresources%2Fstacks%2FBIGTOP%2F0.8&data=05%7C01%7Cbbattula%40visa.com%7C3ad9759845234dc6216008da52cb2e1b%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913329846408523%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=MvxxrjL0F%2B7wLfw5Qr9bstYRBuiwma5mPMEPurCN3vw%3D&reserved=0
> > >
> > > Kengo Seki <sek...@apache.org>
> > >
> > > On Mon, Jun 20, 2022 at 9:26 AM Viraj Jasani
<vjas...@apache.org> wrote:
> > > >
> > > > Thanks for the nice draft, Brahma!
> > > >
> > > > IMHO, as we narrow down to 2.7.7 release, it might be still
worth sticking
> > > > to HDP i.e. plugin the new stack for HDP for 2.7 release
line. For 2.8.0
> > > > (or 3.0.0), we can add entire new stack (BDP sounds good
one) and start off
> > > > with 2.8.0/3.0.0 stack version.
> > > >
> > > > hdp-select can stay on branch-2.7 and it can be moved to
bigtop for trunk
> > > > branch. Thoughts?
> > > >
> > > > Similarly, we can start-off with "python 3 only" support in
trunk, whereas
> > > > 2.7 release line can continue with python 2 to reduce
complexities for
> > > > maintenance releases on 2.7.
> > > >
> > > > All other points look good. Once again, thank you for
drafting the roadmap
> > > > and starting with this thread.
> > > >
> > > >
> > > > On Sun, Jun 19, 2022 at 10:07 AM Battula, Brahma Reddy
> > > > <bbatt...@visa.com.invalid> wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > As Ambari back now, let’s start discussing roadmap. I
will be closely
> > > > > tracking for Jira, GitHub(with Jenkins) and wiki to work
normal.
> > > > > Based on the conclusion of this thread will update
wiki[1].
> > > > >
> > > > > As 2.7.6 is released recently, we need to release 2.7.7
> > > > >
> > > > >
> > > > > 1. Stack Management. There are two approaches for
this. Open to discuss
> > > > > on following..
> > > > > * Entirely new stack
> > > > >
> > > > >
i.
> > > > > Define new stack name. (BDP: BigDataPlatform,
BDS:BigDataStack,
> > > > > OSS:OpenSourceStack,TDP: TuskerDataPlatform…)
> > > > >
> > > > >
ii.
> > > > > Define stack versions
> > > > > (zk-3.5.9,Hadoop-3.2.3,hive-3.1.3,tez-0.10.1,spark-3.2..)
> > > > >
> > > > >
iii. Based
> > > > > above two we need to have select in bigtop code.
> > > > >
> > > > > * Using the existing for current release.
> > > > >
> > > > >
i.
> > > > > Plugin the new stack into HDP self
> > > > >
> > > > >
ii.
> > > > > Still, we can have hdp select
> > > > >
> > > > >
> > > > >
> > > > > I prefer the planA but planB takes less time. Users can
easily adopt it
> > > > > upgrade their stack.
> > > > >
> > > > >
> > > > >
> > > > > Note: Ideally bigtop mpack can be long term approach as
stack id decoupled
> > > > > now, but currently it’s not ready for latest stack
(upgrades can be risk).
> > > > > We’ll work in trunk for same. If this works shortly, we
can choose.
> > > > >
> > > > >
> > > > >
> > > > > 1. Upgrade :
> > > > > * Most of users will be deployed with HDP
> > > > > versions(HDP-2.6.5,HDP-3.1.0), we might need to support
the upgrade to 2.7.7
> > > > > 2. Other work
> > > > > * Any critical/blocker CVE’s Fixes
> > > > > * CVEs Fixes : we’ll try to target some CVE fixes
> > > > > * Review the open PR’s and Jira’s
> > > > >
> > > > >
> > > > >
> > > > > Trunk Code : Parallel we can work for the trunk code,
this we might give
> > > > > as 2.8.
> > > > >
> > > > > * Removal of python2 support
> > > > > * Removal of hard dependency on HDP
> > > > > * Working on bigtop mpack
> > > > >
> > > > > Can add more based on your inputs…
> > > > >
> > > > >
> > > > > Any thoughts on this..? Please feel free to add anything
which I missed.
> > > > >
> > > > >
> > > > >
> > > > > Reference:
> > > > >
> > > > > 1.
> > > > >
https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fpages%2Fviewpage.action%3FpageId%3D30755705&data=05%7C01%7Cbbattula%40visa.com%7C3ad9759845234dc6216008da52cb2e1b%7C38305e12e15d4ee888b9c4db1c477d76%7C0%7C0%7C637913329846408523%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Kv1AicBwjFxEcdYsVjMWgUpefukzFP46lYAs9odJyNM%3D&reserved=0
> > > > >
> > > > >
> > > > >
> >
> >
---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscr...@ambari.apache.org
> > For additional commands, e-mail: dev-h...@ambari.apache.org
> >
> >
> >
> >
---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscr...@ambari.apache.org
> > For additional commands, e-mail: dev-h...@ambari.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscr...@ambari.apache.org
> For additional commands, e-mail: dev-h...@ambari.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@ambari.apache.org
For additional commands, e-mail: dev-h...@ambari.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@ambari.apache.org
For additional commands, e-mail: dev-h...@ambari.apache.org
