Re: [DISCUSS] New subproject: Bigtop Manager

[Evans Ye]

This is really great to hear!

Just curious do we have demo/slide or other materials at hand can be share?
would love to get more holistic understanding :)


Kengo Seki 於 2024年5月17日 週五，20:27寫道：

> Thank you so much for your proposal, Zhiguo!
>
> The lack of user-friendly UI has been a barrier for new and/or light
> users considering adopting Bigtop. Ambari might be a solution, but IMO
> it has a maintainability problem caused from its legacy architecture
> and outdated software stack, and it seems difficult to improve them
> gradually.
> Bigtop Manager is created by Zhiguo and his collaborators from scratch
> to tackle this problem. In my understanding, it's easier to maintain
> since it's built with modern frameworks such as Spring Boot 3 and
> Vue.js 3, and supports recent LTS versions of JDK (17 and 21). I also
> believe his technical skill and sense of responsibility for
> maintaining it, so I'll submit my +1 for accepting it into the Bigtop
> codebase, though it may be required to check if its functionality and
> quality is enough before including it into our release.
>
> Kengo Seki 
>
> On Fri, May 17, 2024 at 2:32 PM 吴治国  wrote:
> >
> > Hello everyone,
> >
> >
> > Due to the fact that Bigtop's native deployment solution (Puppet script)
> is difficult for general users to configure and fully utilize, many of them
> install the components manually or use them in combination with other
> frameworks. To address this, I have developed a project called Bigtop
> Manager, which is inspired by Apache Ambari and Cloudera Manager.
> >
> > The project's repository is hosted on Github[1] and has been developed
> for more than a year, and has also been incubated in OpenEuler community
> for about a half year, while this project has not yet met the standards for
> production-level use, I believe that promoting it as a subproject of Bigtop
> now could bring more benefits to the project and accelerate its development
> progress.
> >
> > I would like to know your opinion about this, thanks.
> >
> > Best Regards,
> > Zhiguo Wu
> >
> > [1]: https://github.com/kevinw66/bigtop-manager
>

Re: [DISCUSS]Drop bigtop-bigpetstore

[Evans Ye]

+1. Thanks for the effort to clean up the codebase.
AFAIK it's not updated for a long time. If its still functional people can
just leverage the old released version of it.

Yuqi Gu  於 2024年1月1日 週一 下午6:28寫道：

> +1
>
> BRs,
> Yuqi
>
> Masatake Iwasaki  于2024年1月1日周一 13:09写道：
>
> > Hi developers,
> >
> >
> > I filed BIGTOP-4057[1] to drop bigtop-bigpetstore which is no longer
> > maintained.
> >
> > Please shout if someone is using this and willing to update the code.
> >
> >
> > [1]https://issues.apache.org/jira/browse/BIGTOP-4057
> >
> >
> > Thanks,
> >
> > Masatake Iwasaki
> >
> >
> >
>

Re: Community over Code?

[Evans Ye]

Not for me. I'm on another side of the country (Vancouver) so if anyone has
the chance to drop by please reach out to me. Would love to get connected ;)

Ganesh Raju  於 2023年8月9日 週三 上午8:25寫道：

> Anybody attending ApcheCon / Community Over Code North America ?
>
> Thanks
> Ganesh
>
> --
> IRC: ganeshraju@#linaro on irc.freenode.ne t
>

Re: Re: [DISCUSS] Release Bigtop 3.2.1

[Evans Ye]

+1. Thanks Masatake!

Yu Hou  於 2023年5月29日 週一 下午7:18寫道：

> +1
>
>
>
>
>
>
>
>
>
> --
>
> Best Regards,
> Yu Hou
>
>
>
> 在 2023-05-29 10:56:50，"Yuqi Gu"  写道：
> >Sounds good to me.
> >
> >Masatake Iwasaki  于2023年5月29日周一 10:30写道：
> >
> >> Hi developers,
> >>
> >> We should start release process of Bigtop 3.2.1
> >> in order to address security issues fixed by Hadoop 3.3.5.
> >>
> >> I volunteer to do release manager work.
> >>
> >> Since BIGTOP-3926 and BIGTOP-3927 are already fixed, BIGTOP-3930 looks
> >> only blocker left open.
> >>
> >>
> https://issues.apache.org/jira/issues/?jql=project%20IN%20(Bigtop)%20AND%20fixVersion%20%3D%203.2.1%20ORDER%20BY%20updated%20DESC
> >>
> >> I think we should limit the scope to just bumping Hadoop to 3.3.5
> >> and critical fixes (if any) to release this quickly.
> >>
> >> Thanks,
> >> Masatake Iwasaki
> >>
>

Re: New Committer: Zhiguo Wu

[Evans Ye]

Congrats!

Yu Hou 於 2023年5月17日 週三，00:57寫道：

> Congratulations  Zhiguo Wu!
>
>
>
> Best Regards,
> Yu Hou
>
>
>
>
>  回复的原邮件 
> | 发件人 | Lei Yao |
> | 日期 | 2023年05月17日 15:54 |
> | 收件人 | dev@bigtop.apache.org |
> | 抄送至 | |
> | 主题 | 回复:Re: New Committer: Zhiguo Wu |
> Congratulations  Zhiguo
>
>
>
>
>
>
>
>
> At 2023-05-17 15:52:59, "Masatake Iwasaki" 
> wrote:
>
> >Congratulations!
> >
> >On 2023/05/17 16:49, chenqiang2080 wrote:
> >>
> >>
> >>
> >>
> >>
> >>
> >> Congratulations  Zhiguo Wu!
> >>
> >>
> >>
> >>
> >>
> >> At 2023-05-17 15:03:24, "Yuqi Gu"  wrote:
> >>> The Project Management Committee (PMC) for Apache Bigtop has asked
> >>> Zhiguo Wu to become a committer and we are pleased to announce
> >>> that he has accepted!
> >>>
> >>> His ASF account has been created as: wuzhi...@apache.org.
> >>>
> >>> Being a committer enables easier contribution to the project
> >>> since there is no need to go via the proxy patch submission process.
> >>> This should enable better productivity.
> >>>
> >>> Apache Bigtop practices CTR (commit-then-review) development process
> >>> which means that you can checkin the code without a +1 from a
> committer.
> >>> This doesn't mean that you're encouraged to do so. Instead, we strongly
> >>> recommend you to seek for feedback before you commit.
> >>> Please read the document before you act:
> >>>
> >>> * https://cwiki.apache.org/confluence/display/BIGTOP/CTR+Model
> >>>
> >>> If in doubt, exercise your best judgement and don't hesitate to ask
> >>> questions on this mailing list.
> >>>
> >>> We're happy to have you on board and looking for many more
> contributions
> >>> to come. Please join me in congratulating Zhiguo Wu!
> >>>
> >>> BRs,
> >>> Yuqi (on behalf of the ASF Bigtop PMC)
>

Re: New Committer: Yoda Leona

[Evans Ye]

Congrats!

吴治国 於 2023年2月15日 週三，23:11寫道：

> Congratulations!
>
>
>  Replied Message 
> | From | Yuqi Gu |
> | Date | 2/13/2023 10:02 |
> | To |  |
> | Subject | Re: New Committer: Yoda Leona |
> Congratulations and welcome Yoda Leona.
>
>
>
> Youngwoo Kim (김영우)  于2023年2月10日周五 12:56写道：
>
> Congrats Yoda!
>
>
> 2023년 2월 10일 (금) 오전 11:29, Kengo Seki 님이 작성:
>
> The Project Management Committee (PMC) for Apache Bigtop has asked
> Yoda Leona to become a committer and we are pleased to announce
> that he has accepted!
>
> His ASF account has been created as: yoda...@apache.org.
>
> Being a committer enables easier contribution to the project
> since there is no need to go via the proxy patch submission process.
> This should enable better productivity.
>
> Apache Bigtop practices CTR (commit-then-review) development process
> which means that you can checkin the code without a +1 from a committer.
> This doesn't mean that you're encouraged to do so. Instead, we strongly
> recommend you to seek for feedback before you commit.
> Please read the document before you act:
>
> * https://cwiki.apache.org/confluence/display/BIGTOP/CTR+Model
>
> If in doubt, exercise your best judgement and don't hesitate to ask
> questions on this mailing list.
>
> We're happy to have you on board and looking for many more contributions
> to come. Please join me in congratulating Yoda Leona!
>
> Regards,
>
> Kengo (on behalf of the ASF Bigtop PMC)
>
>
>

Re: [ANNOUNCE] Apache Bigtop 3.2.0 released

[Evans Ye]

Great release with awesome new features! Thanks team!


Buğra Çakır  於 2023年1月17日 週二 上午7:29寫道：

> Congratulations for this greate release team.
>
> We will also adapt Bigtop installer to the new version of Bigtop repo.
> (https://github.com/beartell/PolePosition)
>
>
> 17 Ocak 2023 Salı 17:55:40 +03 tarihinde Chad W Seys, şunu yazdı:
> > Congratulations and thank you for your work!
> > C.
>
>
>
>
>
>

Re: [DISCUSS] Enable Github Issues for customer-facing questions.

[Evans Ye]

That should be doable.
Worse case scenario is that the committer creates a JIRA and links it to
the github issue. And then we do all the discussion on github.
The downside is we lost any information on the JIRA side. When it comes to
a migration, those links to github issues might be dead.
But think it another way: we link PRs instead of patch files nowadays. So I
guess the strong dependency to github is somewhat OK.

吴治国  於 2022年10月26日 週三 晚上11:48寫道：

> Thank you all for participant the discussion.
>
> > As Masatake said, it's critical to auto generate CHANGELOG and
> RELEASENOTES, which is currently done via JIRA.
>
> Since moving development issues to Github Issues will cause many problems
> (not just CHANGELOG and RELEASENOTES), my intention is not to migrate all
> issues to Github, only customer-facing questions.
>
> > Given that the community is not so active recently, encouraging
> contributors to request JIRA account creation in mailing lists should be a
> viable option.
>
> As far as I know, most Bigtop users are also developers, which is
> different from Ambari, so Masatake's opinion makes sense to me.
>
> But at the same time, I also know that many people are not used to using
> JIRA or Email to ask questions, so I want to enable Github Issues for
> Ambari, at least try it once and see how it works, but we can talk more
> details on Ambari dev mailing list, where I raised an exact same discussion
> thread.
>
> Best Regards,
> Zhiguo Wu
>
> > 2022年10月26日 10:21，Evans Ye  写道：
> >
> > As Masatake said, it's critical to auto generate CHANGELOG and
> > RELEASENOTES, which is currently done via JIRA.
> > I'm more preferred to Masatake's proposal unless there's a solution to
> auto
> > gen from github issues.
> >
> > 李帅 於 2022年10月25日 週二，下午4:13寫道：
> >
> >> I think we should move all issues to github. it is very convenient to
> >> maintain.
> >>
> >> 吴治国  于 2022年10月25日周二 11:40写道：
> >>
> >>> Hello community,
> >>>
> >>> Since JIRA is going to disable public signups, we need to spend more
> time
> >>> and effort to help create accounts for contributors and users, which
> >> makes
> >>> it harder for users to do bug reports. So I'd like to enable Github
> >> Issues
> >>> for customer-facing questions/bug reports/etc., while maintaining
> >>> development issues on Jira, just like Infra suggested.
> >>>
> >>> Or should we move all our works to Github Issues? But this will affect
> >> our
> >>> old commit messages which contains JIRA number.
> >>>
> >>> What's your opinion?
> >>>
> >>> Best Regards,
> >>> Zhiguo Wu
> >>>
> >>
>
>

Re: [DISCUSS] Enable Github Issues for customer-facing questions.

[Evans Ye]

As Masatake said, it's critical to auto generate CHANGELOG and
RELEASENOTES, which is currently done via JIRA.
I'm more preferred to Masatake's proposal unless there's a solution to auto
gen from github issues.

李帅 於 2022年10月25日 週二，下午4:13寫道：

> I think we should move all issues to github. it is very convenient to
> maintain.
>
> 吴治国  于 2022年10月25日周二 11:40写道：
>
> > Hello community,
> >
> > Since JIRA is going to disable public signups, we need to spend more time
> > and effort to help create accounts for contributors and users, which
> makes
> > it harder for users to do bug reports. So I'd like to enable Github
> Issues
> > for customer-facing questions/bug reports/etc., while maintaining
> > development issues on Jira, just like Infra suggested.
> >
> > Or should we move all our works to Github Issues? But this will affect
> our
> > old commit messages which contains JIRA number.
> >
> > What's your opinion?
> >
> > Best Regards,
> > Zhiguo Wu
> >
>

Re: [DISCUSS] Enable autolink redirection from Github commit to Jira for Bigtop

[Evans Ye]

Thanks to Zhiguo and Kengo for the action.

Kengo Seki 於 2022年9月1日 週四，下午8:34寫道：

> Done as INFRA-23651. Thank you for bringing it up, Zhiguo!
>
> Kengo Seki 
>
> On Thu, Sep 1, 2022 at 9:14 PM Kengo Seki  wrote:
> >
> > +1. Considering that we've not received any objection so far, we can
> > move it forward.
> > I'll ask it to the ASF Infra.
> >
> > Kengo Seki 
> >
> > On Tue, Aug 30, 2022 at 10:18 PM Yuqi Gu  wrote:
> > >
> > > +1
> > >
> > > Yu Hou  于2022年8月30日周二 18:28写道：
> > >
> > > > +1
> > > >
> > > >
> > > >
> > > > Best Regards,
> > > > Yu Hou
> > > >
> > > >
> > > >
> > > >
> > > >  回复的原邮件 
> > > > | 发件人 | 吴治国 |
> > > > | 日期 | 2022年08月30日 18:24 |
> > > > | 收件人 | dev@bigtop.apache.org |
> > > > | 抄送至 | |
> > > > | 主题 | [DISCUSS] Enable autolink redirection from Github commit to
> Jira
> > > > for Bigtop |
> > > > Hi Community,
> > > >
> > > > I suggest we can enable autolink from Github commit to Bigtop JIRA
> like
> > > > other communities(eg. HDFS[1], HBase[2], Ambari[3]), it’s helpful
> for issue
> > > > tracking. We can easily access related JIRA page by click the prefix
> of
> > > > commit message like BIGTOP-, which is also highlighted.
> > > >
> > > > Best Regards,
> > > > Zhiguo Wu
> > > >
> > > > [1]: https://github.com/apache/hadoop <
> https://github.com/apache/hadoop
> > > > ;
> > > > [2]: https://github.com/apache/hbase <
> https://github.com/apache/hbase;
> > > > [3]: https://github.com/apache/ambari <
> https://github.com/apache/ambari
> > > > ;
> > > >
> > > >
>

[jira] [Created] (BIGTOP-3774) Drop Docker Sandbox and Vagrant Provisioner

[Evans Ye (Jira)]

Evans Ye created BIGTOP-3774:


 Summary: Drop Docker Sandbox and Vagrant Provisioner
 Key: BIGTOP-3774
 URL: https://issues.apache.org/jira/browse/BIGTOP-3774
 Project: Bigtop
  Issue Type: Improvement
  Components: provisioner, sandbox
Affects Versions: 3.1.1
Reporter: Evans Ye
Assignee: Evans Ye


Per discussion in dev list[1], this JIRA aims to remove non-active maintained 
Vagrant Provisioner and Docker Sandbox.

[1] https://lists.apache.org/thread/x1wxby7mjortw0crybnnww8lhhorovlq.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Propose to drop Docker Sandbox and Vagrant Provisioner

[Evans Ye]

Hi folks,

Both Docker Sandbox[1] and Vagrant Provisioner[2] are not
updated/maintained for a long time. I propose to remove those code to
improve our codebase readability and eliminate potential bugs yield by
non-active maintained code.
I'll wait for 3 days. If there are no objections I'll proceed with a PR.

[1] https://github.com/apache/bigtop/tree/master/docker/sandbox
[2] https://github.com/apache/bigtop/tree/master/provisioner/vagrant

Re: [DISCUSS] Add ClickHouse to Bigtop stack

[Evans Ye]

If both Yuqi questions can be answered I'm open to the clickhouse addition
as long as there's a maintainer for it.

For Distro coverage, I'd like to raise a discussion.
I'm thinking maybe we can have two layers of support: core packages and
extra packages.
Core packages should be fully supported for all the Bigtop Distros, while
extra packages' support can be selective (or open for completion).
Since we do have  the supporting matrix maintained, I think it manages
the user expectation and experience.
Anyhow, any code addition either fully or partially supported should have
CI & smoke test covered (ex: clickhouse on x86 w/ packaging CI, puppet, and
smoke test passed).

- Evans

Yuqi Gu  於 2022年8月4日 週四 上午10:55寫道：

> Agree with Kengo.
> Zhiguo has contributed much to Bigtop including the new Mpack provisioner
> which offers on-click deployment to test Bigtop+Ambari+Mpack.
> I believe he would offer the long-term support to maintain ClickHouse and
> the Mpack services.
>
> To Zhiguo,
> As mentioned by Masatake, ClickHouse is the non-ASF project.
> Before adding it into Bigtop, could you please make clear that
> 1. if there are any license conflicts.
> 2. if there are any restrictions or concerns from upstream ClickHouse
> community.
>
>
> BRs,
> Yuqi
>
>
>
>
> Kengo Seki  于2022年8月2日周二 21:12写道：
>
> > Personally I'd like to encourage Zhiguo to add ClickHouse.
> > He's already done several notable contributions to our codebase,
> > so I think we can expect his long-term commit for keeping ClickHouse
> fresh.
> > In addition, once he gets used to the whole process to add a new
> > component through this development,
> > we also can count on him to add other components and maintain existing
> > ones.
> >
> > Kengo Seki 
> >
> > On Mon, Aug 1, 2022 at 4:17 PM 吴治国  wrote:
> > >
> > > Is there any conclusion?
> > >
> > > Best Regards,
> > > Zhiguo Wu
> > >
> > > > On Jul 29, 2022, at 17:53, 吴治国  wrote:
> > > >
> > > > Thanks Masatake,
> > > >
> > > > I also prefer to use packages provide by them, this will reduce our
> > work.
> > > >
> > > > As I'm trying to use Bigtop as Ambari's default stack, it'll be
> better
> > if we use bigtop repo just like mpack, it makes sense to use BIGTOP stack
> > name with BIGTOP repository.
> > > >
> > > > But add components to Bigtop requires build it from source, provide
> > rpm/deb build scripts, smoke test cases, and puppet manifest, which I'm
> > doing for ClickHouse right now.
> > > >
> > > > And, if ClickHouse end up like ELK, I'll remove it at that time.
> > > >
> > > > Best Regards,
> > > > Zhiguo Wu
> > > >
> > > >> On Jul 29, 2022, at 17:21, Masatake Iwasaki <
> > iwasak...@oss.nttdata.com> wrote:
> > > >>
> > > >> Hi Zhiguo,
> > > >>
> > > >> Honestly saying, I'm not positive to add the product to Bigtop.
> > > >> We do not have enough developer's resource even for maintaining
> > current stack.
> > > >> Since it is non-ASF product driven by dedicated company, it may end
> > up in the same way with ELK.
> > > >>
> > > >> In addition, ClickHouse seems to provide packaging stuff on their
> > own[1] by using nFPM[2]
> > > >> and the packages are published[3].
> > > >> If you want additional coverage, contributing to ClickHouse itself
> > should be the first choice.
> > > >>
> > > >> [1] https://github.com/ClickHouse/ClickHouse/tree/master/packages
> > > >> [2] https://github.com/goreleaser/nfpm
> > > >> [3] https://clickhouse.com/docs/en/getting-started/install/
> > > >>
> > > >> Regards,
> > > >> Masatake Iwasaki
> > > >>
> > > >> On 2022/07/29 15:25, 吴治国 wrote:
> > > >>> Hi community, as the ClickHouse are becoming more and more popular
> > OLAP engine for bigdata, Although it's not related to Hadoop ecosystem, I
> > still want to add it to Bigtop stack.
> > > >>> But I can only take care of it in some distros(mostly CentOS-7,
> > RockyLinux-8 and Ubuntu-20 for now), I don't know if this is enough, if
> > not, can I add it to Bigtop? Or is there anyone also interested in this?
> > > >>> Best Regards,
> > > >>> Zhiguo Wu
> > >
> >
>

Re: Re: About Ambari

[Evans Ye]

Zhiguo,

This Is definitely great if we can have all the folks to work on Apache
Ambari collaboratively.

My two cents:
maybe you can share more details to folks in public as I believe there will
be more users/developers interstinging in it.

At Apache, we encourage: “thing can be public, must be public”.

- Evans

Michiel Verheul 於 2022年6月3日 週五，上午2:29寫道：

> Hi Zhiguo,
>
> I definitely want to be part of this. Please count me in.
>
> Michiel
>
> Op wo 1 jun. 2022 08:51 schreef 吴治国 :
>
> > Hi everyone,
> >
> > I find that most people still rely on Ambari, including myself.
> > Therefore, my team is planning to bring Ambari back to incubator, and we
> > have connected with some IPMCs who are willing to help.
> > Currently, we are preparing the proposal, and we believe that we can
> start
> > voting about this in the Apache Incubator community in the near future.
> > Even if the proposal is rejected, we will still create a fork version on
> > github and keep it running.
> > If you are also interested in this, or want to be part of this, please
> > contact me, thanks!
> >
> > Best Regards,
> > Zhiguo Wu
> >
> > On 2022/05/19 06:41:57 Yuqi Gu wrote:
> > > Bigtop adopted the Ambari stable version (2.7.5) as the cluster
> > management
> > > tools for the potential developers and administrators. It also provided
> > the
> > > bigtop-ambari-mpack(Bgtp-Mpack) to decouple stack management and
> > definition
> > > from Ambari's core.
> > > Currently Bgtp-Mpack still just supports Bigtop-1.5 (Hadoop 2.x), but
> the
> > > Bigtop has already supported Hadoop 3.x in Bigtop 3 release. So we plan
> > to
> > > start working on upgrading Bgtp-Mpack services from Bigtop-1.5 (Hadoop
> > 2.x)
> > > to  Bigtop-3 (Hadoop-3.x).
> > >
> > > BRs,
> > > Yuqi
> > >
> > >
> > >
> > > Michiel Verheul  于2022年5月19日周四 03:56写道：
> > >
> > > > Personally I'm struggling with the exactly the  same issue. There was
> > > > already some kind of discussion on this topic here, earlier:
> > > >
> > > > https://lists.apache.org/thread/wj898zq8q348721xf460mttqlty4v3zw
> > > >
> > > > Personally I have never ran a production cluster without CM before.
> > Bigtop
> > > > 3 with Ambari seemed ideal, but as there was no ambari-mpack for
> > bigtop 3
> > > > yet, I put some energy in porting the HDP mpack to support bigtop.
> > > > But I can understand that maintaining such a component under the
> Bigtop
> > > > project is a no-go, because of Ambari's attic state, Python 2.7 and
> the
> > > > (un)maintainability of such an mpack, so I stopped working on that.
> > > >
> > > > From what I also understand from the above thread, 李帅 is already
> > working on
> > > > some light weight ambari alternative. It feels like that would be a
> > good
> > > > way forward but I don't know how much work has to be done to make
> this
> > work
> > > > and if it's still viable?
> > > >
> > > > The alternative would be running vanilla hadoop/bigtop. I don't have
> > any
> > > > experience with it but I guess the most important gaps for me will
> be:
> > > >
> > > > - initial installation (with support for Kerberos and SSL)
> > > > - (rolling) service restarts
> > > > - basic service monitoring (is a service running or down?)
> > > >
> > > > Maybe we just have to put some energy in creating puppet/Ansible
> > scripts
> > > > for this purpose and just forget about Ambari.
> > > >
> > > >
> > > > Op wo 18 mei 2022 20:07 schreef Battula, Brahma Reddy
> > > > :
> > > >
> > > > > Supposed to ask same question @Martin Blom here.
> > > > > IMO, still Ambari will be good choice for cluster management. I
> > think,
> > > > > most/some amount of people from bigtop(who are using for building
> the
> > > > > packages) still use ambari.
> > > > >
> > > > > Planning to take collective opinion on bring back even with some
> > other
> > > > > name if not with same name.
> > > > >
> > > > > Any thoughts on this..?
> > > > >
> > > > >
> > > > > On 18/05/22, 6:28 PM, "李帅"  wrote:
> > > > >
> > > > > Maintaining Ambari is not easy for its complex architecture.
> > There
> > > > are
> > > > > many
> > > > > Configuration Management Tools such as Puppet, Ansible for
> > > > > alternatives. So
> > > > > for me, I will try to use Puppet or Ansible to deploy and
> monitor
> > > > > cluster.
> > > > > Using Puppet and Ansible without Ambari-like web ui will be a
> > gap for
> > > > > common users.
> > > > >
> > > > >
> > > > >
> > > > > Martin Blom  于 2022年5月18日周三 17:04写道：
> > > > >
> > > > > >
> > > > > > Hi all. I'm new here.
> > > > > >
> > > > > > So at work we provide a service that's been running
> flawlessly
> > on
> > > > > top of
> > > > > > HDP since at least 2014. The recent "events" (which we became
> > aware
> > > > > of only
> > > > > > late last year, when it was already too late) had us
> panicking
> > a
> > > > bit
> > > > > at
> > > > > > first, since we realised we could no longer manage our
> > cluster, but
> > > > > we've
> > > > 

Re: Changes to the Docker provisioner

[Evans Ye]

I think having the option to specify an alternate docker-compose file is a
great idea.
With that we can also put up some example compose files to show users what
provisioner can do.
For example opening up port ranges for services to bind and expose to the
outer network. This is useful for visiting UI pages on browsers.

- Evans

Luca Toscano  於 2022年1月8日 週六 下午7:42寫道：

> Hi everybody,
>
> I filed a code review for the Docker provisioner in
> https://github.com/apache/bigtop/pull/851, Masatake has already
> reviewed it but since a lot of people use it I wanted to bring it to
> your attention. The main issue that I found is that on recent OSes
> like Debian 11, mounting /sys/fs/cgroup to the containers causes
> systemd and dbus to fail when starting. The only explanation that I
> can give is that cgroupsv2 are enabled by default, and recent enough
> versions of Docker compose (10.20+) support them natively. If you have
> a better and more precise explanation please let me know, otherwise
> I'd like to merge the pull request during the next few days. The idea
> is to have, as an experimental feature, a separate docker-compose.yml
> config file that doesn't contain the /sys/fs/cgroup mountpoint, and
> experiment with it.
>
> Let me know what you think :)
>
> Luca
>

Re: [DISCUSS] Release plan of Bigtop 3.1

[Evans Ye]

Overall the release plan is good and +1 to releasing 3.1.

While "release early release often" is a good practice, I'm not sure
whether the CI infra account migration has been done? Doing migration
during release might add a bit more complexity.

Masatake Iwasaki  於 2021年11月22日 週一 下午12:25寫道：

> Hi team,
>
> I filed BIGTOP-3605 as a starting point of discussion.
> https://issues.apache.org/jira/browse/BIGTOP-3605
>
> I propose to release Bigtop 3.1 shortly after Hadoop 3.2.3 is released.
> We can bump Hadoop to 3.3 (or later) on Bigtop 3.2.
>
> * Bumping to Hadoop 3.3 is big change. Providing Hadoop 3.2.3 containing
>almost 300 fixes from 3.2.2 could be useful for users of Bigtop 3.0.
>
> * HBase 2.2.6 and Kafka 2.4.1 are bit old releases.
>We can upgrade them by bumping ZooKeeper to 3.5.
>
> I will appreciate your feedback on BIGTOP-3605 or this thread.
>
> Thanks,
> Masatake Iwasaki
>

Re: [DISCUSS] New Features post Bigtop 3.0

[Evans Ye]

Hi Matt Andruff,
Agree.

Hi Youngwoo,
Could you elaborate what component is needed for the real-time event
streaming? Kafka + Flink in current stack are the solution for it. Pulsar
can be an addition.
Regarding query processing, Could you share more insight on the difference
of Pinot V.S. Presto? Does Pinot suitable for having Looker plugged in
front of it for analytical purposes?

Hi Kengo/Masatake,
Do you have any feature that is needed for your company to move business
forward?


BTW, we have some discussion in the past for this topic you can take as a
reference[1].

[1]
https://docs.google.com/document/d/1F2Gxu8GARQDZXgqHn12LKkQ5wCV_AF4b_tVmjYB6YfA/edit#

Youngwoo Kim (김영우)  於 2021年11月3日 週三 下午4:02寫道：

> Evans,
> Thanks for starting this discussion.
>
> Hopefully, It would be valuable to integrate the real-time event streaming
> and query processing stack e.g., Apache Druid, Pinot, Pulsar and etc.
>
> And 'k8s operator for Bigtop' looks promising for me!
>
> Thanks,
> Youngwoo
>
> On Wed, Nov 3, 2021 at 1:34 AM Evans Ye  wrote:
>
> > Hi folks,
> >
> > With Bigtop 3.0 been released, I think it's time to discuss what's new as
> > our next steps. Of course the open source ver. of unified compatible
> Hadoop
> > Distro. is still our core product going forward. But the surrounding
> value
> > added features might be something that can take us further beyond where
> we
> > were at. Now, let me post some ideas to start the brainstorming.
> >
> > 1. Deployment on K8S: Ambari or Bigtop Puppet as K8S operators.
> > 2. MLOps integrations: MLFlow, Submarine.
> > 3. Data Lake integrations: Hudi, Iceberg, Delta.
> >
> > And for some software engineering stuffs, I think we can do a clean up on
> > out-dated features such as:
> > 1. vagrant provisioner
> > 2. docker sandbox
> > 3. bigtop-ci
> > 4. bigtop-data-generators
> > 5. bigtop-bigpetstore
> >
> > Any thoughts? Would love to hear all of you.
> >
>

[DISCUSS] New Features post Bigtop 3.0

[Evans Ye]

Hi folks,

With Bigtop 3.0 been released, I think it's time to discuss what's new as
our next steps. Of course the open source ver. of unified compatible Hadoop
Distro. is still our core product going forward. But the surrounding value
added features might be something that can take us further beyond where we
were at. Now, let me post some ideas to start the brainstorming.

1. Deployment on K8S: Ambari or Bigtop Puppet as K8S operators.
2. MLOps integrations: MLFlow, Submarine.
3. Data Lake integrations: Hudi, Iceberg, Delta.

And for some software engineering stuffs, I think we can do a clean up on
out-dated features such as:
1. vagrant provisioner
2. docker sandbox
3. bigtop-ci
4. bigtop-data-generators
5. bigtop-bigpetstore

Any thoughts? Would love to hear all of you.

Re: Oozie 5.2.1 and Zookeeper 3.4.14 compatibility issue

[Evans Ye]

Hi Jason,

Good finding and look forward to your PR.

Evans

Jason Wen 於 2021年10月29日 週五，下午9:46寫道：

> Hi All,
>
>
>
> We find the root cause of the issue.
>
> This issue happens when HA is enabled on oozie. oozie-5.2.1 is using
> curator-2.5.0 which depends on guava-11.0.2, but bigtop-3.0.0 apply the
> patch to change guava version to 27.0-jre in order to work with
> Hadoop-3.2.2:
> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/oozie/patch0-hadoop322-hive312.diff#L103
>
> This guava version change breaks the curator-2.5.0 because it depends on
> some API that is not available anymore in guava-27.0-jre, so it will throw
> NoSuchMethod error when starting oozie.
>
> I will file a jira to bigtop and create the PR for fixing the issue.
>
>
>
> Thanks,
>
> Jason
>
>
>
> *From: *Jason Wen 
> *Reply-To: *"u...@bigtop.apache.org" 
> *Date: *Monday, October 25, 2021 at 12:23 PM
> *To: *"u...@bigtop.apache.org" , "
> dev@bigtop.apache.org" 
> *Subject: *Oozie 5.2.1 and Zookeeper 3.4.14 compatibility issue
>
>
>
> Hi All,
>
>
>
> We recently hit an issue when using oozie 5.2.1 with zookeeper 3.4.14 that
> was built from bigtop. The curator version 2.5.0 that oozie 5.2.1 is using
> seems not working with zookeeper 3.4.14. When oozie is started, it tries to
> connect to zookeeper but is always stuck at following log:
> Connecting to ZooKeeper with SASL/Kerberos and using 'sasl' ACLs
>
>
>
> Hadoop 3.2.2 is using curator 2.13.0 version, so we tried to replace
> curator libs in /usr/lib/oozie/lib dir with curator 2.13.0 version jars.
> After the curator jars are replaced with 2.13.0 version, the oozie service
> can be started successfully.
>
>
>
> Has anyone tested Oozie 5.2.1 connection to Zookeeper 3.4.14 with Kerberos
> enabled? This looks like a bug in bigtop 3.0.0
>
>
>
> Thanks,
>
> Jason
>
>
>

Re: Power (ppc64le) CI/CD upgrade

[Evans Ye]

Thanks for the notice, Amir.

MrAsanjar  於 2021年10月22日 週五 下午11:57寫道：

> Hi team,
> IBM plans to upgrade the current Jenkins slave to high-end Power9 with SSD
> or nvme drives, perhaps next week. There will be a short interruption in
> the availability.
>

Re: [VOTE] Release Bigtop version 3.0.0

[Evans Ye]

Here's my late +1 (binding)

   1. SHA 256, 512 evaluated.
   2. signature verified.
   3. Hadoop build on CentOS 7 tested.
   4. smoke test via provisioner tested on CentOS 7 (./docker-hadoop.sh -d
   -c 3 -m 8g -k 'hdfs,mapreduce,yarn,spark,hive,hbase,flink' -s
   'hdfs,mapreduce,yarn,spark,hbase,flink,hive')

I do find things out-of-date such as the Vagrant Provisioner
(provisioner/vagrant). However, that shouldn't cancel RC since that's not
the only thing out-dated in our code base(Docker Sandbox is also out-dated,
etc). The main Bigtop Hadoop 3 distro is a valuable product and its quality
is good enough for a release. Overall, thanks to the great team and great
collaboration!

- Evans

Kengo Seki  於 2021年10月23日 週六 上午8:37寫道：

> > Since this is a hadoop 2 -> hadoop 3 upgrade for Bigtop,
> > is there any documentation/guideline/etc.. that is available
> > to help people to migrate their configs to the new major version?
>
> I'm afraid there is no document like that right now,
> so we have to check the migration guide of each products.
>
> > we'll surely report back any issue that
> > we find when we decide to upgrade.
>
> Thanks! Your feedback is always very helpful.
> Masatake and I will also report back our findings when our customers
> who are using Bigtop 1.5 migrates their platform to 3.0 in the future.
>
> Kengo Seki 
>
> On Thu, Oct 21, 2021 at 12:43 AM Luca Toscano 
> wrote:
> >
> > This is really great, so happy to see 3.0 in RC state!
> >
> > We will probably not be able to test a full upgrade of our testing
> > environment for this vote, but we'll surely report back any issue that
> > we find when we decide to upgrade. Since this is a hadoop 2 -> hadoop
> > 3 upgrade for Bigtop, is there any documentation/guideline/etc.. that
> > is available to help people to migrate their configs to the new major
> > version?
> >
> > Luca
> >
> > On Tue, Oct 19, 2021 at 1:17 AM Kengo Seki  wrote:
> > >
> > > This is the vote for release 3.0.0 of Apache Bigtop.
> > >
> > > It fixes the following issues:
> > >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12311420=12349352
> > >
> > > The vote will be going for at least 72 hours and will be closed on
> Thursday,
> > > October 21st, 2021 at 17:00 PDT. Please download, test and vote with
> > >
> > > [ ] +1, accept RC0 as the official 3.0.0 release of Apache Bigtop
> > > [ ] +0, I don't care either way,
> > > [ ] -1, do not accept RC0 as the official 3.0.0 release of Apache
> > > Bigtop, because...
> > >
> > > Source and binary files:
> > > https://dist.apache.org/repos/dist/dev/bigtop/bigtop-3.0.0-RC0/
> > >
> > > Maven staging repo:
> > >
> https://repository.apache.org/content/repositories/orgapachebigtop-1031
> > >
> > > The git tag to be voted upon is release-3.0.0
> > >
> > > Bigtop's KEYS file containing PGP keys we use to sign the release:
> > > https://dist.apache.org/repos/dist/release/bigtop/KEYS
> > >
> > > You can see the results of packaging, deployment and smoke tests on
> > > the CI environment in the following URLs.
> > > Some builds had to be retried several times since some of our tests
> > > are flaky, but all of them succeeded at last.
> > > (I ran the smoke tests for all distros on x86_64. Regarding aarch64
> and ppc64le,
> > > I ran them only for CentOS and Debian on the former and for Fedora and
> > > Ubuntu on the latter,
> > > since our computation resource is respectively limited.)
> > > https://ci.bigtop.apache.org/view/Releases/job/Bigtop-3.0.0/
> > >
> https://ci.bigtop.apache.org/view/Test/job/Bigtop-3.0.0-smoke-tests/
> > >
> > > Kengo Seki 
>

Re: [VOTE] Release Bigtop version 3.0.0

[Evans Ye]

Thanks Kengo. I'm really excited to see the RC for 3.0 is out!
Let me spend time playing around with it and get back to you.

Kengo Seki  於 2021年10月19日 週二 上午7:17寫道：

> This is the vote for release 3.0.0 of Apache Bigtop.
>
> It fixes the following issues:
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12311420=12349352
>
> The vote will be going for at least 72 hours and will be closed on
> Thursday,
> October 21st, 2021 at 17:00 PDT. Please download, test and vote with
>
> [ ] +1, accept RC0 as the official 3.0.0 release of Apache Bigtop
> [ ] +0, I don't care either way,
> [ ] -1, do not accept RC0 as the official 3.0.0 release of Apache
> Bigtop, because...
>
> Source and binary files:
> https://dist.apache.org/repos/dist/dev/bigtop/bigtop-3.0.0-RC0/
>
> Maven staging repo:
>
> https://repository.apache.org/content/repositories/orgapachebigtop-1031
>
> The git tag to be voted upon is release-3.0.0
>
> Bigtop's KEYS file containing PGP keys we use to sign the release:
> https://dist.apache.org/repos/dist/release/bigtop/KEYS
>
> You can see the results of packaging, deployment and smoke tests on
> the CI environment in the following URLs.
> Some builds had to be retried several times since some of our tests
> are flaky, but all of them succeeded at last.
> (I ran the smoke tests for all distros on x86_64. Regarding aarch64 and
> ppc64le,
> I ran them only for CentOS and Debian on the former and for Fedora and
> Ubuntu on the latter,
> since our computation resource is respectively limited.)
> https://ci.bigtop.apache.org/view/Releases/job/Bigtop-3.0.0/
> https://ci.bigtop.apache.org/view/Test/job/Bigtop-3.0.0-smoke-tests/
>
> Kengo Seki 
>

Re: 3.0.0 release branch cutting notification

[Evans Ye]

Yes. I remember our discussion went back to release early, release often.
For Bullseye support, we can do a 3.0.1 release, which only covers changes
to make Bullseye work.

Luca Toscano  於 2021年8月26日 週四 下午9:07寫道：

> Hi Arnaud,
>
> I'd be interested in Bullseye support as well. IIRC it was already
> discussed, and the plan was to cut another release after 3.0.0 to
> avoid waiting (at the time it wasn't clear when Debian would have
> released).
> I think it is fine to postpone Bullseye support and release 3.0 asap,
> maybe we can open a Jira to start the CI work together??
>
> Luca
>
> On Thu, Aug 26, 2021 at 8:05 AM Arnaud Launay  wrote:
> >
> > Le Thu, Aug 26, 2021 at 11:02:07AM +0900, Kengo Seki a écrit:
> > > and let me know if you have any other fixes that you would like to
> > > merge into 3.0.0 ;)
> >
> > Debian 11 (bullseye) support ? :) It was released two weeks ago. Is it
> at least
> > possible to have an automated test to see what works and what doesn't ?
> >
> > Arnaud.
>

Re: Powered By Bigtop

[Evans Ye]

Sure. That's a good suggestion. But to me it's hard to gather the
information by solely googling it. Even though there are evidences showing
they're using Bigtop, we can't 100% sure so representing them might be
error-phony.

I'd really love to hear if there's any suggestion that can address this :)

Best,
Evans

Ganesh Raju  於 2021年6月29日 週二 12:02 寫道：

> Hi,
> It will be nice if this wiki page - 'Powered by Bigtop'
> 
> shows latest updates. E.g., We know 'Univ of Michigan' is using Bigtop v1.5
> in their production implementation. Likewise there are quite a lot of wiki
> pages which are very old.
>
> Thanks,
> Ganesh
>
> --
> IRC: ganeshraju@#linaro on irc.freenode.ne t
>

Re: [DISCUSS] AWS Graviton

[Evans Ye]

Guys that's fantastic! Then I think we can have more sharings about Bigtop
on Graviton. Something like the deployment / compatibility / performance
test, etc. Maybe on AWS re:invent or something else. What do you think?

Jun HE  於 2021年5月22日 週六 上午10:26寫道：

> Yes, as Ganesh mentioned, Bigtop releases after v1.3 can run on Graviton
> without any issues.
> And AWS released Graviton2 based on Arm's N1 core last year. It is very
> powerful and also works with Bigtop release.
>
> Ganesh Raju  于2021年5月22日周六 上午3:50写道：
>
> > Hi Evans
> > We are very well aware of it and also do quite a lot of testing on AWS
> > Graviton. Our distribution will not have any issues running on it.
> >
> > Regards
> > Ganesh
> >
> > On Fri, May 21, 2021 at 2:17 PM Evans Ye  wrote:
> >
> > > Hi folks,
> > >
> > > I recently came across AWS Graviton and it seems to be an Arm based
> > chip. I
> > > wonder do our Arm folks know more about this? Shall our arm based
> > > distribution able to run on that chip directly? OR, whether we can
> > > support Graviton to further expand our impact to big data world?
> > >
> >
> >
> > --
> > IRC: ganeshraju@#linaro on irc.freenode.ne <http://irc.freenode.net/>t
> >
>

[DISCUSS] AWS Graviton

[Evans Ye]

Hi folks,

I recently came across AWS Graviton and it seems to be an Arm based chip. I
wonder do our Arm folks know more about this? Shall our arm based
distribution able to run on that chip directly? OR, whether we can
support Graviton to further expand our impact to big data world?

Re: Wikimedia blogpost about Bigtop

[Evans Ye]

Hi Luca,

I've finished the review and overall I think it's good. We just need to add
more structures/sections/bullets so that users are easily to navigate
through. The comments are added. Some of them are questions I'm
curious about.

Once we have it polished we can move forward to bigtop wiki and blog posts.


Evans Ye  於 2021年5月13日 週四 下午10:56寫道：

> Hey Luca, just want to make it clear: I think your post on wiki tech blog
> is great! I just want to apologize for being late reviewing.
>
> To spread the word to as much TA as possible, we should post this valuable
> content on every suitable channel, including your tech blog, bigtop
> blog/wiki, and conf talk. Let' make some noise!
>
> Youngwoo Kim (김영우)  於 2021年5月13日 週四 15:03 寫道：
>
>> Hey Luca,
>>
>> Great work!  Hope you can get a chance to give a talk at upcoming online
>> ASF events.
>>
>> Thanks,
>> Youngwoo
>>
>> On Sat, May 8, 2021 at 4:40 PM Luca Toscano 
>> wrote:
>>
>> > Hi everybody,
>> >
>> > I took more than expected but we finally published the blogpost about
>> > our migration to Bigtop 1.5:
>> >
>> https://techblog.wikimedia.org/2021/05/07/upgrading-hadoop-in-just-one-day
>> >
>> > I will also keep working, if you think it is worth it, to
>> >
>> >
>> https://docs.google.com/document/d/1fI1mvbR1mFLV6ohU5cIEnU5hFvEE7EWnKYWOkF55jtE
>> > (I followed up to some comments made by Evans, sorry for the lag in
>> > answering!). Comments and suggestions are very welcome, the goal
>> > should be to eventually move the guide to the website or the wiki (or
>> > something similar).
>> >
>> > If anybody in the community has doubts/suggestions/questions/etc..
>> > about moving to Bigtop I am available to answer any question!
>> >
>> > Now I hope to have some spare time to work on Jiras :P
>> >
>> > Thanks,
>> >
>> > Luca
>> >
>>
>

Re: Wikimedia blogpost about Bigtop

[Evans Ye]

Hey Luca, just want to make it clear: I think your post on wiki tech blog
is great! I just want to apologize for being late reviewing.

To spread the word to as much TA as possible, we should post this valuable
content on every suitable channel, including your tech blog, bigtop
blog/wiki, and conf talk. Let' make some noise!

Youngwoo Kim (김영우)  於 2021年5月13日 週四 15:03 寫道：

> Hey Luca,
>
> Great work!  Hope you can get a chance to give a talk at upcoming online
> ASF events.
>
> Thanks,
> Youngwoo
>
> On Sat, May 8, 2021 at 4:40 PM Luca Toscano 
> wrote:
>
> > Hi everybody,
> >
> > I took more than expected but we finally published the blogpost about
> > our migration to Bigtop 1.5:
> >
> https://techblog.wikimedia.org/2021/05/07/upgrading-hadoop-in-just-one-day
> >
> > I will also keep working, if you think it is worth it, to
> >
> >
> https://docs.google.com/document/d/1fI1mvbR1mFLV6ohU5cIEnU5hFvEE7EWnKYWOkF55jtE
> > (I followed up to some comments made by Evans, sorry for the lag in
> > answering!). Comments and suggestions are very welcome, the goal
> > should be to eventually move the guide to the website or the wiki (or
> > something similar).
> >
> > If anybody in the community has doubts/suggestions/questions/etc..
> > about moving to Bigtop I am available to answer any question!
> >
> > Now I hope to have some spare time to work on Jiras :P
> >
> > Thanks,
> >
> > Luca
> >
>

Re: Starting 3.0.0 release process

[Evans Ye]

I'm +1 to Kengo's plan to release 3.0 w/o waiting.
We can do minor release with required changes later on. Release early,
release often is the key here.
I believe when Debian 11 is out, Luca along with the community is happy to
work together for a release to support the usage at Wiki.
Of course, if our release is delayed and Debian 11 is out, we can
reprioritize then.

Arnaud Launay  於 2021年5月11日 週二 下午6:14寫道：

> Hello,
>
> Le Tue, May 11, 2021 at 09:02:57AM +0900, Kengo Seki a écrit:
> > Is it OK with you, or would you like to include Debian 11 support in
> this release?
>
> That depends on the timeframe for bigtop 3, I think. I suspect Debian 11
> will be
> release in about 2 months. If bigtop3 is to be released this summer or in
> september,
> including it would be a good thing. If B3 is to be released next week, no
> :)
>
> Arnaud.
>

Re: Wikimedia blogpost about Bigtop

[Evans Ye]

NP. I'm still slowly reviewing the doc. Overall I think it's a valuable
content to be made. Please allow me to take more time reviewing it.

Luca Toscano  於 2021年5月8日 週六 下午3:42寫道：

> -users@ +user@, wrong address :)
>
> On Sat, May 8, 2021 at 9:40 AM Luca Toscano 
> wrote:
> >
> > Hi everybody,
> >
> > I took more than expected but we finally published the blogpost about
> > our migration to Bigtop 1.5:
> >
> https://techblog.wikimedia.org/2021/05/07/upgrading-hadoop-in-just-one-day
> >
> > I will also keep working, if you think it is worth it, to
> >
> https://docs.google.com/document/d/1fI1mvbR1mFLV6ohU5cIEnU5hFvEE7EWnKYWOkF55jtE
> > (I followed up to some comments made by Evans, sorry for the lag in
> > answering!). Comments and suggestions are very welcome, the goal
> > should be to eventually move the guide to the website or the wiki (or
> > something similar).
> >
> > If anybody in the community has doubts/suggestions/questions/etc..
> > about moving to Bigtop I am available to answer any question!
> >
> > Now I hope to have some spare time to work on Jiras :P
> >
> > Thanks,
> >
> > Luca
>

Re: ApacheCon Asia 2021

[Evans Ye]

I think there's no conflict and in the arm/ mpack/wiki joint talk, you can
also mention about bigtop 3.0 breifly, and instruct audience that there's a
more detailed talk which will be presented by kengo/masatake, and vice
versa.

Connections in both talks make them look like a series, which creates
synergy of the impact.




Luca Toscano  於 2021年4月30日 週五 22:25 寫道：

> Hi Kengo!
>
> From my point of view both can be submitted at the same time, but I'll
> also let Yuqi comment as well :)
>
> Also it is sooo nice to know that Bigtop 3.0.0 is going to be released
> soon-ish, really nice!
>
> Luca
>
> On Fri, Apr 30, 2021 at 4:19 PM Kengo Seki  wrote:
> >
> > Hi Yuqi and Luca,
> >
> > Sorry for the short notice, but Masatake and I also would like to
> > submit a proposal.
> > Our main topic will be an introduction of Bigtop 3.0.0 (which is
> > supposed to be released before the conference, hopefully),
> > so if you're going to talk about Mpack and Wikimedia's usecase mainly,
> > they don't conflict, I think.
> >
> > We'd like to take part in ApacheCon Asia rather than ApacheCon @Home
> > because of timezone,
> > but just in case, we're planning to submit the same proposal to both of
> them.
> > But if you think our theme seems to conflict with yours, we can avoid
> > the conference to which you're going to submit a proposal
> > and submit only to the other. So let us know if we should do so ;)
> >
> > Kengo Seki 
> >
> > On Thu, Apr 22, 2021 at 2:27 AM Luca Toscano 
> wrote:
> > >
> > > Hi!
> > >
> > > My bad, I am a bit busy these days at work, plus I am finishing the
> > > blog post and I didn't have the time to follow up with Yuqi about
> > > Apachecon. I'll try to review everything during the weekend and get
> > > back to this thread, but I think that another joint talk like the
> > > Linaro's one should be good!
> > >
> > > Luca
> > >
> > > On Wed, Apr 21, 2021 at 11:23 AM YQ  wrote:
> > > >
> > > > Hi Evans,
> > > >
> > > > Luca and I have presented a joint talk in Linaro virtual connect
> > > > 2021(LVC21).
> > > > We also would like to refine this proposal and submit it to
> ApacheCon in
> > > > the next week.
> > > > Thanks.
> > > >
> > > > BRs,
> > > > Yuqi
> > > >
> > > > Evans Ye  于2021年4月21日周三 下午4:27写道：
> > > >
> > > > > The CFP[1] will be closed May 3rd.
> > > > > Does anyone want to present a talk showcasing Bigtop?
> > > > > We got several talks submitted to FOSDEM early this year however
> not
> > > > > accepted. ApacheCon would be a great place to present our works.
> > > > >
> > > > > [1]
> > > > >
> > > > >
> https://apachecon.com/acasia2021/cfp.html?fbclid=IwAR2tZYzSrscpmcJYd6RBzKosht50GO5Vrot7FzvFzyXLufkxJyq7hRYul6c
> > > > >
>

Re: [ANNOUNCE] Kengo Seki is now officially the chair of Apache Bigtop

[Evans Ye]

Congrats! Kengo, I believe this year will be an exciting year with the
convergence of big data landscape. We've a unique position in the open
source ecosystem. I look forward to your lead in the following year. Also
thanks to Jun He for the past year of serving!

Jun HE  於 2021年4月22日 週四 下午3:55寫道：

> Hi all,
>
> I'm very pleased to announce that, with ASF board's approval at April's
> meeting, Kengo Seki is now officially appointed as the chair of Apache
> Bigtop. Please join me in congratulating Kengo Seki!
>
> Regards,
>
> Jun
>

ApacheCon Asia 2021

[Evans Ye]

The CFP[1] will be closed May 3rd.
Does anyone want to present a talk showcasing Bigtop?
We got several talks submitted to FOSDEM early this year however not
accepted. ApacheCon would be a great place to present our works.

[1]
https://apachecon.com/acasia2021/cfp.html?fbclid=IwAR2tZYzSrscpmcJYd6RBzKosht50GO5Vrot7FzvFzyXLufkxJyq7hRYul6c

Re: The Future of Ambari

[Evans Ye]

Great news. I think it's a good move to have deeper integration with
ambari, and making Bigtop the 1st class offering out of box in Ambari.

Ganesh Raju  於 2021年4月19日 週一 上午10:13寫道：

> Just forwarding email from ambari mailing list.
>
> -- Forwarded message -
> From: Szabolcs Beki 
> Date: Sun, Apr 18, 2021 at 11:20 AM
> Subject: Re: The Future of Ambari
> To: 
>
>
> Hi Raed,
>
> Yes, indeed. HDP and CDH  are merging into CDP (Cloudera Data Platform).
> While most of the HDP stack is carried over, Ambari is being replaced by
> Cloudera Manager.  Additionally all the HDP (and CDP) distributions are
> behind paywall.
>
> As most of the users were starting with HDP, they are often considered to
> be coupled, however Ambari is, at its core, a generic cluster
> management tool where you can create your own stack definitions
> .
> Bigtop  is also using Ambari, although an
> older
> version.
>
> The approach we are studying is the possibility of Ambari interoperating
> with the latest Bigtop version
> 
> (as of now Ambari 2.6.1  is used instead of 2.7.5).
>
> Szabi
>
> On Sun, Apr 18, 2021 at 1:28 PM AL HAZME, RAED 
> wrote:
>
> > With the merge of Hortonworks and Cloudera, and the obvious end of the
> > road for HDP. Is there any future for Ambari? Do we use it to manage our
> > newly acquired big data cluster, or shall we look somewhere else? Is
> there
> > another open source solution that could be considered as a replacement
> for
> > Ambari?
> >
> >
> >
> > *Best regards,*
> >
> >
> >
> > *Raed*
> >
> >
> >
> > --
> >
> > This Email and any files transmitted may contain confidential and/or
> > privileged information and is intended solely for the addressee(s) named.
> > If you have received this information in error, or are being posted by
> > accident, please notify the sender by return Email, do not redistribute
> > this email message, delete it immediately and keep no copies of it. All
> > opinions and/or views expressed in this email are solely those of the
> > author and do not necessarily represent those of NGHA. Any purchase
> order,
> > purchase advice or legal commitment is only valid once backed by the
> signed
> > hardcopy by the authorized person from NGHA.
> >
>
>
> --
> IRC: ganeshraju@#linaro on irc.freenode.ne t
>

Re: PPC CI server failure

[Evans Ye]

Let me help. I was busy on a thing.


MrAsanjar .  於 2021年4月15日 週四 下午10:30寫道：

> In order to set up the new Jenkins slave for ppc64le (
> https://issues.apache.org/jira/browse/BIGTOP-3534) we need Jenkins
> master's
> public ssh key. Who can help me here?
>
> On Fri, Apr 2, 2021 at 4:00 PM MrAsanjar  wrote:
>
> > I have verified the state of ppc64le VM, it is operational. Could we
> > enable the ppc64le build before OpenStack flag the VM as ideal again.
> >
> > On Thu, Apr 1, 2021 at 4:08 PM MrAsanjar  wrote:
> >
> >> Hi lads
> >> I just got an email that IBM has reinstated the ppc64le VM.
> >>
> >>
> >> On Mon, Mar 29, 2021 at 12:05 PM Evans Ye  wrote:
> >>
> >>> Great news and thanks, Amir!
> >>>
> >>> Jun HE  於 2021年3月29日 週一 下午1:54寫道：
> >>>
> >>> > Awesome! Looking forward to its back to CI.
> >>> > Thanks a lot for helping on this, Asanjar!
> >>> >
> >>> > Regards,
> >>> >
> >>> > Jun
> >>> >
> >>> > MrAsanjar  于2021年3月29日周一 上午10:18写道：
> >>> >
> >>> > > Hi old friends :)
> >>> > > We should have a ppc64le VM back online sometime this week. I'll
> >>> keep you
> >>> > > all posted.
> >>> > >
> >>> > > On Thu, Nov 19, 2020 at 9:05 PM Evans Ye 
> wrote:
> >>> > >
> >>> > > > Hi rbkrishn,
> >>> > > >
> >>> > > > Would you mind to comment whether those PPC servers for Bigtop CI
> >>> can
> >>> > be
> >>> > > > brought up and unlock our release process?
> >>> > > > Thanks!
> >>> > > >
> >>> > > > Best,
> >>> > > > Evans
> >>> > > >
> >>> > > > Kengo Seki  於 2020年11月18日 週三 上午7:26寫道：
> >>> > > >
> >>> > > > > Thank you for checking, Evans and Amir!
> >>> > > > >
> >>> > > > > Kengo Seki 
> >>> > > > >
> >>> > > > > On Wed, Nov 18, 2020 at 2:09 AM Evans Ye 
> >>> wrote:
> >>> > > > > >
> >>> > > > > > Thank you, Amir.
> >>> > > > > >
> >>> > > > > > MrAsanjar  於 2020年11月18日 週三 00:39 寫道：
> >>> > > > > >
> >>> > > > > > > Hi Evans, let me check with IBM again.
> >>> > > > > > >
> >>> > > > > > >
> >>> > > > > > > On Mon, Nov 16, 2020 at 9:08 PM Evans Ye <
> evan...@apache.org
> >>> >
> >>> > > wrote:
> >>> > > > > > >
> >>> > > > > > > > Hi Amir,
> >>> > > > > > > >
> >>> > > > > > > > We're planning Bigtop 1.5 release and if we don't have
> the
> >>> CI
> >>> > > nodes
> >>> > > > > for
> >>> > > > > > > > PPC, we're not able to release 1.5 with PPC supported.
> >>> > > > > > > > Could you help to confirm again? Thanks!
> >>> > > > > > > >
> >>> > > > > > > > Best,
> >>> > > > > > > > Evans Ye
> >>> > > > > > > >
> >>> > > > > > > >
> >>> > > > > > > >
> >>> > > > > > > > MrAsanjar  於 2020年9月17日 週四 下午8:56寫道：
> >>> > > > > > > >
> >>> > > > > > > > > I have informed IBM management regarding the situation,
> >>> > waiting
> >>> > > > > for a
> >>> > > > > > > > > reply.
> >>> > > > > > > > >
> >>> > > > > > > > > On Thu, Sep 17, 2020 at 3:47 AM Evans Ye <
> >>> evan...@apache.org
> >>> > >
> >>> > > > > wrote:
> >>> > > > > > > > >
> >>> > > > > > > > > > Ok. Thanks for doing this to get the ball rolling.
> >>> > > > > > > > > >
> >>> > > > > > > 

Re: PPC CI server failure

[Evans Ye]

Great news and thanks, Amir!

Jun HE  於 2021年3月29日 週一 下午1:54寫道：

> Awesome! Looking forward to its back to CI.
> Thanks a lot for helping on this, Asanjar!
>
> Regards,
>
> Jun
>
> MrAsanjar  于2021年3月29日周一 上午10:18写道：
>
> > Hi old friends :)
> > We should have a ppc64le VM back online sometime this week. I'll keep you
> > all posted.
> >
> > On Thu, Nov 19, 2020 at 9:05 PM Evans Ye  wrote:
> >
> > > Hi rbkrishn,
> > >
> > > Would you mind to comment whether those PPC servers for Bigtop CI can
> be
> > > brought up and unlock our release process?
> > > Thanks!
> > >
> > > Best,
> > > Evans
> > >
> > > Kengo Seki  於 2020年11月18日 週三 上午7:26寫道：
> > >
> > > > Thank you for checking, Evans and Amir!
> > > >
> > > > Kengo Seki 
> > > >
> > > > On Wed, Nov 18, 2020 at 2:09 AM Evans Ye  wrote:
> > > > >
> > > > > Thank you, Amir.
> > > > >
> > > > > MrAsanjar  於 2020年11月18日 週三 00:39 寫道：
> > > > >
> > > > > > Hi Evans, let me check with IBM again.
> > > > > >
> > > > > >
> > > > > > On Mon, Nov 16, 2020 at 9:08 PM Evans Ye 
> > wrote:
> > > > > >
> > > > > > > Hi Amir,
> > > > > > >
> > > > > > > We're planning Bigtop 1.5 release and if we don't have the CI
> > nodes
> > > > for
> > > > > > > PPC, we're not able to release 1.5 with PPC supported.
> > > > > > > Could you help to confirm again? Thanks!
> > > > > > >
> > > > > > > Best,
> > > > > > > Evans Ye
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > MrAsanjar  於 2020年9月17日 週四 下午8:56寫道：
> > > > > > >
> > > > > > > > I have informed IBM management regarding the situation,
> waiting
> > > > for a
> > > > > > > > reply.
> > > > > > > >
> > > > > > > > On Thu, Sep 17, 2020 at 3:47 AM Evans Ye  >
> > > > wrote:
> > > > > > > >
> > > > > > > > > Ok. Thanks for doing this to get the ball rolling.
> > > > > > > > >
> > > > > > > > > Kengo Seki  於 2020年9月17日 週四 10:29 寫道：
> > > > > > > > >
> > > > > > > > > > Thank you for your help, Amir!
> > > > > > > > > > It's just a heads-up, I temporarily disabled builds for
> ppc
> > > in
> > > > the
> > > > > > > > > > following Jenkins jobs so that they can finish.
> > > > > > > > > >
> > > > > > > > > > * Docker-Puppet-Trunk
> > > > > > > > > > * Docker-Puppet-Trunk-pull
> > > > > > > > > > * Docker-Toolchain-Trunk
> > > > > > > > > > * Docker-Toolchain-Trunk-pull
> > > > > > > > > >
> > > > > > > > > > * Bigtop-trunk-packages
> > > > > > > > > > * Bigtop-trunk-repos
> > > > > > > > > >
> > > > > > > > > > * Remove-All-Docker-Containers-Except-Nexus
> > > > > > > > > > * Remove-Dangling-Docker-Images
> > > > > > > > > > * Remove-Inactive-Containers
> > > > > > > > > >
> > > > > > > > > > Kengo Seki 
> > > > > > > > > >
> > > > > > > > > > On Wed, Sep 16, 2020 at 7:35 PM Evans Ye <
> > evan...@apache.org
> > > >
> > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > Awesome! Nice to hear from you, buddy!
> > > > > > > > > > >
> > > > > > > > > > > MrAsanjar  於 2020年9月16日 週三
> 上午3:54寫道：
> > > > > > > > > > >
> > > > > > > > > > > > Hi Evans,
> > > > > > > > > > > > Let me see what I can do. Give me 24 hr :)
> > > > > > > > > > > >
> > > > > > > > > > > >

Re: Talk by Yuqi and Luca at LinaroConnect

[Evans Ye]

I found the conference link:
https://connect.linaro.org/resources/lvc21/lvc21-204/

Indeed a great work. Thanks Yuqi, Luca!

Ganesh Raju  於 2021年3月29日 週一 下午11:23寫道：

> Great talk Yuqi and Luca at LinaroConnect on Bigtop and it's usecase at
> Wikimedia.
> The talk is published here 
>
> Thanks,
> Ganesh
> --
> IRC: ganeshraju@#linaro on irc.freenode.ne t
>

Re: Wikimedia is running Bigtop 1.5

[Evans Ye]

Let me look into it and get back to you.
And yes, I think we can do it as a series. No need to be just one shot.

Luca Toscano  於 2021年2月28日 週日 下午8:53寫道：

> Forgot to mention - I'll start preparing a blog post for Wikimedia
> during the next few weeks, I'll reach out again to see if we can share
> one (Wikimedia / Apache), but let's keep it as the second step (first
> documentation etc.. then blogpost(s) in my opinion, but lemme know if
> you disagree).
>
> Luca
>
> On Fri, Feb 12, 2021 at 9:52 AM Luca Toscano 
> wrote:
> >
> > Hi everybody,
> >
> > We have finally migrated our CDH cluster to Bigtop 1.5, so I can say
> > that we are now happy Bigtop users :)
> >
> > The upgrade of the production cluster (60 worker nodes, ~50M files on
> > HDFS) was harder than I expected, since we bumped into a strange
> > performance issue that slowed down the HDFS upgrade. I wrote a summary
> > in https://phabricator.wikimedia.org/T273711#6818136 for whoever is
> > interested, it is surely something to highlight in the CDH->Bigtop
> > guide. Speaking of which, the last thing that we did was starting
> >
> https://docs.google.com/document/d/1fI1mvbR1mFLV6ohU5cIEnU5hFvEE7EWnKYWOkF55jtE/edit
> > some time ago, so I am wondering if we could find a more permanent
> > location. Would it make sense to start a wiki page somewhere? Or even
> > a .md file in the github repo, as you prefer (the latter would be more
> > convenient for reviewers etc..).
> >
> > Anyway, thanks a lot to all for the support! It was a looong project
> > but we eventually did it!
> >
> > Luca
>

[ANNOUNCE] New Bigtop PMC member: Yuqi Gu

[Evans Ye]

On behalf of the Apache Bigtop PMC, I am pleased to announce that
Yuqi Gu has been elected to the Bigtop Project Management
Committee. We appreciate Yuqi's contributions thus far, and look
forward to his continued involvement with his new role at Apache Bigtop.

Please join me in congratulating Yuqi Gu!

Re: Bigtop Site: Input needed

[Evans Ye]

I like the user input from Luca, thanks for sharing that.
Overall I think it's the "lack of doc" problem that we're having for a long
time. But pointing out what is important to the users is a valuable info
for us so that we can start from there first and extend further.

I just want to add one more thing: as long as the contribution is good to
the community and align with the community direction, I'm up for it. So
feel free to explore and contribute if "Docsy" is more preferred.



Luca Toscano  於 2020年12月30日 週三 下午6:02寫道：

> Hi everybody,
>
> Thanks a lot Olaf for all the efforts! Going to add 2c from the
> community perspective of a fresh member related to what I was looking
> for when me and my team were deciding if/how to move to Bigtop. What I
> usually like in websites about projects is that there is a quick and
> easy way to get the following information during the first minute of
> navigation (or even less):
> - A clear and concise description about what the project is doing and
> releasing.
> - Quick links about docs containing snippets of code describing few
> simple use cases and how they are handled, or even better quick
> snippets to copy/paste commands to execute to try most common use
> cases.
> - Possibly a date or reference that tells me how often the project is
> releasing, or how to find the info (via github / Jira / etc..).
>
> The http://bigtop.apache.org front page looks good for the first
> point, but it would be great to have a couple of sections like "Here's
> how you build one package with Docker" or "Here's how you create a
> cluster from scratch to test the latest packages via Docker". It is
> very powerful as message since it captures the attention of the user
> looking for some information, and it could be a first jump to
> Confluence. I am saying that since I found Confluence via Google
> search (I know that there is a "Wiki" link in the dropdown menu' but I
> didn't really think to check there at the time) and I personally found
> it a little confusing (not the Bigtop specific one, in general) and I
> felt that the project was not updated a lot since most of the links
> were pointing to 1.1/1.2 releases. At this point it wasn't still clear
> to me what Bigtop was really about, for example the fact that I could
> have rebuilt packages selectively applying patches on top etc.. simply
> using a Docker image. This is an amazing and powerful workflow that
> you all built, and it should be crystal clear to all (new) users how
> simple and effective are the tools available! I know that the info was
> there, but for some reason my brain was not processing it :D (this is
> why I suggested the quick code-snippets in the front page). After a
> bit of research I found Jira, and then I figured out the process of
> releasing etc.. For a new user, official website + Confluence + Jira +
> github are surely overwhelming if there is not a clear link between
> them (at least from my point of view).
>
> Among the examples that Olaf brought up, I think that
> http://arrow.apache.org/ is really neat (especially the docs page,
> very clean) and I also like https://kubernetes.io/ and its
> documentation (that IIUC is docsy-based right?). The
> http://bigtop-docsy.oflebbe.de/ is very nice, I like the fact that the
> vertical color bars are highlighting different concepts/things, and it
> also feels more easy to follow. I completely understand what Evans
> says about the efforts to maintain a new versioned documentation
> though, so ananke is a viable option as well.
>
> Hope that helps!
>
> Luca
>
> On Wed, Dec 30, 2020 at 7:55 AM Kengo Seki  wrote:
> >
> > Thank you for the investigation, Olaf! Both examples are really cool.
> > I personally prefer the second option (docsy) because of its versioned
> > documentation support, but also think Evans' opinion is reasonable so
> > either is OK to me.
> >
> > > All the 3rdparty services used and advertised (twitter, stackoverflow,
> github, google analytics, google search) are hardcoded in the design.
> Either override almost all these layout templates or actually play on these
> channels as well?
> >
> > Just curiosity, isn't commenting out params.links.* in config.toml
> > enough to hide the social network links?
> > I tried the docsy example following the steps described in [1], and
> > the links which are commented out in config.toml (e.g., Stackoverflow
> > and Slack) seemed to be disabled in the footer.
> >
> > [1]: https://themes.gohugo.io/docsy/#documentation
> >
> > Kengo Seki 
> >
> > On Wed, Dec 30, 2020 at 12:56 PM Evans Ye  wrote:
> > >
> > > My two cents:
> > >
> > &

Re: Bigtop Site: Input needed

[Evans Ye]

My two cents:

First of all, thank you for putting in the effort on the bigtop site. The
example looks super great!

I'd vote for Ananke for the following reasons:
1. Seems like this solution strikes the balance between the maintenance
effort and the result we want.
2. The docs are all on confluent now so it's as good as it is if we decide
to stay on confluent.
3. Even though we just have the site updated, I think it's a huge plus for
our branding/imaging. And somewhat telling users that we're keep evolving
and up-to-date.
4. "Docsy" or "Exploring different options" are too much effort for now and
future that we can't carry down the road.

- Evans


Olaf Flebbe  於 2020年12月29日 週二 下午8:06寫道：

> Hi,
>
> As previously discussed I had a look on making the website authoring more
> accessible.
>
> I totally underestimated it, my head hurts now (see below) :)
>
> And I am now at crossroads, need your input:
>
> First I decided to look at static site generaters, these are candidates:
> * Hugo : gohugo.io
> * Zola: https://www.getzola.org
> (* Ninja)
>
> I looked at various other Project’s design-wise:
>
> I liked arrow.apache.org most, it’s clean. It looks like built with ninja
> and a custom layout
> Hadoop.apache.org is built with hugo with a custom layout (not adaptable
> for us), looking not-so-good.
> https://kubernetes.io  and many lot more are built with hugo and
> https://www.docsy.dev/  theme looking very professional.
>
> I didn’t want to use ninja for some reason and decided to look into hugo:
>
> * A plus for hugo was support for asciidoctor files out of the box, as Cos
> was suggested.
>
> * big community repository of ready-made themese. But none if them was a
> 100% fit, so I did example bigtop landing pages:
>
> Examples
> 
>
> Hugo / Ananke theme:
> =
>
> Example:  http://bigtop-ananke.oflebbe.de
> Source:  github.com/oflebbe/bigtop-site
> (Install asciidoctor as an additional prerequisite)
>
> In order to understand Hugo I followed the quickstart and the recommended
> ananke theme:
> Cons:
> * I had to bang my head for two days against the wall to implement a
> pulldown navigation menu. (The ASF Menu)
> * Had to fight sizes and image scaling
> * No support for docs, if we would move docs from confluence with
> versioning support to the site.
> (i.e. seperate bigtop-1.4 from bigtop-1.3 as an example),
> * Allmost no docs.
>
> Pro:
> * Looks ok-ish
> * Clean and easy to understand.
> * Design is Blog-centric
>
> Hugo / docsy theme:
> ===
> Example: http://bigtop-docsy.oflebbe.de
> Source:  github.com/oflebbe/bigtop-site-docsy
>
> There are a lot of Linux Foundation/Cloud projects built on this theme,
> but that’s a pity as well: All the 3rdparty services used and advertised
> (twitter, stackoverflow, github, google analytics, google search ) are
> hardcoded in the design. Either override almost all these layout templates
> or actually play on these channels as well ?
>
> Cons:
> * Harder to use
> * Banging my head against the wall to get PostCSS running correctly.
> * Banging my head for a day to get the "The ASF“ dropdown menu working
> (still doesn’t look right)
> * Fight the social network links and so on in the templates
> * Sometimes it seems to trigger bugs in hugo
>
> Pro:
> * Documentation templating seems to work, support for versioned docs
> * Mature Design system
>
>
> General Observations
> ==
>
> It seems like dropdown navigation menus are not a good idea:
> * From accessibilty perspective it is complex to support for vision
> impaired persons
> * On mobile on docsy theme they aren’t rendered at all (no idea why)
> * Changing that we need to provide more content (hadn’t thought about that
> previously)
>
> We need to decide what additional social media channels we would like to
> use (twitter etc ..)
>
> Need a perspective of how we do documentation: Stay on confluence or move
> to Markdown, into the site.
>
> We offer different Downloads in „Downloads“ and „Releases“. It feels
> weird. And the Instructions how to use are buried deep in confluence and in
> the Readme.md (Readme is not linked from landing page)
>
> => Your input needed: <===
>
> What do you think about what will fit a future landing page best, what
> will your work/our audience supporting most?
>
> * Everything to complex:
>   We should stay on current tooling and focus on content , because of
> limited resources.
> * Docsy:
>   Start with a landing page and move over stuff later
> * Ananke:
>   Only offer a landing page and head over to confluence with everything
> else
> * Exploring different options
>   (suggestions, if possible)
>
> Sadly; I am note able to finish this, I had and still a few days off  now
> the clock is ticking.
>
> What I learned from it (besides tons of CSS/HTML fine points): We should
> improve current content first, work on tooling second.
>
> What conclusion would you draw ?
>
> Best,
>Olaf
>
>
>

Re: HPC, Big Data and Data Science devroom at FOSDEM'21

[Evans Ye]

I think we got too little time to prepare for the submissions. Next time we
can discuss earlier and make a concise proposal for those great
features/stuff in Bigtop 1.5. And for WikiMedia's case an end-to-end
success production migration would be a powerful story to tell. Anyhow,
thank you all for making the submissions!


Yuqi Gu  於 2020年12月30日 週三 上午10:29寫道：

> Just got the reply from Fosdem that the talk proposal was rejected too
> :-(
>
> BRs,
> Yuqi
>
> On Tue, 29 Dec 2020 at 14:19, Yuqi Gu  wrote:
>
>> Hi folks,
>>
>> I haven't got Fosdem's email yet.
>> The 'Event state' is still in 'undecided'.
>> [image: image.png]
>>
>> BRs,
>> Yuqi
>>
>>
>> On Mon, 28 Dec 2020 at 04:52, Kengo Seki  wrote:
>>
>>> Unfortunately, my submission was rejected too... How about you Yuqi?
>>>
>>> Kengo Seki 
>>>
>>> On Sun, Dec 27, 2020 at 7:14 PM Luca Toscano 
>>> wrote:
>>> >
>>> > Hi everybody,
>>> >
>>> > I just got an email from Fosdem's organizers that my talk proposal was
>>> > rejected, if any of yours got accepted and you'll need to present
>>> > briefly the "Wikimedia use case" let me know and I'll gladly help!
>>> >
>>> > Luca
>>> >
>>> > On Wed, Dec 16, 2020 at 1:22 AM Kengo Seki  wrote:
>>> > >
>>> > > Thanks Olaf! I saw the deadline was extended to Dec 18th.
>>> > >
>>> > > Kengo Seki 
>>> > >
>>> > > On Tue, Dec 15, 2020 at 8:36 PM Evans Ye  wrote:
>>> > > >
>>> > > > Thanks! Olaf.
>>> > > >
>>> > > > Olaf Flebbe  於 2020年12月15日 週二 下午7:25寫道：
>>> > > >
>>> > > > > i asked for an extension of the rfp
>>> > > > >
>>> > > > > olaf
>>> > > > >
>>> > > > >
>>> > > > > Anfang der weitergeleiteten Nachricht:
>>> > > > >
>>> > > > > > Von: Kenneth Hoste 
>>> > > > > > Datum: 15. Dezember 2020 um 08:57:12 MEZ
>>> > > > > > An: Olaf Flebbe 
>>> > > > > > Betreff: Aw: HPC, Big Data and Data Science devroom at
>>> FOSDEM'21
>>> > > > > >
>>> > > > > > Hi Olaf,
>>> > > > > >
>>> > > > > > We've extended the deadline yesterday evening until Fri Dec
>>> 18th, see
>>> > > > > https://hpc-bigdata-fosdem21.github.io/ .
>>> > > > > >
>>> > > > > > We won't do any further extensions.
>>> > > > > >
>>> > > > > >
>>> > > > > > regards,
>>> > > > > >
>>> > > > > > Kenneth
>>> > > > > >
>>> > > > > >> On 15/12/2020 07:35, Olaf Flebbe wrote:
>>> > > > > >> Hi Kenneth,
>>> > > > > >> will the deadline extended? The Apache Bigtop community is
>>> likely to
>>> > > > > push an additional proposal otherwise it would like to enhance
>>> the already
>>> > > > > submitted proposal .
>>> > > > > >> best
>>> > > > > >> Olaf
>>> > > > > >>>> Am 12.12.2020 um 11:43 schrieb Kenneth Hoste <
>>> kenneth.ho...@ugent.be
>>> > > > > >:
>>> > > > > >>>
>>> > > > > >>> Dear open source enthusiast,
>>> > > > > >>>
>>> > > > > >>> This is a one time message to promote the HPC, Big Data, and
>>> Data
>>> > > > > Science devroom at FOSDEM'21 (https://fosdem.org).
>>> > > > > >>>
>>> > > > > >>> You are receiving this message because you have submitted a
>>> talk
>>> > > > > proposal for one of the previous editions of the devroom, or
>>> have been
>>> > > > > involved with the organization of it.
>>> > > > > >>>
>>> > > > > >>> FOSDEM'21 will be fully virtual, for well known reasons.
>>> > > > > >>> This makes the organization of the event and individual
>>> devrooms
>>> > > > > particularly challenging.
>>> > > > > >>>
>>> > > > > >>> We hope that you are willing to help promote the HPC, Big
>>> Data, and
>>> > > > > Data Science devroom at FOSDEM'21, or perhaps submit a talk
>>> proposal
>>> > > > > yourself...
>>> > > > > >>>
>>> > > > > >>> The submission deadline is really close (Tue Dec 15th 2020,
>>> which may
>>> > > > > get extended by a couple of days), but very light weight: it
>>> basically
>>> > > > > comes down to a talk title + short description (no paper, etc.).
>>> > > > > >>>
>>> > > > > >>> The actual event is over the weekend of Feb 6-7 2021;
>>> > > > > >>> the HPC devroom itself is planned for Sunday Feb 7th 2021.
>>> > > > > >>>
>>> > > > > >>> Accepted talks will need to be pre-recorded by mid Jan'21.
>>> > > > > >>> Talks will be live streamed during the event with live Q &
>>> > > > > discussions after each talk.
>>> > > > > >>>
>>> > > > > >>> Please visit https://hpc-bigdata-fosdem21.github.io for more
>>> > > > > information.
>>> > > > > >>>
>>> > > > > >>> Feel free to contact me in case of questions!
>>> > > > > >>>
>>> > > > > >>> Please consider forwarding this message to people or
>>> projects who may
>>> > > > > be interested in submitting a talk proposal, either to the HPC
>>> devroom, or
>>> > > > > one of the many other devrooms at FOSDEM'21 (
>>> > > > > https://fosdem.org/2021/schedule/tracks).
>>> > > > > >>>
>>> > > > > >>>
>>> > > > > >>> regards,
>>> > > > > >>>
>>> > > > > >>> Kenneth
>>> > > > >
>>>
>>

Re: Updated the "How to release" wiki

[Evans Ye]

Thanks. Let me look.

Kengo Seki  於 2020年12月17日 週四 07:55 寫道：

> To past release managers (mainly Evans):
>
> I updated the "How to release" wiki
> (https://cwiki.apache.org/confluence/display/BIGTOP/How+to+release)
> based on this release,
> so would you check if there's something wrong or what I missed, when
> it's convenient for you?
>
> Comparing the current and the previous versions directly doesn't work
> well, so checking small diffs in a few steps may be easier, as
> follows:
>
>
> https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=27849974=141=139
>
> https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=27849974=139=138
>
> https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=27849974=138=132
>
> https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=27849974=132=126
>
> Thanks!
>
> Kengo Seki 
>

Re: [ANNOUNCE] Apache Bigtop 1.5.0 released

[Evans Ye]

I just posted the release on ASF Bigtop Blog (blogs.apache.org/bigtop).
I'm guessing that Olaf/Cos was referring to our website at bigtop.apache.org
?

Kengo Seki  於 2020年12月17日 週四 上午7:51寫道：

> Thank you for the comments Olaf and Cos, I'm glad if you could post an
> announcement via @ASFbigtop.
> I'm not sure about "a news page"... Does it refer to the blog post on
> https://blogs.apache.org/bigtop/, or are they different things?
>
> Kengo Seki 
>
> On Thu, Dec 17, 2020 at 3:23 AM Konstantin Boudnik  wrote:
> >
> > Not to start a holiway ;), but just a thought: shall we go straight to
> > asciidoctor instead of markdown? I was lately doing some work with
> > asciidoctor (including writing extensive reports, etc.) and I found it
> > to be way more advanced than markdown.
> >
> > And it integrates perfectly into software development pipeline.
> > --
> > With regards,
> >Cos
> >
> > On 16.12.2020 12:51, Olaf Flebbe wrote:
> > > That’s amazing! Thank you Kengo and all the contributors for all your
> efforts.
> > >
> > > I am asking myself if we shouldn’t announce it on the @ASFbigtop
> twitter handle ?
> > > We need to have a news page first, IMO .
> > >
> > > Unless someone else is stepping up I can try to create a news section
> and eventually convert the Website from xml to markdown, to make it more
> accessible. Beware: I have no web design skills :)
> > >
> > > Best
> > > Olaf
> > >
> > >
> > >> Am 16.12.2020 um 15:08 schrieb Konstantin Boudnik :
> > >>
> > >> Great news! And thanks to everyone who contributed to make this
> release happen!
> > >> I wasn't one of them, but I am getting back to the game as my travels
> getting more predictable now ;)
> > >>
> > >> It is a great way to finish this ... a-ah not so easy year: the
> community and the project are moving forward!
> > >>
> > >> --
> > >> With regards,
> > >>   Cos
> > >>
> > >> On 16.12.2020 08:01, Kengo Seki wrote:
> > >>> On behalf of the Apache Bigtop team, I'd love to announce the general
> > >>> availability of the Bigtop 1.5.0 release.
> > >>> The release is available here:
> > >>>https://bigtop.apache.org/download.html#releases
> > >>> A few highlights of this release include:
> > >>>* Four different Linux distributions are newly supported: CentOS
> 8,
> > >>> Debian 10, Fedora 31, and Ubuntu 18.04
> > >>>* Bigtop Mpack is added as a new feature, that enables users to
> > >>> deploy Bigtop components via Apache Ambari
> > >>>* Livy and ELK stack (Elasticsearch, Logstash, Kibana) are newly
> > >>> added as components
> > >>>* Many component upgrades, e.g., Hadoop 2.10.1, Spark 2.4.5, HBase
> > >>> 1.5.0, Hive 2.3.6, Kafka 2.4.0, Zeppelin 0.8.2
> > >>> With Bigtop 1.5.0 the community continues to deliver the most
> advanced
> > >>> big data stack to date. More details about 1.5.0 release are here:
> > >>>https://bigtop.apache.org/release-notes.html
> > >>> Deploying Bigtop is easy: grab the repo/list file for your favorite
> > >>> Linux distribution:
> > >>>https://www.apache.org/dyn/closer.lua/bigtop/bigtop-1.5.0/repos/
> > >>> and you'll be running your very own bigdata cluster in no time!
> > >>> We welcome your help and feedback. For more information on how to
> > >>> report problems, and to get involved, visit the project website at:
> > >>>https://bigtop.apache.org
> > >>> Lastly, I want to emphasize that this is a collaborative work done by
> > >>> project contributors and other communities,
> > >>> who continue to devote time to make Bigtop a better software. Thank
> > >>> you all for making this release possible!
> > >>> Regards,
> > >>> Kengo Seki, Bigtop 1.5.0 Release Manager
> > >
>

Re: HPC, Big Data and Data Science devroom at FOSDEM'21

[Evans Ye]

Thanks! Olaf.

Olaf Flebbe  於 2020年12月15日 週二 下午7:25寫道：

> i asked for an extension of the rfp
>
> olaf
>
>
> Anfang der weitergeleiteten Nachricht:
>
> > Von: Kenneth Hoste 
> > Datum: 15. Dezember 2020 um 08:57:12 MEZ
> > An: Olaf Flebbe 
> > Betreff: Aw: HPC, Big Data and Data Science devroom at FOSDEM'21
> >
> > Hi Olaf,
> >
> > We've extended the deadline yesterday evening until Fri Dec 18th, see
> https://hpc-bigdata-fosdem21.github.io/ .
> >
> > We won't do any further extensions.
> >
> >
> > regards,
> >
> > Kenneth
> >
> >> On 15/12/2020 07:35, Olaf Flebbe wrote:
> >> Hi Kenneth,
> >> will the deadline extended? The Apache Bigtop community is likely to
> push an additional proposal otherwise it would like to enhance the already
> submitted proposal .
> >> best
> >> Olaf
>  Am 12.12.2020 um 11:43 schrieb Kenneth Hoste  >:
> >>>
> >>> Dear open source enthusiast,
> >>>
> >>> This is a one time message to promote the HPC, Big Data, and Data
> Science devroom at FOSDEM'21 (https://fosdem.org).
> >>>
> >>> You are receiving this message because you have submitted a talk
> proposal for one of the previous editions of the devroom, or have been
> involved with the organization of it.
> >>>
> >>> FOSDEM'21 will be fully virtual, for well known reasons.
> >>> This makes the organization of the event and individual devrooms
> particularly challenging.
> >>>
> >>> We hope that you are willing to help promote the HPC, Big Data, and
> Data Science devroom at FOSDEM'21, or perhaps submit a talk proposal
> yourself...
> >>>
> >>> The submission deadline is really close (Tue Dec 15th 2020, which may
> get extended by a couple of days), but very light weight: it basically
> comes down to a talk title + short description (no paper, etc.).
> >>>
> >>> The actual event is over the weekend of Feb 6-7 2021;
> >>> the HPC devroom itself is planned for Sunday Feb 7th 2021.
> >>>
> >>> Accepted talks will need to be pre-recorded by mid Jan'21.
> >>> Talks will be live streamed during the event with live Q &
> discussions after each talk.
> >>>
> >>> Please visit https://hpc-bigdata-fosdem21.github.io for more
> information.
> >>>
> >>> Feel free to contact me in case of questions!
> >>>
> >>> Please consider forwarding this message to people or projects who may
> be interested in submitting a talk proposal, either to the HPC devroom, or
> one of the many other devrooms at FOSDEM'21 (
> https://fosdem.org/2021/schedule/tracks).
> >>>
> >>>
> >>> regards,
> >>>
> >>> Kenneth
>

Re: Fwd: HPC, Big Data and Data Science devroom at FOSDEM'21

[Evans Ye]

The deadline seems super tight (TODAY![1]). I suggest folks who're
interested in doing the talk just go for a submission and then we can
discuss whether to have a joint talk back in the mailing list.

1. Bigtop at Wikimedia
2. Bigtop MPack
3. What's new in Bigtop 1.5

If we can't make it. We can still shoot for ApacheCon 2021 (though there's
no timeline yet).

Anyhow, glad to see folks are interested in doing talks on Bigtop!

[1] https://lists.fosdem.org/pipermail/fosdem/2020q4/003112.html


Kengo Seki  於 2020年12月14日 週一 下午2:59寫道：

> Sounds great! Masatake and I are also interested in the joint talk.
> We'd like to introduce Bigtop itself breifly, 1.5.0 upgrade overview
> and its basic usages,
> if there's no other folks who'd like to talk the same theme.
>
> Kengo Seki 
>
> On Mon, Dec 14, 2020 at 11:29 AM Yuqi Gu  wrote:
> >
> > *>>> Folks, >>> Any one interested in the joint talk? Though I'm not
> > directly involved in >>> the development. I see 1.5 is a high quality
> > release and worth sharing the >>> story. We can also add a section
> > outlining the future of Bigtop.*
> >
> > That's great. I'd like to share the Bigtop-Mpack feature in our new 1.5
> > release and be glad to participate in this joint talk.
> >
> > BRs,
> > Yuqi
> >
> >
> >
> > On Mon, 14 Dec 2020 at 02:34, Evans Ye  wrote:
> >
> > > That's really a great idea! I think a joint talk of Bigtop 1.5 release
> and
> > > the real-world case from Wikimedia would be a powerful talk to deliver.
> > >
> > > Luca,
> > > Are you comfortable to share the case on  FOSDEM21? I remember that we
> had
> > > a similar discussion before and I think this is a good stage to
> present.
> > >
> > > Folks,
> > > Any one interested in the joint talk? Though I'm not directly involved
> in
> > > the development. I see 1.5 is a high quality release and worth sharing
> the
> > > story. We can also add a section outlining the future of Bigtop.
> > >
> > > Evans
> > >
> > >
> > >
> > > Konstantin Boudnik  於 2020年12月14日 週一 上午12:18寫道：
> > >
> > > > That'd be actually real great! I am in a middle of the move
> (again!), so
> > > > I can't promise anything of a sort. But having something from a
> > > > real-world application would be super to submit and present!
> > > > --
> > > > With regards,
> > > >Cos
> > > >
> > > > On 13.12.2020 05:53, Olaf Flebbe wrote:
> > > > > Hi,
> > > > >
> > > > > Anyone to promote Bigtop at FOSDEM’21 (virtual) ?
> > > > >
> > > > > Would be really great if we could get a Bigtop Version 1.5 Upgrade
> Talk
> > > > or even a Bigtop@Wikimedia (Luca?) presented.
> > > > > FOSDEM is the biggest OSS event in Europe, by large.
> > > > >
> > > > > Best
> > > > > Olaf
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >> Anfang der weitergeleiteten Nachricht:
> > > > >>
> > > > >> Von: Kenneth Hoste 
> > > > >> Betreff: HPC, Big Data and Data Science devroom at FOSDEM'21
> > > > >> Datum: 12. Dezember 2020 um 11:43:25 MEZ
> > > > >> An: undisclosed-recipients: ;
> > > > >>
> > > > >> Dear open source enthusiast,
> > > > >>
> > > > >> This is a one time message to promote the HPC, Big Data, and Data
> > > > Science devroom at FOSDEM'21 (https://fosdem.org <
> https://fosdem.org/>).
> > > > >>
> > > > >> You are receiving this message because you have submitted a talk
> > > > proposal for one of the previous editions of the devroom, or have
> been
> > > > involved with the organization of it.
> > > > >>
> > > > >> FOSDEM'21 will be fully virtual, for well known reasons.
> > > > >> This makes the organization of the event and individual devrooms
> > > > particularly challenging.
> > > > >>
> > > > >> We hope that you are willing to help promote the HPC, Big Data,
> and
> > > > Data Science devroom at FOSDEM'21, or perhaps submit a talk proposal
> > > > yourself...
> > > > >>
> > > > >> The submission deadline is really close (Tue Dec 15th 2020, which
>

Re: [VOTE] Release Bigtop version 1.5.0

[Evans Ye]

Jun He your vote should be a binding one.
See http://www.apache.org/legal/release-policy.html#release-approval

Luca Toscano  於 2020年12月14日 週一 下午11:36寫道：

> Correction sorry - the oozie packages are ok, a downgrade was raised
> when installing the bigtop 1.5 packages since on bigtop 1.4 I have a
> version of oozie (4.3.0-3) with some extra patches in it (IIRC due to
> https://issues.apache.org/jira/browse/BIGTOP-3330).
>
> Luca
>
>
> On Mon, Dec 14, 2020 at 2:28 AM Kengo Seki  wrote:
> >
> > Luca,
> >
> > Thank you for the detailed checks!
> > That's really helpful and informative, so I added some of your
> > findings to the cwiki.
> >
> https://cwiki.apache.org/confluence/display/BIGTOP/Overview+of+Bigtop+1.5.0+Support+Matrix
> >
> > Kengo Seki 
> >
> > On Sun, Dec 13, 2020 at 5:40 PM Luca Toscano 
> wrote:
> > >
> > > On Mon, Dec 7, 2020 at 12:47 AM Kengo Seki  wrote:
> > > >
> > > > This is the vote for release 1.5.0 of Apache Bigtop.
> > > >
> > > > [x] +1, accept RC0 as the official 1.5.0 release of Apache Bigtop
> > >
> > > (Community vote)
> > >
> > > I completed my tests, the new release didn't raise any blocking issue.
> > > My tests were for this use case:
> > > - Hadoop test cluster with Hive/Oozie/Spark/Jupyter/Hue on bare metal
> hosts.
> > > - HDFS and Yarn HA setup (including journal nodes).
> > > - Bigtop 1.4 for Debian 9 for the majority of the nodes, and a client
> > > node on Debian 10.
> > > - In place upgrade from Bigtop 1.4 to 1.5 (upgrading the packages
> > > without removing the old ones first).
> > >
> > > I have tested an in place upgrade, trying to rollback/re-deploy before
> > > the HDFS finalize step, all good. Some interesting things that I
> > > noticed or had to fix in my config after the upgrade:
> > > - the HDFS Namenodes seem to be smarter when dealing with a rollback.
> > > While testing CDH -> Bigtop 1.4 (hdfs 2.6.0 -> 2.8.5) I had to follow
> > > the upstream guidelines and explicitly issue commands to rollback the
> > > hdfs state, but when testing this release it was not needed anymore.
> > > I'll dig a big more into this, but overall the upgrade/rollback
> > > procedure seemed smoother.
> > > - our Hadoop clusters never used the
> > > "yarn.resourcemanager.webapp.address.$rm-id" settings in yarn-site.xml
> > > (that seem to have a good default), but on Bigtop 1.5 it lead to NPE
> > > while running map-reduce jobs, like described in
> > > https://issues.apache.org/jira/browse/YARN-8056. Once I set the value,
> > > everything worked as expected.
> > > - while upgrading the packages, I had a little hiccup with oozie since
> > > the version on Bigtop 1.4 is higher than 1.5 (4.3.0-3 vs 4.3.0-1), so
> > > it needed a downgrade. Nothing really major, just mentioning it.
> > >
> > > About https://dist.apache.org/repos/dist/dev/bigtop/bigtop-1.5.0-RC0:
> > > - verified .asc/.shaXXX, all good.
> > > - all debian packages (9 and 10 versions) worked fine, and Kengo
> > > Seki's gpg signature was ok as well.
> > >
> > > Thanks a lot for this release!
> > >
> > > Luca
>

Re: [VOTE] Release Bigtop version 1.5.0

[Evans Ye]

To unlock the release. I'm +1(binding) to accept RC0 as 1.5.0 release:

   - Checked gpg signature, sha512 sum, sha256 sum
   - build
  - Manual test zookeeper-pkg-ind (with default bigtop-slave images)
   - provision
  - Manual test docker provisioner with default centos-7 and debian-10
  config (with bigtop-puppet images)

Since build, provision, and smoke test features are guarded by our CI
already. I think just running through the checking as an end user is
sufficient.

Evans

Kengo Seki  於 2020年12月14日 週一 上午9:28寫道：

> Luca,
>
> Thank you for the detailed checks!
> That's really helpful and informative, so I added some of your
> findings to the cwiki.
>
> https://cwiki.apache.org/confluence/display/BIGTOP/Overview+of+Bigtop+1.5.0+Support+Matrix
>
> Kengo Seki 
>
> On Sun, Dec 13, 2020 at 5:40 PM Luca Toscano 
> wrote:
> >
> > On Mon, Dec 7, 2020 at 12:47 AM Kengo Seki  wrote:
> > >
> > > This is the vote for release 1.5.0 of Apache Bigtop.
> > >
> > > [x] +1, accept RC0 as the official 1.5.0 release of Apache Bigtop
> >
> > (Community vote)
> >
> > I completed my tests, the new release didn't raise any blocking issue.
> > My tests were for this use case:
> > - Hadoop test cluster with Hive/Oozie/Spark/Jupyter/Hue on bare metal
> hosts.
> > - HDFS and Yarn HA setup (including journal nodes).
> > - Bigtop 1.4 for Debian 9 for the majority of the nodes, and a client
> > node on Debian 10.
> > - In place upgrade from Bigtop 1.4 to 1.5 (upgrading the packages
> > without removing the old ones first).
> >
> > I have tested an in place upgrade, trying to rollback/re-deploy before
> > the HDFS finalize step, all good. Some interesting things that I
> > noticed or had to fix in my config after the upgrade:
> > - the HDFS Namenodes seem to be smarter when dealing with a rollback.
> > While testing CDH -> Bigtop 1.4 (hdfs 2.6.0 -> 2.8.5) I had to follow
> > the upstream guidelines and explicitly issue commands to rollback the
> > hdfs state, but when testing this release it was not needed anymore.
> > I'll dig a big more into this, but overall the upgrade/rollback
> > procedure seemed smoother.
> > - our Hadoop clusters never used the
> > "yarn.resourcemanager.webapp.address.$rm-id" settings in yarn-site.xml
> > (that seem to have a good default), but on Bigtop 1.5 it lead to NPE
> > while running map-reduce jobs, like described in
> > https://issues.apache.org/jira/browse/YARN-8056. Once I set the value,
> > everything worked as expected.
> > - while upgrading the packages, I had a little hiccup with oozie since
> > the version on Bigtop 1.4 is higher than 1.5 (4.3.0-3 vs 4.3.0-1), so
> > it needed a downgrade. Nothing really major, just mentioning it.
> >
> > About https://dist.apache.org/repos/dist/dev/bigtop/bigtop-1.5.0-RC0:
> > - verified .asc/.shaXXX, all good.
> > - all debian packages (9 and 10 versions) worked fine, and Kengo
> > Seki's gpg signature was ok as well.
> >
> > Thanks a lot for this release!
> >
> > Luca
>

Re: [VOTE] Release Bigtop version 1.5.0

[Evans Ye]

ge.
The following SSH command responded with a non-zero exit status.
Vagrant assumes that this means the command failed!

umount /mnt

Stdout from the command:



Stderr from the command:

umount: /mnt: not mounted
---

Kengo Seki  於 2020年12月13日 週日 上午8:37寫道：

> Hi Evans,
>
> In my environment, using the latest official CentOS image instead of
> puppetlabs' one seems to work, as follows.
> So I don't think we have to drop the Vagrant provisioner in this
> release and would like to propose to simply update default values in
> vagrantconfig.yaml,
> but I'm not also strongly against dropping it if you think we should
> reduce maintenance cost on it in this opportunity. :)
>
> ```
> ~/repos/bigtop/provisioner/vagrant$ git diff
> diff --git a/provisioner/vagrant/vagrantconfig.yaml
> b/provisioner/vagrant/vagrantconfig.yaml
> index a25690d2..bee87293 100644
> --- a/provisioner/vagrant/vagrantconfig.yaml
> +++ b/provisioner/vagrant/vagrantconfig.yaml
> @@ -15,8 +15,8 @@
>
>  memory_size: 4096
>  number_cpus: 1
> -box: "puppetlabs/centos-7.2-64-nocm"
> -repo: "http://repos.bigtop.apache.org/releases/1.3.0/centos/7/x86_64;
> +box: "centos/7"
> +repo: "http://repos.bigtop.apache.org/releases/1.5.0/centos/7/x86_64;
>  num_instances: 1
>  distro: centos
>  components: [hdfs, yarn, mapreduce]
> ~/repos/bigtop/provisioner/vagrant$ vagrant up
>
> (snip)
>
> ==> bigtop1: Notice: Roles to deploy: [resourcemanager, nodemanager,
> mapred-app, hadoop-client, mapred-app, namenode, datanode]
>
> (snip)
>
> ==> bigtop1: Notice: Finished catalog run in 364.53 seconds
> ==> bigtop1: Configuring cache buckets...
> ~/repos/bigtop/provisioner/vagrant$ vagrant ssh
>
> (snip)
>
> [vagrant@bigtop1 ~]$ sudo jps
> 3812 JobHistoryServer
> 4212 NameNode
> 5993 Jps
> 3978 NodeManager
> 3531 ResourceManager
> 3438 WebAppProxyServer
> 4447 DataNode
> [vagrant@bigtop1 ~]$ hadoop version
> Hadoop 2.10.1
> Subversion https://gitbox.apache.org/repos/asf/bigtop.git -r
> 81ad7b000cec0503b9a1d5521fdaf0129443b536
> Compiled by jenkins on 2020-11-28T10:07Z
> Compiled with protoc 2.5.0
> From source with checksum 12fc83747448c6d239c8a4e93b4fa294
> This command was run using /usr/lib/hadoop/hadoop-common-2.10.1.jar
> [vagrant@bigtop1 ~]$ sudo -u yarn yarn jar
> /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 1 1
>
> (snip)
>
> Job Finished in 29.655 seconds
> Estimated value of Pi is 4.
> ```
>
> Kengo Seki 
>
> On Sun, Dec 13, 2020 at 12:40 AM Evans Ye  wrote:
> >
> > I think having that malfunction feature removed is better to make 1.5
> > release quality stay untainted. If go for RC1  the effort is small since
> > all the build/repo/artifacts can stay AS-IS. Just the code need to be
> > modified.
> > However, you're right that it's for testing purpose so either way is ok.
> >
> > I don't think this worth to block the release though. I can still fire
> up a
> > JIRA for tracking.
> >
> > Luca Toscano  於 2020年12月12日 週六 下午7:53寫道：
> >
> > > Hi Evans,
> > >
> > > I don't know exactly what it is the typical use case for the Vagrant
> > > provisioner, but I guess it should be mostly testing, so I think it
> > > should be fine for the purposes of the 1.5 release to leave it aside
> > > (and fix it later on).
> > > Do you have more details about the systemd failure? Maybe we could
> > > open a quick jira with details so people can chime in and check, to
> > > speed up the release :)
> > >
> > > Luca
> > >
> > > On Sat, Dec 12, 2020 at 10:55 AM Evans Ye  wrote:
> > > >
> > > > I'm still running tests but one thing need to discuss with folks
> first.
> > > >
> > > > The provisioner/vagrant component is failing in my test.
> > > >
> > > > ==> bigtop1: Error: Could not start Service[hadoop-yarn-proxyserver]:
> > > > Execution of '/bin/systemctl start hadoop-yarn-proxyserver' returned
> 1:
> > > Job
> > > > for hadoop-yarn-proxyserver.service failed. See "systemctl status
> > > > hadoop-yarn-proxyserver.service" and "journalctl -xe" for details.
> > > >
> > > > Seems it's due to the systemd changing. I searched on vagrant box
> but I
> > > see
> > > > NO NEW box for centos/ubuntu from puppetlabs. Also, our CI test does
> not
> > > > cover this component so it's likely to have bug down the road.
> Therefore,
> > > > I'm proposing to remove the component in 1.5 release. How d

Re: [VOTE] Release Bigtop version 1.5.0

[Evans Ye]

I think having that malfunction feature removed is better to make 1.5
release quality stay untainted. If go for RC1  the effort is small since
all the build/repo/artifacts can stay AS-IS. Just the code need to be
modified.
However, you're right that it's for testing purpose so either way is ok.

I don't think this worth to block the release though. I can still fire up a
JIRA for tracking.

Luca Toscano  於 2020年12月12日 週六 下午7:53寫道：

> Hi Evans,
>
> I don't know exactly what it is the typical use case for the Vagrant
> provisioner, but I guess it should be mostly testing, so I think it
> should be fine for the purposes of the 1.5 release to leave it aside
> (and fix it later on).
> Do you have more details about the systemd failure? Maybe we could
> open a quick jira with details so people can chime in and check, to
> speed up the release :)
>
> Luca
>
> On Sat, Dec 12, 2020 at 10:55 AM Evans Ye  wrote:
> >
> > I'm still running tests but one thing need to discuss with folks first.
> >
> > The provisioner/vagrant component is failing in my test.
> >
> > ==> bigtop1: Error: Could not start Service[hadoop-yarn-proxyserver]:
> > Execution of '/bin/systemctl start hadoop-yarn-proxyserver' returned 1:
> Job
> > for hadoop-yarn-proxyserver.service failed. See "systemctl status
> > hadoop-yarn-proxyserver.service" and "journalctl -xe" for details.
> >
> > Seems it's due to the systemd changing. I searched on vagrant box but I
> see
> > NO NEW box for centos/ubuntu from puppetlabs. Also, our CI test does not
> > cover this component so it's likely to have bug down the road. Therefore,
> > I'm proposing to remove the component in 1.5 release. How do the folks
> > think?
> >
> > Evans
> >
> >
> > Kengo Seki  於 2020年12月8日 週二 下午9:55寫道：
> >
> > > Luca and Evans,
> > >
> > > Thank you for testing the RC! Sure, I think we can wait for that until
> > > this weekend :)
> > >
> > > Kengo Seki 
> > >
> > > On Tue, Dec 8, 2020 at 6:24 PM Evans Ye  wrote:
> > > >
> > > > I'm really happy to see the release candidate is finally out.
> > > > With this timing the community will close this year with a strong
> result.
> > > > Thank you all for the efforts!
> > > > Let me find time this week to evaluate and vote.
> > > >
> > > > - Evans
> > > >
> > > > Luca Toscano  於 2020年12月8日 週二 下午4:47寫道：
> > > >
> > > > > Hi!
> > > > >
> > > > > On Mon, Dec 7, 2020 at 12:47 AM Kengo Seki 
> wrote:
> > > > > >
> > > > > > This is the vote for release 1.5.0 of Apache Bigtop.
> > > > > >
> > > > > > It fixes the following issues:
> > > > > >
> > > > >
> > >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12311420=12345178
> > > > > >
> > > > > > The vote will be going for at least 72 hours and will be closed
> on
> > > > > Wednesday,
> > > > > > December 9th, 2020 at 5pm PST. Please download, test and vote
> with
> > > > >
> > > > > This is a great news, really glad to see 1.5.0 finally coming out!
> I
> > > > > might be able to try the debian-9 packages on a testing cluster
> with
> > > > > Bigtop 1.4 + Kerberos by the end of this week, but it would delay a
> > > > > bit the 72h. If this is ok, I'll keep this list posted with result
> (or
> > > > > if I don't manage to test properly).
> > > > >
> > > > > Luca
> > > > >
> > >
>

Re: [VOTE] Release Bigtop version 1.5.0

[Evans Ye]

I'm still running tests but one thing need to discuss with folks first.

The provisioner/vagrant component is failing in my test.

==> bigtop1: Error: Could not start Service[hadoop-yarn-proxyserver]:
Execution of '/bin/systemctl start hadoop-yarn-proxyserver' returned 1: Job
for hadoop-yarn-proxyserver.service failed. See "systemctl status
hadoop-yarn-proxyserver.service" and "journalctl -xe" for details.

Seems it's due to the systemd changing. I searched on vagrant box but I see
NO NEW box for centos/ubuntu from puppetlabs. Also, our CI test does not
cover this component so it's likely to have bug down the road. Therefore,
I'm proposing to remove the component in 1.5 release. How do the folks
think?

Evans


Kengo Seki  於 2020年12月8日 週二 下午9:55寫道：

> Luca and Evans,
>
> Thank you for testing the RC! Sure, I think we can wait for that until
> this weekend :)
>
> Kengo Seki 
>
> On Tue, Dec 8, 2020 at 6:24 PM Evans Ye  wrote:
> >
> > I'm really happy to see the release candidate is finally out.
> > With this timing the community will close this year with a strong result.
> > Thank you all for the efforts!
> > Let me find time this week to evaluate and vote.
> >
> > - Evans
> >
> > Luca Toscano  於 2020年12月8日 週二 下午4:47寫道：
> >
> > > Hi!
> > >
> > > On Mon, Dec 7, 2020 at 12:47 AM Kengo Seki  wrote:
> > > >
> > > > This is the vote for release 1.5.0 of Apache Bigtop.
> > > >
> > > > It fixes the following issues:
> > > >
> > >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12311420=12345178
> > > >
> > > > The vote will be going for at least 72 hours and will be closed on
> > > Wednesday,
> > > > December 9th, 2020 at 5pm PST. Please download, test and vote with
> > >
> > > This is a great news, really glad to see 1.5.0 finally coming out! I
> > > might be able to try the debian-9 packages on a testing cluster with
> > > Bigtop 1.4 + Kerberos by the end of this week, but it would delay a
> > > bit the 72h. If this is ok, I'll keep this list posted with result (or
> > > if I don't manage to test properly).
> > >
> > > Luca
> > >
>

Re: [VOTE] Release Bigtop version 1.5.0

[Evans Ye]

I'm really happy to see the release candidate is finally out.
With this timing the community will close this year with a strong result.
Thank you all for the efforts!
Let me find time this week to evaluate and vote.

- Evans

Luca Toscano  於 2020年12月8日 週二 下午4:47寫道：

> Hi!
>
> On Mon, Dec 7, 2020 at 12:47 AM Kengo Seki  wrote:
> >
> > This is the vote for release 1.5.0 of Apache Bigtop.
> >
> > It fixes the following issues:
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12311420=12345178
> >
> > The vote will be going for at least 72 hours and will be closed on
> Wednesday,
> > December 9th, 2020 at 5pm PST. Please download, test and vote with
>
> This is a great news, really glad to see 1.5.0 finally coming out! I
> might be able to try the debian-9 packages on a testing cluster with
> Bigtop 1.4 + Kerberos by the end of this week, but it would delay a
> bit the 72h. If this is ok, I'll keep this list posted with result (or
> if I don't manage to test properly).
>
> Luca
>

Re: 1.5.0 release update and concern about ppc64le

[Evans Ye]

Hi Kengo,

Seems that there's no response from IBM. I guess we can only move on w/o
PPC supported in 1.5. What do you say?

Evans

Evans Ye  於 2020年11月17日 週二 下午12:15寫道：

> Let's wait for 2~3 days to see whether IBM guys respond.
> If no, unfortunately we can only release w/o PPC supported.
>
> Kengo Seki  於 2020年11月17日 週二 上午7:49寫道：
>
>> We've finally resolved all issues for the 1.5.0 release [1].
>> For a few components (e.g., Kibana (BIGTOP-3428), and maybe also
>> Livy), their smoke tests didn't succeed with all platforms/distros on
>> CI yet, but we confirmed that they worked on our local environment.
>> So I think they are caused by a CI-specific environmental problem and
>> not blockers for this release.
>>
>> I'm going to do the following items this week.
>>
>> * Recreate CI worker nodes following Evans' advice (thanks a lot!),
>> because some hosts are frequently running short of disk capacity and
>> going down. It makes the build unstable.
>> * Run the packaging and deployment/smoke test CI jobs, and ensure the
>> current status is as listed in [2].
>> * Then continue the release process from step 3 in the "How to
>> release" document [3], and hopefully cut a release candidate within
>> this week.
>>
>> My concern is about the ppc64le platform, because I can't build
>> packages without that CI server.
>> Is there any response from the IBM management, Amir? And if it seems
>> difficult to get it back in the near future, is it OK that we publish
>> this release without ppc64le (and do it later once the server comes
>> back in the future)?
>>
>> [1]:
>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20BIGTOP%20AND%20resolution%20%3D%20Unresolved%20AND%20fixVersion%20%3D%201.5.0
>> [2]:
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=165225375
>> [3]: https://cwiki.apache.org/confluence/display/BIGTOP/How+to+release
>>
>> Kengo Seki 
>>
>

Re: Update

[Evans Ye]

Looks awesome! Thank you Kengo. I think this is an important change that
can further support dev efficiency and enable us to release early,  release
often.

Kengo Seki  於 2020年11月19日 週四 上午8:23寫道：

> Thank you for the comment, Evans and Olaf! Following your advices, I
> did the following:
>
> * Removed unused EBS volumes (1TBx2 and 30GBx1).
> * Replaced slaves (02, 03, 06, 07) with newly created EC2 instances.
> Instance types were also upgraded (m3, m4 -> m5).
> * Attached EBS volumes to the above instances. 200GBs to 02 and 03,
> 800GB to 06 and 07.
>   (I said that I was going to separate two 2TB volumes into four
> 500GBs in the past, but slave 06 had used 660GB+ before replacing, so
> I changed the allocation)
>
> Then we're using the following resources now, in accordance with Evans'
> email:
>
> * EC2 instances: one m3.xlarge (master), three m5.xlarge (slave 02, 03
> and 07) and one m5.2xlarge (slave 06)
> * EBS volumes: one 1TB gp2 (master), two 200GB gp2 (slave 02 and 03),
> and two 800GB gp2 (slave 06 and 07)
>
> New servers seem to work fine with Jenkins [1]. I also updated the
> "Bigtop CI Setup Guide" page on cwiki [2].
> Let me know if you find something wrong! :)
>
> [1]: https://ci.bigtop.apache.org/computer/
> [2]:
> https://cwiki.apache.org/confluence/display/BIGTOP/Bigtop+CI+Setup+Guide#BigtopCISetupGuide-SetupJenkinsslaves
>
> Kengo Seki 
>
> On Wed, Nov 11, 2020 at 4:59 AM Olaf Flebbe  wrote:
> >
> > hi,
> >
> > fully supporting evans:
> > the unconnected disk do not contain anything valuable, please remove. it
> might make sense to even recreate the current disks on ssd, a bit larger as
> before if needed.
> >
> > olaf
> >
> > > Am 10.11.2020 um 08:09 schrieb Evans Ye :
> > >
> > > Yes I think overall your plan is good.
> > > What's the purpose of leveraging EBS snapshot? Is it to backup the
> things
> > > we have before migration?
> > > Except for the master node(have jenkins settings stored on disk), all
> those
> > > slaves can be wiped out directly.
> > >
> > >
> > >
> > > Kengo Seki  於 2020年11月10日 週二 下午2:42寫道：
> > >
> > >> Thanks everyone for the information! Now I understand our
> circumstances.
> > >> So we're going to split two 1TB volumes attached to slave06 and 07
> > >> into four 500GB volumes (and change their type to gp2), reattach them
> > >> to 02, 03, 06 and 07, and remove currently unused two 1TB volumes,
> > >> right?
> > >>
> > >>> Kengo would you like to take this, or you need a help?
> > >>
> > >> I think I can do them somehow (maybe using EBS snapshot?), but let me
> > >> ask your help if I'm stuck. :)
> > >>
> > >> Kengo Seki 
> > >>
> > >> On Tue, Nov 10, 2020 at 1:00 AM Evans Ye  wrote:
> > >>>
> > >>> OK. I got it now.
> > >>> So the newly created volumes are currently attached to slave06_2 and
> > >>> slave07_2, respectively.
> > >>> However, they're standard HDD, not GP2 SSD. I think we can take this
> > >> chance
> > >>> to recreate those 2 slaves and do an overhaul of our infrastructure.
> > >>>
> > >>> Kengo would you like to take this, or you need a help?
> > >>>
> > >>> Evans
> > >>>
> > >>> Olaf Flebbe  於 2020年11月6日 週五 上午2:40寫道：
> > >>>
> > >>>> Hi,
> > >>>>
> > >>>> OMG . I think I did it.
> > >>>>
> > >>>> A few years ago two of the instance had a hardware problems and did
> not
> > >>>> reboot any more, filesystem was corrupted and so on.  That was at
> the
> > >> time
> > >>>> of the spectre vulnarability discovery. (2018) . At that time AWS
> had
> > >> major
> > >>>> instabilities since updating firmware seem to have failed for some
> > >> classes
> > >>>> of hardware.
> > >>>>
> > >>>> I tried to recreate them as close as possible but I may have left
> > >>>> accidentely the volumes around. Please lets delete them.
> > >>>>
> > >>>> Olaf
> > >>>>
> > >>>>> Am 05.11.2020 um 14:44 schrieb Konstantin Boudnik  >:
> > >>>>>
> > >>>>> Thanks Evans!
> > >>>>>
> > >>>>> It's great you found th

Re: PPC CI server failure

[Evans Ye]

Hi rbkrishn,

Would you mind to comment whether those PPC servers for Bigtop CI can be
brought up and unlock our release process?
Thanks!

Best,
Evans

Kengo Seki  於 2020年11月18日 週三 上午7:26寫道：

> Thank you for checking, Evans and Amir!
>
> Kengo Seki 
>
> On Wed, Nov 18, 2020 at 2:09 AM Evans Ye  wrote:
> >
> > Thank you, Amir.
> >
> > MrAsanjar  於 2020年11月18日 週三 00:39 寫道：
> >
> > > Hi Evans, let me check with IBM again.
> > >
> > >
> > > On Mon, Nov 16, 2020 at 9:08 PM Evans Ye  wrote:
> > >
> > > > Hi Amir,
> > > >
> > > > We're planning Bigtop 1.5 release and if we don't have the CI nodes
> for
> > > > PPC, we're not able to release 1.5 with PPC supported.
> > > > Could you help to confirm again? Thanks!
> > > >
> > > > Best,
> > > > Evans Ye
> > > >
> > > >
> > > >
> > > > MrAsanjar  於 2020年9月17日 週四 下午8:56寫道：
> > > >
> > > > > I have informed IBM management regarding the situation, waiting
> for a
> > > > > reply.
> > > > >
> > > > > On Thu, Sep 17, 2020 at 3:47 AM Evans Ye 
> wrote:
> > > > >
> > > > > > Ok. Thanks for doing this to get the ball rolling.
> > > > > >
> > > > > > Kengo Seki  於 2020年9月17日 週四 10:29 寫道：
> > > > > >
> > > > > > > Thank you for your help, Amir!
> > > > > > > It's just a heads-up, I temporarily disabled builds for ppc in
> the
> > > > > > > following Jenkins jobs so that they can finish.
> > > > > > >
> > > > > > > * Docker-Puppet-Trunk
> > > > > > > * Docker-Puppet-Trunk-pull
> > > > > > > * Docker-Toolchain-Trunk
> > > > > > > * Docker-Toolchain-Trunk-pull
> > > > > > >
> > > > > > > * Bigtop-trunk-packages
> > > > > > > * Bigtop-trunk-repos
> > > > > > >
> > > > > > > * Remove-All-Docker-Containers-Except-Nexus
> > > > > > > * Remove-Dangling-Docker-Images
> > > > > > > * Remove-Inactive-Containers
> > > > > > >
> > > > > > > Kengo Seki 
> > > > > > >
> > > > > > > On Wed, Sep 16, 2020 at 7:35 PM Evans Ye 
> > > wrote:
> > > > > > > >
> > > > > > > > Awesome! Nice to hear from you, buddy!
> > > > > > > >
> > > > > > > > MrAsanjar  於 2020年9月16日 週三 上午3:54寫道：
> > > > > > > >
> > > > > > > > > Hi Evans,
> > > > > > > > > Let me see what I can do. Give me 24 hr :)
> > > > > > > > >
> > > > > > > > > On Tue, Sep 15, 2020 at 10:51 AM Evans Ye <
> evan...@apache.org>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Yes. I think the action is correct. However [2] might be
> a
> > > > > > different
> > > > > > > > > thing
> > > > > > > > > > for PPC integration in Hadoop.
> > > > > > > > > >
> > > > > > > > > > Amir,
> > > > > > > > > > Could you confirm?
> > > > > > > > > >
> > > > > > > > > > Kengo Seki  於 2020年9月14日 週一 下午9:56寫道：
> > > > > > > > > >
> > > > > > > > > >> Thank you for the advice, Evans!
> > > > > > > > > >> Let me confirm about "PPC machine owners". According to
> > > Amir's
> > > > > > JIRA
> > > > > > > > > >> issues [1][2] and the powered-by list in the OSU site
> [3],
> > > > we're
> > > > > > > using
> > > > > > > > > >> a VM hosted by OSU OSL, right?
> > > > > > > > > >> If it's correct, I'm going to ask them for help via
> > > > > > > > > >> powerdev-requ...@osuosl.org.
> > > > > > > > > >>
> > > > > > > > > >> [1]:
> > > > > > > > > >>
> > > > > > > > >
> > > > > > >
> > &g

Re: PPC CI server failure

[Evans Ye]

Thank you, Amir.

MrAsanjar  於 2020年11月18日 週三 00:39 寫道：

> Hi Evans, let me check with IBM again.
>
>
> On Mon, Nov 16, 2020 at 9:08 PM Evans Ye  wrote:
>
> > Hi Amir,
> >
> > We're planning Bigtop 1.5 release and if we don't have the CI nodes for
> > PPC, we're not able to release 1.5 with PPC supported.
> > Could you help to confirm again? Thanks!
> >
> > Best,
> > Evans Ye
> >
> >
> >
> > MrAsanjar  於 2020年9月17日 週四 下午8:56寫道：
> >
> > > I have informed IBM management regarding the situation, waiting for a
> > > reply.
> > >
> > > On Thu, Sep 17, 2020 at 3:47 AM Evans Ye  wrote:
> > >
> > > > Ok. Thanks for doing this to get the ball rolling.
> > > >
> > > > Kengo Seki  於 2020年9月17日 週四 10:29 寫道：
> > > >
> > > > > Thank you for your help, Amir!
> > > > > It's just a heads-up, I temporarily disabled builds for ppc in the
> > > > > following Jenkins jobs so that they can finish.
> > > > >
> > > > > * Docker-Puppet-Trunk
> > > > > * Docker-Puppet-Trunk-pull
> > > > > * Docker-Toolchain-Trunk
> > > > > * Docker-Toolchain-Trunk-pull
> > > > >
> > > > > * Bigtop-trunk-packages
> > > > > * Bigtop-trunk-repos
> > > > >
> > > > > * Remove-All-Docker-Containers-Except-Nexus
> > > > > * Remove-Dangling-Docker-Images
> > > > > * Remove-Inactive-Containers
> > > > >
> > > > > Kengo Seki 
> > > > >
> > > > > On Wed, Sep 16, 2020 at 7:35 PM Evans Ye 
> wrote:
> > > > > >
> > > > > > Awesome! Nice to hear from you, buddy!
> > > > > >
> > > > > > MrAsanjar  於 2020年9月16日 週三 上午3:54寫道：
> > > > > >
> > > > > > > Hi Evans,
> > > > > > > Let me see what I can do. Give me 24 hr :)
> > > > > > >
> > > > > > > On Tue, Sep 15, 2020 at 10:51 AM Evans Ye 
> > > > wrote:
> > > > > > >
> > > > > > > > Yes. I think the action is correct. However [2] might be a
> > > > different
> > > > > > > thing
> > > > > > > > for PPC integration in Hadoop.
> > > > > > > >
> > > > > > > > Amir,
> > > > > > > > Could you confirm?
> > > > > > > >
> > > > > > > > Kengo Seki  於 2020年9月14日 週一 下午9:56寫道：
> > > > > > > >
> > > > > > > >> Thank you for the advice, Evans!
> > > > > > > >> Let me confirm about "PPC machine owners". According to
> Amir's
> > > > JIRA
> > > > > > > >> issues [1][2] and the powered-by list in the OSU site [3],
> > we're
> > > > > using
> > > > > > > >> a VM hosted by OSU OSL, right?
> > > > > > > >> If it's correct, I'm going to ask them for help via
> > > > > > > >> powerdev-requ...@osuosl.org.
> > > > > > > >>
> > > > > > > >> [1]:
> > > > > > > >>
> > > > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/INFRA-11467?focusedCommentId=15300982=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-15300982
> > > > > > > >> [2]: https://issues.apache.org/jira/browse/INFRA-12014
> > > > > > > >> [3]:
> > > > > > >
> > > https://osuosl.org/services/powerdev/current-projects/#foss-projects
> > > > > > > >>
> > > > > > > >> Kengo Seki 
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> On Mon, Sep 14, 2020 at 2:06 PM Evans Ye <
> evan...@apache.org>
> > > > > wrote:
> > > > > > > >> >
> > > > > > > >> > I'd suggest to reach out to PPC machine owners. Worst case
> > Is
> > > we
> > > > > can
> > > > > > > >> > temporary  drop the PPC support to move the release
> forward.
> > > > > > > >> >
> > > > > > > >> > Kengo Seki  於 2020年9月14日 週一 

Re: 1.5.0 release update and concern about ppc64le

[Evans Ye]

Let's wait for 2~3 days to see whether IBM guys respond.
If no, unfortunately we can only release w/o PPC supported.

Kengo Seki  於 2020年11月17日 週二 上午7:49寫道：

> We've finally resolved all issues for the 1.5.0 release [1].
> For a few components (e.g., Kibana (BIGTOP-3428), and maybe also
> Livy), their smoke tests didn't succeed with all platforms/distros on
> CI yet, but we confirmed that they worked on our local environment.
> So I think they are caused by a CI-specific environmental problem and
> not blockers for this release.
>
> I'm going to do the following items this week.
>
> * Recreate CI worker nodes following Evans' advice (thanks a lot!),
> because some hosts are frequently running short of disk capacity and
> going down. It makes the build unstable.
> * Run the packaging and deployment/smoke test CI jobs, and ensure the
> current status is as listed in [2].
> * Then continue the release process from step 3 in the "How to
> release" document [3], and hopefully cut a release candidate within
> this week.
>
> My concern is about the ppc64le platform, because I can't build
> packages without that CI server.
> Is there any response from the IBM management, Amir? And if it seems
> difficult to get it back in the near future, is it OK that we publish
> this release without ppc64le (and do it later once the server comes
> back in the future)?
>
> [1]:
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20BIGTOP%20AND%20resolution%20%3D%20Unresolved%20AND%20fixVersion%20%3D%201.5.0
> [2]:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=165225375
> [3]: https://cwiki.apache.org/confluence/display/BIGTOP/How+to+release
>
> Kengo Seki 
>

Re: PPC CI server failure

[Evans Ye]

Hi Amir,

We're planning Bigtop 1.5 release and if we don't have the CI nodes for
PPC, we're not able to release 1.5 with PPC supported.
Could you help to confirm again? Thanks!

Best,
Evans Ye



MrAsanjar  於 2020年9月17日 週四 下午8:56寫道：

> I have informed IBM management regarding the situation, waiting for a
> reply.
>
> On Thu, Sep 17, 2020 at 3:47 AM Evans Ye  wrote:
>
> > Ok. Thanks for doing this to get the ball rolling.
> >
> > Kengo Seki  於 2020年9月17日 週四 10:29 寫道：
> >
> > > Thank you for your help, Amir!
> > > It's just a heads-up, I temporarily disabled builds for ppc in the
> > > following Jenkins jobs so that they can finish.
> > >
> > > * Docker-Puppet-Trunk
> > > * Docker-Puppet-Trunk-pull
> > > * Docker-Toolchain-Trunk
> > > * Docker-Toolchain-Trunk-pull
> > >
> > > * Bigtop-trunk-packages
> > > * Bigtop-trunk-repos
> > >
> > > * Remove-All-Docker-Containers-Except-Nexus
> > > * Remove-Dangling-Docker-Images
> > > * Remove-Inactive-Containers
> > >
> > > Kengo Seki 
> > >
> > > On Wed, Sep 16, 2020 at 7:35 PM Evans Ye  wrote:
> > > >
> > > > Awesome! Nice to hear from you, buddy!
> > > >
> > > > MrAsanjar  於 2020年9月16日 週三 上午3:54寫道：
> > > >
> > > > > Hi Evans,
> > > > > Let me see what I can do. Give me 24 hr :)
> > > > >
> > > > > On Tue, Sep 15, 2020 at 10:51 AM Evans Ye 
> > wrote:
> > > > >
> > > > > > Yes. I think the action is correct. However [2] might be a
> > different
> > > > > thing
> > > > > > for PPC integration in Hadoop.
> > > > > >
> > > > > > Amir,
> > > > > > Could you confirm?
> > > > > >
> > > > > > Kengo Seki  於 2020年9月14日 週一 下午9:56寫道：
> > > > > >
> > > > > >> Thank you for the advice, Evans!
> > > > > >> Let me confirm about "PPC machine owners". According to Amir's
> > JIRA
> > > > > >> issues [1][2] and the powered-by list in the OSU site [3], we're
> > > using
> > > > > >> a VM hosted by OSU OSL, right?
> > > > > >> If it's correct, I'm going to ask them for help via
> > > > > >> powerdev-requ...@osuosl.org.
> > > > > >>
> > > > > >> [1]:
> > > > > >>
> > > > >
> > >
> >
> https://issues.apache.org/jira/browse/INFRA-11467?focusedCommentId=15300982=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-15300982
> > > > > >> [2]: https://issues.apache.org/jira/browse/INFRA-12014
> > > > > >> [3]:
> > > > >
> https://osuosl.org/services/powerdev/current-projects/#foss-projects
> > > > > >>
> > > > > >> Kengo Seki 
> > > > > >>
> > > > > >>
> > > > > >> On Mon, Sep 14, 2020 at 2:06 PM Evans Ye 
> > > wrote:
> > > > > >> >
> > > > > >> > I'd suggest to reach out to PPC machine owners. Worst case Is
> we
> > > can
> > > > > >> > temporary  drop the PPC support to move the release forward.
> > > > > >> >
> > > > > >> > Kengo Seki  於 2020年9月14日 週一 12:44 寫道：
> > > > > >> >
> > > > > >> > > Hi everyone,
> > > > > >> > >
> > > > > >> > > Let me share information about the CI environment.
> > > > > >> > > The worker node for ppc64le is currently offlined, so I just
> > > killed
> > > > > >> all
> > > > > >> > > jobs
> > > > > >> > > in the queue waiting for it gets back. Its status is as
> > follows.
> > > > > >> > >
> > > > > >> > > - According to the result of `who -b`, that machine seems to
> > be
> > > > > >> rebooted
> > > > > >> > >   on 2020-09-11 for some reason (probably unexpectedly).
> > > > > >> > >
> > > > > >> > > - According to the result of dmesg, the root volume was
> > mounted
> > > > > >> > >   in read-only mode because of a fsck failure.
> > > > > >> > >
> > > > > >> > >   [   34.840681] EXT4-fs (vda1): Couldn't remount RDWR
> because
> > > of
> > > > > >> > > unprocessed orphan inode list.  Please umount/remount
> instead
> > > > > >> > >   [   60.714110] cgroup: new mount options do not match the
> > > existing
> > > > > >> > > superblock, will be ignored
> > > > > >> > >   [  316.385805] EXT4-fs (vda1): error count since last
> fsck:
> > > 9459
> > > > > >> > >   [  316.385824] EXT4-fs (vda1): initial error at time
> > > 1540294049:
> > > > > >> > > ext4_validate_inode_bitmap:134
> > > > > >> > >   [  316.385826] EXT4-fs (vda1): last error at time
> > 1596881526:
> > > > > >> > > ext4_free_inode:383
> > > > > >> > >
> > > > > >> > > It looks like some fsck work (and replacing the volume, if
> it
> > > fails)
> > > > > >> > > are required,
> > > > > >> > > but I'm not sure if I could run something like `e2fsck -p`,
> > > because
> > > > > >> > > I'm also not sure
> > > > > >> > > where does that machine exist or who's managing it.
> > > > > >> > > (I slightly thought it was running as a VM with QEMU on some
> > EC2
> > > > > >> > > instance, but I couldn't find it)
> > > > > >> > >
> > > > > >> > > > Cos, Evans, Olaf
> > > > > >> > > Would you provide any suggestions?
> > > > > >> > >
> > > > > >> > > Kengo Seki 
> > > > > >> > >
> > > > > >>
> > > > > >
> > > > >
> > >
> >
>

Re: Update

[Evans Ye]

Yes. I think that's enough.
Following is what I was used for bootstrapping a slave w/o volume, you can
take as a REF and modify from there:

adduser jenkins
mkdir /media/ephemeral0/docker
ln -s /media/ephemeral0/docker /var/lib/
yum install -y docker git java ruby
amazon-linux-extras install -y docker
usermod -a -G docker jenkins
mkdir /home/jenkins/.ssh
cat << EOF > /home/jenkins/.ssh/authorized_keys
MASTER_PUBLIC_KEY_HERE
EOF
chmod 700 /home/jenkins/.ssh
chmod 400 /home/jenkins/.ssh/authorized_keys
chown -R jenkins:jenkins /home/jenkins
service docker start
curl -L "
https://github.com/docker/compose/releases/download/1.23.1/docker-compose-$(uname
-s)-$(uname -m)" -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose


Kengo Seki  於 2020年11月10日 週二 下午4:31寫道：

> I intended to keep current data in the storage and only shrink its
> capacity, because I thought slaves also have some settings on the
> disk.
> But are following steps enough for slave setup? If so, I'll wipe and
> setup them from scratch.
>
> * launch EC2 instance
> * create jenkins user
> * configure its authorized_keys so that master can login with current key
> * change slave's IP address on master
> * (then master automatically configure this node, e.g., installing
> agent, right?)
>
> Kengo Seki 
>
> On Tue, Nov 10, 2020 at 4:09 PM Evans Ye  wrote:
> >
> > Yes I think overall your plan is good.
> > What's the purpose of leveraging EBS snapshot? Is it to backup the things
> > we have before migration?
> > Except for the master node(have jenkins settings stored on disk), all
> those
> > slaves can be wiped out directly.
> >
> >
> >
> > Kengo Seki  於 2020年11月10日 週二 下午2:42寫道：
> >
> > > Thanks everyone for the information! Now I understand our
> circumstances.
> > > So we're going to split two 1TB volumes attached to slave06 and 07
> > > into four 500GB volumes (and change their type to gp2), reattach them
> > > to 02, 03, 06 and 07, and remove currently unused two 1TB volumes,
> > > right?
> > >
> > > > Kengo would you like to take this, or you need a help?
> > >
> > > I think I can do them somehow (maybe using EBS snapshot?), but let me
> > > ask your help if I'm stuck. :)
> > >
> > > Kengo Seki 
> > >
> > > On Tue, Nov 10, 2020 at 1:00 AM Evans Ye  wrote:
> > > >
> > > > OK. I got it now.
> > > > So the newly created volumes are currently attached to slave06_2 and
> > > > slave07_2, respectively.
> > > > However, they're standard HDD, not GP2 SSD. I think we can take this
> > > chance
> > > > to recreate those 2 slaves and do an overhaul of our infrastructure.
> > > >
> > > > Kengo would you like to take this, or you need a help?
> > > >
> > > > Evans
> > > >
> > > > Olaf Flebbe  於 2020年11月6日 週五 上午2:40寫道：
> > > >
> > > > > Hi,
> > > > >
> > > > > OMG . I think I did it.
> > > > >
> > > > > A few years ago two of the instance had a hardware problems and
> did not
> > > > > reboot any more, filesystem was corrupted and so on.  That was at
> the
> > > time
> > > > > of the spectre vulnarability discovery. (2018) . At that time AWS
> had
> > > major
> > > > > instabilities since updating firmware seem to have failed for some
> > > classes
> > > > > of hardware.
> > > > >
> > > > > I tried to recreate them as close as possible but I may have left
> > > > > accidentely the volumes around. Please lets delete them.
> > > > >
> > > > > Olaf
> > > > >
> > > > > > Am 05.11.2020 um 14:44 schrieb Konstantin Boudnik <
> c...@apache.org>:
> > > > > >
> > > > > > Thanks Evans!
> > > > > >
> > > > > > It's great you found the details: they are definitely accurate
> as I
> > > am
> > > > > > recalling now. Kengo, do you think splitting the volumes would
> help
> > > us
> > > > > for a
> > > > > > while? Or perhaps we shall try to expand the resource pool (which
> > > might
> > > > > take a
> > > > > > while)?
> > > > > >
> > > > > > Thanks!
> > > > > >  Cos
> > > > > >
> > > > > > On Thu, Nov 05, 2020 at 12:32PM, Evans Ye wrote:
> > > >

Re: Update

[Evans Ye]

Yes I think overall your plan is good.
What's the purpose of leveraging EBS snapshot? Is it to backup the things
we have before migration?
Except for the master node(have jenkins settings stored on disk), all those
slaves can be wiped out directly.



Kengo Seki  於 2020年11月10日 週二 下午2:42寫道：

> Thanks everyone for the information! Now I understand our circumstances.
> So we're going to split two 1TB volumes attached to slave06 and 07
> into four 500GB volumes (and change their type to gp2), reattach them
> to 02, 03, 06 and 07, and remove currently unused two 1TB volumes,
> right?
>
> > Kengo would you like to take this, or you need a help?
>
> I think I can do them somehow (maybe using EBS snapshot?), but let me
> ask your help if I'm stuck. :)
>
> Kengo Seki 
>
> On Tue, Nov 10, 2020 at 1:00 AM Evans Ye  wrote:
> >
> > OK. I got it now.
> > So the newly created volumes are currently attached to slave06_2 and
> > slave07_2, respectively.
> > However, they're standard HDD, not GP2 SSD. I think we can take this
> chance
> > to recreate those 2 slaves and do an overhaul of our infrastructure.
> >
> > Kengo would you like to take this, or you need a help?
> >
> > Evans
> >
> > Olaf Flebbe  於 2020年11月6日 週五 上午2:40寫道：
> >
> > > Hi,
> > >
> > > OMG . I think I did it.
> > >
> > > A few years ago two of the instance had a hardware problems and did not
> > > reboot any more, filesystem was corrupted and so on.  That was at the
> time
> > > of the spectre vulnarability discovery. (2018) . At that time AWS had
> major
> > > instabilities since updating firmware seem to have failed for some
> classes
> > > of hardware.
> > >
> > > I tried to recreate them as close as possible but I may have left
> > > accidentely the volumes around. Please lets delete them.
> > >
> > > Olaf
> > >
> > > > Am 05.11.2020 um 14:44 schrieb Konstantin Boudnik :
> > > >
> > > > Thanks Evans!
> > > >
> > > > It's great you found the details: they are definitely accurate as I
> am
> > > > recalling now. Kengo, do you think splitting the volumes would help
> us
> > > for a
> > > > while? Or perhaps we shall try to expand the resource pool (which
> might
> > > take a
> > > > while)?
> > > >
> > > > Thanks!
> > > >  Cos
> > > >
> > > > On Thu, Nov 05, 2020 at 12:32PM, Evans Ye wrote:
> > > >> In fact, the original deal of our resource is as follows:
> > > >>
> > > >>> 1 m3.2xlarge for CI
> > > >>> 4 m3.xlarge for CI and demo
> > > >>> 3 1TB EBS volumes
> > > >>> 5 elastic IP addresses
> > > >>
> > > >> So technically we should not use that 2 additional 1T volumes
> (created
> > > in
> > > >> 2018).
> > > >> Instead, I think what we can do is to split up one of the existing
> 1TB
> > > >> volumes(ex: attached to slave07) into smaller volumes for slave02,
> 03.
> > > >>
> > > >>
> > > >> Konstantin Boudnik  於 2020年11月4日 週三 下午2:28寫道：
> > > >>
> > > >>> Kengo,
> > > >>>
> > > >>> We had an agreement with EMR folks that we are using the resources
> > > >>> available
> > > >>> to us and it is included into their budget (or something to this
> > > extent).
> > > >>> If
> > > >>> you see some of the resources available under our account - I
> don't see
> > > >>> why we
> > > >>> can't use them.
> > > >>>
> > > >>> If for whatever reason we need to expand the pool, that would
> require a
> > > >>> separate conversation with nice folks from that team, I imagine.
> Please
> > > >>> let me
> > > >>> know if I can help with this going forward.
> > > >>>
> > > >>> Thanks!
> > > >>>  Cos
> > > >>>
> > > >>> On Wed, Nov 04, 2020 at 11:11AM, Kengo Seki wrote:
> > > >>>> Thanks for the comment, Cos! I was able to start docker service on
> > > >>>> docker-slave-02 without replacing and am running some Jenkins
> jobs on
> > > >>>> it now, so I'll replace it in the short future.
> > > >>>> I have a few things that I'd like to ask addi

Re: Update

[Evans Ye]

OK. I got it now.
So the newly created volumes are currently attached to slave06_2 and
slave07_2, respectively.
However, they're standard HDD, not GP2 SSD. I think we can take this chance
to recreate those 2 slaves and do an overhaul of our infrastructure.

Kengo would you like to take this, or you need a help?

Evans

Olaf Flebbe  於 2020年11月6日 週五 上午2:40寫道：

> Hi,
>
> OMG . I think I did it.
>
> A few years ago two of the instance had a hardware problems and did not
> reboot any more, filesystem was corrupted and so on.  That was at the time
> of the spectre vulnarability discovery. (2018) . At that time AWS had major
> instabilities since updating firmware seem to have failed for some classes
> of hardware.
>
> I tried to recreate them as close as possible but I may have left
> accidentely the volumes around. Please lets delete them.
>
> Olaf
>
> > Am 05.11.2020 um 14:44 schrieb Konstantin Boudnik :
> >
> > Thanks Evans!
> >
> > It's great you found the details: they are definitely accurate as I am
> > recalling now. Kengo, do you think splitting the volumes would help us
> for a
> > while? Or perhaps we shall try to expand the resource pool (which might
> take a
> > while)?
> >
> > Thanks!
> >  Cos
> >
> > On Thu, Nov 05, 2020 at 12:32PM, Evans Ye wrote:
> >> In fact, the original deal of our resource is as follows:
> >>
> >>> 1 m3.2xlarge for CI
> >>> 4 m3.xlarge for CI and demo
> >>> 3 1TB EBS volumes
> >>> 5 elastic IP addresses
> >>
> >> So technically we should not use that 2 additional 1T volumes (created
> in
> >> 2018).
> >> Instead, I think what we can do is to split up one of the existing 1TB
> >> volumes(ex: attached to slave07) into smaller volumes for slave02, 03.
> >>
> >>
> >> Konstantin Boudnik  於 2020年11月4日 週三 下午2:28寫道：
> >>
> >>> Kengo,
> >>>
> >>> We had an agreement with EMR folks that we are using the resources
> >>> available
> >>> to us and it is included into their budget (or something to this
> extent).
> >>> If
> >>> you see some of the resources available under our account - I don't see
> >>> why we
> >>> can't use them.
> >>>
> >>> If for whatever reason we need to expand the pool, that would require a
> >>> separate conversation with nice folks from that team, I imagine. Please
> >>> let me
> >>> know if I can help with this going forward.
> >>>
> >>> Thanks!
> >>>  Cos
> >>>
> >>> On Wed, Nov 04, 2020 at 11:11AM, Kengo Seki wrote:
> >>>> Thanks for the comment, Cos! I was able to start docker service on
> >>>> docker-slave-02 without replacing and am running some Jenkins jobs on
> >>>> it now, so I'll replace it in the short future.
> >>>> I have a few things that I'd like to ask additionally:
> >>>>
> >>>> * docker-slave-02 and 03 have a gp2 storage as a root volume that has
> >>>> only 8GiB capacity, and they sometimes run short and stop the CI.
> >>>>  May I increase them to 20 or 30 GiB when I replace those instances?
> >>>> (I'm not sure what is our budget)
> >>>>
> >>>> * They use an instance store with 30GiB to put docker images into it,
> >>>> and they also sometimes run short.
> >>>>  It seems there are two unused volumes with 1TiB (vol-ae71114e and
> >>>> vol-4efa69ae) on AWS console.
> >>>>  May I attach them to 02 and 03 instead of instance stores, or are
> >>>> they backups or something?
> >>>>
> >>>> Kengo Seki 
> >>>>
> >>>> On Mon, Nov 2, 2020 at 6:41 PM Konstantin Boudnik 
> >>> wrote:
> >>>>>
> >>>>> I'd say let replace the broken one. I don't think there's a
> sentimental
> >>>>> value attached ;)
> >>>>>
> >>>>> --
> >>>>> With regards,
> >>>>>   Cos
> >>>>>
> >>>>> On 02.11.2020 08:16, Kengo Seki wrote:
> >>>>>> Thanks for updating Olaf! I've just noticed the Jenkins UI became
> >>> cool :)
> >>>>>> Regarding docker-slave-02, I'll try to replace it after waiting for
> a
> >>>>>> while to make sure there's no objection.
> >>>>>>
> >>>>>> Kengo Seki 
> >&g

Re: Cut the 1.5.0 release branch

[Evans Ye]

Oh yes. You're right for the versioning. Thanks for pointing that out.
For the How To Release doc, ping me once you have it updated!

Kengo Seki  於 2020年11月9日 週一 下午11:35寫道：

> Thank you for adding the permission and the kind offer, Evans!
> I'm going to update the "How to release" page a bit, so let me know if
> there's something wrong.
>
> > Hadoop 3 as discussed is better to be merged in 2.0.
>
> I've thought "Cloud Native Bigtop" (BIGTOP-3225) is assigned to the
> version 2.0, and addressing Hadoop 3.x is to the version 3.0, from the
> discussion in BIGTOP-3287.
> But let's discuss future versioning on another occasion ;)
>
> Kengo Seki 
>
> On Sat, Nov 7, 2020 at 3:31 PM Evans Ye  wrote:
> >
> > BTW, super exciting to see the progress!
> > Let me know what help you need. In previous releases generating and
> signing
> > binary convenient packages is the most time consuming part.
> >
> > Evans Ye  於 2020年11月7日 週六 下午1:40寫道：
> >
> > > Cutting the release branch is OK if we're close to the release in short
> > > time.
> > >
> > > For the wiki permission, I've granted Masatake and you with full
> > > privileges. Let me know if you still have a problem.
> > >
> > > Hadoop 3 as discussed is better to be merged in 2.0.
> > >
> > > Evans
> > >
> > >
> > > Kengo Seki  於 2020年11月6日 週五 下午6:11寫道：
> > >
> > >> Hi everyone (especially committers),
> > >>
> > >> It's a status update of the 1.5.0 release.
> > >> We have only four or five issues that should be resolved before 1.5.0
> > >> at this time [1]
> > >> (let me know if there are other issues that should be included),
> while we
> > >> have
> > >> several PRs waiting to be merged, e.g., upgrading Hadoop to 3.x and
> > >> Flink to 1.11,
> > >> so I've just cut the 1.5.0 release branch (I followed the steps
> > >> described in section two
> > >> of the "How to release" document [2], but let me know if there's any
> > >> mistakes).
> > >> So, if some PR targets the 1.5.0 release, please merge it into
> > >> branch-1.5 in addition to master in future.
> > >>
> > >> By the way, I tried to update [2] on the Confluence wiki, but it seems
> > >> I don't have edit permission.
> > >> Masatake also asks that before [3], but I think he still doesn't have
> > >> it. Would someone give it to us?
> > >>
> > >> [1]:
> > >>
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20Bigtop%20AND%20fixVersion%20%3D%201.5.0%20AND%20resolution%20%3D%20Unresolved
> > >> [2]:
> https://cwiki.apache.org/confluence/display/BIGTOP/How+to+release
> > >> [3]:
> > >>
> https://lists.apache.org/thread.html/r7c98d7c95265827ae2f38d6a6e62b5e4ab7e8d4a325ac44711d89fb2%40%3Cdev.bigtop.apache.org%3E
> > >>
> > >> Kengo Seki 
> > >>
> > >
>

Re: Cut the 1.5.0 release branch

[Evans Ye]

BTW, super exciting to see the progress!
Let me know what help you need. In previous releases generating and signing
binary convenient packages is the most time consuming part.

Evans Ye  於 2020年11月7日 週六 下午1:40寫道：

> Cutting the release branch is OK if we're close to the release in short
> time.
>
> For the wiki permission, I've granted Masatake and you with full
> privileges. Let me know if you still have a problem.
>
> Hadoop 3 as discussed is better to be merged in 2.0.
>
> Evans
>
>
> Kengo Seki  於 2020年11月6日 週五 下午6:11寫道：
>
>> Hi everyone (especially committers),
>>
>> It's a status update of the 1.5.0 release.
>> We have only four or five issues that should be resolved before 1.5.0
>> at this time [1]
>> (let me know if there are other issues that should be included), while we
>> have
>> several PRs waiting to be merged, e.g., upgrading Hadoop to 3.x and
>> Flink to 1.11,
>> so I've just cut the 1.5.0 release branch (I followed the steps
>> described in section two
>> of the "How to release" document [2], but let me know if there's any
>> mistakes).
>> So, if some PR targets the 1.5.0 release, please merge it into
>> branch-1.5 in addition to master in future.
>>
>> By the way, I tried to update [2] on the Confluence wiki, but it seems
>> I don't have edit permission.
>> Masatake also asks that before [3], but I think he still doesn't have
>> it. Would someone give it to us?
>>
>> [1]:
>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20Bigtop%20AND%20fixVersion%20%3D%201.5.0%20AND%20resolution%20%3D%20Unresolved
>> [2]: https://cwiki.apache.org/confluence/display/BIGTOP/How+to+release
>> [3]:
>> https://lists.apache.org/thread.html/r7c98d7c95265827ae2f38d6a6e62b5e4ab7e8d4a325ac44711d89fb2%40%3Cdev.bigtop.apache.org%3E
>>
>> Kengo Seki 
>>
>

Re: Cut the 1.5.0 release branch

[Evans Ye]

Cutting the release branch is OK if we're close to the release in short
time.

For the wiki permission, I've granted Masatake and you with full
privileges. Let me know if you still have a problem.

Hadoop 3 as discussed is better to be merged in 2.0.

Evans


Kengo Seki  於 2020年11月6日 週五 下午6:11寫道：

> Hi everyone (especially committers),
>
> It's a status update of the 1.5.0 release.
> We have only four or five issues that should be resolved before 1.5.0
> at this time [1]
> (let me know if there are other issues that should be included), while we
> have
> several PRs waiting to be merged, e.g., upgrading Hadoop to 3.x and
> Flink to 1.11,
> so I've just cut the 1.5.0 release branch (I followed the steps
> described in section two
> of the "How to release" document [2], but let me know if there's any
> mistakes).
> So, if some PR targets the 1.5.0 release, please merge it into
> branch-1.5 in addition to master in future.
>
> By the way, I tried to update [2] on the Confluence wiki, but it seems
> I don't have edit permission.
> Masatake also asks that before [3], but I think he still doesn't have
> it. Would someone give it to us?
>
> [1]:
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20Bigtop%20AND%20fixVersion%20%3D%201.5.0%20AND%20resolution%20%3D%20Unresolved
> [2]: https://cwiki.apache.org/confluence/display/BIGTOP/How+to+release
> [3]:
> https://lists.apache.org/thread.html/r7c98d7c95265827ae2f38d6a6e62b5e4ab7e8d4a325ac44711d89fb2%40%3Cdev.bigtop.apache.org%3E
>
> Kengo Seki 
>

Re: Update

[Evans Ye]

In fact, the original deal of our resource is as follows:

> 1 m3.2xlarge for CI
> 4 m3.xlarge for CI and demo
> 3 1TB EBS volumes
> 5 elastic IP addresses

So technically we should not use that 2 additional 1T volumes (created in
2018).
Instead, I think what we can do is to split up one of the existing 1TB
volumes(ex: attached to slave07) into smaller volumes for slave02, 03.


Konstantin Boudnik  於 2020年11月4日 週三 下午2:28寫道：

> Kengo,
>
> We had an agreement with EMR folks that we are using the resources
> available
> to us and it is included into their budget (or something to this extent).
> If
> you see some of the resources available under our account - I don't see
> why we
> can't use them.
>
> If for whatever reason we need to expand the pool, that would require a
> separate conversation with nice folks from that team, I imagine. Please
> let me
> know if I can help with this going forward.
>
> Thanks!
>   Cos
>
> On Wed, Nov 04, 2020 at 11:11AM, Kengo Seki wrote:
> > Thanks for the comment, Cos! I was able to start docker service on
> > docker-slave-02 without replacing and am running some Jenkins jobs on
> > it now, so I'll replace it in the short future.
> > I have a few things that I'd like to ask additionally:
> >
> > * docker-slave-02 and 03 have a gp2 storage as a root volume that has
> > only 8GiB capacity, and they sometimes run short and stop the CI.
> >   May I increase them to 20 or 30 GiB when I replace those instances?
> > (I'm not sure what is our budget)
> >
> > * They use an instance store with 30GiB to put docker images into it,
> > and they also sometimes run short.
> >   It seems there are two unused volumes with 1TiB (vol-ae71114e and
> > vol-4efa69ae) on AWS console.
> >   May I attach them to 02 and 03 instead of instance stores, or are
> > they backups or something?
> >
> > Kengo Seki 
> >
> > On Mon, Nov 2, 2020 at 6:41 PM Konstantin Boudnik 
> wrote:
> > >
> > > I'd say let replace the broken one. I don't think there's a sentimental
> > > value attached ;)
> > >
> > > --
> > > With regards,
> > >Cos
> > >
> > > On 02.11.2020 08:16, Kengo Seki wrote:
> > > > Thanks for updating Olaf! I've just noticed the Jenkins UI became
> cool :)
> > > > Regarding docker-slave-02, I'll try to replace it after waiting for a
> > > > while to make sure there's no objection.
> > > >
> > > > Kengo Seki 
> > > >
> > > > On Mon, Nov 2, 2020 at 1:39 PM Jun HE  wrote:
> > > >>
> > > >> Thanks a lot for the update, Olaf!
> > > >>
> > > >> Olaf Flebbe  于2020年10月31日周六 上午3:24写道：
> > > >>
> > > >>> Hi,
> > > >>>
> > > >>> All machines patched. Jenkins and it plugins are updated:
> > > >>>
> > > >>> Things to be noted:
> > > >>>
> > > >>> * Slave 2 seems to be in serious problems. The disk image seems to
> be
> > > >>> corrupt, I would say:
> > > >>> One of the problems: docker does not start any more.
> > > >>> Is there anything important on it ? If yes please contact me. I
> would
> > > >>> recommend to set up slave2 from scratch again.
> > > >>>
> > > >>> * There was a warning regarding Copy Artifacts Plugin. It now
> imposes
> > > >>> stricter rules. Not sure if there is a job depending on it.
> > > >>>
> > > >>> * I removed the CVS plugin.
> > > >>>
> > > >>> Everything else seem to working as usual.
> > > >>>
> > > >>> Best,
> > > >>> Olaf
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > >  Am 30.10.2020 um 19:09 schrieb Olaf Flebbe :
> > > 
> > >  Hi,
> > > 
> > >  I am doing an update of the machines in CI . Seems a couple of
> security
> > > >>> fixes are to be applied.
> > > 
> > >  Olaf
> > > >>>
> > > >>>
>

Re: Current status of the next release

[Evans Ye]

Great! Thanks for the update.

For the release there is still a lot of manual effort needed. Let me know
when you'd like to start the process so I can help.

Kengo Seki  於 2020年9月23日 週三 22:49 寫道：

> Hi everyone,
>
> Let me share the current status of the next release.
> We've steadily moved forward and almost reached the goal.
> I really appreciate your cooperation and contribution.
>
> The attached image is the current packaging matrix [1].
> As you can see, we've succeeded on building most of all combinations
> between components and distros/platforms, though ppc64 is temporarily
> disabled due to the machine problem.
>
> The only build failure is Spark on CentOS 7 for arm64 (and ppc64),
> but it's tracked as BIGTOP-3397 and is supposed to be resolved
> once the next version of snappy-java is released and we add a small patch
> for Spark that uses it (or, if that release doesn't make it, we can
> build it ourselves).
>
> So, our remaining task is to complete the deployment and smoke test matrix
> [2].
> Currently, the following components have not succeeded on any
> distro/platform yet,
> so I'm going to begin with these components.
>
> - GPDB
> - Livy
> - Oozie (already filed as BIGTOP-3406)
> - QFS
> - Spark
>
> I'm planning to cut the release branch once all components passed
> their smoke test
> on at least one distro/platform for deb and rpm respectively
> (e.g., it's regarded as OK if the smoke test succeeds on x86_64/CentOS
> 7 and arm64/Debian 9),
> and hopefully, I'd like to release v1.5.0 within this October.
> Let me know if this schedule is not convenient for you :)
>
> [1]: https://ci.bigtop.apache.org/job/Bigtop-trunk-packages/
> [2]: https://ci.bigtop.apache.org/job/Bigtop-trunk-smoke-tests/
>
> Kengo Seki 
>

Re: Bigtop website

[Evans Ye]

 Yes. That's clear to me now. Thanks for the update. I checked the CI job
and it looks good. Thanks!


Olaf Flebbe  於 2020年9月22日 週二 上午1:44寫道：

> Hi Evans,
>
> ok, let me explain it in detail:
>
> * The website was completly static content previously as well.
> * I did not (until now) do any change to the pages.
> * The website is build by running „mvn site“ in bigtop root.
> * The target/site directory has to be put onto a webserver.
>
> The only thing what is changing is _how_ to put it on a webserver.
> Previously bigtop used the maven site plugin to stage it to a CMS via svn.
> See  in pom.xml.
>
> The now preferred way is  to check the html pages into the git repository
> (rather svn) on a special branch "asf-site" , together with a file
> „.asf.yaml“ in the root with special content. When we push it a special
> handler (gitpubsub) is triggered to push the static content to the website.
>
> On builds.apache.org <http://builds.apache.org/> system this is done in
> job „Bigtop/site“. Additionally there is a special node "git-websites“
> which will allow a Jenkins job to push to git, because git credentials are
> already injected into the job by INFRA.
>
> That’s how to push to bigtop.apache.org <http://bigtop.apache.org/>
> ——
>
> For testing purposes I pushed to bigtop.staged.apache.org <
> http://bigtop.staged.apache.org/> via branch asf-staging and .asf.yaml ,
> but the mechanisms are the same.
>
>
> Hope that clarifies the situation,
> Olaf
>
>
> > Am 21.09.2020 um 17:34 schrieb Evans Ye :
> >
> > Hi Olaf,
> >
> > Thanks for the work!
> > It seems the site code has changed quite a bit. Could you briefly share
> > what changes were made? For example, it seems that the pages are all
> static
> > now. From the CI job it seems that jenkins slave provided by infra has
> the
> > permission to push to any git repository. Have we done something
> > differently? Thanks!
> >
> > - Evans
> >
> >
> > Olaf Flebbe  於 2020年9月20日 週日 下午9:51寫道：
> >
> >> Hi *,
> >>
> >> Finally found time to look into implementing the workflow for pushing
> our
> >> website automatically bigtop.apache.org <http://bigtop.apache.org/>.
> >>
> >> As a test I create a job checkin the git repository once a day for new
> >> commits an deploying to bigtop.staged.apache.org <
> >> http://bigtop.staged.apache.org/>
> >>
> >> Please have a look if you see anything weird / unexpected on it and
> report.
> >> Changing the website is now as easy as having a git commit on src/site .
> >>
> >> If I get no or positive (gasp!)  feedback I will change the job in a few
> >> days to update the prod website instead
> >>
> >> Additional ideas : Using a jenkins pipeline job, refactor pom.xml to
> >> remove now obsolete parts.
> >>
> >> Best
> >>   Olaf
>
>

Re: Developer documenation

[Evans Ye]

Awesome!

Staring from docker images and build up packages would be recommended so
that you can get the idea about how it works.
When you'd like to customize your own big data stack based on bigtop, you
start to build your own docker images instead.

- Evans

Olaf Flebbe  於 2020年9月23日 週三 上午12:05寫道：

> Hi Manfred,
>
> Great!
>
> AFAIK the most up to date information on how to (re)compile an package you
> will find here
>
> https://cwiki.apache.org/confluence/display/BIGTOP/How+to+build+Bigtop-trunk
> <
> https://cwiki.apache.org/confluence/display/BIGTOP/How+to+build+Bigtop-trunk
> >
>
> I am assuming you are targeting linux, not macos, right?
>
> IMO When developing it makes sense to compile „within" the provided docker
> images or improving the docker images.
> The CI’s job is to ensure changes do work on all platforms.
>
> I was using a macbook pro 16GB when I developed packages, so it is
> possible, though not always optimal, since docker filesystem (volume mounts
> from macos to docker guest) is buggy and slow.
>
> Running smoke tests on a mac is possible with workarounds.
>
> Needed workarounds depend very much on what you are trying to accomplish:
> VMware Fusion, docker4mac or depending on CI are at least 3 options I would
> recommend depending on usecase.
>
> Best
> Olaf
>
>
>
>
> > Am 22.09.2020 um 15:48 schrieb Manfred PAUL :
> >
> > Hi,
> >
> > I would like to participate in developing bigtop. For that I am searching
> > for some material (doc, wiki, video, etc) to get all the information
> needed
> > to build the bigtop distribution.
> >
> > All the information I found was for older versions of bigtop (1.1, 1.2
> and
> > 1.3) but might still be up to date, so I am asking.
> >
> > Is written that a developer should use the pre-built docker images for
> > building the distribution and not set up his or her own toolchain. Or
> > should we use only a CI? Is it even possible to build everything on a
> 16GB
> > Macbook Pro? Etc.
> >
> > Thanks
> >
> > Manfred
>
>

Re: Bigtop website

[Evans Ye]

Hi Olaf,

Thanks for the work!
It seems the site code has changed quite a bit. Could you briefly share
what changes were made? For example, it seems that the pages are all static
now. From the CI job it seems that jenkins slave provided by infra has the
permission to push to any git repository. Have we done something
differently? Thanks!

- Evans


Olaf Flebbe  於 2020年9月20日 週日 下午9:51寫道：

> Hi *,
>
> Finally found time to look into implementing the workflow for pushing our
> website automatically bigtop.apache.org .
>
> As a test I create a job checkin the git repository once a day for new
> commits an deploying to bigtop.staged.apache.org <
> http://bigtop.staged.apache.org/>
>
> Please have a look if you see anything weird / unexpected on it and report.
> Changing the website is now as easy as having a git commit on src/site .
>
> If I get no or positive (gasp!)  feedback I will change the job in a few
> days to update the prod website instead
>
> Additional ideas : Using a jenkins pipeline job, refactor pom.xml to
> remove now obsolete parts.
>
> Best
>Olaf

Re: PPC CI server failure

[Evans Ye]

Ok. Thanks for doing this to get the ball rolling.

Kengo Seki  於 2020年9月17日 週四 10:29 寫道：

> Thank you for your help, Amir!
> It's just a heads-up, I temporarily disabled builds for ppc in the
> following Jenkins jobs so that they can finish.
>
> * Docker-Puppet-Trunk
> * Docker-Puppet-Trunk-pull
> * Docker-Toolchain-Trunk
> * Docker-Toolchain-Trunk-pull
>
> * Bigtop-trunk-packages
> * Bigtop-trunk-repos
>
> * Remove-All-Docker-Containers-Except-Nexus
> * Remove-Dangling-Docker-Images
> * Remove-Inactive-Containers
>
> Kengo Seki 
>
> On Wed, Sep 16, 2020 at 7:35 PM Evans Ye  wrote:
> >
> > Awesome! Nice to hear from you, buddy!
> >
> > MrAsanjar  於 2020年9月16日 週三 上午3:54寫道：
> >
> > > Hi Evans,
> > > Let me see what I can do. Give me 24 hr :)
> > >
> > > On Tue, Sep 15, 2020 at 10:51 AM Evans Ye  wrote:
> > >
> > > > Yes. I think the action is correct. However [2] might be a different
> > > thing
> > > > for PPC integration in Hadoop.
> > > >
> > > > Amir,
> > > > Could you confirm?
> > > >
> > > > Kengo Seki  於 2020年9月14日 週一 下午9:56寫道：
> > > >
> > > >> Thank you for the advice, Evans!
> > > >> Let me confirm about "PPC machine owners". According to Amir's JIRA
> > > >> issues [1][2] and the powered-by list in the OSU site [3], we're
> using
> > > >> a VM hosted by OSU OSL, right?
> > > >> If it's correct, I'm going to ask them for help via
> > > >> powerdev-requ...@osuosl.org.
> > > >>
> > > >> [1]:
> > > >>
> > >
> https://issues.apache.org/jira/browse/INFRA-11467?focusedCommentId=15300982=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-15300982
> > > >> [2]: https://issues.apache.org/jira/browse/INFRA-12014
> > > >> [3]:
> > > https://osuosl.org/services/powerdev/current-projects/#foss-projects
> > > >>
> > > >> Kengo Seki 
> > > >>
> > > >>
> > > >> On Mon, Sep 14, 2020 at 2:06 PM Evans Ye 
> wrote:
> > > >> >
> > > >> > I'd suggest to reach out to PPC machine owners. Worst case Is we
> can
> > > >> > temporary  drop the PPC support to move the release forward.
> > > >> >
> > > >> > Kengo Seki  於 2020年9月14日 週一 12:44 寫道：
> > > >> >
> > > >> > > Hi everyone,
> > > >> > >
> > > >> > > Let me share information about the CI environment.
> > > >> > > The worker node for ppc64le is currently offlined, so I just
> killed
> > > >> all
> > > >> > > jobs
> > > >> > > in the queue waiting for it gets back. Its status is as follows.
> > > >> > >
> > > >> > > - According to the result of `who -b`, that machine seems to be
> > > >> rebooted
> > > >> > >   on 2020-09-11 for some reason (probably unexpectedly).
> > > >> > >
> > > >> > > - According to the result of dmesg, the root volume was mounted
> > > >> > >   in read-only mode because of a fsck failure.
> > > >> > >
> > > >> > >   [   34.840681] EXT4-fs (vda1): Couldn't remount RDWR because
> of
> > > >> > > unprocessed orphan inode list.  Please umount/remount instead
> > > >> > >   [   60.714110] cgroup: new mount options do not match the
> existing
> > > >> > > superblock, will be ignored
> > > >> > >   [  316.385805] EXT4-fs (vda1): error count since last fsck:
> 9459
> > > >> > >   [  316.385824] EXT4-fs (vda1): initial error at time
> 1540294049:
> > > >> > > ext4_validate_inode_bitmap:134
> > > >> > >   [  316.385826] EXT4-fs (vda1): last error at time 1596881526:
> > > >> > > ext4_free_inode:383
> > > >> > >
> > > >> > > It looks like some fsck work (and replacing the volume, if it
> fails)
> > > >> > > are required,
> > > >> > > but I'm not sure if I could run something like `e2fsck -p`,
> because
> > > >> > > I'm also not sure
> > > >> > > where does that machine exist or who's managing it.
> > > >> > > (I slightly thought it was running as a VM with QEMU on some EC2
> > > >> > > instance, but I couldn't find it)
> > > >> > >
> > > >> > > > Cos, Evans, Olaf
> > > >> > > Would you provide any suggestions?
> > > >> > >
> > > >> > > Kengo Seki 
> > > >> > >
> > > >>
> > > >
> > >
>

Re: PPC CI server failure

[Evans Ye]

Awesome! Nice to hear from you, buddy!

MrAsanjar  於 2020年9月16日 週三 上午3:54寫道：

> Hi Evans,
> Let me see what I can do. Give me 24 hr :)
>
> On Tue, Sep 15, 2020 at 10:51 AM Evans Ye  wrote:
>
> > Yes. I think the action is correct. However [2] might be a different
> thing
> > for PPC integration in Hadoop.
> >
> > Amir,
> > Could you confirm?
> >
> > Kengo Seki  於 2020年9月14日 週一 下午9:56寫道：
> >
> >> Thank you for the advice, Evans!
> >> Let me confirm about "PPC machine owners". According to Amir's JIRA
> >> issues [1][2] and the powered-by list in the OSU site [3], we're using
> >> a VM hosted by OSU OSL, right?
> >> If it's correct, I'm going to ask them for help via
> >> powerdev-requ...@osuosl.org.
> >>
> >> [1]:
> >>
> https://issues.apache.org/jira/browse/INFRA-11467?focusedCommentId=15300982=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-15300982
> >> [2]: https://issues.apache.org/jira/browse/INFRA-12014
> >> [3]:
> https://osuosl.org/services/powerdev/current-projects/#foss-projects
> >>
> >> Kengo Seki 
> >>
> >>
> >> On Mon, Sep 14, 2020 at 2:06 PM Evans Ye  wrote:
> >> >
> >> > I'd suggest to reach out to PPC machine owners. Worst case Is we can
> >> > temporary  drop the PPC support to move the release forward.
> >> >
> >> > Kengo Seki  於 2020年9月14日 週一 12:44 寫道：
> >> >
> >> > > Hi everyone,
> >> > >
> >> > > Let me share information about the CI environment.
> >> > > The worker node for ppc64le is currently offlined, so I just killed
> >> all
> >> > > jobs
> >> > > in the queue waiting for it gets back. Its status is as follows.
> >> > >
> >> > > - According to the result of `who -b`, that machine seems to be
> >> rebooted
> >> > >   on 2020-09-11 for some reason (probably unexpectedly).
> >> > >
> >> > > - According to the result of dmesg, the root volume was mounted
> >> > >   in read-only mode because of a fsck failure.
> >> > >
> >> > >   [   34.840681] EXT4-fs (vda1): Couldn't remount RDWR because of
> >> > > unprocessed orphan inode list.  Please umount/remount instead
> >> > >   [   60.714110] cgroup: new mount options do not match the existing
> >> > > superblock, will be ignored
> >> > >   [  316.385805] EXT4-fs (vda1): error count since last fsck: 9459
> >> > >   [  316.385824] EXT4-fs (vda1): initial error at time 1540294049:
> >> > > ext4_validate_inode_bitmap:134
> >> > >   [  316.385826] EXT4-fs (vda1): last error at time 1596881526:
> >> > > ext4_free_inode:383
> >> > >
> >> > > It looks like some fsck work (and replacing the volume, if it fails)
> >> > > are required,
> >> > > but I'm not sure if I could run something like `e2fsck -p`, because
> >> > > I'm also not sure
> >> > > where does that machine exist or who's managing it.
> >> > > (I slightly thought it was running as a VM with QEMU on some EC2
> >> > > instance, but I couldn't find it)
> >> > >
> >> > > > Cos, Evans, Olaf
> >> > > Would you provide any suggestions?
> >> > >
> >> > > Kengo Seki 
> >> > >
> >>
> >
>

Re: PPC CI server failure

[Evans Ye]

Yes. I think the action is correct. However [2] might be a different thing
for PPC integration in Hadoop.

Amir,
Could you confirm?

Kengo Seki  於 2020年9月14日 週一 下午9:56寫道：

> Thank you for the advice, Evans!
> Let me confirm about "PPC machine owners". According to Amir's JIRA
> issues [1][2] and the powered-by list in the OSU site [3], we're using
> a VM hosted by OSU OSL, right?
> If it's correct, I'm going to ask them for help via
> powerdev-requ...@osuosl.org.
>
> [1]:
> https://issues.apache.org/jira/browse/INFRA-11467?focusedCommentId=15300982=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-15300982
> [2]: https://issues.apache.org/jira/browse/INFRA-12014
> [3]: https://osuosl.org/services/powerdev/current-projects/#foss-projects
>
> Kengo Seki 
>
>
> On Mon, Sep 14, 2020 at 2:06 PM Evans Ye  wrote:
> >
> > I'd suggest to reach out to PPC machine owners. Worst case Is we can
> > temporary  drop the PPC support to move the release forward.
> >
> > Kengo Seki  於 2020年9月14日 週一 12:44 寫道：
> >
> > > Hi everyone,
> > >
> > > Let me share information about the CI environment.
> > > The worker node for ppc64le is currently offlined, so I just killed all
> > > jobs
> > > in the queue waiting for it gets back. Its status is as follows.
> > >
> > > - According to the result of `who -b`, that machine seems to be
> rebooted
> > >   on 2020-09-11 for some reason (probably unexpectedly).
> > >
> > > - According to the result of dmesg, the root volume was mounted
> > >   in read-only mode because of a fsck failure.
> > >
> > >   [   34.840681] EXT4-fs (vda1): Couldn't remount RDWR because of
> > > unprocessed orphan inode list.  Please umount/remount instead
> > >   [   60.714110] cgroup: new mount options do not match the existing
> > > superblock, will be ignored
> > >   [  316.385805] EXT4-fs (vda1): error count since last fsck: 9459
> > >   [  316.385824] EXT4-fs (vda1): initial error at time 1540294049:
> > > ext4_validate_inode_bitmap:134
> > >   [  316.385826] EXT4-fs (vda1): last error at time 1596881526:
> > > ext4_free_inode:383
> > >
> > > It looks like some fsck work (and replacing the volume, if it fails)
> > > are required,
> > > but I'm not sure if I could run something like `e2fsck -p`, because
> > > I'm also not sure
> > > where does that machine exist or who's managing it.
> > > (I slightly thought it was running as a VM with QEMU on some EC2
> > > instance, but I couldn't find it)
> > >
> > > > Cos, Evans, Olaf
> > > Would you provide any suggestions?
> > >
> > > Kengo Seki 
> > >
>

Re: PPC CI server failure

[Evans Ye]

I'd suggest to reach out to PPC machine owners. Worst case Is we can
temporary  drop the PPC support to move the release forward.

Kengo Seki  於 2020年9月14日 週一 12:44 寫道：

> Hi everyone,
>
> Let me share information about the CI environment.
> The worker node for ppc64le is currently offlined, so I just killed all
> jobs
> in the queue waiting for it gets back. Its status is as follows.
>
> - According to the result of `who -b`, that machine seems to be rebooted
>   on 2020-09-11 for some reason (probably unexpectedly).
>
> - According to the result of dmesg, the root volume was mounted
>   in read-only mode because of a fsck failure.
>
>   [   34.840681] EXT4-fs (vda1): Couldn't remount RDWR because of
> unprocessed orphan inode list.  Please umount/remount instead
>   [   60.714110] cgroup: new mount options do not match the existing
> superblock, will be ignored
>   [  316.385805] EXT4-fs (vda1): error count since last fsck: 9459
>   [  316.385824] EXT4-fs (vda1): initial error at time 1540294049:
> ext4_validate_inode_bitmap:134
>   [  316.385826] EXT4-fs (vda1): last error at time 1596881526:
> ext4_free_inode:383
>
> It looks like some fsck work (and replacing the volume, if it fails)
> are required,
> but I'm not sure if I could run something like `e2fsck -p`, because
> I'm also not sure
> where does that machine exist or who's managing it.
> (I slightly thought it was running as a VM with QEMU on some EC2
> instance, but I couldn't find it)
>
> > Cos, Evans, Olaf
> Would you provide any suggestions?
>
> Kengo Seki 
>

Re: builds.apache.org: bigtop jobs will get lost

[Evans Ye]

To my best knowledge those are dated jobs. I doubt anyone is comsuing the
results. I think we can delete them.

Olaf Flebbe  於 2020年8月21日 週五 03:38 寫道：

> Hi,
>
> honestly I am a bit surprised, looking at builds.apache org:
>
> There are a couple of jobs running there:
>
> One Job:
> "Deploys Bigtop test execution into snapshot repository“
>
> mvn clean
>
> export HADOOP_CONF_DIR=
> export HADOOP_HOME=
> mvn -f bigtop-tests/test-execution/pom.xml -DskipTests -DskipITs deploy
> -DrepositoryId=apache.snapshots.https
>
> Next Job:
> Deploys a top level Bigtop pom file into snapshot repository
> mvn deploy
>
> And a post job of both of them
>
> Essentially doing
> mvn -f bigtop-tests/test-artifacts/pom.xml -U clean deploy
>
> Question from my side:
> Do we need them, since bui...@apache.org  will
> be turned off  in *two* days ?
>
> This seems the result of the job:
>
> https://search.maven.org/search?q=bigtop <
> https://search.maven.org/search?q=bigtop>
>
> IMO this does not make sense at all any more and can be deleted/lost for
> good. Or do I miss something ?
>
> Best
>Olaf

Re: Updated the ci.bigtop.apache.org certificate

[Evans Ye]

Thanks Kengo!

Jun HE  於 2020年6月12日 週五 上午9:54寫道：

> Thanks, Kengo!
>
> To make the life easier, would it be possible to use automatically renewal,
> like crontab, or something like this (
> https://github.com/acmesh-official/acme.sh)?
>
> Olaf Flebbe  于2020年6月11日周四 下午9:40写道：
>
> > Thanks!
> >
> > > Am 11.06.2020 um 14:24 schrieb Kengo Seki :
> > >
> > > Just for your information, I've just renewed the certificate for
> > > ci.bigtop.apache.org in accordance with [1].
> > > Let me know if you find something wrong with it.
> > >
> > > [1]:
> >
> https://cwiki.apache.org/confluence/display/BIGTOP/Bigtop+CI+Setup+Guide#BigtopCISetupGuide-Renewingthecert
> > >
> > > Kengo Seki 
> >
> >
>

Re: New Committer: Yuqi Gu

[Evans Ye]

Congrats Yuqi, well deserved!
We've been working together quite a long time and I'm really impressed by
the passion you have.
Please keep on the what you're doing. The committership is for you to
evolve the features more efficiently.
Furthermore, we look forward to seeing you to take on a more important role
at Bigtop in the future.

- Evans

Jun HE  於 2020年6月11日 週四 上午10:13寫道：

> The Project Management Committee (PMC) for Apache Bigtop has asked Yuqi Gu
> to become a committer and we are pleased to announce that he has accepted!
>
> His ASF account has been created as: guy...@apache.org
>
> Being a committer enables easier contribution to the project since there
> is no
> need to go via the proxy patch submission process. This should enable
> better
> productivity.
>
> Apache Bigtop practices CTR (commit-then-review) development process which
> means that you can checkin the code without a +1 from a committer. This
> doesn't
> mean that you're encouraged to do so. Instead, we strongly recommend you
> to
> seek for feedback before you commit. Please read the document before you
> act:
>
> * https://cwiki.apache.org/confluence/display/BIGTOP/CTR+Model
>
> If in doubt, exercise your best judgement and don't hesitate to ask
> questions on
> this very mailing list.
>
> We're happy to have you on board and looking for many more contributions to
> come. Please join me in congratulating Yuqi Gu!
>
> Regards,
>
> Jun (on behalf of the ASF BigTop PMC)
>

Re: [DISCUSS] [VOTE] Release Ambari-Mpack as a seperate top component in Bigtop

[Evans Ye]

Olaf,

Yuqi was guided by me:
https://github.com/apache/bigtop/pull/555#issuecomment-634819743

Usually we'll put up a DISCUSS first and the a VOTE. So -1 and its reason
will be there in DISCUSS most of the time.
If the discussion is positive, then indeed there's no need to vote as we
pretty much reach the consensus already. In other case that everyone has
expressed there're though but still no consensus reached. We can only put
in into a majority vote and move forward based on the vote result.

I agree with those items you listed as priorities. Kengo as RM is driving
the 1.5 release and we should polish those things.

> Really? MPack is a amabari only technology, that seems not to even be
defined properly. How could this ever end in more contributions to Bigtop ?
We should take more user feedback on this because anyhow it solve user's
problems. And if we have contributors who's willing to maintain it. It
should be something we can adopt. The question we should ask here is are
you willing to be the maintainer going forward?

Best,
Evans

Olaf Flebbe  於 2020年6月8日 週一 上午12:07寫道：

> Hi,
>
> Sorry, I am super lost about the implications of that vote.
>
> Why do we need a vote in the first place?
> I am familiar with votes for additional committers, PMC additions and
> releases. For these votes we have the concept of binding votes by PMC
> members.
>
> Why should I care about this ?
> Does it have legal implications?
>
> Who will maintain this ?
> I am a bit surprised that we do anything with ambari, since we not even
> have a maintainer for it
> grep ambari MAINTAINERS.txt -> nothing
>
> What problem/case is this additional component trying to solve?
> If you are asking for a vote I would like to have a more elaborate
> explanation. I looked at the linked ticket: Sorry I am totally lost about
> what is this about. If you would have simply done it, I wouldn’t have cared.
>
> As I understand it should somehow enable more contributions.
> Really? MPack is a amabari only technology, that seems not to even be
> defined properly. How could this ever end in more contributions to Bigtop ?
>
> What will happen if someone is -1 on this vote ?
>
> If a contributor cares about adopting Bigtop to new business cases, that’s
> very fine to me. Please go ahead. But please don’t ask for a vote. All
> committers still have the chance to reject patches.
>
> Generally I am concerned that we care about an (from Bigtop side)
> unmaintained project while neglecting our core infrastructure and packages
> at the same time.
> 1) The ci.bigtop.apache.org certificate expired again
> 2) The bigtop-trunk-packages do not reflect the architectures we ought to
> support (ubuntu-18.04 for instance)
> 3) Trunk-packages for amd64 looks like garbage
> 4) Still no-one working on Hadoop3
> 5) CI needs to be redone (with pipeline jobs) .
> IMO we might have wrong priorities.
>
> Best,
> Olaf
>
>
>
>
> > Am 04.06.2020 um 19:04 schrieb Ganesh Raju :
> >
> > +1
> >
> > On Thu, Jun 4, 2020 at 12:01 PM Matt Andruff 
> > wrote:
> >
> >> +1
> >>
> >> On Thu., Jun. 4, 2020, 02:35 Yuqi Gu,  wrote:
> >>
> >>> Hi folks,
> >>>
> >>> As the discussion with Evans Ye, Jun He, and Matt Andruff (from Ambari)
> >> on
> >>> https://github.com/apache/bigtop/pull/555#issuecomment-635228180:
> >>>
> >>> For users / Bigtopers could easily modify Mapck and make contributions
> to
> >>> it like bugfix and adding more services,
> >>> How about to decouple Mpack from Ambari and put Mpack as a standalone
> >>> component in Bigtop, like *bigtop-packages/src/common/bigtop-mpack*?
> >>> If so, it's convenient for users to get and install Mpack rpm/deb
> >> packages
> >>> in their own Ambari cluster.
> >>>
> >>> The VOTE:
> >>> [ ] +1, Agree to put Mpack as a standalone component.
> >>> [ ] +0, I don't care either way.
> >>> [ ] -1, do *not * Agree.
> >>>
> >>> Look forward to your comments and feedback, thanks.
> >>>
> >>> BRs,
> >>> Yuqi
> >>>
> >>
> >
> >
> > --
> > IRC: ganeshraju@#linaro on irc.freenode.ne <http://irc.freenode.net/>t
>
>

Re: Bigtop Hadoop 3.2.1 error

[Evans Ye]

> 6) Beside installing deb on my machine and playing with it, what else
tests are needed can be done to ensure it works as intended.

We've a testing framework that runs smoke tests pretty much in the
automatic way. Checkout [1].
Do aware that the deployment and testing feature is not fully covered
across all bigtop components. Checkout the supporting matrix[2]

[1]
https://cwiki.apache.org/confluence/display/BIGTOP/Quickstart+Guide%3A+Bigtop+Integration+Test+Framework+2.0
[2]
https://cwiki.apache.org/confluence/display/BIGTOP/Overview+of+Bigtop+1.4.0+Support+Matrix

As Olaf said, just post your question here and we'd be happy to answer.

- Evans


Olaf Flebbe  於 2020年6月6日 週六 下午9:48寫道：

> Hi
>
> Thank you for you questions!
>
>
>
> Am 06.06.2020 um 10:41 schrieb Jagat Singh :
>
> Hi,
>
> I am trying to use Bigtop to create deb file for Hadoop 3.2.1. I am using
> Ubuntu 18.04.
>
> In the process of doing this, many questions came to my mind which I
> wanted to learn about.
>
> 1) I changed the bom file and ran gradle hadoop-deb command. Is this the
> correct process to create any deb or rpm?
>
>
> As a Bigtop User: no . You are not expected to roll your own Distribution .
> As a Developer: yes somehow, that’s usually the first try to upgrade a
> package.
> Since you already read the instructions: I like to stresst that you are
> supposed to use the docker containers (for git trunk
> bigtop/slaves:trunk-ubuntu-18.04 and run the gradle command within the
> container.
>
>
> 2) In what state you will use patches present in folder common/src/hadoop/
> , is it only usable when upstream system make additional changes after
> making a release?
>
>
> We try to use zero patches but if we find defects which are not fixed in
> the version we want to package, than we do patches. Or the build system is
> to be adapted to our needs… Usually we upstream fixes to the original
> project unless it is very bigtop specific. Most of the time we cherry-pick
> fixes from unreleased versions of the package.
>
> 3) Many times the build fails in between and I had to clean up build and
> output folder both and restart. What is the process to pick from where is
> failed to save time?
>
>
> No way, unfortunately. Neither deb nor rpm packaging have a way to resume
> where it failed.
>
> When I developed packaging I usually trying to do manually do the build,
> dependencies and try to automate stuff in the build scripts. This is very
> time consuming, indeed.
>
> Now you are hitting the ugly part. The upstream Hadoop project decided to
> rework all supporting start scripts, introduced different processes and
> frameworks with Hadoop3 , breaking our complete Hadoop setup.
>
> I did some initial work last year for Hadoop-3.1.2 in the "bigtop-alpha“
> git branch and dropped out of the project because of lack of time and
> changed priorities.
>
> If you like to invest into hadoop-3.2.1 I would recommend to look into
> this and merging with current master first.
>
>
> 4) Follow-up for 3), Building the hadoop and building deb for hadoop are
> two different things, what command can be used to do one after the another
> manually to save time in case one fails.
>
>
> Right, Building Hadoop should be straight forward, but packaging the new
> Hadoop 3.2 layout is hard, since Hadoop project decided to change to a
> monolithic approach. Without rewriting nearly all scripts it is not
> possible anymore to start daemons independently any more.
>
> I would recommend to do a manual installation first and start fresh with
> packaging the individual parts.
>
>
>
> > Task :hadoop-deb FAILED
> dpkg-source: warning: extracting unsigned source package
> (hadoop_3.2.1-1.dsc)
> dpkg-source: error: unpack target exists: hadoop-3.2.1
> dpkg-source: info: extracting hadoop in hadoop-3.2.1
>
> 4) How does Bigtop ensures compatability between products, example Hive is
> compiled with Hadoop version 3.2.1. Based on my understanding in the mvn -D
> commands there is an override and version it takes from the bom file the
> version information. Is this understanding correct? Do I still need to
> change any maven pom file for any software to ensure compatibility?
>
>
> You are right that is the way bigtop currently works. The crucial part is
> that it uses the local maven repository to pick up previous artifacts.
> (Dependency does "mvn install“ and mvn package of an package will pick up
> the previous compiled artifact from ~/.m2/repository ) .
>
> Additionally, some crucial parts like Hadoop or mapreduce jars are
> symlinked in packages so they are only installed once on the target system.
>
> I was proposing to not do this any more since we might break API’s that
> way (and we had that) . My approach in bigtop-alpha was to use the
> dependencies the devs chose and to accept different versions of a jar in
> the system, depending on the *network* API rather the *Java* API. That
> would allow us to compile packages independently.
>
>
> 5) Are there any more resources to 

Re: Starting 1.5.0 release process

[Evans Ye]

I think for the release process it's comprehensive. Let me know if you
have any questions because the whole release process is still quite manual.
We can indeed do it better with some automations.


Kengo Seki  於 2020年6月2日 週二 下午10:56寫道：

> Hi everyone,
>
> I think we've resolved most of the issues for the 1.5.0 release, so
> I'd like to start the release process
> in accordance with
> https://cwiki.apache.org/confluence/display/BIGTOP/How+to+release.
>
> First of all, we have to decide which remaining issues should be fixed.
> I'm seeing the following ones as blockers. Are there any issues to be
> included?
>
> - BIGTOP-3236: Zeppelin smoke tests implementation
> - BIGTOP-3319: Add Logstash as Bigtop component
> - BIGTOP-3343: Add Debian 10 and Ubuntu 18.04 support to the Docker
> provisioner
> - BIGTOP-3355: [Puppet] Make GPG check for repos a configuration and
> default to true
> - BIGTOP-3361: No libcrypt.so.1 in Fedora-31 for Arm64
> - BIGTOP-3362: Unable to install spakr-python package on CentOS8 due
> to lack of python package
> - BIGTOP-3363: Drop Apex, Hama and Tajo in the 1.5.0 release
>
> In addition to the above, I'm going to do the following tasks. Let me
> know if I'm missing something.
>
> - Re-run the Docker-Puppet-Trunk* and Docker-Toolchain-Trunk* Jenkins
> jobs to update and push Docker images, after all docker-related issues
> are resolved.
> - Configure and run the packaging (Bigtop-trunk-packages) and smoke
> test (Bigtop-trunk-smoke-tests) jobs on Jenkins so that we can confirm
> that there's no problem
> - Update support matrix for the 1.5.0 release on the wiki, just like
> 1.4.0 (
> https://cwiki.apache.org/confluence/display/BIGTOP/Overview+of+Bigtop+1.4.0+Support+Matrix
> )
>
> Kengo Seki 
>

Re: [DISCUSS] [VOTE] Release Ambari-Mpack as a seperate top component in Bigtop

[Evans Ye]

I'm +1 to this.

Yuqi Gu  於 2020年6月4日 週四 下午2:35寫道：

> Hi folks,
>
> As the discussion with Evans Ye, Jun He, and Matt Andruff (from Ambari) on
> https://github.com/apache/bigtop/pull/555#issuecomment-635228180:
>
> For users / Bigtopers could easily modify Mapck and make contributions to
> it like bugfix and adding more services,
> How about to decouple Mpack from Ambari and put Mpack as a standalone
> component in Bigtop, like *bigtop-packages/src/common/bigtop-mpack*?
> If so, it's convenient for users to get and install Mpack rpm/deb packages
> in their own Ambari cluster.
>
> The VOTE:
> [ ] +1, Agree to put Mpack as a standalone component.
> [ ] +0, I don't care either way.
> [ ] -1, do *not * Agree.
>
> Look forward to your comments and feedback, thanks.
>
> BRs,
> Yuqi
>

[jira] [Created] (BIGTOP-3355) [Puppet] Make GPG check for repos a configuration and default to true

[Evans Ye (Jira)]

Evans Ye created BIGTOP-3355:


 Summary: [Puppet] Make GPG check for repos a configuration and 
default to true
 Key: BIGTOP-3355
 URL: https://issues.apache.org/jira/browse/BIGTOP-3355
 Project: Bigtop
  Issue Type: Bug
  Components: deployment
Affects Versions: 1.4.0
Reporter: Evans Ye
Assignee: Evans Ye


Currently Bigtop Puppet deploys repos w/o GPG check. This is insecure and need 
to be fixed. We can make disable GPG check a configuration and default to true. 
For testing purpose, one can set it to false to run w/ local repo or the others.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[DISCUSS] Next Generation Bigtop CI Infra

[Evans Ye]

Hi folks,

I've been thinking about the revamp of bigtop's CI infra.
Currently what we have is purely a Jenkins cluster with manual managed:
1. hardware/machine resource
2. build env: docker, docker compose are all manual installed
3. job definition: all jobs are manual created and the definition is not
tracked w/ version control system.

I think for 2,3 there're some tools/solutions that can better manage and
automate the things. So that user can just take the code and build their CI
directly.

At Yahoo! there's a tool called screwdriver[1] which is a framework to do
CICD, however I don't think it fits our scenario well as we're not facing
production CD. Probably Jenkins 2.0 is more suitable? Anyone has experience
and suggestions?

[1] https://screwdriver.cd/

Evans

Re: [DISCUSS] [VOTE] Publish release-1.5.0 without ppc64le support for Zeppelin

[Evans Ye]

So the vote is to just drop Zeppelin support on PPC64LE, am I right?
If so I'm a +1.
In 1.4.0 release, we've a support matrix[1] so that it's more clear to user
what's there in Bigtop.

BTW PPC folks: if you'd like to work on the fix and make Zeppelin
supported. Feel free to -1 on this.
With limited resources, we'd like to focus on the core features. However if
someone would like to work on it its definitely welcomed.

[1]
https://cwiki.apache.org/confluence/display/BIGTOP/Overview+of+Bigtop+1.4.0+Support+Matrix

Evans


Yuqi Gu  於 2020年4月26日 週日 上午10:32寫道：

> Hi folks,
>
> Let's go back to the discussion on PR #631
>  for BIGTOP-3327.
>
> The Zeppelin build failed on ppc64le due to the dependency issue that the
> ppc64le binaries of protoc-gen-grpc-java are still not available.
>
> To make the new release move onward, How about to drop the ppc64le support
> for Zeppelin in the next release?
>
> The VOTE:
> [ ] +1, accept dropping the ppc64le support,
> [ ] +0, I don't care either way,
> [ ] -1, do not accept dropping the ppc64le support, or wanna be a
> volunteer to
> work on it.
>
>
> BR,
> Yuqi
>

[jira] [Created] (BIGTOP-3345) Failed to run smoke test due to jenkins do not have permission to cleanup workspace

[Evans Ye (Jira)]

Evans Ye created BIGTOP-3345:


 Summary: Failed to run smoke test due to jenkins do not have 
permission to cleanup workspace
 Key: BIGTOP-3345
 URL: https://issues.apache.org/jira/browse/BIGTOP-3345
 Project: Bigtop
  Issue Type: Bug
  Components: ci
Affects Versions: 1.4.0
Reporter: Evans Ye



{code:java}
ERROR: Failed to clean the workspace
jenkins.util.io.CompositeIOException: Unable to delete 
'/home/jenkins/workspace/Build-Deploy-Smoke-Test-Pull-Request-All-Distros/OS/centos-7-x86_64-deploy'.
 Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.
at 
jenkins.util.io.PathRemover.forceRemoveDirectoryContents(PathRemover.java:90)
at hudson.Util.deleteContentsRecursive(Util.java:260)
at hudson.Util.deleteContentsRecursive(Util.java:249)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:739)
at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:161)
at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:154)
at hudson.remoting.UserRequest.perform(UserRequest.java:211)
at hudson.remoting.UserRequest.perform(UserRequest.java:54)
at hudson.remoting.Request$2.run(Request.java:369)
at 
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Suppressed: java.nio.file.FileSystemException: 
/home/jenkins/workspace/Build-Deploy-Smoke-Test-Pull-Request-All-Distros/OS/centos-7-x86_64-deploy/bigtop-tests/smoke-tests/logger-test-config/build/resources/main/log4j.properties:
 Operation not permitted
{code}
Ref: 
https://ci.bigtop.apache.org/view/Test/job/Build-Deploy-Smoke-Test-Pull-Request-All-Distros/OS=centos-7-x86_64-deploy/52/consoleFull

Guessing it's because the job error out early before the cleanup command 
getting executed:
{code}
./gradlew realclean docker-provisioner-destroy
{code}

Ref: https://issues.apache.org/jira/browse/BIGTOP-3344



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (BIGTOP-3344) Check-in Smoke Test CI script to the codebase

[Evans Ye (Jira)]

Evans Ye created BIGTOP-3344:


 Summary: Check-in Smoke Test CI script to the codebase 
 Key: BIGTOP-3344
 URL: https://issues.apache.org/jira/browse/BIGTOP-3344
 Project: Bigtop
  Issue Type: Sub-task
  Components: ci
Affects Versions: 1.4.0
Reporter: Evans Ye


The current smoke test script:

{code:java}
OS_WO_ARCH=`echo ${OS} | awk -F"-" '{ print $1"-"$2}'`
# clean up
case $(arch) in
x86_64) IMAGE_SUFIX="";;
*) IMAGE_SUFIX="-$(arch)";;
esac
docker run --rm -v `pwd`:/ws bigtop/slaves:trunk-${OS_WO_ARCH}${IMAGE_SUFIX} 
bash -c "cd /ws;git clean -xdf"

if [ ${COMPONENT} != "custom" ]; then
echo "COMPONENT = ${COMPONENT}"
else
COMPONENT=${CUSTOM_COMPONENT}
fi

if [ ${COMPONENT} != "skip" ]; then
RUN_PACKAGING="allclean ${COMPONENT}-pkg-ind repo-ind"
ENABLE_LOCAL_REPO_PROPERTY="-Penable_local_repo";
fi

if [ ! -z ${STACK} ]; then 
RUN_PROVISIONER="docker-provisioner-destroy docker-provisioner"
STACK_PROPERTY="-Pstack=${STACK}"; 
NUM_INSTANCES_PROPERTY="-Pnum_instances=${NUM_INSTANCES}";
fi

if [ ! -z ${SMOKE_TESTS} ]; then 
SMOKE_TESTS_PROPERTY="-Psmoke_tests=${SMOKE_TESTS}"; 
fi

# Apply patch
curl -L ${PR}.patch | git am

./gradlew ${RUN_PACKAGING} \
-POS=${OS_WO_ARCH} \
-Pprefix=${PREFIX} \
-Pconfig=config_${OS_WO_ARCH}.yaml \
${RUN_PROVISIONER} \
${ENABLE_LOCAL_REPO_PROPERTY} \
${NUM_INSTANCES_PROPERTY} \
${STACK_PROPERTY} \
${SMOKE_TESTS_PROPERTY} \
--info -s;

./gradlew realclean docker-provisioner-destroy
{code}

Jenkins job:
https://ci.bigtop.apache.org/view/Test/job/Build-Deploy-Smoke-Test-Pull-Request-All-Distros/



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: New Committer: Masatake Iwasaki

[Evans Ye]

Congrats Masatake. Well deserved!

Youngwoo Kim (김영우)  於 2020年4月18日 週六 10:45 寫道：

> Congratulations! Masatake
>
> Youngwoo
>
> 2020년 4월 18일 (토) 오전 3:58, Olaf Flebbe 님이 작성:
>
>> The Project Management Committee (PMC) for Apache Bigtop has asked
>> Masatake Iwasaki to become a committer and we are pleased to announce
>> that he has accepted!
>>
>> Being a committer enables easier contribution to the project since there is
>> no need to go via the proxy patch submission process. This should enable 
>> better
>> productivity.
>>
>> Apache Bigtop practices CTR (commit-then-review) development process which
>> means that you can checkin the code without a +1 from a committer. This
>> doesn't mean that you're encouraged to do so. Instead, we strongly recommend 
>> you to
>> seek for feedback before you commit. Please read the document before you
>> act:
>>
>> * https://github.com/apache/bigtop#ctr-model
>>
>> If in doubt, exercise your best judgement and don't hesitate to ask
>> questions on the dev@bigtop.apache.org mailing list.
>>
>> We're happy to have you on board and looking for many more contributions to
>> come. Please join me in congratulating Masatake Iwasaki !
>>
>> Regards,
>>   Olaf  (on behalf of the ASF BigTop PMC)
>>
>>

Re: New Bigtop PMC member: Kengo Seki

[Evans Ye]

congrats, Kengo. Glad to finally have you onboarded.

Youngwoo Kim (김영우)  於 2020年4月18日 週六 10:43 寫道：

> Congratulations & welcome! Kengo
>
> Youngwoo
>
> 2020년 4월 18일 (토) 오전 4:06, Olaf Flebbe 님이 작성:
>
> > On behalf of the Apache Bigtop Project Management Committee, I am pleased
> > to announce that Kengo Seki has accepted our invitation to join the
> > Bigtop PMC.
> >
> > Please join me in congratulating Kengo!
> >
> > Regards,
> >   Olaf  (on behalf of the ASF BigTop PMC)
> >
> >
>

Re: Test your account

[Evans Ye]

It's totally fine to push with 1.5.0 snapshot. When ever needed just push
it :)

Masatake Iwasaki  於 2020年4月18日 週六 13:26 寫道：

> Hi Jun,
>
> I ran `mvn site-deploy` following the wiki and checked the diff.
>
> If I publish the contents based on master branch now,
> version number in the site will be updated from 1.4.0 to 1.5.0-SNAPSHOT.
> This affects not only page headers but contents of
> https://bigtop.apache.org/dependency-info.html
>
> It would be better to publish contents based on release-1.5.0 tag
> after waiting for 1.5.0 release.
>
> Thanks,
> Masatake Iwasaki
>
> On 2020/04/18 11:31, Jun HE wrote:
> > Welcome Masatake! It's great to have you on board.
> >
> > To update the web page, you may follow description here:
> >
> https://cwiki.apache.org/confluence/display/BIGTOP/Deploying+Bigtop%27s+Apache+Website
> >
> >
> > Masatake Iwasaki  于2020年4月18日周六 上午8:56写道：
> >
> >> Hi Olaf,
> >>
> >> Thanks for setting this up.
> >>
> >> I added myself to the member page and pushed.
> >>
> >> On Confluence, it looks like I can not edit existing pages,
> >> while I can create a new page.
> >>
> >> Regards,
> >> Masatake Iwasaki
> >>
> >> On Sat, Apr 18, 2020 at 4:20 AM Olaf Flebbe  wrote:
> >>> Hi Masatake,
> >>>
> >>> please test your account by adding yourself to the member page (Like
> >>> BIGTOP-2622).
> >>>
> >>> You should be able to update the page (see
> >>>
> >>
> https://cwiki.apache.org/confluence/display/BIGTOP/Deploying+Bigtop%27s+Apache+Website
> >> )
> >>> Best,
> >>> Olaf
>

Re: Arm4 build slave is down

[Evans Ye]

Thanks to Jun.and Linaro folks.

Jun HE  於 2020年4月9日 週四 16:14 寫道：

> Sorry for any inconveince caused by this. This node is back on line now.
> Thanks a lot for help from Linaro folks. :)
>
> Evans Ye  于2020年4月8日周三 上午3:50写道：
>
> > Fellow Arm folks,
> >
> > It seems ARM4 build slave is down. Could you help to fix it? Thanks!
> >
> > Evans
> >
>

Arm4 build slave is down

[Evans Ye]

Fellow Arm folks,

It seems ARM4 build slave is down. Could you help to fix it? Thanks!

Evans

Re: Openssl 1.1.1 and Hadoop 2.10.1

[Evans Ye]

There're two ways down the road:

1. If there's a Hadoop release (say, 2.10.1), we just need to bump it up.
2. If Hadoop doesn't release 2.10.1 with HADOOP-16647, then we'll need to
have that patch in Bigtop and do on-the-fly patch.

Either way is fine with me.


Luca Toscano  於 2020年3月23日 週一 下午10:47寫道：

> Hi everybody,
>
> just wanted to leave the pointer for
> https://issues.apache.org/jira/browse/HADOOP-16647, in which it seems
> that some folks are working on openssl 1.1.1 compatibility for Hadoop
> 2.10. From the last comments it seems that a patch should come very
> soon (and it should be easy), it would be great if this was included
> in BigTop 1.5.
> Debian Buster ships with Openssl 1.1.1 for example, and I think that
> other distros may do the same.
>
> Thanks!
>
> Luca
>

Re: Release-1.4.0

[Evans Ye]

You just need to change a bit in bigtop.bom:

-  version { base = '2.10.0'; pkg = base; release = 1 }
+  version { base = '2.10.0'; pkg = base; release = 2 }


Luca Toscano  於 2020年3月18日 週三 下午6:29寫道：

> Hi everybody,
>
> I have successfully rebuild hadoop/oozie packages via gradlew/docker
> from branch-1.4, but now I am wondering if there is a way to force a
> specific version of the packages created. For example, now in my
> internal apt repo I have the 'hadoop' package with version 2.8.5-1
> (synced from the bigtop repos) and the same version for the package
> build by me. I'd like to force the version of the new packages to
> something like 2.8.5-2 (or similar), but didn't find a quick way yet
> (usually I modify the changelog debian file but in this case it is
> autogenerated). Any suggestions?
>
> Thanks in advance,
>
> Luca
>
> Il giorno lun 9 mar 2020 alle ore 08:38 Luca Toscano
>  ha scritto:
> >
> > If possible I'd ask to consider also
> > https://github.com/apache/bigtop/pull/610 for a branch-1.4 backport
> > (if the code looks good).
> >
> > Thanks!
> >
> > Luca
> >
> > Il giorno dom 8 mar 2020 alle ore 20:07 Olaf Flebbe  ha
> scritto:
> > >
> > > Thanks Luca, Evans.
> > >
> > > Release Tags are pushed and two commits to branch-1.4 as well.
> > >
> > > Olaf
> > >
> > > Am 08.03.20 um 13:52 schrieb Luca Toscano:
> > > > Thanks a lot Olaf!
> > > >
> > > > Il giorno dom 8 mar 2020 alle ore 10:21 Evans Ye  >
> > > > ha scritto:
> > > >>
> > > >> It seems that 1.3.0 does not have the final release tag as well.
> > > >> I think make it consistent by adding 1.4.0 and 1.3.0 release tag
> would be a
> > > >> great.
> > > >> Thanks for discovering this.
> > > >>
> > > >> Evans
> > > >>
> > > >> Olaf Flebbe  於 2020年3月8日 週日 下午5:08寫道：
> > > >>
> > > >>> Hi,
> > > >>>
> > > >>> since Luca Toscano asked, I am considering to add two bugfixes to
> the
> > > >>> 1.4 branch with BIGTOP-3308.  I was checking tags first:
> > > >>>
> > > >>> We have
> > > >>> release-1.4.0-RC0
> > > >>> release-1.4.0-RC1
> > > >>> but no
> > > >>> release-1.4.0
> > > >>> tag
> > > >>>
> > > >>> Is it ok to push the "release-1.4.0" tag to latest commit on
> branch-1.4?
> > > >>>
> > > >>> Additionally I would add the needed commits to it as well ?
> > > >>>
> > > >>> I am not considering a bugfix release, though.
> > > >>>
> > > >>> WDYT ?
> > > >>>
> > > >>> Best
> > > >>>   Olaf
> > > >>>
> > > >>>
> > > >>>
>

[DISCUSS] Drop Apache Tajo, Apache Hama, and Apache Apex

[Evans Ye]

Hi folks,

I'd like to raise the discussion about the cleanup of supported component
matrix.

>From [1] the latest release of Tajo is 0.11.3 which was out on 2016-05-18.
Since then there're no further release. Similarly on Hama side the latest
release is on Jan 28, 2016 [2]. Apex is on Attic so there's no chance to
get new release further [3].

Though we'd like to support as many projects as possible, there's simply
not enough effort and resource to do it. Dropping them can also let us more
focus on what's valuable to the users. What do you think?

Kengo Seki
As 1.5 RM what do you think? I think if the discussion is positive this
will be in 1.5 release.

[1] https://tajo.apache.org/
[2] https://hama.apache.org/
[3] https://apex.apache.org/

Evans

[jira] [Created] (BIGTOP-3326) Error out when docker image build failed

[Evans Ye (Jira)]

Evans Ye created BIGTOP-3326:


 Summary: Error out when docker image build failed
 Key: BIGTOP-3326
 URL: https://issues.apache.org/jira/browse/BIGTOP-3326
 Project: Bigtop
  Issue Type: New Feature
  Components: docker
Reporter: Evans Ye


Currently our docker image build script does not detect whether there's an 
error when building. The downside is when library download failed, there's no 
visibility on it and the image is still built and shipped. See BIGTOP-3224 as 
an example(MVN missing). We can:

1. Abort the docker build on non 0 EXEC
2. Post detect the build log and make remediation afterwards. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (BIGTOP-3325) Spark LICENCE files should be exempted by RAT check

[Evans Ye (Jira)]

Evans Ye created BIGTOP-3325:


 Summary: Spark LICENCE files should be exempted by RAT check
 Key: BIGTOP-3325
 URL: https://issues.apache.org/jira/browse/BIGTOP-3325
 Project: Bigtop
  Issue Type: New Feature
Reporter: Evans Ye






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (BIGTOP-3324) Wrap CI scripts into Bigtop codebase

[Evans Ye (Jira)]

Evans Ye created BIGTOP-3324:


 Summary: Wrap CI scripts into Bigtop codebase
 Key: BIGTOP-3324
 URL: https://issues.apache.org/jira/browse/BIGTOP-3324
 Project: Bigtop
  Issue Type: New Feature
  Components: ci
Reporter: Evans Ye


Currently our CI scripts on ci.bigtop.apache.org are pretty complicated. We 
should maintain it in git and fetch them from our repo when triggering CI jobs 
instead of writing them on Jenkins.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Release-1.4.0

[Evans Ye]

It seems that 1.3.0 does not have the final release tag as well.
I think make it consistent by adding 1.4.0 and 1.3.0 release tag would be a
great.
Thanks for discovering this.

Evans

Olaf Flebbe  於 2020年3月8日 週日 下午5:08寫道：

> Hi,
>
> since Luca Toscano asked, I am considering to add two bugfixes to the
> 1.4 branch with BIGTOP-3308.  I was checking tags first:
>
> We have
> release-1.4.0-RC0
> release-1.4.0-RC1
> but no
> release-1.4.0
> tag
>
> Is it ok to push the "release-1.4.0" tag to latest commit on branch-1.4?
>
> Additionally I would add the needed commits to it as well ?
>
> I am not considering a bugfix release, though.
>
> WDYT ?
>
> Best
>   Olaf
>
>
>

Re: WDYT about adding Logstash and Kibana as Bigtop components

[Evans Ye]

+1.
Though it's a separate ecosystem, these are widely adopted tools.

Ganesh Raju  於 2020年3月2日 週一 下午10:53寫道：

> +1
>
> On Mon, Mar 2, 2020 at 4:11 AM Yuqi Gu  wrote:
>
> > Hi folks  ,
> >
> > Elasticsearch was added into Bigtop(BIGTOP-3220).
> >
> > I would like to add Logstash and Kibana into Bigtop to provide the whole
> > ELK Stack.
> >
> > WDYT and thanks for the feedback!
> >
> > BR,
> > Yuqi
> >
>
>
> --
> IRC: ganeshraju@#linaro on irc.freenode.ne t
>

[ANNOUNCE] Jun He is now officially the chair of Apache Bigtop

[Evans Ye]

Hi all,

With ASF board's approval on February's meeting, Jun He is now officially
appointed as the chair of Apache Bigtop. Please join me in congratulating
Jun He!

Meanwhile, thanks to our ex-chair Youngwoo Kim's service, We really glad to
have you together building a great community at Bigtop. What a success!

Evans

Matt Sicker  於 2020年2月20日 週四 上午4:13寫道：

> Dear new PMC chairs,
>
> Congratulations on your new role at Apache. I've changed your LDAP
> privileges to reflect your new status.
>
> Please read this and update the foundation records:
>
> https://svn.apache.org/repos/private/foundation/officers/advice-for-new-pmc-chairs.txt
>
> Warm regards,
>
> Matt Sicker
>

Re: CI has to be updated

[Evans Ye]

Sorry I'm busy but you can go ahead for sure. I'll check back with you.

Olaf Flebbe  於 2020年2月20日 週四 上午6:13寫道：

> Hi Evans,
>
> do you have time to refresh the CI docker image? Seems like jenkins
> urgently has to be updated. In case you don't have time I can try to
> find half an hour in the next days.
>
> Olaf
>