What are build/ and output/ for?

2018-10-25 Thread Dagang Wei
Hi folks, I am having difficulty understanding the difference between build/ and output/. From the names, I assume build/ is used for build and output/ is used for keeping build result. But I do find package sources are copied to output/ and maven builds happen there. Dagang

Re: Installing scripts in /usr/lib/hadoop/libexec/

2018-10-25 Thread Dagang Wei
ch may take time to merge back into master. > Which has longer release cycle. > > Dagang Wei 於 2018年10月21日 週日 上午3:46寫道: > > > Sounds good. I can file a ticket for Hadoop 3 support. But for "you > should > > lay out your plan how to attack the issue", I don&#x

Re: Installing scripts in /usr/lib/hadoop/libexec/

2018-10-20 Thread Dagang Wei
the issue, in order to > make it easier to follow your work. > > I did some initial investigations (still visible in github branch > "bigtop-alpha") but I you may like to have a fresh start to handle hadoop-3. > > Olaf > > > > > > Am 19.10.2018 u

Re: Installing scripts in /usr/lib/hadoop/libexec/

2018-10-19 Thread Dagang Wei
; we have a great chance to overhaul the hadoop packaging in Bigtop. Are you > planning to add support of Hadoop 3? > > Dagang Wei 於 2018年10月11日 週四 上午6:40寫道: > > > Hi folks, > > > > I have a question regarding installing scripts under > > /usr/lib/hadoo

Re: Why the dependencies on bigtop-* packages are not reflected in bigtop.bom?

2018-10-17 Thread Dagang Wei
Flebbe wrote: > Hi > > in bigtop.bom we note build time dependencies. > > package dependencies are for install time and runtime. > > best regards, > olaf > > > > Am 16.10.2018 um 22:57 schrieb Dagang Wei : > > > > Hi folks, > > > > I notic

Why the dependencies on bigtop-* packages are not reflected in bigtop.bom?

2018-10-16 Thread Dagang Wei
Hi folks, I noticed that hadoop depends on bigtop-utils [1]: Depends: ${shlibs:Depends}, ${misc:Depends}, adduser, bigtop-utils (>= 0.7), zookeeper (>= 3.4.0), psmisc, netcat-openbsd But it is not reflected in bigtop.bom [2]: dependencies = [ zookeeper:['hadoop', 'hbase'], hadoop:['

Installing scripts in /usr/lib/hadoop/libexec/

2018-10-10 Thread Dagang Wei
Hi folks, I have a question regarding installing scripts under /usr/lib/hadoop/libexec/ in [hadoop.install]( https://github.com/apache/bigtop/blob/master/bigtop-packages/src/deb/hadoop/hadoop.install ): /usr/lib/hadoop/libexec/hadoop-config.sh /usr/lib/hadoop/libexec/hadoop-layout.sh Why

How is multiple debs/rpms for a component implemented?

2018-10-04 Thread Dagang Wei
Hi folks, Some components (e.g., Hadoop) produces multiple debs/rpms, I am having difficulty figuring out how it is implemented. Could you point me in the right direction? Thanks! Dagang

Re: Does Bigtop support local tarballs/git repos?

2018-10-04 Thread Dagang Wei
have to tweak > > compilation if you want to reuse local maven repository. > > > > Best regards, > > Olaf > > > > > > > Am 01.10.2018 um 19:35 schrieb Dagang Wei : > > > > > > Hi folks, > > > > > > In bigtop.bom, packa

Does Bigtop support local tarballs/git repos?

2018-10-01 Thread Dagang Wei
Hi folks, In bigtop.bom, packages are configured to be downloaded from either Apache site/mirror or a remote Git repo. This requires internet access and sometimes unreliable. Does it support local tarballs or git repos? So that we can download tarballs/git repos onces and reuse it for every build.

Generating APT repo from a list of debs without changes files

2018-09-28 Thread Dagang Wei
Hi folks, I noticed Bigtop generates APT repo based on changes files [1] which in turn are generated during deb build IIUC, but the changes files are not in the output APT repo. In my case, say package abc-1.2.3 was built one month ago, but now it is broken for external dependencies, I want to re

Re: Hello world example for adding a new package

2018-09-20 Thread Dagang Wei
I just created this draft based on my understanding https://docs.google.com/document/d/1kJeWXRVD0pdttm0LFSUMwZkjomBNEQru3fGUJxha81A/edit# . On Thu, Sep 20, 2018 at 7:33 PM Evans Ye wrote: > Great idea! That can also serve as a test case for packaging stuffs in > Bigtop. > > Dagang W

Re: Hello world example for adding a new package

2018-09-20 Thread Dagang Wei
template. > > -- > Regards, > Cos > > On September 20, 2018 12:07:13 PM PDT, Dagang Wei > wrote: > >Hi folks, > > > >I am wondering is there a step by step guide to create a hello-world > >package (a package with only a hello-world.sh script) in Bigtop? T

[jira] [Created] (BIGTOP-3085) Hello world example for adding a new package

2018-09-20 Thread Dagang Wei (JIRA)
Dagang Wei created BIGTOP-3085: -- Summary: Hello world example for adding a new package Key: BIGTOP-3085 URL: https://issues.apache.org/jira/browse/BIGTOP-3085 Project: Bigtop Issue Type

Hello world example for adding a new package

2018-09-20 Thread Dagang Wei
Hi folks, I am wondering is there a step by step guide to create a hello-world package (a package with only a hello-world.sh script) in Bigtop? That would help beginners to learn how the Bigtop works. If we don't have now, and that makes sense, I can file a JIRA issue and work on it. Thanks! Dag

Re: Why there is no do-component-build and install_bigtop_utils.sh for bigtop-utils?

2018-09-20 Thread Dagang Wei
ls by > >script, > >say bigtop-detect-javahome. That's why there is no need to build them > >and > >install them. > >See: > > > https://github.com/apache/bigtop/tree/master/bigtop-packages/src/common/bigtop-utils > > > >These helper scripts will be

Why there is no do-component-build and install_bigtop_utils.sh for bigtop-utils?

2018-09-19 Thread Dagang Wei
Hi folks, I am trying to understand the package bigtop-utils, one interesting thing I noticed is that unlike other packages, there is no do-component-build and install_bigtop_utils.sh under bigtop-packages/src/common/bigtop-utils. Why is that? And how that works? Thanks! Dagang

Re: Package versioning with patches

2018-09-17 Thread Dagang Wei
CAC 8312 4870 D885 8616 6115 220F 6980 1F27 E622 > > Disclaimer: Opinions expressed in this email are those of the author, > and do not necessarily represent the views of any company the author > might be affiliated with at the moment of writing. > > > On Mon, Sep 17, 2018

Package versioning with patches

2018-09-17 Thread Dagang Wei
Hi folks, Seems after applying a patch, the package versioning will not change. For example, if I add a patch to Hadoop 2.8.4 in Bigtop, the result package still gets the version 2.8.4, so it is not able to tell from the version number whether a package contains a patch or not. Is there a way I c

Doc error

2018-07-22 Thread Dagang Wei
Hi folks, I noticed an error in the Packaging Cookbook page [1]: > The bigtop.mk file defines all components in bigtop I believe "bigtop.mk" should be "bigtop.bom". I can help fix it, but I didn't find instructions on how to do it. Thanks! Dagang [1]: https://cwiki.apache.org/confluence/displa

Re: Speed up Bigtop build through parallelism?

2018-07-22 Thread Dagang Wei
Hi Cos, Is there a guide or could you point me to the source code of "splitting the build by components"? The problem I am experience now is that, sometimes component build (e.g., hbase) is flaky (due to network issue, etc), then I need to retry the whole build. I am thinking about splitting pack