Hello Bigtop Community, I've created a pull request for packing of Apache Apex into bigtop that I'd like to get reviewd and committed to Bigtop repository: https://github.com/apache/bigtop/pull/92
The pull request contains code for packing of apex for both deb and rpm packing. This does not include smoke tests which will be the very next steps once this gets accepted. If you have some time, could you please review this pull request and provide feedback so that we can get these merged in? Also, there is a question related to what should be included in package with respect to packing of runtime dependent jars. Currently, I've included license files of non-ASF based licenses in package. Is there any specific guidelines from bigtop that integrating projects needs to follow on that front? In the meantime, I would like to get some feedback on pull request for if there are any code change required. Thank you in advance!! Chinmay. On Wed, Feb 17, 2016 at 11:10 AM, Chinmay Kolhatkar <[email protected]> wrote: > Cos, > Thank for a quick reply. We (apex community) has decided to follow your > recommendation i.e. drop "-incubating". > > Roman, > Apologies, I did not understand your last question earlier. > Yes, I can certainly provide the prototype for rpm+deb packing of Apex > with bigtop. > Here is the link to GH where development is WIP: > https://github.com/chinmaykolhatkar/bigtop/tree/BIGTOP-2314 > > Thanks, > Chinmay. > > > On Tue, Feb 16, 2016 at 9:15 PM, Konstantin Boudnik <[email protected]> > wrote: > >> Dude to known limitations of RPMs "-" aren't allowed for sure. Deb doesn't >> have such idiosyncrasy, but there are ppl who are using RPM, which is >> beyond >> me. Anyway... >> >> This is a long standing issue for us as well, as we have a few incubating >> projects in the stack. My recommendation would be to drop "-incubating" >> part >> from the package version. Check out how it is done for zeppelin. >> >> Cos >> >> On Tue, Feb 16, 2016 at 09:04PM, Chinmay Kolhatkar wrote: >> > Hello Everyone, >> > >> > I'm facing an issue while setting base version for rpm package. >> > We want to have version of Apache Apex as "3.3.0-incubating". >> > But when I try to set version in bigtop.bom file, I get error while >> > generating rpm package as rpm does not allow hyphen ('-') in version. >> > >> > I also tried to set 'incubating' as a value as release: version { base = >> > '3.3.0'; pkg = base; release = 'incubating' } >> > But that doesn't seem to take any effect. >> > >> > Can you please suggest how can we achieve this? >> > >> > Thanks, >> > Chinmay. >> > >> > >> > On Mon, Feb 15, 2016 at 11:38 AM, Chinmay Kolhatkar <[email protected] >> > >> > wrote: >> > >> > > Hi Roman, >> > > >> > > Thanks for the feedback. >> > > Here are the answers to the question in the same order: >> > > 1. Apex is a YARN-native platform. Hence apex needs YARN to be there. >> > > 2. conf/ directory recently got removed. This is not needed anymore >> in the >> > > package. >> > > 3. Here is the link to GH repo for apex platform: >> > > https://github.com/apache/incubator-apex-core >> > > The source as of now does not output any package. do-component >> build >> > > would compile the apex platform and generate a rpm/deb package using >> files >> > > present in source code. Hence there is no prototype of the package. >> > > Is that the prototype you're looking for? >> > > >> > > Thanks, >> > > Chinmay. >> > > >> > > >> > > On Mon, Feb 15, 2016 at 11:11 AM, Roman Shaposhnik < >> [email protected]> >> > > wrote: >> > > >> > >> Looks good to me. Three questions: >> > >> 1. Does apex provide standalone daemon or does only run on YARN ? >> > >> 2. What's the intended use for content under /usr/lib/apex/conf ? >> > >> 3. Any chance you can provide a pointer to GH repo with a >> prototype? >> > >> >> > >> Thanks, >> > >> Roman. >> > >> >> > >> >> > >> On Sun, Feb 14, 2016 at 8:36 PM, Chinmay Kolhatkar < >> [email protected]> >> > >> wrote: >> > >> > Hi Everyone, >> > >> > >> > >> > Here is some details regarding packaging apex component into >> bigtop: >> > >> > 1. Apex component will dependent on hadoop. This will be mentioned >> in >> > >> > bigtop.bom file. >> > >> > 2. The component added to bom file will look like following: >> > >> > >> > >> > 'apex' { >> > >> > name = 'apex' >> > >> > relNotes = 'Apache Apex (incubating)' >> > >> > version { base = '3.3.0-incubating'; pkg = base; release = 1 >> } >> > >> > tarball { source = >> > >> "$name-${version.base}-source-release.tar.gz" >> > >> > destination = "$name-${version.base}.tar.gz" } >> > >> > url { download_path = >> "/incubator/$name/v${version.base}/" >> > >> > site = "${apache.APACHE_MIRROR}/${download_path}" >> > >> > archive = >> "${apache.APACHE_ARCHIVE}/${download_path}" } >> > >> > } >> > >> > >> > >> > 3. After installation of package (deb+rpm), the directory structure >> > >> related >> > >> > to apex will look like following: >> > >> > >> > >> > / >> > >> > `-- usr >> > >> > |-- bin >> > >> > | |-- apex -> ../lib/apex/bin/apex >> > >> > | `-- dtcli -> ../lib/apex/bin/apex >> > >> > |-- lib >> > >> > | `-- apex >> > >> > | |-- bin >> > >> > | | |-- apex >> > >> > | | `-- dtcli -> apex >> > >> > | |-- conf >> > >> > | | `-- dt-env.sh >> > >> > | `-- lib >> > >> > | |-- apex-api-3.3.0-incubating.jar >> > >> > | |-- apex-bufferserver-3.3.0-incubating.jar >> > >> > | |-- apex-common-3.3.0-incubating.jar >> > >> > | |-- apex-engine.jar >> > >> > | `-- <other dependency jars> >> > >> > `-- share >> > >> > `-- doc >> > >> > `-- apex >> > >> > |-- changelog.Debian.gz >> > >> > |-- copyright >> > >> > `-- license >> > >> > `-- LICENSE.gz >> > >> > >> > >> > >> > >> > Please share your thoughts on above approach. >> > >> > >> > >> > >> > >> > Thanks, >> > >> > Chinmay. >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > On Sat, Feb 13, 2016 at 8:38 AM, Thomas Weise <[email protected]> >> wrote: >> > >> > >> > >> >> Chinmay, >> > >> >> >> > >> >> Can you share some details regarding the packaging? >> > >> >> >> > >> >> Thanks >> > >> >> >> > >> >> On Thu, Feb 11, 2016 at 12:35 AM, Chinmay Kolhatkar < >> > >> >> [email protected] >> > >> >> > wrote: >> > >> >> >> > >> >> > Dear Community, >> > >> >> > >> > >> >> > Here is the Jira that is created for adding Apex into Bigtop. >> > >> >> > https://issues.apache.org/jira/browse/BIGTOP-2313 >> > >> >> > >> > >> >> > I've started working on creating packages for Apex. >> > >> >> > >> > >> >> > Please share your thoughts on approach as well as on questions >> > >> related to >> > >> >> > packaging in my previous mail. >> > >> >> > >> > >> >> > Thanks, >> > >> >> > Chinmay. >> > >> >> > >> > >> >> > >> > >> >> > >> > >> >> > On Wed, Feb 10, 2016 at 9:55 PM, Chinmay Kolhatkar < >> > >> >> > [email protected]> >> > >> >> > wrote: >> > >> >> > >> > >> >> > > Hello Bigtop Community, >> > >> >> > > >> > >> >> > > Continuing with the packing effort, here is how we're >> planning to >> > >> >> package >> > >> >> > > Apache Apex. >> > >> >> > > >> > >> >> > > Tentative package directory structure for apex would look like >> > >> >> following: >> > >> >> > > >> > >> >> > > /usr/lib/apex/bin/dtcli >> > >> >> > > /usr/lib/apex/lib/apex-api-3.3.0-incubating.jar >> > >> >> > > /usr/lib/apex/lib/apex-engine-3.3.0-incubating.jar >> > >> >> > > /usr/lib/apex/lib/apex-bufferserver-3.3.0-incubating.jar >> > >> >> > > /usr/lib/apex/lib/apex-common-3.3.0-incubating.jar >> > >> >> > > /usr/lib/apex/lib/<other dependent jars> >> > >> >> > > /usr/lib/apex/conf/dt-env.sh >> > >> >> > > /usr/bin/dtcli -> /usr/lib/apex/bin/dtcli >> > >> >> > > /usr/share/doc/man/man1/dtcli.1.gz >> > >> >> > > /usr/share/doc/apex/license/LICENSE.txt >> > >> >> > > /usr/share/doc/apex/license/<package>-LICENSE.txt >> > >> >> > > /usr/share/doc/apex/CHANGELOG >> > >> >> > > /usr/share/doc/apex/NOTICE >> > >> >> > > >> > >> >> > > Addition contents to bigtop.bom: >> > >> >> > > 1. Dependency of apex will be on "hadoop" >> > >> >> > > 2. Component addition would looks like this: >> > >> >> > > 'apex' { >> > >> >> > > name = 'apex' >> > >> >> > > relNotes = 'Apache Apex (incubating)' >> > >> >> > > version { base = '3.3.0-incubating'; pkg = base; >> release = 1 >> > >> } >> > >> >> > > tarball { source = >> > >> >> > "$name-${version.base}-source-release.tar.gz" >> > >> >> > > destination = "$name-${version.base}.tar.gz" } >> > >> >> > > url { download_path = >> > >> "/incubator/$name/v${version.base}/" >> > >> >> > > site = >> "${apache.APACHE_MIRROR}/${download_path}" >> > >> >> > > archive = >> > >> "${apache.APACHE_ARCHIVE}/${download_path}" } >> > >> >> > > } >> > >> >> > > >> > >> >> > > >> > >> >> > > I have couple of questions related to packaging: >> > >> >> > > 1. What is the best possible way to test the package locally? >> > >> >> > > 2. Do you have any specific size limitation on size of the >> package >> > >> (rpm >> > >> >> > OR >> > >> >> > > deb)? >> > >> >> > > 3. Is above proposed packaging information inline with bigtop >> > >> >> standards? >> > >> >> > > 4. Am I missing anything there? >> > >> >> > > >> > >> >> > > Thanks, >> > >> >> > > Chinmay. >> > >> >> > > >> > >> >> > > >> > >> >> > > >> > >> >> > > On Fri, Feb 5, 2016 at 12:01 AM, Chinmay Kolhatkar < >> > >> >> > > [email protected]> wrote: >> > >> >> > > >> > >> >> > >> Hello Bigtop community! >> > >> >> > >> >> > >> >> > >> My name is Chinmay Kolhatkar and I am a committer in Apache >> Apex >> > >> >> > >> (incubating) project. >> > >> >> > >> Our website is at: http://apex.incubator.apache.org/ >> > >> >> > >> >> > >> >> > >> We want to explore and integrate Apache Apex into the Bigtop >> as a >> > >> >> > >> component. >> > >> >> > >> >> > >> >> > >> Please treat this mail as a continuation of a mail thread: >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> > >> >> >> > >> >> http://mail-archives.apache.org/mod_mbox/bigtop-dev/201601.mbox/%3CCA%2B5xAo20CAHuYtdxw3mtH2X9whREKd7fLdz4xmuTMqkaTiiskg%40mail.gmail.com%3E >> > >> >> > >> >> > >> >> > >> We're currently confirming with requirements for components >> and >> > >> making >> > >> >> > >> sure that we meet the expectations: >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> > >> >> >> > >> >> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution >> > >> >> > >> >> > >> >> > >> Once requirements are through, we're planning to add >> component >> > >> into >> > >> >> > >> Apache Bigtop in following phases (as separate patches/pull >> > >> requests): >> > >> >> > >> 1. Packaging >> > >> >> > >> - Creating of package i.e. rpm & deb files. >> > >> >> > >> - documentations/READMEs, LICENSE, DISCLAMER, NOTES etc >> if any >> > >> >> > needed. >> > >> >> > >> - Any documentation that need to be added to >> distribution of >> > >> our >> > >> >> > >> software. >> > >> >> > >> - Any license information of dependencies required to be >> > >> added to >> > >> >> > >> package >> > >> >> > >> 2. Smoke tests >> > >> >> > >> - Adding smoke test for packaging. >> > >> >> > >> 3. Puppet recipes for automatic deployment and configuration >> > >> >> > >> - Add puppet recipes for automatic deployment >> > >> >> > >> 4. license clearance: >> > >> >> > >> - Run 'gradle rat' to make sure all new stuff is >> compliant >> > >> with >> > >> >> ASF >> > >> >> > >> license requirements. >> > >> >> > >> >> > >> >> > >> Please share your thoughts on above approach. >> > >> >> > >> >> > >> >> > >> If the approach seems fine, we can create a Jira items and >> > >> >> corresponding >> > >> >> > >> subtasks to start working. >> > >> >> > >> >> > >> >> > >> Thanks, >> > >> >> > >> Chinmay. >> > >> >> > >> >> > >> >> > >> >> > >> >> > > >> > >> >> > >> > >> >> >> > >> >> > > >> > > >> > >
