Sorry for another mail. Just want to mention how I tested the patch before creating PR:
Apex package was built using with following docker images: 1) Ubuntu trusty: docker run --rm -v `pwd`:/ws --workdir /ws bigtop/slaves:trunk-ubuntu-14.04 bash -l -c './gradlew allclean ; ./gradlew apex-pkg' 2) CentOS 7 docker run --rm -v `pwd`:/ws --workdir /ws bigtop/slaves:trunk-centos-7 bash -l -c './gradlew allclean ; ./gradlew apex-pkg' 3) CentOS 6 docker run --rm -v `pwd`:/ws --workdir /ws bigtop/slaves:trunk-centos-6 bash -l -c './gradlew allclean ; ./gradlew apex-pkg' 4) OpenSUSE 13.2 docker run --rm -v `pwd`:/ws --workdir /ws bigtop/slaves:trunk-opensuse-13.2 -l -c './gradlew allclean ; ./gradlew apex-pkg' Apex package was installed on following docker containers and verified for sanity: 1) bigtop/slaves:trunk-ubuntu-14.04 2) bigtop/slaves:trunk-centos-6 3) bigtop/slaves:trunk-centos-7 4) bigtop/slaves:trunk-opensuse-13.2 When you have some time, please provide the feedback on the pull request ( https://github.com/apache/bigtop/pull/92) Thank you, Chinmay. On Mon, Feb 22, 2016 at 9:43 PM, Chinmay Kolhatkar <[email protected]> wrote: > Hello Bigtop Community, > > I've created a pull request for packing of Apache Apex into bigtop that > I'd like to get reviewd and committed to Bigtop repository: > https://github.com/apache/bigtop/pull/92 > > The pull request contains code for packing of apex for both deb and rpm > packing. > This does not include smoke tests which will be the very next steps once > this gets accepted. > > If you have some time, could you please review this pull request and > provide feedback so that we can get these merged in? > > Also, there is a question related to what should be included in package > with respect to packing of runtime dependent jars. > Currently, I've included license files of non-ASF based licenses in > package. > Is there any specific guidelines from bigtop that integrating projects > needs to follow on that front? > > In the meantime, I would like to get some feedback on pull request for if > there are any code change required. > > Thank you in advance!! > > Chinmay. > > > > On Wed, Feb 17, 2016 at 11:10 AM, Chinmay Kolhatkar <[email protected]> > wrote: > >> Cos, >> Thank for a quick reply. We (apex community) has decided to follow your >> recommendation i.e. drop "-incubating". >> >> Roman, >> Apologies, I did not understand your last question earlier. >> Yes, I can certainly provide the prototype for rpm+deb packing of Apex >> with bigtop. >> Here is the link to GH where development is WIP: >> https://github.com/chinmaykolhatkar/bigtop/tree/BIGTOP-2314 >> >> Thanks, >> Chinmay. >> >> >> On Tue, Feb 16, 2016 at 9:15 PM, Konstantin Boudnik <[email protected]> >> wrote: >> >>> Dude to known limitations of RPMs "-" aren't allowed for sure. Deb >>> doesn't >>> have such idiosyncrasy, but there are ppl who are using RPM, which is >>> beyond >>> me. Anyway... >>> >>> This is a long standing issue for us as well, as we have a few incubating >>> projects in the stack. My recommendation would be to drop "-incubating" >>> part >>> from the package version. Check out how it is done for zeppelin. >>> >>> Cos >>> >>> On Tue, Feb 16, 2016 at 09:04PM, Chinmay Kolhatkar wrote: >>> > Hello Everyone, >>> > >>> > I'm facing an issue while setting base version for rpm package. >>> > We want to have version of Apache Apex as "3.3.0-incubating". >>> > But when I try to set version in bigtop.bom file, I get error while >>> > generating rpm package as rpm does not allow hyphen ('-') in version. >>> > >>> > I also tried to set 'incubating' as a value as release: version { base >>> = >>> > '3.3.0'; pkg = base; release = 'incubating' } >>> > But that doesn't seem to take any effect. >>> > >>> > Can you please suggest how can we achieve this? >>> > >>> > Thanks, >>> > Chinmay. >>> > >>> > >>> > On Mon, Feb 15, 2016 at 11:38 AM, Chinmay Kolhatkar < >>> [email protected]> >>> > wrote: >>> > >>> > > Hi Roman, >>> > > >>> > > Thanks for the feedback. >>> > > Here are the answers to the question in the same order: >>> > > 1. Apex is a YARN-native platform. Hence apex needs YARN to be there. >>> > > 2. conf/ directory recently got removed. This is not needed anymore >>> in the >>> > > package. >>> > > 3. Here is the link to GH repo for apex platform: >>> > > https://github.com/apache/incubator-apex-core >>> > > The source as of now does not output any package. do-component >>> build >>> > > would compile the apex platform and generate a rpm/deb package using >>> files >>> > > present in source code. Hence there is no prototype of the package. >>> > > Is that the prototype you're looking for? >>> > > >>> > > Thanks, >>> > > Chinmay. >>> > > >>> > > >>> > > On Mon, Feb 15, 2016 at 11:11 AM, Roman Shaposhnik < >>> [email protected]> >>> > > wrote: >>> > > >>> > >> Looks good to me. Three questions: >>> > >> 1. Does apex provide standalone daemon or does only run on YARN ? >>> > >> 2. What's the intended use for content under /usr/lib/apex/conf ? >>> > >> 3. Any chance you can provide a pointer to GH repo with a >>> prototype? >>> > >> >>> > >> Thanks, >>> > >> Roman. >>> > >> >>> > >> >>> > >> On Sun, Feb 14, 2016 at 8:36 PM, Chinmay Kolhatkar < >>> [email protected]> >>> > >> wrote: >>> > >> > Hi Everyone, >>> > >> > >>> > >> > Here is some details regarding packaging apex component into >>> bigtop: >>> > >> > 1. Apex component will dependent on hadoop. This will be >>> mentioned in >>> > >> > bigtop.bom file. >>> > >> > 2. The component added to bom file will look like following: >>> > >> > >>> > >> > 'apex' { >>> > >> > name = 'apex' >>> > >> > relNotes = 'Apache Apex (incubating)' >>> > >> > version { base = '3.3.0-incubating'; pkg = base; release = >>> 1 } >>> > >> > tarball { source = >>> > >> "$name-${version.base}-source-release.tar.gz" >>> > >> > destination = "$name-${version.base}.tar.gz" } >>> > >> > url { download_path = >>> "/incubator/$name/v${version.base}/" >>> > >> > site = "${apache.APACHE_MIRROR}/${download_path}" >>> > >> > archive = >>> "${apache.APACHE_ARCHIVE}/${download_path}" } >>> > >> > } >>> > >> > >>> > >> > 3. After installation of package (deb+rpm), the directory >>> structure >>> > >> related >>> > >> > to apex will look like following: >>> > >> > >>> > >> > / >>> > >> > `-- usr >>> > >> > |-- bin >>> > >> > | |-- apex -> ../lib/apex/bin/apex >>> > >> > | `-- dtcli -> ../lib/apex/bin/apex >>> > >> > |-- lib >>> > >> > | `-- apex >>> > >> > | |-- bin >>> > >> > | | |-- apex >>> > >> > | | `-- dtcli -> apex >>> > >> > | |-- conf >>> > >> > | | `-- dt-env.sh >>> > >> > | `-- lib >>> > >> > | |-- apex-api-3.3.0-incubating.jar >>> > >> > | |-- apex-bufferserver-3.3.0-incubating.jar >>> > >> > | |-- apex-common-3.3.0-incubating.jar >>> > >> > | |-- apex-engine.jar >>> > >> > | `-- <other dependency jars> >>> > >> > `-- share >>> > >> > `-- doc >>> > >> > `-- apex >>> > >> > |-- changelog.Debian.gz >>> > >> > |-- copyright >>> > >> > `-- license >>> > >> > `-- LICENSE.gz >>> > >> > >>> > >> > >>> > >> > Please share your thoughts on above approach. >>> > >> > >>> > >> > >>> > >> > Thanks, >>> > >> > Chinmay. >>> > >> > >>> > >> > >>> > >> > >>> > >> > >>> > >> > On Sat, Feb 13, 2016 at 8:38 AM, Thomas Weise <[email protected]> >>> wrote: >>> > >> > >>> > >> >> Chinmay, >>> > >> >> >>> > >> >> Can you share some details regarding the packaging? >>> > >> >> >>> > >> >> Thanks >>> > >> >> >>> > >> >> On Thu, Feb 11, 2016 at 12:35 AM, Chinmay Kolhatkar < >>> > >> >> [email protected] >>> > >> >> > wrote: >>> > >> >> >>> > >> >> > Dear Community, >>> > >> >> > >>> > >> >> > Here is the Jira that is created for adding Apex into Bigtop. >>> > >> >> > https://issues.apache.org/jira/browse/BIGTOP-2313 >>> > >> >> > >>> > >> >> > I've started working on creating packages for Apex. >>> > >> >> > >>> > >> >> > Please share your thoughts on approach as well as on questions >>> > >> related to >>> > >> >> > packaging in my previous mail. >>> > >> >> > >>> > >> >> > Thanks, >>> > >> >> > Chinmay. >>> > >> >> > >>> > >> >> > >>> > >> >> > >>> > >> >> > On Wed, Feb 10, 2016 at 9:55 PM, Chinmay Kolhatkar < >>> > >> >> > [email protected]> >>> > >> >> > wrote: >>> > >> >> > >>> > >> >> > > Hello Bigtop Community, >>> > >> >> > > >>> > >> >> > > Continuing with the packing effort, here is how we're >>> planning to >>> > >> >> package >>> > >> >> > > Apache Apex. >>> > >> >> > > >>> > >> >> > > Tentative package directory structure for apex would look >>> like >>> > >> >> following: >>> > >> >> > > >>> > >> >> > > /usr/lib/apex/bin/dtcli >>> > >> >> > > /usr/lib/apex/lib/apex-api-3.3.0-incubating.jar >>> > >> >> > > /usr/lib/apex/lib/apex-engine-3.3.0-incubating.jar >>> > >> >> > > /usr/lib/apex/lib/apex-bufferserver-3.3.0-incubating.jar >>> > >> >> > > /usr/lib/apex/lib/apex-common-3.3.0-incubating.jar >>> > >> >> > > /usr/lib/apex/lib/<other dependent jars> >>> > >> >> > > /usr/lib/apex/conf/dt-env.sh >>> > >> >> > > /usr/bin/dtcli -> /usr/lib/apex/bin/dtcli >>> > >> >> > > /usr/share/doc/man/man1/dtcli.1.gz >>> > >> >> > > /usr/share/doc/apex/license/LICENSE.txt >>> > >> >> > > /usr/share/doc/apex/license/<package>-LICENSE.txt >>> > >> >> > > /usr/share/doc/apex/CHANGELOG >>> > >> >> > > /usr/share/doc/apex/NOTICE >>> > >> >> > > >>> > >> >> > > Addition contents to bigtop.bom: >>> > >> >> > > 1. Dependency of apex will be on "hadoop" >>> > >> >> > > 2. Component addition would looks like this: >>> > >> >> > > 'apex' { >>> > >> >> > > name = 'apex' >>> > >> >> > > relNotes = 'Apache Apex (incubating)' >>> > >> >> > > version { base = '3.3.0-incubating'; pkg = base; >>> release = 1 >>> > >> } >>> > >> >> > > tarball { source = >>> > >> >> > "$name-${version.base}-source-release.tar.gz" >>> > >> >> > > destination = "$name-${version.base}.tar.gz" >>> } >>> > >> >> > > url { download_path = >>> > >> "/incubator/$name/v${version.base}/" >>> > >> >> > > site = >>> "${apache.APACHE_MIRROR}/${download_path}" >>> > >> >> > > archive = >>> > >> "${apache.APACHE_ARCHIVE}/${download_path}" } >>> > >> >> > > } >>> > >> >> > > >>> > >> >> > > >>> > >> >> > > I have couple of questions related to packaging: >>> > >> >> > > 1. What is the best possible way to test the package locally? >>> > >> >> > > 2. Do you have any specific size limitation on size of the >>> package >>> > >> (rpm >>> > >> >> > OR >>> > >> >> > > deb)? >>> > >> >> > > 3. Is above proposed packaging information inline with bigtop >>> > >> >> standards? >>> > >> >> > > 4. Am I missing anything there? >>> > >> >> > > >>> > >> >> > > Thanks, >>> > >> >> > > Chinmay. >>> > >> >> > > >>> > >> >> > > >>> > >> >> > > >>> > >> >> > > On Fri, Feb 5, 2016 at 12:01 AM, Chinmay Kolhatkar < >>> > >> >> > > [email protected]> wrote: >>> > >> >> > > >>> > >> >> > >> Hello Bigtop community! >>> > >> >> > >> >>> > >> >> > >> My name is Chinmay Kolhatkar and I am a committer in Apache >>> Apex >>> > >> >> > >> (incubating) project. >>> > >> >> > >> Our website is at: http://apex.incubator.apache.org/ >>> > >> >> > >> >>> > >> >> > >> We want to explore and integrate Apache Apex into the >>> Bigtop as a >>> > >> >> > >> component. >>> > >> >> > >> >>> > >> >> > >> Please treat this mail as a continuation of a mail thread: >>> > >> >> > >> >>> > >> >> > >> >>> > >> >> > >>> > >> >> >>> > >> >>> http://mail-archives.apache.org/mod_mbox/bigtop-dev/201601.mbox/%3CCA%2B5xAo20CAHuYtdxw3mtH2X9whREKd7fLdz4xmuTMqkaTiiskg%40mail.gmail.com%3E >>> > >> >> > >> >>> > >> >> > >> We're currently confirming with requirements for components >>> and >>> > >> making >>> > >> >> > >> sure that we meet the expectations: >>> > >> >> > >> >>> > >> >> > >> >>> > >> >> > >>> > >> >> >>> > >> >>> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution >>> > >> >> > >> >>> > >> >> > >> Once requirements are through, we're planning to add >>> component >>> > >> into >>> > >> >> > >> Apache Bigtop in following phases (as separate patches/pull >>> > >> requests): >>> > >> >> > >> 1. Packaging >>> > >> >> > >> - Creating of package i.e. rpm & deb files. >>> > >> >> > >> - documentations/READMEs, LICENSE, DISCLAMER, NOTES etc >>> if any >>> > >> >> > needed. >>> > >> >> > >> - Any documentation that need to be added to >>> distribution of >>> > >> our >>> > >> >> > >> software. >>> > >> >> > >> - Any license information of dependencies required to be >>> > >> added to >>> > >> >> > >> package >>> > >> >> > >> 2. Smoke tests >>> > >> >> > >> - Adding smoke test for packaging. >>> > >> >> > >> 3. Puppet recipes for automatic deployment and configuration >>> > >> >> > >> - Add puppet recipes for automatic deployment >>> > >> >> > >> 4. license clearance: >>> > >> >> > >> - Run 'gradle rat' to make sure all new stuff is >>> compliant >>> > >> with >>> > >> >> ASF >>> > >> >> > >> license requirements. >>> > >> >> > >> >>> > >> >> > >> Please share your thoughts on above approach. >>> > >> >> > >> >>> > >> >> > >> If the approach seems fine, we can create a Jira items and >>> > >> >> corresponding >>> > >> >> > >> subtasks to start working. >>> > >> >> > >> >>> > >> >> > >> Thanks, >>> > >> >> > >> Chinmay. >>> > >> >> > >> >>> > >> >> > >> >>> > >> >> > > >>> > >> >> > >>> > >> >> >>> > >> >>> > > >>> > > >>> >> >> >
