Thank you Bigtop community for efficiently consuming the first patch (packing of apex). For the second phase, I'll soon be creating a PR for Smoke tests for apex package.
Thanks, Chinmay. On Thu, Feb 25, 2016 at 12:30 PM, Chinmay Kolhatkar <[email protected]> wrote: > Thanks Jay. > There are already some comments from YoungWoo Kim which are taken care of > in that PR. > > On Tue, Feb 23, 2016 at 6:05 PM, Jay Vyas <[email protected]> > wrote: > >> Thanks again for this , don't apologize for the email. >> >> We actually are excited about apex, thanks for this! We will get it >> reviewed within the next couple of days I'm sure (I'll look some time after >> work if nobody else gets to it first). >> >> > On Feb 23, 2016, at 6:09 AM, Chinmay Kolhatkar <[email protected]> >> wrote: >> > >> > Sorry for another mail. Just want to mention how I tested the patch >> before >> > creating PR: >> > >> > Apex package was built using with following docker images: >> > 1) Ubuntu trusty: >> > docker run --rm -v `pwd`:/ws --workdir /ws >> bigtop/slaves:trunk-ubuntu-14.04 >> > bash -l -c './gradlew allclean ; ./gradlew apex-pkg' >> > >> > 2) CentOS 7 >> > docker run --rm -v `pwd`:/ws --workdir /ws bigtop/slaves:trunk-centos-7 >> > bash -l -c './gradlew allclean ; ./gradlew apex-pkg' >> > >> > 3) CentOS 6 >> > docker run --rm -v `pwd`:/ws --workdir /ws bigtop/slaves:trunk-centos-6 >> > bash -l -c './gradlew allclean ; ./gradlew apex-pkg' >> > >> > 4) OpenSUSE 13.2 >> > docker run --rm -v `pwd`:/ws --workdir /ws >> > bigtop/slaves:trunk-opensuse-13.2 -l -c './gradlew allclean ; ./gradlew >> > apex-pkg' >> > >> > >> > Apex package was installed on following docker containers and verified >> for >> > sanity: >> > 1) bigtop/slaves:trunk-ubuntu-14.04 >> > 2) bigtop/slaves:trunk-centos-6 >> > 3) bigtop/slaves:trunk-centos-7 >> > 4) bigtop/slaves:trunk-opensuse-13.2 >> > >> > When you have some time, please provide the feedback on the pull >> request ( >> > https://github.com/apache/bigtop/pull/92) >> > >> > Thank you, >> > Chinmay. >> > >> > >> > >> > >> > On Mon, Feb 22, 2016 at 9:43 PM, Chinmay Kolhatkar <[email protected]> >> > wrote: >> > >> >> Hello Bigtop Community, >> >> >> >> I've created a pull request for packing of Apache Apex into bigtop that >> >> I'd like to get reviewd and committed to Bigtop repository: >> >> https://github.com/apache/bigtop/pull/92 >> >> >> >> The pull request contains code for packing of apex for both deb and rpm >> >> packing. >> >> This does not include smoke tests which will be the very next steps >> once >> >> this gets accepted. >> >> >> >> If you have some time, could you please review this pull request and >> >> provide feedback so that we can get these merged in? >> >> >> >> Also, there is a question related to what should be included in package >> >> with respect to packing of runtime dependent jars. >> >> Currently, I've included license files of non-ASF based licenses in >> >> package. >> >> Is there any specific guidelines from bigtop that integrating projects >> >> needs to follow on that front? >> >> >> >> In the meantime, I would like to get some feedback on pull request for >> if >> >> there are any code change required. >> >> >> >> Thank you in advance!! >> >> >> >> Chinmay. >> >> >> >> >> >> >> >> On Wed, Feb 17, 2016 at 11:10 AM, Chinmay Kolhatkar < >> [email protected]> >> >> wrote: >> >> >> >>> Cos, >> >>> Thank for a quick reply. We (apex community) has decided to follow >> your >> >>> recommendation i.e. drop "-incubating". >> >>> >> >>> Roman, >> >>> Apologies, I did not understand your last question earlier. >> >>> Yes, I can certainly provide the prototype for rpm+deb packing of Apex >> >>> with bigtop. >> >>> Here is the link to GH where development is WIP: >> >>> https://github.com/chinmaykolhatkar/bigtop/tree/BIGTOP-2314 >> >>> >> >>> Thanks, >> >>> Chinmay. >> >>> >> >>> >> >>> On Tue, Feb 16, 2016 at 9:15 PM, Konstantin Boudnik <[email protected]> >> >>> wrote: >> >>> >> >>>> Dude to known limitations of RPMs "-" aren't allowed for sure. Deb >> >>>> doesn't >> >>>> have such idiosyncrasy, but there are ppl who are using RPM, which is >> >>>> beyond >> >>>> me. Anyway... >> >>>> >> >>>> This is a long standing issue for us as well, as we have a few >> incubating >> >>>> projects in the stack. My recommendation would be to drop >> "-incubating" >> >>>> part >> >>>> from the package version. Check out how it is done for zeppelin. >> >>>> >> >>>> Cos >> >>>> >> >>>>> On Tue, Feb 16, 2016 at 09:04PM, Chinmay Kolhatkar wrote: >> >>>>> Hello Everyone, >> >>>>> >> >>>>> I'm facing an issue while setting base version for rpm package. >> >>>>> We want to have version of Apache Apex as "3.3.0-incubating". >> >>>>> But when I try to set version in bigtop.bom file, I get error while >> >>>>> generating rpm package as rpm does not allow hyphen ('-') in >> version. >> >>>>> >> >>>>> I also tried to set 'incubating' as a value as release: version { >> base >> >>>> = >> >>>>> '3.3.0'; pkg = base; release = 'incubating' } >> >>>>> But that doesn't seem to take any effect. >> >>>>> >> >>>>> Can you please suggest how can we achieve this? >> >>>>> >> >>>>> Thanks, >> >>>>> Chinmay. >> >>>>> >> >>>>> >> >>>>> On Mon, Feb 15, 2016 at 11:38 AM, Chinmay Kolhatkar < >> >>>> [email protected]> >> >>>>> wrote: >> >>>>> >> >>>>>> Hi Roman, >> >>>>>> >> >>>>>> Thanks for the feedback. >> >>>>>> Here are the answers to the question in the same order: >> >>>>>> 1. Apex is a YARN-native platform. Hence apex needs YARN to be >> there. >> >>>>>> 2. conf/ directory recently got removed. This is not needed anymore >> >>>> in the >> >>>>>> package. >> >>>>>> 3. Here is the link to GH repo for apex platform: >> >>>>>> https://github.com/apache/incubator-apex-core >> >>>>>> The source as of now does not output any package. do-component >> >>>> build >> >>>>>> would compile the apex platform and generate a rpm/deb package >> using >> >>>> files >> >>>>>> present in source code. Hence there is no prototype of the package. >> >>>>>> Is that the prototype you're looking for? >> >>>>>> >> >>>>>> Thanks, >> >>>>>> Chinmay. >> >>>>>> >> >>>>>> >> >>>>>> On Mon, Feb 15, 2016 at 11:11 AM, Roman Shaposhnik < >> >>>> [email protected]> >> >>>>>> wrote: >> >>>>>> >> >>>>>>> Looks good to me. Three questions: >> >>>>>>> 1. Does apex provide standalone daemon or does only run on YARN >> ? >> >>>>>>> 2. What's the intended use for content under /usr/lib/apex/conf >> ? >> >>>>>>> 3. Any chance you can provide a pointer to GH repo with a >> >>>> prototype? >> >>>>>>> >> >>>>>>> Thanks, >> >>>>>>> Roman. >> >>>>>>> >> >>>>>>> >> >>>>>>> On Sun, Feb 14, 2016 at 8:36 PM, Chinmay Kolhatkar < >> >>>> [email protected]> >> >>>>>>> wrote: >> >>>>>>>> Hi Everyone, >> >>>>>>>> >> >>>>>>>> Here is some details regarding packaging apex component into >> >>>> bigtop: >> >>>>>>>> 1. Apex component will dependent on hadoop. This will be >> >>>> mentioned in >> >>>>>>>> bigtop.bom file. >> >>>>>>>> 2. The component added to bom file will look like following: >> >>>>>>>> >> >>>>>>>> 'apex' { >> >>>>>>>> name = 'apex' >> >>>>>>>> relNotes = 'Apache Apex (incubating)' >> >>>>>>>> version { base = '3.3.0-incubating'; pkg = base; release = >> >>>> 1 } >> >>>>>>>> tarball { source = >> >>>>>>> "$name-${version.base}-source-release.tar.gz" >> >>>>>>>> destination = "$name-${version.base}.tar.gz" } >> >>>>>>>> url { download_path = >> >>>> "/incubator/$name/v${version.base}/" >> >>>>>>>> site = "${apache.APACHE_MIRROR}/${download_path}" >> >>>>>>>> archive = >> >>>> "${apache.APACHE_ARCHIVE}/${download_path}" } >> >>>>>>>> } >> >>>>>>>> >> >>>>>>>> 3. After installation of package (deb+rpm), the directory >> >>>> structure >> >>>>>>> related >> >>>>>>>> to apex will look like following: >> >>>>>>>> >> >>>>>>>> / >> >>>>>>>> `-- usr >> >>>>>>>> |-- bin >> >>>>>>>> | |-- apex -> ../lib/apex/bin/apex >> >>>>>>>> | `-- dtcli -> ../lib/apex/bin/apex >> >>>>>>>> |-- lib >> >>>>>>>> | `-- apex >> >>>>>>>> | |-- bin >> >>>>>>>> | | |-- apex >> >>>>>>>> | | `-- dtcli -> apex >> >>>>>>>> | |-- conf >> >>>>>>>> | | `-- dt-env.sh >> >>>>>>>> | `-- lib >> >>>>>>>> | |-- apex-api-3.3.0-incubating.jar >> >>>>>>>> | |-- apex-bufferserver-3.3.0-incubating.jar >> >>>>>>>> | |-- apex-common-3.3.0-incubating.jar >> >>>>>>>> | |-- apex-engine.jar >> >>>>>>>> | `-- <other dependency jars> >> >>>>>>>> `-- share >> >>>>>>>> `-- doc >> >>>>>>>> `-- apex >> >>>>>>>> |-- changelog.Debian.gz >> >>>>>>>> |-- copyright >> >>>>>>>> `-- license >> >>>>>>>> `-- LICENSE.gz >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> Please share your thoughts on above approach. >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> Thanks, >> >>>>>>>> Chinmay. >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> On Sat, Feb 13, 2016 at 8:38 AM, Thomas Weise <[email protected]> >> >>>> wrote: >> >>>>>>>> >> >>>>>>>>> Chinmay, >> >>>>>>>>> >> >>>>>>>>> Can you share some details regarding the packaging? >> >>>>>>>>> >> >>>>>>>>> Thanks >> >>>>>>>>> >> >>>>>>>>> On Thu, Feb 11, 2016 at 12:35 AM, Chinmay Kolhatkar < >> >>>>>>>>> [email protected] >> >>>>>>>>>> wrote: >> >>>>>>>>> >> >>>>>>>>>> Dear Community, >> >>>>>>>>>> >> >>>>>>>>>> Here is the Jira that is created for adding Apex into Bigtop. >> >>>>>>>>>> https://issues.apache.org/jira/browse/BIGTOP-2313 >> >>>>>>>>>> >> >>>>>>>>>> I've started working on creating packages for Apex. >> >>>>>>>>>> >> >>>>>>>>>> Please share your thoughts on approach as well as on questions >> >>>>>>> related to >> >>>>>>>>>> packaging in my previous mail. >> >>>>>>>>>> >> >>>>>>>>>> Thanks, >> >>>>>>>>>> Chinmay. >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> On Wed, Feb 10, 2016 at 9:55 PM, Chinmay Kolhatkar < >> >>>>>>>>>> [email protected]> >> >>>>>>>>>> wrote: >> >>>>>>>>>> >> >>>>>>>>>>> Hello Bigtop Community, >> >>>>>>>>>>> >> >>>>>>>>>>> Continuing with the packing effort, here is how we're >> >>>> planning to >> >>>>>>>>> package >> >>>>>>>>>>> Apache Apex. >> >>>>>>>>>>> >> >>>>>>>>>>> Tentative package directory structure for apex would look >> >>>> like >> >>>>>>>>> following: >> >>>>>>>>>>> >> >>>>>>>>>>> /usr/lib/apex/bin/dtcli >> >>>>>>>>>>> /usr/lib/apex/lib/apex-api-3.3.0-incubating.jar >> >>>>>>>>>>> /usr/lib/apex/lib/apex-engine-3.3.0-incubating.jar >> >>>>>>>>>>> /usr/lib/apex/lib/apex-bufferserver-3.3.0-incubating.jar >> >>>>>>>>>>> /usr/lib/apex/lib/apex-common-3.3.0-incubating.jar >> >>>>>>>>>>> /usr/lib/apex/lib/<other dependent jars> >> >>>>>>>>>>> /usr/lib/apex/conf/dt-env.sh >> >>>>>>>>>>> /usr/bin/dtcli -> /usr/lib/apex/bin/dtcli >> >>>>>>>>>>> /usr/share/doc/man/man1/dtcli.1.gz >> >>>>>>>>>>> /usr/share/doc/apex/license/LICENSE.txt >> >>>>>>>>>>> /usr/share/doc/apex/license/<package>-LICENSE.txt >> >>>>>>>>>>> /usr/share/doc/apex/CHANGELOG >> >>>>>>>>>>> /usr/share/doc/apex/NOTICE >> >>>>>>>>>>> >> >>>>>>>>>>> Addition contents to bigtop.bom: >> >>>>>>>>>>> 1. Dependency of apex will be on "hadoop" >> >>>>>>>>>>> 2. Component addition would looks like this: >> >>>>>>>>>>> 'apex' { >> >>>>>>>>>>> name = 'apex' >> >>>>>>>>>>> relNotes = 'Apache Apex (incubating)' >> >>>>>>>>>>> version { base = '3.3.0-incubating'; pkg = base; >> >>>> release = 1 >> >>>>>>> } >> >>>>>>>>>>> tarball { source = >> >>>>>>>>>> "$name-${version.base}-source-release.tar.gz" >> >>>>>>>>>>> destination = "$name-${version.base}.tar.gz" >> >>>> } >> >>>>>>>>>>> url { download_path = >> >>>>>>> "/incubator/$name/v${version.base}/" >> >>>>>>>>>>> site = >> >>>> "${apache.APACHE_MIRROR}/${download_path}" >> >>>>>>>>>>> archive = >> >>>>>>> "${apache.APACHE_ARCHIVE}/${download_path}" } >> >>>>>>>>>>> } >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> I have couple of questions related to packaging: >> >>>>>>>>>>> 1. What is the best possible way to test the package locally? >> >>>>>>>>>>> 2. Do you have any specific size limitation on size of the >> >>>> package >> >>>>>>> (rpm >> >>>>>>>>>> OR >> >>>>>>>>>>> deb)? >> >>>>>>>>>>> 3. Is above proposed packaging information inline with bigtop >> >>>>>>>>> standards? >> >>>>>>>>>>> 4. Am I missing anything there? >> >>>>>>>>>>> >> >>>>>>>>>>> Thanks, >> >>>>>>>>>>> Chinmay. >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> On Fri, Feb 5, 2016 at 12:01 AM, Chinmay Kolhatkar < >> >>>>>>>>>>> [email protected]> wrote: >> >>>>>>>>>>> >> >>>>>>>>>>>> Hello Bigtop community! >> >>>>>>>>>>>> >> >>>>>>>>>>>> My name is Chinmay Kolhatkar and I am a committer in Apache >> >>>> Apex >> >>>>>>>>>>>> (incubating) project. >> >>>>>>>>>>>> Our website is at: http://apex.incubator.apache.org/ >> >>>>>>>>>>>> >> >>>>>>>>>>>> We want to explore and integrate Apache Apex into the >> >>>> Bigtop as a >> >>>>>>>>>>>> component. >> >>>>>>>>>>>> >> >>>>>>>>>>>> Please treat this mail as a continuation of a mail thread: >> >>>> >> http://mail-archives.apache.org/mod_mbox/bigtop-dev/201601.mbox/%3CCA%2B5xAo20CAHuYtdxw3mtH2X9whREKd7fLdz4xmuTMqkaTiiskg%40mail.gmail.com%3E >> >>>>>>>>>>>> >> >>>>>>>>>>>> We're currently confirming with requirements for components >> >>>> and >> >>>>>>> making >> >>>>>>>>>>>> sure that we meet the expectations: >> >>>> >> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution >> >>>>>>>>>>>> >> >>>>>>>>>>>> Once requirements are through, we're planning to add >> >>>> component >> >>>>>>> into >> >>>>>>>>>>>> Apache Bigtop in following phases (as separate patches/pull >> >>>>>>> requests): >> >>>>>>>>>>>> 1. Packaging >> >>>>>>>>>>>> - Creating of package i.e. rpm & deb files. >> >>>>>>>>>>>> - documentations/READMEs, LICENSE, DISCLAMER, NOTES etc >> >>>> if any >> >>>>>>>>>> needed. >> >>>>>>>>>>>> - Any documentation that need to be added to >> >>>> distribution of >> >>>>>>> our >> >>>>>>>>>>>> software. >> >>>>>>>>>>>> - Any license information of dependencies required to be >> >>>>>>> added to >> >>>>>>>>>>>> package >> >>>>>>>>>>>> 2. Smoke tests >> >>>>>>>>>>>> - Adding smoke test for packaging. >> >>>>>>>>>>>> 3. Puppet recipes for automatic deployment and configuration >> >>>>>>>>>>>> - Add puppet recipes for automatic deployment >> >>>>>>>>>>>> 4. license clearance: >> >>>>>>>>>>>> - Run 'gradle rat' to make sure all new stuff is >> >>>> compliant >> >>>>>>> with >> >>>>>>>>> ASF >> >>>>>>>>>>>> license requirements. >> >>>>>>>>>>>> >> >>>>>>>>>>>> Please share your thoughts on above approach. >> >>>>>>>>>>>> >> >>>>>>>>>>>> If the approach seems fine, we can create a Jira items and >> >>>>>>>>> corresponding >> >>>>>>>>>>>> subtasks to start working. >> >>>>>>>>>>>> >> >>>>>>>>>>>> Thanks, >> >>>>>>>>>>>> Chinmay. >> >> >> > >
