Hello Bigtop Community, The last PR for adding apex as component is bigtop is merged. This completes the Jira BIGTOP-2313.
A huge thanks to everyone for accepting the patches and adding apex as a component in bigtop github repository. We're excited to share "yum/apt-get install apex" commands with users of apex. Any idea on when will be the next release of bigtop and when yum/apt-get repositories will get updated? Thanks, Chinmay. On Fri, Mar 4, 2016 at 12:24 PM, Chinmay Kolhatkar <[email protected]> wrote: > ​Hello Bigtop Community, > > As a part of second phase of integration, I've created a pull request for > adding smoke test of Apex package to Bigtop that I would like to get > reviewed and committed to Bigtop repository: > https://github.com/apache/bigtop/pull/94 > > The pull request contains 2 basic tests which validates correctness of the > installed apex package. > > If you have some time, could you please review the pull request and > provide the feedback so that we can get these tests merged in? > > Thanks a ton!! > > - Chinmay. > > > > On Fri, Feb 26, 2016 at 9:22 PM, Konstantin Boudnik <[email protected]> > wrote: > >> On Fri, Feb 26, 2016 at 02:26PM, Chinmay Kolhatkar wrote: >> > Thank you Bigtop community for efficiently consuming the first patch >> > (packing of apex). >> >> yeah, you don't want to see the regurgitation process ;) >> >> > For the second phase, I'll soon be creating a PR for Smoke tests for >> apex >> > package. >> >> Keep 'em coming and thanks for the contributions! >> Cos >> >> > On Thu, Feb 25, 2016 at 12:30 PM, Chinmay Kolhatkar <[email protected] >> > >> > wrote: >> > >> > > Thanks Jay. >> > > There are already some comments from YoungWoo Kim which are taken >> care of >> > > in that PR. >> > > >> > > On Tue, Feb 23, 2016 at 6:05 PM, Jay Vyas < >> [email protected]> >> > > wrote: >> > > >> > >> Thanks again for this , don't apologize for the email. >> > >> >> > >> We actually are excited about apex, thanks for this! We will get it >> > >> reviewed within the next couple of days I'm sure (I'll look some >> time after >> > >> work if nobody else gets to it first). >> > >> >> > >> > On Feb 23, 2016, at 6:09 AM, Chinmay Kolhatkar <[email protected] >> > >> > >> wrote: >> > >> > >> > >> > Sorry for another mail. Just want to mention how I tested the patch >> > >> before >> > >> > creating PR: >> > >> > >> > >> > Apex package was built using with following docker images: >> > >> > 1) Ubuntu trusty: >> > >> > docker run --rm -v `pwd`:/ws --workdir /ws >> > >> bigtop/slaves:trunk-ubuntu-14.04 >> > >> > bash -l -c './gradlew allclean ; ./gradlew apex-pkg' >> > >> > >> > >> > 2) CentOS 7 >> > >> > docker run --rm -v `pwd`:/ws --workdir /ws >> bigtop/slaves:trunk-centos-7 >> > >> > bash -l -c './gradlew allclean ; ./gradlew apex-pkg' >> > >> > >> > >> > 3) CentOS 6 >> > >> > docker run --rm -v `pwd`:/ws --workdir /ws >> bigtop/slaves:trunk-centos-6 >> > >> > bash -l -c './gradlew allclean ; ./gradlew apex-pkg' >> > >> > >> > >> > 4) OpenSUSE 13.2 >> > >> > docker run --rm -v `pwd`:/ws --workdir /ws >> > >> > bigtop/slaves:trunk-opensuse-13.2 -l -c './gradlew allclean ; >> ./gradlew >> > >> > apex-pkg' >> > >> > >> > >> > >> > >> > Apex package was installed on following docker containers and >> verified >> > >> for >> > >> > sanity: >> > >> > 1) bigtop/slaves:trunk-ubuntu-14.04 >> > >> > 2) bigtop/slaves:trunk-centos-6 >> > >> > 3) bigtop/slaves:trunk-centos-7 >> > >> > 4) bigtop/slaves:trunk-opensuse-13.2 >> > >> > >> > >> > When you have some time, please provide the feedback on the pull >> > >> request ( >> > >> > https://github.com/apache/bigtop/pull/92) >> > >> > >> > >> > Thank you, >> > >> > Chinmay. >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > On Mon, Feb 22, 2016 at 9:43 PM, Chinmay Kolhatkar < >> [email protected]> >> > >> > wrote: >> > >> > >> > >> >> Hello Bigtop Community, >> > >> >> >> > >> >> I've created a pull request for packing of Apache Apex into >> bigtop that >> > >> >> I'd like to get reviewd and committed to Bigtop repository: >> > >> >> https://github.com/apache/bigtop/pull/92 >> > >> >> >> > >> >> The pull request contains code for packing of apex for both deb >> and rpm >> > >> >> packing. >> > >> >> This does not include smoke tests which will be the very next >> steps >> > >> once >> > >> >> this gets accepted. >> > >> >> >> > >> >> If you have some time, could you please review this pull request >> and >> > >> >> provide feedback so that we can get these merged in? >> > >> >> >> > >> >> Also, there is a question related to what should be included in >> package >> > >> >> with respect to packing of runtime dependent jars. >> > >> >> Currently, I've included license files of non-ASF based licenses >> in >> > >> >> package. >> > >> >> Is there any specific guidelines from bigtop that integrating >> projects >> > >> >> needs to follow on that front? >> > >> >> >> > >> >> In the meantime, I would like to get some feedback on pull >> request for >> > >> if >> > >> >> there are any code change required. >> > >> >> >> > >> >> Thank you in advance!! >> > >> >> >> > >> >> Chinmay. >> > >> >> >> > >> >> >> > >> >> >> > >> >> On Wed, Feb 17, 2016 at 11:10 AM, Chinmay Kolhatkar < >> > >> [email protected]> >> > >> >> wrote: >> > >> >> >> > >> >>> Cos, >> > >> >>> Thank for a quick reply. We (apex community) has decided to >> follow >> > >> your >> > >> >>> recommendation i.e. drop "-incubating". >> > >> >>> >> > >> >>> Roman, >> > >> >>> Apologies, I did not understand your last question earlier. >> > >> >>> Yes, I can certainly provide the prototype for rpm+deb packing >> of Apex >> > >> >>> with bigtop. >> > >> >>> Here is the link to GH where development is WIP: >> > >> >>> https://github.com/chinmaykolhatkar/bigtop/tree/BIGTOP-2314 >> > >> >>> >> > >> >>> Thanks, >> > >> >>> Chinmay. >> > >> >>> >> > >> >>> >> > >> >>> On Tue, Feb 16, 2016 at 9:15 PM, Konstantin Boudnik < >> [email protected]> >> > >> >>> wrote: >> > >> >>> >> > >> >>>> Dude to known limitations of RPMs "-" aren't allowed for sure. >> Deb >> > >> >>>> doesn't >> > >> >>>> have such idiosyncrasy, but there are ppl who are using RPM, >> which is >> > >> >>>> beyond >> > >> >>>> me. Anyway... >> > >> >>>> >> > >> >>>> This is a long standing issue for us as well, as we have a few >> > >> incubating >> > >> >>>> projects in the stack. My recommendation would be to drop >> > >> "-incubating" >> > >> >>>> part >> > >> >>>> from the package version. Check out how it is done for zeppelin. >> > >> >>>> >> > >> >>>> Cos >> > >> >>>> >> > >> >>>>> On Tue, Feb 16, 2016 at 09:04PM, Chinmay Kolhatkar wrote: >> > >> >>>>> Hello Everyone, >> > >> >>>>> >> > >> >>>>> I'm facing an issue while setting base version for rpm package. >> > >> >>>>> We want to have version of Apache Apex as "3.3.0-incubating". >> > >> >>>>> But when I try to set version in bigtop.bom file, I get error >> while >> > >> >>>>> generating rpm package as rpm does not allow hyphen ('-') in >> > >> version. >> > >> >>>>> >> > >> >>>>> I also tried to set 'incubating' as a value as release: >> version { >> > >> base >> > >> >>>> = >> > >> >>>>> '3.3.0'; pkg = base; release = 'incubating' } >> > >> >>>>> But that doesn't seem to take any effect. >> > >> >>>>> >> > >> >>>>> Can you please suggest how can we achieve this? >> > >> >>>>> >> > >> >>>>> Thanks, >> > >> >>>>> Chinmay. >> > >> >>>>> >> > >> >>>>> >> > >> >>>>> On Mon, Feb 15, 2016 at 11:38 AM, Chinmay Kolhatkar < >> > >> >>>> [email protected]> >> > >> >>>>> wrote: >> > >> >>>>> >> > >> >>>>>> Hi Roman, >> > >> >>>>>> >> > >> >>>>>> Thanks for the feedback. >> > >> >>>>>> Here are the answers to the question in the same order: >> > >> >>>>>> 1. Apex is a YARN-native platform. Hence apex needs YARN to be >> > >> there. >> > >> >>>>>> 2. conf/ directory recently got removed. This is not needed >> anymore >> > >> >>>> in the >> > >> >>>>>> package. >> > >> >>>>>> 3. Here is the link to GH repo for apex platform: >> > >> >>>>>> https://github.com/apache/incubator-apex-core >> > >> >>>>>> The source as of now does not output any package. >> do-component >> > >> >>>> build >> > >> >>>>>> would compile the apex platform and generate a rpm/deb package >> > >> using >> > >> >>>> files >> > >> >>>>>> present in source code. Hence there is no prototype of the >> package. >> > >> >>>>>> Is that the prototype you're looking for? >> > >> >>>>>> >> > >> >>>>>> Thanks, >> > >> >>>>>> Chinmay. >> > >> >>>>>> >> > >> >>>>>> >> > >> >>>>>> On Mon, Feb 15, 2016 at 11:11 AM, Roman Shaposhnik < >> > >> >>>> [email protected]> >> > >> >>>>>> wrote: >> > >> >>>>>> >> > >> >>>>>>> Looks good to me. Three questions: >> > >> >>>>>>> 1. Does apex provide standalone daemon or does only run on >> YARN >> > >> ? >> > >> >>>>>>> 2. What's the intended use for content under >> /usr/lib/apex/conf >> > >> ? >> > >> >>>>>>> 3. Any chance you can provide a pointer to GH repo with a >> > >> >>>> prototype? >> > >> >>>>>>> >> > >> >>>>>>> Thanks, >> > >> >>>>>>> Roman. >> > >> >>>>>>> >> > >> >>>>>>> >> > >> >>>>>>> On Sun, Feb 14, 2016 at 8:36 PM, Chinmay Kolhatkar < >> > >> >>>> [email protected]> >> > >> >>>>>>> wrote: >> > >> >>>>>>>> Hi Everyone, >> > >> >>>>>>>> >> > >> >>>>>>>> Here is some details regarding packaging apex component into >> > >> >>>> bigtop: >> > >> >>>>>>>> 1. Apex component will dependent on hadoop. This will be >> > >> >>>> mentioned in >> > >> >>>>>>>> bigtop.bom file. >> > >> >>>>>>>> 2. The component added to bom file will look like following: >> > >> >>>>>>>> >> > >> >>>>>>>> 'apex' { >> > >> >>>>>>>> name = 'apex' >> > >> >>>>>>>> relNotes = 'Apache Apex (incubating)' >> > >> >>>>>>>> version { base = '3.3.0-incubating'; pkg = base; >> release = >> > >> >>>> 1 } >> > >> >>>>>>>> tarball { source = >> > >> >>>>>>> "$name-${version.base}-source-release.tar.gz" >> > >> >>>>>>>> destination = "$name-${version.base}.tar.gz" >> } >> > >> >>>>>>>> url { download_path = >> > >> >>>> "/incubator/$name/v${version.base}/" >> > >> >>>>>>>> site = >> "${apache.APACHE_MIRROR}/${download_path}" >> > >> >>>>>>>> archive = >> > >> >>>> "${apache.APACHE_ARCHIVE}/${download_path}" } >> > >> >>>>>>>> } >> > >> >>>>>>>> >> > >> >>>>>>>> 3. After installation of package (deb+rpm), the directory >> > >> >>>> structure >> > >> >>>>>>> related >> > >> >>>>>>>> to apex will look like following: >> > >> >>>>>>>> >> > >> >>>>>>>> / >> > >> >>>>>>>> `-- usr >> > >> >>>>>>>> |-- bin >> > >> >>>>>>>> | |-- apex -> ../lib/apex/bin/apex >> > >> >>>>>>>> | `-- dtcli -> ../lib/apex/bin/apex >> > >> >>>>>>>> |-- lib >> > >> >>>>>>>> | `-- apex >> > >> >>>>>>>> | |-- bin >> > >> >>>>>>>> | | |-- apex >> > >> >>>>>>>> | | `-- dtcli -> apex >> > >> >>>>>>>> | |-- conf >> > >> >>>>>>>> | | `-- dt-env.sh >> > >> >>>>>>>> | `-- lib >> > >> >>>>>>>> | |-- apex-api-3.3.0-incubating.jar >> > >> >>>>>>>> | |-- apex-bufferserver-3.3.0-incubating.jar >> > >> >>>>>>>> | |-- apex-common-3.3.0-incubating.jar >> > >> >>>>>>>> | |-- apex-engine.jar >> > >> >>>>>>>> | `-- <other dependency jars> >> > >> >>>>>>>> `-- share >> > >> >>>>>>>> `-- doc >> > >> >>>>>>>> `-- apex >> > >> >>>>>>>> |-- changelog.Debian.gz >> > >> >>>>>>>> |-- copyright >> > >> >>>>>>>> `-- license >> > >> >>>>>>>> `-- LICENSE.gz >> > >> >>>>>>>> >> > >> >>>>>>>> >> > >> >>>>>>>> Please share your thoughts on above approach. >> > >> >>>>>>>> >> > >> >>>>>>>> >> > >> >>>>>>>> Thanks, >> > >> >>>>>>>> Chinmay. >> > >> >>>>>>>> >> > >> >>>>>>>> >> > >> >>>>>>>> >> > >> >>>>>>>> >> > >> >>>>>>>> On Sat, Feb 13, 2016 at 8:38 AM, Thomas Weise < >> [email protected]> >> > >> >>>> wrote: >> > >> >>>>>>>> >> > >> >>>>>>>>> Chinmay, >> > >> >>>>>>>>> >> > >> >>>>>>>>> Can you share some details regarding the packaging? >> > >> >>>>>>>>> >> > >> >>>>>>>>> Thanks >> > >> >>>>>>>>> >> > >> >>>>>>>>> On Thu, Feb 11, 2016 at 12:35 AM, Chinmay Kolhatkar < >> > >> >>>>>>>>> [email protected] >> > >> >>>>>>>>>> wrote: >> > >> >>>>>>>>> >> > >> >>>>>>>>>> Dear Community, >> > >> >>>>>>>>>> >> > >> >>>>>>>>>> Here is the Jira that is created for adding Apex into >> Bigtop. >> > >> >>>>>>>>>> https://issues.apache.org/jira/browse/BIGTOP-2313 >> > >> >>>>>>>>>> >> > >> >>>>>>>>>> I've started working on creating packages for Apex. >> > >> >>>>>>>>>> >> > >> >>>>>>>>>> Please share your thoughts on approach as well as on >> questions >> > >> >>>>>>> related to >> > >> >>>>>>>>>> packaging in my previous mail. >> > >> >>>>>>>>>> >> > >> >>>>>>>>>> Thanks, >> > >> >>>>>>>>>> Chinmay. >> > >> >>>>>>>>>> >> > >> >>>>>>>>>> >> > >> >>>>>>>>>> >> > >> >>>>>>>>>> On Wed, Feb 10, 2016 at 9:55 PM, Chinmay Kolhatkar < >> > >> >>>>>>>>>> [email protected]> >> > >> >>>>>>>>>> wrote: >> > >> >>>>>>>>>> >> > >> >>>>>>>>>>> Hello Bigtop Community, >> > >> >>>>>>>>>>> >> > >> >>>>>>>>>>> Continuing with the packing effort, here is how we're >> > >> >>>> planning to >> > >> >>>>>>>>> package >> > >> >>>>>>>>>>> Apache Apex. >> > >> >>>>>>>>>>> >> > >> >>>>>>>>>>> Tentative package directory structure for apex would look >> > >> >>>> like >> > >> >>>>>>>>> following: >> > >> >>>>>>>>>>> >> > >> >>>>>>>>>>> /usr/lib/apex/bin/dtcli >> > >> >>>>>>>>>>> /usr/lib/apex/lib/apex-api-3.3.0-incubating.jar >> > >> >>>>>>>>>>> /usr/lib/apex/lib/apex-engine-3.3.0-incubating.jar >> > >> >>>>>>>>>>> /usr/lib/apex/lib/apex-bufferserver-3.3.0-incubating.jar >> > >> >>>>>>>>>>> /usr/lib/apex/lib/apex-common-3.3.0-incubating.jar >> > >> >>>>>>>>>>> /usr/lib/apex/lib/<other dependent jars> >> > >> >>>>>>>>>>> /usr/lib/apex/conf/dt-env.sh >> > >> >>>>>>>>>>> /usr/bin/dtcli -> /usr/lib/apex/bin/dtcli >> > >> >>>>>>>>>>> /usr/share/doc/man/man1/dtcli.1.gz >> > >> >>>>>>>>>>> /usr/share/doc/apex/license/LICENSE.txt >> > >> >>>>>>>>>>> /usr/share/doc/apex/license/<package>-LICENSE.txt >> > >> >>>>>>>>>>> /usr/share/doc/apex/CHANGELOG >> > >> >>>>>>>>>>> /usr/share/doc/apex/NOTICE >> > >> >>>>>>>>>>> >> > >> >>>>>>>>>>> Addition contents to bigtop.bom: >> > >> >>>>>>>>>>> 1. Dependency of apex will be on "hadoop" >> > >> >>>>>>>>>>> 2. Component addition would looks like this: >> > >> >>>>>>>>>>> 'apex' { >> > >> >>>>>>>>>>> name = 'apex' >> > >> >>>>>>>>>>> relNotes = 'Apache Apex (incubating)' >> > >> >>>>>>>>>>> version { base = '3.3.0-incubating'; pkg = base; >> > >> >>>> release = 1 >> > >> >>>>>>> } >> > >> >>>>>>>>>>> tarball { source = >> > >> >>>>>>>>>> "$name-${version.base}-source-release.tar.gz" >> > >> >>>>>>>>>>> destination = >> "$name-${version.base}.tar.gz" >> > >> >>>> } >> > >> >>>>>>>>>>> url { download_path = >> > >> >>>>>>> "/incubator/$name/v${version.base}/" >> > >> >>>>>>>>>>> site = >> > >> >>>> "${apache.APACHE_MIRROR}/${download_path}" >> > >> >>>>>>>>>>> archive = >> > >> >>>>>>> "${apache.APACHE_ARCHIVE}/${download_path}" } >> > >> >>>>>>>>>>> } >> > >> >>>>>>>>>>> >> > >> >>>>>>>>>>> >> > >> >>>>>>>>>>> I have couple of questions related to packaging: >> > >> >>>>>>>>>>> 1. What is the best possible way to test the package >> locally? >> > >> >>>>>>>>>>> 2. Do you have any specific size limitation on size of >> the >> > >> >>>> package >> > >> >>>>>>> (rpm >> > >> >>>>>>>>>> OR >> > >> >>>>>>>>>>> deb)? >> > >> >>>>>>>>>>> 3. Is above proposed packaging information inline with >> bigtop >> > >> >>>>>>>>> standards? >> > >> >>>>>>>>>>> 4. Am I missing anything there? >> > >> >>>>>>>>>>> >> > >> >>>>>>>>>>> Thanks, >> > >> >>>>>>>>>>> Chinmay. >> > >> >>>>>>>>>>> >> > >> >>>>>>>>>>> >> > >> >>>>>>>>>>> >> > >> >>>>>>>>>>> On Fri, Feb 5, 2016 at 12:01 AM, Chinmay Kolhatkar < >> > >> >>>>>>>>>>> [email protected]> wrote: >> > >> >>>>>>>>>>> >> > >> >>>>>>>>>>>> Hello Bigtop community! >> > >> >>>>>>>>>>>> >> > >> >>>>>>>>>>>> My name is Chinmay Kolhatkar and I am a committer in >> Apache >> > >> >>>> Apex >> > >> >>>>>>>>>>>> (incubating) project. >> > >> >>>>>>>>>>>> Our website is at: http://apex.incubator.apache.org/ >> > >> >>>>>>>>>>>> >> > >> >>>>>>>>>>>> We want to explore and integrate Apache Apex into the >> > >> >>>> Bigtop as a >> > >> >>>>>>>>>>>> component. >> > >> >>>>>>>>>>>> >> > >> >>>>>>>>>>>> Please treat this mail as a continuation of a mail >> thread: >> > >> >>>> >> > >> >> http://mail-archives.apache.org/mod_mbox/bigtop-dev/201601.mbox/%3CCA%2B5xAo20CAHuYtdxw3mtH2X9whREKd7fLdz4xmuTMqkaTiiskg%40mail.gmail.com%3E >> > >> >>>>>>>>>>>> >> > >> >>>>>>>>>>>> We're currently confirming with requirements for >> components >> > >> >>>> and >> > >> >>>>>>> making >> > >> >>>>>>>>>>>> sure that we meet the expectations: >> > >> >>>> >> > >> >> https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution >> > >> >>>>>>>>>>>> >> > >> >>>>>>>>>>>> Once requirements are through, we're planning to add >> > >> >>>> component >> > >> >>>>>>> into >> > >> >>>>>>>>>>>> Apache Bigtop in following phases (as separate >> patches/pull >> > >> >>>>>>> requests): >> > >> >>>>>>>>>>>> 1. Packaging >> > >> >>>>>>>>>>>> - Creating of package i.e. rpm & deb files. >> > >> >>>>>>>>>>>> - documentations/READMEs, LICENSE, DISCLAMER, NOTES >> etc >> > >> >>>> if any >> > >> >>>>>>>>>> needed. >> > >> >>>>>>>>>>>> - Any documentation that need to be added to >> > >> >>>> distribution of >> > >> >>>>>>> our >> > >> >>>>>>>>>>>> software. >> > >> >>>>>>>>>>>> - Any license information of dependencies required >> to be >> > >> >>>>>>> added to >> > >> >>>>>>>>>>>> package >> > >> >>>>>>>>>>>> 2. Smoke tests >> > >> >>>>>>>>>>>> - Adding smoke test for packaging. >> > >> >>>>>>>>>>>> 3. Puppet recipes for automatic deployment and >> configuration >> > >> >>>>>>>>>>>> - Add puppet recipes for automatic deployment >> > >> >>>>>>>>>>>> 4. license clearance: >> > >> >>>>>>>>>>>> - Run 'gradle rat' to make sure all new stuff is >> > >> >>>> compliant >> > >> >>>>>>> with >> > >> >>>>>>>>> ASF >> > >> >>>>>>>>>>>> license requirements. >> > >> >>>>>>>>>>>> >> > >> >>>>>>>>>>>> Please share your thoughts on above approach. >> > >> >>>>>>>>>>>> >> > >> >>>>>>>>>>>> If the approach seems fine, we can create a Jira items >> and >> > >> >>>>>>>>> corresponding >> > >> >>>>>>>>>>>> subtasks to start working. >> > >> >>>>>>>>>>>> >> > >> >>>>>>>>>>>> Thanks, >> > >> >>>>>>>>>>>> Chinmay. >> > >> >> >> > >> >> > > >> > > >> > >
