​Hello Bigtop Community, As a part of second phase of integration, I've created a pull request for adding smoke test of Apex package to Bigtop that I would like to get reviewed and committed to Bigtop repository: https://github.com/apache/bigtop/pull/94
The pull request contains 2 basic tests which validates correctness of the installed apex package. If you have some time, could you please review the pull request and provide the feedback so that we can get these tests merged in? Thanks a ton!! - Chinmay. On Fri, Feb 26, 2016 at 9:22 PM, Konstantin Boudnik <[email protected]> wrote: > On Fri, Feb 26, 2016 at 02:26PM, Chinmay Kolhatkar wrote: > > Thank you Bigtop community for efficiently consuming the first patch > > (packing of apex). > > yeah, you don't want to see the regurgitation process ;) > > > For the second phase, I'll soon be creating a PR for Smoke tests for apex > > package. > > Keep 'em coming and thanks for the contributions! > Cos > > > On Thu, Feb 25, 2016 at 12:30 PM, Chinmay Kolhatkar <[email protected]> > > wrote: > > > > > Thanks Jay. > > > There are already some comments from YoungWoo Kim which are taken care > of > > > in that PR. > > > > > > On Tue, Feb 23, 2016 at 6:05 PM, Jay Vyas <[email protected] > > > > > wrote: > > > > > >> Thanks again for this , don't apologize for the email. > > >> > > >> We actually are excited about apex, thanks for this! We will get it > > >> reviewed within the next couple of days I'm sure (I'll look some time > after > > >> work if nobody else gets to it first). > > >> > > >> > On Feb 23, 2016, at 6:09 AM, Chinmay Kolhatkar <[email protected]> > > >> wrote: > > >> > > > >> > Sorry for another mail. Just want to mention how I tested the patch > > >> before > > >> > creating PR: > > >> > > > >> > Apex package was built using with following docker images: > > >> > 1) Ubuntu trusty: > > >> > docker run --rm -v `pwd`:/ws --workdir /ws > > >> bigtop/slaves:trunk-ubuntu-14.04 > > >> > bash -l -c './gradlew allclean ; ./gradlew apex-pkg' > > >> > > > >> > 2) CentOS 7 > > >> > docker run --rm -v `pwd`:/ws --workdir /ws > bigtop/slaves:trunk-centos-7 > > >> > bash -l -c './gradlew allclean ; ./gradlew apex-pkg' > > >> > > > >> > 3) CentOS 6 > > >> > docker run --rm -v `pwd`:/ws --workdir /ws > bigtop/slaves:trunk-centos-6 > > >> > bash -l -c './gradlew allclean ; ./gradlew apex-pkg' > > >> > > > >> > 4) OpenSUSE 13.2 > > >> > docker run --rm -v `pwd`:/ws --workdir /ws > > >> > bigtop/slaves:trunk-opensuse-13.2 -l -c './gradlew allclean ; > ./gradlew > > >> > apex-pkg' > > >> > > > >> > > > >> > Apex package was installed on following docker containers and > verified > > >> for > > >> > sanity: > > >> > 1) bigtop/slaves:trunk-ubuntu-14.04 > > >> > 2) bigtop/slaves:trunk-centos-6 > > >> > 3) bigtop/slaves:trunk-centos-7 > > >> > 4) bigtop/slaves:trunk-opensuse-13.2 > > >> > > > >> > When you have some time, please provide the feedback on the pull > > >> request ( > > >> > https://github.com/apache/bigtop/pull/92) > > >> > > > >> > Thank you, > > >> > Chinmay. > > >> > > > >> > > > >> > > > >> > > > >> > On Mon, Feb 22, 2016 at 9:43 PM, Chinmay Kolhatkar < > [email protected]> > > >> > wrote: > > >> > > > >> >> Hello Bigtop Community, > > >> >> > > >> >> I've created a pull request for packing of Apache Apex into bigtop > that > > >> >> I'd like to get reviewd and committed to Bigtop repository: > > >> >> https://github.com/apache/bigtop/pull/92 > > >> >> > > >> >> The pull request contains code for packing of apex for both deb > and rpm > > >> >> packing. > > >> >> This does not include smoke tests which will be the very next steps > > >> once > > >> >> this gets accepted. > > >> >> > > >> >> If you have some time, could you please review this pull request > and > > >> >> provide feedback so that we can get these merged in? > > >> >> > > >> >> Also, there is a question related to what should be included in > package > > >> >> with respect to packing of runtime dependent jars. > > >> >> Currently, I've included license files of non-ASF based licenses in > > >> >> package. > > >> >> Is there any specific guidelines from bigtop that integrating > projects > > >> >> needs to follow on that front? > > >> >> > > >> >> In the meantime, I would like to get some feedback on pull request > for > > >> if > > >> >> there are any code change required. > > >> >> > > >> >> Thank you in advance!! > > >> >> > > >> >> Chinmay. > > >> >> > > >> >> > > >> >> > > >> >> On Wed, Feb 17, 2016 at 11:10 AM, Chinmay Kolhatkar < > > >> [email protected]> > > >> >> wrote: > > >> >> > > >> >>> Cos, > > >> >>> Thank for a quick reply. We (apex community) has decided to follow > > >> your > > >> >>> recommendation i.e. drop "-incubating". > > >> >>> > > >> >>> Roman, > > >> >>> Apologies, I did not understand your last question earlier. > > >> >>> Yes, I can certainly provide the prototype for rpm+deb packing of > Apex > > >> >>> with bigtop. > > >> >>> Here is the link to GH where development is WIP: > > >> >>> https://github.com/chinmaykolhatkar/bigtop/tree/BIGTOP-2314 > > >> >>> > > >> >>> Thanks, > > >> >>> Chinmay. > > >> >>> > > >> >>> > > >> >>> On Tue, Feb 16, 2016 at 9:15 PM, Konstantin Boudnik < > [email protected]> > > >> >>> wrote: > > >> >>> > > >> >>>> Dude to known limitations of RPMs "-" aren't allowed for sure. > Deb > > >> >>>> doesn't > > >> >>>> have such idiosyncrasy, but there are ppl who are using RPM, > which is > > >> >>>> beyond > > >> >>>> me. Anyway... > > >> >>>> > > >> >>>> This is a long standing issue for us as well, as we have a few > > >> incubating > > >> >>>> projects in the stack. My recommendation would be to drop > > >> "-incubating" > > >> >>>> part > > >> >>>> from the package version. Check out how it is done for zeppelin. > > >> >>>> > > >> >>>> Cos > > >> >>>> > > >> >>>>> On Tue, Feb 16, 2016 at 09:04PM, Chinmay Kolhatkar wrote: > > >> >>>>> Hello Everyone, > > >> >>>>> > > >> >>>>> I'm facing an issue while setting base version for rpm package. > > >> >>>>> We want to have version of Apache Apex as "3.3.0-incubating". > > >> >>>>> But when I try to set version in bigtop.bom file, I get error > while > > >> >>>>> generating rpm package as rpm does not allow hyphen ('-') in > > >> version. > > >> >>>>> > > >> >>>>> I also tried to set 'incubating' as a value as release: version > { > > >> base > > >> >>>> = > > >> >>>>> '3.3.0'; pkg = base; release = 'incubating' } > > >> >>>>> But that doesn't seem to take any effect. > > >> >>>>> > > >> >>>>> Can you please suggest how can we achieve this? > > >> >>>>> > > >> >>>>> Thanks, > > >> >>>>> Chinmay. > > >> >>>>> > > >> >>>>> > > >> >>>>> On Mon, Feb 15, 2016 at 11:38 AM, Chinmay Kolhatkar < > > >> >>>> [email protected]> > > >> >>>>> wrote: > > >> >>>>> > > >> >>>>>> Hi Roman, > > >> >>>>>> > > >> >>>>>> Thanks for the feedback. > > >> >>>>>> Here are the answers to the question in the same order: > > >> >>>>>> 1. Apex is a YARN-native platform. Hence apex needs YARN to be > > >> there. > > >> >>>>>> 2. conf/ directory recently got removed. This is not needed > anymore > > >> >>>> in the > > >> >>>>>> package. > > >> >>>>>> 3. Here is the link to GH repo for apex platform: > > >> >>>>>> https://github.com/apache/incubator-apex-core > > >> >>>>>> The source as of now does not output any package. > do-component > > >> >>>> build > > >> >>>>>> would compile the apex platform and generate a rpm/deb package > > >> using > > >> >>>> files > > >> >>>>>> present in source code. Hence there is no prototype of the > package. > > >> >>>>>> Is that the prototype you're looking for? > > >> >>>>>> > > >> >>>>>> Thanks, > > >> >>>>>> Chinmay. > > >> >>>>>> > > >> >>>>>> > > >> >>>>>> On Mon, Feb 15, 2016 at 11:11 AM, Roman Shaposhnik < > > >> >>>> [email protected]> > > >> >>>>>> wrote: > > >> >>>>>> > > >> >>>>>>> Looks good to me. Three questions: > > >> >>>>>>> 1. Does apex provide standalone daemon or does only run on > YARN > > >> ? > > >> >>>>>>> 2. What's the intended use for content under > /usr/lib/apex/conf > > >> ? > > >> >>>>>>> 3. Any chance you can provide a pointer to GH repo with a > > >> >>>> prototype? > > >> >>>>>>> > > >> >>>>>>> Thanks, > > >> >>>>>>> Roman. > > >> >>>>>>> > > >> >>>>>>> > > >> >>>>>>> On Sun, Feb 14, 2016 at 8:36 PM, Chinmay Kolhatkar < > > >> >>>> [email protected]> > > >> >>>>>>> wrote: > > >> >>>>>>>> Hi Everyone, > > >> >>>>>>>> > > >> >>>>>>>> Here is some details regarding packaging apex component into > > >> >>>> bigtop: > > >> >>>>>>>> 1. Apex component will dependent on hadoop. This will be > > >> >>>> mentioned in > > >> >>>>>>>> bigtop.bom file. > > >> >>>>>>>> 2. The component added to bom file will look like following: > > >> >>>>>>>> > > >> >>>>>>>> 'apex' { > > >> >>>>>>>> name = 'apex' > > >> >>>>>>>> relNotes = 'Apache Apex (incubating)' > > >> >>>>>>>> version { base = '3.3.0-incubating'; pkg = base; > release = > > >> >>>> 1 } > > >> >>>>>>>> tarball { source = > > >> >>>>>>> "$name-${version.base}-source-release.tar.gz" > > >> >>>>>>>> destination = "$name-${version.base}.tar.gz" } > > >> >>>>>>>> url { download_path = > > >> >>>> "/incubator/$name/v${version.base}/" > > >> >>>>>>>> site = > "${apache.APACHE_MIRROR}/${download_path}" > > >> >>>>>>>> archive = > > >> >>>> "${apache.APACHE_ARCHIVE}/${download_path}" } > > >> >>>>>>>> } > > >> >>>>>>>> > > >> >>>>>>>> 3. After installation of package (deb+rpm), the directory > > >> >>>> structure > > >> >>>>>>> related > > >> >>>>>>>> to apex will look like following: > > >> >>>>>>>> > > >> >>>>>>>> / > > >> >>>>>>>> `-- usr > > >> >>>>>>>> |-- bin > > >> >>>>>>>> | |-- apex -> ../lib/apex/bin/apex > > >> >>>>>>>> | `-- dtcli -> ../lib/apex/bin/apex > > >> >>>>>>>> |-- lib > > >> >>>>>>>> | `-- apex > > >> >>>>>>>> | |-- bin > > >> >>>>>>>> | | |-- apex > > >> >>>>>>>> | | `-- dtcli -> apex > > >> >>>>>>>> | |-- conf > > >> >>>>>>>> | | `-- dt-env.sh > > >> >>>>>>>> | `-- lib > > >> >>>>>>>> | |-- apex-api-3.3.0-incubating.jar > > >> >>>>>>>> | |-- apex-bufferserver-3.3.0-incubating.jar > > >> >>>>>>>> | |-- apex-common-3.3.0-incubating.jar > > >> >>>>>>>> | |-- apex-engine.jar > > >> >>>>>>>> | `-- <other dependency jars> > > >> >>>>>>>> `-- share > > >> >>>>>>>> `-- doc > > >> >>>>>>>> `-- apex > > >> >>>>>>>> |-- changelog.Debian.gz > > >> >>>>>>>> |-- copyright > > >> >>>>>>>> `-- license > > >> >>>>>>>> `-- LICENSE.gz > > >> >>>>>>>> > > >> >>>>>>>> > > >> >>>>>>>> Please share your thoughts on above approach. > > >> >>>>>>>> > > >> >>>>>>>> > > >> >>>>>>>> Thanks, > > >> >>>>>>>> Chinmay. > > >> >>>>>>>> > > >> >>>>>>>> > > >> >>>>>>>> > > >> >>>>>>>> > > >> >>>>>>>> On Sat, Feb 13, 2016 at 8:38 AM, Thomas Weise < > [email protected]> > > >> >>>> wrote: > > >> >>>>>>>> > > >> >>>>>>>>> Chinmay, > > >> >>>>>>>>> > > >> >>>>>>>>> Can you share some details regarding the packaging? > > >> >>>>>>>>> > > >> >>>>>>>>> Thanks > > >> >>>>>>>>> > > >> >>>>>>>>> On Thu, Feb 11, 2016 at 12:35 AM, Chinmay Kolhatkar < > > >> >>>>>>>>> [email protected] > > >> >>>>>>>>>> wrote: > > >> >>>>>>>>> > > >> >>>>>>>>>> Dear Community, > > >> >>>>>>>>>> > > >> >>>>>>>>>> Here is the Jira that is created for adding Apex into > Bigtop. > > >> >>>>>>>>>> https://issues.apache.org/jira/browse/BIGTOP-2313 > > >> >>>>>>>>>> > > >> >>>>>>>>>> I've started working on creating packages for Apex. > > >> >>>>>>>>>> > > >> >>>>>>>>>> Please share your thoughts on approach as well as on > questions > > >> >>>>>>> related to > > >> >>>>>>>>>> packaging in my previous mail. > > >> >>>>>>>>>> > > >> >>>>>>>>>> Thanks, > > >> >>>>>>>>>> Chinmay. > > >> >>>>>>>>>> > > >> >>>>>>>>>> > > >> >>>>>>>>>> > > >> >>>>>>>>>> On Wed, Feb 10, 2016 at 9:55 PM, Chinmay Kolhatkar < > > >> >>>>>>>>>> [email protected]> > > >> >>>>>>>>>> wrote: > > >> >>>>>>>>>> > > >> >>>>>>>>>>> Hello Bigtop Community, > > >> >>>>>>>>>>> > > >> >>>>>>>>>>> Continuing with the packing effort, here is how we're > > >> >>>> planning to > > >> >>>>>>>>> package > > >> >>>>>>>>>>> Apache Apex. > > >> >>>>>>>>>>> > > >> >>>>>>>>>>> Tentative package directory structure for apex would look > > >> >>>> like > > >> >>>>>>>>> following: > > >> >>>>>>>>>>> > > >> >>>>>>>>>>> /usr/lib/apex/bin/dtcli > > >> >>>>>>>>>>> /usr/lib/apex/lib/apex-api-3.3.0-incubating.jar > > >> >>>>>>>>>>> /usr/lib/apex/lib/apex-engine-3.3.0-incubating.jar > > >> >>>>>>>>>>> /usr/lib/apex/lib/apex-bufferserver-3.3.0-incubating.jar > > >> >>>>>>>>>>> /usr/lib/apex/lib/apex-common-3.3.0-incubating.jar > > >> >>>>>>>>>>> /usr/lib/apex/lib/<other dependent jars> > > >> >>>>>>>>>>> /usr/lib/apex/conf/dt-env.sh > > >> >>>>>>>>>>> /usr/bin/dtcli -> /usr/lib/apex/bin/dtcli > > >> >>>>>>>>>>> /usr/share/doc/man/man1/dtcli.1.gz > > >> >>>>>>>>>>> /usr/share/doc/apex/license/LICENSE.txt > > >> >>>>>>>>>>> /usr/share/doc/apex/license/<package>-LICENSE.txt > > >> >>>>>>>>>>> /usr/share/doc/apex/CHANGELOG > > >> >>>>>>>>>>> /usr/share/doc/apex/NOTICE > > >> >>>>>>>>>>> > > >> >>>>>>>>>>> Addition contents to bigtop.bom: > > >> >>>>>>>>>>> 1. Dependency of apex will be on "hadoop" > > >> >>>>>>>>>>> 2. Component addition would looks like this: > > >> >>>>>>>>>>> 'apex' { > > >> >>>>>>>>>>> name = 'apex' > > >> >>>>>>>>>>> relNotes = 'Apache Apex (incubating)' > > >> >>>>>>>>>>> version { base = '3.3.0-incubating'; pkg = base; > > >> >>>> release = 1 > > >> >>>>>>> } > > >> >>>>>>>>>>> tarball { source = > > >> >>>>>>>>>> "$name-${version.base}-source-release.tar.gz" > > >> >>>>>>>>>>> destination = > "$name-${version.base}.tar.gz" > > >> >>>> } > > >> >>>>>>>>>>> url { download_path = > > >> >>>>>>> "/incubator/$name/v${version.base}/" > > >> >>>>>>>>>>> site = > > >> >>>> "${apache.APACHE_MIRROR}/${download_path}" > > >> >>>>>>>>>>> archive = > > >> >>>>>>> "${apache.APACHE_ARCHIVE}/${download_path}" } > > >> >>>>>>>>>>> } > > >> >>>>>>>>>>> > > >> >>>>>>>>>>> > > >> >>>>>>>>>>> I have couple of questions related to packaging: > > >> >>>>>>>>>>> 1. What is the best possible way to test the package > locally? > > >> >>>>>>>>>>> 2. Do you have any specific size limitation on size of the > > >> >>>> package > > >> >>>>>>> (rpm > > >> >>>>>>>>>> OR > > >> >>>>>>>>>>> deb)? > > >> >>>>>>>>>>> 3. Is above proposed packaging information inline with > bigtop > > >> >>>>>>>>> standards? > > >> >>>>>>>>>>> 4. Am I missing anything there? > > >> >>>>>>>>>>> > > >> >>>>>>>>>>> Thanks, > > >> >>>>>>>>>>> Chinmay. > > >> >>>>>>>>>>> > > >> >>>>>>>>>>> > > >> >>>>>>>>>>> > > >> >>>>>>>>>>> On Fri, Feb 5, 2016 at 12:01 AM, Chinmay Kolhatkar < > > >> >>>>>>>>>>> [email protected]> wrote: > > >> >>>>>>>>>>> > > >> >>>>>>>>>>>> Hello Bigtop community! > > >> >>>>>>>>>>>> > > >> >>>>>>>>>>>> My name is Chinmay Kolhatkar and I am a committer in > Apache > > >> >>>> Apex > > >> >>>>>>>>>>>> (incubating) project. > > >> >>>>>>>>>>>> Our website is at: http://apex.incubator.apache.org/ > > >> >>>>>>>>>>>> > > >> >>>>>>>>>>>> We want to explore and integrate Apache Apex into the > > >> >>>> Bigtop as a > > >> >>>>>>>>>>>> component. > > >> >>>>>>>>>>>> > > >> >>>>>>>>>>>> Please treat this mail as a continuation of a mail > thread: > > >> >>>> > > >> > http://mail-archives.apache.org/mod_mbox/bigtop-dev/201601.mbox/%3CCA%2B5xAo20CAHuYtdxw3mtH2X9whREKd7fLdz4xmuTMqkaTiiskg%40mail.gmail.com%3E > > >> >>>>>>>>>>>> > > >> >>>>>>>>>>>> We're currently confirming with requirements for > components > > >> >>>> and > > >> >>>>>>> making > > >> >>>>>>>>>>>> sure that we meet the expectations: > > >> >>>> > > >> > https://cwiki.apache.org/confluence/display/BIGTOP/Requirement+for+adding+a+new+component+to+Bigtop+distribution > > >> >>>>>>>>>>>> > > >> >>>>>>>>>>>> Once requirements are through, we're planning to add > > >> >>>> component > > >> >>>>>>> into > > >> >>>>>>>>>>>> Apache Bigtop in following phases (as separate > patches/pull > > >> >>>>>>> requests): > > >> >>>>>>>>>>>> 1. Packaging > > >> >>>>>>>>>>>> - Creating of package i.e. rpm & deb files. > > >> >>>>>>>>>>>> - documentations/READMEs, LICENSE, DISCLAMER, NOTES > etc > > >> >>>> if any > > >> >>>>>>>>>> needed. > > >> >>>>>>>>>>>> - Any documentation that need to be added to > > >> >>>> distribution of > > >> >>>>>>> our > > >> >>>>>>>>>>>> software. > > >> >>>>>>>>>>>> - Any license information of dependencies required to > be > > >> >>>>>>> added to > > >> >>>>>>>>>>>> package > > >> >>>>>>>>>>>> 2. Smoke tests > > >> >>>>>>>>>>>> - Adding smoke test for packaging. > > >> >>>>>>>>>>>> 3. Puppet recipes for automatic deployment and > configuration > > >> >>>>>>>>>>>> - Add puppet recipes for automatic deployment > > >> >>>>>>>>>>>> 4. license clearance: > > >> >>>>>>>>>>>> - Run 'gradle rat' to make sure all new stuff is > > >> >>>> compliant > > >> >>>>>>> with > > >> >>>>>>>>> ASF > > >> >>>>>>>>>>>> license requirements. > > >> >>>>>>>>>>>> > > >> >>>>>>>>>>>> Please share your thoughts on above approach. > > >> >>>>>>>>>>>> > > >> >>>>>>>>>>>> If the approach seems fine, we can create a Jira items > and > > >> >>>>>>>>> corresponding > > >> >>>>>>>>>>>> subtasks to start working. > > >> >>>>>>>>>>>> > > >> >>>>>>>>>>>> Thanks, > > >> >>>>>>>>>>>> Chinmay. > > >> >> > > >> > > > > > > >
