Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3
+1 approve. Tested multiple times and after a long round of fixing and testing over and over. Cheers, Nadav On 29 May 2018 at 07:38, Yaniv Rodenski wrote: > Hi everyone, > > We have fixed the legal issues, as well as a bug found by @Nadav please > review and vote on the release candidate #3 for the version > 0.2.0-incubating, as follows > > [ ] +1, Approve the release > [ ] -1, Do not approve the release (please provide specific comments) > > The complete staging area is available for your review, which includes: > > * JIRA release notes [1], > * the official Apache source release to be deployed to dist.apache.org > [2], > which is signed with the key with fingerprint [3], > * source code tag "version-0.2.0-incubating-rc3" [4], > * Java artifacts were built with Gradle 3.1 and OpenJDK/Oracle JDK > 1.8.0_151 > > The vote will be open for at least 72 hours. It is adopted by majority > approval, with at least 3 PMC affirmative votes. > > Thanks, > Yaniv > > [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa? > projectId=12321521&version=12342793 > [2] https://dist.apache.org/repos/dist/dev/incubator/amaterasu/0.2.0rc3/ > [3] https://dist.apache.org/repos/dist/dev/incubator/amaterasu/KEYS > [4] https://github.com/apache/incubator-amaterasu/tags >
Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3
+1 looks good to me On Tue, May 29, 2018, 14:39 Nadav Har Tzvi wrote: > +1 approve. Tested multiple times and after a long round of fixing and > testing over and over. > > Cheers, > Nadav > > > On 29 May 2018 at 07:38, Yaniv Rodenski wrote: > > > Hi everyone, > > > > We have fixed the legal issues, as well as a bug found by @Nadav please > > review and vote on the release candidate #3 for the version > > 0.2.0-incubating, as follows > > > > [ ] +1, Approve the release > > [ ] -1, Do not approve the release (please provide specific comments) > > > > The complete staging area is available for your review, which includes: > > > > * JIRA release notes [1], > > * the official Apache source release to be deployed to dist.apache.org > > [2], > > which is signed with the key with fingerprint [3], > > * source code tag "version-0.2.0-incubating-rc3" [4], > > * Java artifacts were built with Gradle 3.1 and OpenJDK/Oracle JDK > > 1.8.0_151 > > > > The vote will be open for at least 72 hours. It is adopted by majority > > approval, with at least 3 PMC affirmative votes. > > > > Thanks, > > Yaniv > > > > [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa? > > projectId=12321521&version=12342793 > > [2] https://dist.apache.org/repos/dist/dev/incubator/amaterasu/0.2.0rc3/ > > [3] https://dist.apache.org/repos/dist/dev/incubator/amaterasu/KEYS > > [4] https://github.com/apache/incubator-amaterasu/tags > > >
Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3
Unit tests passing and build was successful on the branch "version-0.2.0-incubating-rc3" +1 from me Cheers, Kirupa On Tue, May 29, 2018 at 3:06 PM, guy peleg wrote: > +1 looks good to me > > On Tue, May 29, 2018, 14:39 Nadav Har Tzvi wrote: > > > +1 approve. Tested multiple times and after a long round of fixing and > > testing over and over. > > > > Cheers, > > Nadav > > > > > > On 29 May 2018 at 07:38, Yaniv Rodenski wrote: > > > > > Hi everyone, > > > > > > We have fixed the legal issues, as well as a bug found by @Nadav please > > > review and vote on the release candidate #3 for the version > > > 0.2.0-incubating, as follows > > > > > > [ ] +1, Approve the release > > > [ ] -1, Do not approve the release (please provide specific comments) > > > > > > The complete staging area is available for your review, which includes: > > > > > > * JIRA release notes [1], > > > * the official Apache source release to be deployed to dist.apache.org > > > [2], > > > which is signed with the key with fingerprint [3], > > > * source code tag "version-0.2.0-incubating-rc3" [4], > > > * Java artifacts were built with Gradle 3.1 and OpenJDK/Oracle JDK > > > 1.8.0_151 > > > > > > The vote will be open for at least 72 hours. It is adopted by majority > > > approval, with at least 3 PMC affirmative votes. > > > > > > Thanks, > > > Yaniv > > > > > > [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa? > > > projectId=12321521&version=12342793 > > > [2] https://dist.apache.org/repos/dist/dev/incubator/amaterasu/ > 0.2.0rc3/ > > > [3] https://dist.apache.org/repos/dist/dev/incubator/amaterasu/KEYS > > > [4] https://github.com/apache/incubator-amaterasu/tags > > > > > >
Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3
+1 from me Unit Tests and Build ran fine. Tested on HDP (VM) but had trouble allocating containers (didn't have that before). Apparently Centos VMs are known to have this problem. Disabled physical memory check (yarn.nodemanager.pmem-check-enabled) and ran jobs successfully. On Tue, May 29, 2018 at 10:42 PM Kirupa Devarajan wrote: > Unit tests passing and build was successful on the branch > "version-0.2.0-incubating-rc3" > > +1 from me > > Cheers, > Kirupa > > > On Tue, May 29, 2018 at 3:06 PM, guy peleg wrote: > > > +1 looks good to me > > > > On Tue, May 29, 2018, 14:39 Nadav Har Tzvi > wrote: > > > > > +1 approve. Tested multiple times and after a long round of fixing and > > > testing over and over. > > > > > > Cheers, > > > Nadav > > > > > > > > > On 29 May 2018 at 07:38, Yaniv Rodenski wrote: > > > > > > > Hi everyone, > > > > > > > > We have fixed the legal issues, as well as a bug found by @Nadav > please > > > > review and vote on the release candidate #3 for the version > > > > 0.2.0-incubating, as follows > > > > > > > > [ ] +1, Approve the release > > > > [ ] -1, Do not approve the release (please provide specific comments) > > > > > > > > The complete staging area is available for your review, which > includes: > > > > > > > > * JIRA release notes [1], > > > > * the official Apache source release to be deployed to > dist.apache.org > > > > [2], > > > > which is signed with the key with fingerprint [3], > > > > * source code tag "version-0.2.0-incubating-rc3" [4], > > > > * Java artifacts were built with Gradle 3.1 and OpenJDK/Oracle JDK > > > > 1.8.0_151 > > > > > > > > The vote will be open for at least 72 hours. It is adopted by > majority > > > > approval, with at least 3 PMC affirmative votes. > > > > > > > > Thanks, > > > > Yaniv > > > > > > > > [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa? > > > > projectId=12321521&version=12342793 > > > > [2] https://dist.apache.org/repos/dist/dev/incubator/amaterasu/ > > 0.2.0rc3/ > > > > [3] https://dist.apache.org/repos/dist/dev/incubator/amaterasu/KEYS > > > > [4] https://github.com/apache/incubator-amaterasu/tags > > > > > > > > > >
Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3
Yaniv, Eyal, this might be related to the same issue you faced with HDP. Can you confirm? On Tue, May 29, 2018, 17:58 Arun Manivannan wrote: > +1 from me > > Unit Tests and Build ran fine. > > Tested on HDP (VM) but had trouble allocating containers (didn't have that > before). Apparently Centos VMs are known to have this problem. Disabled > physical memory check (yarn.nodemanager.pmem-check-enabled) and ran jobs > successfully. > > > > > > On Tue, May 29, 2018 at 10:42 PM Kirupa Devarajan > > wrote: > > > Unit tests passing and build was successful on the branch > > "version-0.2.0-incubating-rc3" > > > > +1 from me > > > > Cheers, > > Kirupa > > > > > > On Tue, May 29, 2018 at 3:06 PM, guy peleg wrote: > > > > > +1 looks good to me > > > > > > On Tue, May 29, 2018, 14:39 Nadav Har Tzvi > > wrote: > > > > > > > +1 approve. Tested multiple times and after a long round of fixing > and > > > > testing over and over. > > > > > > > > Cheers, > > > > Nadav > > > > > > > > > > > > On 29 May 2018 at 07:38, Yaniv Rodenski wrote: > > > > > > > > > Hi everyone, > > > > > > > > > > We have fixed the legal issues, as well as a bug found by @Nadav > > please > > > > > review and vote on the release candidate #3 for the version > > > > > 0.2.0-incubating, as follows > > > > > > > > > > [ ] +1, Approve the release > > > > > [ ] -1, Do not approve the release (please provide specific > comments) > > > > > > > > > > The complete staging area is available for your review, which > > includes: > > > > > > > > > > * JIRA release notes [1], > > > > > * the official Apache source release to be deployed to > > dist.apache.org > > > > > [2], > > > > > which is signed with the key with fingerprint [3], > > > > > * source code tag "version-0.2.0-incubating-rc3" [4], > > > > > * Java artifacts were built with Gradle 3.1 and OpenJDK/Oracle JDK > > > > > 1.8.0_151 > > > > > > > > > > The vote will be open for at least 72 hours. It is adopted by > > majority > > > > > approval, with at least 3 PMC affirmative votes. > > > > > > > > > > Thanks, > > > > > Yaniv > > > > > > > > > > [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa? > > > > > projectId=12321521&version=12342793 > > > > > [2] https://dist.apache.org/repos/dist/dev/incubator/amaterasu/ > > > 0.2.0rc3/ > > > > > [3] > https://dist.apache.org/repos/dist/dev/incubator/amaterasu/KEYS > > > > > [4] https://github.com/apache/incubator-amaterasu/tags > > > > > > > > > > > > > > >
Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3
The pmem disabling is just temporary. I'll do a detailed analysis and get back with a proper solution. Any hints on this front is highly appreciated. Cheers Arun On Wed, May 30, 2018, 01:10 Nadav Har Tzvi wrote: > Yaniv, Eyal, this might be related to the same issue you faced with HDP. > Can you confirm? > > On Tue, May 29, 2018, 17:58 Arun Manivannan wrote: > > > +1 from me > > > > Unit Tests and Build ran fine. > > > > Tested on HDP (VM) but had trouble allocating containers (didn't have > that > > before). Apparently Centos VMs are known to have this problem. Disabled > > physical memory check (yarn.nodemanager.pmem-check-enabled) and ran jobs > > successfully. > > > > > > > > > > > > On Tue, May 29, 2018 at 10:42 PM Kirupa Devarajan < > kirupagara...@gmail.com > > > > > wrote: > > > > > Unit tests passing and build was successful on the branch > > > "version-0.2.0-incubating-rc3" > > > > > > +1 from me > > > > > > Cheers, > > > Kirupa > > > > > > > > > On Tue, May 29, 2018 at 3:06 PM, guy peleg wrote: > > > > > > > +1 looks good to me > > > > > > > > On Tue, May 29, 2018, 14:39 Nadav Har Tzvi > > > wrote: > > > > > > > > > +1 approve. Tested multiple times and after a long round of fixing > > and > > > > > testing over and over. > > > > > > > > > > Cheers, > > > > > Nadav > > > > > > > > > > > > > > > On 29 May 2018 at 07:38, Yaniv Rodenski wrote: > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > We have fixed the legal issues, as well as a bug found by @Nadav > > > please > > > > > > review and vote on the release candidate #3 for the version > > > > > > 0.2.0-incubating, as follows > > > > > > > > > > > > [ ] +1, Approve the release > > > > > > [ ] -1, Do not approve the release (please provide specific > > comments) > > > > > > > > > > > > The complete staging area is available for your review, which > > > includes: > > > > > > > > > > > > * JIRA release notes [1], > > > > > > * the official Apache source release to be deployed to > > > dist.apache.org > > > > > > [2], > > > > > > which is signed with the key with fingerprint [3], > > > > > > * source code tag "version-0.2.0-incubating-rc3" [4], > > > > > > * Java artifacts were built with Gradle 3.1 and OpenJDK/Oracle > JDK > > > > > > 1.8.0_151 > > > > > > > > > > > > The vote will be open for at least 72 hours. It is adopted by > > > majority > > > > > > approval, with at least 3 PMC affirmative votes. > > > > > > > > > > > > Thanks, > > > > > > Yaniv > > > > > > > > > > > > [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa? > > > > > > projectId=12321521&version=12342793 > > > > > > [2] https://dist.apache.org/repos/dist/dev/incubator/amaterasu/ > > > > 0.2.0rc3/ > > > > > > [3] > > https://dist.apache.org/repos/dist/dev/incubator/amaterasu/KEYS > > > > > > [4] https://github.com/apache/incubator-amaterasu/tags > > > > > > > > > > > > > > > > > > > > >
Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3
Yaniv and I just tested it. It worked flawlessly on my end (HDP docker on AWS). Both Spark-Scala and PySpark. It worked on Yaniv's HDP cluster as well. Worth noting: 1. HDP 2.6.4 2. Cluster has total of 32GB memory available 3. Each container is allocated 1G memory. 4. Amaterasu.properties: zk=sandbox-hdp.hortonworks.com version=0.2.0-incubating-rc3 master=192.168.33.11 user=root mode=yarn webserver.port=8000 webserver.root=dist spark.version=2.6.4.0-91 yarn.queue=default yarn.jarspath=hdfs:///apps/amaterasu spark.home=/usr/hdp/current/spark2-client #spark.home=/opt/cloudera/parcels/SPARK2-2.1.0.cloudera2-1.cdh5.7.0.p0.171658/lib/spark2 yarn.hadoop.home.dir=/etc/hadoop spark.opts.spark.yarn.am.extraJavaOptions="-Dhdp.version=2.6.4.0-91" spark.opts.spark.driver.extraJavaOptions="-Dhdp.version=2.6.4.0-91" Arun, please share: 1. YARN memory configurations 2. amaterasu.properties content 3. HDP version. Cheers, Nadav On 30 May 2018 at 07:11, Arun Manivannan wrote: > The pmem disabling is just temporary. I'll do a detailed analysis and get > back with a proper solution. > > Any hints on this front is highly appreciated. > > Cheers > Arun > > On Wed, May 30, 2018, 01:10 Nadav Har Tzvi wrote: > > > Yaniv, Eyal, this might be related to the same issue you faced with HDP. > > Can you confirm? > > > > On Tue, May 29, 2018, 17:58 Arun Manivannan wrote: > > > > > +1 from me > > > > > > Unit Tests and Build ran fine. > > > > > > Tested on HDP (VM) but had trouble allocating containers (didn't have > > that > > > before). Apparently Centos VMs are known to have this problem. > Disabled > > > physical memory check (yarn.nodemanager.pmem-check-enabled) and ran > jobs > > > successfully. > > > > > > > > > > > > > > > > > > On Tue, May 29, 2018 at 10:42 PM Kirupa Devarajan < > > kirupagara...@gmail.com > > > > > > > wrote: > > > > > > > Unit tests passing and build was successful on the branch > > > > "version-0.2.0-incubating-rc3" > > > > > > > > +1 from me > > > > > > > > Cheers, > > > > Kirupa > > > > > > > > > > > > On Tue, May 29, 2018 at 3:06 PM, guy peleg > wrote: > > > > > > > > > +1 looks good to me > > > > > > > > > > On Tue, May 29, 2018, 14:39 Nadav Har Tzvi > > > > > wrote: > > > > > > > > > > > +1 approve. Tested multiple times and after a long round of > fixing > > > and > > > > > > testing over and over. > > > > > > > > > > > > Cheers, > > > > > > Nadav > > > > > > > > > > > > > > > > > > On 29 May 2018 at 07:38, Yaniv Rodenski wrote: > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > We have fixed the legal issues, as well as a bug found by > @Nadav > > > > please > > > > > > > review and vote on the release candidate #3 for the version > > > > > > > 0.2.0-incubating, as follows > > > > > > > > > > > > > > [ ] +1, Approve the release > > > > > > > [ ] -1, Do not approve the release (please provide specific > > > comments) > > > > > > > > > > > > > > The complete staging area is available for your review, which > > > > includes: > > > > > > > > > > > > > > * JIRA release notes [1], > > > > > > > * the official Apache source release to be deployed to > > > > dist.apache.org > > > > > > > [2], > > > > > > > which is signed with the key with fingerprint [3], > > > > > > > * source code tag "version-0.2.0-incubating-rc3" [4], > > > > > > > * Java artifacts were built with Gradle 3.1 and OpenJDK/Oracle > > JDK > > > > > > > 1.8.0_151 > > > > > > > > > > > > > > The vote will be open for at least 72 hours. It is adopted by > > > > majority > > > > > > > approval, with at least 3 PMC affirmative votes. > > > > > > > > > > > > > > Thanks, > > > > > > > Yaniv > > > > > > > > > > > > > > [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa? > > > > > > > projectId=12321521&version=12342793 > > > > > > > [2] https://dist.apache.org/repos/ > dist/dev/incubator/amaterasu/ > > > > > 0.2.0rc3/ > > > > > > > [3] > > > https://dist.apache.org/repos/dist/dev/incubator/amaterasu/KEYS > > > > > > > [4] https://github.com/apache/incubator-amaterasu/tags > > > > > > > > > > > > > > > > > > > > > > > > > > > >
Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3
Thanks a lot, Nadav. Will get home and spend some more time on this. I was in a rush and did this poor workaround. My VM is just 8 GB. Cheers Arun On Wed, May 30, 2018, 12:27 Nadav Har Tzvi wrote: > Yaniv and I just tested it. It worked flawlessly on my end (HDP docker on > AWS). Both Spark-Scala and PySpark. > It worked on Yaniv's HDP cluster as well. > Worth noting: > 1. HDP 2.6.4 > 2. Cluster has total of 32GB memory available > 3. Each container is allocated 1G memory. > 4. Amaterasu.properties: > > zk=sandbox-hdp.hortonworks.com > version=0.2.0-incubating-rc3 > master=192.168.33.11 > user=root > mode=yarn > webserver.port=8000 > webserver.root=dist > spark.version=2.6.4.0-91 > yarn.queue=default > yarn.jarspath=hdfs:///apps/amaterasu > spark.home=/usr/hdp/current/spark2-client > > #spark.home=/opt/cloudera/parcels/SPARK2-2.1.0.cloudera2-1.cdh5.7.0.p0.171658/lib/spark2 > yarn.hadoop.home.dir=/etc/hadoop > spark.opts.spark.yarn.am.extraJavaOptions="-Dhdp.version=2.6.4.0-91" > spark.opts.spark.driver.extraJavaOptions="-Dhdp.version=2.6.4.0-91" > > > Arun, please share: > 1. YARN memory configurations > 2. amaterasu.properties content > 3. HDP version. > > Cheers, > Nadav > > > On 30 May 2018 at 07:11, Arun Manivannan wrote: > > > The pmem disabling is just temporary. I'll do a detailed analysis and get > > back with a proper solution. > > > > Any hints on this front is highly appreciated. > > > > Cheers > > Arun > > > > On Wed, May 30, 2018, 01:10 Nadav Har Tzvi > wrote: > > > > > Yaniv, Eyal, this might be related to the same issue you faced with > HDP. > > > Can you confirm? > > > > > > On Tue, May 29, 2018, 17:58 Arun Manivannan wrote: > > > > > > > +1 from me > > > > > > > > Unit Tests and Build ran fine. > > > > > > > > Tested on HDP (VM) but had trouble allocating containers (didn't have > > > that > > > > before). Apparently Centos VMs are known to have this problem. > > Disabled > > > > physical memory check (yarn.nodemanager.pmem-check-enabled) and ran > > jobs > > > > successfully. > > > > > > > > > > > > > > > > > > > > > > > > On Tue, May 29, 2018 at 10:42 PM Kirupa Devarajan < > > > kirupagara...@gmail.com > > > > > > > > > wrote: > > > > > > > > > Unit tests passing and build was successful on the branch > > > > > "version-0.2.0-incubating-rc3" > > > > > > > > > > +1 from me > > > > > > > > > > Cheers, > > > > > Kirupa > > > > > > > > > > > > > > > On Tue, May 29, 2018 at 3:06 PM, guy peleg > > wrote: > > > > > > > > > > > +1 looks good to me > > > > > > > > > > > > On Tue, May 29, 2018, 14:39 Nadav Har Tzvi < > nadavhart...@gmail.com > > > > > > > > wrote: > > > > > > > > > > > > > +1 approve. Tested multiple times and after a long round of > > fixing > > > > and > > > > > > > testing over and over. > > > > > > > > > > > > > > Cheers, > > > > > > > Nadav > > > > > > > > > > > > > > > > > > > > > On 29 May 2018 at 07:38, Yaniv Rodenski > wrote: > > > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > > > We have fixed the legal issues, as well as a bug found by > > @Nadav > > > > > please > > > > > > > > review and vote on the release candidate #3 for the version > > > > > > > > 0.2.0-incubating, as follows > > > > > > > > > > > > > > > > [ ] +1, Approve the release > > > > > > > > [ ] -1, Do not approve the release (please provide specific > > > > comments) > > > > > > > > > > > > > > > > The complete staging area is available for your review, which > > > > > includes: > > > > > > > > > > > > > > > > * JIRA release notes [1], > > > > > > > > * the official Apache source release to be deployed to > > > > > dist.apache.org > > > > > > > > [2], > > > > > > > > which is signed with the key with fingerprint [3], > > > > > > > > * source code tag "version-0.2.0-incubating-rc3" [4], > > > > > > > > * Java artifacts were built with Gradle 3.1 and > OpenJDK/Oracle > > > JDK > > > > > > > > 1.8.0_151 > > > > > > > > > > > > > > > > The vote will be open for at least 72 hours. It is adopted by > > > > > majority > > > > > > > > approval, with at least 3 PMC affirmative votes. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Yaniv > > > > > > > > > > > > > > > > [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa? > > > > > > > > projectId=12321521&version=12342793 > > > > > > > > [2] https://dist.apache.org/repos/ > > dist/dev/incubator/amaterasu/ > > > > > > 0.2.0rc3/ > > > > > > > > [3] > > > > https://dist.apache.org/repos/dist/dev/incubator/amaterasu/KEYS > > > > > > > > [4] https://github.com/apache/incubator-amaterasu/tags > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3
Hi All, Just a kind reminder for our mentors, we need your vote on this one :) Cheers, Yaniv On Wed, May 30, 2018 at 3:25 PM, Arun Manivannan wrote: > Thanks a lot, Nadav. Will get home and spend some more time on this. I was > in a rush and did this poor workaround. My VM is just 8 GB. > > Cheers > Arun > > On Wed, May 30, 2018, 12:27 Nadav Har Tzvi wrote: > > > Yaniv and I just tested it. It worked flawlessly on my end (HDP docker on > > AWS). Both Spark-Scala and PySpark. > > It worked on Yaniv's HDP cluster as well. > > Worth noting: > > 1. HDP 2.6.4 > > 2. Cluster has total of 32GB memory available > > 3. Each container is allocated 1G memory. > > 4. Amaterasu.properties: > > > > zk=sandbox-hdp.hortonworks.com > > version=0.2.0-incubating-rc3 > > master=192.168.33.11 > > user=root > > mode=yarn > > webserver.port=8000 > > webserver.root=dist > > spark.version=2.6.4.0-91 > > yarn.queue=default > > yarn.jarspath=hdfs:///apps/amaterasu > > spark.home=/usr/hdp/current/spark2-client > > > > #spark.home=/opt/cloudera/parcels/SPARK2-2.1.0.cloudera2-1.cdh5.7.0.p0. > 171658/lib/spark2 > > yarn.hadoop.home.dir=/etc/hadoop > > spark.opts.spark.yarn.am.extraJavaOptions="-Dhdp.version=2.6.4.0-91" > > spark.opts.spark.driver.extraJavaOptions="-Dhdp.version=2.6.4.0-91" > > > > > > Arun, please share: > > 1. YARN memory configurations > > 2. amaterasu.properties content > > 3. HDP version. > > > > Cheers, > > Nadav > > > > > > On 30 May 2018 at 07:11, Arun Manivannan wrote: > > > > > The pmem disabling is just temporary. I'll do a detailed analysis and > get > > > back with a proper solution. > > > > > > Any hints on this front is highly appreciated. > > > > > > Cheers > > > Arun > > > > > > On Wed, May 30, 2018, 01:10 Nadav Har Tzvi > > wrote: > > > > > > > Yaniv, Eyal, this might be related to the same issue you faced with > > HDP. > > > > Can you confirm? > > > > > > > > On Tue, May 29, 2018, 17:58 Arun Manivannan wrote: > > > > > > > > > +1 from me > > > > > > > > > > Unit Tests and Build ran fine. > > > > > > > > > > Tested on HDP (VM) but had trouble allocating containers (didn't > have > > > > that > > > > > before). Apparently Centos VMs are known to have this problem. > > > Disabled > > > > > physical memory check (yarn.nodemanager.pmem-check-enabled) and > ran > > > jobs > > > > > successfully. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, May 29, 2018 at 10:42 PM Kirupa Devarajan < > > > > kirupagara...@gmail.com > > > > > > > > > > > wrote: > > > > > > > > > > > Unit tests passing and build was successful on the branch > > > > > > "version-0.2.0-incubating-rc3" > > > > > > > > > > > > +1 from me > > > > > > > > > > > > Cheers, > > > > > > Kirupa > > > > > > > > > > > > > > > > > > On Tue, May 29, 2018 at 3:06 PM, guy peleg > > > wrote: > > > > > > > > > > > > > +1 looks good to me > > > > > > > > > > > > > > On Tue, May 29, 2018, 14:39 Nadav Har Tzvi < > > nadavhart...@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > > > > > > +1 approve. Tested multiple times and after a long round of > > > fixing > > > > > and > > > > > > > > testing over and over. > > > > > > > > > > > > > > > > Cheers, > > > > > > > > Nadav > > > > > > > > > > > > > > > > > > > > > > > > On 29 May 2018 at 07:38, Yaniv Rodenski > > wrote: > > > > > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > > > > > We have fixed the legal issues, as well as a bug found by > > > @Nadav > > > > > > please > > > > > > > > > review and vote on the release candidate #3 for the version > > > > > > > > > 0.2.0-incubating, as follows > > > > > > > > > > > > > > > > > > [ ] +1, Approve the release > > > > > > > > > [ ] -1, Do not approve the release (please provide specific > > > > > comments) > > > > > > > > > > > > > > > > > > The complete staging area is available for your review, > which > > > > > > includes: > > > > > > > > > > > > > > > > > > * JIRA release notes [1], > > > > > > > > > * the official Apache source release to be deployed to > > > > > > dist.apache.org > > > > > > > > > [2], > > > > > > > > > which is signed with the key with fingerprint [3], > > > > > > > > > * source code tag "version-0.2.0-incubating-rc3" [4], > > > > > > > > > * Java artifacts were built with Gradle 3.1 and > > OpenJDK/Oracle > > > > JDK > > > > > > > > > 1.8.0_151 > > > > > > > > > > > > > > > > > > The vote will be open for at least 72 hours. It is adopted > by > > > > > > majority > > > > > > > > > approval, with at least 3 PMC affirmative votes. > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Yaniv > > > > > > > > > > > > > > > > > > [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa > ? > > > > > > > > > projectId=12321521&version=12342793 > > > > > > > > > [2] https://dist.apache.org/repos/ > > > dist/dev/incubator/amaterasu/ > > > > > > > 0.2.0rc3/ > > > > > > > > > [3] > > > > > https://dist.apache.org/repos/dist/dev/incubator/amaterasu/KEYS > > > > > > > > > [4] https://githu
Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3
Great improvement! license/notice/disclaimer: ok headers: ~fine (missing in some non-code files, it would be great to add those -- non-blocking) binaries: there's a gradle binary, but that may be ruled as acceptable (non-blocking from me) Not sure why we need both .tar and .tar.gz. I'd get rid of one of them. Also, shipping private keys in the distribution looks weird as well. For the first release, this is good. So, +1 to send onwards for an Incubator review, and polish a few of these things for a subsequent release. On Wed, May 30, 2018 at 7:12 PM, Yaniv Rodenski wrote: > Hi All, > > Just a kind reminder for our mentors, we need your vote on this one :) > > Cheers, > Yaniv > > On Wed, May 30, 2018 at 3:25 PM, Arun Manivannan wrote: > > > Thanks a lot, Nadav. Will get home and spend some more time on this. I > was > > in a rush and did this poor workaround. My VM is just 8 GB. > > > > Cheers > > Arun > > > > On Wed, May 30, 2018, 12:27 Nadav Har Tzvi > wrote: > > > > > Yaniv and I just tested it. It worked flawlessly on my end (HDP docker > on > > > AWS). Both Spark-Scala and PySpark. > > > It worked on Yaniv's HDP cluster as well. > > > Worth noting: > > > 1. HDP 2.6.4 > > > 2. Cluster has total of 32GB memory available > > > 3. Each container is allocated 1G memory. > > > 4. Amaterasu.properties: > > > > > > zk=sandbox-hdp.hortonworks.com > > > version=0.2.0-incubating-rc3 > > > master=192.168.33.11 > > > user=root > > > mode=yarn > > > webserver.port=8000 > > > webserver.root=dist > > > spark.version=2.6.4.0-91 > > > yarn.queue=default > > > yarn.jarspath=hdfs:///apps/amaterasu > > > spark.home=/usr/hdp/current/spark2-client > > > > > > #spark.home=/opt/cloudera/parcels/SPARK2-2.1.0. > cloudera2-1.cdh5.7.0.p0. > > 171658/lib/spark2 > > > yarn.hadoop.home.dir=/etc/hadoop > > > spark.opts.spark.yarn.am.extraJavaOptions="-Dhdp.version=2.6.4.0-91" > > > spark.opts.spark.driver.extraJavaOptions="-Dhdp.version=2.6.4.0-91" > > > > > > > > > Arun, please share: > > > 1. YARN memory configurations > > > 2. amaterasu.properties content > > > 3. HDP version. > > > > > > Cheers, > > > Nadav > > > > > > > > > On 30 May 2018 at 07:11, Arun Manivannan wrote: > > > > > > > The pmem disabling is just temporary. I'll do a detailed analysis and > > get > > > > back with a proper solution. > > > > > > > > Any hints on this front is highly appreciated. > > > > > > > > Cheers > > > > Arun > > > > > > > > On Wed, May 30, 2018, 01:10 Nadav Har Tzvi > > > wrote: > > > > > > > > > Yaniv, Eyal, this might be related to the same issue you faced with > > > HDP. > > > > > Can you confirm? > > > > > > > > > > On Tue, May 29, 2018, 17:58 Arun Manivannan > wrote: > > > > > > > > > > > +1 from me > > > > > > > > > > > > Unit Tests and Build ran fine. > > > > > > > > > > > > Tested on HDP (VM) but had trouble allocating containers (didn't > > have > > > > > that > > > > > > before). Apparently Centos VMs are known to have this problem. > > > > Disabled > > > > > > physical memory check (yarn.nodemanager.pmem-check-enabled) and > > ran > > > > jobs > > > > > > successfully. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, May 29, 2018 at 10:42 PM Kirupa Devarajan < > > > > > kirupagara...@gmail.com > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > Unit tests passing and build was successful on the branch > > > > > > > "version-0.2.0-incubating-rc3" > > > > > > > > > > > > > > +1 from me > > > > > > > > > > > > > > Cheers, > > > > > > > Kirupa > > > > > > > > > > > > > > > > > > > > > On Tue, May 29, 2018 at 3:06 PM, guy peleg > > > > > wrote: > > > > > > > > > > > > > > > +1 looks good to me > > > > > > > > > > > > > > > > On Tue, May 29, 2018, 14:39 Nadav Har Tzvi < > > > nadavhart...@gmail.com > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > +1 approve. Tested multiple times and after a long round of > > > > fixing > > > > > > and > > > > > > > > > testing over and over. > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > Nadav > > > > > > > > > > > > > > > > > > > > > > > > > > > On 29 May 2018 at 07:38, Yaniv Rodenski > > > wrote: > > > > > > > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > > > > > > > We have fixed the legal issues, as well as a bug found by > > > > @Nadav > > > > > > > please > > > > > > > > > > review and vote on the release candidate #3 for the > version > > > > > > > > > > 0.2.0-incubating, as follows > > > > > > > > > > > > > > > > > > > > [ ] +1, Approve the release > > > > > > > > > > [ ] -1, Do not approve the release (please provide > specific > > > > > > comments) > > > > > > > > > > > > > > > > > > > > The complete staging area is available for your review, > > which > > > > > > > includes: > > > > > > > > > > > > > > > > > > > > * JIRA release notes [1], > > > > > > > > > > * the official Apache source release to be deployed to > > > > > > > dist.apache.org > > > > > > > > > > [2], > > > > > > > > > > which
Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3
Thanks Davor, We will fix those for the next release. Cheers, Yaniv On Fri, Jun 1, 2018 at 7:23 AM, Davor Bonaci wrote: > Great improvement! > > license/notice/disclaimer: ok > headers: ~fine (missing in some non-code files, it would be great to add > those -- non-blocking) > binaries: there's a gradle binary, but that may be ruled as acceptable > (non-blocking from me) > > Not sure why we need both .tar and .tar.gz. I'd get rid of one of them. > Also, shipping private keys in the distribution looks weird as well. > > For the first release, this is good. So, +1 to send onwards for an > Incubator review, and polish a few of these things for a subsequent > release. > > On Wed, May 30, 2018 at 7:12 PM, Yaniv Rodenski wrote: > > > Hi All, > > > > Just a kind reminder for our mentors, we need your vote on this one :) > > > > Cheers, > > Yaniv > > > > On Wed, May 30, 2018 at 3:25 PM, Arun Manivannan > wrote: > > > > > Thanks a lot, Nadav. Will get home and spend some more time on this. I > > was > > > in a rush and did this poor workaround. My VM is just 8 GB. > > > > > > Cheers > > > Arun > > > > > > On Wed, May 30, 2018, 12:27 Nadav Har Tzvi > > wrote: > > > > > > > Yaniv and I just tested it. It worked flawlessly on my end (HDP > docker > > on > > > > AWS). Both Spark-Scala and PySpark. > > > > It worked on Yaniv's HDP cluster as well. > > > > Worth noting: > > > > 1. HDP 2.6.4 > > > > 2. Cluster has total of 32GB memory available > > > > 3. Each container is allocated 1G memory. > > > > 4. Amaterasu.properties: > > > > > > > > zk=sandbox-hdp.hortonworks.com > > > > version=0.2.0-incubating-rc3 > > > > master=192.168.33.11 > > > > user=root > > > > mode=yarn > > > > webserver.port=8000 > > > > webserver.root=dist > > > > spark.version=2.6.4.0-91 > > > > yarn.queue=default > > > > yarn.jarspath=hdfs:///apps/amaterasu > > > > spark.home=/usr/hdp/current/spark2-client > > > > > > > > #spark.home=/opt/cloudera/parcels/SPARK2-2.1.0. > > cloudera2-1.cdh5.7.0.p0. > > > 171658/lib/spark2 > > > > yarn.hadoop.home.dir=/etc/hadoop > > > > spark.opts.spark.yarn.am.extraJavaOptions="-Dhdp.version=2.6.4.0-91" > > > > spark.opts.spark.driver.extraJavaOptions="-Dhdp.version=2.6.4.0-91" > > > > > > > > > > > > Arun, please share: > > > > 1. YARN memory configurations > > > > 2. amaterasu.properties content > > > > 3. HDP version. > > > > > > > > Cheers, > > > > Nadav > > > > > > > > > > > > On 30 May 2018 at 07:11, Arun Manivannan wrote: > > > > > > > > > The pmem disabling is just temporary. I'll do a detailed analysis > and > > > get > > > > > back with a proper solution. > > > > > > > > > > Any hints on this front is highly appreciated. > > > > > > > > > > Cheers > > > > > Arun > > > > > > > > > > On Wed, May 30, 2018, 01:10 Nadav Har Tzvi > > > > > wrote: > > > > > > > > > > > Yaniv, Eyal, this might be related to the same issue you faced > with > > > > HDP. > > > > > > Can you confirm? > > > > > > > > > > > > On Tue, May 29, 2018, 17:58 Arun Manivannan > > wrote: > > > > > > > > > > > > > +1 from me > > > > > > > > > > > > > > Unit Tests and Build ran fine. > > > > > > > > > > > > > > Tested on HDP (VM) but had trouble allocating containers > (didn't > > > have > > > > > > that > > > > > > > before). Apparently Centos VMs are known to have this problem. > > > > > Disabled > > > > > > > physical memory check (yarn.nodemanager.pmem-check-enabled) > and > > > ran > > > > > jobs > > > > > > > successfully. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, May 29, 2018 at 10:42 PM Kirupa Devarajan < > > > > > > kirupagara...@gmail.com > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Unit tests passing and build was successful on the branch > > > > > > > > "version-0.2.0-incubating-rc3" > > > > > > > > > > > > > > > > +1 from me > > > > > > > > > > > > > > > > Cheers, > > > > > > > > Kirupa > > > > > > > > > > > > > > > > > > > > > > > > On Tue, May 29, 2018 at 3:06 PM, guy peleg < > whisr...@gmail.com > > > > > > > > wrote: > > > > > > > > > > > > > > > > > +1 looks good to me > > > > > > > > > > > > > > > > > > On Tue, May 29, 2018, 14:39 Nadav Har Tzvi < > > > > nadavhart...@gmail.com > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > +1 approve. Tested multiple times and after a long round > of > > > > > fixing > > > > > > > and > > > > > > > > > > testing over and over. > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > Nadav > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 29 May 2018 at 07:38, Yaniv Rodenski > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > > > > > > > > > We have fixed the legal issues, as well as a bug found > by > > > > > @Nadav > > > > > > > > please > > > > > > > > > > > review and vote on the release candidate #3 for the > > version > > > > > > > > > > > 0.2.0-incubating, as follows > > > > > > > > > > > > > > > > > > > > >
Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3
One thing to clarify, the tar.gz is the source code as I've seen this is released in other projects. if that is not needed I'm happy to drop it from future releases. On Fri, Jun 1, 2018 at 5:01 PM, Yaniv Rodenski wrote: > Thanks Davor, > > We will fix those for the next release. > > Cheers, > Yaniv > > On Fri, Jun 1, 2018 at 7:23 AM, Davor Bonaci wrote: > >> Great improvement! >> >> license/notice/disclaimer: ok >> headers: ~fine (missing in some non-code files, it would be great to add >> those -- non-blocking) >> binaries: there's a gradle binary, but that may be ruled as acceptable >> (non-blocking from me) >> >> Not sure why we need both .tar and .tar.gz. I'd get rid of one of them. >> Also, shipping private keys in the distribution looks weird as well. >> >> For the first release, this is good. So, +1 to send onwards for an >> Incubator review, and polish a few of these things for a subsequent >> release. >> >> On Wed, May 30, 2018 at 7:12 PM, Yaniv Rodenski wrote: >> >> > Hi All, >> > >> > Just a kind reminder for our mentors, we need your vote on this one :) >> > >> > Cheers, >> > Yaniv >> > >> > On Wed, May 30, 2018 at 3:25 PM, Arun Manivannan >> wrote: >> > >> > > Thanks a lot, Nadav. Will get home and spend some more time on this. I >> > was >> > > in a rush and did this poor workaround. My VM is just 8 GB. >> > > >> > > Cheers >> > > Arun >> > > >> > > On Wed, May 30, 2018, 12:27 Nadav Har Tzvi >> > wrote: >> > > >> > > > Yaniv and I just tested it. It worked flawlessly on my end (HDP >> docker >> > on >> > > > AWS). Both Spark-Scala and PySpark. >> > > > It worked on Yaniv's HDP cluster as well. >> > > > Worth noting: >> > > > 1. HDP 2.6.4 >> > > > 2. Cluster has total of 32GB memory available >> > > > 3. Each container is allocated 1G memory. >> > > > 4. Amaterasu.properties: >> > > > >> > > > zk=sandbox-hdp.hortonworks.com >> > > > version=0.2.0-incubating-rc3 >> > > > master=192.168.33.11 >> > > > user=root >> > > > mode=yarn >> > > > webserver.port=8000 >> > > > webserver.root=dist >> > > > spark.version=2.6.4.0-91 >> > > > yarn.queue=default >> > > > yarn.jarspath=hdfs:///apps/amaterasu >> > > > spark.home=/usr/hdp/current/spark2-client >> > > > >> > > > #spark.home=/opt/cloudera/parcels/SPARK2-2.1.0. >> > cloudera2-1.cdh5.7.0.p0. >> > > 171658/lib/spark2 >> > > > yarn.hadoop.home.dir=/etc/hadoop >> > > > spark.opts.spark.yarn.am.extraJavaOptions="-Dhdp.version=2. >> 6.4.0-91" >> > > > spark.opts.spark.driver.extraJavaOptions="-Dhdp.version=2.6.4.0-91" >> > > > >> > > > >> > > > Arun, please share: >> > > > 1. YARN memory configurations >> > > > 2. amaterasu.properties content >> > > > 3. HDP version. >> > > > >> > > > Cheers, >> > > > Nadav >> > > > >> > > > >> > > > On 30 May 2018 at 07:11, Arun Manivannan wrote: >> > > > >> > > > > The pmem disabling is just temporary. I'll do a detailed analysis >> and >> > > get >> > > > > back with a proper solution. >> > > > > >> > > > > Any hints on this front is highly appreciated. >> > > > > >> > > > > Cheers >> > > > > Arun >> > > > > >> > > > > On Wed, May 30, 2018, 01:10 Nadav Har Tzvi < >> nadavhart...@gmail.com> >> > > > wrote: >> > > > > >> > > > > > Yaniv, Eyal, this might be related to the same issue you faced >> with >> > > > HDP. >> > > > > > Can you confirm? >> > > > > > >> > > > > > On Tue, May 29, 2018, 17:58 Arun Manivannan >> > wrote: >> > > > > > >> > > > > > > +1 from me >> > > > > > > >> > > > > > > Unit Tests and Build ran fine. >> > > > > > > >> > > > > > > Tested on HDP (VM) but had trouble allocating containers >> (didn't >> > > have >> > > > > > that >> > > > > > > before). Apparently Centos VMs are known to have this >> problem. >> > > > > Disabled >> > > > > > > physical memory check (yarn.nodemanager.pmem-check-enabled) >> and >> > > ran >> > > > > jobs >> > > > > > > successfully. >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > On Tue, May 29, 2018 at 10:42 PM Kirupa Devarajan < >> > > > > > kirupagara...@gmail.com >> > > > > > > > >> > > > > > > wrote: >> > > > > > > >> > > > > > > > Unit tests passing and build was successful on the branch >> > > > > > > > "version-0.2.0-incubating-rc3" >> > > > > > > > >> > > > > > > > +1 from me >> > > > > > > > >> > > > > > > > Cheers, >> > > > > > > > Kirupa >> > > > > > > > >> > > > > > > > >> > > > > > > > On Tue, May 29, 2018 at 3:06 PM, guy peleg < >> whisr...@gmail.com >> > > >> > > > > wrote: >> > > > > > > > >> > > > > > > > > +1 looks good to me >> > > > > > > > > >> > > > > > > > > On Tue, May 29, 2018, 14:39 Nadav Har Tzvi < >> > > > nadavhart...@gmail.com >> > > > > > >> > > > > > > > wrote: >> > > > > > > > > >> > > > > > > > > > +1 approve. Tested multiple times and after a long >> round of >> > > > > fixing >> > > > > > > and >> > > > > > > > > > testing over and over. >> > > > > > > > > > >> > > > > > > > > > Cheers, >> > > > > > > > > > Nadav >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > On 29 Ma
Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3
No problem with .tar.gz. But, you don't need .tar then. On Fri, Jun 1, 2018 at 12:04 AM, Yaniv Rodenski wrote: > One thing to clarify, the tar.gz is the source code as I've seen this is > released in other projects. if that is not needed I'm happy to drop it from > future releases. > > On Fri, Jun 1, 2018 at 5:01 PM, Yaniv Rodenski wrote: > > > Thanks Davor, > > > > We will fix those for the next release. > > > > Cheers, > > Yaniv > > > > On Fri, Jun 1, 2018 at 7:23 AM, Davor Bonaci wrote: > > > >> Great improvement! > >> > >> license/notice/disclaimer: ok > >> headers: ~fine (missing in some non-code files, it would be great to add > >> those -- non-blocking) > >> binaries: there's a gradle binary, but that may be ruled as acceptable > >> (non-blocking from me) > >> > >> Not sure why we need both .tar and .tar.gz. I'd get rid of one of them. > >> Also, shipping private keys in the distribution looks weird as well. > >> > >> For the first release, this is good. So, +1 to send onwards for an > >> Incubator review, and polish a few of these things for a subsequent > >> release. > >> > >> On Wed, May 30, 2018 at 7:12 PM, Yaniv Rodenski > wrote: > >> > >> > Hi All, > >> > > >> > Just a kind reminder for our mentors, we need your vote on this one :) > >> > > >> > Cheers, > >> > Yaniv > >> > > >> > On Wed, May 30, 2018 at 3:25 PM, Arun Manivannan > >> wrote: > >> > > >> > > Thanks a lot, Nadav. Will get home and spend some more time on > this. I > >> > was > >> > > in a rush and did this poor workaround. My VM is just 8 GB. > >> > > > >> > > Cheers > >> > > Arun > >> > > > >> > > On Wed, May 30, 2018, 12:27 Nadav Har Tzvi > >> > wrote: > >> > > > >> > > > Yaniv and I just tested it. It worked flawlessly on my end (HDP > >> docker > >> > on > >> > > > AWS). Both Spark-Scala and PySpark. > >> > > > It worked on Yaniv's HDP cluster as well. > >> > > > Worth noting: > >> > > > 1. HDP 2.6.4 > >> > > > 2. Cluster has total of 32GB memory available > >> > > > 3. Each container is allocated 1G memory. > >> > > > 4. Amaterasu.properties: > >> > > > > >> > > > zk=sandbox-hdp.hortonworks.com > >> > > > version=0.2.0-incubating-rc3 > >> > > > master=192.168.33.11 > >> > > > user=root > >> > > > mode=yarn > >> > > > webserver.port=8000 > >> > > > webserver.root=dist > >> > > > spark.version=2.6.4.0-91 > >> > > > yarn.queue=default > >> > > > yarn.jarspath=hdfs:///apps/amaterasu > >> > > > spark.home=/usr/hdp/current/spark2-client > >> > > > > >> > > > #spark.home=/opt/cloudera/parcels/SPARK2-2.1.0. > >> > cloudera2-1.cdh5.7.0.p0. > >> > > 171658/lib/spark2 > >> > > > yarn.hadoop.home.dir=/etc/hadoop > >> > > > spark.opts.spark.yarn.am.extraJavaOptions="-Dhdp.version=2. > >> 6.4.0-91" > >> > > > spark.opts.spark.driver.extraJavaOptions="-Dhdp. > version=2.6.4.0-91" > >> > > > > >> > > > > >> > > > Arun, please share: > >> > > > 1. YARN memory configurations > >> > > > 2. amaterasu.properties content > >> > > > 3. HDP version. > >> > > > > >> > > > Cheers, > >> > > > Nadav > >> > > > > >> > > > > >> > > > On 30 May 2018 at 07:11, Arun Manivannan wrote: > >> > > > > >> > > > > The pmem disabling is just temporary. I'll do a detailed > analysis > >> and > >> > > get > >> > > > > back with a proper solution. > >> > > > > > >> > > > > Any hints on this front is highly appreciated. > >> > > > > > >> > > > > Cheers > >> > > > > Arun > >> > > > > > >> > > > > On Wed, May 30, 2018, 01:10 Nadav Har Tzvi < > >> nadavhart...@gmail.com> > >> > > > wrote: > >> > > > > > >> > > > > > Yaniv, Eyal, this might be related to the same issue you faced > >> with > >> > > > HDP. > >> > > > > > Can you confirm? > >> > > > > > > >> > > > > > On Tue, May 29, 2018, 17:58 Arun Manivannan > >> > wrote: > >> > > > > > > >> > > > > > > +1 from me > >> > > > > > > > >> > > > > > > Unit Tests and Build ran fine. > >> > > > > > > > >> > > > > > > Tested on HDP (VM) but had trouble allocating containers > >> (didn't > >> > > have > >> > > > > > that > >> > > > > > > before). Apparently Centos VMs are known to have this > >> problem. > >> > > > > Disabled > >> > > > > > > physical memory check (yarn.nodemanager.pmem-check- > enabled) > >> and > >> > > ran > >> > > > > jobs > >> > > > > > > successfully. > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > On Tue, May 29, 2018 at 10:42 PM Kirupa Devarajan < > >> > > > > > kirupagara...@gmail.com > >> > > > > > > > > >> > > > > > > wrote: > >> > > > > > > > >> > > > > > > > Unit tests passing and build was successful on the branch > >> > > > > > > > "version-0.2.0-incubating-rc3" > >> > > > > > > > > >> > > > > > > > +1 from me > >> > > > > > > > > >> > > > > > > > Cheers, > >> > > > > > > > Kirupa > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > On Tue, May 29, 2018 at 3:06 PM, guy peleg < > >> whisr...@gmail.com > >> > > > >> > > > > wrote: > >> > > > > > > > > >> > > > > > > > > +1 looks good to me > >> > > > > > > > > > >> > > > > > > > > On
Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3
Hi Davor, Sorry for keeping my emails short and undescriptive, I have only one hand for the next month due to a skateboarding accident. Based on what I've seen in other projects, it seems common to release bot the binaries (in Amaterasu those are released in the .tar file) as well as a source (which are in the -source.tar.gz file) The main question is if we do or do not need to release both. Cheers, Yaniv On Sat, Jun 2, 2018 at 1:13 AM, Davor Bonaci wrote: > No problem with .tar.gz. But, you don't need .tar then. > > On Fri, Jun 1, 2018 at 12:04 AM, Yaniv Rodenski wrote: > > > One thing to clarify, the tar.gz is the source code as I've seen this is > > released in other projects. if that is not needed I'm happy to drop it > from > > future releases. > > > > On Fri, Jun 1, 2018 at 5:01 PM, Yaniv Rodenski wrote: > > > > > Thanks Davor, > > > > > > We will fix those for the next release. > > > > > > Cheers, > > > Yaniv > > > > > > On Fri, Jun 1, 2018 at 7:23 AM, Davor Bonaci > wrote: > > > > > >> Great improvement! > > >> > > >> license/notice/disclaimer: ok > > >> headers: ~fine (missing in some non-code files, it would be great to > add > > >> those -- non-blocking) > > >> binaries: there's a gradle binary, but that may be ruled as acceptable > > >> (non-blocking from me) > > >> > > >> Not sure why we need both .tar and .tar.gz. I'd get rid of one of > them. > > >> Also, shipping private keys in the distribution looks weird as well. > > >> > > >> For the first release, this is good. So, +1 to send onwards for an > > >> Incubator review, and polish a few of these things for a subsequent > > >> release. > > >> > > >> On Wed, May 30, 2018 at 7:12 PM, Yaniv Rodenski > > wrote: > > >> > > >> > Hi All, > > >> > > > >> > Just a kind reminder for our mentors, we need your vote on this one > :) > > >> > > > >> > Cheers, > > >> > Yaniv > > >> > > > >> > On Wed, May 30, 2018 at 3:25 PM, Arun Manivannan > > >> wrote: > > >> > > > >> > > Thanks a lot, Nadav. Will get home and spend some more time on > > this. I > > >> > was > > >> > > in a rush and did this poor workaround. My VM is just 8 GB. > > >> > > > > >> > > Cheers > > >> > > Arun > > >> > > > > >> > > On Wed, May 30, 2018, 12:27 Nadav Har Tzvi < > nadavhart...@gmail.com> > > >> > wrote: > > >> > > > > >> > > > Yaniv and I just tested it. It worked flawlessly on my end (HDP > > >> docker > > >> > on > > >> > > > AWS). Both Spark-Scala and PySpark. > > >> > > > It worked on Yaniv's HDP cluster as well. > > >> > > > Worth noting: > > >> > > > 1. HDP 2.6.4 > > >> > > > 2. Cluster has total of 32GB memory available > > >> > > > 3. Each container is allocated 1G memory. > > >> > > > 4. Amaterasu.properties: > > >> > > > > > >> > > > zk=sandbox-hdp.hortonworks.com > > >> > > > version=0.2.0-incubating-rc3 > > >> > > > master=192.168.33.11 > > >> > > > user=root > > >> > > > mode=yarn > > >> > > > webserver.port=8000 > > >> > > > webserver.root=dist > > >> > > > spark.version=2.6.4.0-91 > > >> > > > yarn.queue=default > > >> > > > yarn.jarspath=hdfs:///apps/amaterasu > > >> > > > spark.home=/usr/hdp/current/spark2-client > > >> > > > > > >> > > > #spark.home=/opt/cloudera/parcels/SPARK2-2.1.0. > > >> > cloudera2-1.cdh5.7.0.p0. > > >> > > 171658/lib/spark2 > > >> > > > yarn.hadoop.home.dir=/etc/hadoop > > >> > > > spark.opts.spark.yarn.am.extraJavaOptions="-Dhdp.version=2. > > >> 6.4.0-91" > > >> > > > spark.opts.spark.driver.extraJavaOptions="-Dhdp. > > version=2.6.4.0-91" > > >> > > > > > >> > > > > > >> > > > Arun, please share: > > >> > > > 1. YARN memory configurations > > >> > > > 2. amaterasu.properties content > > >> > > > 3. HDP version. > > >> > > > > > >> > > > Cheers, > > >> > > > Nadav > > >> > > > > > >> > > > > > >> > > > On 30 May 2018 at 07:11, Arun Manivannan > wrote: > > >> > > > > > >> > > > > The pmem disabling is just temporary. I'll do a detailed > > analysis > > >> and > > >> > > get > > >> > > > > back with a proper solution. > > >> > > > > > > >> > > > > Any hints on this front is highly appreciated. > > >> > > > > > > >> > > > > Cheers > > >> > > > > Arun > > >> > > > > > > >> > > > > On Wed, May 30, 2018, 01:10 Nadav Har Tzvi < > > >> nadavhart...@gmail.com> > > >> > > > wrote: > > >> > > > > > > >> > > > > > Yaniv, Eyal, this might be related to the same issue you > faced > > >> with > > >> > > > HDP. > > >> > > > > > Can you confirm? > > >> > > > > > > > >> > > > > > On Tue, May 29, 2018, 17:58 Arun Manivannan < > a...@arunma.com> > > >> > wrote: > > >> > > > > > > > >> > > > > > > +1 from me > > >> > > > > > > > > >> > > > > > > Unit Tests and Build ran fine. > > >> > > > > > > > > >> > > > > > > Tested on HDP (VM) but had trouble allocating containers > > >> (didn't > > >> > > have > > >> > > > > > that > > >> > > > > > > before). Apparently Centos VMs are known to have this > > >> problem. > > >> > > > > Disabled > > >> > > > > > > physical memory check (yarn.nodemanager.pmem-check- > > enabled) > > >> and > > >> > > ran > >
Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3
Gentlemen, Apologies for coming back late. The issue was just the minimum container size that was configured in my cluster (yarn.scheduler.maximum-allocation-mb). It was set at 1 GB. I didn't specify any spark specific memory parameters during my run (the memory defaults that the SparkSetupProvider was looking at) and to top it the code was setting the Xmx at 1 GB causing the overallocation and failure. I have one minor proposal. If this is agreeable, I can raise a quick PR. Can we pull out the executor java options as a property in the amaterasu.properties? amaterasu.executor.extra.java.opts = "-Xmx1G -Dscala.usejavacp=true -Dhdp.version=2.6.5.0-292" As a side effect, we must provide the flexibility to allow quotes around the parameter but passing the quotes to the java command would fail. I have stripped off the extra quotes in a dirty way at the moment. Should we consider proper command parsing (and possibly convert them to be bash compatible strings)? s"java -cp spark/jars/*:executor.jar:spark/conf/:${config.YARN.hadoopHomeDir}/conf/ " + s" ${config.amaterasuExecutorJavaOpts.replaceAll("\"","")} "+ Meanwhile, I'll also update by PR for Amaterasu-24 after pulling the latest from the branch. Cheers, Arun On Wed, May 30, 2018 at 1:25 PM Arun Manivannan wrote: > Thanks a lot, Nadav. Will get home and spend some more time on this. I was > in a rush and did this poor workaround. My VM is just 8 GB. > > Cheers > Arun > > > On Wed, May 30, 2018, 12:27 Nadav Har Tzvi wrote: > >> Yaniv and I just tested it. It worked flawlessly on my end (HDP docker on >> AWS). Both Spark-Scala and PySpark. >> It worked on Yaniv's HDP cluster as well. >> Worth noting: >> 1. HDP 2.6.4 >> 2. Cluster has total of 32GB memory available >> 3. Each container is allocated 1G memory. >> 4. Amaterasu.properties: >> >> zk=sandbox-hdp.hortonworks.com >> version=0.2.0-incubating-rc3 >> master=192.168.33.11 >> user=root >> mode=yarn >> webserver.port=8000 >> webserver.root=dist >> spark.version=2.6.4.0-91 >> yarn.queue=default >> yarn.jarspath=hdfs:///apps/amaterasu >> spark.home=/usr/hdp/current/spark2-client >> >> #spark.home=/opt/cloudera/parcels/SPARK2-2.1.0.cloudera2-1.cdh5.7.0.p0.171658/lib/spark2 >> yarn.hadoop.home.dir=/etc/hadoop >> spark.opts.spark.yarn.am.extraJavaOptions="-Dhdp.version=2.6.4.0-91" >> spark.opts.spark.driver.extraJavaOptions="-Dhdp.version=2.6.4.0-91" >> >> >> Arun, please share: >> 1. YARN memory configurations >> 2. amaterasu.properties content >> 3. HDP version. >> >> Cheers, >> Nadav >> >> >> On 30 May 2018 at 07:11, Arun Manivannan wrote: >> >> > The pmem disabling is just temporary. I'll do a detailed analysis and >> get >> > back with a proper solution. >> > >> > Any hints on this front is highly appreciated. >> > >> > Cheers >> > Arun >> > >> > On Wed, May 30, 2018, 01:10 Nadav Har Tzvi >> wrote: >> > >> > > Yaniv, Eyal, this might be related to the same issue you faced with >> HDP. >> > > Can you confirm? >> > > >> > > On Tue, May 29, 2018, 17:58 Arun Manivannan wrote: >> > > >> > > > +1 from me >> > > > >> > > > Unit Tests and Build ran fine. >> > > > >> > > > Tested on HDP (VM) but had trouble allocating containers (didn't >> have >> > > that >> > > > before). Apparently Centos VMs are known to have this problem. >> > Disabled >> > > > physical memory check (yarn.nodemanager.pmem-check-enabled) and ran >> > jobs >> > > > successfully. >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > On Tue, May 29, 2018 at 10:42 PM Kirupa Devarajan < >> > > kirupagara...@gmail.com >> > > > > >> > > > wrote: >> > > > >> > > > > Unit tests passing and build was successful on the branch >> > > > > "version-0.2.0-incubating-rc3" >> > > > > >> > > > > +1 from me >> > > > > >> > > > > Cheers, >> > > > > Kirupa >> > > > > >> > > > > >> > > > > On Tue, May 29, 2018 at 3:06 PM, guy peleg >> > wrote: >> > > > > >> > > > > > +1 looks good to me >> > > > > > >> > > > > > On Tue, May 29, 2018, 14:39 Nadav Har Tzvi < >> nadavhart...@gmail.com >> > > >> > > > > wrote: >> > > > > > >> > > > > > > +1 approve. Tested multiple times and after a long round of >> > fixing >> > > > and >> > > > > > > testing over and over. >> > > > > > > >> > > > > > > Cheers, >> > > > > > > Nadav >> > > > > > > >> > > > > > > >> > > > > > > On 29 May 2018 at 07:38, Yaniv Rodenski >> wrote: >> > > > > > > >> > > > > > > > Hi everyone, >> > > > > > > > >> > > > > > > > We have fixed the legal issues, as well as a bug found by >> > @Nadav >> > > > > please >> > > > > > > > review and vote on the release candidate #3 for the version >> > > > > > > > 0.2.0-incubating, as follows >> > > > > > > > >> > > > > > > > [ ] +1, Approve the release >> > > > > > > > [ ] -1, Do not approve the release (please provide specific >> > > > comments) >> > > > > > > > >> > > > > > > > The complete staging area is available for your review, >> which >> > > > > includes: >> > > > > > > > >> > > > > > > > * JIRA release notes [1], >> > > > > > > > *
Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3
Hi Arun, Fair point, but I think it can be configured via env/[env_name]/spark.yml am I wrong? Anyway, ideally I think we should try and have per job configurations in the environment rather than in the amaterasu.properties Cheers, Yaniv On Sun, 3 Jun 2018 at 1:32 am, Arun Manivannan wrote: > Gentlemen, > > Apologies for coming back late. The issue was just the minimum container > size that was configured in my cluster > (yarn.scheduler.maximum-allocation-mb). It was set at 1 GB. > > I didn't specify any spark specific memory parameters during my run (the > memory defaults that the SparkSetupProvider was looking at) and to top it > the code was setting the Xmx at 1 GB causing the overallocation and > failure. > > I have one minor proposal. If this is agreeable, I can raise a quick PR. > > Can we pull out the executor java options as a property in the > amaterasu.properties? > > amaterasu.executor.extra.java.opts = "-Xmx1G -Dscala.usejavacp=true > -Dhdp.version=2.6.5.0-292" > > > As a side effect, we must provide the flexibility to allow quotes around > the parameter but passing the quotes to the java command would fail. I > have stripped off the extra quotes in a dirty way at the moment. Should we > consider proper command parsing (and possibly convert them to be bash > compatible strings)? > > s"java -cp > spark/jars/*:executor.jar:spark/conf/:${config.YARN.hadoopHomeDir}/conf/ > " + > s" ${config.amaterasuExecutorJavaOpts.replaceAll("\"","")} "+ > > > Meanwhile, I'll also update by PR for Amaterasu-24 after pulling the latest > from the branch. > > Cheers, > Arun > > On Wed, May 30, 2018 at 1:25 PM Arun Manivannan wrote: > > > Thanks a lot, Nadav. Will get home and spend some more time on this. I > was > > in a rush and did this poor workaround. My VM is just 8 GB. > > > > Cheers > > Arun > > > > > > On Wed, May 30, 2018, 12:27 Nadav Har Tzvi > wrote: > > > >> Yaniv and I just tested it. It worked flawlessly on my end (HDP docker > on > >> AWS). Both Spark-Scala and PySpark. > >> It worked on Yaniv's HDP cluster as well. > >> Worth noting: > >> 1. HDP 2.6.4 > >> 2. Cluster has total of 32GB memory available > >> 3. Each container is allocated 1G memory. > >> 4. Amaterasu.properties: > >> > >> zk=sandbox-hdp.hortonworks.com > >> version=0.2.0-incubating-rc3 > >> master=192.168.33.11 > >> user=root > >> mode=yarn > >> webserver.port=8000 > >> webserver.root=dist > >> spark.version=2.6.4.0-91 > >> yarn.queue=default > >> yarn.jarspath=hdfs:///apps/amaterasu > >> spark.home=/usr/hdp/current/spark2-client > >> > >> > #spark.home=/opt/cloudera/parcels/SPARK2-2.1.0.cloudera2-1.cdh5.7.0.p0.171658/lib/spark2 > >> yarn.hadoop.home.dir=/etc/hadoop > >> spark.opts.spark.yarn.am.extraJavaOptions="-Dhdp.version=2.6.4.0-91" > >> spark.opts.spark.driver.extraJavaOptions="-Dhdp.version=2.6.4.0-91" > >> > >> > >> Arun, please share: > >> 1. YARN memory configurations > >> 2. amaterasu.properties content > >> 3. HDP version. > >> > >> Cheers, > >> Nadav > >> > >> > >> On 30 May 2018 at 07:11, Arun Manivannan wrote: > >> > >> > The pmem disabling is just temporary. I'll do a detailed analysis and > >> get > >> > back with a proper solution. > >> > > >> > Any hints on this front is highly appreciated. > >> > > >> > Cheers > >> > Arun > >> > > >> > On Wed, May 30, 2018, 01:10 Nadav Har Tzvi > >> wrote: > >> > > >> > > Yaniv, Eyal, this might be related to the same issue you faced with > >> HDP. > >> > > Can you confirm? > >> > > > >> > > On Tue, May 29, 2018, 17:58 Arun Manivannan > wrote: > >> > > > >> > > > +1 from me > >> > > > > >> > > > Unit Tests and Build ran fine. > >> > > > > >> > > > Tested on HDP (VM) but had trouble allocating containers (didn't > >> have > >> > > that > >> > > > before). Apparently Centos VMs are known to have this problem. > >> > Disabled > >> > > > physical memory check (yarn.nodemanager.pmem-check-enabled) and > ran > >> > jobs > >> > > > successfully. > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > On Tue, May 29, 2018 at 10:42 PM Kirupa Devarajan < > >> > > kirupagara...@gmail.com > >> > > > > > >> > > > wrote: > >> > > > > >> > > > > Unit tests passing and build was successful on the branch > >> > > > > "version-0.2.0-incubating-rc3" > >> > > > > > >> > > > > +1 from me > >> > > > > > >> > > > > Cheers, > >> > > > > Kirupa > >> > > > > > >> > > > > > >> > > > > On Tue, May 29, 2018 at 3:06 PM, guy peleg > >> > wrote: > >> > > > > > >> > > > > > +1 looks good to me > >> > > > > > > >> > > > > > On Tue, May 29, 2018, 14:39 Nadav Har Tzvi < > >> nadavhart...@gmail.com > >> > > > >> > > > > wrote: > >> > > > > > > >> > > > > > > +1 approve. Tested multiple times and after a long round of > >> > fixing > >> > > > and > >> > > > > > > testing over and over. > >> > > > > > > > >> > > > > > > Cheers, > >> > > > > > > Nadav > >> > > > > > > > >> > > > > > > > >> > > > > > > On 29 May 2018 at 07:38, Yaniv Rodenski > >> wrote: > >> > > > > > > > >> > > > > > > > H
Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3
Ah. Missed the part where we could configure parameters at a job level. Thanks a lot for clarifying, Yaniv. Cheers, Arun On Sun, Jun 3, 2018 at 6:37 PM Yaniv Rodenski wrote: > Hi Arun, > > Fair point, but I think it can be configured via env/[env_name]/spark.yml > am I wrong? > Anyway, ideally I think we should try and have per job configurations in > the environment rather than in the amaterasu.properties > > Cheers, > Yaniv > > On Sun, 3 Jun 2018 at 1:32 am, Arun Manivannan wrote: > > > Gentlemen, > > > > Apologies for coming back late. The issue was just the minimum container > > size that was configured in my cluster > > (yarn.scheduler.maximum-allocation-mb). It was set at 1 GB. > > > > I didn't specify any spark specific memory parameters during my run (the > > memory defaults that the SparkSetupProvider was looking at) and to top it > > the code was setting the Xmx at 1 GB causing the overallocation and > > failure. > > > > I have one minor proposal. If this is agreeable, I can raise a quick PR. > > > > Can we pull out the executor java options as a property in the > > amaterasu.properties? > > > > amaterasu.executor.extra.java.opts = "-Xmx1G -Dscala.usejavacp=true > > -Dhdp.version=2.6.5.0-292" > > > > > > As a side effect, we must provide the flexibility to allow quotes around > > the parameter but passing the quotes to the java command would fail. I > > have stripped off the extra quotes in a dirty way at the moment. Should > we > > consider proper command parsing (and possibly convert them to be bash > > compatible strings)? > > > > s"java -cp > > spark/jars/*:executor.jar:spark/conf/:${config.YARN.hadoopHomeDir}/conf/ > > " + > > s" ${config.amaterasuExecutorJavaOpts.replaceAll("\"","")} "+ > > > > > > Meanwhile, I'll also update by PR for Amaterasu-24 after pulling the > latest > > from the branch. > > > > Cheers, > > Arun > > > > On Wed, May 30, 2018 at 1:25 PM Arun Manivannan wrote: > > > > > Thanks a lot, Nadav. Will get home and spend some more time on this. I > > was > > > in a rush and did this poor workaround. My VM is just 8 GB. > > > > > > Cheers > > > Arun > > > > > > > > > On Wed, May 30, 2018, 12:27 Nadav Har Tzvi > > wrote: > > > > > >> Yaniv and I just tested it. It worked flawlessly on my end (HDP docker > > on > > >> AWS). Both Spark-Scala and PySpark. > > >> It worked on Yaniv's HDP cluster as well. > > >> Worth noting: > > >> 1. HDP 2.6.4 > > >> 2. Cluster has total of 32GB memory available > > >> 3. Each container is allocated 1G memory. > > >> 4. Amaterasu.properties: > > >> > > >> zk=sandbox-hdp.hortonworks.com > > >> version=0.2.0-incubating-rc3 > > >> master=192.168.33.11 > > >> user=root > > >> mode=yarn > > >> webserver.port=8000 > > >> webserver.root=dist > > >> spark.version=2.6.4.0-91 > > >> yarn.queue=default > > >> yarn.jarspath=hdfs:///apps/amaterasu > > >> spark.home=/usr/hdp/current/spark2-client > > >> > > >> > > > #spark.home=/opt/cloudera/parcels/SPARK2-2.1.0.cloudera2-1.cdh5.7.0.p0.171658/lib/spark2 > > >> yarn.hadoop.home.dir=/etc/hadoop > > >> spark.opts.spark.yarn.am.extraJavaOptions="-Dhdp.version=2.6.4.0-91" > > >> spark.opts.spark.driver.extraJavaOptions="-Dhdp.version=2.6.4.0-91" > > >> > > >> > > >> Arun, please share: > > >> 1. YARN memory configurations > > >> 2. amaterasu.properties content > > >> 3. HDP version. > > >> > > >> Cheers, > > >> Nadav > > >> > > >> > > >> On 30 May 2018 at 07:11, Arun Manivannan wrote: > > >> > > >> > The pmem disabling is just temporary. I'll do a detailed analysis > and > > >> get > > >> > back with a proper solution. > > >> > > > >> > Any hints on this front is highly appreciated. > > >> > > > >> > Cheers > > >> > Arun > > >> > > > >> > On Wed, May 30, 2018, 01:10 Nadav Har Tzvi > > >> wrote: > > >> > > > >> > > Yaniv, Eyal, this might be related to the same issue you faced > with > > >> HDP. > > >> > > Can you confirm? > > >> > > > > >> > > On Tue, May 29, 2018, 17:58 Arun Manivannan > > wrote: > > >> > > > > >> > > > +1 from me > > >> > > > > > >> > > > Unit Tests and Build ran fine. > > >> > > > > > >> > > > Tested on HDP (VM) but had trouble allocating containers (didn't > > >> have > > >> > > that > > >> > > > before). Apparently Centos VMs are known to have this problem. > > >> > Disabled > > >> > > > physical memory check (yarn.nodemanager.pmem-check-enabled) and > > ran > > >> > jobs > > >> > > > successfully. > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > On Tue, May 29, 2018 at 10:42 PM Kirupa Devarajan < > > >> > > kirupagara...@gmail.com > > >> > > > > > > >> > > > wrote: > > >> > > > > > >> > > > > Unit tests passing and build was successful on the branch > > >> > > > > "version-0.2.0-incubating-rc3" > > >> > > > > > > >> > > > > +1 from me > > >> > > > > > > >> > > > > Cheers, > > >> > > > > Kirupa > > >> > > > > > > >> > > > > > > >> > > > > On Tue, May 29, 2018 at 3:06 PM, guy peleg < > whisr...@gmail.com> > > >> > wrote: > > >> > > > > > > >> > > >
Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3
Hi All, Just following up with @JB and @Olivier, any chance you guys can have a look and vote? Cheers, Yaniv On Mon, Jun 4, 2018 at 12:08 AM, Arun Manivannan wrote: > Ah. Missed the part where we could configure parameters at a job level. > Thanks a lot for clarifying, Yaniv. > > Cheers, > Arun > > On Sun, Jun 3, 2018 at 6:37 PM Yaniv Rodenski wrote: > > > Hi Arun, > > > > Fair point, but I think it can be configured via env/[env_name]/spark.yml > > am I wrong? > > Anyway, ideally I think we should try and have per job configurations in > > the environment rather than in the amaterasu.properties > > > > Cheers, > > Yaniv > > > > On Sun, 3 Jun 2018 at 1:32 am, Arun Manivannan wrote: > > > > > Gentlemen, > > > > > > Apologies for coming back late. The issue was just the minimum > container > > > size that was configured in my cluster > > > (yarn.scheduler.maximum-allocation-mb). It was set at 1 GB. > > > > > > I didn't specify any spark specific memory parameters during my run > (the > > > memory defaults that the SparkSetupProvider was looking at) and to top > it > > > the code was setting the Xmx at 1 GB causing the overallocation and > > > failure. > > > > > > I have one minor proposal. If this is agreeable, I can raise a quick > PR. > > > > > > Can we pull out the executor java options as a property in the > > > amaterasu.properties? > > > > > > amaterasu.executor.extra.java.opts = "-Xmx1G -Dscala.usejavacp=true > > > -Dhdp.version=2.6.5.0-292" > > > > > > > > > As a side effect, we must provide the flexibility to allow quotes > around > > > the parameter but passing the quotes to the java command would fail. I > > > have stripped off the extra quotes in a dirty way at the moment. Should > > we > > > consider proper command parsing (and possibly convert them to be bash > > > compatible strings)? > > > > > > s"java -cp > > > > spark/jars/*:executor.jar:spark/conf/:${config.YARN.hadoopHomeDir}/conf/ > > > " + > > > s" ${config.amaterasuExecutorJavaOpts.replaceAll("\"","")} "+ > > > > > > > > > Meanwhile, I'll also update by PR for Amaterasu-24 after pulling the > > latest > > > from the branch. > > > > > > Cheers, > > > Arun > > > > > > On Wed, May 30, 2018 at 1:25 PM Arun Manivannan > wrote: > > > > > > > Thanks a lot, Nadav. Will get home and spend some more time on this. > I > > > was > > > > in a rush and did this poor workaround. My VM is just 8 GB. > > > > > > > > Cheers > > > > Arun > > > > > > > > > > > > On Wed, May 30, 2018, 12:27 Nadav Har Tzvi > > > wrote: > > > > > > > >> Yaniv and I just tested it. It worked flawlessly on my end (HDP > docker > > > on > > > >> AWS). Both Spark-Scala and PySpark. > > > >> It worked on Yaniv's HDP cluster as well. > > > >> Worth noting: > > > >> 1. HDP 2.6.4 > > > >> 2. Cluster has total of 32GB memory available > > > >> 3. Each container is allocated 1G memory. > > > >> 4. Amaterasu.properties: > > > >> > > > >> zk=sandbox-hdp.hortonworks.com > > > >> version=0.2.0-incubating-rc3 > > > >> master=192.168.33.11 > > > >> user=root > > > >> mode=yarn > > > >> webserver.port=8000 > > > >> webserver.root=dist > > > >> spark.version=2.6.4.0-91 > > > >> yarn.queue=default > > > >> yarn.jarspath=hdfs:///apps/amaterasu > > > >> spark.home=/usr/hdp/current/spark2-client > > > >> > > > >> > > > > > > #spark.home=/opt/cloudera/parcels/SPARK2-2.1.0.cloudera2-1.cdh5.7.0.p0.171658/lib/spark2 > > > >> yarn.hadoop.home.dir=/etc/hadoop > > > >> spark.opts.spark.yarn.am > .extraJavaOptions="-Dhdp.version=2.6.4.0-91" > > > >> spark.opts.spark.driver.extraJavaOptions="-Dhdp.version=2.6.4.0-91" > > > >> > > > >> > > > >> Arun, please share: > > > >> 1. YARN memory configurations > > > >> 2. amaterasu.properties content > > > >> 3. HDP version. > > > >> > > > >> Cheers, > > > >> Nadav > > > >> > > > >> > > > >> On 30 May 2018 at 07:11, Arun Manivannan wrote: > > > >> > > > >> > The pmem disabling is just temporary. I'll do a detailed analysis > > and > > > >> get > > > >> > back with a proper solution. > > > >> > > > > >> > Any hints on this front is highly appreciated. > > > >> > > > > >> > Cheers > > > >> > Arun > > > >> > > > > >> > On Wed, May 30, 2018, 01:10 Nadav Har Tzvi < > nadavhart...@gmail.com> > > > >> wrote: > > > >> > > > > >> > > Yaniv, Eyal, this might be related to the same issue you faced > > with > > > >> HDP. > > > >> > > Can you confirm? > > > >> > > > > > >> > > On Tue, May 29, 2018, 17:58 Arun Manivannan > > > wrote: > > > >> > > > > > >> > > > +1 from me > > > >> > > > > > > >> > > > Unit Tests and Build ran fine. > > > >> > > > > > > >> > > > Tested on HDP (VM) but had trouble allocating containers > (didn't > > > >> have > > > >> > > that > > > >> > > > before). Apparently Centos VMs are known to have this > problem. > > > >> > Disabled > > > >> > > > physical memory check (yarn.nodemanager.pmem-check-enabled) > and > > > ran > > > >> > jobs > > > >> > > > successfully. > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> >
Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3
> > Based on what I've seen in other projects, it seems common to release bot > the binaries (in Amaterasu those are released in the .tar file) as well as > a source (which are in the -source.tar.gz file) > The main question is if we do or do not need to release both. > Sorry, my mistake. It is perfectly fine to release convenience binaries in addition to source code. You may want to name them better and refer to both from the vote email, however ;-)
Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3
+1 On Mon, 4 Jun 2018 at 17:02, Yaniv Rodenski wrote: > Hi All, > > Just following up with @JB and @Olivier, any chance you guys can have a > look and vote? > > Cheers, > Yaniv > > On Mon, Jun 4, 2018 at 12:08 AM, Arun Manivannan wrote: > > > Ah. Missed the part where we could configure parameters at a job level. > > Thanks a lot for clarifying, Yaniv. > > > > Cheers, > > Arun > > > > On Sun, Jun 3, 2018 at 6:37 PM Yaniv Rodenski wrote: > > > > > Hi Arun, > > > > > > Fair point, but I think it can be configured via > env/[env_name]/spark.yml > > > am I wrong? > > > Anyway, ideally I think we should try and have per job configurations > in > > > the environment rather than in the amaterasu.properties > > > > > > Cheers, > > > Yaniv > > > > > > On Sun, 3 Jun 2018 at 1:32 am, Arun Manivannan > wrote: > > > > > > > Gentlemen, > > > > > > > > Apologies for coming back late. The issue was just the minimum > > container > > > > size that was configured in my cluster > > > > (yarn.scheduler.maximum-allocation-mb). It was set at 1 GB. > > > > > > > > I didn't specify any spark specific memory parameters during my run > > (the > > > > memory defaults that the SparkSetupProvider was looking at) and to > top > > it > > > > the code was setting the Xmx at 1 GB causing the overallocation and > > > > failure. > > > > > > > > I have one minor proposal. If this is agreeable, I can raise a quick > > PR. > > > > > > > > Can we pull out the executor java options as a property in the > > > > amaterasu.properties? > > > > > > > > amaterasu.executor.extra.java.opts = "-Xmx1G -Dscala.usejavacp=true > > > > -Dhdp.version=2.6.5.0-292" > > > > > > > > > > > > As a side effect, we must provide the flexibility to allow quotes > > around > > > > the parameter but passing the quotes to the java command would > fail. I > > > > have stripped off the extra quotes in a dirty way at the moment. > Should > > > we > > > > consider proper command parsing (and possibly convert them to be bash > > > > compatible strings)? > > > > > > > > s"java -cp > > > > > > spark/jars/*:executor.jar:spark/conf/:${config.YARN.hadoopHomeDir}/conf/ > > > > " + > > > > s" ${config.amaterasuExecutorJavaOpts.replaceAll("\"","")} "+ > > > > > > > > > > > > Meanwhile, I'll also update by PR for Amaterasu-24 after pulling the > > > latest > > > > from the branch. > > > > > > > > Cheers, > > > > Arun > > > > > > > > On Wed, May 30, 2018 at 1:25 PM Arun Manivannan > > wrote: > > > > > > > > > Thanks a lot, Nadav. Will get home and spend some more time on > this. > > I > > > > was > > > > > in a rush and did this poor workaround. My VM is just 8 GB. > > > > > > > > > > Cheers > > > > > Arun > > > > > > > > > > > > > > > On Wed, May 30, 2018, 12:27 Nadav Har Tzvi > > > > > wrote: > > > > > > > > > >> Yaniv and I just tested it. It worked flawlessly on my end (HDP > > docker > > > > on > > > > >> AWS). Both Spark-Scala and PySpark. > > > > >> It worked on Yaniv's HDP cluster as well. > > > > >> Worth noting: > > > > >> 1. HDP 2.6.4 > > > > >> 2. Cluster has total of 32GB memory available > > > > >> 3. Each container is allocated 1G memory. > > > > >> 4. Amaterasu.properties: > > > > >> > > > > >> zk=sandbox-hdp.hortonworks.com > > > > >> version=0.2.0-incubating-rc3 > > > > >> master=192.168.33.11 > > > > >> user=root > > > > >> mode=yarn > > > > >> webserver.port=8000 > > > > >> webserver.root=dist > > > > >> spark.version=2.6.4.0-91 > > > > >> yarn.queue=default > > > > >> yarn.jarspath=hdfs:///apps/amaterasu > > > > >> spark.home=/usr/hdp/current/spark2-client > > > > >> > > > > >> > > > > > > > > > > #spark.home=/opt/cloudera/parcels/SPARK2-2.1.0.cloudera2-1.cdh5.7.0.p0.171658/lib/spark2 > > > > >> yarn.hadoop.home.dir=/etc/hadoop > > > > >> spark.opts.spark.yarn.am > > .extraJavaOptions="-Dhdp.version=2.6.4.0-91" > > > > >> > spark.opts.spark.driver.extraJavaOptions="-Dhdp.version=2.6.4.0-91" > > > > >> > > > > >> > > > > >> Arun, please share: > > > > >> 1. YARN memory configurations > > > > >> 2. amaterasu.properties content > > > > >> 3. HDP version. > > > > >> > > > > >> Cheers, > > > > >> Nadav > > > > >> > > > > >> > > > > >> On 30 May 2018 at 07:11, Arun Manivannan wrote: > > > > >> > > > > >> > The pmem disabling is just temporary. I'll do a detailed > analysis > > > and > > > > >> get > > > > >> > back with a proper solution. > > > > >> > > > > > >> > Any hints on this front is highly appreciated. > > > > >> > > > > > >> > Cheers > > > > >> > Arun > > > > >> > > > > > >> > On Wed, May 30, 2018, 01:10 Nadav Har Tzvi < > > nadavhart...@gmail.com> > > > > >> wrote: > > > > >> > > > > > >> > > Yaniv, Eyal, this might be related to the same issue you faced > > > with > > > > >> HDP. > > > > >> > > Can you confirm? > > > > >> > > > > > > >> > > On Tue, May 29, 2018, 17:58 Arun Manivannan > > > > wrote: > > > > >> > > > > > > >> > > > +1 from me > > > > >> > > > > > > > >> > > > Unit Tests and Build ran fine. > > > > >> > > > > > > > >> > > > Tested
Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3
+1 I checked the signature, the headers: it looks good. Thanks ! Regards JB On 02/06/2018 17:32, Arun Manivannan wrote: > Gentlemen, > > Apologies for coming back late. The issue was just the minimum container > size that was configured in my cluster > (yarn.scheduler.maximum-allocation-mb). It was set at 1 GB. > > I didn't specify any spark specific memory parameters during my run (the > memory defaults that the SparkSetupProvider was looking at) and to top it > the code was setting the Xmx at 1 GB causing the overallocation and > failure. > > I have one minor proposal. If this is agreeable, I can raise a quick PR. > > Can we pull out the executor java options as a property in the > amaterasu.properties? > > amaterasu.executor.extra.java.opts = "-Xmx1G -Dscala.usejavacp=true > -Dhdp.version=2.6.5.0-292" > > > As a side effect, we must provide the flexibility to allow quotes around > the parameter but passing the quotes to the java command would fail. I > have stripped off the extra quotes in a dirty way at the moment. Should we > consider proper command parsing (and possibly convert them to be bash > compatible strings)? > > s"java -cp > spark/jars/*:executor.jar:spark/conf/:${config.YARN.hadoopHomeDir}/conf/ > " + > s" ${config.amaterasuExecutorJavaOpts.replaceAll("\"","")} "+ > > > Meanwhile, I'll also update by PR for Amaterasu-24 after pulling the latest > from the branch. > > Cheers, > Arun > > On Wed, May 30, 2018 at 1:25 PM Arun Manivannan wrote: > >> Thanks a lot, Nadav. Will get home and spend some more time on this. I was >> in a rush and did this poor workaround. My VM is just 8 GB. >> >> Cheers >> Arun >> >> >> On Wed, May 30, 2018, 12:27 Nadav Har Tzvi wrote: >> >>> Yaniv and I just tested it. It worked flawlessly on my end (HDP docker on >>> AWS). Both Spark-Scala and PySpark. >>> It worked on Yaniv's HDP cluster as well. >>> Worth noting: >>> 1. HDP 2.6.4 >>> 2. Cluster has total of 32GB memory available >>> 3. Each container is allocated 1G memory. >>> 4. Amaterasu.properties: >>> >>> zk=sandbox-hdp.hortonworks.com >>> version=0.2.0-incubating-rc3 >>> master=192.168.33.11 >>> user=root >>> mode=yarn >>> webserver.port=8000 >>> webserver.root=dist >>> spark.version=2.6.4.0-91 >>> yarn.queue=default >>> yarn.jarspath=hdfs:///apps/amaterasu >>> spark.home=/usr/hdp/current/spark2-client >>> >>> #spark.home=/opt/cloudera/parcels/SPARK2-2.1.0.cloudera2-1.cdh5.7.0.p0.171658/lib/spark2 >>> yarn.hadoop.home.dir=/etc/hadoop >>> spark.opts.spark.yarn.am.extraJavaOptions="-Dhdp.version=2.6.4.0-91" >>> spark.opts.spark.driver.extraJavaOptions="-Dhdp.version=2.6.4.0-91" >>> >>> >>> Arun, please share: >>> 1. YARN memory configurations >>> 2. amaterasu.properties content >>> 3. HDP version. >>> >>> Cheers, >>> Nadav >>> >>> >>> On 30 May 2018 at 07:11, Arun Manivannan wrote: >>> The pmem disabling is just temporary. I'll do a detailed analysis and >>> get back with a proper solution. Any hints on this front is highly appreciated. Cheers Arun On Wed, May 30, 2018, 01:10 Nadav Har Tzvi >>> wrote: > Yaniv, Eyal, this might be related to the same issue you faced with >>> HDP. > Can you confirm? > > On Tue, May 29, 2018, 17:58 Arun Manivannan wrote: > >> +1 from me >> >> Unit Tests and Build ran fine. >> >> Tested on HDP (VM) but had trouble allocating containers (didn't >>> have > that >> before). Apparently Centos VMs are known to have this problem. Disabled >> physical memory check (yarn.nodemanager.pmem-check-enabled) and ran jobs >> successfully. >> >> >> >> >> >> On Tue, May 29, 2018 at 10:42 PM Kirupa Devarajan < > kirupagara...@gmail.com >>> >> wrote: >> >>> Unit tests passing and build was successful on the branch >>> "version-0.2.0-incubating-rc3" >>> >>> +1 from me >>> >>> Cheers, >>> Kirupa >>> >>> >>> On Tue, May 29, 2018 at 3:06 PM, guy peleg wrote: >>> +1 looks good to me On Tue, May 29, 2018, 14:39 Nadav Har Tzvi < >>> nadavhart...@gmail.com > >>> wrote: > +1 approve. Tested multiple times and after a long round of fixing >> and > testing over and over. > > Cheers, > Nadav > > > On 29 May 2018 at 07:38, Yaniv Rodenski >>> wrote: > >> Hi everyone, >> >> We have fixed the legal issues, as well as a bug found by @Nadav >>> please >> review and vote on the release candidate #3 for the version >> 0.2.0-incubating, as follows >> >> [ ] +1, Approve the release >> [ ] -1, Do not approve the release (please provide specific >> comments) >> >> The complete staging area is available for your review, >>> which >>> includes: >> >> * JIRA re