Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3
Unit tests passing and build was successful on the branch "version-0.2.0-incubating-rc3" +1 from me Cheers, Kirupa On Tue, May 29, 2018 at 3:06 PM, guy peleg wrote: > +1 looks good to me > > On Tue, May 29, 2018, 14:39 Nadav Har Tzvi wrote: > > > +1 approve. Tested multiple times and after a long round of fixing and > > testing over and over. > > > > Cheers, > > Nadav > > > > > > On 29 May 2018 at 07:38, Yaniv Rodenski wrote: > > > > > Hi everyone, > > > > > > We have fixed the legal issues, as well as a bug found by @Nadav please > > > review and vote on the release candidate #3 for the version > > > 0.2.0-incubating, as follows > > > > > > [ ] +1, Approve the release > > > [ ] -1, Do not approve the release (please provide specific comments) > > > > > > The complete staging area is available for your review, which includes: > > > > > > * JIRA release notes [1], > > > * the official Apache source release to be deployed to dist.apache.org > > > [2], > > > which is signed with the key with fingerprint [3], > > > * source code tag "version-0.2.0-incubating-rc3" [4], > > > * Java artifacts were built with Gradle 3.1 and OpenJDK/Oracle JDK > > > 1.8.0_151 > > > > > > The vote will be open for at least 72 hours. It is adopted by majority > > > approval, with at least 3 PMC affirmative votes. > > > > > > Thanks, > > > Yaniv > > > > > > [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa? > > > projectId=12321521&version=12342793 > > > [2] https://dist.apache.org/repos/dist/dev/incubator/amaterasu/ > 0.2.0rc3/ > > > [3] https://dist.apache.org/repos/dist/dev/incubator/amaterasu/KEYS > > > [4] https://github.com/apache/incubator-amaterasu/tags > > > > > >
Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3
+1 from me Unit Tests and Build ran fine. Tested on HDP (VM) but had trouble allocating containers (didn't have that before). Apparently Centos VMs are known to have this problem. Disabled physical memory check (yarn.nodemanager.pmem-check-enabled) and ran jobs successfully. On Tue, May 29, 2018 at 10:42 PM Kirupa Devarajan wrote: > Unit tests passing and build was successful on the branch > "version-0.2.0-incubating-rc3" > > +1 from me > > Cheers, > Kirupa > > > On Tue, May 29, 2018 at 3:06 PM, guy peleg wrote: > > > +1 looks good to me > > > > On Tue, May 29, 2018, 14:39 Nadav Har Tzvi > wrote: > > > > > +1 approve. Tested multiple times and after a long round of fixing and > > > testing over and over. > > > > > > Cheers, > > > Nadav > > > > > > > > > On 29 May 2018 at 07:38, Yaniv Rodenski wrote: > > > > > > > Hi everyone, > > > > > > > > We have fixed the legal issues, as well as a bug found by @Nadav > please > > > > review and vote on the release candidate #3 for the version > > > > 0.2.0-incubating, as follows > > > > > > > > [ ] +1, Approve the release > > > > [ ] -1, Do not approve the release (please provide specific comments) > > > > > > > > The complete staging area is available for your review, which > includes: > > > > > > > > * JIRA release notes [1], > > > > * the official Apache source release to be deployed to > dist.apache.org > > > > [2], > > > > which is signed with the key with fingerprint [3], > > > > * source code tag "version-0.2.0-incubating-rc3" [4], > > > > * Java artifacts were built with Gradle 3.1 and OpenJDK/Oracle JDK > > > > 1.8.0_151 > > > > > > > > The vote will be open for at least 72 hours. It is adopted by > majority > > > > approval, with at least 3 PMC affirmative votes. > > > > > > > > Thanks, > > > > Yaniv > > > > > > > > [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa? > > > > projectId=12321521&version=12342793 > > > > [2] https://dist.apache.org/repos/dist/dev/incubator/amaterasu/ > > 0.2.0rc3/ > > > > [3] https://dist.apache.org/repos/dist/dev/incubator/amaterasu/KEYS > > > > [4] https://github.com/apache/incubator-amaterasu/tags > > > > > > > > > >
Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3
Yaniv, Eyal, this might be related to the same issue you faced with HDP. Can you confirm? On Tue, May 29, 2018, 17:58 Arun Manivannan wrote: > +1 from me > > Unit Tests and Build ran fine. > > Tested on HDP (VM) but had trouble allocating containers (didn't have that > before). Apparently Centos VMs are known to have this problem. Disabled > physical memory check (yarn.nodemanager.pmem-check-enabled) and ran jobs > successfully. > > > > > > On Tue, May 29, 2018 at 10:42 PM Kirupa Devarajan > > wrote: > > > Unit tests passing and build was successful on the branch > > "version-0.2.0-incubating-rc3" > > > > +1 from me > > > > Cheers, > > Kirupa > > > > > > On Tue, May 29, 2018 at 3:06 PM, guy peleg wrote: > > > > > +1 looks good to me > > > > > > On Tue, May 29, 2018, 14:39 Nadav Har Tzvi > > wrote: > > > > > > > +1 approve. Tested multiple times and after a long round of fixing > and > > > > testing over and over. > > > > > > > > Cheers, > > > > Nadav > > > > > > > > > > > > On 29 May 2018 at 07:38, Yaniv Rodenski wrote: > > > > > > > > > Hi everyone, > > > > > > > > > > We have fixed the legal issues, as well as a bug found by @Nadav > > please > > > > > review and vote on the release candidate #3 for the version > > > > > 0.2.0-incubating, as follows > > > > > > > > > > [ ] +1, Approve the release > > > > > [ ] -1, Do not approve the release (please provide specific > comments) > > > > > > > > > > The complete staging area is available for your review, which > > includes: > > > > > > > > > > * JIRA release notes [1], > > > > > * the official Apache source release to be deployed to > > dist.apache.org > > > > > [2], > > > > > which is signed with the key with fingerprint [3], > > > > > * source code tag "version-0.2.0-incubating-rc3" [4], > > > > > * Java artifacts were built with Gradle 3.1 and OpenJDK/Oracle JDK > > > > > 1.8.0_151 > > > > > > > > > > The vote will be open for at least 72 hours. It is adopted by > > majority > > > > > approval, with at least 3 PMC affirmative votes. > > > > > > > > > > Thanks, > > > > > Yaniv > > > > > > > > > > [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa? > > > > > projectId=12321521&version=12342793 > > > > > [2] https://dist.apache.org/repos/dist/dev/incubator/amaterasu/ > > > 0.2.0rc3/ > > > > > [3] > https://dist.apache.org/repos/dist/dev/incubator/amaterasu/KEYS > > > > > [4] https://github.com/apache/incubator-amaterasu/tags > > > > > > > > > > > > > > >
Re: Speeding up Anaconda deployment on executors
Hi Nadav, First of all, a small correction, in Mesos Amaterasu chooses to reuse the same container if it makes sense (there is an existing container that makes sense to reuseon the node that gave us the offer, currently that's all of the time but will change shortly. With that in mind there are two solutions we should probably think of: - Utilizing docker as Docker is available from YARN 3.0 (docker cache is per node) - Pre-packaged pipelines. this was actually raised by Karel a long time ago. If we add a stage to build and package pipelines we can have the dependencies pre-packaged (as an option) with the pipeline and hance we save the double download. I think both are probebly options that are useful, and we should probably add them to the project's backlog, but would love to hear other opinions. Cheers, Yaniv On Mon, May 28, 2018 at 7:17 AM, Nadav Har Tzvi wrote: > Hey everyone, > > So we have this issue, Anaconda takes forever to deploy on the executors, > whether it is YARN or Mesos. > > Let's first discuss why is it like this right now. > > First, let's see for each platform, how Apache Amaterasu interacts with the > underlying platform, in regard to what smallest independent unit that is > awarded its own isolated execution environment. > > *Apache Mesos:* > In Apache Mesos, we get our own nifty set of instances and executors. An > instance obviously can host multiple executors. depending on its capacity. > Thus the smallest independent unit here is the executor itself. > > *Apache Hadoop YARN*: > On YARN, we have a similar set of resources, we have nodes, each node is a > host to containers. > > Great, so far it sounds similar, right? Here is where Apache Amaterasu > takes things a bit differently for each platform. > > In Apache Mesos, everything is run on the same executor, regardless of how > many actions the job has. So if the job has 20 actions, they will run > sequentially on the same executor, resulting in the smallest independent > unit being the job itself, as only the job deserves its own running > environment. > > On Hadoop, things are different, a lot. > To start, each action is treated by YARN as a different application, with > its own set of containers. This means that on YARN, action is the smallest > independent unit. > > So what's the problem actually? So the problem in general is that we cannot > rely on the existence of 3rd party utilities, libraries, you name it, on > the target execution environment. This forces us to bundle anything we need > along with the job execution process. > Anaconda is exactly such 3rd party utility that we desperately need in > order to run PySpark code that has dependencies on more than PySpark itself > and pure Python. (Pandas, numpy, sklearn, there are more than enough > examples out there) > We need to install Anaconda once for each execution environment. In Apache > Mesos our smallest reliable execution environment is the executor itself, > thus we need to install Anaconda once per job. > In YARN, our smallest execution environment is the container, hence we need > to install Anaconda over and over for each action. > This obviously poses a problem because of numerous reasons: > 1. While we can make an excuse in the first action that it is setup time, > it is obvious that for the second action we are wasting time, a lot. To > compare Mesos and YARN, starting the second action on Mesos is a matter of > seconds. In YARN it is measured in minutes. > 2. We do the same thing over and over again, even if we run on the same > machine. This makes no sense whatsoever! We are losing the ability to cache > things. So for example, if I need numpy and that takes about 20-30 seconds > to download and install, why do I need to install it from scratch over and > over again? > 3. It causes code reliability issues. If Miniconda isn't there and I need > to roll a PySpark job, I now have to setup guards and fallbacks and what > not? Even worse, I have to find weird tricks to even get access to the > Miniconda environment, and that is different on Mesos and YARN, so now I > have a jungle in the code! > 4. On YARN, PySpark runs on yet a different container! Guess what?! This > container has no access to miniconda! We currently use --py-files to send a > list of gazzilion packages. This is different in Mesos, where PySpark > itself runs in the same executor as the main Amaterasu process. > So guess what? I now have a jungle in my PySpark invocation code too! > > Also take a note that the current implementation for Python 3rd party > dependencies resolution is Anaconda, this gives us an isolated environment > that doesn't rely on the existing Python (cause maybe, for some reason, you > have Python 2.5 on your cluster, which is not supported by new versions of > data libraries such as pandas, numpy and so forth), in addition it gives us > the nifty Conda package manager. > However, it doesn't mean that it has to stay that way. If the need or > r
Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3
The pmem disabling is just temporary. I'll do a detailed analysis and get back with a proper solution. Any hints on this front is highly appreciated. Cheers Arun On Wed, May 30, 2018, 01:10 Nadav Har Tzvi wrote: > Yaniv, Eyal, this might be related to the same issue you faced with HDP. > Can you confirm? > > On Tue, May 29, 2018, 17:58 Arun Manivannan wrote: > > > +1 from me > > > > Unit Tests and Build ran fine. > > > > Tested on HDP (VM) but had trouble allocating containers (didn't have > that > > before). Apparently Centos VMs are known to have this problem. Disabled > > physical memory check (yarn.nodemanager.pmem-check-enabled) and ran jobs > > successfully. > > > > > > > > > > > > On Tue, May 29, 2018 at 10:42 PM Kirupa Devarajan < > kirupagara...@gmail.com > > > > > wrote: > > > > > Unit tests passing and build was successful on the branch > > > "version-0.2.0-incubating-rc3" > > > > > > +1 from me > > > > > > Cheers, > > > Kirupa > > > > > > > > > On Tue, May 29, 2018 at 3:06 PM, guy peleg wrote: > > > > > > > +1 looks good to me > > > > > > > > On Tue, May 29, 2018, 14:39 Nadav Har Tzvi > > > wrote: > > > > > > > > > +1 approve. Tested multiple times and after a long round of fixing > > and > > > > > testing over and over. > > > > > > > > > > Cheers, > > > > > Nadav > > > > > > > > > > > > > > > On 29 May 2018 at 07:38, Yaniv Rodenski wrote: > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > We have fixed the legal issues, as well as a bug found by @Nadav > > > please > > > > > > review and vote on the release candidate #3 for the version > > > > > > 0.2.0-incubating, as follows > > > > > > > > > > > > [ ] +1, Approve the release > > > > > > [ ] -1, Do not approve the release (please provide specific > > comments) > > > > > > > > > > > > The complete staging area is available for your review, which > > > includes: > > > > > > > > > > > > * JIRA release notes [1], > > > > > > * the official Apache source release to be deployed to > > > dist.apache.org > > > > > > [2], > > > > > > which is signed with the key with fingerprint [3], > > > > > > * source code tag "version-0.2.0-incubating-rc3" [4], > > > > > > * Java artifacts were built with Gradle 3.1 and OpenJDK/Oracle > JDK > > > > > > 1.8.0_151 > > > > > > > > > > > > The vote will be open for at least 72 hours. It is adopted by > > > majority > > > > > > approval, with at least 3 PMC affirmative votes. > > > > > > > > > > > > Thanks, > > > > > > Yaniv > > > > > > > > > > > > [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa? > > > > > > projectId=12321521&version=12342793 > > > > > > [2] https://dist.apache.org/repos/dist/dev/incubator/amaterasu/ > > > > 0.2.0rc3/ > > > > > > [3] > > https://dist.apache.org/repos/dist/dev/incubator/amaterasu/KEYS > > > > > > [4] https://github.com/apache/incubator-amaterasu/tags > > > > > > > > > > > > > > > > > > > > >
Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3
Yaniv and I just tested it. It worked flawlessly on my end (HDP docker on AWS). Both Spark-Scala and PySpark. It worked on Yaniv's HDP cluster as well. Worth noting: 1. HDP 2.6.4 2. Cluster has total of 32GB memory available 3. Each container is allocated 1G memory. 4. Amaterasu.properties: zk=sandbox-hdp.hortonworks.com version=0.2.0-incubating-rc3 master=192.168.33.11 user=root mode=yarn webserver.port=8000 webserver.root=dist spark.version=2.6.4.0-91 yarn.queue=default yarn.jarspath=hdfs:///apps/amaterasu spark.home=/usr/hdp/current/spark2-client #spark.home=/opt/cloudera/parcels/SPARK2-2.1.0.cloudera2-1.cdh5.7.0.p0.171658/lib/spark2 yarn.hadoop.home.dir=/etc/hadoop spark.opts.spark.yarn.am.extraJavaOptions="-Dhdp.version=2.6.4.0-91" spark.opts.spark.driver.extraJavaOptions="-Dhdp.version=2.6.4.0-91" Arun, please share: 1. YARN memory configurations 2. amaterasu.properties content 3. HDP version. Cheers, Nadav On 30 May 2018 at 07:11, Arun Manivannan wrote: > The pmem disabling is just temporary. I'll do a detailed analysis and get > back with a proper solution. > > Any hints on this front is highly appreciated. > > Cheers > Arun > > On Wed, May 30, 2018, 01:10 Nadav Har Tzvi wrote: > > > Yaniv, Eyal, this might be related to the same issue you faced with HDP. > > Can you confirm? > > > > On Tue, May 29, 2018, 17:58 Arun Manivannan wrote: > > > > > +1 from me > > > > > > Unit Tests and Build ran fine. > > > > > > Tested on HDP (VM) but had trouble allocating containers (didn't have > > that > > > before). Apparently Centos VMs are known to have this problem. > Disabled > > > physical memory check (yarn.nodemanager.pmem-check-enabled) and ran > jobs > > > successfully. > > > > > > > > > > > > > > > > > > On Tue, May 29, 2018 at 10:42 PM Kirupa Devarajan < > > kirupagara...@gmail.com > > > > > > > wrote: > > > > > > > Unit tests passing and build was successful on the branch > > > > "version-0.2.0-incubating-rc3" > > > > > > > > +1 from me > > > > > > > > Cheers, > > > > Kirupa > > > > > > > > > > > > On Tue, May 29, 2018 at 3:06 PM, guy peleg > wrote: > > > > > > > > > +1 looks good to me > > > > > > > > > > On Tue, May 29, 2018, 14:39 Nadav Har Tzvi > > > > > wrote: > > > > > > > > > > > +1 approve. Tested multiple times and after a long round of > fixing > > > and > > > > > > testing over and over. > > > > > > > > > > > > Cheers, > > > > > > Nadav > > > > > > > > > > > > > > > > > > On 29 May 2018 at 07:38, Yaniv Rodenski wrote: > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > We have fixed the legal issues, as well as a bug found by > @Nadav > > > > please > > > > > > > review and vote on the release candidate #3 for the version > > > > > > > 0.2.0-incubating, as follows > > > > > > > > > > > > > > [ ] +1, Approve the release > > > > > > > [ ] -1, Do not approve the release (please provide specific > > > comments) > > > > > > > > > > > > > > The complete staging area is available for your review, which > > > > includes: > > > > > > > > > > > > > > * JIRA release notes [1], > > > > > > > * the official Apache source release to be deployed to > > > > dist.apache.org > > > > > > > [2], > > > > > > > which is signed with the key with fingerprint [3], > > > > > > > * source code tag "version-0.2.0-incubating-rc3" [4], > > > > > > > * Java artifacts were built with Gradle 3.1 and OpenJDK/Oracle > > JDK > > > > > > > 1.8.0_151 > > > > > > > > > > > > > > The vote will be open for at least 72 hours. It is adopted by > > > > majority > > > > > > > approval, with at least 3 PMC affirmative votes. > > > > > > > > > > > > > > Thanks, > > > > > > > Yaniv > > > > > > > > > > > > > > [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa? > > > > > > > projectId=12321521&version=12342793 > > > > > > > [2] https://dist.apache.org/repos/ > dist/dev/incubator/amaterasu/ > > > > > 0.2.0rc3/ > > > > > > > [3] > > > https://dist.apache.org/repos/dist/dev/incubator/amaterasu/KEYS > > > > > > > [4] https://github.com/apache/incubator-amaterasu/tags > > > > > > > > > > > > > > > > > > > > > > > > > > > >
Re: [VOTE] Amaterasu release 0.2.0-incubating, release candidate #3
Thanks a lot, Nadav. Will get home and spend some more time on this. I was in a rush and did this poor workaround. My VM is just 8 GB. Cheers Arun On Wed, May 30, 2018, 12:27 Nadav Har Tzvi wrote: > Yaniv and I just tested it. It worked flawlessly on my end (HDP docker on > AWS). Both Spark-Scala and PySpark. > It worked on Yaniv's HDP cluster as well. > Worth noting: > 1. HDP 2.6.4 > 2. Cluster has total of 32GB memory available > 3. Each container is allocated 1G memory. > 4. Amaterasu.properties: > > zk=sandbox-hdp.hortonworks.com > version=0.2.0-incubating-rc3 > master=192.168.33.11 > user=root > mode=yarn > webserver.port=8000 > webserver.root=dist > spark.version=2.6.4.0-91 > yarn.queue=default > yarn.jarspath=hdfs:///apps/amaterasu > spark.home=/usr/hdp/current/spark2-client > > #spark.home=/opt/cloudera/parcels/SPARK2-2.1.0.cloudera2-1.cdh5.7.0.p0.171658/lib/spark2 > yarn.hadoop.home.dir=/etc/hadoop > spark.opts.spark.yarn.am.extraJavaOptions="-Dhdp.version=2.6.4.0-91" > spark.opts.spark.driver.extraJavaOptions="-Dhdp.version=2.6.4.0-91" > > > Arun, please share: > 1. YARN memory configurations > 2. amaterasu.properties content > 3. HDP version. > > Cheers, > Nadav > > > On 30 May 2018 at 07:11, Arun Manivannan wrote: > > > The pmem disabling is just temporary. I'll do a detailed analysis and get > > back with a proper solution. > > > > Any hints on this front is highly appreciated. > > > > Cheers > > Arun > > > > On Wed, May 30, 2018, 01:10 Nadav Har Tzvi > wrote: > > > > > Yaniv, Eyal, this might be related to the same issue you faced with > HDP. > > > Can you confirm? > > > > > > On Tue, May 29, 2018, 17:58 Arun Manivannan wrote: > > > > > > > +1 from me > > > > > > > > Unit Tests and Build ran fine. > > > > > > > > Tested on HDP (VM) but had trouble allocating containers (didn't have > > > that > > > > before). Apparently Centos VMs are known to have this problem. > > Disabled > > > > physical memory check (yarn.nodemanager.pmem-check-enabled) and ran > > jobs > > > > successfully. > > > > > > > > > > > > > > > > > > > > > > > > On Tue, May 29, 2018 at 10:42 PM Kirupa Devarajan < > > > kirupagara...@gmail.com > > > > > > > > > wrote: > > > > > > > > > Unit tests passing and build was successful on the branch > > > > > "version-0.2.0-incubating-rc3" > > > > > > > > > > +1 from me > > > > > > > > > > Cheers, > > > > > Kirupa > > > > > > > > > > > > > > > On Tue, May 29, 2018 at 3:06 PM, guy peleg > > wrote: > > > > > > > > > > > +1 looks good to me > > > > > > > > > > > > On Tue, May 29, 2018, 14:39 Nadav Har Tzvi < > nadavhart...@gmail.com > > > > > > > > wrote: > > > > > > > > > > > > > +1 approve. Tested multiple times and after a long round of > > fixing > > > > and > > > > > > > testing over and over. > > > > > > > > > > > > > > Cheers, > > > > > > > Nadav > > > > > > > > > > > > > > > > > > > > > On 29 May 2018 at 07:38, Yaniv Rodenski > wrote: > > > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > > > We have fixed the legal issues, as well as a bug found by > > @Nadav > > > > > please > > > > > > > > review and vote on the release candidate #3 for the version > > > > > > > > 0.2.0-incubating, as follows > > > > > > > > > > > > > > > > [ ] +1, Approve the release > > > > > > > > [ ] -1, Do not approve the release (please provide specific > > > > comments) > > > > > > > > > > > > > > > > The complete staging area is available for your review, which > > > > > includes: > > > > > > > > > > > > > > > > * JIRA release notes [1], > > > > > > > > * the official Apache source release to be deployed to > > > > > dist.apache.org > > > > > > > > [2], > > > > > > > > which is signed with the key with fingerprint [3], > > > > > > > > * source code tag "version-0.2.0-incubating-rc3" [4], > > > > > > > > * Java artifacts were built with Gradle 3.1 and > OpenJDK/Oracle > > > JDK > > > > > > > > 1.8.0_151 > > > > > > > > > > > > > > > > The vote will be open for at least 72 hours. It is adopted by > > > > > majority > > > > > > > > approval, with at least 3 PMC affirmative votes. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Yaniv > > > > > > > > > > > > > > > > [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa? > > > > > > > > projectId=12321521&version=12342793 > > > > > > > > [2] https://dist.apache.org/repos/ > > dist/dev/incubator/amaterasu/ > > > > > > 0.2.0rc3/ > > > > > > > > [3] > > > > https://dist.apache.org/repos/dist/dev/incubator/amaterasu/KEYS > > > > > > > > [4] https://github.com/apache/incubator-amaterasu/tags > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
