JIRA ticket # 239 On Wed, Aug 17, 2016 at 2:21 PM, John Yost <hokiege...@gmail.com> wrote:
> Hi Guys, > > Sorry, just checked email, setting role to * does indeed cause this > error. Matt -> if you set it to nothing or whatever role you have your > mesos slaves configured as, that will fix this. My apologies for failing to > log and fix this. Darin -> I will enter a JIRA ticket for this. > > To echo Darin's sentiments--thanks a bunch for checking Myriad out! :) > > --John > > On Wed, Aug 17, 2016 at 1:25 PM, Darin Johnson <dbjohnson1...@gmail.com> > wrote: > >> Hey Matt, >> >> Looking through the code, I think setting myriadFrameworkRole to "*" might >> be the problem. Can you try commenting out that line in your config? >> I'll >> double check this in a little while too. If that works I'll submit a >> patch >> that checks that. >> >> Sorry - Myriad is still a pretty young project! Thanks for checking it >> out >> though! >> >> Darin >> >> On Wed, Aug 17, 2016 at 11:25 AM, Matthew J. Loppatto < >> mloppa...@keywcorp.com> wrote: >> >> > Hey Darin, >> > >> > Pulling from master got rid of the errors I was seeing, however I'm >> > running into a new issue. After starting the resource manager, I see >> this >> > in the logs: >> > >> > 2016-08-17 10:56:40,709 INFO org.apache.myriad.Main: Launching 1 NM(s) >> > with profile medium >> > 2016-08-17 10:56:40,710 INFO org.apache.myriad.scheduler.My >> riadOperations: >> > Adding 1 NM instances to cluster >> > 2016-08-17 10:56:40,733 ERROR org.apache.myriad.scheduler.ev >> ent.handlers.ErrorEventHandler: >> > Role '' is not present in the master's --roles >> > >> > My Mesos cluster has the default "*" role so I tried setting >> > frameworkRole: "*" in myriad-config-default.yml, restarted the resource >> > manager and got this error: >> > >> > 2016-08-17 11:06:28,244 ERROR org.apache.myriad.scheduler.ev >> ent.handlers.ResourceOffersEventHandler: >> > Exception thrown while trying to create a task for nm >> > java.lang.IllegalArgumentException: n must be positive >> > at java.util.Random.nextInt(Random.java:300) >> > at org.apache.myriad.scheduler.resource.RangeResource. >> > getRandomValues(RangeResource.java:128) >> > at org.apache.myriad.scheduler.resource.RangeResource. >> > consumeResource(RangeResource.java:99) >> > at org.apache.myriad.scheduler.resource.ResourceOfferContainer. >> > consumePorts(ResourceOfferContainer.java:171) >> > at org.apache.myriad.scheduler.NMTaskFactory.createTask( >> > NMTaskFactory.java:45) >> > at org.apache.myriad.scheduler.event.handlers. >> > ResourceOffersEventHandler.onEvent(ResourceOffersEventHandler.java:119) >> > at org.apache.myriad.scheduler.event.handlers. >> > ResourceOffersEventHandler.onEvent(ResourceOffersEventHandler.java:49) >> > at com.lmax.disruptor.BatchEventProcessor.run( >> > BatchEventProcessor.java:128) >> > at java.util.concurrent.ThreadPoolExecutor.runWorker( >> > ThreadPoolExecutor.java:1145) >> > at java.util.concurrent.ThreadPoolExecutor$Worker.run( >> > ThreadPoolExecutor.java:615) >> > at java.lang.Thread.run(Thread.java:745) >> > >> > Does Myriad require its own role in Mesos? >> > >> > Thanks, >> > Matt >> > >> > >> > -----Original Message----- >> > From: Darin Johnson [mailto:dbjohnson1...@gmail.com] >> > Sent: Tuesday, August 16, 2016 6:18 PM >> > To: Dev >> > Subject: Re: Resource manager error >> > >> > Hey Mathew, my coworker found the same issue recently, I fixed it on my >> > last pull request, if you'd like to pull from master. >> > >> > Alternatively, you could comment out the appendCgroups line in >> > myriad-scheduler <https://urldefense.proofpoint.com/v2/url?u=https- >> > 3A__github.com_apache_incubator-2Dmyriad_tree_0.2.x_ >> > myriad-2Dscheduler&d=CwIFaQ&c=31nHN1tvZeuWBT6LwDN4Ngk1qezfsY >> > HyolgGeY2ZhlU&r=D2bc6ANY3sIFSxaGDnPt52V5PqDlabKIPtzNhAIfJCs&m= >> > ibxhOZQSsKtyVi5iruY8ImkW7bQ8zOrHcuDTLL7GBwA&s=5zc_OUK_ >> > qwnVQoC2kVCcAgvb4ZmZrVKF-iHca_dif4Y&e= >/src <https://urldefense. >> > proofpoint.com/v2/url?u=https-3A__github.com_apache_ >> > incubator-2Dmyriad_tree_0.2.x_myriad-2Dscheduler_src&d=CwIFaQ&c= >> > 31nHN1tvZeuWBT6LwDN4Ngk1qezfsYHyolgGeY2ZhlU&r= >> > D2bc6ANY3sIFSxaGDnPt52V5PqDlabKIPtzNhAIfJCs&m= >> > ibxhOZQSsKtyVi5iruY8ImkW7bQ8zOrHcuDTLL7GBwA&s= >> > 6CJWEHP2t7cY2oTmNz9Aq9AV39VEkUOKpMoRuz1q9nY&e= > /main < >> > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_ >> > incubator-2Dmyriad_tree_0.2.x_myriad-2Dscheduler_src_main&d=CwIFaQ&c= >> > 31nHN1tvZeuWBT6LwDN4Ngk1qezfsYHyolgGeY2ZhlU&r= >> > D2bc6ANY3sIFSxaGDnPt52V5PqDlabKIPtzNhAIfJCs&m= >> > ibxhOZQSsKtyVi5iruY8ImkW7bQ8zOrHcuDTLL7GBwA&s= >> > BtYphvuvvNZ5owUTfiRd4hW90jq0Ib8GGtKiHU0fTB4&e= > /java < >> > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_ >> > incubator-2Dmyriad_tree_0.2.x_myriad-2Dscheduler_src_main_ja >> va&d=CwIFaQ&c= >> > 31nHN1tvZeuWBT6LwDN4Ngk1qezfsYHyolgGeY2ZhlU&r= >> > D2bc6ANY3sIFSxaGDnPt52V5PqDlabKIPtzNhAIfJCs&m= >> > ibxhOZQSsKtyVi5iruY8ImkW7bQ8zOrHcuDTLL7GBwA&s=5o45_ >> > 8zSN96rSaQJ8oCWfhCvmqhSbLpz9fMnV9Fk4WI&e= > /org <https://urldefense. >> > proofpoint.com/v2/url?u=https-3A__github.com_apache_ >> > incubator-2Dmyriad_tree_0.2.x_myriad-2Dscheduler_src_main_ >> > java_org&d=CwIFaQ&c=31nHN1tvZeuWBT6LwDN4Ngk1qezfsYHyolgGeY2ZhlU&r= >> > D2bc6ANY3sIFSxaGDnPt52V5PqDlabKIPtzNhAIfJCs&m= >> > ibxhOZQSsKtyVi5iruY8ImkW7bQ8zOrHcuDTLL7GBwA&s= >> > KUTlDXsl6Okj5nCNNyCqnHcE2ePwEEOsYkf2ASzQP2Y&e= > /apache < >> > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_ >> > incubator-2Dmyriad_tree_0.2.x_myriad-2Dscheduler_src_main_ >> > java_org_apache&d=CwIFaQ&c=31nHN1tvZeuWBT6LwDN4Ngk1qezfsYHyo >> lgGeY2ZhlU&r= >> > D2bc6ANY3sIFSxaGDnPt52V5PqDlabKIPtzNhAIfJCs&m= >> > ibxhOZQSsKtyVi5iruY8ImkW7bQ8zOrHcuDTLL7GBwA&s=465af32H1JItcea_ >> > tp5hz7zxwpqWgAqbVA8APaWmSUE&e= > /myriad <https://urldefense. >> > proofpoint.com/v2/url?u=https-3A__github.com_apache_ >> > incubator-2Dmyriad_tree_0.2.x_myriad-2Dscheduler_src_main_ >> > java_org_apache_myriad&d=CwIFaQ&c=31nHN1tvZeuWBT6LwDN4Ngk1qezfsY >> > HyolgGeY2ZhlU&r=D2bc6ANY3sIFSxaGDnPt52V5PqDlabKIPtzNhAIfJCs&m= >> > ibxhOZQSsKtyVi5iruY8ImkW7bQ8zOrHcuDTLL7GBwA&s=QEZka3G6qwIsYzvPtXx- >> > w4uVek0Bt2D3bD4M4160Dnk&e= > /scheduler <https://urldefense. >> > proofpoint.com/v2/url?u=https-3A__github.com_apache_ >> > incubator-2Dmyriad_tree_0.2.x_myriad-2Dscheduler_src_main_ >> > java_org_apache_myriad_scheduler&d=CwIFaQ&c=31nHN1tvZeuWBT6L >> wDN4Ngk1qezfsY >> > HyolgGeY2ZhlU&r=D2bc6ANY3sIFSxaGDnPt52V5PqDlabKIPtzNhAIfJCs&m= >> > ibxhOZQSsKtyVi5iruY8ImkW7bQ8zOrHcuDTLL7GBwA&s= >> > 2EubhJ2JLwuGjY6DBZXpauvyuXJ0xefgOFHC8lEo5JE&e= > >> > /*NMExecutorCLGenImpl* and rebuild. >> > >> > Sorry that missed my QA unfortunately I'm always using cgroups and >> didn't >> > test that. We may do a 0.2.1 release but I can say when. >> > >> > Darin >> > >> > On Aug 16, 2016 8:49 AM, "Matthew J. Loppatto" <mloppa...@keywcorp.com> >> > wrote: >> > >> > > Hi, >> > > >> > > >> > > >> > > I’m setting up Myriad 0.2.0 on my Mesos cluster following this guide: >> > > https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.a >> pache.org_ >> > > confluence_display_MYRIAD_&d=CwIFaQ&c=31nHN1tvZeuWBT6LwDN4Ng >> k1qezfsYHy >> > > olgGeY2ZhlU&r=D2bc6ANY3sIFSxaGDnPt52V5PqDlabKIPtzNhAIfJCs&m= >> ibxhOZQSsK >> > > tyVi5iruY8ImkW7bQ8zOrHcuDTLL7GBwA&s=LaQad9p3ZI3Rt5cTn3kHAb58 >> BuSD5etwIm >> > > WZHzFz6Sk&e= >> > > Installing+for+Developers >> > > >> > > >> > > >> > > And I get the following error in the resource manager executor log in >> > > mesos after starting it with `/opt/hadoop-2.7.2/bin/yarn >> > resourcemanager`: >> > > >> > > >> > > >> > > chown: cannot access ‘/sys/fs/cgroup/cpu/mesos/f5d6 >> c530-c13d-4b1d-bc30- >> > f298affb6442’: >> > > No such file or directory >> > > >> > > env: /bin/yarn: No such file or directory >> > > >> > > ory >> > > >> > > >> > > >> > > It appears the ‘mesos’ directory doesn’t exist under >> /sys/fs/cgroup/cpu. >> > > Any ideas what the issue could be? >> > > >> > > >> > > >> > > This is my yarn-site.xml: >> > > >> > > >> > > >> > > <configuration> >> > > >> > > <!-- Site-specific YARN configuration properties --> >> > > >> > > <property> >> > > >> > > <name>yarn.nodemanager.aux-services</name> >> > > >> > > <value>mapreduce_shuffle,myriad_executor</value> >> > > >> > > <!-- If using MapR distro, please use the following value: >> > > >> > > >> > > <value>mapreduce_shuffle,mapr_direct_shuffle,myriad_executor</value> >> > > --> >> > > >> > > </property> >> > > >> > > <property> >> > > >> > > >> > > <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name> >> > > >> > > <value>org.apache.hadoop.mapred.ShuffleHandler</value> >> > > >> > > </property> >> > > >> > > <property> >> > > >> > > >> > > <name>yarn.nodemanager.aux-services.myriad_executor.class</name> >> > > >> > > >> > > <value>org.apache.myriad.executor.MyriadExecutorAuxService</value> >> > > >> > > </property> >> > > >> > > <property> >> > > >> > > <name>yarn.nm.liveness-monitor.expiry-interval-ms</name> >> > > >> > > <value>2000</value> >> > > >> > > </property> >> > > >> > > <property> >> > > >> > > <name>yarn.am.liveness-monitor.expiry-interval-ms</name> >> > > >> > > <value>10000</value> >> > > >> > > </property> >> > > >> > > <property> >> > > >> > > >> > > <name>yarn.resourcemanager.nm.liveness-monitor.interval-ms</name> >> > > >> > > <value>1000</value> >> > > >> > > </property> >> > > >> > > <!-- Needed for Fine Grain Scaling --> >> > > >> > > <property> >> > > >> > > <name>yarn.scheduler.minimum-allocation-vcores</name> >> > > >> > > <value>0</value> >> > > >> > > </property> >> > > >> > > <property> >> > > >> > > <name>yarn.scheduler.minimum-allocation-mb</name> >> > > >> > > <value>0</value> >> > > >> > > </property> >> > > >> > > <!-- Site specific YARN configuration properties --> >> > > >> > > <property> >> > > >> > > <name>yarn.nodemanager.resource.cpu-vcores</name> >> > > >> > > <value>${nodemanager.resource.cpu-vcores}</value> >> > > >> > > </property> >> > > >> > > <property> >> > > >> > > <name>yarn.nodemanager.resource.memory-mb</name> >> > > >> > > <value>${nodemanager.resource.memory-mb}</value> >> > > >> > > </property> >> > > >> > > <!--These options enable dynamic port assignment by mesos --> >> > > >> > > <property> >> > > >> > > <name>yarn.nodemanager.address</name> >> > > >> > > <value>${myriad.yarn.nodemanager.address}</value> >> > > >> > > </property> >> > > >> > > <property> >> > > >> > > <name>yarn.nodemanager.webapp.address</name> >> > > >> > > <value>${myriad.yarn.nodemanager.webapp.address}</value> >> > > >> > > </property> >> > > >> > > <property> >> > > >> > > <name>yarn.nodemanager.webapp.https.address</name> >> > > >> > > <value>${myriad.yarn.nodemanager.webapp.address}</value> >> > > >> > > </property> >> > > >> > > <property> >> > > >> > > <name>yarn.nodemanager.localizer.address</name> >> > > >> > > <value>${myriad.yarn.nodemanager.localizer.address}</value> >> > > >> > > </property> >> > > >> > > <!-- Configure Myriad Scheduler here --> >> > > >> > > <property> >> > > >> > > <name>yarn.resourcemanager.scheduler.class</name> >> > > >> > > <value>org.apache.myriad.scheduler.yarn.MyriadFairScheduler< >> /value> >> > > >> > > <description>One can configure other scehdulers as well from >> > > following >> > > list: org.apache.myriad.scheduler.yarn.MyriadCapacityScheduler, >> > > org.apache.myriad.scheduler.yarn.MyriadFifoScheduler</description> >> > > >> > > </property> >> > > >> > > <!-- Disable PMem/VMem checks for Hadoop 2.7.2 --> >> > > >> > > <property> >> > > >> > > <name>yarn.nodemanager.pmem-check-enabled</name> >> > > >> > > <value>false</value> >> > > >> > > </property> >> > > >> > > <property> >> > > >> > > <name>yarn.nodemanager.vmem-check-enabled</name> >> > > >> > > <value>false</value> >> > > >> > > </property> >> > > >> > > </configuration> >> > > >> > > >> > > >> > > >> > > >> > > My myriad-config-default.yml: >> > > >> > > >> > > >> > > mesosMaster: zk://myip:2181/mesos >> > > >> > > checkpoint: false >> > > >> > > frameworkFailoverTimeout: 43200000 >> > > >> > > frameworkName: MyriadAlpha >> > > >> > > frameworkRole: >> > > >> > > frameworkUser: root # User the Node Manager runs as, required if >> > > nodeManagerURI set, otherwise defaults to the user >> > > >> > > # running the resource manager. >> > > >> > > frameworkSuperUser: root # To be depricated, currently permissions >> > > need set by a superuser due to Mesos-1790. Must be >> > > >> > > # root or have passwordless sudo. Required if >> > > nodeManagerURI set, ignored otherwise. >> > > >> > > nativeLibrary: /usr/local/lib/libmesos.so >> > > >> > > zkServers: myip:2181 >> > > >> > > zkTimeout: 20000 >> > > >> > > restApiPort: 8192 >> > > >> > > servedConfigPath: dist/config.tgz >> > > >> > > servedBinaryPath: dist/binary.tgz >> > > >> > > profiles: >> > > >> > > zero: # NMs launched with this profile dynamically obtain cpu/mem >> > > from Mesos >> > > >> > > cpu: 0 >> > > >> > > mem: 0 >> > > >> > > small: >> > > >> > > cpu: 2 >> > > >> > > mem: 2048 >> > > >> > > medium: >> > > >> > > cpu: 4 >> > > >> > > mem: 4096 >> > > >> > > large: >> > > >> > > cpu: 10 >> > > >> > > mem: 12288 >> > > >> > > nmInstances: # NMs to start with. Requires at least 1 NM with a >> > > non-zero profile. >> > > >> > > medium: 1 # <profile_name : instances> >> > > >> > > rebalancer: false >> > > >> > > haEnabled: false >> > > >> > > nodemanager: >> > > >> > > jvmMaxMemoryMB: 1024 >> > > >> > > cpus: 0.2 >> > > >> > > cgroups: false >> > > >> > > executor: >> > > >> > > jvmMaxMemoryMB: 256 >> > > >> > > path: >> > > file:///usr/local/libexec/mesos/myriad-executor-runnable-0.1.0.jar >> > > >> > > #The following should be used for a remotely distributed URI, hdfs >> > > assumed but other URI types valid. >> > > >> > > #nodeManagerUri: hdfs://namenode:port/dist/hadoop-2.7.0.tar.gz >> > > >> > > #configUri: >> > > https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0. >> 1_api_arif >> > > acts_config.tgz&d=CwIFaQ&c=31nHN1tvZeuWBT6LwDN4Ngk1qezfsYHyo >> lgGeY2ZhlU >> > > &r=D2bc6ANY3sIFSxaGDnPt52V5PqDlabKIPtzNhAIfJCs&m=ibxhOZQSsKt >> yVi5iruY8I >> > > mkW7bQ8zOrHcuDTLL7GBwA&s=IpOqhUOtwJsdorbAOeoY7GgHalMJ1s9EUju >> RUfRsmew&e >> > > = >> > > >> > > #jvmUri: >> > > https://urldefense.proofpoint.com/v2/url?u=https-3A__downloa >> ds.mycompa >> > > ny.com_java_jre-2D7u76-2Dlinux-2Dx64.tar.gz&d=CwIFaQ&c= >> 31nHN1tvZeuWBT6 >> > > LwDN4Ngk1qezfsYHyolgGeY2ZhlU&r=D2bc6ANY3sIFSxaGDnPt52V5PqDla >> bKIPtzNhAI >> > > fJCs&m=ibxhOZQSsKtyVi5iruY8ImkW7bQ8zOrHcuDTLL7GBwA&s=jPB2677 >> RH3k3CLsgl >> > > 4Zj3tGawuCLVB1a2WXBUOWEelU&e= >> > > >> > > yarnEnvironment: >> > > >> > > YARN_HOME: /opt/hadoop-2.7.2 >> > > >> > > >> > > >> > > >> > > >> > > Thanks! >> > > >> > > Matt >> > > >> > >> > >
