Hey Darin, Commenting out myriadFrameworkRole got rid of the log message about the missing role, but I'm still seeing the "n must be positive" exception.
The only other thing of interest I see in the log is WARN fair.AllocationFileLoaderService: fair-scheduler.xml not found on the classpath. Not sure if that is causing any issue though. Matt -----Original Message----- From: Darin Johnson [mailto:dbjohnson1...@gmail.com] Sent: Wednesday, August 17, 2016 1:26 PM To: Dev Subject: Re: Resource manager error Hey Matt, Looking through the code, I think setting myriadFrameworkRole to "*" might be the problem. Can you try commenting out that line in your config? I'll double check this in a little while too. If that works I'll submit a patch that checks that. Sorry - Myriad is still a pretty young project! Thanks for checking it out though! Darin On Wed, Aug 17, 2016 at 11:25 AM, Matthew J. Loppatto < mloppa...@keywcorp.com> wrote: > Hey Darin, > > Pulling from master got rid of the errors I was seeing, however I'm > running into a new issue. After starting the resource manager, I see > this in the logs: > > 2016-08-17 10:56:40,709 INFO org.apache.myriad.Main: Launching 1 NM(s) > with profile medium > 2016-08-17 10:56:40,710 INFO org.apache.myriad.scheduler.MyriadOperations: > Adding 1 NM instances to cluster > 2016-08-17 10:56:40,733 ERROR > org.apache.myriad.scheduler.event.handlers.ErrorEventHandler: > Role '' is not present in the master's --roles > > My Mesos cluster has the default "*" role so I tried setting > frameworkRole: "*" in myriad-config-default.yml, restarted the > resource manager and got this error: > > 2016-08-17 11:06:28,244 ERROR > org.apache.myriad.scheduler.event.handlers.ResourceOffersEventHandler: > Exception thrown while trying to create a task for nm > java.lang.IllegalArgumentException: n must be positive > at java.util.Random.nextInt(Random.java:300) > at org.apache.myriad.scheduler.resource.RangeResource. > getRandomValues(RangeResource.java:128) > at org.apache.myriad.scheduler.resource.RangeResource. > consumeResource(RangeResource.java:99) > at org.apache.myriad.scheduler.resource.ResourceOfferContainer. > consumePorts(ResourceOfferContainer.java:171) > at org.apache.myriad.scheduler.NMTaskFactory.createTask( > NMTaskFactory.java:45) > at org.apache.myriad.scheduler.event.handlers. > ResourceOffersEventHandler.onEvent(ResourceOffersEventHandler.java:119) > at org.apache.myriad.scheduler.event.handlers. > ResourceOffersEventHandler.onEvent(ResourceOffersEventHandler.java:49) > at com.lmax.disruptor.BatchEventProcessor.run( > BatchEventProcessor.java:128) > at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > > Does Myriad require its own role in Mesos? > > Thanks, > Matt > > > -----Original Message----- > From: Darin Johnson [mailto:dbjohnson1...@gmail.com] > Sent: Tuesday, August 16, 2016 6:18 PM > To: Dev > Subject: Re: Resource manager error > > Hey Mathew, my coworker found the same issue recently, I fixed it on > my last pull request, if you'd like to pull from master. > > Alternatively, you could comment out the appendCgroups line in > myriad-scheduler <https://urldefense.proofpoint.com/v2/url?u=https- > 3A__github.com_apache_incubator-2Dmyriad_tree_0.2.x_ > myriad-2Dscheduler&d=CwIFaQ&c=31nHN1tvZeuWBT6LwDN4Ngk1qezfsY > HyolgGeY2ZhlU&r=D2bc6ANY3sIFSxaGDnPt52V5PqDlabKIPtzNhAIfJCs&m= > ibxhOZQSsKtyVi5iruY8ImkW7bQ8zOrHcuDTLL7GBwA&s=5zc_OUK_ > qwnVQoC2kVCcAgvb4ZmZrVKF-iHca_dif4Y&e= >/src > <https://urldefense.proofpoint.com/v2/url?u=https-3A__urldefense&d=CwI > FaQ&c=31nHN1tvZeuWBT6LwDN4Ngk1qezfsYHyolgGeY2ZhlU&r=D2bc6ANY3sIFSxaGDn > Pt52V5PqDlabKIPtzNhAIfJCs&m=sX9u4FJdfE4P4b24cRwGMPuyeT4XkeQRP5t8wCZTjV > w&s=DeQq-jARIja9dGEYfjeIQMd6jGkf_tNUyvQn7PIMieU&e= .> > proofpoint.com/v2/url?u=https-3A__github.com_apache_ > incubator-2Dmyriad_tree_0.2.x_myriad-2Dscheduler_src&d=CwIFaQ&c= > 31nHN1tvZeuWBT6LwDN4Ngk1qezfsYHyolgGeY2ZhlU&r= > D2bc6ANY3sIFSxaGDnPt52V5PqDlabKIPtzNhAIfJCs&m= > ibxhOZQSsKtyVi5iruY8ImkW7bQ8zOrHcuDTLL7GBwA&s= > 6CJWEHP2t7cY2oTmNz9Aq9AV39VEkUOKpMoRuz1q9nY&e= > /main < > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache > _ > incubator-2Dmyriad_tree_0.2.x_myriad-2Dscheduler_src_main&d=CwIFaQ&c= > 31nHN1tvZeuWBT6LwDN4Ngk1qezfsYHyolgGeY2ZhlU&r= > D2bc6ANY3sIFSxaGDnPt52V5PqDlabKIPtzNhAIfJCs&m= > ibxhOZQSsKtyVi5iruY8ImkW7bQ8zOrHcuDTLL7GBwA&s= > BtYphvuvvNZ5owUTfiRd4hW90jq0Ib8GGtKiHU0fTB4&e= > /java < > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache > _ > incubator-2Dmyriad_tree_0.2.x_myriad-2Dscheduler_src_main_java&d=CwIFa > Q&c= 31nHN1tvZeuWBT6LwDN4Ngk1qezfsYHyolgGeY2ZhlU&r= > D2bc6ANY3sIFSxaGDnPt52V5PqDlabKIPtzNhAIfJCs&m= > ibxhOZQSsKtyVi5iruY8ImkW7bQ8zOrHcuDTLL7GBwA&s=5o45_ > 8zSN96rSaQJ8oCWfhCvmqhSbLpz9fMnV9Fk4WI&e= > /org > <https://urldefense.proofpoint.com/v2/url?u=https-3A__urldefense&d=CwI > FaQ&c=31nHN1tvZeuWBT6LwDN4Ngk1qezfsYHyolgGeY2ZhlU&r=D2bc6ANY3sIFSxaGDn > Pt52V5PqDlabKIPtzNhAIfJCs&m=sX9u4FJdfE4P4b24cRwGMPuyeT4XkeQRP5t8wCZTjV > w&s=DeQq-jARIja9dGEYfjeIQMd6jGkf_tNUyvQn7PIMieU&e= .> > proofpoint.com/v2/url?u=https-3A__github.com_apache_ > incubator-2Dmyriad_tree_0.2.x_myriad-2Dscheduler_src_main_ > java_org&d=CwIFaQ&c=31nHN1tvZeuWBT6LwDN4Ngk1qezfsYHyolgGeY2ZhlU&r= > D2bc6ANY3sIFSxaGDnPt52V5PqDlabKIPtzNhAIfJCs&m= > ibxhOZQSsKtyVi5iruY8ImkW7bQ8zOrHcuDTLL7GBwA&s= > KUTlDXsl6Okj5nCNNyCqnHcE2ePwEEOsYkf2ASzQP2Y&e= > /apache < > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache > _ incubator-2Dmyriad_tree_0.2.x_myriad-2Dscheduler_src_main_ > java_org_apache&d=CwIFaQ&c=31nHN1tvZeuWBT6LwDN4Ngk1qezfsYHyolgGeY2ZhlU > &r= D2bc6ANY3sIFSxaGDnPt52V5PqDlabKIPtzNhAIfJCs&m= > ibxhOZQSsKtyVi5iruY8ImkW7bQ8zOrHcuDTLL7GBwA&s=465af32H1JItcea_ > tp5hz7zxwpqWgAqbVA8APaWmSUE&e= > /myriad > <https://urldefense.proofpoint.com/v2/url?u=https-3A__urldefense&d=CwI > FaQ&c=31nHN1tvZeuWBT6LwDN4Ngk1qezfsYHyolgGeY2ZhlU&r=D2bc6ANY3sIFSxaGDn > Pt52V5PqDlabKIPtzNhAIfJCs&m=sX9u4FJdfE4P4b24cRwGMPuyeT4XkeQRP5t8wCZTjV > w&s=DeQq-jARIja9dGEYfjeIQMd6jGkf_tNUyvQn7PIMieU&e= .> > proofpoint.com/v2/url?u=https-3A__github.com_apache_ > incubator-2Dmyriad_tree_0.2.x_myriad-2Dscheduler_src_main_ > java_org_apache_myriad&d=CwIFaQ&c=31nHN1tvZeuWBT6LwDN4Ngk1qezfsY > HyolgGeY2ZhlU&r=D2bc6ANY3sIFSxaGDnPt52V5PqDlabKIPtzNhAIfJCs&m= > ibxhOZQSsKtyVi5iruY8ImkW7bQ8zOrHcuDTLL7GBwA&s=QEZka3G6qwIsYzvPtXx- > w4uVek0Bt2D3bD4M4160Dnk&e= > /scheduler > <https://urldefense.proofpoint.com/v2/url?u=https-3A__urldefense&d=CwI > FaQ&c=31nHN1tvZeuWBT6LwDN4Ngk1qezfsYHyolgGeY2ZhlU&r=D2bc6ANY3sIFSxaGDn > Pt52V5PqDlabKIPtzNhAIfJCs&m=sX9u4FJdfE4P4b24cRwGMPuyeT4XkeQRP5t8wCZTjV > w&s=DeQq-jARIja9dGEYfjeIQMd6jGkf_tNUyvQn7PIMieU&e= .> > proofpoint.com/v2/url?u=https-3A__github.com_apache_ > incubator-2Dmyriad_tree_0.2.x_myriad-2Dscheduler_src_main_ > java_org_apache_myriad_scheduler&d=CwIFaQ&c=31nHN1tvZeuWBT6LwDN4Ngk1qe > zfsY HyolgGeY2ZhlU&r=D2bc6ANY3sIFSxaGDnPt52V5PqDlabKIPtzNhAIfJCs&m= > ibxhOZQSsKtyVi5iruY8ImkW7bQ8zOrHcuDTLL7GBwA&s= > 2EubhJ2JLwuGjY6DBZXpauvyuXJ0xefgOFHC8lEo5JE&e= > > /*NMExecutorCLGenImpl* and rebuild. > > Sorry that missed my QA unfortunately I'm always using cgroups and > didn't test that. We may do a 0.2.1 release but I can say when. > > Darin > > On Aug 16, 2016 8:49 AM, "Matthew J. Loppatto" > <mloppa...@keywcorp.com> > wrote: > > > Hi, > > > > > > > > I’m setting up Myriad 0.2.0 on my Mesos cluster following this guide: > > https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.or > > g_ > > confluence_display_MYRIAD_&d=CwIFaQ&c=31nHN1tvZeuWBT6LwDN4Ngk1qezfsY > > Hy > > olgGeY2ZhlU&r=D2bc6ANY3sIFSxaGDnPt52V5PqDlabKIPtzNhAIfJCs&m=ibxhOZQS > > sK > > tyVi5iruY8ImkW7bQ8zOrHcuDTLL7GBwA&s=LaQad9p3ZI3Rt5cTn3kHAb58BuSD5etw > > Im > > WZHzFz6Sk&e= > > Installing+for+Developers > > > > > > > > And I get the following error in the resource manager executor log > > in mesos after starting it with `/opt/hadoop-2.7.2/bin/yarn > resourcemanager`: > > > > > > > > chown: cannot access > > ‘/sys/fs/cgroup/cpu/mesos/f5d6c530-c13d-4b1d-bc30- > f298affb6442’: > > No such file or directory > > > > env: /bin/yarn: No such file or directory > > > > ory > > > > > > > > It appears the ‘mesos’ directory doesn’t exist under /sys/fs/cgroup/cpu. > > Any ideas what the issue could be? > > > > > > > > This is my yarn-site.xml: > > > > > > > > <configuration> > > > > <!-- Site-specific YARN configuration properties --> > > > > <property> > > > > <name>yarn.nodemanager.aux-services</name> > > > > <value>mapreduce_shuffle,myriad_executor</value> > > > > <!-- If using MapR distro, please use the following value: > > > > > > <value>mapreduce_shuffle,mapr_direct_shuffle,myriad_executor</value> > > --> > > > > </property> > > > > <property> > > > > > > <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name> > > > > <value>org.apache.hadoop.mapred.ShuffleHandler</value> > > > > </property> > > > > <property> > > > > > > <name>yarn.nodemanager.aux-services.myriad_executor.class</name> > > > > > > <value>org.apache.myriad.executor.MyriadExecutorAuxService</value> > > > > </property> > > > > <property> > > > > <name>yarn.nm.liveness-monitor.expiry-interval-ms</name> > > > > <value>2000</value> > > > > </property> > > > > <property> > > > > <name>yarn.am.liveness-monitor.expiry-interval-ms</name> > > > > <value>10000</value> > > > > </property> > > > > <property> > > > > > > <name>yarn.resourcemanager.nm.liveness-monitor.interval-ms</name> > > > > <value>1000</value> > > > > </property> > > > > <!-- Needed for Fine Grain Scaling --> > > > > <property> > > > > <name>yarn.scheduler.minimum-allocation-vcores</name> > > > > <value>0</value> > > > > </property> > > > > <property> > > > > <name>yarn.scheduler.minimum-allocation-mb</name> > > > > <value>0</value> > > > > </property> > > > > <!-- Site specific YARN configuration properties --> > > > > <property> > > > > <name>yarn.nodemanager.resource.cpu-vcores</name> > > > > <value>${nodemanager.resource.cpu-vcores}</value> > > > > </property> > > > > <property> > > > > <name>yarn.nodemanager.resource.memory-mb</name> > > > > <value>${nodemanager.resource.memory-mb}</value> > > > > </property> > > > > <!--These options enable dynamic port assignment by mesos --> > > > > <property> > > > > <name>yarn.nodemanager.address</name> > > > > <value>${myriad.yarn.nodemanager.address}</value> > > > > </property> > > > > <property> > > > > <name>yarn.nodemanager.webapp.address</name> > > > > <value>${myriad.yarn.nodemanager.webapp.address}</value> > > > > </property> > > > > <property> > > > > <name>yarn.nodemanager.webapp.https.address</name> > > > > <value>${myriad.yarn.nodemanager.webapp.address}</value> > > > > </property> > > > > <property> > > > > <name>yarn.nodemanager.localizer.address</name> > > > > <value>${myriad.yarn.nodemanager.localizer.address}</value> > > > > </property> > > > > <!-- Configure Myriad Scheduler here --> > > > > <property> > > > > <name>yarn.resourcemanager.scheduler.class</name> > > > > > > <value>org.apache.myriad.scheduler.yarn.MyriadFairScheduler</value> > > > > <description>One can configure other scehdulers as well from > > following > > list: org.apache.myriad.scheduler.yarn.MyriadCapacityScheduler, > > org.apache.myriad.scheduler.yarn.MyriadFifoScheduler</description> > > > > </property> > > > > <!-- Disable PMem/VMem checks for Hadoop 2.7.2 --> > > > > <property> > > > > <name>yarn.nodemanager.pmem-check-enabled</name> > > > > <value>false</value> > > > > </property> > > > > <property> > > > > <name>yarn.nodemanager.vmem-check-enabled</name> > > > > <value>false</value> > > > > </property> > > > > </configuration> > > > > > > > > > > > > My myriad-config-default.yml: > > > > > > > > mesosMaster: zk://myip:2181/mesos > > > > checkpoint: false > > > > frameworkFailoverTimeout: 43200000 > > > > frameworkName: MyriadAlpha > > > > frameworkRole: > > > > frameworkUser: root # User the Node Manager runs as, required if > > nodeManagerURI set, otherwise defaults to the user > > > > # running the resource manager. > > > > frameworkSuperUser: root # To be depricated, currently permissions > > need set by a superuser due to Mesos-1790. Must be > > > > # root or have passwordless sudo. Required > > if nodeManagerURI set, ignored otherwise. > > > > nativeLibrary: /usr/local/lib/libmesos.so > > > > zkServers: myip:2181 > > > > zkTimeout: 20000 > > > > restApiPort: 8192 > > > > servedConfigPath: dist/config.tgz > > > > servedBinaryPath: dist/binary.tgz > > > > profiles: > > > > zero: # NMs launched with this profile dynamically obtain cpu/mem > > from Mesos > > > > cpu: 0 > > > > mem: 0 > > > > small: > > > > cpu: 2 > > > > mem: 2048 > > > > medium: > > > > cpu: 4 > > > > mem: 4096 > > > > large: > > > > cpu: 10 > > > > mem: 12288 > > > > nmInstances: # NMs to start with. Requires at least 1 NM with a > > non-zero profile. > > > > medium: 1 # <profile_name : instances> > > > > rebalancer: false > > > > haEnabled: false > > > > nodemanager: > > > > jvmMaxMemoryMB: 1024 > > > > cpus: 0.2 > > > > cgroups: false > > > > executor: > > > > jvmMaxMemoryMB: 256 > > > > path: > > file:///usr/local/libexec/mesos/myriad-executor-runnable-0.1.0.jar > > > > #The following should be used for a remotely distributed URI, hdfs > > assumed but other URI types valid. > > > > #nodeManagerUri: hdfs://namenode:port/dist/hadoop-2.7.0.tar.gz > > > > #configUri: > > https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1_api_ar > > if > > acts_config.tgz&d=CwIFaQ&c=31nHN1tvZeuWBT6LwDN4Ngk1qezfsYHyolgGeY2Zh > > lU > > &r=D2bc6ANY3sIFSxaGDnPt52V5PqDlabKIPtzNhAIfJCs&m=ibxhOZQSsKtyVi5iruY > > 8I > > mkW7bQ8zOrHcuDTLL7GBwA&s=IpOqhUOtwJsdorbAOeoY7GgHalMJ1s9EUjuRUfRsmew > > &e > > = > > > > #jvmUri: > > https://urldefense.proofpoint.com/v2/url?u=https-3A__downloads.mycom > > pa > > ny.com_java_jre-2D7u76-2Dlinux-2Dx64.tar.gz&d=CwIFaQ&c=31nHN1tvZeuWB > > T6 > > LwDN4Ngk1qezfsYHyolgGeY2ZhlU&r=D2bc6ANY3sIFSxaGDnPt52V5PqDlabKIPtzNh > > AI > > fJCs&m=ibxhOZQSsKtyVi5iruY8ImkW7bQ8zOrHcuDTLL7GBwA&s=jPB2677RH3k3CLs > > gl > > 4Zj3tGawuCLVB1a2WXBUOWEelU&e= > > > > yarnEnvironment: > > > > YARN_HOME: /opt/hadoop-2.7.2 > > > > > > > > > > > > Thanks! > > > > Matt > > >
smime.p7s
Description: S/MIME cryptographic signature
