Hey Darin, Pulling from master got rid of the errors I was seeing, however I'm running into a new issue. After starting the resource manager, I see this in the logs:
2016-08-17 10:56:40,709 INFO org.apache.myriad.Main: Launching 1 NM(s) with
profile medium
2016-08-17 10:56:40,710 INFO org.apache.myriad.scheduler.MyriadOperations:
Adding 1 NM instances to cluster
2016-08-17 10:56:40,733 ERROR
org.apache.myriad.scheduler.event.handlers.ErrorEventHandler: Role '' is not
present in the master's --roles
My Mesos cluster has the default "*" role so I tried setting frameworkRole: "*"
in myriad-config-default.yml, restarted the resource manager and got this error:
2016-08-17 11:06:28,244 ERROR
org.apache.myriad.scheduler.event.handlers.ResourceOffersEventHandler:
Exception thrown while trying to create a task for nm
java.lang.IllegalArgumentException: n must be positive
at java.util.Random.nextInt(Random.java:300)
at
org.apache.myriad.scheduler.resource.RangeResource.getRandomValues(RangeResource.java:128)
at
org.apache.myriad.scheduler.resource.RangeResource.consumeResource(RangeResource.java:99)
at
org.apache.myriad.scheduler.resource.ResourceOfferContainer.consumePorts(ResourceOfferContainer.java:171)
at
org.apache.myriad.scheduler.NMTaskFactory.createTask(NMTaskFactory.java:45)
at
org.apache.myriad.scheduler.event.handlers.ResourceOffersEventHandler.onEvent(ResourceOffersEventHandler.java:119)
at
org.apache.myriad.scheduler.event.handlers.ResourceOffersEventHandler.onEvent(ResourceOffersEventHandler.java:49)
at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Does Myriad require its own role in Mesos?
Thanks,
Matt
-----Original Message-----
From: Darin Johnson [mailto:dbjohnson1...@gmail.com]
Sent: Tuesday, August 16, 2016 6:18 PM
To: Dev
Subject: Re: Resource manager error
Hey Mathew, my coworker found the same issue recently, I fixed it on my last
pull request, if you'd like to pull from master.
Alternatively, you could comment out the appendCgroups line in myriad-scheduler
<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_incubator-2Dmyriad_tree_0.2.x_myriad-2Dscheduler&d=CwIFaQ&c=31nHN1tvZeuWBT6LwDN4Ngk1qezfsYHyolgGeY2ZhlU&r=D2bc6ANY3sIFSxaGDnPt52V5PqDlabKIPtzNhAIfJCs&m=ibxhOZQSsKtyVi5iruY8ImkW7bQ8zOrHcuDTLL7GBwA&s=5zc_OUK_qwnVQoC2kVCcAgvb4ZmZrVKF-iHca_dif4Y&e=
>/src
<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_incubator-2Dmyriad_tree_0.2.x_myriad-2Dscheduler_src&d=CwIFaQ&c=31nHN1tvZeuWBT6LwDN4Ngk1qezfsYHyolgGeY2ZhlU&r=D2bc6ANY3sIFSxaGDnPt52V5PqDlabKIPtzNhAIfJCs&m=ibxhOZQSsKtyVi5iruY8ImkW7bQ8zOrHcuDTLL7GBwA&s=6CJWEHP2t7cY2oTmNz9Aq9AV39VEkUOKpMoRuz1q9nY&e=
> /main
<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_incubator-2Dmyriad_tree_0.2.x_myriad-2Dscheduler_src_main&d=CwIFaQ&c=31nHN1tvZeuWBT6LwDN4Ngk1qezfsYHyolgGeY2ZhlU&r=D2bc6ANY3sIFSxaGDnPt52V5PqDlabKIPtzNhAIfJCs&m=ibxhOZQSsKtyVi5iruY8ImkW7bQ8zOrHcuDTLL7GBwA&s=BtYphvuvvNZ5owUTfiRd4hW90jq0Ib8GGtKiHU0fTB4&e=
> /java
<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_incubator-2Dmyriad_tree_0.2.x_myriad-2Dscheduler_src_main_java&d=CwIFaQ&c=31nHN1tvZeuWBT6LwDN4Ngk1qezfsYHyolgGeY2ZhlU&r=D2bc6ANY3sIFSxaGDnPt52V5PqDlabKIPtzNhAIfJCs&m=ibxhOZQSsKtyVi5iruY8ImkW7bQ8zOrHcuDTLL7GBwA&s=5o45_8zSN96rSaQJ8oCWfhCvmqhSbLpz9fMnV9Fk4WI&e=
> /org
<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_incubator-2Dmyriad_tree_0.2.x_myriad-2Dscheduler_src_main_java_org&d=CwIFaQ&c=31nHN1tvZeuWBT6LwDN4Ngk1qezfsYHyolgGeY2ZhlU&r=D2bc6ANY3sIFSxaGDnPt52V5PqDlabKIPtzNhAIfJCs&m=ibxhOZQSsKtyVi5iruY8ImkW7bQ8zOrHcuDTLL7GBwA&s=KUTlDXsl6Okj5nCNNyCqnHcE2ePwEEOsYkf2ASzQP2Y&e=
> /apache
<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_incubator-2Dmyriad_tree_0.2.x_myriad-2Dscheduler_src_main_java_org_apache&d=CwIFaQ&c=31nHN1tvZeuWBT6LwDN4Ngk1qezfsYHyolgGeY2ZhlU&r=D2bc6ANY3sIFSxaGDnPt52V5PqDlabKIPtzNhAIfJCs&m=ibxhOZQSsKtyVi5iruY8ImkW7bQ8zOrHcuDTLL7GBwA&s=465af32H1JItcea_tp5hz7zxwpqWgAqbVA8APaWmSUE&e=
> /myriad
<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_incubator-2Dmyriad_tree_0.2.x_myriad-2Dscheduler_src_main_java_org_apache_myriad&d=CwIFaQ&c=31nHN1tvZeuWBT6LwDN4Ngk1qezfsYHyolgGeY2ZhlU&r=D2bc6ANY3sIFSxaGDnPt52V5PqDlabKIPtzNhAIfJCs&m=ibxhOZQSsKtyVi5iruY8ImkW7bQ8zOrHcuDTLL7GBwA&s=QEZka3G6qwIsYzvPtXx-w4uVek0Bt2D3bD4M4160Dnk&e=
> /scheduler
<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_incubator-2Dmyriad_tree_0.2.x_myriad-2Dscheduler_src_main_java_org_apache_myriad_scheduler&d=CwIFaQ&c=31nHN1tvZeuWBT6LwDN4Ngk1qezfsYHyolgGeY2ZhlU&r=D2bc6ANY3sIFSxaGDnPt52V5PqDlabKIPtzNhAIfJCs&m=ibxhOZQSsKtyVi5iruY8ImkW7bQ8zOrHcuDTLL7GBwA&s=2EubhJ2JLwuGjY6DBZXpauvyuXJ0xefgOFHC8lEo5JE&e=
>
/*NMExecutorCLGenImpl* and rebuild.
Sorry that missed my QA unfortunately I'm always using cgroups and didn't test
that. We may do a 0.2.1 release but I can say when.
Darin
On Aug 16, 2016 8:49 AM, "Matthew J. Loppatto" <mloppa...@keywcorp.com>
wrote:
> Hi,
>
>
>
> I’m setting up Myriad 0.2.0 on my Mesos cluster following this guide:
> https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_
> confluence_display_MYRIAD_&d=CwIFaQ&c=31nHN1tvZeuWBT6LwDN4Ngk1qezfsYHy
> olgGeY2ZhlU&r=D2bc6ANY3sIFSxaGDnPt52V5PqDlabKIPtzNhAIfJCs&m=ibxhOZQSsK
> tyVi5iruY8ImkW7bQ8zOrHcuDTLL7GBwA&s=LaQad9p3ZI3Rt5cTn3kHAb58BuSD5etwIm
> WZHzFz6Sk&e=
> Installing+for+Developers
>
>
>
> And I get the following error in the resource manager executor log in
> mesos after starting it with `/opt/hadoop-2.7.2/bin/yarn resourcemanager`:
>
>
>
> chown: cannot access
> ‘/sys/fs/cgroup/cpu/mesos/f5d6c530-c13d-4b1d-bc30-f298affb6442’:
> No such file or directory
>
> env: /bin/yarn: No such file or directory
>
> ory
>
>
>
> It appears the ‘mesos’ directory doesn’t exist under /sys/fs/cgroup/cpu.
> Any ideas what the issue could be?
>
>
>
> This is my yarn-site.xml:
>
>
>
> <configuration>
>
> <!-- Site-specific YARN configuration properties -->
>
> <property>
>
> <name>yarn.nodemanager.aux-services</name>
>
> <value>mapreduce_shuffle,myriad_executor</value>
>
> <!-- If using MapR distro, please use the following value:
>
>
> <value>mapreduce_shuffle,mapr_direct_shuffle,myriad_executor</value>
> -->
>
> </property>
>
> <property>
>
>
> <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
>
> <value>org.apache.hadoop.mapred.ShuffleHandler</value>
>
> </property>
>
> <property>
>
>
> <name>yarn.nodemanager.aux-services.myriad_executor.class</name>
>
>
> <value>org.apache.myriad.executor.MyriadExecutorAuxService</value>
>
> </property>
>
> <property>
>
> <name>yarn.nm.liveness-monitor.expiry-interval-ms</name>
>
> <value>2000</value>
>
> </property>
>
> <property>
>
> <name>yarn.am.liveness-monitor.expiry-interval-ms</name>
>
> <value>10000</value>
>
> </property>
>
> <property>
>
>
> <name>yarn.resourcemanager.nm.liveness-monitor.interval-ms</name>
>
> <value>1000</value>
>
> </property>
>
> <!-- Needed for Fine Grain Scaling -->
>
> <property>
>
> <name>yarn.scheduler.minimum-allocation-vcores</name>
>
> <value>0</value>
>
> </property>
>
> <property>
>
> <name>yarn.scheduler.minimum-allocation-mb</name>
>
> <value>0</value>
>
> </property>
>
> <!-- Site specific YARN configuration properties -->
>
> <property>
>
> <name>yarn.nodemanager.resource.cpu-vcores</name>
>
> <value>${nodemanager.resource.cpu-vcores}</value>
>
> </property>
>
> <property>
>
> <name>yarn.nodemanager.resource.memory-mb</name>
>
> <value>${nodemanager.resource.memory-mb}</value>
>
> </property>
>
> <!--These options enable dynamic port assignment by mesos -->
>
> <property>
>
> <name>yarn.nodemanager.address</name>
>
> <value>${myriad.yarn.nodemanager.address}</value>
>
> </property>
>
> <property>
>
> <name>yarn.nodemanager.webapp.address</name>
>
> <value>${myriad.yarn.nodemanager.webapp.address}</value>
>
> </property>
>
> <property>
>
> <name>yarn.nodemanager.webapp.https.address</name>
>
> <value>${myriad.yarn.nodemanager.webapp.address}</value>
>
> </property>
>
> <property>
>
> <name>yarn.nodemanager.localizer.address</name>
>
> <value>${myriad.yarn.nodemanager.localizer.address}</value>
>
> </property>
>
> <!-- Configure Myriad Scheduler here -->
>
> <property>
>
> <name>yarn.resourcemanager.scheduler.class</name>
>
> <value>org.apache.myriad.scheduler.yarn.MyriadFairScheduler</value>
>
> <description>One can configure other scehdulers as well from
> following
> list: org.apache.myriad.scheduler.yarn.MyriadCapacityScheduler,
> org.apache.myriad.scheduler.yarn.MyriadFifoScheduler</description>
>
> </property>
>
> <!-- Disable PMem/VMem checks for Hadoop 2.7.2 -->
>
> <property>
>
> <name>yarn.nodemanager.pmem-check-enabled</name>
>
> <value>false</value>
>
> </property>
>
> <property>
>
> <name>yarn.nodemanager.vmem-check-enabled</name>
>
> <value>false</value>
>
> </property>
>
> </configuration>
>
>
>
>
>
> My myriad-config-default.yml:
>
>
>
> mesosMaster: zk://myip:2181/mesos
>
> checkpoint: false
>
> frameworkFailoverTimeout: 43200000
>
> frameworkName: MyriadAlpha
>
> frameworkRole:
>
> frameworkUser: root # User the Node Manager runs as, required if
> nodeManagerURI set, otherwise defaults to the user
>
> # running the resource manager.
>
> frameworkSuperUser: root # To be depricated, currently permissions
> need set by a superuser due to Mesos-1790. Must be
>
> # root or have passwordless sudo. Required if
> nodeManagerURI set, ignored otherwise.
>
> nativeLibrary: /usr/local/lib/libmesos.so
>
> zkServers: myip:2181
>
> zkTimeout: 20000
>
> restApiPort: 8192
>
> servedConfigPath: dist/config.tgz
>
> servedBinaryPath: dist/binary.tgz
>
> profiles:
>
> zero: # NMs launched with this profile dynamically obtain cpu/mem
> from Mesos
>
> cpu: 0
>
> mem: 0
>
> small:
>
> cpu: 2
>
> mem: 2048
>
> medium:
>
> cpu: 4
>
> mem: 4096
>
> large:
>
> cpu: 10
>
> mem: 12288
>
> nmInstances: # NMs to start with. Requires at least 1 NM with a
> non-zero profile.
>
> medium: 1 # <profile_name : instances>
>
> rebalancer: false
>
> haEnabled: false
>
> nodemanager:
>
> jvmMaxMemoryMB: 1024
>
> cpus: 0.2
>
> cgroups: false
>
> executor:
>
> jvmMaxMemoryMB: 256
>
> path:
> file:///usr/local/libexec/mesos/myriad-executor-runnable-0.1.0.jar
>
> #The following should be used for a remotely distributed URI, hdfs
> assumed but other URI types valid.
>
> #nodeManagerUri: hdfs://namenode:port/dist/hadoop-2.7.0.tar.gz
>
> #configUri:
> https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1_api_arif
> acts_config.tgz&d=CwIFaQ&c=31nHN1tvZeuWBT6LwDN4Ngk1qezfsYHyolgGeY2ZhlU
> &r=D2bc6ANY3sIFSxaGDnPt52V5PqDlabKIPtzNhAIfJCs&m=ibxhOZQSsKtyVi5iruY8I
> mkW7bQ8zOrHcuDTLL7GBwA&s=IpOqhUOtwJsdorbAOeoY7GgHalMJ1s9EUjuRUfRsmew&e
> =
>
> #jvmUri:
> https://urldefense.proofpoint.com/v2/url?u=https-3A__downloads.mycompa
> ny.com_java_jre-2D7u76-2Dlinux-2Dx64.tar.gz&d=CwIFaQ&c=31nHN1tvZeuWBT6
> LwDN4Ngk1qezfsYHyolgGeY2ZhlU&r=D2bc6ANY3sIFSxaGDnPt52V5PqDlabKIPtzNhAI
> fJCs&m=ibxhOZQSsKtyVi5iruY8ImkW7bQ8zOrHcuDTLL7GBwA&s=jPB2677RH3k3CLsgl
> 4Zj3tGawuCLVB1a2WXBUOWEelU&e=
>
> yarnEnvironment:
>
> YARN_HOME: /opt/hadoop-2.7.2
>
>
>
>
>
> Thanks!
>
> Matt
>
smime.p7s
Description: S/MIME cryptographic signature
