答复: 答复: Hello Guys
Darin, Thank you for your reply. As you mentioned that, Myriad NM is launched by Myriad Framework, while CM could be launching the NM in a totally different way. I think I should figure out the procedure first. Since Myriad could utilize resource from Mesos, and at the same time it is a YARN FairScheducler. Although it's impractical to perfect integrate CM with Mesos，there still could be a chance to intergrate them under this specific scene. With the help of Myriad, may be it is possible to share the resource between these two framework. Regards, Yongyu 发件人: Darin Johnson [mailto:dbjohnson1...@gmail.com] 发送时间: 2016年4月27日 22:47 收件人: 陈泳宇; Dev 抄送: c...@linkernetworks.com; s...@linkernetworks.com; Sam Chen 主题: Re: 答复: Hello Guys Yongyu, Sounds like you've made some progress! I'm not sure you'll be able to get the cluster completely under CM management and the node managers launched by Myriad aren't running the Cloudera agent (I don't think we'll ever officially support that, but you could tweak the source to run it in the background, if you could make it a general process it might be a feature we'd add). Though if you're launching the Resource Manager via CM I'd expect it to show up. I haven't attempted to put my resource manager under the CM though. Darin On Tue, Apr 26, 2016 at 4:45 AM, 陈泳宇 mailto:yc...@linkernetworks.com> > wrote: Hi Darin, This is Yongyu from Linkernetworks, I have proved that myriad can work well with CDH. Currently, I am also trying to myriad our hadoop cluster under the management of CM. I noticed that you have already complete the mission, but I still got some problem here: The status shown on CM dashboard will not change… Here are my steps: 1. Add YARN service on CM dashboard.. 2. Stop the YARN service 3. Cd to /opt/cloudera/parcels/CDH-5.7.0-1.cdh5.7.0.p0.45 4. Copy native libs 、myriad jars as well we the config files. 5. Tar this package and upload it to hdfs. 6. Start the resourcemanager. Myriad service will be launched successfully, but nothing changes on the dashboard, which means the cluster is not under the management of CM.. Regards, Yongyu Begin forwarded message: From: Darin Johnson mailto:dbjohnson1...@gmail.com> > Date: April 21, 2016 at 8:50:22 PM GMT+8 To: Dev mailto:dev@myriad.incubator.apache.org> > Subject: Re: Hello Guys Reply-To: dev@myriad.incubator.apache.org <mailto:dev@myriad.incubator.apache.org> Hey Sam, I'm already myriad alongside a CM managed hadoop cluster. It's a little hacky right now, I'm working on stream lining this, it may involve CM and/or some docker integration. Here are my current steps: 0. I strongly recommend pulling of master and building from source - it has some really useful patches. We're working on another release now (couple weeks out though). 1. Let Cloudera Manager configure hdfs - it does a good job of this. 2. Grab a cloudera tar ball from here: http://www.cloudera.com/documentation/enterprise/release-notes/topics/cdh_vd_cdh_package_tarball_eom.html#topic_3 (I've also just used apache-hadoop tarballs). 3. Extract the tar ball, copy the native libraries cdh install on your system in hadoop-*/lib/native 4 cp myriad/myriad-*/build/libs/* hadoop-*/share/hadoop/yarn/lib 5 copy your hadoop configs to hadoop-*/etc/hadoop/ 6 create a myriad-default-config.yml in hadoop-*/etc/hadoop follow instructions on wiki for remote distribution for edits to myriad-default-config.yml. NB: don't enable yarn cgroups yet - I'm fixing a bug. 7 chown -R root:root hadoop-*y 8 chown root:yarn hadoop-*/bin/container-executor ; chmod g+s hadoop*/bin/container-executor 9 mv hadoop-<..> hadoop-myriad 10 tar -zxvf hadoop-myriad.tgz hadoop-myriad ; hadoop fs -put hadoop-myriad.tgz /dist/ 11 cd hadoop-myriad/ && sudo -u yarn bin/yarn resource manager 12 hit the web-ui at host:8192 and flexup some node managers This has been pretty stable. Alternatively, if you're running mesos and docker you could look at PR https://github.com/apache/incubator-myriad/pull/64, it's still WIP but avoids a lot of setup - would be happy to work with you as I document and harden this feature for the 0.2.0 release. Currently that runs off generic hadoop put we could certainly create distribution specific dockerfiles. Let me know if you need help. Keep in mind this is still an alpha project, so expect some issues. Would love to get feedback, use cases and feature ideas. Also a mesos tip: if your running services like hdfs outside of mesos you should adjust you're mesos resources appropriately or you'll end up over subscribed and processes will slow or die. Thanks for trying myriad! Darin On Apr 21, 2016 4:22 AM, "rchen" mailto:rc...@linkernetworks.com> > wrote: Hi Guys,
Re: 答复: Hello Guys
Yongyu, Sounds like you've made some progress! I'm not sure you'll be able to get the cluster completely under CM management and the node managers launched by Myriad aren't running the Cloudera agent (I don't think we'll ever officially support that, but you could tweak the source to run it in the background, if you could make it a general process it might be a feature we'd add). Though if you're launching the Resource Manager via CM I'd expect it to show up. I haven't attempted to put my resource manager under the CM though. Darin On Tue, Apr 26, 2016 at 4:45 AM, 陈泳宇 wrote: > Hi Darin, This is Yongyu from Linkernetworks, I have proved that myriad > can work well with CDH. Currently, I am also trying to myriad our hadoop > cluster under the management of CM. > > I noticed that you have already complete the mission, but I still got some > problem here: The status shown on CM dashboard will not change… > > Here are my steps: > > 1. Add YARN service on CM dashboard.. > > 2. Stop the YARN service > > 3. Cd to /opt/cloudera/parcels/CDH-5.7.0-1.cdh5.7.0.p0.45 > > 4. Copy native libs 、myriad jars as well we the config files. > > 5. Tar this package and upload it to hdfs. > > 6. Start the resourcemanager. > > Myriad service will be launched successfully, but nothing changes on the > dashboard, which means the cluster is not under the management of CM.. > > > > > > Regards, > > Yongyu > > > Begin forwarded message: > > *From:* Darin Johnson > *Date:* April 21, 2016 at 8:50:22 PM GMT+8 > *To:* Dev > *Subject:* *Re: Hello Guys* > *Reply-To:* dev@myriad.incubator.apache.org > > Hey Sam, > > I'm already myriad alongside a CM managed hadoop cluster. It's a little > hacky right now, I'm working on stream lining this, it may involve CM > and/or some docker integration. Here are my current steps: > > 0. I strongly recommend pulling of master and building from source - it has > some really useful patches. We're working on another release now (couple > weeks out though). > > 1. Let Cloudera Manager configure hdfs - it does a good job of this. > > 2. Grab a cloudera tar ball from here: > > http://www.cloudera.com/documentation/enterprise/release-notes/topics/cdh_vd_cdh_package_tarball_eom.html#topic_3 > (I've also just used apache-hadoop tarballs). > > 3. Extract the tar ball, copy the native libraries cdh install on your > system in hadoop-*/lib/native > > 4 cp myriad/myriad-*/build/libs/* hadoop-*/share/hadoop/yarn/lib > > 5 copy your hadoop configs to hadoop-*/etc/hadoop/ > > 6 create a myriad-default-config.yml in hadoop-*/etc/hadoop follow > instructions on wiki for remote distribution for edits to > myriad-default-config.yml. NB: don't enable yarn cgroups yet - I'm fixing > a bug. > > 7 chown -R root:root hadoop-*y > > 8 chown root:yarn hadoop-*/bin/container-executor ; chmod g+s > hadoop*/bin/container-executor > > 9 mv hadoop-<..> hadoop-myriad > > 10 tar -zxvf hadoop-myriad.tgz hadoop-myriad ; hadoop fs -put > hadoop-myriad.tgz /dist/ > > 11 cd hadoop-myriad/ && sudo -u yarn bin/yarn resource manager > > 12 hit the web-ui at host:8192 and flexup some node managers > > This has been pretty stable. > > Alternatively, if you're running mesos and docker you could look at PR > https://github.com/apache/incubator-myriad/pull/64, it's still WIP but > avoids a lot of setup - would be happy to work with you as I document and > harden this feature for the 0.2.0 release. Currently that runs off generic > hadoop put we could certainly create distribution specific dockerfiles. > > Let me know if you need help. Keep in mind this is still an alpha project, > so expect some issues. Would love to get feedback, use cases and feature > ideas. > > Also a mesos tip: if your running services like hdfs outside of mesos you > should adjust you're mesos resources appropriately or you'll end up over > subscribed and processes will slow or die. > > Thanks for trying myriad! > > Darin > > > On Apr 21, 2016 4:22 AM, "rchen" wrote: > > Hi Guys, > Currrently, we are working on Myriad with Cloudera distribution. And we > have proved that Mesos and work with Yarn. > However talking about CM, how to integrate ? Any comments is welcome. > appreciated. > > > Regards > Sam > >
Re: Hello Guys
Hey Sam, I'm already myriad alongside a CM managed hadoop cluster. It's a little hacky right now, I'm working on stream lining this, it may involve CM and/or some docker integration. Here are my current steps: 0. I strongly recommend pulling of master and building from source - it has some really useful patches. We're working on another release now (couple weeks out though). 1. Let Cloudera Manager configure hdfs - it does a good job of this. 2. Grab a cloudera tar ball from here: http://www.cloudera.com/documentation/enterprise/release-notes/topics/cdh_vd_cdh_package_tarball_eom.html#topic_3 (I've also just used apache-hadoop tarballs). 3. Extract the tar ball, copy the native libraries cdh install on your system in hadoop-*/lib/native 4 cp myriad/myriad-*/build/libs/* hadoop-*/share/hadoop/yarn/lib 5 copy your hadoop configs to hadoop-*/etc/hadoop/ 6 create a myriad-default-config.yml in hadoop-*/etc/hadoop follow instructions on wiki for remote distribution for edits to myriad-default-config.yml. NB: don't enable yarn cgroups yet - I'm fixing a bug. 7 chown -R root:root hadoop-*y 8 chown root:yarn hadoop-*/bin/container-executor ; chmod g+s hadoop*/bin/container-executor 9 mv hadoop-<..> hadoop-myriad 10 tar -zxvf hadoop-myriad.tgz hadoop-myriad ; hadoop fs -put hadoop-myriad.tgz /dist/ 11 cd hadoop-myriad/ && sudo -u yarn bin/yarn resource manager 12 hit the web-ui at host:8192 and flexup some node managers This has been pretty stable. Alternatively, if you're running mesos and docker you could look at PR https://github.com/apache/incubator-myriad/pull/64, it's still WIP but avoids a lot of setup - would be happy to work with you as I document and harden this feature for the 0.2.0 release. Currently that runs off generic hadoop put we could certainly create distribution specific dockerfiles. Let me know if you need help. Keep in mind this is still an alpha project, so expect some issues. Would love to get feedback, use cases and feature ideas. Also a mesos tip: if your running services like hdfs outside of mesos you should adjust you're mesos resources appropriately or you'll end up over subscribed and processes will slow or die. Thanks for trying myriad! Darin On Apr 21, 2016 4:22 AM, "rchen" wrote: Hi Guys, Currrently, we are working on Myriad with Cloudera distribution. And we have proved that Mesos and work with Yarn. However talking about CM, how to integrate ? Any comments is welcome. appreciated. Regards Sam
Hello Guys
Hi Guys, Currrently, we are working on Myriad with Cloudera distribution. And we have proved that Mesos and work with Yarn. However talking about CM, how to integrate ? Any comments is welcome. appreciated. Regards Sam
