Re: Yarn-Site
OK edited, Ruth I decided to do the absolute path of /opt/hadoop-2.7.1 vs
$YARN_HOME, in case the environment variable was off and someone
copy/pasted.
Still need to add the instructions for running the RM in Marathon.
Darin
On Mon, Oct 5, 2015 at 4:39 PM, Darin Johnson
wrote:
> sudo rm $YARN_HOME/etc/hadoop/yarn-site.xml <- This is slightly off, I'll
> go correct.
>
> On Mon, Oct 5, 2015 at 4:38 PM, Darin Johnson
> wrote:
>
>> John, I'm running off: https://github.com/apache/incubator-myriad, it
>> seems to run OK there's a couple of NPE issues though they're rare events
>> (I had to invent a way to make one occur), I've got a PR for one if you
>> want to test/review it for me :).
>>
>> I'll try to update the wiki with instructions on running the resource
>> manager via marathon.
>>
>> Darin
>>
>>
>> On Mon, Oct 5, 2015 at 4:27 PM, John Omernik wrote:
>>
>>> I see. That makes sense. Thanks for the tip.
>>>
>>> Is it safe to pull down a recent version at this point? Are we using the
>>> official "master" or phase1? (the lazy man in me is asking for a link to
>>> the current repo so I don't have to read back over emails to see where I
>>> should go :)
>>>
>>>
>>>
>>> On Mon, Oct 5, 2015 at 3:25 PM, Darin Johnson
>>> wrote:
>>>
>>> > Hey John,
>>> >
>>> > Are you trying to run the resource manager from the tar ball via
>>> marathon?
>>> > It's doable, my suggested approach would be to use a json like this:
>>> >
>>> > {
>>> > "id": "resource-manager",
>>> > "uris": ["hdfs://namenode:port/dist/hadoop-2.7.0.tgz",
>>> > "hdfs://namenode:port/dist/conf/hadoop/yarn-site.xml",
>>> > "hdfs://namenode:port/dist/conf/hadoop/hdfs-site.xml",
>>> > "hdfs:///dist/conf/hadoop/core-site.xml",
>>> > "hdfs://namenode:port/dist/conf/hadoop/mapred-site.xml"],
>>> > "cmd": "cp *.xml hadoop-2.7.0/etc/hadoop && cd hadoop-2.7.0 &&
>>> bin/yarn
>>> > resourcemanager",
>>> > "mem": 16,
>>> > "cpu": 1
>>> > "instances" : 1,
>>> > "user": "yarn"
>>> > }
>>> >
>>> > Basically it keeps you from redoing the tar ball every time you edit a
>>> > config, instead you just upload the new yarn-site.xml. The Node
>>> Manager
>>> > gets it's config from the Resource Manager (I'm assuming this is all
>>> for
>>> > remote distribution, otherwise creating the tar ball is optional).
>>> >
>>> > Darin
>>> >
>>> > On Mon, Oct 5, 2015 at 2:36 PM, John Omernik wrote:
>>> >
>>> > > Hey all, I've been waiting until the chaos of the code move has died
>>> > down.
>>> > > I am looking to get this working on my MapR cluster now, and would
>>> like
>>> > > some clarification on instructions here:
>>> > >
>>> > >
>>> > >
>>> > >
>>> >
>>> https://cwiki.apache.org/confluence/display/MYRIAD/Installing+for+Administrators
>>> > >
>>> > > Basically, in the instructions below, it has the "remove the
>>> > > yarn-site.xml. Yet to run the resource manager with myriad, you
>>> need the
>>> > > yarn-site to be packaged with things (unless I am reading that
>>> > incorrectly)
>>> > > Is the only option right now to created a tarball for nodemanagers,
>>> and
>>> > > have this be different from the tarball for the resource manager?
>>> > >
>>> > > Step 5: Create the Tarball
>>> > >
>>> > > The tarball has all of the files needed for the Node Managers and
>>> > Resource
>>> > > Managers. The following shows how to create the tarball and place it
>>> in
>>> > > HDFS:
>>> > > cd ~
>>> > > sudo cp -rp $YARN_HOME .
>>> > > sudo rm $YARN_HOME/etc/hadoop/yarn-site.xml
>>> > > sudo tar -zcpf ~/hadoop-2.7.1.tar.gz hadoop-2.7.1
>>> > > hadoop fs -put ~/hadoop-2.7.1.tar.gz /dist
>>> > >
>>> >
>>>
>>
>>
>
Re: Yarn-Site
sudo rm $YARN_HOME/etc/hadoop/yarn-site.xml <- This is slightly off, I'll
go correct.
On Mon, Oct 5, 2015 at 4:38 PM, Darin Johnson
wrote:
> John, I'm running off: https://github.com/apache/incubator-myriad, it
> seems to run OK there's a couple of NPE issues though they're rare events
> (I had to invent a way to make one occur), I've got a PR for one if you
> want to test/review it for me :).
>
> I'll try to update the wiki with instructions on running the resource
> manager via marathon.
>
> Darin
>
>
> On Mon, Oct 5, 2015 at 4:27 PM, John Omernik wrote:
>
>> I see. That makes sense. Thanks for the tip.
>>
>> Is it safe to pull down a recent version at this point? Are we using the
>> official "master" or phase1? (the lazy man in me is asking for a link to
>> the current repo so I don't have to read back over emails to see where I
>> should go :)
>>
>>
>>
>> On Mon, Oct 5, 2015 at 3:25 PM, Darin Johnson
>> wrote:
>>
>> > Hey John,
>> >
>> > Are you trying to run the resource manager from the tar ball via
>> marathon?
>> > It's doable, my suggested approach would be to use a json like this:
>> >
>> > {
>> > "id": "resource-manager",
>> > "uris": ["hdfs://namenode:port/dist/hadoop-2.7.0.tgz",
>> > "hdfs://namenode:port/dist/conf/hadoop/yarn-site.xml",
>> > "hdfs://namenode:port/dist/conf/hadoop/hdfs-site.xml",
>> > "hdfs:///dist/conf/hadoop/core-site.xml",
>> > "hdfs://namenode:port/dist/conf/hadoop/mapred-site.xml"],
>> > "cmd": "cp *.xml hadoop-2.7.0/etc/hadoop && cd hadoop-2.7.0 &&
>> bin/yarn
>> > resourcemanager",
>> > "mem": 16,
>> > "cpu": 1
>> > "instances" : 1,
>> > "user": "yarn"
>> > }
>> >
>> > Basically it keeps you from redoing the tar ball every time you edit a
>> > config, instead you just upload the new yarn-site.xml. The Node Manager
>> > gets it's config from the Resource Manager (I'm assuming this is all for
>> > remote distribution, otherwise creating the tar ball is optional).
>> >
>> > Darin
>> >
>> > On Mon, Oct 5, 2015 at 2:36 PM, John Omernik wrote:
>> >
>> > > Hey all, I've been waiting until the chaos of the code move has died
>> > down.
>> > > I am looking to get this working on my MapR cluster now, and would
>> like
>> > > some clarification on instructions here:
>> > >
>> > >
>> > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/MYRIAD/Installing+for+Administrators
>> > >
>> > > Basically, in the instructions below, it has the "remove the
>> > > yarn-site.xml. Yet to run the resource manager with myriad, you need
>> the
>> > > yarn-site to be packaged with things (unless I am reading that
>> > incorrectly)
>> > > Is the only option right now to created a tarball for nodemanagers,
>> and
>> > > have this be different from the tarball for the resource manager?
>> > >
>> > > Step 5: Create the Tarball
>> > >
>> > > The tarball has all of the files needed for the Node Managers and
>> > Resource
>> > > Managers. The following shows how to create the tarball and place it
>> in
>> > > HDFS:
>> > > cd ~
>> > > sudo cp -rp $YARN_HOME .
>> > > sudo rm $YARN_HOME/etc/hadoop/yarn-site.xml
>> > > sudo tar -zcpf ~/hadoop-2.7.1.tar.gz hadoop-2.7.1
>> > > hadoop fs -put ~/hadoop-2.7.1.tar.gz /dist
>> > >
>> >
>>
>
>
Re: Yarn-Site
John, I'm running off: https://github.com/apache/incubator-myriad, it seems
to run OK there's a couple of NPE issues though they're rare events (I had
to invent a way to make one occur), I've got a PR for one if you want to
test/review it for me :).
I'll try to update the wiki with instructions on running the resource
manager via marathon.
Darin
On Mon, Oct 5, 2015 at 4:27 PM, John Omernik wrote:
> I see. That makes sense. Thanks for the tip.
>
> Is it safe to pull down a recent version at this point? Are we using the
> official "master" or phase1? (the lazy man in me is asking for a link to
> the current repo so I don't have to read back over emails to see where I
> should go :)
>
>
>
> On Mon, Oct 5, 2015 at 3:25 PM, Darin Johnson
> wrote:
>
> > Hey John,
> >
> > Are you trying to run the resource manager from the tar ball via
> marathon?
> > It's doable, my suggested approach would be to use a json like this:
> >
> > {
> > "id": "resource-manager",
> > "uris": ["hdfs://namenode:port/dist/hadoop-2.7.0.tgz",
> > "hdfs://namenode:port/dist/conf/hadoop/yarn-site.xml",
> > "hdfs://namenode:port/dist/conf/hadoop/hdfs-site.xml",
> > "hdfs:///dist/conf/hadoop/core-site.xml",
> > "hdfs://namenode:port/dist/conf/hadoop/mapred-site.xml"],
> > "cmd": "cp *.xml hadoop-2.7.0/etc/hadoop && cd hadoop-2.7.0 && bin/yarn
> > resourcemanager",
> > "mem": 16,
> > "cpu": 1
> > "instances" : 1,
> > "user": "yarn"
> > }
> >
> > Basically it keeps you from redoing the tar ball every time you edit a
> > config, instead you just upload the new yarn-site.xml. The Node Manager
> > gets it's config from the Resource Manager (I'm assuming this is all for
> > remote distribution, otherwise creating the tar ball is optional).
> >
> > Darin
> >
> > On Mon, Oct 5, 2015 at 2:36 PM, John Omernik wrote:
> >
> > > Hey all, I've been waiting until the chaos of the code move has died
> > down.
> > > I am looking to get this working on my MapR cluster now, and would like
> > > some clarification on instructions here:
> > >
> > >
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/MYRIAD/Installing+for+Administrators
> > >
> > > Basically, in the instructions below, it has the "remove the
> > > yarn-site.xml. Yet to run the resource manager with myriad, you need
> the
> > > yarn-site to be packaged with things (unless I am reading that
> > incorrectly)
> > > Is the only option right now to created a tarball for nodemanagers, and
> > > have this be different from the tarball for the resource manager?
> > >
> > > Step 5: Create the Tarball
> > >
> > > The tarball has all of the files needed for the Node Managers and
> > Resource
> > > Managers. The following shows how to create the tarball and place it in
> > > HDFS:
> > > cd ~
> > > sudo cp -rp $YARN_HOME .
> > > sudo rm $YARN_HOME/etc/hadoop/yarn-site.xml
> > > sudo tar -zcpf ~/hadoop-2.7.1.tar.gz hadoop-2.7.1
> > > hadoop fs -put ~/hadoop-2.7.1.tar.gz /dist
> > >
> >
>
Re: Yarn-Site
hi John, I made that change because, after re-checking the original, I indicated removing the yarn-site.xml. I interpreted this snippet to mean that "sudo cp -rp $YARN_HOME ." copies everything to the current location and that a tarball is created. I also assumed (hopefully, not by mistake), that having the yarn-site.xml file there would have conflict issues since everything would be distributed via /dist. Anyway, Darin is taking a look at this page. --ruth On Mon, Oct 5, 2015 at 11:36 AM, John Omernik wrote: > Hey all, I've been waiting until the chaos of the code move has died down. > I am looking to get this working on my MapR cluster now, and would like > some clarification on instructions here: > > > > https://cwiki.apache.org/confluence/display/MYRIAD/Installing+for+Administrators > > Basically, in the instructions below, it has the "remove the > yarn-site.xml. Yet to run the resource manager with myriad, you need the > yarn-site to be packaged with things (unless I am reading that incorrectly) > Is the only option right now to created a tarball for nodemanagers, and > have this be different from the tarball for the resource manager? > > Step 5: Create the Tarball > > The tarball has all of the files needed for the Node Managers and Resource > Managers. The following shows how to create the tarball and place it in > HDFS: > cd ~ > sudo cp -rp $YARN_HOME . > sudo rm $YARN_HOME/etc/hadoop/yarn-site.xml > sudo tar -zcpf ~/hadoop-2.7.1.tar.gz hadoop-2.7.1 > hadoop fs -put ~/hadoop-2.7.1.tar.gz /dist > -- Ruth Harris Sr. Technical Writer, MapR
Re: Yarn-Site
I see. That makes sense. Thanks for the tip.
Is it safe to pull down a recent version at this point? Are we using the
official "master" or phase1? (the lazy man in me is asking for a link to
the current repo so I don't have to read back over emails to see where I
should go :)
On Mon, Oct 5, 2015 at 3:25 PM, Darin Johnson
wrote:
> Hey John,
>
> Are you trying to run the resource manager from the tar ball via marathon?
> It's doable, my suggested approach would be to use a json like this:
>
> {
> "id": "resource-manager",
> "uris": ["hdfs://namenode:port/dist/hadoop-2.7.0.tgz",
> "hdfs://namenode:port/dist/conf/hadoop/yarn-site.xml",
> "hdfs://namenode:port/dist/conf/hadoop/hdfs-site.xml",
> "hdfs:///dist/conf/hadoop/core-site.xml",
> "hdfs://namenode:port/dist/conf/hadoop/mapred-site.xml"],
> "cmd": "cp *.xml hadoop-2.7.0/etc/hadoop && cd hadoop-2.7.0 && bin/yarn
> resourcemanager",
> "mem": 16,
> "cpu": 1
> "instances" : 1,
> "user": "yarn"
> }
>
> Basically it keeps you from redoing the tar ball every time you edit a
> config, instead you just upload the new yarn-site.xml. The Node Manager
> gets it's config from the Resource Manager (I'm assuming this is all for
> remote distribution, otherwise creating the tar ball is optional).
>
> Darin
>
> On Mon, Oct 5, 2015 at 2:36 PM, John Omernik wrote:
>
> > Hey all, I've been waiting until the chaos of the code move has died
> down.
> > I am looking to get this working on my MapR cluster now, and would like
> > some clarification on instructions here:
> >
> >
> >
> >
> https://cwiki.apache.org/confluence/display/MYRIAD/Installing+for+Administrators
> >
> > Basically, in the instructions below, it has the "remove the
> > yarn-site.xml. Yet to run the resource manager with myriad, you need the
> > yarn-site to be packaged with things (unless I am reading that
> incorrectly)
> > Is the only option right now to created a tarball for nodemanagers, and
> > have this be different from the tarball for the resource manager?
> >
> > Step 5: Create the Tarball
> >
> > The tarball has all of the files needed for the Node Managers and
> Resource
> > Managers. The following shows how to create the tarball and place it in
> > HDFS:
> > cd ~
> > sudo cp -rp $YARN_HOME .
> > sudo rm $YARN_HOME/etc/hadoop/yarn-site.xml
> > sudo tar -zcpf ~/hadoop-2.7.1.tar.gz hadoop-2.7.1
> > hadoop fs -put ~/hadoop-2.7.1.tar.gz /dist
> >
>
Re: Yarn-Site
Hey John,
Are you trying to run the resource manager from the tar ball via marathon?
It's doable, my suggested approach would be to use a json like this:
{
"id": "resource-manager",
"uris": ["hdfs://namenode:port/dist/hadoop-2.7.0.tgz",
"hdfs://namenode:port/dist/conf/hadoop/yarn-site.xml",
"hdfs://namenode:port/dist/conf/hadoop/hdfs-site.xml",
"hdfs:///dist/conf/hadoop/core-site.xml",
"hdfs://namenode:port/dist/conf/hadoop/mapred-site.xml"],
"cmd": "cp *.xml hadoop-2.7.0/etc/hadoop && cd hadoop-2.7.0 && bin/yarn
resourcemanager",
"mem": 16,
"cpu": 1
"instances" : 1,
"user": "yarn"
}
Basically it keeps you from redoing the tar ball every time you edit a
config, instead you just upload the new yarn-site.xml. The Node Manager
gets it's config from the Resource Manager (I'm assuming this is all for
remote distribution, otherwise creating the tar ball is optional).
Darin
On Mon, Oct 5, 2015 at 2:36 PM, John Omernik wrote:
> Hey all, I've been waiting until the chaos of the code move has died down.
> I am looking to get this working on my MapR cluster now, and would like
> some clarification on instructions here:
>
>
>
> https://cwiki.apache.org/confluence/display/MYRIAD/Installing+for+Administrators
>
> Basically, in the instructions below, it has the "remove the
> yarn-site.xml. Yet to run the resource manager with myriad, you need the
> yarn-site to be packaged with things (unless I am reading that incorrectly)
> Is the only option right now to created a tarball for nodemanagers, and
> have this be different from the tarball for the resource manager?
>
> Step 5: Create the Tarball
>
> The tarball has all of the files needed for the Node Managers and Resource
> Managers. The following shows how to create the tarball and place it in
> HDFS:
> cd ~
> sudo cp -rp $YARN_HOME .
> sudo rm $YARN_HOME/etc/hadoop/yarn-site.xml
> sudo tar -zcpf ~/hadoop-2.7.1.tar.gz hadoop-2.7.1
> hadoop fs -put ~/hadoop-2.7.1.tar.gz /dist
>
Yarn-Site
Hey all, I've been waiting until the chaos of the code move has died down. I am looking to get this working on my MapR cluster now, and would like some clarification on instructions here: https://cwiki.apache.org/confluence/display/MYRIAD/Installing+for+Administrators Basically, in the instructions below, it has the "remove the yarn-site.xml. Yet to run the resource manager with myriad, you need the yarn-site to be packaged with things (unless I am reading that incorrectly) Is the only option right now to created a tarball for nodemanagers, and have this be different from the tarball for the resource manager? Step 5: Create the Tarball The tarball has all of the files needed for the Node Managers and Resource Managers. The following shows how to create the tarball and place it in HDFS: cd ~ sudo cp -rp $YARN_HOME . sudo rm $YARN_HOME/etc/hadoop/yarn-site.xml sudo tar -zcpf ~/hadoop-2.7.1.tar.gz hadoop-2.7.1 hadoop fs -put ~/hadoop-2.7.1.tar.gz /dist
Incubator PMC/Board report for Oct 2015 ([ppmc])
Dear podling, This email was sent by an automated system on behalf of the Apache Incubator PMC. It is an initial reminder to give you plenty of time to prepare your quarterly board report. The board meeting is scheduled for Wed, 21 October 2015, 10:30 am PST. The report for your podling will form a part of the Incubator PMC report. The Incubator PMC requires your report to be submitted 2 weeks before the board meeting, to allow sufficient time for review and submission (Wed, Oct 7th). Please submit your report with sufficient time to allow the incubator PMC, and subsequently board members to review and digest. Again, the very latest you should submit your report is 2 weeks prior to the board meeting. Thanks, The Apache Incubator PMC Submitting your Report -- Your report should contain the following: * Your project name * A brief description of your project, which assumes no knowledge of the project or necessarily of its field * A list of the three most important issues to address in the move towards graduation. * Any issues that the Incubator PMC or ASF Board might wish/need to be aware of * How has the community developed since the last report * How has the project developed since the last report. This should be appended to the Incubator Wiki page at: http://wiki.apache.org/incubator/October2015 Note: This is manually populated. You may need to wait a little before this page is created from a template. Mentors --- Mentors should review reports for their project(s) and sign them off on the Incubator wiki page. Signing off reports shows that you are following the project - projects that are not signed may raise alarms for the Incubator PMC. Incubator PMC
