Re: add to wiki

[Adam Bordelon]

Darin, you now have edit permissions on the wiki.
Santosh, you now have edit/admin permissions.

On Wed, Sep 9, 2015 at 12:37 PM, Santosh Marella 
wrote:

> Hi Darin,
>
>   I've added you to JIRA. Please let me know if you face problems using
> MYRIAD JIRA.
>
> Thanks,
> Santosh
>
> On Wed, Sep 9, 2015 at 12:01 PM, Darin Johnson 
> wrote:
>
> > Also, can I be added to JIRA.  Would like to assign some issues to myself
> > and close a few I completed.
> >
> > On Wed, Sep 9, 2015 at 1:53 PM, Santosh Marella 
> > wrote:
> >
> > > Please give me edit perms as well. Username: smarella
> > >
> > > Santosh
> > >
> > > On Wed, Sep 9, 2015 at 11:40 AM, Darin Johnson <
> dbjohnson1...@gmail.com>
> > > wrote:
> > >
> > > > could someone give me edit perm to the wiki?  Thanks, Darin.
> Username:
> > > > DarinJ
> > > >
> > >
> >
>

[jira] [Commented] (MYRIAD-130) Launching a profile missing from the myriad-config-default.yml throws null pointer exception

[Santosh Marella (JIRA)]

[ 
https://issues.apache.org/jira/browse/MYRIAD-130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14737537#comment-14737537
 ] 

Santosh Marella commented on MYRIAD-130:


[~Aashreya], I think this issue might have been fixed with the merge of PR 111 
(https://github.com/mesos/myriad/pull/111).

With PR 111, if you try to flexup using a profile name that's not defined in 
the myriad config (.yml), then the API request should fail.

> Launching a profile missing from the myriad-config-default.yml throws null 
> pointer exception
> 
>
> Key: MYRIAD-130
> URL: https://issues.apache.org/jira/browse/MYRIAD-130
> Project: Myriad
>  Issue Type: Bug
>  Components: Scheduler
>Reporter: Aashreya Ravi Shankar
>Assignee: Santosh Marella
>
> When trying to flex-up a profile which is missing from the myriad-config 
> file, null pointer exception is thrown.
> 15/09/08 10:34:38 ERROR mortbay.log: /api/cluster/flexup
> java.lang.NullPointerException
>   at com.ebay.myriad.state.SchedulerState.addNodes(SchedulerState.java:63)
>   at 
> com.ebay.myriad.scheduler.MyriadOperations.flexUpCluster(MyriadOperations.java:65)
>   at com.ebay.myriad.api.ClustersResource.flexUp(ClustersResource.java:83)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
>   at 
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
>   at 
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
>   at 
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
>   at 
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
>   at 
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
>   at 
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
>   at 
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
>   at 
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
>   at 
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
>   at 
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
>   at 
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
>   at 
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
>   at 
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
>   at 
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
>   at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>   at 
> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (MYRIAD-130) Launching a profile missing from the myriad-config-default.yml throws null pointer exception

[Santosh Marella (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MYRIAD-130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Santosh Marella updated MYRIAD-130:
---
Assignee: Santosh Marella  (was: Swapnil Daingade)

> Launching a profile missing from the myriad-config-default.yml throws null 
> pointer exception
> 
>
> Key: MYRIAD-130
> URL: https://issues.apache.org/jira/browse/MYRIAD-130
> Project: Myriad
>  Issue Type: Bug
>  Components: Scheduler
>Reporter: Aashreya Ravi Shankar
>Assignee: Santosh Marella
>
> When trying to flex-up a profile which is missing from the myriad-config 
> file, null pointer exception is thrown.
> 15/09/08 10:34:38 ERROR mortbay.log: /api/cluster/flexup
> java.lang.NullPointerException
>   at com.ebay.myriad.state.SchedulerState.addNodes(SchedulerState.java:63)
>   at 
> com.ebay.myriad.scheduler.MyriadOperations.flexUpCluster(MyriadOperations.java:65)
>   at com.ebay.myriad.api.ClustersResource.flexUp(ClustersResource.java:83)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
>   at 
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
>   at 
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
>   at 
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
>   at 
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
>   at 
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
>   at 
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
>   at 
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
>   at 
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
>   at 
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
>   at 
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
>   at 
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
>   at 
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
>   at 
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
>   at 
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
>   at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>   at 
> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (MYRIAD-130) Launching a profile missing from the myriad-config-default.yml throws null pointer exception

[Santosh Marella (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MYRIAD-130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Santosh Marella updated MYRIAD-130:
---
Assignee: Swapnil Daingade

> Launching a profile missing from the myriad-config-default.yml throws null 
> pointer exception
> 
>
> Key: MYRIAD-130
> URL: https://issues.apache.org/jira/browse/MYRIAD-130
> Project: Myriad
>  Issue Type: Bug
>  Components: Scheduler
>Reporter: Aashreya Ravi Shankar
>Assignee: Swapnil Daingade
>
> When trying to flex-up a profile which is missing from the myriad-config 
> file, null pointer exception is thrown.
> 15/09/08 10:34:38 ERROR mortbay.log: /api/cluster/flexup
> java.lang.NullPointerException
>   at com.ebay.myriad.state.SchedulerState.addNodes(SchedulerState.java:63)
>   at 
> com.ebay.myriad.scheduler.MyriadOperations.flexUpCluster(MyriadOperations.java:65)
>   at com.ebay.myriad.api.ClustersResource.flexUp(ClustersResource.java:83)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
>   at 
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
>   at 
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
>   at 
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
>   at 
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
>   at 
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
>   at 
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
>   at 
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
>   at 
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
>   at 
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
>   at 
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
>   at 
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
>   at 
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
>   at 
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
>   at 
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
>   at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>   at 
> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Question about the Wiki Instructions on yarn-site.xml

[Ruth Harris]

Thanks Darin1

--Ruth

On Wed, Sep 9, 2015 at 12:10 PM, John Omernik  wrote:

> As to editing directly, I am at a new employer, and we are trying to hash
> out if I can sign the Apache Committer Agreement stuff.  Thus, my thoughts,
> if the group wants them, will have to be in informal forum posts which I
> can't make any claim to from an IP perspective. I will work on the
> committers document approved, and do more directly with the Wiki, sorry for
> the roundaboutness.
>
> On Wed, Sep 9, 2015 at 1:07 PM, Ruth Harris  wrote:
>
> > Hi all,
> >
> > If you can clarify for me also about what the original instructions for
> the
> > Admin was trying to do and provide clearer information, I can update with
> > wiki information and then update the .markdown file in github.
> > Alternatively, the SME can update the wiki directly.
> >
> > I only walked through the config and build information associated with
> the
> > Developer information.
> >
> > Thanks, Ruth
> >
> > On Wed, Sep 9, 2015 at 8:52 AM, Darin Johnson 
> > wrote:
> >
> > > John,
> > > Understood I don't think making the tempdir be setup that way is ideal.
> > > We've had issues with other frameworks in the past.
> > > Darin
> > > On Sep 9, 2015 11:48 AM, "John Omernik"  wrote:
> > >
> > > > Well at this point my biggest issue the root user stuff in the other
> > > thread
> > > > and figuring out how to get it to work without making my slave's
> mesos
> > > temp
> > > > only writable by root (is there a work around? And is this a best
> > > practice
> > > > anyhow? what are the down stream effects of this etc)
> > > >
> > > > On Wed, Sep 9, 2015 at 10:45 AM, Darin Johnson <
> > dbjohnson1...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hey John I'm going to try to recreate issue using vanilla hadoop
> > later
> > > > > today.  Any other settings I should know about?
> > > > > Darin
> > > > > On Sep 9, 2015 9:42 AM, "John Omernik"  wrote:
> > > > >
> > > > > > This was another "slipped in" question in my other thread, I am
> > > > breaking
> > > > > > out for specific instructions.  Basically, I was struggling with
> > with
> > > > > some
> > > > > > things in the wiki on this page:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/MYRIAD/Installing+for+Administrators
> > > > > >
> > > > > > In step 5:
> > > > > > Step 5: Configure YARN to use Myriad
> > > > > >
> > > > > > Modify the */opt/hadoop-2.7.0/etc/hadoop/yarn-site.xml* file as
> > > > > instructed
> > > > > > in Sample: myriad-config-default.yml
> > > > > > <
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/MYRIAD/Sample%3A+myriad-config-default.yml
> > > > > > >
> > > > > > .
> > > > > >
> > > > > >
> > > > > > Issue 1: It should link to the yarn-site.xml page, not hte
> > > > > > myriad-config.default.yml page
> > > > > >
> > > > > > Issue 2:
> > > > > > It has us put that information in the yarn-site.xml This makes
> > sense.
> > > > > The
> > > > > > resource manager needs to be aware of the myriad stuff.
> > > > > >
> > > > > > Then I go to create a tarball, (which I SHOULD be able to use for
> > > both
> > > > > > resource manager and nodemanager... right?) However, the
> > instructions
> > > > > state
> > > > > > to remove the *.xml files.
> > > > > >
> > > > > > Step 6: Create the Tarball
> > > > > >
> > > > > > The tarball has all of the files needed for the Node Managers and
> > > > > Resource
> > > > > > Managers. The following shows how to create the tarball and place
> > it
> > > in
> > > > > > HDFS:
> > > > > > cd ~
> > > > > > sudo cp -rp /opt/hadoop-2.7.0 .
> > > > > > sudo rm hadoop-2.7.0/etc/hadoop/*.xml
> > > > > > sudo tar -zcpf ~/hadoop-2.7.0.tar.gz hadoop-2.7.0
> > > > > > hadoop fs -put ~/hadoop-2.7.0.tar.gz /dist
> > > > > >
> > > > > >
> > > > > > What I ended up doing... since I am running the resourcemanager
> > > > (myriad)
> > > > > in
> > > > > > marathon, is I created two tarballs. One is my
> > hadoop-2.7.0-RM.tar.gz
> > > > > which
> > > > > > has the all the xml files still in the tar ball for shipping to
> > > > marathon.
> > > > > > Then other is hadoop-2.7.0-NM.tar.gz which per the instructions
> > > removes
> > > > > the
> > > > > > *.xml files from the /etc/hadoop/ directory.
> > > > > >
> > > > > >
> > > > > > I guess... my logic is that myriad creates the conf directory for
> > the
> > > > > > nodemanagers... but then I thought, and I overthinking something?
> > Am
> > > I
> > > > > > missing something? Could that be factoring into what I am doing
> > here?
> > > > > >
> > > > > >
> > > > > > Obviously my first steps are to add the extra yarn-site.xml
> > entries,
> > > > but
> > > > > in
> > > > > > this current setup, they are only going into the resource manager
> > > > > yarn-site
> > > > > > as the the node-managers don't have a yarn-site in their
> > directories.
> > > > > Am I
> > > > > > looking at this correctly?  Perhaps we could rethink the removal
> > > > proc

Re: add to wiki

[Santosh Marella]

Hi Darin,

  I've added you to JIRA. Please let me know if you face problems using
MYRIAD JIRA.

Thanks,
Santosh

On Wed, Sep 9, 2015 at 12:01 PM, Darin Johnson 
wrote:

> Also, can I be added to JIRA.  Would like to assign some issues to myself
> and close a few I completed.
>
> On Wed, Sep 9, 2015 at 1:53 PM, Santosh Marella 
> wrote:
>
> > Please give me edit perms as well. Username: smarella
> >
> > Santosh
> >
> > On Wed, Sep 9, 2015 at 11:40 AM, Darin Johnson 
> > wrote:
> >
> > > could someone give me edit perm to the wiki?  Thanks, Darin.  Username:
> > > DarinJ
> > >
> >
>

Re: Question about the Wiki Instructions on yarn-site.xml

[John Omernik]

As to editing directly, I am at a new employer, and we are trying to hash
out if I can sign the Apache Committer Agreement stuff.  Thus, my thoughts,
if the group wants them, will have to be in informal forum posts which I
can't make any claim to from an IP perspective. I will work on the
committers document approved, and do more directly with the Wiki, sorry for
the roundaboutness.

On Wed, Sep 9, 2015 at 1:07 PM, Ruth Harris  wrote:

> Hi all,
>
> If you can clarify for me also about what the original instructions for the
> Admin was trying to do and provide clearer information, I can update with
> wiki information and then update the .markdown file in github.
> Alternatively, the SME can update the wiki directly.
>
> I only walked through the config and build information associated with the
> Developer information.
>
> Thanks, Ruth
>
> On Wed, Sep 9, 2015 at 8:52 AM, Darin Johnson 
> wrote:
>
> > John,
> > Understood I don't think making the tempdir be setup that way is ideal.
> > We've had issues with other frameworks in the past.
> > Darin
> > On Sep 9, 2015 11:48 AM, "John Omernik"  wrote:
> >
> > > Well at this point my biggest issue the root user stuff in the other
> > thread
> > > and figuring out how to get it to work without making my slave's mesos
> > temp
> > > only writable by root (is there a work around? And is this a best
> > practice
> > > anyhow? what are the down stream effects of this etc)
> > >
> > > On Wed, Sep 9, 2015 at 10:45 AM, Darin Johnson <
> dbjohnson1...@gmail.com>
> > > wrote:
> > >
> > > > Hey John I'm going to try to recreate issue using vanilla hadoop
> later
> > > > today.  Any other settings I should know about?
> > > > Darin
> > > > On Sep 9, 2015 9:42 AM, "John Omernik"  wrote:
> > > >
> > > > > This was another "slipped in" question in my other thread, I am
> > > breaking
> > > > > out for specific instructions.  Basically, I was struggling with
> with
> > > > some
> > > > > things in the wiki on this page:
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/MYRIAD/Installing+for+Administrators
> > > > >
> > > > > In step 5:
> > > > > Step 5: Configure YARN to use Myriad
> > > > >
> > > > > Modify the */opt/hadoop-2.7.0/etc/hadoop/yarn-site.xml* file as
> > > > instructed
> > > > > in Sample: myriad-config-default.yml
> > > > > <
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/MYRIAD/Sample%3A+myriad-config-default.yml
> > > > > >
> > > > > .
> > > > >
> > > > >
> > > > > Issue 1: It should link to the yarn-site.xml page, not hte
> > > > > myriad-config.default.yml page
> > > > >
> > > > > Issue 2:
> > > > > It has us put that information in the yarn-site.xml This makes
> sense.
> > > > The
> > > > > resource manager needs to be aware of the myriad stuff.
> > > > >
> > > > > Then I go to create a tarball, (which I SHOULD be able to use for
> > both
> > > > > resource manager and nodemanager... right?) However, the
> instructions
> > > > state
> > > > > to remove the *.xml files.
> > > > >
> > > > > Step 6: Create the Tarball
> > > > >
> > > > > The tarball has all of the files needed for the Node Managers and
> > > > Resource
> > > > > Managers. The following shows how to create the tarball and place
> it
> > in
> > > > > HDFS:
> > > > > cd ~
> > > > > sudo cp -rp /opt/hadoop-2.7.0 .
> > > > > sudo rm hadoop-2.7.0/etc/hadoop/*.xml
> > > > > sudo tar -zcpf ~/hadoop-2.7.0.tar.gz hadoop-2.7.0
> > > > > hadoop fs -put ~/hadoop-2.7.0.tar.gz /dist
> > > > >
> > > > >
> > > > > What I ended up doing... since I am running the resourcemanager
> > > (myriad)
> > > > in
> > > > > marathon, is I created two tarballs. One is my
> hadoop-2.7.0-RM.tar.gz
> > > > which
> > > > > has the all the xml files still in the tar ball for shipping to
> > > marathon.
> > > > > Then other is hadoop-2.7.0-NM.tar.gz which per the instructions
> > removes
> > > > the
> > > > > *.xml files from the /etc/hadoop/ directory.
> > > > >
> > > > >
> > > > > I guess... my logic is that myriad creates the conf directory for
> the
> > > > > nodemanagers... but then I thought, and I overthinking something?
> Am
> > I
> > > > > missing something? Could that be factoring into what I am doing
> here?
> > > > >
> > > > >
> > > > > Obviously my first steps are to add the extra yarn-site.xml
> entries,
> > > but
> > > > in
> > > > > this current setup, they are only going into the resource manager
> > > > yarn-site
> > > > > as the the node-managers don't have a yarn-site in their
> directories.
> > > > Am I
> > > > > looking at this correctly?  Perhaps we could rethink the removal
> > > process
> > > > of
> > > > > the XML files in the tarball to allow this to work correctly with a
> > > > single
> > > > > tarball?
> > > > >
> > > > > If I am missing something here, please advise!
> > > > >
> > > > >
> > > > > John
> > > > >
> > > >
> > >
> >
>
>
>
> --
> Ruth Harris
> Sr. Technical Writer, MapR
>

Re: add to wiki

[Darin Johnson]

Also, can I be added to JIRA.  Would like to assign some issues to myself
and close a few I completed.

On Wed, Sep 9, 2015 at 1:53 PM, Santosh Marella 
wrote:

> Please give me edit perms as well. Username: smarella
>
> Santosh
>
> On Wed, Sep 9, 2015 at 11:40 AM, Darin Johnson 
> wrote:
>
> > could someone give me edit perm to the wiki?  Thanks, Darin.  Username:
> > DarinJ
> >
>

Re: add to wiki

[Santosh Marella]

Please give me edit perms as well. Username: smarella

Santosh

On Wed, Sep 9, 2015 at 11:40 AM, Darin Johnson 
wrote:

> could someone give me edit perm to the wiki?  Thanks, Darin.  Username:
> DarinJ
>

add to wiki

[Darin Johnson]

could someone give me edit perm to the wiki?  Thanks, Darin.  Username:
DarinJ

Re: Question about the Wiki Instructions on yarn-site.xml

[Darin Johnson]

Ruth, I wrote most of the remote distribution code and am going over John's
issue now.  Will inform you later today/tomorrow and/or update the wiki.

Darin

On Wed, Sep 9, 2015 at 2:07 PM, Ruth Harris  wrote:

> Hi all,
>
> If you can clarify for me also about what the original instructions for the
> Admin was trying to do and provide clearer information, I can update with
> wiki information and then update the .markdown file in github.
> Alternatively, the SME can update the wiki directly.
>
> I only walked through the config and build information associated with the
> Developer information.
>
> Thanks, Ruth
>
> On Wed, Sep 9, 2015 at 8:52 AM, Darin Johnson 
> wrote:
>
> > John,
> > Understood I don't think making the tempdir be setup that way is ideal.
> > We've had issues with other frameworks in the past.
> > Darin
> > On Sep 9, 2015 11:48 AM, "John Omernik"  wrote:
> >
> > > Well at this point my biggest issue the root user stuff in the other
> > thread
> > > and figuring out how to get it to work without making my slave's mesos
> > temp
> > > only writable by root (is there a work around? And is this a best
> > practice
> > > anyhow? what are the down stream effects of this etc)
> > >
> > > On Wed, Sep 9, 2015 at 10:45 AM, Darin Johnson <
> dbjohnson1...@gmail.com>
> > > wrote:
> > >
> > > > Hey John I'm going to try to recreate issue using vanilla hadoop
> later
> > > > today.  Any other settings I should know about?
> > > > Darin
> > > > On Sep 9, 2015 9:42 AM, "John Omernik"  wrote:
> > > >
> > > > > This was another "slipped in" question in my other thread, I am
> > > breaking
> > > > > out for specific instructions.  Basically, I was struggling with
> with
> > > > some
> > > > > things in the wiki on this page:
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/MYRIAD/Installing+for+Administrators
> > > > >
> > > > > In step 5:
> > > > > Step 5: Configure YARN to use Myriad
> > > > >
> > > > > Modify the */opt/hadoop-2.7.0/etc/hadoop/yarn-site.xml* file as
> > > > instructed
> > > > > in Sample: myriad-config-default.yml
> > > > > <
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/MYRIAD/Sample%3A+myriad-config-default.yml
> > > > > >
> > > > > .
> > > > >
> > > > >
> > > > > Issue 1: It should link to the yarn-site.xml page, not hte
> > > > > myriad-config.default.yml page
> > > > >
> > > > > Issue 2:
> > > > > It has us put that information in the yarn-site.xml This makes
> sense.
> > > > The
> > > > > resource manager needs to be aware of the myriad stuff.
> > > > >
> > > > > Then I go to create a tarball, (which I SHOULD be able to use for
> > both
> > > > > resource manager and nodemanager... right?) However, the
> instructions
> > > > state
> > > > > to remove the *.xml files.
> > > > >
> > > > > Step 6: Create the Tarball
> > > > >
> > > > > The tarball has all of the files needed for the Node Managers and
> > > > Resource
> > > > > Managers. The following shows how to create the tarball and place
> it
> > in
> > > > > HDFS:
> > > > > cd ~
> > > > > sudo cp -rp /opt/hadoop-2.7.0 .
> > > > > sudo rm hadoop-2.7.0/etc/hadoop/*.xml
> > > > > sudo tar -zcpf ~/hadoop-2.7.0.tar.gz hadoop-2.7.0
> > > > > hadoop fs -put ~/hadoop-2.7.0.tar.gz /dist
> > > > >
> > > > >
> > > > > What I ended up doing... since I am running the resourcemanager
> > > (myriad)
> > > > in
> > > > > marathon, is I created two tarballs. One is my
> hadoop-2.7.0-RM.tar.gz
> > > > which
> > > > > has the all the xml files still in the tar ball for shipping to
> > > marathon.
> > > > > Then other is hadoop-2.7.0-NM.tar.gz which per the instructions
> > removes
> > > > the
> > > > > *.xml files from the /etc/hadoop/ directory.
> > > > >
> > > > >
> > > > > I guess... my logic is that myriad creates the conf directory for
> the
> > > > > nodemanagers... but then I thought, and I overthinking something?
> Am
> > I
> > > > > missing something? Could that be factoring into what I am doing
> here?
> > > > >
> > > > >
> > > > > Obviously my first steps are to add the extra yarn-site.xml
> entries,
> > > but
> > > > in
> > > > > this current setup, they are only going into the resource manager
> > > > yarn-site
> > > > > as the the node-managers don't have a yarn-site in their
> directories.
> > > > Am I
> > > > > looking at this correctly?  Perhaps we could rethink the removal
> > > process
> > > > of
> > > > > the XML files in the tarball to allow this to work correctly with a
> > > > single
> > > > > tarball?
> > > > >
> > > > > If I am missing something here, please advise!
> > > > >
> > > > >
> > > > > John
> > > > >
> > > >
> > >
> >
>
>
>
> --
> Ruth Harris
> Sr. Technical Writer, MapR
>

Re: Question about the Wiki Instructions on yarn-site.xml

[Ruth Harris]

Hi all,

If you can clarify for me also about what the original instructions for the
Admin was trying to do and provide clearer information, I can update with
wiki information and then update the .markdown file in github.
Alternatively, the SME can update the wiki directly.

I only walked through the config and build information associated with the
Developer information.

Thanks, Ruth

On Wed, Sep 9, 2015 at 8:52 AM, Darin Johnson 
wrote:

> John,
> Understood I don't think making the tempdir be setup that way is ideal.
> We've had issues with other frameworks in the past.
> Darin
> On Sep 9, 2015 11:48 AM, "John Omernik"  wrote:
>
> > Well at this point my biggest issue the root user stuff in the other
> thread
> > and figuring out how to get it to work without making my slave's mesos
> temp
> > only writable by root (is there a work around? And is this a best
> practice
> > anyhow? what are the down stream effects of this etc)
> >
> > On Wed, Sep 9, 2015 at 10:45 AM, Darin Johnson 
> > wrote:
> >
> > > Hey John I'm going to try to recreate issue using vanilla hadoop later
> > > today.  Any other settings I should know about?
> > > Darin
> > > On Sep 9, 2015 9:42 AM, "John Omernik"  wrote:
> > >
> > > > This was another "slipped in" question in my other thread, I am
> > breaking
> > > > out for specific instructions.  Basically, I was struggling with with
> > > some
> > > > things in the wiki on this page:
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/MYRIAD/Installing+for+Administrators
> > > >
> > > > In step 5:
> > > > Step 5: Configure YARN to use Myriad
> > > >
> > > > Modify the */opt/hadoop-2.7.0/etc/hadoop/yarn-site.xml* file as
> > > instructed
> > > > in Sample: myriad-config-default.yml
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/MYRIAD/Sample%3A+myriad-config-default.yml
> > > > >
> > > > .
> > > >
> > > >
> > > > Issue 1: It should link to the yarn-site.xml page, not hte
> > > > myriad-config.default.yml page
> > > >
> > > > Issue 2:
> > > > It has us put that information in the yarn-site.xml This makes sense.
> > > The
> > > > resource manager needs to be aware of the myriad stuff.
> > > >
> > > > Then I go to create a tarball, (which I SHOULD be able to use for
> both
> > > > resource manager and nodemanager... right?) However, the instructions
> > > state
> > > > to remove the *.xml files.
> > > >
> > > > Step 6: Create the Tarball
> > > >
> > > > The tarball has all of the files needed for the Node Managers and
> > > Resource
> > > > Managers. The following shows how to create the tarball and place it
> in
> > > > HDFS:
> > > > cd ~
> > > > sudo cp -rp /opt/hadoop-2.7.0 .
> > > > sudo rm hadoop-2.7.0/etc/hadoop/*.xml
> > > > sudo tar -zcpf ~/hadoop-2.7.0.tar.gz hadoop-2.7.0
> > > > hadoop fs -put ~/hadoop-2.7.0.tar.gz /dist
> > > >
> > > >
> > > > What I ended up doing... since I am running the resourcemanager
> > (myriad)
> > > in
> > > > marathon, is I created two tarballs. One is my hadoop-2.7.0-RM.tar.gz
> > > which
> > > > has the all the xml files still in the tar ball for shipping to
> > marathon.
> > > > Then other is hadoop-2.7.0-NM.tar.gz which per the instructions
> removes
> > > the
> > > > *.xml files from the /etc/hadoop/ directory.
> > > >
> > > >
> > > > I guess... my logic is that myriad creates the conf directory for the
> > > > nodemanagers... but then I thought, and I overthinking something? Am
> I
> > > > missing something? Could that be factoring into what I am doing here?
> > > >
> > > >
> > > > Obviously my first steps are to add the extra yarn-site.xml entries,
> > but
> > > in
> > > > this current setup, they are only going into the resource manager
> > > yarn-site
> > > > as the the node-managers don't have a yarn-site in their directories.
> > > Am I
> > > > looking at this correctly?  Perhaps we could rethink the removal
> > process
> > > of
> > > > the XML files in the tarball to allow this to work correctly with a
> > > single
> > > > tarball?
> > > >
> > > > If I am missing something here, please advise!
> > > >
> > > >
> > > > John
> > > >
> > >
> >
>



-- 
Ruth Harris
Sr. Technical Writer, MapR

Re: Hangout today?

[Adam Bordelon]

I'm in the hangout now, and I've opened it up to others.
Please join: https://plus.google.com/hangouts/_/mesosphere.io/myriad

On Wed, Sep 9, 2015 at 8:55 AM, Darin Johnson 
wrote:

> Eom
>

Re: Getting Nodes to be "Running" in Mesos

[Santosh Marella]

> ready right away (1 minute delay after kicking off Myriad)
That's true. The reason a medium NM instance is kicked off at RM's startup
was to allow non-zero capacity in the cluster, without which YARN seems
to reject app submissions. I'll look at YARN's code base more carefully
and see if this behavior can be disabled via configuration
(I didn't notice such option last timeI looked).

The other workaround might be to reduce the 1 min delay in mesos-dns
to create DNS entries for mesos tasks. Not sure if that's recommended
in production, but sometimes I felt 1 min is too long to create a DNS entry.
If RM failsover, it means the new RM instance can't be discovered for 1 min.
Job's that were previously running have to wait > 1 min to resume.

Santosh

On Tue, Sep 8, 2015 at 1:23 PM, John Omernik  wrote:

> Also a side note:  The Flexing up and now having to have at least one node
> manager specified at startup:
>
> nmInstances: # NMs to start with. Requires at least 1 NM with a non-zero
> profile.
>
>   medium: 1 # 
>
>
> Is going to lead to task failures with mesos dns because the name won't be
> ready right away (1 minute delay after kicking off Myriad) do we NEED to
> have a non-0 profile nodemanager startup with the resource manager?
>
> On Tue, Sep 8, 2015 at 3:16 PM, John Omernik  wrote:
>
> > Cool.  Question about the yarn-site.xml in general.
> >
> > I was struggling with some things in the wiki on this page:
> >
> https://cwiki.apache.org/confluence/display/MYRIAD/Installing+for+Administrators
> >
> > Basically in step 5:
> > Step 5: Configure YARN to use Myriad
> >
> > Modify the */opt/hadoop-2.7.0/etc/hadoop/yarn-site.xml* file as
> > instructed in Sample: myriad-config-default.yml
> > <
> https://cwiki.apache.org/confluence/display/MYRIAD/Sample%3A+myriad-config-default.yml
> >
> > .
> >
> >
> > (It should not link to the yml, but to the yarn site, side issue) it has
> > us put that information in the yarn-site.xml This makes sense.  The
> > resource manager needs to be aware of the myriad stuff.
> >
> > Then I go to create a tarbal, (which I SHOULD be able to use for both
> > resource manager and nodemanager... right?) However, the instructions
> state
> > to remove the *.xml files.
> >
> > Step 6: Create the Tarball
> >
> > The tarball has all of the files needed for the Node Managers and
> > Resource Managers. The following shows how to create the tarball and
> place
> > it in HDFS:
> > cd ~
> > sudo cp -rp /opt/hadoop-2.7.0 .
> > sudo rm hadoop-2.7.0/etc/hadoop/*.xml
> > sudo tar -zcpf ~/hadoop-2.7.0.tar.gz hadoop-2.7.0
> > hadoop fs -put ~/hadoop-2.7.0.tar.gz /dist
> >
> >
> > What I ended up doing... since I am running the resourcemanager (myriad)
> > in marathon, is I created two tarballs. One is my hadoop-2.7.0-RM.tar.gz
> > which has the all the xml files still in the tar ball for shipping to
> > marathon. Then other is hadoop-2.7.0-NM.tar.gz which per the instructions
> > removes the *.xml files from the /etc/hadoop/ directory.
> >
> >
> > I guess... my logic is that myriad creates the conf directory for the
> > nodemanagers... but then I thought, and I overthinking something? Am I
> > missing something? Could that be factoring into what I am doing here?
> >
> >
> > Obviously my first steps are to add the extra yarn-site.xml entries, but
> > in this current setup, they are only going into the resource manager
> > yarn-site as the the node-managers don't have a yarn-site in their
> > directories.
> >
> >
> >
> >
> >
> >
> >
> > On Tue, Sep 8, 2015 at 3:09 PM, yuliya Feldman <
> > yufeld...@yahoo.com.invalid> wrote:
> >
> >> Take a look at :   https://github.com/mesos/myriad/pull/128
> >> for yarn-site.xml updates
> >>
> >>   From: John Omernik 
> >>  To: dev@myriad.incubator.apache.org
> >>  Sent: Tuesday, September 8, 2015 12:38 PM
> >>  Subject: Getting Nodes to be "Running" in Mesos
> >>
> >> So I am playing around with a recent build of Myriad, and I am using
> MapR
> >> 5.0 (hadoop-2.7.0) I hate to use the dev list as a "help Myriad won't
> run"
> >> forum, so please forgive me if I am using the list wrong.
> >>
> >> Basically, I seem to be able to get myriad running, and the things up,
> and
> >> it tries to start a nodemanager.
> >>
> >> In mesos, the status of the nodemanager task never gets past staging,
> and
> >> eventually, fails.  The logs for both the node manager and myriad, seem
> to
> >> look healthy, and I am not sure where I should look next to troubleshoot
> >> what is happening. Basically you can see the registration of the
> >> nodemanager, and then it fails with no error in the logs... Any thoughts
> >> would be appreciated on where I can look next for troubleshooting.
> >>
> >>
> >> Node Manager Logs (complete)
> >>
> >> STARTUP_MSG:  build = g...@github.com:mapr/private-hadoop-common.git
> >> -r fc95119f587541fb3a9af0dbeeed23c974178115; compiled by 'root' on
> >> 2015-08-19T20:02Z
> >> STARTUP_MSG:  java = 1.8.0_45-internal
> >> ***

Hangout today?

[Darin Johnson]

Eom

Re: Question about the Wiki Instructions on yarn-site.xml

[Darin Johnson]

John,
Understood I don't think making the tempdir be setup that way is ideal.
We've had issues with other frameworks in the past.
Darin
On Sep 9, 2015 11:48 AM, "John Omernik"  wrote:

> Well at this point my biggest issue the root user stuff in the other thread
> and figuring out how to get it to work without making my slave's mesos temp
> only writable by root (is there a work around? And is this a best practice
> anyhow? what are the down stream effects of this etc)
>
> On Wed, Sep 9, 2015 at 10:45 AM, Darin Johnson 
> wrote:
>
> > Hey John I'm going to try to recreate issue using vanilla hadoop later
> > today.  Any other settings I should know about?
> > Darin
> > On Sep 9, 2015 9:42 AM, "John Omernik"  wrote:
> >
> > > This was another "slipped in" question in my other thread, I am
> breaking
> > > out for specific instructions.  Basically, I was struggling with with
> > some
> > > things in the wiki on this page:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/MYRIAD/Installing+for+Administrators
> > >
> > > In step 5:
> > > Step 5: Configure YARN to use Myriad
> > >
> > > Modify the */opt/hadoop-2.7.0/etc/hadoop/yarn-site.xml* file as
> > instructed
> > > in Sample: myriad-config-default.yml
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/MYRIAD/Sample%3A+myriad-config-default.yml
> > > >
> > > .
> > >
> > >
> > > Issue 1: It should link to the yarn-site.xml page, not hte
> > > myriad-config.default.yml page
> > >
> > > Issue 2:
> > > It has us put that information in the yarn-site.xml This makes sense.
> > The
> > > resource manager needs to be aware of the myriad stuff.
> > >
> > > Then I go to create a tarball, (which I SHOULD be able to use for both
> > > resource manager and nodemanager... right?) However, the instructions
> > state
> > > to remove the *.xml files.
> > >
> > > Step 6: Create the Tarball
> > >
> > > The tarball has all of the files needed for the Node Managers and
> > Resource
> > > Managers. The following shows how to create the tarball and place it in
> > > HDFS:
> > > cd ~
> > > sudo cp -rp /opt/hadoop-2.7.0 .
> > > sudo rm hadoop-2.7.0/etc/hadoop/*.xml
> > > sudo tar -zcpf ~/hadoop-2.7.0.tar.gz hadoop-2.7.0
> > > hadoop fs -put ~/hadoop-2.7.0.tar.gz /dist
> > >
> > >
> > > What I ended up doing... since I am running the resourcemanager
> (myriad)
> > in
> > > marathon, is I created two tarballs. One is my hadoop-2.7.0-RM.tar.gz
> > which
> > > has the all the xml files still in the tar ball for shipping to
> marathon.
> > > Then other is hadoop-2.7.0-NM.tar.gz which per the instructions removes
> > the
> > > *.xml files from the /etc/hadoop/ directory.
> > >
> > >
> > > I guess... my logic is that myriad creates the conf directory for the
> > > nodemanagers... but then I thought, and I overthinking something? Am I
> > > missing something? Could that be factoring into what I am doing here?
> > >
> > >
> > > Obviously my first steps are to add the extra yarn-site.xml entries,
> but
> > in
> > > this current setup, they are only going into the resource manager
> > yarn-site
> > > as the the node-managers don't have a yarn-site in their directories.
> > Am I
> > > looking at this correctly?  Perhaps we could rethink the removal
> process
> > of
> > > the XML files in the tarball to allow this to work correctly with a
> > single
> > > tarball?
> > >
> > > If I am missing something here, please advise!
> > >
> > >
> > > John
> > >
> >
>

Re: Question about the Wiki Instructions on yarn-site.xml

[John Omernik]

Well at this point my biggest issue the root user stuff in the other thread
and figuring out how to get it to work without making my slave's mesos temp
only writable by root (is there a work around? And is this a best practice
anyhow? what are the down stream effects of this etc)

On Wed, Sep 9, 2015 at 10:45 AM, Darin Johnson 
wrote:

> Hey John I'm going to try to recreate issue using vanilla hadoop later
> today.  Any other settings I should know about?
> Darin
> On Sep 9, 2015 9:42 AM, "John Omernik"  wrote:
>
> > This was another "slipped in" question in my other thread, I am breaking
> > out for specific instructions.  Basically, I was struggling with with
> some
> > things in the wiki on this page:
> >
> >
> https://cwiki.apache.org/confluence/display/MYRIAD/Installing+for+Administrators
> >
> > In step 5:
> > Step 5: Configure YARN to use Myriad
> >
> > Modify the */opt/hadoop-2.7.0/etc/hadoop/yarn-site.xml* file as
> instructed
> > in Sample: myriad-config-default.yml
> > <
> >
> https://cwiki.apache.org/confluence/display/MYRIAD/Sample%3A+myriad-config-default.yml
> > >
> > .
> >
> >
> > Issue 1: It should link to the yarn-site.xml page, not hte
> > myriad-config.default.yml page
> >
> > Issue 2:
> > It has us put that information in the yarn-site.xml This makes sense.
> The
> > resource manager needs to be aware of the myriad stuff.
> >
> > Then I go to create a tarball, (which I SHOULD be able to use for both
> > resource manager and nodemanager... right?) However, the instructions
> state
> > to remove the *.xml files.
> >
> > Step 6: Create the Tarball
> >
> > The tarball has all of the files needed for the Node Managers and
> Resource
> > Managers. The following shows how to create the tarball and place it in
> > HDFS:
> > cd ~
> > sudo cp -rp /opt/hadoop-2.7.0 .
> > sudo rm hadoop-2.7.0/etc/hadoop/*.xml
> > sudo tar -zcpf ~/hadoop-2.7.0.tar.gz hadoop-2.7.0
> > hadoop fs -put ~/hadoop-2.7.0.tar.gz /dist
> >
> >
> > What I ended up doing... since I am running the resourcemanager (myriad)
> in
> > marathon, is I created two tarballs. One is my hadoop-2.7.0-RM.tar.gz
> which
> > has the all the xml files still in the tar ball for shipping to marathon.
> > Then other is hadoop-2.7.0-NM.tar.gz which per the instructions removes
> the
> > *.xml files from the /etc/hadoop/ directory.
> >
> >
> > I guess... my logic is that myriad creates the conf directory for the
> > nodemanagers... but then I thought, and I overthinking something? Am I
> > missing something? Could that be factoring into what I am doing here?
> >
> >
> > Obviously my first steps are to add the extra yarn-site.xml entries, but
> in
> > this current setup, they are only going into the resource manager
> yarn-site
> > as the the node-managers don't have a yarn-site in their directories.
> Am I
> > looking at this correctly?  Perhaps we could rethink the removal process
> of
> > the XML files in the tarball to allow this to work correctly with a
> single
> > tarball?
> >
> > If I am missing something here, please advise!
> >
> >
> > John
> >
>

Re: Question about the Wiki Instructions on yarn-site.xml

[Darin Johnson]

Hey John I'm going to try to recreate issue using vanilla hadoop later
today.  Any other settings I should know about?
Darin
On Sep 9, 2015 9:42 AM, "John Omernik"  wrote:

> This was another "slipped in" question in my other thread, I am breaking
> out for specific instructions.  Basically, I was struggling with with some
> things in the wiki on this page:
>
> https://cwiki.apache.org/confluence/display/MYRIAD/Installing+for+Administrators
>
> In step 5:
> Step 5: Configure YARN to use Myriad
>
> Modify the */opt/hadoop-2.7.0/etc/hadoop/yarn-site.xml* file as instructed
> in Sample: myriad-config-default.yml
> <
> https://cwiki.apache.org/confluence/display/MYRIAD/Sample%3A+myriad-config-default.yml
> >
> .
>
>
> Issue 1: It should link to the yarn-site.xml page, not hte
> myriad-config.default.yml page
>
> Issue 2:
> It has us put that information in the yarn-site.xml This makes sense.  The
> resource manager needs to be aware of the myriad stuff.
>
> Then I go to create a tarball, (which I SHOULD be able to use for both
> resource manager and nodemanager... right?) However, the instructions state
> to remove the *.xml files.
>
> Step 6: Create the Tarball
>
> The tarball has all of the files needed for the Node Managers and  Resource
> Managers. The following shows how to create the tarball and place it in
> HDFS:
> cd ~
> sudo cp -rp /opt/hadoop-2.7.0 .
> sudo rm hadoop-2.7.0/etc/hadoop/*.xml
> sudo tar -zcpf ~/hadoop-2.7.0.tar.gz hadoop-2.7.0
> hadoop fs -put ~/hadoop-2.7.0.tar.gz /dist
>
>
> What I ended up doing... since I am running the resourcemanager (myriad) in
> marathon, is I created two tarballs. One is my hadoop-2.7.0-RM.tar.gz which
> has the all the xml files still in the tar ball for shipping to marathon.
> Then other is hadoop-2.7.0-NM.tar.gz which per the instructions removes the
> *.xml files from the /etc/hadoop/ directory.
>
>
> I guess... my logic is that myriad creates the conf directory for the
> nodemanagers... but then I thought, and I overthinking something? Am I
> missing something? Could that be factoring into what I am doing here?
>
>
> Obviously my first steps are to add the extra yarn-site.xml entries, but in
> this current setup, they are only going into the resource manager yarn-site
> as the the node-managers don't have a yarn-site in their directories.  Am I
> looking at this correctly?  Perhaps we could rethink the removal process of
> the XML files in the tarball to allow this to work correctly with a single
> tarball?
>
> If I am missing something here, please advise!
>
>
> John
>

Re: Getting Nodes to be "Running" in Mesos

[John Omernik]

So focusing on this issue to run Myriad at this point, we would need to

1. Run Myriad as root (i.e. in marathon "user":"root",  must be added to
the json so it runs as root)
2. Have the frameworkUser be root
3. Have the frameoworkSuperUser either be root or be someone who can
passwordlessly sudo to root.
4. Have the entire path of the slave work-dir be owned by root and only
writable by root up to where the container-executor.cfg exists.

On point 4, so for me I am running my slaves pointing to a work directory
that is /opt/mapr/mesos/tmp/slave in that I have some space issues on some
of my nodes /.   Even if I ran it to /tmp I would run into the same
problem. If I found a new place to put the work directly on every slave,
where it was root writable from / to the .cfg file, then it would work.
But, would other frameworks fail? Or would their chown process actually fix
things so they could write?  This seems like a huge work around to get
Myriad running.

At this point is there another way to get Myriad or is running all as root
the only way? Just trying to get myriad back up and running here.



On Tue, Sep 8, 2015 at 9:30 PM, Darin Johnson 
wrote:

> Yuliya, the reason for the chown framework user . is that the the executor
> (as frameworkUser) must write some files the the MESOS_DIRECTORY,
> specifically stderr, stdout and at the time the capsule dir (now
> obsolete).  I suppose we could touch these files and then give them the
> proper permissions.
>
> I was planning to remove a lot of the code once MESOS-1790 is resolved, Jim
> submitted a patch already.  In particular, there would no longer be a
> frameworkSuperUser (it's there so we can extract the tarball and preserve
> ownership/permissions for container-executor), and the frameworkUser would
> just run the yarn nodemanger.  If we continue to require the
> MESOS_DIRECTORY to be owned by root and we'll be required to continue to
> run it in a way similar to it is currently.  I really don't like the idea
> of running frameworks as root or even with passwordless sudo if I can help
> it, but at the time it was the only work around.
>
> So I guess the question is frameworkSuperUser something that we'd like to
> eventually depricate or is it here for good?  Also, I should comment on
> Mesos-1790 to see what's going on with the patch.
>
> Darin
>
>
>
> On Sep 8, 2015 7:12 PM, "yuliya Feldman" 
> wrote:
>
> > John,
> > It is a problem with permissions for container-executor.cfg - it requires
> > whole path to it to be owned by root.
> > One step is to change work-dir for mesos-slave to point to a different
> > directory (not tmp) that is writable only by root.
> > It still does not solve full issue since binary distro is changing
> > permissions of the distro directory to a framework user.
> > If framework user is root and myriad is running as root it can be solved,
> > otherwise we need changes to binary distro code.
> > I was planning to do it, but got distracted by other stuff. Will try to
> > look at it this week.
> > Thanks,Yuliya
> >   From: John Omernik 
> >  To: dev@myriad.incubator.apache.org; yuliya Feldman <
> yufeld...@yahoo.com>
> >  Sent: Tuesday, September 8, 2015 1:31 PM
> >  Subject: Re: Getting Nodes to be "Running" in Mesos
> >
> > interesting... when I did root as the framework user then I got this:
> >
> > ExitCodeException exitCode=24: File /tmp must not be world or group
> > writable, but is 1777
> >
> > at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
> > at org.apache.hadoop.util.Shell.run(Shell.java:456)
> > at
> > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
> > at
> >
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:182)
> > at
> >
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:210)
> > at
> > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> > at
> >
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:463)
> > at
> >
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:511)
> > 15/09/08 15:30:38 INFO nodemanager.ContainerExecutor:
> > 15/09/08 15:30:38 INFO service.AbstractService: Service NodeManager
> > failed in state INITED; cause:
> > org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to
> > initialize container executor
> > org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to
> > initialize container executor
> > at
> >
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:212)
> > at
> > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> > at
> >
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:463)
> > at
> >
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:511)
> > Caused by: java.io.IOExceptio

Question about the Wiki Instructions on yarn-site.xml

[John Omernik]

This was another "slipped in" question in my other thread, I am breaking
out for specific instructions.  Basically, I was struggling with with some
things in the wiki on this page:
https://cwiki.apache.org/confluence/display/MYRIAD/Installing+for+Administrators

In step 5:
Step 5: Configure YARN to use Myriad

Modify the */opt/hadoop-2.7.0/etc/hadoop/yarn-site.xml* file as instructed
in Sample: myriad-config-default.yml

.


Issue 1: It should link to the yarn-site.xml page, not hte
myriad-config.default.yml page

Issue 2:
It has us put that information in the yarn-site.xml This makes sense.  The
resource manager needs to be aware of the myriad stuff.

Then I go to create a tarball, (which I SHOULD be able to use for both
resource manager and nodemanager... right?) However, the instructions state
to remove the *.xml files.

Step 6: Create the Tarball

The tarball has all of the files needed for the Node Managers and  Resource
Managers. The following shows how to create the tarball and place it in
HDFS:
cd ~
sudo cp -rp /opt/hadoop-2.7.0 .
sudo rm hadoop-2.7.0/etc/hadoop/*.xml
sudo tar -zcpf ~/hadoop-2.7.0.tar.gz hadoop-2.7.0
hadoop fs -put ~/hadoop-2.7.0.tar.gz /dist


What I ended up doing... since I am running the resourcemanager (myriad) in
marathon, is I created two tarballs. One is my hadoop-2.7.0-RM.tar.gz which
has the all the xml files still in the tar ball for shipping to marathon.
Then other is hadoop-2.7.0-NM.tar.gz which per the instructions removes the
*.xml files from the /etc/hadoop/ directory.


I guess... my logic is that myriad creates the conf directory for the
nodemanagers... but then I thought, and I overthinking something? Am I
missing something? Could that be factoring into what I am doing here?


Obviously my first steps are to add the extra yarn-site.xml entries, but in
this current setup, they are only going into the resource manager yarn-site
as the the node-managers don't have a yarn-site in their directories.  Am I
looking at this correctly?  Perhaps we could rethink the removal process of
the XML files in the tarball to allow this to work correctly with a single
tarball?

If I am missing something here, please advise!


John

Requirement for Active Profile at Startup - DNS Delay

[John Omernik]

I tossed a few small things into my larger thread about getting Myriad
running, so I am going to start separate threads to break them out.


When starting Myriad, it seems we now need to have at least one node
manager specified at startup (based on the config file as seen below)

nmInstances: # NMs to start with. Requires at least 1 NM with a non-zero
profile.

  medium: 1 # 


This is going to lead to task failures with mesos dns because the name
won't be ready right away (potentially a 1 minute delay after kicking off
Myriad) do we NEED to have a non-0 profile nodemanager startup with the
resource manager? Can't we start Myriad with no nodemanagers, and then have
automation (i.e. a startup procedure with a script to flex up after the
myriad.marathon.mesos names begins to resolve) to flex up?