Re: Struggling with Permissions

[yuliya Feldman]

Darrin,
Unfortunately it is not the property.
Though definitely passing property you are talking about would be useful - it 
is not used right now by our AuxServices for local volume creation.
  From: Darin Johnson 
 To: Dev ; yuliya Feldman 
 
 Sent: Tuesday, November 17, 2015 12:51 PM
 Subject: Re: Struggling with Permissions
   
Yuliya: Are you referencing yarn.nodemanager.hostname or a mapr specific
option?

I'm working right now on passing a
-Dyarn.nodemanager.hostname=offer.getHostName().  Useful if you've got
extra ip's for a san or management network.

John: Yeah the permissions on the tarball are a pain to get right.  I'm
working on Docker Support and a build script for the tarball, which should
make things easier.  Also, to the point of using world writable directories
it's a little scary from the security side of things to allow executables
to run there, especially things running as privileged users.  Many distro's
of linux will mount /tmp noexec.

Darin



On Tue, Nov 17, 2015 at 2:53 PM, yuliya Feldman  wrote:

> Please change workdir directory for mesos slave to one that is not /tmp
> and make sure that dir is owned by root.
> There is one more caveat with binary distro and MapR - in Myriad code for
> binary distro configuration is copied from RM to NMs - it doe snot work for
> MapR since we need hostname (yes for the sake of local volumes) to be
> unique.
> MapR will have Myriad release to handle this situation.
>      From: John Omernik 
>  To: dev@myriad.incubator.apache.org
>  Sent: Tuesday, November 17, 2015 11:37 AM
>  Subject: Re: Struggling with Permissions
>
> Oh hey, I found a post by me back on Sept 9.  I looked at the Jiras and
> followed the instructions with the same errors. At this point do I still
> need to have a place where the entire path is owned by root? That seems
> like a an odd requirement (a changed of each node to facilitate a
> framework)
>
>
>
>
>
> On Tue, Nov 17, 2015 at 1:25 PM, John Omernik  wrote:
>
> > Hey all, I am struggling with permissions on myriad, trying to get the
> > right permissions in the tgz as well as who to run as.  I am running in
> > MapR, which means I need to run as mapr or root (otherwise my volume
> > creation scripts will fail on MapR, MapR folks, we should talk more about
> > those scripts)
> >
> > But back to the code, I've had lots issues. When I run the Frameworkuser
> > and Superuser as mapr, it unpacks everything as MapR and I get a
> > "/bin/container-executor" must be owned by root but is owned by 700 (my
> > mapr UID).
> >
> > So now I am running as root, and I am getting the error below as it
> > relates to /tmp. I am not sure which /tmp this refers to. the /tmp that
> my
> > slave is executing in? (i.e. my local mesos agent /tmp directory) or my
> > MaprFS /tmp directory (both of which are world writable, as /tmp
> typically
> > is... or am I mistaken here?)
> >
> > Any thoughts on how to get this to resolve? This is when nodemanager is
> > trying to start running as root and root for both of my Myriad users.
> >
> > Thanks!
> >
> >
> > Caused by: ExitCodeException exitCode=24: File /tmp must not be world or
> group writable, but is 1777
> >
> >
> >
> >
>
>
>
>


  

Re: Struggling with Permissions

[yuliya Feldman]

Hadoop (not Mapr) requires whole path starting from "/" be owned by root and 
writable only by root
The second problem is exactly what I was talking about configuration being 
taken from RM that overwrites local one
I can give you a patch to mitigate the issue for Mapr if you are building from 
source.
Thanks,Yuliya
  From: John Omernik 
 To: dev@myriad.incubator.apache.org 
 Sent: Tuesday, November 17, 2015 1:15 PM
 Subject: Re: Struggling with Permissions
   
Well sure /tmp is world writeable but /tmp/mesos is not world writable thus
there is a sandbox to play in there... or am I missing something. Not to
mention my tmp is rwt which is world writable but only the creator or root
can modify (based on the googles).
Yuliya:

I am seeing a weird behavior with MapR as it relates to (I believe) the
mapr_direct_shuffle.

In the Node Manager logs, I see things starting and it saying "Checking for
local volume, if local volume is not present command will create and mount
it"

Command invoked is : /opt/mapr/server/createTTVolume.sh
hadoopmapr7.brewingintel.com /var/mapr/local/
hadoopmapr2.brewingintel.com/mapred /var/mapr/local/
hadoopmapr2.brewingintel.com/mapred/nodeManager yarn


What is interesting here is hadoopmapr7 is the nodemanager it's trying to
start on, however the mount point it's trying to create is hadoopmapr2
which is the node the resource manager happened to fall on...  I was very
confused by that because in no place should hadoopmapr2 be "known" to the
nodemanager, because it thinks the resource manager hostname is
myriad.marathon.mesos.

So why was it hard coding to the node the resource manager is running on?

Well if I look at the conf file in the sandbox (the file that gets copied
to be yarn-site.xml for node managers.  There ARE four references the
hadoopmapr2. Three of the four say "source programatically" and one is just
set... that's mapr.host.  Could there be some down stream hinkyness going
on with how MapR is setting hostnames?  All of these variables seem "wrong"
in that mapr.host (on the node manager) should be hadoopmapr7 in this case,
and the resource managers should all be myriad.marathon.mesos.  I'd be
interested in your thoughts here, because I am stumped at how these are
getting set.



yarn.resourcemanager.addresshadoopmapr2:8032programatically
mapr.hosthadoopmapr2.brewingintel.com

yarn.resourcemanager.resource-tracker.addresshadoopmapr2:8031programatically
yarn.resourcemanager.admin.addresshadoopmapr2:8033programatically







On Tue, Nov 17, 2015 at 2:51 PM, Darin Johnson 
wrote:

> Yuliya: Are you referencing yarn.nodemanager.hostname or a mapr specific
> option?
>
> I'm working right now on passing a
> -Dyarn.nodemanager.hostname=offer.getHostName().  Useful if you've got
> extra ip's for a san or management network.
>
> John: Yeah the permissions on the tarball are a pain to get right.  I'm
> working on Docker Support and a build script for the tarball, which should
> make things easier.  Also, to the point of using world writable directories
> it's a little scary from the security side of things to allow executables
> to run there, especially things running as privileged users.  Many distro's
> of linux will mount /tmp noexec.
>
> Darin
>
> On Tue, Nov 17, 2015 at 2:53 PM, yuliya Feldman
>  > wrote:
>
> > Please change workdir directory for mesos slave to one that is not /tmp
> > and make sure that dir is owned by root.
> > There is one more caveat with binary distro and MapR - in Myriad code for
> > binary distro configuration is copied from RM to NMs - it doe snot work
> for
> > MapR since we need hostname (yes for the sake of local volumes) to be
> > unique.
> > MapR will have Myriad release to handle this situation.
> >      From: John Omernik 
> >  To: dev@myriad.incubator.apache.org
> >  Sent: Tuesday, November 17, 2015 11:37 AM
> >  Subject: Re: Struggling with Permissions
> >
> > Oh hey, I found a post by me back on Sept 9.  I looked at the Jiras and
> > followed the instructions with the same errors. At this point do I still
> > need to have a place where the entire path is owned by root? That seems
> > like a an odd requirement (a changed of each node to facilitate a
> > framework)
> >
> >
> >
> >
> >
> > On Tue, Nov 17, 2015 at 1:25 PM, John Omernik  wrote:
> >
> > > Hey all, I am struggling with permissions on myriad, trying to get the
> > > right permissions in the tgz as well as who to run as.  I am running in
> > > MapR, which means I need to run as mapr or root (otherwise my volume
> > > creation scripts will fail on MapR, MapR folks, we should talk more
> about
> > > those scripts)
> > >
> > > But back to the code, I've had lots issues. When I run the
> Frameworkuser
> > > and Superuser as mapr, it unpacks everything as MapR and I get a
> > > "/bin/container-executor" must be owned by root but is owned by 700 (my
> > > mapr UID).
> > >
> > > So now I am running as root, and I am getting the error below as it
> > > relates to /tmp. I am not sure whi

Re: Struggling with Permissions

[Darin Johnson]

John,

I'm not super familiar with MapR, but I think I might have some thought and
the the MapR people can chime it :).

I think the mapr.host thing is due to the fact in the remote distribution,
Myriad pulls it's config from the resource manager.  As I mentioned in my
note to Yuyila, I'm working on adding the ability to add
yarn.nodemanager.hostname as a -D option, I think the right thing may by to
expose an environment variable $HOSTNAME and then in yarnEnvironment: you
could set a YARN_OPTS=-Dmapr.hostname=$HOSTNAME
-Dyarn.nodemanager.hostname=$HOSTNAME ... option.

One could imagine a similar option for ports as this is kind of what
Marathon does.

Maybe best to JIRA this, as I don't think we necessarily expose a lot of
things we should just yet.


On Tue, Nov 17, 2015 at 4:41 PM, John Omernik  wrote:

> What's even stranger is I can't for life of me find where "mapr.host" gets
> set or used.  I did a grep -P -R "mapr\.host" ./*  in /opt/mapr (which
> included me pulling down the myriad code into
> /opt/mapr/myriad/incubator-myriad) and found only one reference in
> /opt/mapr/server/mapr_yarn_install.sh
>
> 
>
>   yarn.nodemanager.hostname
>
>   \${mapr.host}
>
> " | sudo tee -a ${YARN_CONF_FILE}
>
>
> But I don't think that is being called at all by the resource manager...
>
>
> (Note when I create my tarball from /opt/mapr/hadoop/hadoop-2.7.0 directory
> I am using tar -zcfhp  to both preserver permissions and include the files
> that symlinked... not sure if that affects things here.. )
>
>
>
>
>
> On Tue, Nov 17, 2015 at 3:15 PM, John Omernik  wrote:
>
> > Well sure /tmp is world writeable but /tmp/mesos is not world writable
> > thus there is a sandbox to play in there... or am I missing something.
> Not
> > to mention my tmp is rwt which is world writable but only the creator or
> > root can modify (based on the googles).
> > Yuliya:
> >
> > I am seeing a weird behavior with MapR as it relates to (I believe) the
> > mapr_direct_shuffle.
> >
> > In the Node Manager logs, I see things starting and it saying "Checking
> > for local volume, if local volume is not present command will create and
> > mount it"
> >
> > Command invoked is : /opt/mapr/server/createTTVolume.sh
> > hadoopmapr7.brewingintel.com /var/mapr/local/
> > hadoopmapr2.brewingintel.com/mapred /var/mapr/local/
> > hadoopmapr2.brewingintel.com/mapred/nodeManager yarn
> >
> >
> > What is interesting here is hadoopmapr7 is the nodemanager it's trying to
> > start on, however the mount point it's trying to create is hadoopmapr2
> > which is the node the resource manager happened to fall on...  I was very
> > confused by that because in no place should hadoopmapr2 be "known" to the
> > nodemanager, because it thinks the resource manager hostname is
> > myriad.marathon.mesos.
> >
> > So why was it hard coding to the node the resource manager is running on?
> >
> > Well if I look at the conf file in the sandbox (the file that gets copied
> > to be yarn-site.xml for node managers.  There ARE four references the
> > hadoopmapr2. Three of the four say "source programatically" and one is
> just
> > set... that's mapr.host.  Could there be some down stream hinkyness going
> > on with how MapR is setting hostnames?  All of these variables seem
> "wrong"
> > in that mapr.host (on the node manager) should be hadoopmapr7 in this
> case,
> > and the resource managers should all be myriad.marathon.mesos.   I'd be
> > interested in your thoughts here, because I am stumped at how these are
> > getting set.
> >
> >
> >
> >
> >
> yarn.resourcemanager.addresshadoopmapr2:8032programatically
> > mapr.hosthadoopmapr2.brewingintel.com
> > 
> >
> >
> yarn.resourcemanager.resource-tracker.addresshadoopmapr2:8031programatically
> >
> >
> yarn.resourcemanager.admin.addresshadoopmapr2:8033programatically
> >
> >
> >
> >
> >
> > On Tue, Nov 17, 2015 at 2:51 PM, Darin Johnson 
> > wrote:
> >
> >> Yuliya: Are you referencing yarn.nodemanager.hostname or a mapr specific
> >> option?
> >>
> >> I'm working right now on passing a
> >> -Dyarn.nodemanager.hostname=offer.getHostName().  Useful if you've got
> >> extra ip's for a san or management network.
> >>
> >> John: Yeah the permissions on the tarball are a pain to get right.  I'm
> >> working on Docker Support and a build script for the tarball, which
> should
> >> make things easier.  Also, to the point of using world writable
> >> directories
> >> it's a little scary from the security side of things to allow
> executables
> >> to run there, especially things running as privileged users.  Many
> >> distro's
> >> of linux will mount /tmp noexec.
> >>
> >> Darin
> >>
> >> On Tue, Nov 17, 2015 at 2:53 PM, yuliya Feldman
> >>  >> > wrote:
> >>
> >> > Please change workdir directory for mesos slave to one that is not
> /tmp
> >> > and make sure that dir is owned by root.
> >> > There is one more caveat with binary distro and MapR - in Myriad code
> >> for
> >> > binary distro configuration is copied from RM to NMs - it doe 

Re: Struggling with Permissions

[John Omernik]

What's even stranger is I can't for life of me find where "mapr.host" gets
set or used.  I did a grep -P -R "mapr\.host" ./*  in /opt/mapr (which
included me pulling down the myriad code into
/opt/mapr/myriad/incubator-myriad) and found only one reference in
/opt/mapr/server/mapr_yarn_install.sh



  yarn.nodemanager.hostname

  \${mapr.host}

" | sudo tee -a ${YARN_CONF_FILE}


But I don't think that is being called at all by the resource manager...


(Note when I create my tarball from /opt/mapr/hadoop/hadoop-2.7.0 directory
I am using tar -zcfhp  to both preserver permissions and include the files
that symlinked... not sure if that affects things here.. )





On Tue, Nov 17, 2015 at 3:15 PM, John Omernik  wrote:

> Well sure /tmp is world writeable but /tmp/mesos is not world writable
> thus there is a sandbox to play in there... or am I missing something. Not
> to mention my tmp is rwt which is world writable but only the creator or
> root can modify (based on the googles).
> Yuliya:
>
> I am seeing a weird behavior with MapR as it relates to (I believe) the
> mapr_direct_shuffle.
>
> In the Node Manager logs, I see things starting and it saying "Checking
> for local volume, if local volume is not present command will create and
> mount it"
>
> Command invoked is : /opt/mapr/server/createTTVolume.sh
> hadoopmapr7.brewingintel.com /var/mapr/local/
> hadoopmapr2.brewingintel.com/mapred /var/mapr/local/
> hadoopmapr2.brewingintel.com/mapred/nodeManager yarn
>
>
> What is interesting here is hadoopmapr7 is the nodemanager it's trying to
> start on, however the mount point it's trying to create is hadoopmapr2
> which is the node the resource manager happened to fall on...  I was very
> confused by that because in no place should hadoopmapr2 be "known" to the
> nodemanager, because it thinks the resource manager hostname is
> myriad.marathon.mesos.
>
> So why was it hard coding to the node the resource manager is running on?
>
> Well if I look at the conf file in the sandbox (the file that gets copied
> to be yarn-site.xml for node managers.  There ARE four references the
> hadoopmapr2. Three of the four say "source programatically" and one is just
> set... that's mapr.host.  Could there be some down stream hinkyness going
> on with how MapR is setting hostnames?  All of these variables seem "wrong"
> in that mapr.host (on the node manager) should be hadoopmapr7 in this case,
> and the resource managers should all be myriad.marathon.mesos.   I'd be
> interested in your thoughts here, because I am stumped at how these are
> getting set.
>
>
>
>
> yarn.resourcemanager.addresshadoopmapr2:8032programatically
> mapr.hosthadoopmapr2.brewingintel.com
> 
>
> yarn.resourcemanager.resource-tracker.addresshadoopmapr2:8031programatically
>
> yarn.resourcemanager.admin.addresshadoopmapr2:8033programatically
>
>
>
>
>
> On Tue, Nov 17, 2015 at 2:51 PM, Darin Johnson 
> wrote:
>
>> Yuliya: Are you referencing yarn.nodemanager.hostname or a mapr specific
>> option?
>>
>> I'm working right now on passing a
>> -Dyarn.nodemanager.hostname=offer.getHostName().  Useful if you've got
>> extra ip's for a san or management network.
>>
>> John: Yeah the permissions on the tarball are a pain to get right.  I'm
>> working on Docker Support and a build script for the tarball, which should
>> make things easier.  Also, to the point of using world writable
>> directories
>> it's a little scary from the security side of things to allow executables
>> to run there, especially things running as privileged users.  Many
>> distro's
>> of linux will mount /tmp noexec.
>>
>> Darin
>>
>> On Tue, Nov 17, 2015 at 2:53 PM, yuliya Feldman
>> > > wrote:
>>
>> > Please change workdir directory for mesos slave to one that is not /tmp
>> > and make sure that dir is owned by root.
>> > There is one more caveat with binary distro and MapR - in Myriad code
>> for
>> > binary distro configuration is copied from RM to NMs - it doe snot work
>> for
>> > MapR since we need hostname (yes for the sake of local volumes) to be
>> > unique.
>> > MapR will have Myriad release to handle this situation.
>> >   From: John Omernik 
>> >  To: dev@myriad.incubator.apache.org
>> >  Sent: Tuesday, November 17, 2015 11:37 AM
>> >  Subject: Re: Struggling with Permissions
>> >
>> > Oh hey, I found a post by me back on Sept 9.  I looked at the Jiras and
>> > followed the instructions with the same errors. At this point do I still
>> > need to have a place where the entire path is owned by root? That seems
>> > like a an odd requirement (a changed of each node to facilitate a
>> > framework)
>> >
>> >
>> >
>> >
>> >
>> > On Tue, Nov 17, 2015 at 1:25 PM, John Omernik  wrote:
>> >
>> > > Hey all, I am struggling with permissions on myriad, trying to get the
>> > > right permissions in the tgz as well as who to run as.  I am running
>> in
>> > > MapR, which means I need to run as mapr or root (otherwise my volume
>> > > creation scripts will fail on MapR, MapR folks, we shoul

Re: Struggling with Permissions

[John Omernik]

Well sure /tmp is world writeable but /tmp/mesos is not world writable thus
there is a sandbox to play in there... or am I missing something. Not to
mention my tmp is rwt which is world writable but only the creator or root
can modify (based on the googles).
Yuliya:

I am seeing a weird behavior with MapR as it relates to (I believe) the
mapr_direct_shuffle.

In the Node Manager logs, I see things starting and it saying "Checking for
local volume, if local volume is not present command will create and mount
it"

Command invoked is : /opt/mapr/server/createTTVolume.sh
hadoopmapr7.brewingintel.com /var/mapr/local/
hadoopmapr2.brewingintel.com/mapred /var/mapr/local/
hadoopmapr2.brewingintel.com/mapred/nodeManager yarn


What is interesting here is hadoopmapr7 is the nodemanager it's trying to
start on, however the mount point it's trying to create is hadoopmapr2
which is the node the resource manager happened to fall on...  I was very
confused by that because in no place should hadoopmapr2 be "known" to the
nodemanager, because it thinks the resource manager hostname is
myriad.marathon.mesos.

So why was it hard coding to the node the resource manager is running on?

Well if I look at the conf file in the sandbox (the file that gets copied
to be yarn-site.xml for node managers.  There ARE four references the
hadoopmapr2. Three of the four say "source programatically" and one is just
set... that's mapr.host.  Could there be some down stream hinkyness going
on with how MapR is setting hostnames?  All of these variables seem "wrong"
in that mapr.host (on the node manager) should be hadoopmapr7 in this case,
and the resource managers should all be myriad.marathon.mesos.   I'd be
interested in your thoughts here, because I am stumped at how these are
getting set.



yarn.resourcemanager.addresshadoopmapr2:8032programatically
mapr.hosthadoopmapr2.brewingintel.com

yarn.resourcemanager.resource-tracker.addresshadoopmapr2:8031programatically
yarn.resourcemanager.admin.addresshadoopmapr2:8033programatically





On Tue, Nov 17, 2015 at 2:51 PM, Darin Johnson 
wrote:

> Yuliya: Are you referencing yarn.nodemanager.hostname or a mapr specific
> option?
>
> I'm working right now on passing a
> -Dyarn.nodemanager.hostname=offer.getHostName().  Useful if you've got
> extra ip's for a san or management network.
>
> John: Yeah the permissions on the tarball are a pain to get right.  I'm
> working on Docker Support and a build script for the tarball, which should
> make things easier.  Also, to the point of using world writable directories
> it's a little scary from the security side of things to allow executables
> to run there, especially things running as privileged users.  Many distro's
> of linux will mount /tmp noexec.
>
> Darin
>
> On Tue, Nov 17, 2015 at 2:53 PM, yuliya Feldman
>  > wrote:
>
> > Please change workdir directory for mesos slave to one that is not /tmp
> > and make sure that dir is owned by root.
> > There is one more caveat with binary distro and MapR - in Myriad code for
> > binary distro configuration is copied from RM to NMs - it doe snot work
> for
> > MapR since we need hostname (yes for the sake of local volumes) to be
> > unique.
> > MapR will have Myriad release to handle this situation.
> >   From: John Omernik 
> >  To: dev@myriad.incubator.apache.org
> >  Sent: Tuesday, November 17, 2015 11:37 AM
> >  Subject: Re: Struggling with Permissions
> >
> > Oh hey, I found a post by me back on Sept 9.  I looked at the Jiras and
> > followed the instructions with the same errors. At this point do I still
> > need to have a place where the entire path is owned by root? That seems
> > like a an odd requirement (a changed of each node to facilitate a
> > framework)
> >
> >
> >
> >
> >
> > On Tue, Nov 17, 2015 at 1:25 PM, John Omernik  wrote:
> >
> > > Hey all, I am struggling with permissions on myriad, trying to get the
> > > right permissions in the tgz as well as who to run as.  I am running in
> > > MapR, which means I need to run as mapr or root (otherwise my volume
> > > creation scripts will fail on MapR, MapR folks, we should talk more
> about
> > > those scripts)
> > >
> > > But back to the code, I've had lots issues. When I run the
> Frameworkuser
> > > and Superuser as mapr, it unpacks everything as MapR and I get a
> > > "/bin/container-executor" must be owned by root but is owned by 700 (my
> > > mapr UID).
> > >
> > > So now I am running as root, and I am getting the error below as it
> > > relates to /tmp. I am not sure which /tmp this refers to. the /tmp that
> > my
> > > slave is executing in? (i.e. my local mesos agent /tmp directory) or my
> > > MaprFS /tmp directory (both of which are world writable, as /tmp
> > typically
> > > is... or am I mistaken here?)
> > >
> > > Any thoughts on how to get this to resolve? This is when nodemanager is
> > > trying to start running as root and root for both of my Myriad users.
> > >
> > > Thanks!
> > >
> > >
> > > Caused by: ExitCodeException

Re: Struggling with Permissions

[Darin Johnson]

Yuliya: Are you referencing yarn.nodemanager.hostname or a mapr specific
option?

I'm working right now on passing a
-Dyarn.nodemanager.hostname=offer.getHostName().  Useful if you've got
extra ip's for a san or management network.

John: Yeah the permissions on the tarball are a pain to get right.  I'm
working on Docker Support and a build script for the tarball, which should
make things easier.  Also, to the point of using world writable directories
it's a little scary from the security side of things to allow executables
to run there, especially things running as privileged users.  Many distro's
of linux will mount /tmp noexec.

Darin

On Tue, Nov 17, 2015 at 2:53 PM, yuliya Feldman  wrote:

> Please change workdir directory for mesos slave to one that is not /tmp
> and make sure that dir is owned by root.
> There is one more caveat with binary distro and MapR - in Myriad code for
> binary distro configuration is copied from RM to NMs - it doe snot work for
> MapR since we need hostname (yes for the sake of local volumes) to be
> unique.
> MapR will have Myriad release to handle this situation.
>   From: John Omernik 
>  To: dev@myriad.incubator.apache.org
>  Sent: Tuesday, November 17, 2015 11:37 AM
>  Subject: Re: Struggling with Permissions
>
> Oh hey, I found a post by me back on Sept 9.  I looked at the Jiras and
> followed the instructions with the same errors. At this point do I still
> need to have a place where the entire path is owned by root? That seems
> like a an odd requirement (a changed of each node to facilitate a
> framework)
>
>
>
>
>
> On Tue, Nov 17, 2015 at 1:25 PM, John Omernik  wrote:
>
> > Hey all, I am struggling with permissions on myriad, trying to get the
> > right permissions in the tgz as well as who to run as.  I am running in
> > MapR, which means I need to run as mapr or root (otherwise my volume
> > creation scripts will fail on MapR, MapR folks, we should talk more about
> > those scripts)
> >
> > But back to the code, I've had lots issues. When I run the Frameworkuser
> > and Superuser as mapr, it unpacks everything as MapR and I get a
> > "/bin/container-executor" must be owned by root but is owned by 700 (my
> > mapr UID).
> >
> > So now I am running as root, and I am getting the error below as it
> > relates to /tmp. I am not sure which /tmp this refers to. the /tmp that
> my
> > slave is executing in? (i.e. my local mesos agent /tmp directory) or my
> > MaprFS /tmp directory (both of which are world writable, as /tmp
> typically
> > is... or am I mistaken here?)
> >
> > Any thoughts on how to get this to resolve? This is when nodemanager is
> > trying to start running as root and root for both of my Myriad users.
> >
> > Thanks!
> >
> >
> > Caused by: ExitCodeException exitCode=24: File /tmp must not be world or
> group writable, but is 1777
> >
> >
> >
> >
>
>
>
>

Re: Struggling with Permissions

[John Omernik]

Is this change going to be required for all mesos installations that would
run Myriad with remote distribution?  I guess I'd like to respectfully
challenge that notion that to use remote distribution we need to make a
cluster wide change to all of our slave nodes that would require a restart
of the mesos slaves.  That seems to me to be quite a large requirement.

What's strange to me, is at one point I had this working without the that
requirement, but I can't reproduce how I created the tgz to make it work.
Where does the requirement to have the /tmp not be world writable come into
play? This is seems like a strange requirement in that as a executor, it's
world should start at the root of it's container.  I.e. how does it even
know that that it's parent directory has different permissions? Can we just
just traverse frameworks up? If I had a command that said rm -rf ../../*
would that work?  Maybe I just didn't dig into what the sandbox was before
this, but my thought that from the perspective of the executors / was, for
example:

/tmp/mesos/slaves/20151007-102829-1660987584-5050-15078-S5/frameworks/d9aab75d-1a74-489d-976d-805ce55364ff-0011/executors/myriad_executord9aab75d-1a74-489d-976d-805ce55364ff-0011d9aab75d-1a74-489d-976d-805ce55364ff-O18052420151007-102829-1660987584-5050-15078-S5/runs/70c6ba08-5c7a-4399-bf5a-34ac328e66e1/


as a ls of that directory shows me the unpacked tgz etc.  How does the
nodemanager know that the parent directories, specifically /tmp is world
writable, and why does it care?

I am not trying to be belligerent, I just want to understand this without
just changing a cluster wide setting/restarting mesos.



On Tue, Nov 17, 2015 at 1:53 PM, yuliya Feldman  wrote:

> Please change workdir directory for mesos slave to one that is not /tmp
> and make sure that dir is owned by root.
> There is one more caveat with binary distro and MapR - in Myriad code for
> binary distro configuration is copied from RM to NMs - it doe snot work for
> MapR since we need hostname (yes for the sake of local volumes) to be
> unique.
> MapR will have Myriad release to handle this situation.
>   From: John Omernik 
>  To: dev@myriad.incubator.apache.org
>  Sent: Tuesday, November 17, 2015 11:37 AM
>  Subject: Re: Struggling with Permissions
>
> Oh hey, I found a post by me back on Sept 9.  I looked at the Jiras and
> followed the instructions with the same errors. At this point do I still
> need to have a place where the entire path is owned by root? That seems
> like a an odd requirement (a changed of each node to facilitate a
> framework)
>
>
>
>
>
> On Tue, Nov 17, 2015 at 1:25 PM, John Omernik  wrote:
>
> > Hey all, I am struggling with permissions on myriad, trying to get the
> > right permissions in the tgz as well as who to run as.  I am running in
> > MapR, which means I need to run as mapr or root (otherwise my volume
> > creation scripts will fail on MapR, MapR folks, we should talk more about
> > those scripts)
> >
> > But back to the code, I've had lots issues. When I run the Frameworkuser
> > and Superuser as mapr, it unpacks everything as MapR and I get a
> > "/bin/container-executor" must be owned by root but is owned by 700 (my
> > mapr UID).
> >
> > So now I am running as root, and I am getting the error below as it
> > relates to /tmp. I am not sure which /tmp this refers to. the /tmp that
> my
> > slave is executing in? (i.e. my local mesos agent /tmp directory) or my
> > MaprFS /tmp directory (both of which are world writable, as /tmp
> typically
> > is... or am I mistaken here?)
> >
> > Any thoughts on how to get this to resolve? This is when nodemanager is
> > trying to start running as root and root for both of my Myriad users.
> >
> > Thanks!
> >
> >
> > Caused by: ExitCodeException exitCode=24: File /tmp must not be world or
> group writable, but is 1777
> >
> >
> >
> >
>
>
>
>

Re: Struggling with Permissions

[yuliya Feldman]

Please change workdir directory for mesos slave to one that is not /tmp and 
make sure that dir is owned by root.
There is one more caveat with binary distro and MapR - in Myriad code for 
binary distro configuration is copied from RM to NMs - it doe snot work for 
MapR since we need hostname (yes for the sake of local volumes) to be unique.
MapR will have Myriad release to handle this situation.
  From: John Omernik 
 To: dev@myriad.incubator.apache.org 
 Sent: Tuesday, November 17, 2015 11:37 AM
 Subject: Re: Struggling with Permissions
   
Oh hey, I found a post by me back on Sept 9.  I looked at the Jiras and
followed the instructions with the same errors. At this point do I still
need to have a place where the entire path is owned by root? That seems
like a an odd requirement (a changed of each node to facilitate a framework)





On Tue, Nov 17, 2015 at 1:25 PM, John Omernik  wrote:

> Hey all, I am struggling with permissions on myriad, trying to get the
> right permissions in the tgz as well as who to run as.  I am running in
> MapR, which means I need to run as mapr or root (otherwise my volume
> creation scripts will fail on MapR, MapR folks, we should talk more about
> those scripts)
>
> But back to the code, I've had lots issues. When I run the Frameworkuser
> and Superuser as mapr, it unpacks everything as MapR and I get a
> "/bin/container-executor" must be owned by root but is owned by 700 (my
> mapr UID).
>
> So now I am running as root, and I am getting the error below as it
> relates to /tmp. I am not sure which /tmp this refers to. the /tmp that my
> slave is executing in? (i.e. my local mesos agent /tmp directory) or my
> MaprFS /tmp directory (both of which are world writable, as /tmp typically
> is... or am I mistaken here?)
>
> Any thoughts on how to get this to resolve? This is when nodemanager is
> trying to start running as root and root for both of my Myriad users.
>
> Thanks!
>
>
> Caused by: ExitCodeException exitCode=24: File /tmp must not be world or 
> group writable, but is 1777
>
>
>
>


  

Re: Struggling with Permissions

[John Omernik]

Oh hey, I found a post by me back on Sept 9.  I looked at the Jiras and
followed the instructions with the same errors. At this point do I still
need to have a place where the entire path is owned by root? That seems
like a an odd requirement (a changed of each node to facilitate a framework)



On Tue, Nov 17, 2015 at 1:25 PM, John Omernik  wrote:

> Hey all, I am struggling with permissions on myriad, trying to get the
> right permissions in the tgz as well as who to run as.  I am running in
> MapR, which means I need to run as mapr or root (otherwise my volume
> creation scripts will fail on MapR, MapR folks, we should talk more about
> those scripts)
>
> But back to the code, I've had lots issues. When I run the Frameworkuser
> and Superuser as mapr, it unpacks everything as MapR and I get a
> "/bin/container-executor" must be owned by root but is owned by 700 (my
> mapr UID).
>
> So now I am running as root, and I am getting the error below as it
> relates to /tmp. I am not sure which /tmp this refers to. the /tmp that my
> slave is executing in? (i.e. my local mesos agent /tmp directory) or my
> MaprFS /tmp directory (both of which are world writable, as /tmp typically
> is... or am I mistaken here?)
>
> Any thoughts on how to get this to resolve? This is when nodemanager is
> trying to start running as root and root for both of my Myriad users.
>
> Thanks!
>
>
> Caused by: ExitCodeException exitCode=24: File /tmp must not be world or 
> group writable, but is 1777
>
>
>
>

Struggling with Permissions

[John Omernik]

Hey all, I am struggling with permissions on myriad, trying to get the
right permissions in the tgz as well as who to run as.  I am running in
MapR, which means I need to run as mapr or root (otherwise my volume
creation scripts will fail on MapR, MapR folks, we should talk more about
those scripts)

But back to the code, I've had lots issues. When I run the Frameworkuser
and Superuser as mapr, it unpacks everything as MapR and I get a
"/bin/container-executor" must be owned by root but is owned by 700 (my
mapr UID).

So now I am running as root, and I am getting the error below as it relates
to /tmp. I am not sure which /tmp this refers to. the /tmp that my slave is
executing in? (i.e. my local mesos agent /tmp directory) or my MaprFS /tmp
directory (both of which are world writable, as /tmp typically is... or am
I mistaken here?)

Any thoughts on how to get this to resolve? This is when nodemanager is
trying to start running as root and root for both of my Myriad users.

Thanks!


Caused by: ExitCodeException exitCode=24: File /tmp must not be world
or group writable, but is 1777

Re: [Help needed from Mentors] SVN PubSub

[Adam Bordelon]

Thanks, Luciano. And yes, I am m...@apache.org

On Tue, Nov 17, 2015 at 8:11 AM, Luciano Resende 
wrote:

> I have pushed a "myriad" branch, and waiting for Infra to review/merge.
>
> On Tue, Nov 17, 2015 at 7:55 AM, Santosh Marella 
> wrote:
>
>> More Apache Ids need to be added: me,darinj,sdaingade
>>
>> --
>> Sent from mobile
>> On Nov 17, 2015 7:44 AM, "Luciano Resende"  wrote:
>>
>> > Are we missing anyone ?
>> >
>> > myriad=benh,danese,kensipe,lresende,mohit,smarella,tdunning
>> >
>> >
>> > On Mon, Nov 16, 2015 at 5:23 AM, Benjamin Hindman 
>> > wrote:
>> >
>> > > I've never edited an "svn authorization template" before, is there
>> some
>> > > documentation that one of the more experienced mentors can point us/me
>> > to?
>> > >
>> > > On Fri, Nov 13, 2015 at 7:47 AM, Adam Bordelon 
>> > wrote:
>> > >
>> > >> Correct. As mentioned in the Nov 4-5 comments on INFRA-10696
>> > >> <
>> >
>> https://issues.apache.org/jira/browse/INFRA-10696?focusedCommentId=14990668&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14990668
>> > >,
>> > >> we got 403 Forbidden errors when trying to commit. Geoffrey from
>> INFRA
>> > >> tells us, "You need to get your mentor (or someone with enough
>> karma) to
>> > >> edit the svn authorization template and add a list of names from
>> myriad
>> > >> that you want to be able to commit."
>> > >>
>> > >> On Thu, Nov 12, 2015 at 6:37 PM, Jim Klucar 
>> wrote:
>> > >>
>> > >>> Ted,
>> > >>>
>> > >>> Santosh and Adam both tried to commit to the svn repo but got
>> > permission
>> > >>> denied. Looks like they just need some kind of svn permissions to
>> push
>> > our
>> > >>> site out.
>> > >>>
>> > >>> Jim
>> > >>>
>> > >>> On Thursday, November 12, 2015, Ted Dunning 
>> > >>> wrote:
>> > >>>
>> >  OK.
>> > 
>> >  I am really confused.  I hear you need SVN something.  And you need
>> > some
>> >  kind of web action.
>> > 
>> >  What specifically do you need?
>> > 
>> >  (I am heavily loaded right now and in a difficult time zone)
>> > 
>> > 
>> >  On Fri, Nov 13, 2015 at 5:34 AM, Adam Bordelon > >
>> >  wrote:
>> > 
>> >  > + Mentor email addresses, in case filters are failing us.
>> >  > Any mentor, help! We still need help with this.
>> >  >
>> >  > Don't you want to see this content:
>> >  http://klucar.github.io/myriad-site/
>> >  > at this address: http://myriad.incubator.apache.org/
>> >  > ?
>> >  >
>> >  > On Tue, Nov 10, 2015 at 9:34 AM, Santosh Marella <
>> >  smare...@maprtech.com>
>> >  > wrote:
>> >  >
>> >  >> Gentle reminder... Appreciate your help with this.
>> >  >>
>> >  >> --
>> >  >> Sent from mobile
>> >  >> On Nov 8, 2015 7:02 PM, "Santosh Marella" <
>> smare...@maprtech.com>
>> >  wrote:
>> >  >>
>> >  >> > Hi Ben,Danese,Luciano,Ted,
>> >  >> >
>> >  >> >   In INFRA-10696 [1], we've requested for a SVN directory to
>> >  publish
>> >  >> > Myriad's website. However, it seems to need a svn
>> authorization
>> >  >> template to
>> >  >> > be submitted by one of the Mentors in order to push website
>> >  commits to
>> >  >> SVN.
>> >  >> > Can one of you please help with that?
>> >  >> >
>> >  >> >   [1] https://issues.apache.org/jira/browse/INFRA-10696
>> >  >> >
>> >  >> > Thanks,
>> >  >> > Santosh
>> >  >> >
>> >  >>
>> >  >
>> >  >
>> > 
>> > >>>
>> > >>
>> > >
>> > >
>> > > --
>> > > Benjamin Hindman
>> > > Founder of Mesosphere and Co-Creator of Apache Mesos
>> > > Mesosphere Inc.  
>> > >
>> > > Follow us on Twitter: @mesosphere 
>> > >
>> >
>> >
>> >
>> > --
>> > Luciano Resende
>> > http://people.apache.org/~lresende
>> > http://twitter.com/lresende1975
>> > http://lresende.blogspot.com/
>> >
>>
>
>
>
> --
> Luciano Resende
> http://people.apache.org/~lresende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>

Re: [Help needed from Mentors] SVN PubSub

[Luciano Resende]

I have pushed a "myriad" branch, and waiting for Infra to review/merge.

On Tue, Nov 17, 2015 at 7:55 AM, Santosh Marella 
wrote:

> More Apache Ids need to be added: me,darinj,sdaingade
>
> --
> Sent from mobile
> On Nov 17, 2015 7:44 AM, "Luciano Resende"  wrote:
>
> > Are we missing anyone ?
> >
> > myriad=benh,danese,kensipe,lresende,mohit,smarella,tdunning
> >
> >
> > On Mon, Nov 16, 2015 at 5:23 AM, Benjamin Hindman 
> > wrote:
> >
> > > I've never edited an "svn authorization template" before, is there some
> > > documentation that one of the more experienced mentors can point us/me
> > to?
> > >
> > > On Fri, Nov 13, 2015 at 7:47 AM, Adam Bordelon 
> > wrote:
> > >
> > >> Correct. As mentioned in the Nov 4-5 comments on INFRA-10696
> > >> <
> >
> https://issues.apache.org/jira/browse/INFRA-10696?focusedCommentId=14990668&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14990668
> > >,
> > >> we got 403 Forbidden errors when trying to commit. Geoffrey from INFRA
> > >> tells us, "You need to get your mentor (or someone with enough karma)
> to
> > >> edit the svn authorization template and add a list of names from
> myriad
> > >> that you want to be able to commit."
> > >>
> > >> On Thu, Nov 12, 2015 at 6:37 PM, Jim Klucar  wrote:
> > >>
> > >>> Ted,
> > >>>
> > >>> Santosh and Adam both tried to commit to the svn repo but got
> > permission
> > >>> denied. Looks like they just need some kind of svn permissions to
> push
> > our
> > >>> site out.
> > >>>
> > >>> Jim
> > >>>
> > >>> On Thursday, November 12, 2015, Ted Dunning 
> > >>> wrote:
> > >>>
> >  OK.
> > 
> >  I am really confused.  I hear you need SVN something.  And you need
> > some
> >  kind of web action.
> > 
> >  What specifically do you need?
> > 
> >  (I am heavily loaded right now and in a difficult time zone)
> > 
> > 
> >  On Fri, Nov 13, 2015 at 5:34 AM, Adam Bordelon 
> >  wrote:
> > 
> >  > + Mentor email addresses, in case filters are failing us.
> >  > Any mentor, help! We still need help with this.
> >  >
> >  > Don't you want to see this content:
> >  http://klucar.github.io/myriad-site/
> >  > at this address: http://myriad.incubator.apache.org/
> >  > ?
> >  >
> >  > On Tue, Nov 10, 2015 at 9:34 AM, Santosh Marella <
> >  smare...@maprtech.com>
> >  > wrote:
> >  >
> >  >> Gentle reminder... Appreciate your help with this.
> >  >>
> >  >> --
> >  >> Sent from mobile
> >  >> On Nov 8, 2015 7:02 PM, "Santosh Marella"  >
> >  wrote:
> >  >>
> >  >> > Hi Ben,Danese,Luciano,Ted,
> >  >> >
> >  >> >   In INFRA-10696 [1], we've requested for a SVN directory to
> >  publish
> >  >> > Myriad's website. However, it seems to need a svn authorization
> >  >> template to
> >  >> > be submitted by one of the Mentors in order to push website
> >  commits to
> >  >> SVN.
> >  >> > Can one of you please help with that?
> >  >> >
> >  >> >   [1] https://issues.apache.org/jira/browse/INFRA-10696
> >  >> >
> >  >> > Thanks,
> >  >> > Santosh
> >  >> >
> >  >>
> >  >
> >  >
> > 
> > >>>
> > >>
> > >
> > >
> > > --
> > > Benjamin Hindman
> > > Founder of Mesosphere and Co-Creator of Apache Mesos
> > > Mesosphere Inc.  
> > >
> > > Follow us on Twitter: @mesosphere 
> > >
> >
> >
> >
> > --
> > Luciano Resende
> > http://people.apache.org/~lresende
> > http://twitter.com/lresende1975
> > http://lresende.blogspot.com/
> >
>



-- 
Luciano Resende
http://people.apache.org/~lresende
http://twitter.com/lresende1975
http://lresende.blogspot.com/

Re: [Help needed from Mentors] SVN PubSub

[Santosh Marella]

More Apache Ids need to be added: me,darinj,sdaingade

--
Sent from mobile
On Nov 17, 2015 7:44 AM, "Luciano Resende"  wrote:

> Are we missing anyone ?
>
> myriad=benh,danese,kensipe,lresende,mohit,smarella,tdunning
>
>
> On Mon, Nov 16, 2015 at 5:23 AM, Benjamin Hindman 
> wrote:
>
> > I've never edited an "svn authorization template" before, is there some
> > documentation that one of the more experienced mentors can point us/me
> to?
> >
> > On Fri, Nov 13, 2015 at 7:47 AM, Adam Bordelon 
> wrote:
> >
> >> Correct. As mentioned in the Nov 4-5 comments on INFRA-10696
> >> <
> https://issues.apache.org/jira/browse/INFRA-10696?focusedCommentId=14990668&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14990668
> >,
> >> we got 403 Forbidden errors when trying to commit. Geoffrey from INFRA
> >> tells us, "You need to get your mentor (or someone with enough karma) to
> >> edit the svn authorization template and add a list of names from myriad
> >> that you want to be able to commit."
> >>
> >> On Thu, Nov 12, 2015 at 6:37 PM, Jim Klucar  wrote:
> >>
> >>> Ted,
> >>>
> >>> Santosh and Adam both tried to commit to the svn repo but got
> permission
> >>> denied. Looks like they just need some kind of svn permissions to push
> our
> >>> site out.
> >>>
> >>> Jim
> >>>
> >>> On Thursday, November 12, 2015, Ted Dunning 
> >>> wrote:
> >>>
>  OK.
> 
>  I am really confused.  I hear you need SVN something.  And you need
> some
>  kind of web action.
> 
>  What specifically do you need?
> 
>  (I am heavily loaded right now and in a difficult time zone)
> 
> 
>  On Fri, Nov 13, 2015 at 5:34 AM, Adam Bordelon 
>  wrote:
> 
>  > + Mentor email addresses, in case filters are failing us.
>  > Any mentor, help! We still need help with this.
>  >
>  > Don't you want to see this content:
>  http://klucar.github.io/myriad-site/
>  > at this address: http://myriad.incubator.apache.org/
>  > ?
>  >
>  > On Tue, Nov 10, 2015 at 9:34 AM, Santosh Marella <
>  smare...@maprtech.com>
>  > wrote:
>  >
>  >> Gentle reminder... Appreciate your help with this.
>  >>
>  >> --
>  >> Sent from mobile
>  >> On Nov 8, 2015 7:02 PM, "Santosh Marella" 
>  wrote:
>  >>
>  >> > Hi Ben,Danese,Luciano,Ted,
>  >> >
>  >> >   In INFRA-10696 [1], we've requested for a SVN directory to
>  publish
>  >> > Myriad's website. However, it seems to need a svn authorization
>  >> template to
>  >> > be submitted by one of the Mentors in order to push website
>  commits to
>  >> SVN.
>  >> > Can one of you please help with that?
>  >> >
>  >> >   [1] https://issues.apache.org/jira/browse/INFRA-10696
>  >> >
>  >> > Thanks,
>  >> > Santosh
>  >> >
>  >>
>  >
>  >
> 
> >>>
> >>
> >
> >
> > --
> > Benjamin Hindman
> > Founder of Mesosphere and Co-Creator of Apache Mesos
> > Mesosphere Inc.  
> >
> > Follow us on Twitter: @mesosphere 
> >
>
>
>
> --
> Luciano Resende
> http://people.apache.org/~lresende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>

Re: [Help needed from Mentors] SVN PubSub

[Luciano Resende]

Are we missing anyone ?

myriad=benh,danese,kensipe,lresende,mohit,smarella,tdunning


On Mon, Nov 16, 2015 at 5:23 AM, Benjamin Hindman 
wrote:

> I've never edited an "svn authorization template" before, is there some
> documentation that one of the more experienced mentors can point us/me to?
>
> On Fri, Nov 13, 2015 at 7:47 AM, Adam Bordelon  wrote:
>
>> Correct. As mentioned in the Nov 4-5 comments on INFRA-10696
>> ,
>> we got 403 Forbidden errors when trying to commit. Geoffrey from INFRA
>> tells us, "You need to get your mentor (or someone with enough karma) to
>> edit the svn authorization template and add a list of names from myriad
>> that you want to be able to commit."
>>
>> On Thu, Nov 12, 2015 at 6:37 PM, Jim Klucar  wrote:
>>
>>> Ted,
>>>
>>> Santosh and Adam both tried to commit to the svn repo but got permission
>>> denied. Looks like they just need some kind of svn permissions to push our
>>> site out.
>>>
>>> Jim
>>>
>>> On Thursday, November 12, 2015, Ted Dunning 
>>> wrote:
>>>
 OK.

 I am really confused.  I hear you need SVN something.  And you need some
 kind of web action.

 What specifically do you need?

 (I am heavily loaded right now and in a difficult time zone)


 On Fri, Nov 13, 2015 at 5:34 AM, Adam Bordelon 
 wrote:

 > + Mentor email addresses, in case filters are failing us.
 > Any mentor, help! We still need help with this.
 >
 > Don't you want to see this content:
 http://klucar.github.io/myriad-site/
 > at this address: http://myriad.incubator.apache.org/
 > ?
 >
 > On Tue, Nov 10, 2015 at 9:34 AM, Santosh Marella <
 smare...@maprtech.com>
 > wrote:
 >
 >> Gentle reminder... Appreciate your help with this.
 >>
 >> --
 >> Sent from mobile
 >> On Nov 8, 2015 7:02 PM, "Santosh Marella" 
 wrote:
 >>
 >> > Hi Ben,Danese,Luciano,Ted,
 >> >
 >> >   In INFRA-10696 [1], we've requested for a SVN directory to
 publish
 >> > Myriad's website. However, it seems to need a svn authorization
 >> template to
 >> > be submitted by one of the Mentors in order to push website
 commits to
 >> SVN.
 >> > Can one of you please help with that?
 >> >
 >> >   [1] https://issues.apache.org/jira/browse/INFRA-10696
 >> >
 >> > Thanks,
 >> > Santosh
 >> >
 >>
 >
 >

>>>
>>
>
>
> --
> Benjamin Hindman
> Founder of Mesosphere and Co-Creator of Apache Mesos
> Mesosphere Inc.  
>
> Follow us on Twitter: @mesosphere 
>



-- 
Luciano Resende
http://people.apache.org/~lresende
http://twitter.com/lresende1975
http://lresende.blogspot.com/