Hi,
thanks for answer, I'll open a ticket.
In the meantime - I have found a workaround. The recipe is the following:
1. Create a new account/group on all machines (lets call it sparkuser).
Run spark from this account.
2. Add your user to group sparkuser.
3. If you decide to write
Hi,
thanks for answer. The rights are
drwxr-xr-x 3 tfruboes all 5632 05-19 15:40 test19EE/
I have tried setting the rights to 777 for this directory prior to
execution. This does not get propagated down the chain, ie the directory
created as a result of the save call (namesAndAges.parquet2
You could try setting `SPARK_USER` to the user under which your workers are
running. I couldn't find many references to this variable, but at least
Yarn and Mesos take it into account when spawning executors. Chances are
that standalone mode also does it.
iulian
On Wed, May 20, 2015 at 9:29 AM,
Thanks for a suggestion. I have tried playing with it, sc.sparkUser()
gives me expected user name, but it doesnt solve the problem. From a
quick search through the spark code it seems to me, that this setting is
effective only for yarn and mesos.
I think the workaround for the problem could
Could you file a JIRA for this?
The executor should run under the user who submit a job, I think.
On Wed, May 20, 2015 at 2:40 AM, Tomasz Fruboes
tomasz.frub...@fuw.edu.pl wrote:
Thanks for a suggestion. I have tried playing with it, sc.sparkUser() gives
me expected user name, but it doesnt
Dear Experts,
we have a spark cluster (standalone mode) in which master and workers
are started from root account. Everything runs correctly to the point
when we try doing operations such as
dataFrame.select(name, age).save(ofile, parquet)
or
rdd.saveAsPickleFile(ofile)
, where
It surprises me, could you list the owner information of
/mnt/lustre/bigdata/med_home/tmp/test19EE/ ?
On Tue, May 19, 2015 at 8:15 AM, Tomasz Fruboes
tomasz.frub...@fuw.edu.pl wrote:
Dear Experts,
we have a spark cluster (standalone mode) in which master and workers are
started from root
