Is there any consideration of allowing the application to pass in
arbitrary username/password credentials for writing to hdfs? Perhaps the
usefulness of this evaporates when I'm not in the habit of running my
application in local mode for testing purposes.
On 12/12/2013 6:10 PM, Shao, Saisai wrote:
Hi Koert,
Spark with multi-user support has been merged in master branch with
patch (https://github.com/apache/incubator-spark/pull/23), you can
check out the master branch. These patch can support to access hdfs
with the username you start the Spark application, not the one who
starts Spark service.
Thanks
Jerry
*From:*Koert Kuipers [mailto:ko...@tresata.com]
*Sent:* Friday, December 13, 2013 8:39 AM
*To:* user@spark.incubator.apache.org
*Subject:* Re: writing to HDFS with a given username
Hey Philip,
how do you get spark to write to hdfs with your user name? When i use
spark it writes to hdfs as the user that runs the spark services... i
wish it read and wrote as me.
On Thu, Dec 12, 2013 at 6:37 PM, Philip Ogren <philip.og...@oracle.com
<mailto:philip.og...@oracle.com>> wrote:
When I call rdd.saveAsTextFile("hdfs://...") it uses my username to
write to the HDFS drive. If I try to write to an HDFS directory that
I do not have permissions to, then I get an error like this:
Permission denied: user=me, access=WRITE,
inode="/user/you/":you:us:drwxr-xr-x
I can obviously get around this by changing the permissions on the
directory /user/you. However, is it possible to call rdd.saveAsText
with an alternate username and password?
Thanks,
Philip
