Re: writing to HDFS with a given username
Well, it only uses my user name when I run my application in local mode (i.e. spark is running on my laptop with a master url of local.) Not a general solution for you I'm afraid! On 12/12/2013 5:38 PM, Koert Kuipers wrote: Hey Philip, how do you get spark to write to hdfs with your user name? When i use spark it writes to hdfs as the user that runs the spark services... i wish it read and wrote as me. On Thu, Dec 12, 2013 at 6:37 PM, Philip Ogren philip.og...@oracle.com mailto:philip.og...@oracle.com wrote: When I call rdd.saveAsTextFile(hdfs://...) it uses my username to write to the HDFS drive. If I try to write to an HDFS directory that I do not have permissions to, then I get an error like this: Permission denied: user=me, access=WRITE, inode=/user/you/:you:us:drwxr-xr-x I can obviously get around this by changing the permissions on the directory /user/you. However, is it possible to call rdd.saveAsText with an alternate username and password? Thanks, Philip
Re: writing to HDFS with a given username
thats great. didn't realize this was in master already. On Thu, Dec 12, 2013 at 8:10 PM, Shao, Saisai saisai.s...@intel.com wrote: Hi Koert, Spark with multi-user support has been merged in master branch with patch ( https://github.com/apache/incubator-spark/pull/23), you can check out the master branch. These patch can support to access hdfs with the username you start the Spark application, not the one who starts Spark service. Thanks Jerry *From:* Koert Kuipers [mailto:ko...@tresata.com] *Sent:* Friday, December 13, 2013 8:39 AM *To:* user@spark.incubator.apache.org *Subject:* Re: writing to HDFS with a given username Hey Philip, how do you get spark to write to hdfs with your user name? When i use spark it writes to hdfs as the user that runs the spark services... i wish it read and wrote as me. On Thu, Dec 12, 2013 at 6:37 PM, Philip Ogren philip.og...@oracle.com wrote: When I call rdd.saveAsTextFile(hdfs://...) it uses my username to write to the HDFS drive. If I try to write to an HDFS directory that I do not have permissions to, then I get an error like this: Permission denied: user=me, access=WRITE, inode=/user/you/:you:us:drwxr-xr-x I can obviously get around this by changing the permissions on the directory /user/you. However, is it possible to call rdd.saveAsText with an alternate username and password? Thanks, Philip
Re: writing to HDFS with a given username
Yup, this should be in Spark 0.9 and 0.8.1. Matei On Dec 13, 2013, at 9:41 AM, Koert Kuipers ko...@tresata.com wrote: thats great. didn't realize this was in master already. On Thu, Dec 12, 2013 at 8:10 PM, Shao, Saisai saisai.s...@intel.com wrote: Hi Koert, Spark with multi-user support has been merged in master branch with patch (https://github.com/apache/incubator-spark/pull/23), you can check out the master branch. These patch can support to access hdfs with the username you start the Spark application, not the one who starts Spark service. Thanks Jerry From: Koert Kuipers [mailto:ko...@tresata.com] Sent: Friday, December 13, 2013 8:39 AM To: user@spark.incubator.apache.org Subject: Re: writing to HDFS with a given username Hey Philip, how do you get spark to write to hdfs with your user name? When i use spark it writes to hdfs as the user that runs the spark services... i wish it read and wrote as me. On Thu, Dec 12, 2013 at 6:37 PM, Philip Ogren philip.og...@oracle.com wrote: When I call rdd.saveAsTextFile(hdfs://...) it uses my username to write to the HDFS drive. If I try to write to an HDFS directory that I do not have permissions to, then I get an error like this: Permission denied: user=me, access=WRITE, inode=/user/you/:you:us:drwxr-xr-x I can obviously get around this by changing the permissions on the directory /user/you. However, is it possible to call rdd.saveAsText with an alternate username and password? Thanks, Philip
Re: writing to HDFS with a given username
Hey Philip, how do you get spark to write to hdfs with your user name? When i use spark it writes to hdfs as the user that runs the spark services... i wish it read and wrote as me. On Thu, Dec 12, 2013 at 6:37 PM, Philip Ogren philip.og...@oracle.comwrote: When I call rdd.saveAsTextFile(hdfs://...) it uses my username to write to the HDFS drive. If I try to write to an HDFS directory that I do not have permissions to, then I get an error like this: Permission denied: user=me, access=WRITE, inode=/user/you/:you:us: drwxr-xr-x I can obviously get around this by changing the permissions on the directory /user/you. However, is it possible to call rdd.saveAsText with an alternate username and password? Thanks, Philip
