[ https://issues.apache.org/jira/browse/SPARK-7141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14512794#comment-14512794 ]
Eric O. LEBIGOT (EOL) commented on SPARK-7141: ---------------------------------------------- Ah, thanks: it's good to know where the issue comes from. As far as I understand, it does not look like it's going to be fixed any soon ("minor" issue, and Amazon deprecating s3://…). I will try s3n:// instead and see if this works and changes anything. > saveAsTextFile() on S3 first creates empty prefix > ------------------------------------------------- > > Key: SPARK-7141 > URL: https://issues.apache.org/jira/browse/SPARK-7141 > Project: Spark > Issue Type: Bug > Components: PySpark > Affects Versions: 1.3.1 > Environment: OS X 10.10 > Reporter: Eric O. LEBIGOT (EOL) > > Using {{saveAsTextFile("s3://bucket/prefix"}} actually adds an empty prefix, > i.e. it writes to {{s3://bucket//prefix}} (note the double slash). > Example code (in a {{pyspark}} shell): > {{rdd = sc.parallelize("abcd")}} > {{rdd.saveAsTextFile("s3://bucket/prefix")}} > This is quite annoying, as the files cannot be saved in the intended location > (they can be read, though, with the original path: > {{sc.textFile("s3://bucket/prefix"}}, but the AWS console does not show them > in the right place). > Also, many {{block_*}} files are created directly in the bucket: shouldn't > they be deleted? (This may be a separate issue, but maybe it is a path issue > as well.) -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org