[ https://issues.apache.org/jira/browse/SPARK-7141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric O. LEBIGOT (EOL) updated SPARK-7141: ----------------------------------------- Description: Using {{saveAsTextFile("s3://bucket/prefix"}} actually adds an empty prefix, i.e. it writes to {{s3://bucket//prefix}} (note the double slash). Example code (in a {{pyspark}} shell): {{rdd = sc.parallelize("abcd")}} {{rdd.saveAsTextFile("s3://bucket/prefix")}} This is quite annoying, as the files cannot be saved in the intended location (they can be read, though, with the original path: {{sc.textFile("s3://bucket/prefix"}}, but the AWS console does not show them in the right place). Also, many {{block_*}} files are created directly in the bucket: shouldn't they be deleted? (This may be a separate issue, but maybe it is a path issue as well.) was: Using {{saveAsTextFile("s3://bucket/prefix"}} actually adds an empty prefix, i.e. it writes to {{s3://bucket//prefix}} (note the double slash). Example code (in a {{pyspark}} shell): {{rdd = sc.parallelize("abcd")}} {{rdd.saveAsTextFile("s3://bucket/prefix")}) This is quite annoying, as the files cannot be saved in the intended location (they can be read, though, with the original path: {{sc.textFile("s3://bucket/prefix"}}, but the AWS console does not show them in the right place). Also, many {{block_*}} files are created directly in the bucket: shouldn't they be deleted? (This may be a separate issue, but maybe it is a path issue as well.) > saveAsTextFile() on S3 first creates empty prefix > ------------------------------------------------- > > Key: SPARK-7141 > URL: https://issues.apache.org/jira/browse/SPARK-7141 > Project: Spark > Issue Type: Bug > Components: PySpark > Affects Versions: 1.3.1 > Environment: OS X 10.10 > Reporter: Eric O. LEBIGOT (EOL) > > Using {{saveAsTextFile("s3://bucket/prefix"}} actually adds an empty prefix, > i.e. it writes to {{s3://bucket//prefix}} (note the double slash). > Example code (in a {{pyspark}} shell): > {{rdd = sc.parallelize("abcd")}} > {{rdd.saveAsTextFile("s3://bucket/prefix")}} > This is quite annoying, as the files cannot be saved in the intended location > (they can be read, though, with the original path: > {{sc.textFile("s3://bucket/prefix"}}, but the AWS console does not show them > in the right place). > Also, many {{block_*}} files are created directly in the bucket: shouldn't > they be deleted? (This may be a separate issue, but maybe it is a path issue > as well.) -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org