[ https://issues.apache.org/jira/browse/SPARK-7847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14557728#comment-14557728 ]
Apache Spark commented on SPARK-7847: ------------------------------------- User 'liancheng' has created a pull request for this issue: https://github.com/apache/spark/pull/6389 > Fix dynamic partition path escaping > ----------------------------------- > > Key: SPARK-7847 > URL: https://issues.apache.org/jira/browse/SPARK-7847 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 1.3.0, 1.3.1, 1.4.0 > Reporter: Cheng Lian > Assignee: Cheng Lian > Priority: Critical > > Background: when writing dynamic partitions, partition values are converted > to string and escaped if necessary. For example, a partition column {{p}} of > type {{String}} may have a value {{A/B}}, then the corresponding partition > directory name is escaped into {{p=A%2fB}}. > Currently, there are two issues regarding to dynamic partition path escaping. > The first issue is that, when reading back partition values, escaped strings > are not unescaped. This one is easy to fix. > The second issue is more subtle. In [PR > #5381|https://github.com/apache/spark/pull/5381/files#diff-c69b9e667e93b7e4693812cc72abb65fR492] > we tried to use {{Path.toUri.toString}} to fix an escaping issue related to > S3 credentials with {{/}} character. Unfortunately, {{Path.toUri.toString}} > also escapes {{%}} characters in the path. Thus, using the dynamic > partitioning case mentioned above, {{p=A%2fB}} is double escaped into > {{p=A%252fB}} ({{%}} escaped into {{%25}}). > The expected behavior here should be, only escaping the URI user info part > (S3 key and secret) but leave all other components untouched. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org