[ https://issues.apache.org/jira/browse/SPARK-19832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wenchen Fan resolved SPARK-19832. --------------------------------- Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 17173 [https://github.com/apache/spark/pull/17173] > DynamicPartitionWriteTask should escape the partition name > ----------------------------------------------------------- > > Key: SPARK-19832 > URL: https://issues.apache.org/jira/browse/SPARK-19832 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.2.0 > Reporter: Song Jun > Fix For: 2.2.0 > > > Currently in DynamicPartitionWriteTask, when we get the paritionPath of a > parition, we just escape the partition value, not escape the partition name. > this will cause some problems for some special partition name situation, for > example : > 1) if the partition name contains '%' etc, there will be two partition path > created in the filesytem, one is for escaped path like '/path/a%25b=1', > another is for unescaped path like '/path/a%b=1'. > and the data inserted stored in unescaped path, while the show partitions > table will return 'a%25b=1' which the partition name is escaped. So here it > is not consist. And I think the data should be stored in the escaped path in > filesystem, which Hive2.0.0 also have the same action. > 2) if the partition name contains ':', there will throw exception that new > Path("/path","a:b"), this is illegal which has a colon in the relative path. > {code} > java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative > path in absolute URI: a:b > at org.apache.hadoop.fs.Path.initialize(Path.java:205) > at org.apache.hadoop.fs.Path.<init>(Path.java:171) > at org.apache.hadoop.fs.Path.<init>(Path.java:88) > ... 48 elided > Caused by: java.net.URISyntaxException: Relative path in absolute URI: a:b > at java.net.URI.checkPath(URI.java:1823) > at java.net.URI.<init>(URI.java:745) > at org.apache.hadoop.fs.Path.initialize(Path.java:202) > ... 50 more > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org