[jira] [Assigned] (SPARK-19832) DynamicPartitionWriteTask should escape the partition name
[
https://issues.apache.org/jira/browse/SPARK-19832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenchen Fan reassigned SPARK-19832:
---
Assignee: Song Jun
> DynamicPartitionWriteTask should escape the partition name
> ---
>
> Key: SPARK-19832
> URL: https://issues.apache.org/jira/browse/SPARK-19832
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.2.0
>Reporter: Song Jun
>Assignee: Song Jun
> Fix For: 2.2.0
>
>
> Currently in DynamicPartitionWriteTask, when we get the paritionPath of a
> parition, we just escape the partition value, not escape the partition name.
> this will cause some problems for some special partition name situation, for
> example :
> 1) if the partition name contains '%' etc, there will be two partition path
> created in the filesytem, one is for escaped path like '/path/a%25b=1',
> another is for unescaped path like '/path/a%b=1'.
> and the data inserted stored in unescaped path, while the show partitions
> table will return 'a%25b=1' which the partition name is escaped. So here it
> is not consist. And I think the data should be stored in the escaped path in
> filesystem, which Hive2.0.0 also have the same action.
> 2) if the partition name contains ':', there will throw exception that new
> Path("/path","a:b"), this is illegal which has a colon in the relative path.
> {code}
> java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative
> path in absolute URI: a:b
> at org.apache.hadoop.fs.Path.initialize(Path.java:205)
> at org.apache.hadoop.fs.Path.(Path.java:171)
> at org.apache.hadoop.fs.Path.(Path.java:88)
> ... 48 elided
> Caused by: java.net.URISyntaxException: Relative path in absolute URI: a:b
> at java.net.URI.checkPath(URI.java:1823)
> at java.net.URI.(URI.java:745)
> at org.apache.hadoop.fs.Path.initialize(Path.java:202)
> ... 50 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-19832) DynamicPartitionWriteTask should escape the partition name
[
https://issues.apache.org/jira/browse/SPARK-19832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-19832:
Assignee: (was: Apache Spark)
> DynamicPartitionWriteTask should escape the partition name
> ---
>
> Key: SPARK-19832
> URL: https://issues.apache.org/jira/browse/SPARK-19832
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.2.0
>Reporter: Song Jun
>
> Currently in DynamicPartitionWriteTask, when we get the paritionPath of a
> parition, we just escape the partition value, not escape the partition name.
> this will cause some problems for some special partition name situation, for
> example :
> 1) if the partition name contains '%' etc, there will be two partition path
> created in the filesytem, one is for escaped path like '/path/a%25b=1',
> another is for unescaped path like '/path/a%b=1'.
> and the data inserted stored in unescaped path, while the show partitions
> table will return 'a%25b=1' which the partition name is escaped. So here it
> is not consist. And I think the data should be stored in the escaped path in
> filesystem, which Hive2.0.0 also have the same action.
> 2) if the partition name contains ':', there will throw exception that new
> Path("/path","a:b"), this is illegal which has a colon in the relative path.
> {code}
> java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative
> path in absolute URI: a:b
> at org.apache.hadoop.fs.Path.initialize(Path.java:205)
> at org.apache.hadoop.fs.Path.(Path.java:171)
> at org.apache.hadoop.fs.Path.(Path.java:88)
> ... 48 elided
> Caused by: java.net.URISyntaxException: Relative path in absolute URI: a:b
> at java.net.URI.checkPath(URI.java:1823)
> at java.net.URI.(URI.java:745)
> at org.apache.hadoop.fs.Path.initialize(Path.java:202)
> ... 50 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-19832) DynamicPartitionWriteTask should escape the partition name
[
https://issues.apache.org/jira/browse/SPARK-19832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-19832:
Assignee: Apache Spark
> DynamicPartitionWriteTask should escape the partition name
> ---
>
> Key: SPARK-19832
> URL: https://issues.apache.org/jira/browse/SPARK-19832
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.2.0
>Reporter: Song Jun
>Assignee: Apache Spark
>
> Currently in DynamicPartitionWriteTask, when we get the paritionPath of a
> parition, we just escape the partition value, not escape the partition name.
> this will cause some problems for some special partition name situation, for
> example :
> 1) if the partition name contains '%' etc, there will be two partition path
> created in the filesytem, one is for escaped path like '/path/a%25b=1',
> another is for unescaped path like '/path/a%b=1'.
> and the data inserted stored in unescaped path, while the show partitions
> table will return 'a%25b=1' which the partition name is escaped. So here it
> is not consist. And I think the data should be stored in the escaped path in
> filesystem, which Hive2.0.0 also have the same action.
> 2) if the partition name contains ':', there will throw exception that new
> Path("/path","a:b"), this is illegal which has a colon in the relative path.
> {code}
> java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative
> path in absolute URI: a:b
> at org.apache.hadoop.fs.Path.initialize(Path.java:205)
> at org.apache.hadoop.fs.Path.(Path.java:171)
> at org.apache.hadoop.fs.Path.(Path.java:88)
> ... 48 elided
> Caused by: java.net.URISyntaxException: Relative path in absolute URI: a:b
> at java.net.URI.checkPath(URI.java:1823)
> at java.net.URI.(URI.java:745)
> at org.apache.hadoop.fs.Path.initialize(Path.java:202)
> ... 50 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
