[jira] [Assigned] (SPARK-8887) Explicitly define which data types can be used as dynamic partition columns
[
https://issues.apache.org/jira/browse/SPARK-8887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-8887:
---
Assignee: Cheng Lian (was: Apache Spark)
Explicitly define which data types can be used as dynamic partition columns
---
Key: SPARK-8887
URL: https://issues.apache.org/jira/browse/SPARK-8887
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 1.4.0
Reporter: Cheng Lian
Assignee: Cheng Lian
{{InsertIntoHadoopFsRelation}} implements Hive compatible dynamic
partitioning insertion, which uses {{String.valueOf}} to write encode
partition column values into dynamic partition directories. This actually
limits the data types that can be used in partition column. For example,
string representation of {{StructType}} values is not well defined. However,
this limitation is not explicitly enforced.
There are several things we can improve:
# Enforce dynamic column data type requirements by adding analysis rules and
throws {{AnalysisException}} when violation occurs.
# Abstract away string representation of various data types, so that we don't
need to convert internal representation types (e.g. {{UTF8String}}) to
external types (e.g. {{String}}). A set of Hive compatible implementations
should be provided to ensure compatibility with Hive.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-8887) Explicitly define which data types can be used as dynamic partition columns
[
https://issues.apache.org/jira/browse/SPARK-8887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-8887:
---
Assignee: Apache Spark (was: Cheng Lian)
Explicitly define which data types can be used as dynamic partition columns
---
Key: SPARK-8887
URL: https://issues.apache.org/jira/browse/SPARK-8887
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 1.4.0
Reporter: Cheng Lian
Assignee: Apache Spark
{{InsertIntoHadoopFsRelation}} implements Hive compatible dynamic
partitioning insertion, which uses {{String.valueOf}} to write encode
partition column values into dynamic partition directories. This actually
limits the data types that can be used in partition column. For example,
string representation of {{StructType}} values is not well defined. However,
this limitation is not explicitly enforced.
There are several things we can improve:
# Enforce dynamic column data type requirements by adding analysis rules and
throws {{AnalysisException}} when violation occurs.
# Abstract away string representation of various data types, so that we don't
need to convert internal representation types (e.g. {{UTF8String}}) to
external types (e.g. {{String}}). A set of Hive compatible implementations
should be provided to ensure compatibility with Hive.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-8887) Explicitly define which data types can be used as dynamic partition columns
[
https://issues.apache.org/jira/browse/SPARK-8887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cheng Lian reassigned SPARK-8887:
-
Assignee: Cheng Lian
Explicitly define which data types can be used as dynamic partition columns
---
Key: SPARK-8887
URL: https://issues.apache.org/jira/browse/SPARK-8887
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 1.4.0
Reporter: Cheng Lian
Assignee: Cheng Lian
{{InsertIntoHadoopFsRelation}} implements Hive compatible dynamic
partitioning insertion, which uses {{String.valueOf}} to write encode
partition column values into dynamic partition directories. This actually
limits the data types that can be used in partition column. For example,
string representation of {{StructType}} values is not well defined. However,
this limitation is not explicitly enforced.
There are several things we can improve:
# Enforce dynamic column data type requirements by adding analysis rules and
throws {{AnalysisException}} when violation occurs.
# Abstract away string representation of various data types, so that we don't
need to convert internal representation types (e.g. {{UTF8String}}) to
external types (e.g. {{String}}). A set of Hive compatible implementations
should be provided to ensure compatibility with Hive.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
