[jira] [Updated] (SPARK-10186) Add support for more postgres column types
[
https://issues.apache.org/jira/browse/SPARK-10186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Reynold Xin updated SPARK-10186:
Labels: (was: array json postgres sql struct)
> Add support for more postgres column types
> --
>
> Key: SPARK-10186
> URL: https://issues.apache.org/jira/browse/SPARK-10186
> Project: Spark
> Issue Type: New Feature
> Components: SQL
>Affects Versions: 1.4.1
> Environment: Ubuntu on AWS
>Reporter: Simeon Simeonov
>
> The specific observations below are based on Postgres 9.4 tables accessed via
> the postgresql-9.4-1201.jdbc41.jar driver. However, based on the behavior, I
> would expect the problem to exists for all external SQL databases.
> - *json and jsonb columns generate {{java.sql.SQLException: Unsupported type
> }}*. While it is reasonable to not support dynamic schema discovery of
> JSON columns automatically (it requires two passes over the data), a better
> behavior would be to create a String column and return the JSON.
> - *Array columns generate {{java.sql.SQLException: Unsupported type 2003}}*.
> This is true even for simple types, e.g., {{text[]}}. A better behavior would
> be be create an Array column.
> - *Custom type columns are mapped to a String column.* This behavior is
> harder to understand as the schema of a custom type is fixed and therefore
> mappable to a Struct column. The automatic conversion to a string is also
> inconsistent when compared to json and array column handling.
> The exceptions are thrown by
> {{org.apache.spark.sql.jdbc.JDBCRDD$.org$apache$spark$sql$jdbc$JDBCRDD$$getCatalystType(JDBCRDD.scala:100)}}
> so this definitely looks like a Spark SQL and not a JDBC problem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-10186) Add support for more postgres column types
[
https://issues.apache.org/jira/browse/SPARK-10186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Reynold Xin updated SPARK-10186:
Assignee: Marius Van Niekerk
> Add support for more postgres column types
> --
>
> Key: SPARK-10186
> URL: https://issues.apache.org/jira/browse/SPARK-10186
> Project: Spark
> Issue Type: New Feature
> Components: SQL
>Affects Versions: 1.4.1
> Environment: Ubuntu on AWS
>Reporter: Simeon Simeonov
>Assignee: Marius Van Niekerk
> Fix For: 1.6.0
>
>
> The specific observations below are based on Postgres 9.4 tables accessed via
> the postgresql-9.4-1201.jdbc41.jar driver. However, based on the behavior, I
> would expect the problem to exists for all external SQL databases.
> - *json and jsonb columns generate {{java.sql.SQLException: Unsupported type
> }}*. While it is reasonable to not support dynamic schema discovery of
> JSON columns automatically (it requires two passes over the data), a better
> behavior would be to create a String column and return the JSON.
> - *Array columns generate {{java.sql.SQLException: Unsupported type 2003}}*.
> This is true even for simple types, e.g., {{text[]}}. A better behavior would
> be be create an Array column.
> - *Custom type columns are mapped to a String column.* This behavior is
> harder to understand as the schema of a custom type is fixed and therefore
> mappable to a Struct column. The automatic conversion to a string is also
> inconsistent when compared to json and array column handling.
> The exceptions are thrown by
> {{org.apache.spark.sql.jdbc.JDBCRDD$.org$apache$spark$sql$jdbc$JDBCRDD$$getCatalystType(JDBCRDD.scala:100)}}
> so this definitely looks like a Spark SQL and not a JDBC problem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-10186) Add support for more postgres column types
[
https://issues.apache.org/jira/browse/SPARK-10186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Armbrust updated SPARK-10186:
-
Summary: Add support for more postgres column types (was: Inconsistent
handling of complex column types in external databases)
> Add support for more postgres column types
> --
>
> Key: SPARK-10186
> URL: https://issues.apache.org/jira/browse/SPARK-10186
> Project: Spark
> Issue Type: New Feature
> Components: SQL
>Affects Versions: 1.4.1
> Environment: Ubuntu on AWS
>Reporter: Simeon Simeonov
> Labels: array, json, postgres, sql, struct
>
> The specific observations below are based on Postgres 9.4 tables accessed via
> the postgresql-9.4-1201.jdbc41.jar driver. However, based on the behavior, I
> would expect the problem to exists for all external SQL databases.
> - *json and jsonb columns generate {{java.sql.SQLException: Unsupported type
> }}*. While it is reasonable to not support dynamic schema discovery of
> JSON columns automatically (it requires two passes over the data), a better
> behavior would be to create a String column and return the JSON.
> - *Array columns generate {{java.sql.SQLException: Unsupported type 2003}}*.
> This is true even for simple types, e.g., {{text[]}}. A better behavior would
> be be create an Array column.
> - *Custom type columns are mapped to a String column.* This behavior is
> harder to understand as the schema of a custom type is fixed and therefore
> mappable to a Struct column. The automatic conversion to a string is also
> inconsistent when compared to json and array column handling.
> The exceptions are throw by
> {{org.apache.spark.sql.jdbc.JDBCRDD$.org$apache$spark$sql$jdbc$JDBCRDD$$getCatalystType(JDBCRDD.scala:100)}}
> so this definitely looks like a Spark SQL and not a JDBC problem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-10186) Add support for more postgres column types
[
https://issues.apache.org/jira/browse/SPARK-10186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Simeon Simeonov updated SPARK-10186:
Description:
The specific observations below are based on Postgres 9.4 tables accessed via
the postgresql-9.4-1201.jdbc41.jar driver. However, based on the behavior, I
would expect the problem to exists for all external SQL databases.
- *json and jsonb columns generate {{java.sql.SQLException: Unsupported type
}}*. While it is reasonable to not support dynamic schema discovery of JSON
columns automatically (it requires two passes over the data), a better behavior
would be to create a String column and return the JSON.
- *Array columns generate {{java.sql.SQLException: Unsupported type 2003}}*.
This is true even for simple types, e.g., {{text[]}}. A better behavior would
be be create an Array column.
- *Custom type columns are mapped to a String column.* This behavior is harder
to understand as the schema of a custom type is fixed and therefore mappable to
a Struct column. The automatic conversion to a string is also inconsistent when
compared to json and array column handling.
The exceptions are thrown by
{{org.apache.spark.sql.jdbc.JDBCRDD$.org$apache$spark$sql$jdbc$JDBCRDD$$getCatalystType(JDBCRDD.scala:100)}}
so this definitely looks like a Spark SQL and not a JDBC problem.
was:
The specific observations below are based on Postgres 9.4 tables accessed via
the postgresql-9.4-1201.jdbc41.jar driver. However, based on the behavior, I
would expect the problem to exists for all external SQL databases.
- *json and jsonb columns generate {{java.sql.SQLException: Unsupported type
}}*. While it is reasonable to not support dynamic schema discovery of JSON
columns automatically (it requires two passes over the data), a better behavior
would be to create a String column and return the JSON.
- *Array columns generate {{java.sql.SQLException: Unsupported type 2003}}*.
This is true even for simple types, e.g., {{text[]}}. A better behavior would
be be create an Array column.
- *Custom type columns are mapped to a String column.* This behavior is harder
to understand as the schema of a custom type is fixed and therefore mappable to
a Struct column. The automatic conversion to a string is also inconsistent when
compared to json and array column handling.
The exceptions are throw by
{{org.apache.spark.sql.jdbc.JDBCRDD$.org$apache$spark$sql$jdbc$JDBCRDD$$getCatalystType(JDBCRDD.scala:100)}}
so this definitely looks like a Spark SQL and not a JDBC problem.
> Add support for more postgres column types
> --
>
> Key: SPARK-10186
> URL: https://issues.apache.org/jira/browse/SPARK-10186
> Project: Spark
> Issue Type: New Feature
> Components: SQL
>Affects Versions: 1.4.1
> Environment: Ubuntu on AWS
>Reporter: Simeon Simeonov
> Labels: array, json, postgres, sql, struct
>
> The specific observations below are based on Postgres 9.4 tables accessed via
> the postgresql-9.4-1201.jdbc41.jar driver. However, based on the behavior, I
> would expect the problem to exists for all external SQL databases.
> - *json and jsonb columns generate {{java.sql.SQLException: Unsupported type
> }}*. While it is reasonable to not support dynamic schema discovery of
> JSON columns automatically (it requires two passes over the data), a better
> behavior would be to create a String column and return the JSON.
> - *Array columns generate {{java.sql.SQLException: Unsupported type 2003}}*.
> This is true even for simple types, e.g., {{text[]}}. A better behavior would
> be be create an Array column.
> - *Custom type columns are mapped to a String column.* This behavior is
> harder to understand as the schema of a custom type is fixed and therefore
> mappable to a Struct column. The automatic conversion to a string is also
> inconsistent when compared to json and array column handling.
> The exceptions are thrown by
> {{org.apache.spark.sql.jdbc.JDBCRDD$.org$apache$spark$sql$jdbc$JDBCRDD$$getCatalystType(JDBCRDD.scala:100)}}
> so this definitely looks like a Spark SQL and not a JDBC problem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
