[jira] [Updated] (SPARK-29644) ShortType is wrongly set as Int in JDBCUtils.scala
[ https://issues.apache.org/jira/browse/SPARK-29644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shiv Prashant Sood updated SPARK-29644: --- Description: @maropu pointed out this issue during [PR 25344|https://github.com/apache/spark/pull/25344] review discussion. In [JDBCUtils.scala|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala] line number 547 case ShortType => (stmt: PreparedStatement, row: Row, pos: Int) => stmt.setInt(pos + 1, row.getShort(pos)) I dont see any reproducible issue, but this is clearly a problem that must be fixed. was: @maropu pointed out this issue in [PR 25344|https://github.com/apache/spark/pull/25344] In [JDBCUtils.scala|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala] line number 547 case ShortType => (stmt: PreparedStatement, row: Row, pos: Int) => stmt.setInt(pos + 1, row.getShort(pos)) > ShortType is wrongly set as Int in JDBCUtils.scala > -- > > Key: SPARK-29644 > URL: https://issues.apache.org/jira/browse/SPARK-29644 > Project: Spark > Issue Type: New Feature > Components: SQL >Affects Versions: 2.4.4, 3.0.0 >Reporter: Shiv Prashant Sood >Priority: Minor > > @maropu pointed out this issue during [PR > 25344|https://github.com/apache/spark/pull/25344] review discussion. > In > [JDBCUtils.scala|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala] > line number 547 > case ShortType => > (stmt: PreparedStatement, row: Row, pos: Int) => > stmt.setInt(pos + 1, row.getShort(pos)) > I dont see any reproducible issue, but this is clearly a problem that must be > fixed. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-29644) ShortType is wrongly set as Int in JDBCUtils.scala
Shiv Prashant Sood created SPARK-29644: -- Summary: ShortType is wrongly set as Int in JDBCUtils.scala Key: SPARK-29644 URL: https://issues.apache.org/jira/browse/SPARK-29644 Project: Spark Issue Type: New Feature Components: SQL Affects Versions: 2.4.4, 3.0.0 Reporter: Shiv Prashant Sood @maropu pointed out this issue in [PR 25344|https://github.com/apache/spark/pull/25344] In [JDBCUtils.scala|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala] line number 547 case ShortType => (stmt: PreparedStatement, row: Row, pos: Int) => stmt.setInt(pos + 1, row.getShort(pos)) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28151) [JDBC Connector] ByteType is not correctly mapped for read/write of SQLServer tables
[
https://issues.apache.org/jira/browse/SPARK-28151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16899323#comment-16899323
]
Shiv Prashant Sood commented on SPARK-28151:
Created Work In Progress PR for this JIRA here.
https://github.com/apache/spark/pull/25344
As of now my fix is failing with a different error. Have created a PR to get
help in resolving the new error
> [JDBC Connector] ByteType is not correctly mapped for read/write of SQLServer
> tables
>
>
> Key: SPARK-28151
> URL: https://issues.apache.org/jira/browse/SPARK-28151
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.0.0, 2.4.3
>Reporter: Shiv Prashant Sood
>Priority: Minor
>
> ##ByteType issue
> Writing dataframe with column type BYTETYPE fails when using JDBC connector
> for SQL Server. Append and Read of tables also fail. The problem is due
> 1. (Write path) Incorrect mapping of BYTETYPE in getCommonJDBCType() in
> jdbcutils.scala where BYTETYPE gets mapped to BYTE text. It should be mapped
> to TINYINT
> {color:#cc7832}case {color}ByteType =>
> Option(JdbcType({color:#6a8759}"BYTE"{color}{color:#cc7832},
> {color}java.sql.Types.{color:#9876aa}TINYINT{color}))
> In getCatalystType() ( JDBC to Catalyst type mapping) TINYINT is mapped to
> INTEGER, while it should be mapped to BYTETYPE. Mapping to integer is ok from
> the point of view of upcasting, but will lead to 4 byte allocation rather
> than 1 byte for BYTETYPE.
> 2. (read path) Read path ends up calling makeGetter(dt: DataType, metadata:
> Metadata). The function sets the value in RDD row. The value is set per the
> data type. Here there is no mapping for BYTETYPE and thus results will result
> in an error when getCatalystType() is fixed.
> Note : These issues were found when reading/writing with SQLServer. Will be
> submitting a PR soon to fix these mappings in MSSQLServerDialect.
> Error seen when writing table
> (JDBC Write failed,com.microsoft.sqlserver.jdbc.SQLServerException: Column,
> parameter, or variable #2: *Cannot find data type BYTE*.)
> com.microsoft.sqlserver.jdbc.SQLServerException: Column, parameter, or
> variable #2: Cannot find data type BYTE.
> com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:254)
> com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(SQLServerStatement.java:1608)
> com.microsoft.sqlserver.jdbc.SQLServerStatement.doExecuteStatement(SQLServerStatement.java:859)
> ..
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24907) Migrate JDBC data source to DataSource API v2
[ https://issues.apache.org/jira/browse/SPARK-24907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16889552#comment-16889552 ] Shiv Prashant Sood commented on SPARK-24907: Pushed a PR with scaffolding for read/write paths and first draft implementation of write( append) path based on current APIs https://github.com/apache/spark/pull/25211 > Migrate JDBC data source to DataSource API v2 > - > > Key: SPARK-24907 > URL: https://issues.apache.org/jira/browse/SPARK-24907 > Project: Spark > Issue Type: New Feature > Components: SQL >Affects Versions: 2.3.0, 3.0.0 >Reporter: Teng Peng >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-24907) Migrate JDBC data source to DataSource API v2
[ https://issues.apache.org/jira/browse/SPARK-24907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shiv Prashant Sood updated SPARK-24907: --- Affects Version/s: 3.0.0 > Migrate JDBC data source to DataSource API v2 > - > > Key: SPARK-24907 > URL: https://issues.apache.org/jira/browse/SPARK-24907 > Project: Spark > Issue Type: New Feature > Components: SQL >Affects Versions: 2.3.0, 3.0.0 >Reporter: Teng Peng >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28152) [JDBC Connector] ShortType and FloatTypes are not mapped correctly for read/write of SQLServer Tables
[ https://issues.apache.org/jira/browse/SPARK-28152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16884460#comment-16884460 ] Shiv Prashant Sood commented on SPARK-28152: Pull request for fix created. https://github.com/apache/spark/pull/25146 > [JDBC Connector] ShortType and FloatTypes are not mapped correctly for > read/write of SQLServer Tables > - > > Key: SPARK-28152 > URL: https://issues.apache.org/jira/browse/SPARK-28152 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.0.0, 2.4.3 >Reporter: Shiv Prashant Sood >Priority: Minor > > ShortType and FloatTypes are not correctly mapped to right JDBC types when > using JDBC connector. This results in tables and spark data frame being > created with unintended types. The issue was observed when validating against > SQLServer. > Some example issue > * Write from df with column type results in a SQL table of with column type > as INTEGER as opposed to SMALLINT. Thus a larger table that expected. > * read results in a dataframe with type INTEGER as opposed to ShortType > FloatTypes have a issue with read path. In the write path Spark data type > 'FloatType' is correctly mapped to JDBC equivalent data type 'Real'. But in > the read path when JDBC data types need to be converted to Catalyst data > types ( getCatalystType) 'Real' gets incorrectly gets mapped to 'DoubleType' > rather than 'FloatType'. > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-28152) [JDBC Connector] ShortType and FloatTypes are not mapped correctly for read/write of SQLServer Tables
[ https://issues.apache.org/jira/browse/SPARK-28152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shiv Prashant Sood updated SPARK-28152: --- Summary: [JDBC Connector] ShortType and FloatTypes are not mapped correctly for read/write of SQLServer Tables (was: [JDBC Connector] ShortType and FloatTypes are not mapped correctly) > [JDBC Connector] ShortType and FloatTypes are not mapped correctly for > read/write of SQLServer Tables > - > > Key: SPARK-28152 > URL: https://issues.apache.org/jira/browse/SPARK-28152 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.0.0, 2.4.3 >Reporter: Shiv Prashant Sood >Priority: Minor > > ShortType and FloatTypes are not correctly mapped to right JDBC types when > using JDBC connector. This results in tables and spark data frame being > created with unintended types. The issue was observed when validating against > SQLServer. > Some example issue > * Write from df with column type results in a SQL table of with column type > as INTEGER as opposed to SMALLINT. Thus a larger table that expected. > * read results in a dataframe with type INTEGER as opposed to ShortType > FloatTypes have a issue with read path. In the write path Spark data type > 'FloatType' is correctly mapped to JDBC equivalent data type 'Real'. But in > the read path when JDBC data types need to be converted to Catalyst data > types ( getCatalystType) 'Real' gets incorrectly gets mapped to 'DoubleType' > rather than 'FloatType'. > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28151) [JDBC Connector] ByteType is not correctly mapped for read/write of SQLServer tables
[
https://issues.apache.org/jira/browse/SPARK-28151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16884454#comment-16884454
]
Shiv Prashant Sood commented on SPARK-28151:
Removed the FloatType and Short Type fix description as that would be handled
by separate PR (SPARK-28152)
> [JDBC Connector] ByteType is not correctly mapped for read/write of SQLServer
> tables
>
>
> Key: SPARK-28151
> URL: https://issues.apache.org/jira/browse/SPARK-28151
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.0.0, 2.4.3
>Reporter: Shiv Prashant Sood
>Priority: Minor
>
> ##ByteType issue
> Writing dataframe with column type BYTETYPE fails when using JDBC connector
> for SQL Server. Append and Read of tables also fail. The problem is due
> 1. (Write path) Incorrect mapping of BYTETYPE in getCommonJDBCType() in
> jdbcutils.scala where BYTETYPE gets mapped to BYTE text. It should be mapped
> to TINYINT
> {color:#cc7832}case {color}ByteType =>
> Option(JdbcType({color:#6a8759}"BYTE"{color}{color:#cc7832},
> {color}java.sql.Types.{color:#9876aa}TINYINT{color}))
> In getCatalystType() ( JDBC to Catalyst type mapping) TINYINT is mapped to
> INTEGER, while it should be mapped to BYTETYPE. Mapping to integer is ok from
> the point of view of upcasting, but will lead to 4 byte allocation rather
> than 1 byte for BYTETYPE.
> 2. (read path) Read path ends up calling makeGetter(dt: DataType, metadata:
> Metadata). The function sets the value in RDD row. The value is set per the
> data type. Here there is no mapping for BYTETYPE and thus results will result
> in an error when getCatalystType() is fixed.
> Note : These issues were found when reading/writing with SQLServer. Will be
> submitting a PR soon to fix these mappings in MSSQLServerDialect.
> Error seen when writing table
> (JDBC Write failed,com.microsoft.sqlserver.jdbc.SQLServerException: Column,
> parameter, or variable #2: *Cannot find data type BYTE*.)
> com.microsoft.sqlserver.jdbc.SQLServerException: Column, parameter, or
> variable #2: Cannot find data type BYTE.
> com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:254)
> com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(SQLServerStatement.java:1608)
> com.microsoft.sqlserver.jdbc.SQLServerStatement.doExecuteStatement(SQLServerStatement.java:859)
> ..
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-28151) [JDBC Connector] ByteType is not correctly mapped for read/write of SQLServer tables
[
https://issues.apache.org/jira/browse/SPARK-28151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shiv Prashant Sood updated SPARK-28151:
---
Description:
##ByteType issue
Writing dataframe with column type BYTETYPE fails when using JDBC connector for
SQL Server. Append and Read of tables also fail. The problem is due
1. (Write path) Incorrect mapping of BYTETYPE in getCommonJDBCType() in
jdbcutils.scala where BYTETYPE gets mapped to BYTE text. It should be mapped to
TINYINT
{color:#cc7832}case {color}ByteType =>
Option(JdbcType({color:#6a8759}"BYTE"{color}{color:#cc7832},
{color}java.sql.Types.{color:#9876aa}TINYINT{color}))
In getCatalystType() ( JDBC to Catalyst type mapping) TINYINT is mapped to
INTEGER, while it should be mapped to BYTETYPE. Mapping to integer is ok from
the point of view of upcasting, but will lead to 4 byte allocation rather than
1 byte for BYTETYPE.
2. (read path) Read path ends up calling makeGetter(dt: DataType, metadata:
Metadata). The function sets the value in RDD row. The value is set per the
data type. Here there is no mapping for BYTETYPE and thus results will result
in an error when getCatalystType() is fixed.
Note : These issues were found when reading/writing with SQLServer. Will be
submitting a PR soon to fix these mappings in MSSQLServerDialect.
Error seen when writing table
(JDBC Write failed,com.microsoft.sqlserver.jdbc.SQLServerException: Column,
parameter, or variable #2: *Cannot find data type BYTE*.)
com.microsoft.sqlserver.jdbc.SQLServerException: Column, parameter, or variable
#2: Cannot find data type BYTE.
com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:254)
com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(SQLServerStatement.java:1608)
com.microsoft.sqlserver.jdbc.SQLServerStatement.doExecuteStatement(SQLServerStatement.java:859)
..
was:
##ByteType issue
Writing dataframe with column type BYTETYPE fails when using JDBC connector for
SQL Server. Append and Read of tables also fail. The problem is due
1. (Write path) Incorrect mapping of BYTETYPE in getCommonJDBCType() in
jdbcutils.scala where BYTETYPE gets mapped to BYTE text. It should be mapped to
TINYINT
{color:#cc7832}case {color}ByteType =>
Option(JdbcType({color:#6a8759}"BYTE"{color}{color:#cc7832},
{color}java.sql.Types.{color:#9876aa}TINYINT{color}))
In getCatalystType() ( JDBC to Catalyst type mapping) TINYINT is mapped to
INTEGER, while it should be mapped to BYTETYPE. Mapping to integer is ok from
the point of view of upcasting, but will lead to 4 byte allocation rather than
1 byte for BYTETYPE.
2. (read path) Read path ends up calling makeGetter(dt: DataType, metadata:
Metadata). The function sets the value in RDD row. The value is set per the
data type. Here there is no mapping for BYTETYPE and thus results will result
in an error when getCatalystType() is fixed.
Note : These issues were found when reading/writing with SQLServer. Will be
submitting a PR soon to fix these mappings in MSSQLServerDialect.
Error seen when writing table
(JDBC Write failed,com.microsoft.sqlserver.jdbc.SQLServerException: Column,
parameter, or variable #2: *Cannot find data type BYTE*.)
com.microsoft.sqlserver.jdbc.SQLServerException: Column, parameter, or variable
#2: Cannot find data type BYTE.
com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:254)
com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(SQLServerStatement.java:1608)
com.microsoft.sqlserver.jdbc.SQLServerStatement.doExecuteStatement(SQLServerStatement.java:859)
..
##ShortType and FloatType issue
ShortType and FloatTypes are not correctly mapped to right JDBC types when
using JDBC connector. This results in tables and spark data frame being created
with unintended types.
Some example issue
Write from df with column type results in a SQL table of with column type
as INTEGER as opposed to SMALLINT. Thus a larger table that expected.
read results in a dataframe with type INTEGER as opposed to ShortType
FloatTypes have a issue with read path. In the write path Spark data type
'FloatType' is correctly mapped to JDBC equivalent data type 'Real'. But in the
read path when JDBC data types need to be converted to Catalyst data types (
getCatalystType) 'Real' gets incorrectly gets mapped to 'DoubleType' rather
than 'FloatType'.
> [JDBC Connector] ByteType is not correctly mapped for read/write of SQLServer
> tables
>
>
> Key: SPARK-28151
> URL: https://issues.apache.org/jira/browse/SPARK-28151
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.0.0, 2.4.3
>Reporter: Shiv Prashant Sood
>Priori
[jira] [Updated] (SPARK-28151) [JDBC Connector] ByteType is not correctly mapped for read/write of SQLServer tables
[
https://issues.apache.org/jira/browse/SPARK-28151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shiv Prashant Sood updated SPARK-28151:
---
Summary: [JDBC Connector] ByteType is not correctly mapped for read/write
of SQLServer tables (was: ByteType, ShortType and FloatTypes are not correctly
mapped for read/write of SQLServer tables)
> [JDBC Connector] ByteType is not correctly mapped for read/write of SQLServer
> tables
>
>
> Key: SPARK-28151
> URL: https://issues.apache.org/jira/browse/SPARK-28151
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.0.0, 2.4.3
>Reporter: Shiv Prashant Sood
>Priority: Minor
>
> ##ByteType issue
> Writing dataframe with column type BYTETYPE fails when using JDBC connector
> for SQL Server. Append and Read of tables also fail. The problem is due
> 1. (Write path) Incorrect mapping of BYTETYPE in getCommonJDBCType() in
> jdbcutils.scala where BYTETYPE gets mapped to BYTE text. It should be mapped
> to TINYINT
> {color:#cc7832}case {color}ByteType =>
> Option(JdbcType({color:#6a8759}"BYTE"{color}{color:#cc7832},
> {color}java.sql.Types.{color:#9876aa}TINYINT{color}))
> In getCatalystType() ( JDBC to Catalyst type mapping) TINYINT is mapped to
> INTEGER, while it should be mapped to BYTETYPE. Mapping to integer is ok from
> the point of view of upcasting, but will lead to 4 byte allocation rather
> than 1 byte for BYTETYPE.
> 2. (read path) Read path ends up calling makeGetter(dt: DataType, metadata:
> Metadata). The function sets the value in RDD row. The value is set per the
> data type. Here there is no mapping for BYTETYPE and thus results will result
> in an error when getCatalystType() is fixed.
> Note : These issues were found when reading/writing with SQLServer. Will be
> submitting a PR soon to fix these mappings in MSSQLServerDialect.
> Error seen when writing table
> (JDBC Write failed,com.microsoft.sqlserver.jdbc.SQLServerException: Column,
> parameter, or variable #2: *Cannot find data type BYTE*.)
> com.microsoft.sqlserver.jdbc.SQLServerException: Column, parameter, or
> variable #2: Cannot find data type BYTE.
> com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:254)
> com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(SQLServerStatement.java:1608)
> com.microsoft.sqlserver.jdbc.SQLServerStatement.doExecuteStatement(SQLServerStatement.java:859)
> ..
> ##ShortType and FloatType issue
> ShortType and FloatTypes are not correctly mapped to right JDBC types when
> using JDBC connector. This results in tables and spark data frame being
> created with unintended types.
> Some example issue
> Write from df with column type results in a SQL table of with column type
> as INTEGER as opposed to SMALLINT. Thus a larger table that expected.
> read results in a dataframe with type INTEGER as opposed to ShortType
> FloatTypes have a issue with read path. In the write path Spark data type
> 'FloatType' is correctly mapped to JDBC equivalent data type 'Real'. But in
> the read path when JDBC data types need to be converted to Catalyst data
> types ( getCatalystType) 'Real' gets incorrectly gets mapped to 'DoubleType'
> rather than 'FloatType'.
>
>
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-28152) [JDBC Connector] ShortType and FloatTypes are not mapped correctly
[ https://issues.apache.org/jira/browse/SPARK-28152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shiv Prashant Sood updated SPARK-28152: --- Description: ShortType and FloatTypes are not correctly mapped to right JDBC types when using JDBC connector. This results in tables and spark data frame being created with unintended types. The issue was observed when validating against SQLServer. Some example issue * Write from df with column type results in a SQL table of with column type as INTEGER as opposed to SMALLINT. Thus a larger table that expected. * read results in a dataframe with type INTEGER as opposed to ShortType FloatTypes have a issue with read path. In the write path Spark data type 'FloatType' is correctly mapped to JDBC equivalent data type 'Real'. But in the read path when JDBC data types need to be converted to Catalyst data types ( getCatalystType) 'Real' gets incorrectly gets mapped to 'DoubleType' rather than 'FloatType'. was: ShortType and FloatTypes are not correctly mapped to right JDBC types when using JDBC connector. This results in tables and spark data frame being created with unintended types. Some example issue * Write from df with column type results in a SQL table of with column type as INTEGER as opposed to SMALLINT. Thus a larger table that expected. * read results in a dataframe with type INTEGER as opposed to ShortType FloatTypes have a issue with read path. In the write path Spark data type 'FloatType' is correctly mapped to JDBC equivalent data type 'Real'. But in the read path when JDBC data types need to be converted to Catalyst data types ( getCatalystType) 'Real' gets incorrectly gets mapped to 'DoubleType' rather than 'FloatType'. > [JDBC Connector] ShortType and FloatTypes are not mapped correctly > -- > > Key: SPARK-28152 > URL: https://issues.apache.org/jira/browse/SPARK-28152 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.0.0, 2.4.3 >Reporter: Shiv Prashant Sood >Priority: Minor > > ShortType and FloatTypes are not correctly mapped to right JDBC types when > using JDBC connector. This results in tables and spark data frame being > created with unintended types. The issue was observed when validating against > SQLServer. > Some example issue > * Write from df with column type results in a SQL table of with column type > as INTEGER as opposed to SMALLINT. Thus a larger table that expected. > * read results in a dataframe with type INTEGER as opposed to ShortType > FloatTypes have a issue with read path. In the write path Spark data type > 'FloatType' is correctly mapped to JDBC equivalent data type 'Real'. But in > the read path when JDBC data types need to be converted to Catalyst data > types ( getCatalystType) 'Real' gets incorrectly gets mapped to 'DoubleType' > rather than 'FloatType'. > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-28152) [JDBC Connector] ShortType and FloatTypes are not mapped correctly
[ https://issues.apache.org/jira/browse/SPARK-28152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shiv Prashant Sood updated SPARK-28152: --- Summary: [JDBC Connector] ShortType and FloatTypes are not mapped correctly (was: [JDBC Connector] ShortType and FloatTypes are not correctly mapped correctly) > [JDBC Connector] ShortType and FloatTypes are not mapped correctly > -- > > Key: SPARK-28152 > URL: https://issues.apache.org/jira/browse/SPARK-28152 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.0.0, 2.4.3 >Reporter: Shiv Prashant Sood >Priority: Minor > > ShortType and FloatTypes are not correctly mapped to right JDBC types when > using JDBC connector. This results in tables and spark data frame being > created with unintended types. > Some example issue > * Write from df with column type results in a SQL table of with column type > as INTEGER as opposed to SMALLINT. Thus a larger table that expected. > * read results in a dataframe with type INTEGER as opposed to ShortType > FloatTypes have a issue with read path. In the write path Spark data type > 'FloatType' is correctly mapped to JDBC equivalent data type 'Real'. But in > the read path when JDBC data types need to be converted to Catalyst data > types ( getCatalystType) 'Real' gets incorrectly gets mapped to 'DoubleType' > rather than 'FloatType'. > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-28152) [JDBC Connector] ShortType and FloatTypes are not correctly mapped correctly
[ https://issues.apache.org/jira/browse/SPARK-28152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shiv Prashant Sood updated SPARK-28152: --- Summary: [JDBC Connector] ShortType and FloatTypes are not correctly mapped correctly (was: ShortType and FloatTypes are not correctly mapped correctly) > [JDBC Connector] ShortType and FloatTypes are not correctly mapped correctly > > > Key: SPARK-28152 > URL: https://issues.apache.org/jira/browse/SPARK-28152 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.0.0, 2.4.3 >Reporter: Shiv Prashant Sood >Priority: Minor > > ShortType and FloatTypes are not correctly mapped to right JDBC types when > using JDBC connector. This results in tables and spark data frame being > created with unintended types. > Some example issue > * Write from df with column type results in a SQL table of with column type > as INTEGER as opposed to SMALLINT. Thus a larger table that expected. > * read results in a dataframe with type INTEGER as opposed to ShortType > FloatTypes have a issue with read path. In the write path Spark data type > 'FloatType' is correctly mapped to JDBC equivalent data type 'Real'. But in > the read path when JDBC data types need to be converted to Catalyst data > types ( getCatalystType) 'Real' gets incorrectly gets mapped to 'DoubleType' > rather than 'FloatType'. > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-28152) ShortType and FloatTypes are not correctly mapped correctly
[ https://issues.apache.org/jira/browse/SPARK-28152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shiv Prashant Sood updated SPARK-28152: --- Summary: ShortType and FloatTypes are not correctly mapped correctly (was: ShortType and FloatTypes are not correctly mapped to right JDBC types when using JDBC connector) > ShortType and FloatTypes are not correctly mapped correctly > --- > > Key: SPARK-28152 > URL: https://issues.apache.org/jira/browse/SPARK-28152 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.0.0, 2.4.3 >Reporter: Shiv Prashant Sood >Priority: Minor > > ShortType and FloatTypes are not correctly mapped to right JDBC types when > using JDBC connector. This results in tables and spark data frame being > created with unintended types. > Some example issue > * Write from df with column type results in a SQL table of with column type > as INTEGER as opposed to SMALLINT. Thus a larger table that expected. > * read results in a dataframe with type INTEGER as opposed to ShortType > FloatTypes have a issue with read path. In the write path Spark data type > 'FloatType' is correctly mapped to JDBC equivalent data type 'Real'. But in > the read path when JDBC data types need to be converted to Catalyst data > types ( getCatalystType) 'Real' gets incorrectly gets mapped to 'DoubleType' > rather than 'FloatType'. > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Reopened] (SPARK-28152) ShortType and FloatTypes are not correctly mapped to right JDBC types when using JDBC connector
[ https://issues.apache.org/jira/browse/SPARK-28152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shiv Prashant Sood reopened SPARK-28152: Reopening this issue to submit this change as a separate PR for clarity. Earlier this change for made part of the ByteType PR ( 28151) > ShortType and FloatTypes are not correctly mapped to right JDBC types when > using JDBC connector > --- > > Key: SPARK-28152 > URL: https://issues.apache.org/jira/browse/SPARK-28152 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.0.0, 2.4.3 >Reporter: Shiv Prashant Sood >Priority: Minor > > ShortType and FloatTypes are not correctly mapped to right JDBC types when > using JDBC connector. This results in tables and spark data frame being > created with unintended types. > Some example issue > * Write from df with column type results in a SQL table of with column type > as INTEGER as opposed to SMALLINT. Thus a larger table that expected. > * read results in a dataframe with type INTEGER as opposed to ShortType > FloatTypes have a issue with read path. In the write path Spark data type > 'FloatType' is correctly mapped to JDBC equivalent data type 'Real'. But in > the read path when JDBC data types need to be converted to Catalyst data > types ( getCatalystType) 'Real' gets incorrectly gets mapped to 'DoubleType' > rather than 'FloatType'. > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-28380) DataSourceV2 API based JDBC connector
Shiv Prashant Sood created SPARK-28380: -- Summary: DataSourceV2 API based JDBC connector Key: SPARK-28380 URL: https://issues.apache.org/jira/browse/SPARK-28380 Project: Spark Issue Type: New Feature Components: SQL Affects Versions: 3.0.0 Reporter: Shiv Prashant Sood JIRA for DataSourceV2 API based JDBC connector. Goals : - Generic connector based on JDBC that supports all databases (min bar is support for all V1 data bases). - Reference implementation and Interface for any specialized JDBC connectors. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28152) ShortType and FloatTypes are not correctly mapped to right JDBC types when using JDBC connector
[ https://issues.apache.org/jira/browse/SPARK-28152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16872739#comment-16872739 ] Shiv Prashant Sood commented on SPARK-28152: Resolved as part of https://issues.apache.org/jira/browse/SPARK-28151 > ShortType and FloatTypes are not correctly mapped to right JDBC types when > using JDBC connector > --- > > Key: SPARK-28152 > URL: https://issues.apache.org/jira/browse/SPARK-28152 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.0.0, 2.4.3 >Reporter: Shiv Prashant Sood >Priority: Minor > > ShortType and FloatTypes are not correctly mapped to right JDBC types when > using JDBC connector. This results in tables and spark data frame being > created with unintended types. > Some example issue > * Write from df with column type results in a SQL table of with column type > as INTEGER as opposed to SMALLINT. Thus a larger table that expected. > * read results in a dataframe with type INTEGER as opposed to ShortType > FloatTypes have a issue with read path. In the write path Spark data type > 'FloatType' is correctly mapped to JDBC equivalent data type 'Real'. But in > the read path when JDBC data types need to be converted to Catalyst data > types ( getCatalystType) 'Real' gets incorrectly gets mapped to 'DoubleType' > rather than 'FloatType'. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-28151) ByteType, ShortType and FloatTypes are not correctly mapped for read/write of SQLServer tables
[
https://issues.apache.org/jira/browse/SPARK-28151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shiv Prashant Sood updated SPARK-28151:
---
Description:
##ByteType issue
Writing dataframe with column type BYTETYPE fails when using JDBC connector for
SQL Server. Append and Read of tables also fail. The problem is due
1. (Write path) Incorrect mapping of BYTETYPE in getCommonJDBCType() in
jdbcutils.scala where BYTETYPE gets mapped to BYTE text. It should be mapped to
TINYINT
{color:#cc7832}case {color}ByteType =>
Option(JdbcType({color:#6a8759}"BYTE"{color}{color:#cc7832},
{color}java.sql.Types.{color:#9876aa}TINYINT{color}))
In getCatalystType() ( JDBC to Catalyst type mapping) TINYINT is mapped to
INTEGER, while it should be mapped to BYTETYPE. Mapping to integer is ok from
the point of view of upcasting, but will lead to 4 byte allocation rather than
1 byte for BYTETYPE.
2. (read path) Read path ends up calling makeGetter(dt: DataType, metadata:
Metadata). The function sets the value in RDD row. The value is set per the
data type. Here there is no mapping for BYTETYPE and thus results will result
in an error when getCatalystType() is fixed.
Note : These issues were found when reading/writing with SQLServer. Will be
submitting a PR soon to fix these mappings in MSSQLServerDialect.
Error seen when writing table
(JDBC Write failed,com.microsoft.sqlserver.jdbc.SQLServerException: Column,
parameter, or variable #2: *Cannot find data type BYTE*.)
com.microsoft.sqlserver.jdbc.SQLServerException: Column, parameter, or variable
#2: Cannot find data type BYTE.
com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:254)
com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(SQLServerStatement.java:1608)
com.microsoft.sqlserver.jdbc.SQLServerStatement.doExecuteStatement(SQLServerStatement.java:859)
..
##ShortType and FloatType issue
ShortType and FloatTypes are not correctly mapped to right JDBC types when
using JDBC connector. This results in tables and spark data frame being created
with unintended types.
Some example issue
Write from df with column type results in a SQL table of with column type
as INTEGER as opposed to SMALLINT. Thus a larger table that expected.
read results in a dataframe with type INTEGER as opposed to ShortType
FloatTypes have a issue with read path. In the write path Spark data type
'FloatType' is correctly mapped to JDBC equivalent data type 'Real'. But in the
read path when JDBC data types need to be converted to Catalyst data types (
getCatalystType) 'Real' gets incorrectly gets mapped to 'DoubleType' rather
than 'FloatType'.
was:
Writing dataframe with column type BYTETYPE fails when using JDBC connector for
SQL Server. Append and Read of tables also fail. The problem is due
1. (Write path) Incorrect mapping of BYTETYPE in getCommonJDBCType() in
jdbcutils.scala where BYTETYPE gets mapped to BYTE text. It should be mapped to
TINYINT
{color:#cc7832}case {color}ByteType =>
Option(JdbcType({color:#6a8759}"BYTE"{color}{color:#cc7832},
{color}java.sql.Types.{color:#9876aa}TINYINT{color}))
In getCatalystType() ( JDBC to Catalyst type mapping) TINYINT is mapped to
INTEGER, while it should be mapped to BYTETYPE. Mapping to integer is ok from
the point of view of upcasting, but will lead to 4 byte allocation rather than
1 byte for BYTETYPE.
2. (read path) Read path ends up calling makeGetter(dt: DataType, metadata:
Metadata). The function sets the value in RDD row. The value is set per the
data type. Here there is no mapping for BYTETYPE and thus results will result
in an error when getCatalystType() is fixed.
Note : These issues were found when reading/writing with SQLServer. Will be
submitting a PR soon to fix these mappings in MSSQLServerDialect.
Error seen when writing table
(JDBC Write failed,com.microsoft.sqlserver.jdbc.SQLServerException: Column,
parameter, or variable #2: *Cannot find data type BYTE*.)
com.microsoft.sqlserver.jdbc.SQLServerException: Column, parameter, or variable
#2: Cannot find data type BYTE.
com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:254)
com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(SQLServerStatement.java:1608)
com.microsoft.sqlserver.jdbc.SQLServerStatement.doExecuteStatement(SQLServerStatement.java:859)
..
> ByteType, ShortType and FloatTypes are not correctly mapped for read/write of
> SQLServer tables
> --
>
> Key: SPARK-28151
> URL: https://issues.apache.org/jira/browse/SPARK-28151
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.0.0, 2.4.3
>Reporter: Shiv Prashant Sood
>
[jira] [Updated] (SPARK-28151) ByteType, ShortType and FloatTypes are not correctly mapped for read/write of SQLServer tables
[
https://issues.apache.org/jira/browse/SPARK-28151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shiv Prashant Sood updated SPARK-28151:
---
Summary: ByteType, ShortType and FloatTypes are not correctly mapped for
read/write of SQLServer tables (was: ByteType is not correctly mapped for
read/write of SQLServer tables)
> ByteType, ShortType and FloatTypes are not correctly mapped for read/write of
> SQLServer tables
> --
>
> Key: SPARK-28151
> URL: https://issues.apache.org/jira/browse/SPARK-28151
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.0.0, 2.4.3
>Reporter: Shiv Prashant Sood
>Priority: Minor
>
> Writing dataframe with column type BYTETYPE fails when using JDBC connector
> for SQL Server. Append and Read of tables also fail. The problem is due
> 1. (Write path) Incorrect mapping of BYTETYPE in getCommonJDBCType() in
> jdbcutils.scala where BYTETYPE gets mapped to BYTE text. It should be mapped
> to TINYINT
> {color:#cc7832}case {color}ByteType =>
> Option(JdbcType({color:#6a8759}"BYTE"{color}{color:#cc7832},
> {color}java.sql.Types.{color:#9876aa}TINYINT{color}))
> In getCatalystType() ( JDBC to Catalyst type mapping) TINYINT is mapped to
> INTEGER, while it should be mapped to BYTETYPE. Mapping to integer is ok from
> the point of view of upcasting, but will lead to 4 byte allocation rather
> than 1 byte for BYTETYPE.
> 2. (read path) Read path ends up calling makeGetter(dt: DataType, metadata:
> Metadata). The function sets the value in RDD row. The value is set per the
> data type. Here there is no mapping for BYTETYPE and thus results will result
> in an error when getCatalystType() is fixed.
> Note : These issues were found when reading/writing with SQLServer. Will be
> submitting a PR soon to fix these mappings in MSSQLServerDialect.
> Error seen when writing table
> (JDBC Write failed,com.microsoft.sqlserver.jdbc.SQLServerException: Column,
> parameter, or variable #2: *Cannot find data type BYTE*.)
> com.microsoft.sqlserver.jdbc.SQLServerException: Column, parameter, or
> variable #2: Cannot find data type BYTE.
> com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:254)
> com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(SQLServerStatement.java:1608)
> com.microsoft.sqlserver.jdbc.SQLServerStatement.doExecuteStatement(SQLServerStatement.java:859)
> ..
>
>
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-28152) ShortType and FloatTypes are not correctly mapped to right JDBC types when using JDBC connector
[ https://issues.apache.org/jira/browse/SPARK-28152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shiv Prashant Sood updated SPARK-28152: --- Description: ShortType and FloatTypes are not correctly mapped to right JDBC types when using JDBC connector. This results in tables and spark data frame being created with unintended types. Some example issue * Write from df with column type results in a SQL table of with column type as INTEGER as opposed to SMALLINT. Thus a larger table that expected. * read results in a dataframe with type INTEGER as opposed to ShortType FloatTypes have a issue with read path. In the write path Spark data type 'FloatType' is correctly mapped to JDBC equivalent data type 'Real'. But in the read path when JDBC data types need to be converted to Catalyst data types ( getCatalystType) 'Real' gets incorrectly gets mapped to 'DoubleType' rather than 'FloatType'. was: ShortType and FloatTypes are not correctly mapped to right JDBC types when using JDBC connector. This results in tables or spark data frame being created with unintended types. Some example issue * Write from df with column type results in a SQL table of with column type as INTEGER as opposed to SMALLINT. Thus a larger table that expected. * read results in a dataframe with type INTEGER as opposed to ShortType FloatTypes have a issue with read path. In the write path Spark data type 'FloatType' is correctly mapped to JDBC equivalent data type 'Real'. But in the read path when JDBC data types need to be converted to Catalyst data types ( getCatalystType) 'Real' gets incorrectly gets mapped to 'DoubleType' rather than 'FloatType'. > ShortType and FloatTypes are not correctly mapped to right JDBC types when > using JDBC connector > --- > > Key: SPARK-28152 > URL: https://issues.apache.org/jira/browse/SPARK-28152 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.0.0, 2.4.3 >Reporter: Shiv Prashant Sood >Priority: Minor > > ShortType and FloatTypes are not correctly mapped to right JDBC types when > using JDBC connector. This results in tables and spark data frame being > created with unintended types. > Some example issue > * Write from df with column type results in a SQL table of with column type > as INTEGER as opposed to SMALLINT. Thus a larger table that expected. > * read results in a dataframe with type INTEGER as opposed to ShortType > FloatTypes have a issue with read path. In the write path Spark data type > 'FloatType' is correctly mapped to JDBC equivalent data type 'Real'. But in > the read path when JDBC data types need to be converted to Catalyst data > types ( getCatalystType) 'Real' gets incorrectly gets mapped to 'DoubleType' > rather than 'FloatType'. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28151) ByteType is not correctly mapped for read/write of SQLServer tables
[
https://issues.apache.org/jira/browse/SPARK-28151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16872577#comment-16872577
]
Shiv Prashant Sood commented on SPARK-28151:
Fixed by [https://github.com/apache/spark/pull/24969]
> ByteType is not correctly mapped for read/write of SQLServer tables
> ---
>
> Key: SPARK-28151
> URL: https://issues.apache.org/jira/browse/SPARK-28151
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.0.0, 2.4.3
>Reporter: Shiv Prashant Sood
>Priority: Minor
>
> Writing dataframe with column type BYTETYPE fails when using JDBC connector
> for SQL Server. Append and Read of tables also fail. The problem is due
> 1. (Write path) Incorrect mapping of BYTETYPE in getCommonJDBCType() in
> jdbcutils.scala where BYTETYPE gets mapped to BYTE text. It should be mapped
> to TINYINT
> {color:#cc7832}case {color}ByteType =>
> Option(JdbcType({color:#6a8759}"BYTE"{color}{color:#cc7832},
> {color}java.sql.Types.{color:#9876aa}TINYINT{color}))
> In getCatalystType() ( JDBC to Catalyst type mapping) TINYINT is mapped to
> INTEGER, while it should be mapped to BYTETYPE. Mapping to integer is ok from
> the point of view of upcasting, but will lead to 4 byte allocation rather
> than 1 byte for BYTETYPE.
> 2. (read path) Read path ends up calling makeGetter(dt: DataType, metadata:
> Metadata). The function sets the value in RDD row. The value is set per the
> data type. Here there is no mapping for BYTETYPE and thus results will result
> in an error when getCatalystType() is fixed.
> Note : These issues were found when reading/writing with SQLServer. Will be
> submitting a PR soon to fix these mappings in MSSQLServerDialect.
> Error seen when writing table
> (JDBC Write failed,com.microsoft.sqlserver.jdbc.SQLServerException: Column,
> parameter, or variable #2: *Cannot find data type BYTE*.)
> com.microsoft.sqlserver.jdbc.SQLServerException: Column, parameter, or
> variable #2: Cannot find data type BYTE.
> com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:254)
> com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(SQLServerStatement.java:1608)
> com.microsoft.sqlserver.jdbc.SQLServerStatement.doExecuteStatement(SQLServerStatement.java:859)
> ..
>
>
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-28152) ShortType and FloatTypes are not correctly mapped to right JDBC types when using JDBC connector
Shiv Prashant Sood created SPARK-28152: -- Summary: ShortType and FloatTypes are not correctly mapped to right JDBC types when using JDBC connector Key: SPARK-28152 URL: https://issues.apache.org/jira/browse/SPARK-28152 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 2.4.3, 3.0.0 Reporter: Shiv Prashant Sood ShortType and FloatTypes are not correctly mapped to right JDBC types when using JDBC connector. This results in tables or spark data frame being created with unintended types. Some example issue * Write from df with column type results in a SQL table of with column type as INTEGER as opposed to SMALLINT. Thus a larger table that expected. * read results in a dataframe with type INTEGER as opposed to ShortType FloatTypes have a issue with read path. In the write path Spark data type 'FloatType' is correctly mapped to JDBC equivalent data type 'Real'. But in the read path when JDBC data types need to be converted to Catalyst data types ( getCatalystType) 'Real' gets incorrectly gets mapped to 'DoubleType' rather than 'FloatType'. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-28151) ByteType is not correctly mapped for read/write of SQLServer tables
Shiv Prashant Sood created SPARK-28151:
--
Summary: ByteType is not correctly mapped for read/write of
SQLServer tables
Key: SPARK-28151
URL: https://issues.apache.org/jira/browse/SPARK-28151
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.4.3, 3.0.0
Reporter: Shiv Prashant Sood
Writing dataframe with column type BYTETYPE fails when using JDBC connector for
SQL Server. Append and Read of tables also fail. The problem is due
1. (Write path) Incorrect mapping of BYTETYPE in getCommonJDBCType() in
jdbcutils.scala where BYTETYPE gets mapped to BYTE text. It should be mapped to
TINYINT
{color:#cc7832}case {color}ByteType =>
Option(JdbcType({color:#6a8759}"BYTE"{color}{color:#cc7832},
{color}java.sql.Types.{color:#9876aa}TINYINT{color}))
In getCatalystType() ( JDBC to Catalyst type mapping) TINYINT is mapped to
INTEGER, while it should be mapped to BYTETYPE. Mapping to integer is ok from
the point of view of upcasting, but will lead to 4 byte allocation rather than
1 byte for BYTETYPE.
2. (read path) Read path ends up calling makeGetter(dt: DataType, metadata:
Metadata). The function sets the value in RDD row. The value is set per the
data type. Here there is no mapping for BYTETYPE and thus results will result
in an error when getCatalystType() is fixed.
Note : These issues were found when reading/writing with SQLServer. Will be
submitting a PR soon to fix these mappings in MSSQLServerDialect.
Error seen when writing table
(JDBC Write failed,com.microsoft.sqlserver.jdbc.SQLServerException: Column,
parameter, or variable #2: *Cannot find data type BYTE*.)
com.microsoft.sqlserver.jdbc.SQLServerException: Column, parameter, or variable
#2: Cannot find data type BYTE.
com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:254)
com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(SQLServerStatement.java:1608)
com.microsoft.sqlserver.jdbc.SQLServerStatement.doExecuteStatement(SQLServerStatement.java:859)
..
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
