This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 9d715ba49171 [SPARK-47938][SQL] MsSQLServer: Cannot find data type
BYTE error
9d715ba49171 is described below
commit 9d715ba491710969340d9e8a49a21d11f51ef7d3
Author: Kent Yao <y...@apache.org>
AuthorDate: Mon Apr 22 22:31:13 2024 -0700
[SPARK-47938][SQL] MsSQLServer: Cannot find data type BYTE error
### What changes were proposed in this pull request?
This PR uses SMALLINT (as TINYINT ranges [0, 255]) instead of BYTE to fix
the ByteType mapping for MsSQLServer JDBC
```java
[info] com.microsoft.sqlserver.jdbc.SQLServerException: Column,
parameter, or variable #1: Cannot find data type BYTE.
[info] at
com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:265)
[info] at
com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(SQLServerStatement.java:1662)
[info] at
com.microsoft.sqlserver.jdbc.SQLServerStatement.doExecuteStatement(SQLServerStatement.java:898)
[info] at
com.microsoft.sqlserver.jdbc.SQLServerStatement$StmtExecCmd.doExecute(SQLServerStatement.java:793)
[info] at
com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:7417)
[info] at
com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:3488)
[info] at
com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:262)
[info] at
com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLServerStatement.java:237)
[info] at
com.microsoft.sqlserver.jdbc.SQLServerStatement.executeUpdate(SQLServerStatement.java:733)
[info] at
org.apache.spark.sql.jdbc.JdbcDialect.createTable(JdbcDialects.scala:267)
```
### Why are the changes needed?
bugfix
### Does this PR introduce _any_ user-facing change?
no
### How was this patch tested?
new tests
### Was this patch authored or co-authored using generative AI tooling?
no
Closes #46164 from yaooqinn/SPARK-47938.
Lead-authored-by: Kent Yao <y...@apache.org>
Co-authored-by: Dongjoon Hyun <dongj...@apache.org>
Signed-off-by: Dongjoon Hyun <dh...@apple.com>
---
.../org/apache/spark/sql/jdbc/MsSqlServerIntegrationSuite.scala | 8 ++++++++
.../main/scala/org/apache/spark/sql/jdbc/MsSqlServerDialect.scala | 1 +
2 files changed, 9 insertions(+)
diff --git
a/connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/MsSqlServerIntegrationSuite.scala
b/connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/MsSqlServerIntegrationSuite.scala
index 8bceb9506e85..273e8c35dd07 100644
---
a/connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/MsSqlServerIntegrationSuite.scala
+++
b/connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/MsSqlServerIntegrationSuite.scala
@@ -437,4 +437,12 @@ class MsSqlServerIntegrationSuite extends
DockerJDBCIntegrationSuite {
.load()
assert(df.collect().toSet === expectedResult)
}
+
+ test("SPARK-47938: Fix 'Cannot find data type BYTE' in SQL Server") {
+ spark.sql("select cast(1 as byte) as c0")
+ .write
+ .jdbc(jdbcUrl, "test_byte", new Properties)
+ val df = spark.read.jdbc(jdbcUrl, "test_byte", new Properties)
+ checkAnswer(df, Row(1.toShort))
+ }
}
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/jdbc/MsSqlServerDialect.scala
b/sql/core/src/main/scala/org/apache/spark/sql/jdbc/MsSqlServerDialect.scala
index 862e99adc3b0..1d05c0d7c24e 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/jdbc/MsSqlServerDialect.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/jdbc/MsSqlServerDialect.scala
@@ -136,6 +136,7 @@ private case class MsSqlServerDialect() extends JdbcDialect
{
case BinaryType => Some(JdbcType("VARBINARY(MAX)",
java.sql.Types.VARBINARY))
case ShortType if !SQLConf.get.legacyMsSqlServerNumericMappingEnabled =>
Some(JdbcType("SMALLINT", java.sql.Types.SMALLINT))
+ case ByteType => Some(JdbcType("SMALLINT", java.sql.Types.TINYINT))
case _ => None
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
