Peng Cheng created SPARK-10625:
----------------------------------
Summary: Spark SQL JDBC read/write is unable to handle JDBC
Drivers that adds unserializable objects into connection properties
Key: SPARK-10625
URL: https://issues.apache.org/jira/browse/SPARK-10625
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 1.5.0, 1.4.1
Environment: Ubuntu 14.04
Reporter: Peng Cheng
Some JDBC drivers (e.g. SAP HANA) tries to optimize connection pooling by
adding new objects into the connection properties, which is then reused by
Spark to be deployed to workers. When some of these new objects are unable to
be serializable it will trigger an org.apache.spark.SparkException: Task not
serializable. The following test code snippet demonstrate this problem by using
a modified H2 driver:
test("INSERT to JDBC Datasource with UnserializableH2Driver") {
object UnserializableH2Driver extends org.h2.Driver {
override def connect(url: String, info: Properties): Connection = {
val result = super.connect(url, info)
info.put("unserializableDriver", this)
result
}
override def getParentLogger: Logger = ???
}
import scala.collection.JavaConversions._
val oldDrivers =
DriverManager.getDrivers.filter(_.acceptsURL("jdbc:h2:")).toSeq
oldDrivers.foreach{
DriverManager.deregisterDriver
}
DriverManager.registerDriver(UnserializableH2Driver)
sql("INSERT INTO TABLE PEOPLE1 SELECT * FROM PEOPLE")
assert(2 === sqlContext.read.jdbc(url1, "TEST.PEOPLE1", properties).count)
assert(2 === sqlContext.read.jdbc(url1, "TEST.PEOPLE1",
properties).collect()(0).length)
DriverManager.deregisterDriver(UnserializableH2Driver)
oldDrivers.foreach{
DriverManager.registerDriver
}
}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
