R. H. created SPARK-12048:
-----------------------------
Summary: JDBCRDD calls close() twice - SQLite then throws an
exception
Key: SPARK-12048
URL: https://issues.apache.org/jira/browse/SPARK-12048
Project: Spark
Issue Type: Bug
Components: Input/Output
Affects Versions: 1.5.1
Reporter: R. H.
Priority: Trivial
The following code works:
val tableData = sqlContext.read.format("jdbc")
.options(
Map(
"url" -> "jdbc:sqlite:/tmp/test.db",
"dbtable" -> "testtable")).load()
but an exception gets reported. From the log:
15/11/30 12:13:02 INFO jdbc.JDBCRDD: closed connection
15/11/30 12:13:02 WARN jdbc.JDBCRDD: Exception closing statement
java.sql.SQLException: [SQLITE_ERROR] SQL error or missing database (Connection
is closed) at org.sqlite.core.DB.newSQLException(DB.java:890) at
org.sqlite.core.CoreStatement.internalClose(CoreStatement.java:109) at
org.sqlite.jdbc3.JDBC3Statement.close(JDBC3Statement.java:35) at
org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$$anon$1.org$apache$spark$sql$execution$datasources$jdbc$JDBCRDD$$anon$$close(JDBCRDD.scala:454)
So Spark succeeded to close the JDBC connection, and then it fails to close the
JDBC statement.
Looking at the source, close() seems to be called twice.
If you look at the close() method (around line 443)
def close() {
if (closed) return
you can see that it checks the variable closed, but that value is never set to
true.
So a trivial fix should be to set "closed = true" at the end of close().
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
