[
https://issues.apache.org/jira/browse/SPARK-12048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
R. H. updated SPARK-12048:
--------------------------
Description:
The following code works:
val tableData = sqlContext.read.format("jdbc")
.options(
Map(
"url" -> "jdbc:sqlite:/tmp/test.db",
"dbtable" -> "testtable")).load()
but an exception gets reported. From the log:
15/11/30 12:13:02 INFO jdbc.JDBCRDD: closed connection
15/11/30 12:13:02 WARN jdbc.JDBCRDD: Exception closing statement
java.sql.SQLException: [SQLITE_ERROR] SQL error or missing database (Connection
is closed) at org.sqlite.core.DB.newSQLException(DB.java:890) at
org.sqlite.core.CoreStatement.internalClose(CoreStatement.java:109) at
org.sqlite.jdbc3.JDBC3Statement.close(JDBC3Statement.java:35) at
org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$$anon$1.org$apache$spark$sql$execution$datasources$jdbc$JDBCRDD$$anon$$close(JDBCRDD.scala:454)
So Spark succeeded to close the JDBC connection, and then it fails to close the
JDBC statement.
Looking at the source, close() seems to be called twice:
context.addTaskCompletionListener{ context => close() }
and
def hasNext
If you look at the close() method (around line 443)
def close() {
if (closed) return
you can see that it checks the variable closed, but that value is never set to
true.
So a trivial fix should be to set "closed = true" at the end of close().
was:
The following code works:
val tableData = sqlContext.read.format("jdbc")
.options(
Map(
"url" -> "jdbc:sqlite:/tmp/test.db",
"dbtable" -> "testtable")).load()
but an exception gets reported. From the log:
15/11/30 12:13:02 INFO jdbc.JDBCRDD: closed connection
15/11/30 12:13:02 WARN jdbc.JDBCRDD: Exception closing statement
java.sql.SQLException: [SQLITE_ERROR] SQL error or missing database (Connection
is closed) at org.sqlite.core.DB.newSQLException(DB.java:890) at
org.sqlite.core.CoreStatement.internalClose(CoreStatement.java:109) at
org.sqlite.jdbc3.JDBC3Statement.close(JDBC3Statement.java:35) at
org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$$anon$1.org$apache$spark$sql$execution$datasources$jdbc$JDBCRDD$$anon$$close(JDBCRDD.scala:454)
So Spark succeeded to close the JDBC connection, and then it fails to close the
JDBC statement.
Looking at the source, close() seems to be called twice.
If you look at the close() method (around line 443)
def close() {
if (closed) return
you can see that it checks the variable closed, but that value is never set to
true.
So a trivial fix should be to set "closed = true" at the end of close().
> JDBCRDD calls close() twice - SQLite then throws an exception
> -------------------------------------------------------------
>
> Key: SPARK-12048
> URL: https://issues.apache.org/jira/browse/SPARK-12048
> Project: Spark
> Issue Type: Bug
> Components: Input/Output
> Affects Versions: 1.5.1
> Reporter: R. H.
> Priority: Trivial
>
> The following code works:
> val tableData = sqlContext.read.format("jdbc")
> .options(
> Map(
> "url" -> "jdbc:sqlite:/tmp/test.db",
> "dbtable" -> "testtable")).load()
> but an exception gets reported. From the log:
> 15/11/30 12:13:02 INFO jdbc.JDBCRDD: closed connection
> 15/11/30 12:13:02 WARN jdbc.JDBCRDD: Exception closing statement
> java.sql.SQLException: [SQLITE_ERROR] SQL error or missing database
> (Connection is closed) at org.sqlite.core.DB.newSQLException(DB.java:890) at
> org.sqlite.core.CoreStatement.internalClose(CoreStatement.java:109) at
> org.sqlite.jdbc3.JDBC3Statement.close(JDBC3Statement.java:35) at
> org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$$anon$1.org$apache$spark$sql$execution$datasources$jdbc$JDBCRDD$$anon$$close(JDBCRDD.scala:454)
> So Spark succeeded to close the JDBC connection, and then it fails to close
> the JDBC statement.
> Looking at the source, close() seems to be called twice:
> context.addTaskCompletionListener{ context => close() }
> and
> def hasNext
> If you look at the close() method (around line 443)
> def close() {
> if (closed) return
> you can see that it checks the variable closed, but that value is never set
> to true.
> So a trivial fix should be to set "closed = true" at the end of close().
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
