[
https://issues.apache.org/jira/browse/SPARK-12048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15035486#comment-15035486
]
R. H. commented on SPARK-12048:
-------------------------------
I have added the missing line, built a distribution, and tested it
successfully: SQLite no longer throws the exception (and a test against
PostgreSQL was OK as well).
The PR is in https://github.com/rh99/spark
- What else can I do with the PR?
> JDBCRDD calls close() twice - SQLite then throws an exception
> -------------------------------------------------------------
>
> Key: SPARK-12048
> URL: https://issues.apache.org/jira/browse/SPARK-12048
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.5.1
> Reporter: R. H.
> Priority: Minor
> Labels: starter
>
> The following code works:
> {quote}
> val tableData = sqlContext.read.format("jdbc")
> .options(
> Map(
> "url" -> "jdbc:sqlite:/tmp/test.db",
> "dbtable" -> "testtable")).load()
> {quote}
> but an exception gets reported. From the log:
> {quote}
> 15/11/30 12:13:02 INFO jdbc.JDBCRDD: closed connection
> 15/11/30 12:13:02 WARN jdbc.JDBCRDD: Exception closing statement
> java.sql.SQLException: [SQLITE_ERROR] SQL error or missing database
> (Connection is closed) at org.sqlite.core.DB.newSQLException(DB.java:890) at
> org.sqlite.core.CoreStatement.internalClose(CoreStatement.java:109) at
> org.sqlite.jdbc3.JDBC3Statement.close(JDBC3Statement.java:35) at
> org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$$anon$1.org$apache$spark$sql$execution$datasources$jdbc$JDBCRDD$$anon$$close(JDBCRDD.scala:454)
> {quote}
> So Spark succeeded to close the JDBC connection, and then it fails to close
> the JDBC statement.
> Looking at the source, close() seems to be called twice:
> {quote}
> context.addTaskCompletionListener\{ context => close() \}
> {quote}
> and in
> {quote}
> def hasNext
> {quote}
> If you look at the close() method (around line 443)
> {quote}
> def close() {
> if (closed) return
> {quote}
> you can see that it checks the variable closed, but that value is never set
> to true.
> So a trivial fix should be to set "closed = true" at the end of close().
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
