[jira] [Commented] (SPARK-4908) Spark SQL built for Hive 13 fails under concurrent metadata queries
[
https://issues.apache.org/jira/browse/SPARK-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14273271#comment-14273271
]
Apache Spark commented on SPARK-4908:
-
User 'baishuo' has created a pull request for this issue:
https://github.com/apache/spark/pull/4001
Spark SQL built for Hive 13 fails under concurrent metadata queries
---
Key: SPARK-4908
URL: https://issues.apache.org/jira/browse/SPARK-4908
Project: Spark
Issue Type: Bug
Components: SQL
Reporter: David Ross
Assignee: Cheng Lian
Priority: Blocker
Fix For: 1.3.0, 1.2.1
We are trunk: {{1.3.0-SNAPSHOT}}, as of this commit:
https://github.com/apache/spark/commit/3d0c37b8118f6057a663f959321a79b8061132b6
We are using Spark built for Hive 13, using this option:
{{-Phive-0.13.1}}
In single-threaded mode, normal operations look fine. However, under
concurrency, with at least 2 concurrent connections, metadata queries fail.
For example, {{USE some_db}}, {{SHOW TABLES}}, and the implicit {{USE}}
statement when you pass a default schema in the JDBC URL, all fail.
{{SELECT}} queries like {{SELECT * FROM some_table}} do not have this issue.
Here is some example code:
{code}
object main extends App {
import java.sql._
import scala.concurrent._
import scala.concurrent.duration._
import scala.concurrent.ExecutionContext.Implicits.global
Class.forName(org.apache.hive.jdbc.HiveDriver)
val host = localhost // update this
val url = sjdbc:hive2://${host}:10511/some_db // update this
val future = Future.traverse(1 to 3) { i =
Future {
println(Starting: + i)
try {
val conn = DriverManager.getConnection(url)
} catch {
case e: Throwable = e.printStackTrace()
println(Failed: + i)
}
println(Finishing: + i)
}
}
Await.result(future, 2.minutes)
println(done!)
}
{code}
Here is the output:
{code}
Starting: 1
Starting: 3
Starting: 2
java.sql.SQLException:
org.apache.spark.sql.execution.QueryExecutionException: FAILED: Operation
cancelled
at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:121)
at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:109)
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:231)
at
org.apache.hive.jdbc.HiveConnection.configureConnection(HiveConnection.java:451)
at org.apache.hive.jdbc.HiveConnection.init(HiveConnection.java:195)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:270)
at
com.atscale.engine.connection.pool.main$$anonfun$30$$anonfun$apply$2.apply$mcV$sp(ConnectionManager.scala:896)
at
com.atscale.engine.connection.pool.main$$anonfun$30$$anonfun$apply$2.apply(ConnectionManager.scala:893)
at
com.atscale.engine.connection.pool.main$$anonfun$30$$anonfun$apply$2.apply(ConnectionManager.scala:893)
at
scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at
scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at
scala.concurrent.impl.ExecutionContextImpl$AdaptedForkJoinTask.exec(ExecutionContextImpl.scala:121)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Failed: 3
Finishing: 3
java.sql.SQLException:
org.apache.spark.sql.execution.QueryExecutionException: FAILED: Operation
cancelled
at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:121)
at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:109)
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:231)
at
org.apache.hive.jdbc.HiveConnection.configureConnection(HiveConnection.java:451)
at org.apache.hive.jdbc.HiveConnection.init(HiveConnection.java:195)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:270)
at
com.atscale.engine.connection.pool.main$$anonfun$30$$anonfun$apply$2.apply$mcV$sp(ConnectionManager.scala:896)
at
com.atscale.engine.connection.pool.main$$anonfun$30$$anonfun$apply$2.apply(ConnectionManager.scala:893)
at
[jira] [Commented] (SPARK-4908) Spark SQL built for Hive 13 fails under concurrent metadata queries
[
https://issues.apache.org/jira/browse/SPARK-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14268894#comment-14268894
]
Cheng Lian commented on SPARK-4908:
---
It was considered as a quick fix because we hadn't figured out the root cause
when the PR was submitted. But now it turned out to be a valid fix :)
Spark SQL built for Hive 13 fails under concurrent metadata queries
---
Key: SPARK-4908
URL: https://issues.apache.org/jira/browse/SPARK-4908
Project: Spark
Issue Type: Bug
Components: SQL
Reporter: David Ross
Assignee: Cheng Lian
Priority: Blocker
Fix For: 1.3.0, 1.2.1
We are trunk: {{1.3.0-SNAPSHOT}}, as of this commit:
https://github.com/apache/spark/commit/3d0c37b8118f6057a663f959321a79b8061132b6
We are using Spark built for Hive 13, using this option:
{{-Phive-0.13.1}}
In single-threaded mode, normal operations look fine. However, under
concurrency, with at least 2 concurrent connections, metadata queries fail.
For example, {{USE some_db}}, {{SHOW TABLES}}, and the implicit {{USE}}
statement when you pass a default schema in the JDBC URL, all fail.
{{SELECT}} queries like {{SELECT * FROM some_table}} do not have this issue.
Here is some example code:
{code}
object main extends App {
import java.sql._
import scala.concurrent._
import scala.concurrent.duration._
import scala.concurrent.ExecutionContext.Implicits.global
Class.forName(org.apache.hive.jdbc.HiveDriver)
val host = localhost // update this
val url = sjdbc:hive2://${host}:10511/some_db // update this
val future = Future.traverse(1 to 3) { i =
Future {
println(Starting: + i)
try {
val conn = DriverManager.getConnection(url)
} catch {
case e: Throwable = e.printStackTrace()
println(Failed: + i)
}
println(Finishing: + i)
}
}
Await.result(future, 2.minutes)
println(done!)
}
{code}
Here is the output:
{code}
Starting: 1
Starting: 3
Starting: 2
java.sql.SQLException:
org.apache.spark.sql.execution.QueryExecutionException: FAILED: Operation
cancelled
at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:121)
at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:109)
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:231)
at
org.apache.hive.jdbc.HiveConnection.configureConnection(HiveConnection.java:451)
at org.apache.hive.jdbc.HiveConnection.init(HiveConnection.java:195)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:270)
at
com.atscale.engine.connection.pool.main$$anonfun$30$$anonfun$apply$2.apply$mcV$sp(ConnectionManager.scala:896)
at
com.atscale.engine.connection.pool.main$$anonfun$30$$anonfun$apply$2.apply(ConnectionManager.scala:893)
at
com.atscale.engine.connection.pool.main$$anonfun$30$$anonfun$apply$2.apply(ConnectionManager.scala:893)
at
scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at
scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at
scala.concurrent.impl.ExecutionContextImpl$AdaptedForkJoinTask.exec(ExecutionContextImpl.scala:121)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Failed: 3
Finishing: 3
java.sql.SQLException:
org.apache.spark.sql.execution.QueryExecutionException: FAILED: Operation
cancelled
at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:121)
at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:109)
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:231)
at
org.apache.hive.jdbc.HiveConnection.configureConnection(HiveConnection.java:451)
at org.apache.hive.jdbc.HiveConnection.init(HiveConnection.java:195)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:270)
at
com.atscale.engine.connection.pool.main$$anonfun$30$$anonfun$apply$2.apply$mcV$sp(ConnectionManager.scala:896)
at
com.atscale.engine.connection.pool.main$$anonfun$30$$anonfun$apply$2.apply(ConnectionManager.scala:893)
at
[jira] [Commented] (SPARK-4908) Spark SQL built for Hive 13 fails under concurrent metadata queries
[
https://issues.apache.org/jira/browse/SPARK-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14268892#comment-14268892
]
David Ross commented on SPARK-4908:
---
I've verified that this is fixed on trunk. Since his commit says just a quick
fix, I will let [~marmbrus] decide whether or not to keep this JIRA open.
Spark SQL built for Hive 13 fails under concurrent metadata queries
---
Key: SPARK-4908
URL: https://issues.apache.org/jira/browse/SPARK-4908
Project: Spark
Issue Type: Bug
Components: SQL
Reporter: David Ross
Assignee: Cheng Lian
Priority: Blocker
Fix For: 1.3.0, 1.2.1
We are trunk: {{1.3.0-SNAPSHOT}}, as of this commit:
https://github.com/apache/spark/commit/3d0c37b8118f6057a663f959321a79b8061132b6
We are using Spark built for Hive 13, using this option:
{{-Phive-0.13.1}}
In single-threaded mode, normal operations look fine. However, under
concurrency, with at least 2 concurrent connections, metadata queries fail.
For example, {{USE some_db}}, {{SHOW TABLES}}, and the implicit {{USE}}
statement when you pass a default schema in the JDBC URL, all fail.
{{SELECT}} queries like {{SELECT * FROM some_table}} do not have this issue.
Here is some example code:
{code}
object main extends App {
import java.sql._
import scala.concurrent._
import scala.concurrent.duration._
import scala.concurrent.ExecutionContext.Implicits.global
Class.forName(org.apache.hive.jdbc.HiveDriver)
val host = localhost // update this
val url = sjdbc:hive2://${host}:10511/some_db // update this
val future = Future.traverse(1 to 3) { i =
Future {
println(Starting: + i)
try {
val conn = DriverManager.getConnection(url)
} catch {
case e: Throwable = e.printStackTrace()
println(Failed: + i)
}
println(Finishing: + i)
}
}
Await.result(future, 2.minutes)
println(done!)
}
{code}
Here is the output:
{code}
Starting: 1
Starting: 3
Starting: 2
java.sql.SQLException:
org.apache.spark.sql.execution.QueryExecutionException: FAILED: Operation
cancelled
at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:121)
at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:109)
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:231)
at
org.apache.hive.jdbc.HiveConnection.configureConnection(HiveConnection.java:451)
at org.apache.hive.jdbc.HiveConnection.init(HiveConnection.java:195)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:270)
at
com.atscale.engine.connection.pool.main$$anonfun$30$$anonfun$apply$2.apply$mcV$sp(ConnectionManager.scala:896)
at
com.atscale.engine.connection.pool.main$$anonfun$30$$anonfun$apply$2.apply(ConnectionManager.scala:893)
at
com.atscale.engine.connection.pool.main$$anonfun$30$$anonfun$apply$2.apply(ConnectionManager.scala:893)
at
scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at
scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at
scala.concurrent.impl.ExecutionContextImpl$AdaptedForkJoinTask.exec(ExecutionContextImpl.scala:121)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Failed: 3
Finishing: 3
java.sql.SQLException:
org.apache.spark.sql.execution.QueryExecutionException: FAILED: Operation
cancelled
at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:121)
at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:109)
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:231)
at
org.apache.hive.jdbc.HiveConnection.configureConnection(HiveConnection.java:451)
at org.apache.hive.jdbc.HiveConnection.init(HiveConnection.java:195)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:270)
at
com.atscale.engine.connection.pool.main$$anonfun$30$$anonfun$apply$2.apply$mcV$sp(ConnectionManager.scala:896)
at
com.atscale.engine.connection.pool.main$$anonfun$30$$anonfun$apply$2.apply(ConnectionManager.scala:893)
at
[jira] [Commented] (SPARK-4908) Spark SQL built for Hive 13 fails under concurrent metadata queries
[
https://issues.apache.org/jira/browse/SPARK-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14264469#comment-14264469
]
Cheng Lian commented on SPARK-4908:
---
Would like to add a comment about the root cause of this issue. When serving a
HiveQL query, Spark SQL's {{HiveContext.runHive}} method gets a
{{org.apache.hadoop.hive.ql.Driver}} instance via
{{CommandProcessFactory.get}}, which creates and caches {{Driver}} instances.
In the case of {{HiveThriftServer2}}, {{HiveContext.runHive}} is called by
multiple threads owned by a threaded executor of the Thrift server. However,
{{Driver}} is not thread safe, but cached {{Driver}} instance can be accessed
by multiple threads, thus causes problem. PR #3834 fixes this issue by
synchronizing {{HiveContext.runHive}}, which is valid. On the other hand,
HiveServer2 actually create a new {{Driver}} instance for every served SQL
query when initializing a {{SQLOperation}}.
[~dyross] When built against Hive 0.12.0, Spark SQL 1.2.0 also suffers this
issue. The snippet doesn't show this because Hive 0.12.0 JDBC driver doesn't
execute a {{USE db}} statement to switch current database even if the JDBC
connection URL specifies a database name. If you replace the lines in the
{{try}} block with:
{code}
val conn = DriverManager.getConnection(url)
val stmt = conn.createStatement()
stmt.execute(use hello;)
stmt.close()
println(Finished: + i)
{code}
you'll see exactly the same exceptions.
Spark SQL built for Hive 13 fails under concurrent metadata queries
---
Key: SPARK-4908
URL: https://issues.apache.org/jira/browse/SPARK-4908
Project: Spark
Issue Type: Bug
Components: SQL
Reporter: David Ross
Assignee: Cheng Lian
Priority: Blocker
Fix For: 1.3.0, 1.2.1
We are trunk: {{1.3.0-SNAPSHOT}}, as of this commit:
https://github.com/apache/spark/commit/3d0c37b8118f6057a663f959321a79b8061132b6
We are using Spark built for Hive 13, using this option:
{{-Phive-0.13.1}}
In single-threaded mode, normal operations look fine. However, under
concurrency, with at least 2 concurrent connections, metadata queries fail.
For example, {{USE some_db}}, {{SHOW TABLES}}, and the implicit {{USE}}
statement when you pass a default schema in the JDBC URL, all fail.
{{SELECT}} queries like {{SELECT * FROM some_table}} do not have this issue.
Here is some example code:
{code}
object main extends App {
import java.sql._
import scala.concurrent._
import scala.concurrent.duration._
import scala.concurrent.ExecutionContext.Implicits.global
Class.forName(org.apache.hive.jdbc.HiveDriver)
val host = localhost // update this
val url = sjdbc:hive2://${host}:10511/some_db // update this
val future = Future.traverse(1 to 3) { i =
Future {
println(Starting: + i)
try {
val conn = DriverManager.getConnection(url)
} catch {
case e: Throwable = e.printStackTrace()
println(Failed: + i)
}
println(Finishing: + i)
}
}
Await.result(future, 2.minutes)
println(done!)
}
{code}
Here is the output:
{code}
Starting: 1
Starting: 3
Starting: 2
java.sql.SQLException:
org.apache.spark.sql.execution.QueryExecutionException: FAILED: Operation
cancelled
at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:121)
at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:109)
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:231)
at
org.apache.hive.jdbc.HiveConnection.configureConnection(HiveConnection.java:451)
at org.apache.hive.jdbc.HiveConnection.init(HiveConnection.java:195)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:270)
at
com.atscale.engine.connection.pool.main$$anonfun$30$$anonfun$apply$2.apply$mcV$sp(ConnectionManager.scala:896)
at
com.atscale.engine.connection.pool.main$$anonfun$30$$anonfun$apply$2.apply(ConnectionManager.scala:893)
at
com.atscale.engine.connection.pool.main$$anonfun$30$$anonfun$apply$2.apply(ConnectionManager.scala:893)
at
scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at
scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at
scala.concurrent.impl.ExecutionContextImpl$AdaptedForkJoinTask.exec(ExecutionContextImpl.scala:121)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
[jira] [Commented] (SPARK-4908) Spark SQL built for Hive 13 fails under concurrent metadata queries
[
https://issues.apache.org/jira/browse/SPARK-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260490#comment-14260490
]
Michael Armbrust commented on SPARK-4908:
-
You don't even need the JDBC server to cause the problem. This seems to fail
as well:
{code}
(1 to 100).par.map { _ =
sql(USE default)
sql(SHOW TABLES)
}
{code}
Spark SQL built for Hive 13 fails under concurrent metadata queries
---
Key: SPARK-4908
URL: https://issues.apache.org/jira/browse/SPARK-4908
Project: Spark
Issue Type: Bug
Components: SQL
Reporter: David Ross
Priority: Critical
We are trunk: {{1.3.0-SNAPSHOT}}, as of this commit:
https://github.com/apache/spark/commit/3d0c37b8118f6057a663f959321a79b8061132b6
We are using Spark built for Hive 13, using this option:
{{-Phive-0.13.1}}
In single-threaded mode, normal operations look fine. However, under
concurrency, with at least 2 concurrent connections, metadata queries fail.
For example, {{USE some_db}}, {{SHOW TABLES}}, and the implicit {{USE}}
statement when you pass a default schema in the JDBC URL, all fail.
{{SELECT}} queries like {{SELECT * FROM some_table}} do not have this issue.
Here is some example code:
{code}
object main extends App {
import java.sql._
import scala.concurrent._
import scala.concurrent.duration._
import scala.concurrent.ExecutionContext.Implicits.global
Class.forName(org.apache.hive.jdbc.HiveDriver)
val host = localhost // update this
val url = sjdbc:hive2://${host}:10511/some_db // update this
val future = Future.traverse(1 to 3) { i =
Future {
println(Starting: + i)
try {
val conn = DriverManager.getConnection(url)
} catch {
case e: Throwable = e.printStackTrace()
println(Failed: + i)
}
println(Finishing: + i)
}
}
Await.result(future, 2.minutes)
println(done!)
}
{code}
Here is the output:
{code}
Starting: 1
Starting: 3
Starting: 2
java.sql.SQLException:
org.apache.spark.sql.execution.QueryExecutionException: FAILED: Operation
cancelled
at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:121)
at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:109)
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:231)
at
org.apache.hive.jdbc.HiveConnection.configureConnection(HiveConnection.java:451)
at org.apache.hive.jdbc.HiveConnection.init(HiveConnection.java:195)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:270)
at
com.atscale.engine.connection.pool.main$$anonfun$30$$anonfun$apply$2.apply$mcV$sp(ConnectionManager.scala:896)
at
com.atscale.engine.connection.pool.main$$anonfun$30$$anonfun$apply$2.apply(ConnectionManager.scala:893)
at
com.atscale.engine.connection.pool.main$$anonfun$30$$anonfun$apply$2.apply(ConnectionManager.scala:893)
at
scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at
scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at
scala.concurrent.impl.ExecutionContextImpl$AdaptedForkJoinTask.exec(ExecutionContextImpl.scala:121)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Failed: 3
Finishing: 3
java.sql.SQLException:
org.apache.spark.sql.execution.QueryExecutionException: FAILED: Operation
cancelled
at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:121)
at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:109)
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:231)
at
org.apache.hive.jdbc.HiveConnection.configureConnection(HiveConnection.java:451)
at org.apache.hive.jdbc.HiveConnection.init(HiveConnection.java:195)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:270)
at
com.atscale.engine.connection.pool.main$$anonfun$30$$anonfun$apply$2.apply$mcV$sp(ConnectionManager.scala:896)
at
com.atscale.engine.connection.pool.main$$anonfun$30$$anonfun$apply$2.apply(ConnectionManager.scala:893)
at
[jira] [Commented] (SPARK-4908) Spark SQL built for Hive 13 fails under concurrent metadata queries
[
https://issues.apache.org/jira/browse/SPARK-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260714#comment-14260714
]
Apache Spark commented on SPARK-4908:
-
User 'marmbrus' has created a pull request for this issue:
https://github.com/apache/spark/pull/3834
Spark SQL built for Hive 13 fails under concurrent metadata queries
---
Key: SPARK-4908
URL: https://issues.apache.org/jira/browse/SPARK-4908
Project: Spark
Issue Type: Bug
Components: SQL
Reporter: David Ross
Assignee: Cheng Lian
Priority: Blocker
We are trunk: {{1.3.0-SNAPSHOT}}, as of this commit:
https://github.com/apache/spark/commit/3d0c37b8118f6057a663f959321a79b8061132b6
We are using Spark built for Hive 13, using this option:
{{-Phive-0.13.1}}
In single-threaded mode, normal operations look fine. However, under
concurrency, with at least 2 concurrent connections, metadata queries fail.
For example, {{USE some_db}}, {{SHOW TABLES}}, and the implicit {{USE}}
statement when you pass a default schema in the JDBC URL, all fail.
{{SELECT}} queries like {{SELECT * FROM some_table}} do not have this issue.
Here is some example code:
{code}
object main extends App {
import java.sql._
import scala.concurrent._
import scala.concurrent.duration._
import scala.concurrent.ExecutionContext.Implicits.global
Class.forName(org.apache.hive.jdbc.HiveDriver)
val host = localhost // update this
val url = sjdbc:hive2://${host}:10511/some_db // update this
val future = Future.traverse(1 to 3) { i =
Future {
println(Starting: + i)
try {
val conn = DriverManager.getConnection(url)
} catch {
case e: Throwable = e.printStackTrace()
println(Failed: + i)
}
println(Finishing: + i)
}
}
Await.result(future, 2.minutes)
println(done!)
}
{code}
Here is the output:
{code}
Starting: 1
Starting: 3
Starting: 2
java.sql.SQLException:
org.apache.spark.sql.execution.QueryExecutionException: FAILED: Operation
cancelled
at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:121)
at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:109)
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:231)
at
org.apache.hive.jdbc.HiveConnection.configureConnection(HiveConnection.java:451)
at org.apache.hive.jdbc.HiveConnection.init(HiveConnection.java:195)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:270)
at
com.atscale.engine.connection.pool.main$$anonfun$30$$anonfun$apply$2.apply$mcV$sp(ConnectionManager.scala:896)
at
com.atscale.engine.connection.pool.main$$anonfun$30$$anonfun$apply$2.apply(ConnectionManager.scala:893)
at
com.atscale.engine.connection.pool.main$$anonfun$30$$anonfun$apply$2.apply(ConnectionManager.scala:893)
at
scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at
scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at
scala.concurrent.impl.ExecutionContextImpl$AdaptedForkJoinTask.exec(ExecutionContextImpl.scala:121)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Failed: 3
Finishing: 3
java.sql.SQLException:
org.apache.spark.sql.execution.QueryExecutionException: FAILED: Operation
cancelled
at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:121)
at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:109)
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:231)
at
org.apache.hive.jdbc.HiveConnection.configureConnection(HiveConnection.java:451)
at org.apache.hive.jdbc.HiveConnection.init(HiveConnection.java:195)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:270)
at
com.atscale.engine.connection.pool.main$$anonfun$30$$anonfun$apply$2.apply$mcV$sp(ConnectionManager.scala:896)
at
com.atscale.engine.connection.pool.main$$anonfun$30$$anonfun$apply$2.apply(ConnectionManager.scala:893)
at
com.atscale.engine.connection.pool.main$$anonfun$30$$anonfun$apply$2.apply(ConnectionManager.scala:893)
at
[jira] [Commented] (SPARK-4908) Spark SQL built for Hive 13 fails under concurrent metadata queries
[
https://issues.apache.org/jira/browse/SPARK-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14256174#comment-14256174
]
David Ross commented on SPARK-4908:
---
Note that noticed this line from native Hive logging:
{code}
14/12/19 21:44:55 INFO ql.Driver: Concurrency mode is disabled, not creating a
lock manager
{code}
It seems to be tied to this config:
https://github.com/apache/hive/blob/branch-0.13/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java#L719
I have this to our {{hive-site.xml}} in the spark {{conf}} directory:
{code}
property
namehive.support.concurrency/name
valuetrue/value
/property
{code}
And I still have the issue.
Perhaps there is more I need to do to support concurrency?
Spark SQL built for Hive 13 fails under concurrent metadata queries
---
Key: SPARK-4908
URL: https://issues.apache.org/jira/browse/SPARK-4908
Project: Spark
Issue Type: Bug
Reporter: David Ross
We are trunk: {{1.3.0-SNAPSHOT}}, as of this commit:
https://github.com/apache/spark/commit/3d0c37b8118f6057a663f959321a79b8061132b6
We are using Spark built for Hive 13, using this option:
{{-Phive-0.13.1}}
In single-threaded mode, normal operations look fine. However, under
concurrency, with at least 2 concurrent connections, metadata queries fail.
For example, {{USE some_db}}, {{SHOW TABLES}}, and the implicit {{USE}}
statement when you pass a default schema in the JDBC URL, all fail.
{{SELECT}} queries like {{SELECT * FROM some_table}} do not have this issue.
Here is some example code:
{code}
object main extends App {
import java.sql._
import scala.concurrent._
import scala.concurrent.duration._
import scala.concurrent.ExecutionContext.Implicits.global
Class.forName(org.apache.hive.jdbc.HiveDriver)
val host = localhost // update this
val url = sjdbc:hive2://${host}:10511/some_db // update this
val future = Future.traverse(1 to 3) { i =
Future {
println(Starting: + i)
try {
val conn = DriverManager.getConnection(url)
} catch {
case e: Throwable = e.printStackTrace()
println(Failed: + i)
}
println(Finishing: + i)
}
}
Await.result(future, 2.minutes)
println(done!)
}
{code}
Here is the output:
{code}
Starting: 1
Starting: 3
Starting: 2
java.sql.SQLException:
org.apache.spark.sql.execution.QueryExecutionException: FAILED: Operation
cancelled
at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:121)
at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:109)
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:231)
at
org.apache.hive.jdbc.HiveConnection.configureConnection(HiveConnection.java:451)
at org.apache.hive.jdbc.HiveConnection.init(HiveConnection.java:195)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:270)
at
com.atscale.engine.connection.pool.main$$anonfun$30$$anonfun$apply$2.apply$mcV$sp(ConnectionManager.scala:896)
at
com.atscale.engine.connection.pool.main$$anonfun$30$$anonfun$apply$2.apply(ConnectionManager.scala:893)
at
com.atscale.engine.connection.pool.main$$anonfun$30$$anonfun$apply$2.apply(ConnectionManager.scala:893)
at
scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at
scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at
scala.concurrent.impl.ExecutionContextImpl$AdaptedForkJoinTask.exec(ExecutionContextImpl.scala:121)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Failed: 3
Finishing: 3
java.sql.SQLException:
org.apache.spark.sql.execution.QueryExecutionException: FAILED: Operation
cancelled
at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:121)
at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:109)
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:231)
at
org.apache.hive.jdbc.HiveConnection.configureConnection(HiveConnection.java:451)
at org.apache.hive.jdbc.HiveConnection.init(HiveConnection.java:195)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:270)
at
