Thank you Yin Huai. This is probably true.
I saw in the hive-site.xml, Liu has changed the entry, which is default should
be false.
<property>
<name>hive.support.concurrency</name>
<description>Enable Hive's Table Lock Manager Service</description>
<value>true</value>
</property>
Someone is working on upgrading the Hive to 0.13 for SparkSQL
(https://github.com/apache/spark/pull/2241), not sure if you can wait for this.
☺
From: Yin Huai [mailto:[email protected]]
Sent: Wednesday, September 17, 2014 1:50 AM
To: Cheng, Hao
Cc: linkpatrickliu; [email protected]
Subject: Re: SparkSQL 1.1 hang when "DROP" or "LOAD"
I meant it may be a Hive bug since we also call Hive's drop table internally.
On Tue, Sep 16, 2014 at 1:44 PM, Yin Huai
<[email protected]<mailto:[email protected]>> wrote:
Seems https://issues.apache.org/jira/browse/HIVE-5474 is related?
On Tue, Sep 16, 2014 at 4:49 AM, Cheng, Hao
<[email protected]<mailto:[email protected]>> wrote:
Thank you for pasting the steps, I will look at this, hopefully come out with a
solution soon.
-----Original Message-----
From: linkpatrickliu
[mailto:[email protected]<mailto:[email protected]>]
Sent: Tuesday, September 16, 2014 3:17 PM
To: [email protected]<mailto:[email protected]>
Subject: RE: SparkSQL 1.1 hang when "DROP" or "LOAD"
Hi, Hao Cheng.
I have done other tests. And the result shows the thriftServer can connect to
Zookeeper.
However, I found some more interesting things. And I think I have found a bug!
Test procedure:
Test1:
(0) Use beeline to connect to thriftServer.
(1) Switch database "use dw_op1"; (OK)
The logs show that the thriftServer connected with Zookeeper and acquired locks.
(2) Drop table "drop table src;" (Blocked) The logs show that the thriftServer
is "acquireReadWriteLocks".
Doubt:
The reason why I cannot drop table src is because the first SQL "use dw_op1"
have left locks in Zookeeper unsuccessfully released.
So when the second SQL is acquiring locks in Zookeeper, it will block.
Test2:
Restart thriftServer.
Instead of switching to another database, I just drop the table in the default
database;
(0) Restart thriftServer & use beeline to connect to thriftServer.
(1) Drop table "drop table src"; (OK)
Amazing! Succeed!
(2) Drop again! "drop table src2;" (Blocked) Same error: the thriftServer is
blocked in the "acquireReadWriteLocks"
phrase.
As you can see.
Only the first SQL requiring locks can succeed.
So I think the reason is that the thriftServer cannot release locks correctly
in Zookeeper.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/SparkSQL-1-1-hang-when-DROP-or-LOAD-tp14222p14339.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail:
[email protected]<mailto:[email protected]> For
additional commands, e-mail:
[email protected]<mailto:[email protected]>
---------------------------------------------------------------------
To unsubscribe, e-mail:
[email protected]<mailto:[email protected]>
For additional commands, e-mail:
[email protected]<mailto:[email protected]>
