[
https://issues.apache.org/jira/browse/SPARK-3544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen resolved SPARK-3544.
------------------------------
Resolution: Incomplete
I think this is at least obsolete now; no follow up or other reports of this,
but not particular resolution either.
> SparkSQL thriftServer cannot release locks correctly in Zookeeper
> -----------------------------------------------------------------
>
> Key: SPARK-3544
> URL: https://issues.apache.org/jira/browse/SPARK-3544
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.1.0
> Reporter: Patrick Liu
> Priority: Critical
>
> Bug description:
> The thriftServer cannot release locks correctly in Zookeeper.
> Once a thriftServer is started, the 1st SQL submited by Beeline or JDBC
> requiring locks can be successfully computed.
> However, the 2rd SQL requiring locks will be blocked. And the thriftServer
> log shows: INFO Driver: <PERFLOG method=acquireReadWriteLocks>.
> 2 Tests to illustrate the problem:
> Test 1:
> (0) Start thriftServer & use beeline to connect to it.
> (1) Switch database(require locks);(OK)
> (2) Drop table(require locks) “drop table src"; (Blocked)
> Test 2:
> (0) Start thriftServer & use beeline to connect to it.
> (1) Drop table(require locks) "drop table src"; (OK)
> (2) Drop another table(require locks) "drop table src2;"(Blocked)
> Basic Information:
> Spark 1.1.0
> Hadoop 2.0.0-cdh4.6.0
> Compile Command:
> export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M
> -XX:ReservedCodeCacheSize=512m"
> mvn -Dhadoop.version=2.0.0-cdh4.6.0 -Phive -Pspark-ganglia-lgpl -DskipTests
> package
> hive-site.xml:
> <configuration>
> <property>
> <name>fs.default.name</name>
> <value>hdfs://ns</value>
> </property>
> <property>
> <name>dfs.nameservices</name>
> <value>ns</value>
> </property>
>
> <property>
> <name>dfs.ha.namenodes.ns</name>
> <value>machine01,machine02</value>
> </property>
>
> <property>
> <name>dfs.namenode.rpc-address.ns.machine01</name>
> <value>machine01:54310</value>
>
> </property>
> <property>
> <name>dfs.namenode.rpc-address.ns.machine02</name>
> <value>machine02:54310</value>
>
> </property>
>
> <property>
> <name>dfs.client.failover.proxy.provider.ns</name>
>
> <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
>
> </property>
> <property>
> <name>javax.jdo.option.ConnectionURL</name>
> <value>jdbc:mysql://localhost:3306/metastore</value>
> <description>JDBC connect string for a JDBC metastore</description>
> </property>
> <property>
> <name>javax.jdo.option.ConnectionDriverName</name>
> <value>com.mysql.jdbc.Driver</value>
> <description>Driver class name for a JDBC metastore</description>
> </property>
> <property>
> <name>javax.jdo.option.ConnectionUserName</name>
> <value>hive_user</value>
> </property>
> <property>
> <name>javax.jdo.option.ConnectionPassword</name>
> <value>hive_123</value>
> </property>
> <property>
> <name>datanucleus.autoCreateSchema</name>
> <value>false</value>
> </property>
> <property>
> <name>datanucleus.autoCreateTables</name>
> <value>true</value>
> </property>
> <property>
> <name>datanucleus.fixedDatastore</name>
> <value>false</value>
> </property>
> <property>
> <name>hive.support.concurrency</name>
> <description>Enable Hive's Table Lock Manager Service</description>
> <value>true</value>
> </property>
> <property>
> <name>hive.zookeeper.quorum</name>
> <value>machine01,machine02,machine03</value>
> <description>Zookeeper quorum used by Hive's Table Lock
> Manager</description>
> </property>
> <property>
> <name>hive.metastore.warehouse.dir</name>
> <value>/user/hive/warehouse</value>
> <description>Hive warehouse directory</description>
> </property>
> <property>
> <name>mapred.job.tracker</name>
> <value>machine01:8032</value>
> </property>
> <property>
> <name>io.compression.codecs</name>
> <value>org.apache.hadoop.io.compress.SnappyCodec</value>
> </property>
> <property>
> <name>mapreduce.output.fileoutputformat.compress.codec</name>
> <value>org.apache.hadoop.io.compress.SnappyCodec</value>
> </property>
> <property>
> <name>mapreduce.output.fileoutputformat.compress.type</name>
> <value>BLOCK</value>
> </property>
> <property>
> <name>hive.exec.show.job.failure.debug.info</name>
> <value>true</value>
> <description>
> If a job fails, whether to provide a link in the CLI to the task with the
> most failures, along with debugging hints if applicable.
> </description>
> </property>
> <property>
> <name>hive.hwi.listen.host</name>
> <value>0.0.0.0</value>
> <description>This is the host address the Hive Web Interface will listen
> on</description>
> </property>
> <property>
> <name>hive.hwi.listen.port</name>
> <value>9999</value>
> <description>This is the port the Hive Web Interface will listen
> on</description>
> </property>
> <property>
> <name>hive.hwi.war.file</name>
> <value>/lib/hive-hwi-0.10.0-cdh4.2.0.war</value>
> <description>This is the WAR file with the jsp content for Hive Web
> Interface</description>
> </property>
> <property>
> <name>hive.aux.jars.path</name>
>
> <value>file:///usr/lib/hive/lib/hive-hbase-handler-0.10.0-cdh4.6.0.jar,file:///usr/lib/hbase/hbase-0.94.15-cdh4.6.0-security.jar,file:///usr/lib/zookeeper/zookeeper.jar</value>
>
> </property>
> <property>
> <name>hbase.zookeeper.quorum</name>
> <value>machine01,machine02,machine03</value>
> </property>
> <property>
> <name>hive.cli.print.header</name>
> <value>true</value>
> </property>
> <property>
> <name>hive.metastore.execute.setugi</name>
> <value>true</value>
> <description>In unsecure mode, setting this property to true will cause
> the metastore to execute DFS operations using the client's reported user and
> group permissions. Note that this property must be set on both the client and
> server sides. Further note that its best effort. If client sets its to true
> and server sets it to false, client setting will be ignored.</description>
> </property>
> <property>
> <name>hive.security.authorization.enabled</name>
> <value>true</value>
> <description>enable or disable the hive client
> authorization</description>
> </property>
> <property>
> <name>hive.metastore.authorization.storage.checks</name>
> <value>true</value>
> </property>
> <property>
> <name>hive.security.authorization.createtable.owner.grants</name>
> <value>ALL</value>
> <description>the privileges automatically granted to the owner whenever a
> table gets created.
> An example like "select,drop" will grant select and drop privilege to the
> owner of the table</description>
> </property>
> </configuration>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
