[ https://issues.apache.org/jira/browse/SPARK-3544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen resolved SPARK-3544. ------------------------------ Resolution: Incomplete I think this is at least obsolete now; no follow up or other reports of this, but not particular resolution either. > SparkSQL thriftServer cannot release locks correctly in Zookeeper > ----------------------------------------------------------------- > > Key: SPARK-3544 > URL: https://issues.apache.org/jira/browse/SPARK-3544 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 1.1.0 > Reporter: Patrick Liu > Priority: Critical > > Bug description: > The thriftServer cannot release locks correctly in Zookeeper. > Once a thriftServer is started, the 1st SQL submited by Beeline or JDBC > requiring locks can be successfully computed. > However, the 2rd SQL requiring locks will be blocked. And the thriftServer > log shows: INFO Driver: <PERFLOG method=acquireReadWriteLocks>. > 2 Tests to illustrate the problem: > Test 1: > (0) Start thriftServer & use beeline to connect to it. > (1) Switch database(require locks);(OK) > (2) Drop table(require locks) “drop table src"; (Blocked) > Test 2: > (0) Start thriftServer & use beeline to connect to it. > (1) Drop table(require locks) "drop table src"; (OK) > (2) Drop another table(require locks) "drop table src2;"(Blocked) > Basic Information: > Spark 1.1.0 > Hadoop 2.0.0-cdh4.6.0 > Compile Command: > export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M > -XX:ReservedCodeCacheSize=512m" > mvn -Dhadoop.version=2.0.0-cdh4.6.0 -Phive -Pspark-ganglia-lgpl -DskipTests > package > hive-site.xml: > <configuration> > <property> > <name>fs.default.name</name> > <value>hdfs://ns</value> > </property> > <property> > <name>dfs.nameservices</name> > <value>ns</value> > </property> > > <property> > <name>dfs.ha.namenodes.ns</name> > <value>machine01,machine02</value> > </property> > > <property> > <name>dfs.namenode.rpc-address.ns.machine01</name> > <value>machine01:54310</value> > > </property> > <property> > <name>dfs.namenode.rpc-address.ns.machine02</name> > <value>machine02:54310</value> > > </property> > > <property> > <name>dfs.client.failover.proxy.provider.ns</name> > > <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value> > > </property> > <property> > <name>javax.jdo.option.ConnectionURL</name> > <value>jdbc:mysql://localhost:3306/metastore</value> > <description>JDBC connect string for a JDBC metastore</description> > </property> > <property> > <name>javax.jdo.option.ConnectionDriverName</name> > <value>com.mysql.jdbc.Driver</value> > <description>Driver class name for a JDBC metastore</description> > </property> > <property> > <name>javax.jdo.option.ConnectionUserName</name> > <value>hive_user</value> > </property> > <property> > <name>javax.jdo.option.ConnectionPassword</name> > <value>hive_123</value> > </property> > <property> > <name>datanucleus.autoCreateSchema</name> > <value>false</value> > </property> > <property> > <name>datanucleus.autoCreateTables</name> > <value>true</value> > </property> > <property> > <name>datanucleus.fixedDatastore</name> > <value>false</value> > </property> > <property> > <name>hive.support.concurrency</name> > <description>Enable Hive's Table Lock Manager Service</description> > <value>true</value> > </property> > <property> > <name>hive.zookeeper.quorum</name> > <value>machine01,machine02,machine03</value> > <description>Zookeeper quorum used by Hive's Table Lock > Manager</description> > </property> > <property> > <name>hive.metastore.warehouse.dir</name> > <value>/user/hive/warehouse</value> > <description>Hive warehouse directory</description> > </property> > <property> > <name>mapred.job.tracker</name> > <value>machine01:8032</value> > </property> > <property> > <name>io.compression.codecs</name> > <value>org.apache.hadoop.io.compress.SnappyCodec</value> > </property> > <property> > <name>mapreduce.output.fileoutputformat.compress.codec</name> > <value>org.apache.hadoop.io.compress.SnappyCodec</value> > </property> > <property> > <name>mapreduce.output.fileoutputformat.compress.type</name> > <value>BLOCK</value> > </property> > <property> > <name>hive.exec.show.job.failure.debug.info</name> > <value>true</value> > <description> > If a job fails, whether to provide a link in the CLI to the task with the > most failures, along with debugging hints if applicable. > </description> > </property> > <property> > <name>hive.hwi.listen.host</name> > <value>0.0.0.0</value> > <description>This is the host address the Hive Web Interface will listen > on</description> > </property> > <property> > <name>hive.hwi.listen.port</name> > <value>9999</value> > <description>This is the port the Hive Web Interface will listen > on</description> > </property> > <property> > <name>hive.hwi.war.file</name> > <value>/lib/hive-hwi-0.10.0-cdh4.2.0.war</value> > <description>This is the WAR file with the jsp content for Hive Web > Interface</description> > </property> > <property> > <name>hive.aux.jars.path</name> > > <value>file:///usr/lib/hive/lib/hive-hbase-handler-0.10.0-cdh4.6.0.jar,file:///usr/lib/hbase/hbase-0.94.15-cdh4.6.0-security.jar,file:///usr/lib/zookeeper/zookeeper.jar</value> > > </property> > <property> > <name>hbase.zookeeper.quorum</name> > <value>machine01,machine02,machine03</value> > </property> > <property> > <name>hive.cli.print.header</name> > <value>true</value> > </property> > <property> > <name>hive.metastore.execute.setugi</name> > <value>true</value> > <description>In unsecure mode, setting this property to true will cause > the metastore to execute DFS operations using the client's reported user and > group permissions. Note that this property must be set on both the client and > server sides. Further note that its best effort. If client sets its to true > and server sets it to false, client setting will be ignored.</description> > </property> > <property> > <name>hive.security.authorization.enabled</name> > <value>true</value> > <description>enable or disable the hive client > authorization</description> > </property> > <property> > <name>hive.metastore.authorization.storage.checks</name> > <value>true</value> > </property> > <property> > <name>hive.security.authorization.createtable.owner.grants</name> > <value>ALL</value> > <description>the privileges automatically granted to the owner whenever a > table gets created. > An example like "select,drop" will grant select and drop privilege to the > owner of the table</description> > </property> > </configuration> -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org