[ https://issues.apache.org/jira/browse/SPARK-10529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ZhengYaofeng updated SPARK-10529: --------------------------------- Flags: Patch > When creating multiple HiveContext objects in one jvm, jdbc connections to > metastore cann't be released and it may cause PermGen OutOfMemoryError. > -------------------------------------------------------------------------------------------------------------------------------------------------- > > Key: SPARK-10529 > URL: https://issues.apache.org/jira/browse/SPARK-10529 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 1.4.1 > Reporter: ZhengYaofeng > Attachments: IsolatedClientLoader.scala > > > Test code as follows: > object SqlTest { > def main (args: Array[String]) { > var port = 4042 > def createSc = { > val sparkConf = new SparkConf().setAppName(s"SqlTest_$port") > .setMaster("spark://zdh221:7077") > .set("spark.ui.port",port.toString) > .set("spark.executor.memory","4g") > .set("spark.executor.cores","2") > .set("spark.cores.max","6") > port += 1 > new SparkContext(sparkConf) > } > > while(port - 4042 < 200){ > println(s"============Current app id:${port - 4042}=============") > > val hc = new HiveContext(createSc) > hc.sql("show databases").collect().foreach(println) > hc.sparkContext.stop() > //Hive.closeCurrent() > //hc = null > Thread.sleep(3000) > } > > Thread.sleep(1000000) > } > } > > Testing on spark 1.4.1 with run cmd bellow. > export > CLASSPATH="$CLASSPATH:/home/hadoop/spark/conf:/home/hadoop/spark/lib/*:/home/hadoop/zyf/lib/*" > java -Xmx8096m -Xms1024m -XX:MaxPermSize=1024m -cp $CLASSPATH SqlTest > Files list: > > /home/hadoop/spark/conf:core-site.xml;hdfs-site.xml;hive-site.xml;slaves;spark-defaults.conf;spark-env.sh > > /home/hadoop/zyf/lib:hadoop-lzo-0.4.20.jar;mysql-connector-java-5.1.28-bin.jar;sqltest-1.0-SNAPSHOT.jar > > MySQL is used as the metastore. You can obviously see that jdbc connections > to MySQL grow constantly through command 'show status like > 'Threads_connected';' when my test app is running. Even if you invoke > 'Hive.closeCurrent()', it cann't release current jdbc connections. Besides I > can not find another possible way. If you take spark 1.3.1 to test, jdbc > connections won't grow. > Meanwhile, it ends with 'java.lang.OutOfMemoryError: PermGen space' when > cycling 45 times, which means 45 HiveContext objects are created. It's > interesting that if you set MaxPermSize to '2048m', it can cycle 93 times, if > you set MaxPermSize to '3072m', it can cycle 141 times. So,it indicates that > each time creating one HiveContext object, it loads the same amount of new > classes and they won't be released. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org