xinzhang created SPARK-23022: -------------------------------- Summary: Spark Thrift Server always cache resource issues Key: SPARK-23022 URL: https://issues.apache.org/jira/browse/SPARK-23022 Project: Spark Issue Type: Bug Components: Deploy Affects Versions: 2.2.1 Environment: CentOS6.x Spark2.x JDK1.8 Reporter: xinzhang
Hi. I use the Thrift Server for SparkSQL . I query muiltle query.I deply the Spark on Yarn. When I finish my query.Thrift Server always cache the Yarn Resources. Any suggests will be helpful. ------------------------------------------------------------------------------------ Here is the img . ------------------------------------------------------------------------------------ !https://user-images.githubusercontent.com/8244097/34752652-8d224416-f5ee-11e7-89d3-5868c128378d.png! !https://user-images.githubusercontent.com/8244097/34752397-215bea08-f5ed-11e7-89f4-13ef9ab78904.png! !https://user-images.githubusercontent.com/8244097/34752403-2756d224-f5ed-11e7-97d3-9c7d21c48f3a.png! !https://user-images.githubusercontent.com/8244097/34752409-2ba1d3ce-f5ed-11e7-85d6-9e46ae8a3e2b.png! ------------------------------------------------------------------------------------ Here is the Spark Conf . ------------------------------------------------------------------------------------ {code:java} spark.dynamicAllocation.minExecutors 1 spark.dynamicAllocation.maxExecutors 500 spark.dynamicAllocation.enabled true spark.shuffle.service.enabled true spark.executor.instances 2 spark.executor.memory 6g #serializer spark.serializer org.apache.spark.serializer.KryoSerializer spark.kryoserializer.buffer.max 512m #spark.kryo.unsafe true spark.kryo.referenceTracking false spark.rdd.compress true spark.memory.offHeap.enabled true spark.memory.offHeap.size 1g spark.yarn.archive hdfs://ns/data1/hadooptmp/spark2.2.1/jars spark.yarn.am.memory 2g spark.driver.memory 4g spark.driver.maxResultSize 2g #SPARK SQL spark.sql.shuffle.partitions 500 spark.sql.statistics.fallBackToHdfs true spark.sql.orc.filterPushdown true spark.sql.autoBroadcastJoinThreshold 104857600 spark.sql.adaptive.enabled true spark.history.fs.logDirectory hdfs://ns/data4/hadooptmp/spark-history spark.eventLog.enabled true spark.eventLog.compress true spark.eventLog.dir hdfs://ns/data4/hadooptmp/spark-history spark.yarn.historyServer.address 172.31.10.119:18080 spark.io.compression.codec snappy spark.executor.logs.rolling.enableCompression true spark.dynamicAllocation.executorIdleTimeout 10s spark.network.timeout 600s spark.sql.parquet.writeLegacyFormat true {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org