[ https://issues.apache.org/jira/browse/SPARK-26914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-26914: ------------------------------------ Assignee: (was: Apache Spark) > ThriftServer scheduler pool may be unpredictably when using fair schedule mode > ------------------------------------------------------------------------------ > > Key: SPARK-26914 > URL: https://issues.apache.org/jira/browse/SPARK-26914 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.4.0 > Reporter: zhoukang > Priority: Major > Attachments: 26914-01.png, 26914-02.png, 26914-03.png > > > When using fair scheduler mode for thrift server, we may have unpredictable > result. > {code:java} > val pool = sessionToActivePool.get(parentSession.getSessionHandle) > if (pool != null) { > > sqlContext.sparkContext.setLocalProperty(SparkContext.SPARK_SCHEDULER_POOL, > pool) > } > {code} > Here is an example: > We have some query will use default pool, however it submit to 'normal' pool. > !26914-02.png! > I changed code and add some log.Got some strange result. > !26914-01.png! > !26914-03.png! > Then i found out that the localProperties of SparkContext may has > unpredictable result when call setLocalProperty. And since thriftserver use > thread pool to execute queries, it will trigger this bug sometimes. > {code:java} > /** > * Set a local property that affects jobs submitted from this thread, such > as the Spark fair > * scheduler pool. User-defined properties may also be set here. These > properties are propagated > * through to worker tasks and can be accessed there via > * [[org.apache.spark.TaskContext#getLocalProperty]]. > * > * These properties are inherited by child threads spawned from this > thread. This > * may have unexpected consequences when working with thread pools. The > standard java > * implementation of thread pools have worker threads spawn other worker > threads. > * As a result, local properties may propagate unpredictably. > */ > def setLocalProperty(key: String, value: String) { > if (value == null) { > localProperties.get.remove(key) > } else { > localProperties.get.setProperty(key, value) > } > } > {code} > -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org