[ https://issues.apache.org/jira/browse/SPARK-44458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kapil Singh updated SPARK-44458: -------------------------------- Attachment: (was: image-2023-07-17-16-57-09-669.png) > ThreadLocal not being copied to child thread when child thread is reused from > pool > ----------------------------------------------------------------------------------- > > Key: SPARK-44458 > URL: https://issues.apache.org/jira/browse/SPARK-44458 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 3.4.1 > Reporter: Kapil Singh > Priority: Major > > Currently setting the Spark configuration using the statement > `spark.conf.set("spark.sql.caseSensitive", "true")` and subsequently > executing a Spark action in a separate thread. An intermittent observation > reveals that occasionally the configuration set in the main thread fails to > propagate to the child thread > Steps to repro: [Spark SQL configs can't get propagated properly to a new > thread in Spark 3.1 issue · > GitHub|https://gist.github.com/t-rufang/25341b3678e5d7c74e3a209457fce0e9] > > This is limitation due to ThreadLocal not being copied to child thread when > child thread is reused from pool and only done instead in thread creation or > when idle thread has expired KeepAlive in Executor > Refer: > [https://users.scala-lang.org/t/future-executioncontext-and-threadlocal/7675/2] > > [https://www.stevenskelton.ca/threadlocal-variables-scala-futures/] -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org