[
https://issues.apache.org/jira/browse/SPARK-44458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kapil Singh updated SPARK-44458:
--------------------------------
Description:
Currently setting the Spark configuration using the statement
`spark.conf.set("spark.sql.caseSensitive", "true")` and subsequently executing
a Spark action in a separate thread. An intermittent observation reveals that
occasionally the configuration set in the main thread fails to propagate to the
child thread
Steps to repro: [Spark SQL configs can't get propagated properly to a new
thread in Spark 3.1 issue ·
GitHub|https://gist.github.com/t-rufang/25341b3678e5d7c74e3a209457fce0e9]
This is limitation due to ThreadLocal not being copied to child thread when
child thread is reused from pool and only done instead in thread creation or
when idle thread has expired KeepAlive in Executor
Refer:
[https://users.scala-lang.org/t/future-executioncontext-and-threadlocal/7675/2]
[https://www.stevenskelton.ca/threadlocal-variables-scala-futures/]
was:
Currently setting the Spark configuration using the statement
`spark.conf.set("spark.sql.caseSensitive", "true")` and subsequently executing
a Spark action in a separate thread. An intermittent observation reveals that
occasionally the configuration set in the main thread fails to propagate to the
child thread.
!image-2023-07-17-16-56-59-388.png!
!image-2023-07-17-16-57-09-669.png!
This is limitation due to ThreadLocal not being copied to child thread when
child thread is reused from pool and only done instead in thread creation or
when idle thread has expired KeepAlive in Executor
Refer:
[https://users.scala-lang.org/t/future-executioncontext-and-threadlocal/7675/2]
[https://www.stevenskelton.ca/threadlocal-variables-scala-futures/]
> ThreadLocal not being copied to child thread when child thread is reused from
> pool
> -----------------------------------------------------------------------------------
>
> Key: SPARK-44458
> URL: https://issues.apache.org/jira/browse/SPARK-44458
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 3.4.1
> Reporter: Kapil Singh
> Priority: Major
> Attachments: image-2023-07-17-16-56-59-388.png,
> image-2023-07-17-16-57-09-669.png
>
>
> Currently setting the Spark configuration using the statement
> `spark.conf.set("spark.sql.caseSensitive", "true")` and subsequently
> executing a Spark action in a separate thread. An intermittent observation
> reveals that occasionally the configuration set in the main thread fails to
> propagate to the child thread
> Steps to repro: [Spark SQL configs can't get propagated properly to a new
> thread in Spark 3.1 issue ·
> GitHub|https://gist.github.com/t-rufang/25341b3678e5d7c74e3a209457fce0e9]
>
> This is limitation due to ThreadLocal not being copied to child thread when
> child thread is reused from pool and only done instead in thread creation or
> when idle thread has expired KeepAlive in Executor
> Refer:
> [https://users.scala-lang.org/t/future-executioncontext-and-threadlocal/7675/2]
>
> [https://www.stevenskelton.ca/threadlocal-variables-scala-futures/]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
