[
https://issues.apache.org/jira/browse/SPARK-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16034427#comment-16034427
]
Robert Kruszewski commented on SPARK-20952:
-------------------------------------------
You're right that this needs a bit of clarification. There's a bit more
subtlety with respect to that actual threadlocal and it's use. First of all
this makes behaviour same as on the driver where you have localProperties on
SparkContext which are inheritable. Secondly I believe the issue you're
describing will not arise since a) executor tasks are uninterruptible and b)
the thread pool used to run them is a cachedThreadPool and not a ForkJoinPool,
hence given task thread will not inherit from another task thread. Let me know
if I am missing something here though.
> TaskContext should be an InheritableThreadLocal
> -----------------------------------------------
>
> Key: SPARK-20952
> URL: https://issues.apache.org/jira/browse/SPARK-20952
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.1.1
> Reporter: Robert Kruszewski
> Priority: Minor
>
> TaskContext is a ThreadLocal as a result when you fork a thread inside your
> executor task you lose the handle on the original context set by the
> executor. We should change it to InheritableThreadLocal so we can access it
> inside thread pools on executors.
> See ParquetFileFormat#readFootersInParallel for example of code that uses
> thread pools inside the tasks.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
