[
https://issues.apache.org/jira/browse/TINKERPOP3-988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marko A. Rodriguez closed TINKERPOP3-988.
-----------------------------------------
Resolution: Fixed
> SparkGraphComputer.submit shouldn't use ForkJoinPool.commonPool
> ---------------------------------------------------------------
>
> Key: TINKERPOP3-988
> URL: https://issues.apache.org/jira/browse/TINKERPOP3-988
> Project: TinkerPop 3
> Issue Type: Improvement
> Components: hadoop
> Affects Versions: 3.1.0-incubating
> Reporter: Dan LaRocque
> Assignee: Dan LaRocque
> Fix For: 3.1.1-incubating
>
>
> {{SparkGraphComputer.submit}} delegates most of its work to a closure that
> executes on the common forkjoin pool. The closure does a lot of stuff. It
> calls into both Spark and Hadoop.
> This approach has two problems:
> 1. Inability to customize the context classloader used within the closure
> The context classloader of the thread that called {{submit}} is not
> necessarily the same as the context classloader common forkjoin pool threads.
> This matters because multiple bits of code reachable from {{submit}}'s
> closure rely on the context classloader. SparkMemory is one; Hadoop's
> UserGroupInformation is another, depending on the credentials configuration
> (UGI is reached indirectly via {{FileSystem.get}}). This basically means
> that the caller has to use whatever context classloader is currently in use
> by the fork join common pool, or else bad things can happen, such as
> nonsensical-looking ClassCastExceptions.
> 2. Inability to override the context classloader inside the closure
> When {{System.getSecurityManager() != null}}, the common forkjoin pool
> switches from its default worker thread factory implementation to a more
> restrictive alternative called InnocuousForkJoinWorkerThreadFactory. Threads
> created by this factory can't call {{setContextClassLoader}}. Attempting to
> do so throws a SecurityException. However,
> UserGroupInformation.newLoginContext must be able to call
> {{setContextClassLoader}}. It saves the CCL to a variable, does some work,
> then restores the CCL from a variable. This is impossible if the method
> throws a SecurityException. So, if a security manager is present in the VM,
> {{submit}}'s closure can die in {{FileSystem.get}} -> UGI before any useful
> work even begins.
> I set the Affects Version to the version on which I observed it, but it might
> affect earlier versions too.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
