[ https://issues.apache.org/jira/browse/SPARK-12216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16744675#comment-16744675 ]
Kingsley Jones edited comment on SPARK-12216 at 1/17/19 4:51 AM: ----------------------------------------------------------------- Okay, so I think we have a candidate for what is actually causing the problem. There is an open bug on the scala language site for a class within the scala REPL IMain.scala [https://github.com/scala/bug/issues/10045] `scala.tools.nsc.interpreter.IMain.TranslatingClassLoader` calls a non-thread-safe method `{{translateSimpleResource`}} (this method calls `{{SymbolTable.enteringPhase`}}), which makes it become non-thread-safe. "However, a ClassLoader must be thread-safe since the class can be loaded in arbitrary thread." In my REPL reflection experiment above the relevant class is: {color:#333333}`scala> val loader = Thread.currentThread.getContextClassLoader()`{color} {color:#333333}`loader: ClassLoader = scala.tools.nsc.interpreter.IMain$TranslatingClassLoader@3a1a20f`{color} {color:#333333}Ergo ... the same class in the above bug being marked as non threadsafe.{color} {color:#333333}See this stack overflow for a discussion of thread safety:{color} [https://stackoverflow.com/questions/46258558/scala-objects-and-thread-safety] The scala REPL code has some internal classloaders which are used to compile and execute any code entered into the REPL. On Windows 10, if you simply start a spark-shell from the command line, do nothing, and then :quit the REPL will barf with a stacktrace to this particular class reference (namely TranslatingClassLoader) which is identified as an open bug in the scala language issues marked "non threadsafe". {color:#24292e}I am gonna try and contact the person who raised the bug on the scala issues thread and get some input.{color} It seemed like he could only produce it with a complicated SQL script. Here we have with Apache Spark a simple, and on my tests 100% reproducible, instance of the bug on Windows 10 and in my tests Windows Server 2016. UPDATE: I cross-posted on [https://github.com/scala/bug/issues/10045] with an explanation of the observations made here and a link back to this issue. was (Author: kingsley): Okay, so I think we have a candidate for what is actually causing the problem. There is an open bug on the scala language site for a class within the scala REPL IMain.scala [https://github.com/scala/bug/issues/10045] 'scala.tools.nsc.interpreter.IMain.TranslatingClassLoader calls a non-thread-safe method {{translateSimpleResource}} (this method calls {{SymbolTable.enteringPhase}}), which makes it become non-thread-safe. However, a ClassLoader must be thread-safe since the class can be loaded in arbitrary thread. In my REPL reflection experiment above the relevant class is: {color:#333333}scala> val loader = Thread.currentThread.getContextClassLoader(){color} {color:#333333} loader: ClassLoader = scala.tools.nsc.interpreter.IMain$TranslatingClassLoader@3a1a20f{color} {color:#333333}Ergo ... the very class in the above bug being marked as non threadsafe.{color} {color:#333333}See this stack overflow for a discussion of thread safety:{color} [https://stackoverflow.com/questions/46258558/scala-objects-and-thread-safety] The scala REPL code has some internal classloaders which are used to compile and execute any code entered into the REPL. On Windows 10, if you simply start a spark-shell from the command line, do nothing, and then :quit the REPL will barf with a stacktrace to this particular class reference (namely TranslatingClassLoader) which is identified as an open bug in the scala language issues marked "non threadsafe". {color:#24292e}I am gonna try and contact the person who raised the bug on the scala issues thread and get some input.{color} It seemed like he could only produce it with a complicated SQL script. Here we have with Apache Spark a simple, and on my tests 100% reproducible, instance of the bug on Windows 10 and in my tests Windows Server 2016. UPDATE: I cross-posted on [https://github.com/scala/bug/issues/10045] with an explanation of the observations made here and a link back to this issue. > Spark failed to delete temp directory > -------------------------------------- > > Key: SPARK-12216 > URL: https://issues.apache.org/jira/browse/SPARK-12216 > Project: Spark > Issue Type: Bug > Components: Spark Shell > Environment: windows 7 64 bit > Spark 1.52 > Java 1.8.0.65 > PATH includes: > C:\Users\Stefan\spark-1.5.2-bin-hadoop2.6\bin > C:\ProgramData\Oracle\Java\javapath > C:\Users\Stefan\scala\bin > SYSTEM variables set are: > JAVA_HOME=C:\Program Files\Java\jre1.8.0_65 > HADOOP_HOME=C:\Users\Stefan\hadoop-2.6.0\bin > (where the bin\winutils resides) > both \tmp and \tmp\hive have permissions > drwxrwxrwx as detected by winutils ls > Reporter: stefan > Priority: Minor > > The mailing list archives have no obvious solution to this: > scala> :q > Stopping spark context. > 15/12/08 16:24:22 ERROR ShutdownHookManager: Exception while deleting Spark > temp dir: > C:\Users\Stefan\AppData\Local\Temp\spark-18f2a418-e02f-458b-8325-60642868fdff > java.io.IOException: Failed to delete: > C:\Users\Stefan\AppData\Local\Temp\spark-18f2a418-e02f-458b-8325-60642868fdff > at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:884) > at > org.apache.spark.util.ShutdownHookManager$$anonfun$1$$anonfun$apply$mcV$sp$3.apply(ShutdownHookManager.scala:63) > at > org.apache.spark.util.ShutdownHookManager$$anonfun$1$$anonfun$apply$mcV$sp$3.apply(ShutdownHookManager.scala:60) > at scala.collection.mutable.HashSet.foreach(HashSet.scala:79) > at > org.apache.spark.util.ShutdownHookManager$$anonfun$1.apply$mcV$sp(ShutdownHookManager.scala:60) > at > org.apache.spark.util.SparkShutdownHook.run(ShutdownHookManager.scala:264) > at > org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ShutdownHookManager.scala:234) > at > org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:234) > at > org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:234) > at > org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1699) > at > org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply$mcV$sp(ShutdownHookManager.scala:234) > at > org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:234) > at > org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:234) > at scala.util.Try$.apply(Try.scala:161) > at > org.apache.spark.util.SparkShutdownHookManager.runAll(ShutdownHookManager.scala:234) > at > org.apache.spark.util.SparkShutdownHookManager$$anon$2.run(ShutdownHookManager.scala:216) > at > org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54) -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org