Github user NiharS commented on the issue: https://github.com/apache/spark/pull/22192 I'll change the config documentation to specify YARN only, hopefully it's not a huge issue. It seems like the line `Thread.currentThread().setContextClassLoader(replClassLoader)` is causing the pyspark failures, they pass when I remove it. I'm looking at the test cases but I really don't see how this is affecting them...it seems that in both test cases, the DStreams monitor a directory but don't pick up the changes they're supposed to, and just time out. I checked that I can bypass this issue by changing back to having the plugins loaded on a separate thread (and setting that thread's contextClassLoader instead of the current thread) and it passes tests and continues to work. That said this issue does seem to be indicative of some problem in pyspark streaming
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org