Spark Streaming uses threadpools so you need to remove ThreadLocal when it's not used.
On Fri, Jan 29, 2016 at 12:55 PM, N B <nb.nos...@gmail.com> wrote: > Thanks for the response Ryan. So I would say that it is in fact the > purpose of a ThreadLocal i.e. to have a copy of the variable as long as the > thread lives. I guess my concern is around usage of threadpools and whether > Spark streaming will internally create many threads that rotate between > tasks on purpose thereby holding onto ThreadLocals that may actually never > be used again. > > Thanks > > On Fri, Jan 29, 2016 at 12:12 PM, Shixiong(Ryan) Zhu < > shixi...@databricks.com> wrote: > >> Of cause. If you use a ThreadLocal in a long living thread and forget to >> remove it, it's definitely a memory leak. >> >> On Thu, Jan 28, 2016 at 9:31 PM, N B <nb.nos...@gmail.com> wrote: >> >>> Hello, >>> >>> Does anyone know if there are any potential pitfalls associated with >>> using ThreadLocal variables in a Spark streaming application? One things I >>> have seen mentioned in the context of app servers that use thread pools is >>> that ThreadLocals can leak memory. Could this happen in Spark streaming >>> also? >>> >>> Thanks >>> Nikunj >>> >>> >> >
