Possibly https://issues.apache.org/jira/browse/SPARK-17396 ?
On Tue, Nov 1, 2016 at 2:11 AM kant kodali <kanth...@gmail.com> wrote: > Hi Ryan, > > I think you are right. This may not be related to the Receiver. I have > attached jstack dump here. I do a simple MapToPair and reduceByKey and I > have a window Interval of 1 minute (60000ms) and batch interval of 1s ( > 1000) This is generating lot of threads atleast 5 to 8 threads per second > and the total number of threads is monotonically increasing. So just for > tweaking purpose I changed my window interval to 1min (60000ms) and batch > interval of 10s (10000) this looked lot better but still not ideal at > very least it is not monotonic anymore (It goes up and down). Now my > question really is how do I tune such that my number of threads are > optimal while satisfying the window Interval of 1 minute (60000ms) and > batch interval of 1s (1000) ? > > This jstack dump is taken after running my spark driver program for 2 mins > and there are about 1000 threads. > > Thanks! >
