VitoMakarevich commented on PR #42893: URL: https://github.com/apache/spark/pull/42893#issuecomment-1718054124
It's hard to say they were from netty, let me describe more about the facts and assumptions I made. I checked heapdump and found that: 1. There were hundred of millions of entries to finalizer queue/`java.util.zip.*` entries. 2. It's impossible to find where they come from once they GCed(so moved to finalizer queue and the only reference to `java.util.zip.*` package is coming from the finalizer queue. 3. When I turned off spark UI - the issue disappeared. I was playing with settings like number of tasks/stages/jobs and so on, but it didn't help. But what I found is that there were at least a couple of thousands of references to `java.util.zip.*` from `netty` - that's why I assume finally finalizer queue is fed by `netty` - but I may be wrong here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
