Since it's a stop the world activity, it's supposed to finish as soon as possible. it's parallel marking phase, so 23 threads (which is picked by JVM) should be fine. I don't have any other apps running on this machine. Spark is working perfectly fine except for one job, The eden size (8G) is also reasonable given that this host has 160G, it's actually much lower than normal time. While running that job, JVM shrinks the size of eden, that's why it's using relatively smaller eden size.
But the unique thing about this is that it's taking quite a long time processing in Ext Root Scanning, which has something to do with native space. And from the thread dumps, it's running generated code (codegen from query) while showing this symptom. I'll continue my investigation ... Thanks -Kohki On Mon, Nov 15, 2021 at 8:27 AM Sean Owen <sro...@gmail.com> wrote: > I'm not a GC tuning expert, but 23 GC threads sounds like a lot (?). Turn > that down maybe. > What are your actual heap region sizes, from -verbose:gc? > This kind of sounds like the young generation is too ... large I think? > not sure. > > None of this is particularly Spark related, but maybe some ideas will > surface. > Of course, reducing memory allocation in your app if possible always helps. > > > On Mon, Nov 15, 2021 at 10:18 AM Kohki Nishio <tarop...@gmail.com> wrote: > >> it's a VM, but it has 16 cores and 32 processors. >> >> -Kohki >> >> On Mon, Nov 15, 2021 at 12:53 AM Deepak Goel <deic...@gmail.com> wrote: >> >>> How many 'hardware threads' do you have? >>> >>> >>> Deepak >>> "The greatness of a nation can be judged by the way its animals are >>> treated - Mahatma Gandhi" >>> >>> +91 73500 12833 >>> deic...@gmail.com >>> >>> Facebook: https://www.facebook.com/deicool >>> LinkedIn: www.linkedin.com/in/deicool >>> >>> "Plant a Tree, Go Green" >>> >>> Make In India : http://www.makeinindia.com/home >>> >>> >>> On Mon, Nov 15, 2021 at 11:02 AM Kohki Nishio <tarop...@gmail.com> >>> wrote: >>> >>>> Hello, I'm seeing continuous long pauses (>5s) while running a job in >>>> Spark (3.1.2). I'm using G1GC but every GC(young) takes around 5 seconds >>>> and it happens continuously, anybody has seen anything like that ? Thanks >>>> >>>> ---- >>>> 2021-11-14T05:03:11.084+0000: 372441.741: [GC pause (G1 Evacuation >>>> Pause) (young), 5.7183644 secs] >>>> [Parallel Time: 5692.3 ms, GC Workers: 23] >>>> [GC Worker Start (ms): Min: 372441741.8, Avg: 372441742.2, Max: >>>> 372441742.5, Diff: 0.7] >>>> [Ext Root Scanning (ms): Min: 4109.6, Avg: 4854.2, Max: 5691.1, >>>> Diff: 1581.5, Sum: 111646.0] >>>> [Update RS (ms): Min: 0.0, Avg: 0.6, Max: 13.4, Diff: 13.4, Sum: >>>> 13.4] >>>> [Processed Buffers: Min: 0, Avg: 9.9, Max: 227, Diff: 227, >>>> Sum: 227] >>>> [Scan RS (ms): Min: 0.1, Avg: 0.1, Max: 0.2, Diff: 0.1, Sum: 1.8] >>>> [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: >>>> 0.0, Sum: 0.0] >>>> [Object Copy (ms): Min: 0.0, Avg: 2.4, Max: 39.7, Diff: 39.6, >>>> Sum: 54.1] >>>> [Termination (ms): Min: 0.0, Avg: 833.8, Max: 1528.3, Diff: >>>> 1528.3, Sum: 19176.7] >>>> [Termination Attempts: Min: 1, Avg: 7.1, Max: 14, Diff: 13, >>>> Sum: 164] >>>> [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.4, Diff: 0.4, >>>> Sum: 2.9] >>>> [GC Worker Total (ms): Min: 5690.7, Avg: 5691.1, Max: 5691.4, >>>> Diff: 0.7, Sum: 130894.9] >>>> [GC Worker End (ms): Min: 372447433.1, Avg: 372447433.3, Max: >>>> 372447433.5, Diff: 0.4] >>>> [Code Root Fixup: 0.1 ms] >>>> [Code Root Purge: 0.0 ms] >>>> [String Dedup Fixup: 1.5 ms, GC Workers: 23] >>>> [Queue Fixup (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: >>>> 0.0] >>>> [Table Fixup (ms): Min: 0.1, Avg: 0.5, Max: 0.7, Diff: 0.6, Sum: >>>> 10.5] >>>> [Clear CT: 1.4 ms] >>>> [Other: 23.0 ms] >>>> [Choose CSet: 0.0 ms] >>>> [Ref Proc: 16.9 ms] >>>> [Ref Enq: 1.6 ms] >>>> [Redirty Cards: 1.1 ms] >>>> [Humongous Register: 0.7 ms] >>>> [Humongous Reclaim: 0.3 ms] >>>> [Free CSet: 0.7 ms] >>>> [Eden: 8096.0M(8096.0M)->0.0B(8096.0M) Survivors: 96.0M->96.0M Heap: >>>> 23.3G(160.0G)->15.4G(160.0G)] >>>> [Times: user=23.46 sys=1.03, real=5.72 secs] >>>> >>>> >>>> -- >>>> Kohki Nishio >>>> >>> >> >> -- >> Kohki Nishio >> > -- Kohki Nishio
