I am having problems with a SolrCloud where both nodes become unresponsive. I
am wondering whether there is some sort of memory leak. Attached is a portion
of the solr_gc.log from around the time that the problem starts. Have you any
suggestions how to diagnose and address this issue?
Thanks
2019-07-17T04:24:35.810-0400: 65828.261: Total time for which application
threads were stopped: 0.8963997 seconds, Stopping threads took: 0.8958437
seconds
2019-07-17T04:24:42.059-0400: 65834.510: [GC (CMS Initial Mark) [1
CMS-initial-mark: 20963525K(35389440K)] 24965285K(45219840K), 0.2193216 secs]
[Times: user=0.61 sys=0.27, real=0.22 secs]
2019-07-17T04:24:42.279-0400: 65834.730: Total time for which application
threads were stopped: 3.4630816 seconds, Stopping threads took: 3.2432089
seconds
2019-07-17T04:24:42.279-0400: 65834.730: [CMS-concurrent-mark-start]
2019-07-17T04:24:43.281-0400: 65835.732: Total time for which application
threads were stopped: 0.0005248 seconds, Stopping threads took: 0.0001216
seconds
2019-07-17T04:24:43.282-0400: 65835.733: [CMS-concurrent-mark: 1.002/1.003
secs] [Times: user=5.18 sys=0.19, real=1.01 secs]
2019-07-17T04:24:43.282-0400: 65835.733: [CMS-concurrent-preclean-start]
2019-07-17T04:24:43.377-0400: 65835.829: [CMS-concurrent-preclean: 0.096/0.096
secs] [Times: user=0.14 sys=0.05, real=0.09 secs]
2019-07-17T04:24:43.378-0400: 65835.829:
[CMS-concurrent-abortable-preclean-start]
2019-07-17T04:24:44.282-0400: 65836.733: Total time for which application
threads were stopped: 0.0005190 seconds, Stopping threads took: 0.0001509
seconds
2019-07-17T04:24:46.282-0400: 65838.733: Total time for which application
threads were stopped: 0.0005607 seconds, Stopping threads took: 0.0001641
seconds
CMS: abort preclean due to time 2019-07-17T04:24:49.495-0400: 65841.946:
[CMS-concurrent-abortable-preclean: 5.343/6.117 secs] [Times: user=11.07
sys=0.79, real=6.12 secs]
2019-07-17T04:24:49.496-0400: 65841.947: [GC (CMS Final Remark) [YG occupancy:
4287360 K (9830400 K)]{Heap before GC invocations=4 (full 3):
par new generation total 9830400K, used 4287360K [0x00007fd060000000,
0x00007fd330000000, 0x00007fd330000000)
eden space 7864320K, 51% used [0x00007fd060000000, 0x00007fd1564c5bf8,
0x00007fd240000000)
from space 1966080K, 12% used [0x00007fd240000000, 0x00007fd24f61a718,
0x00007fd2b8000000)
to space 1966080K, 0% used [0x00007fd2b8000000, 0x00007fd2b8000000,
0x00007fd330000000)
concurrent mark-sweep generation total 35389440K, used 20963525K
[0x00007fd330000000, 0x00007fdba0000000, 0x00007fdba0000000)
Metaspace used 47785K, capacity 49718K, committed 49972K, reserved 51200K
2019-07-17T04:24:49.496-0400: 65841.947: [GC (CMS Final Remark)
2019-07-17T04:24:49.496-0400: 65841.947: [ParNew
Desired survivor size 1811939328 bytes, new threshold 8 (max 8)
- age 1: 305474592 bytes, 305474592 total
- age 2: 101513064 bytes, 406987656 total
- age 3: 118054640 bytes, 525042296 total
- age 4: 792 bytes, 525043088 total
- age 5: 12209536 bytes, 537252624 total
: 4287360K->543113K(9830400K), 0.7519541 secs] 25250886K->21506639K(45219840K),
0.7521144 secs] [Times: user=2.62 sys=0.39, real=0.75 secs]
Heap after GC invocations=5 (full 3):
par new generation total 9830400K, used 543113K [0x00007fd060000000,
0x00007fd330000000, 0x00007fd330000000)
eden space 7864320K, 0% used [0x00007fd060000000, 0x00007fd060000000,
0x00007fd240000000)
from space 1966080K, 27% used [0x00007fd2b8000000, 0x00007fd2d9262690,
0x00007fd330000000)
to space 1966080K, 0% used [0x00007fd240000000, 0x00007fd240000000,
0x00007fd2b8000000)
concurrent mark-sweep generation total 35389440K, used 20963525K
[0x00007fd330000000, 0x00007fdba0000000, 0x00007fdba0000000)
Metaspace used 47785K, capacity 49718K, committed 49972K, reserved 51200K
}
2019-07-17T04:24:50.248-0400: 65842.699: [Rescan (parallel) , 0.1138935
secs]2019-07-17T04:24:50.362-0400: 65842.813: [weak refs processing, 0.0007099
secs]2019-07-17T04:24:50.363-0400: 65842.814: [class unloading, 0.0343465
secs]2019-07-17T04:24:50.397-0400: 65842.848: [scrub symbol table, 0.0137971
secs]2019-07-17T04:24:50.411-0400: 65842.862: [scrub string table, 0.0009367
secs][1 CMS-remark: 20963525K(35389440K)] 21506639K(45219840K), 0.9220238 secs]
[Times: user=3.09 sys=0.42, real=0.92 secs]
2019-07-17T04:24:50.418-0400: 65842.869: Total time for which application
threads were stopped: 0.9225790 seconds, Stopping threads took: 0.0001272
seconds
2019-07-17T04:24:50.418-0400: 65842.869: [CMS-concurrent-sweep-start]
2019-07-17T04:24:50.418-0400: 65842.869: [CMS-concurrent-sweep: 0.000/0.000
secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
2019-07-17T04:24:50.418-0400: 65842.869: [CMS-concurrent-reset-start]
2019-07-17T04:24:50.663-0400: 65843.114: [CMS-concurrent-reset: 0.245/0.245
secs] [Times: user=0.33 sys=0.16, real=0.25 secs]
2019-07-17T04:24:52.663-0400: 65845.114: [GC (CMS Initial Mark) [1
CMS-initial-mark: 20963525K(35389440K)] 21834541K(45219840K), 0.0556868 secs]
[Times: user=0.20 sys=0.00, real=0.05 secs]
2019-07-17T04:24:52.719-0400: 65845.170: Total time for which application
threads were stopped: 0.0564015 seconds, Stopping threads took: 0.0001193
seconds
2019-07-17T04:24:52.719-0400: 65845.170: [CMS-concurrent-mark-start]
2019-07-17T04:24:54.744-0400: 65847.195: [CMS-concurrent-mark: 2.025/2.025
secs] [Times: user=9.16 sys=0.79, real=2.03 secs]
2019-07-17T04:24:54.744-0400: 65847.196: [CMS-concurrent-preclean-start]
2019-07-17T04:24:55.292-0400: 65847.744: [CMS-concurrent-preclean: 0.520/0.548
secs] [Times: user=0.98 sys=0.11, real=0.55 secs]
2019-07-17T04:24:55.292-0400: 65847.744:
[CMS-concurrent-abortable-preclean-start]
CMS: abort preclean due to time 2019-07-17T04:25:01.914-0400: 65854.365:
[CMS-concurrent-abortable-preclean: 6.546/6.622 secs] [Times: user=12.57
sys=0.75, real=6.62 secs]
2019-07-17T04:25:01.916-0400: 65854.367: [GC (CMS Final Remark) [YG occupancy:
2987215 K (9830400 K)]{Heap before GC invocations=5 (full 4):
par new generation total 9830400K, used 2987215K [0x00007fd060000000,
0x00007fd330000000, 0x00007fd330000000)
eden space 7864320K, 31% used [0x00007fd060000000, 0x00007fd0f52d1728,
0x00007fd240000000)
from space 1966080K, 27% used [0x00007fd2b8000000, 0x00007fd2d9262690,
0x00007fd330000000)
to space 1966080K, 0% used [0x00007fd240000000, 0x00007fd240000000,
0x00007fd2b8000000)
concurrent mark-sweep generation total 35389440K, used 20963525K
[0x00007fd330000000, 0x00007fdba0000000, 0x00007fdba0000000)
Metaspace used 47785K, capacity 49718K, committed 49972K, reserved 51200K
2019-07-17T04:25:01.916-0400: 65854.367: [GC (CMS Final Remark)
2019-07-17T04:25:01.916-0400: 65854.367: [ParNew
Desired survivor size 1811939328 bytes, new threshold 3 (max 8)
- age 1: 1555619728 bytes, 1555619728 total
- age 2: 227157032 bytes, 1782776760 total
- age 3: 101487192 bytes, 1884263952 total
- age 4: 117294712 bytes, 2001558664 total
- age 5: 792 bytes, 2001559456 total
- age 6: 11707352 bytes, 2013266808 total
: 2987215K->1966080K(9830400K), 7.9346367 secs]
23950741K->23611379K(45219840K), 7.9349270 secs] [Times: user=27.01 sys=2.58,
real=7.93 secs]
Heap after GC invocations=6 (full 4):
par new generation total 9830400K, used 1966080K [0x00007fd060000000,
0x00007fd330000000, 0x00007fd330000000)
eden space 7864320K, 0% used [0x00007fd060000000, 0x00007fd060000000,
0x00007fd240000000)
from space 1966080K, 100% used [0x00007fd240000000, 0x00007fd2b8000000,
0x00007fd2b8000000)
to space 1966080K, 0% used [0x00007fd2b8000000, 0x00007fd2b8000000,
0x00007fd330000000)
concurrent mark-sweep generation total 35389440K, used 21645299K
[0x00007fd330000000, 0x00007fdba0000000, 0x00007fdba0000000)
Metaspace used 47785K, capacity 49718K, committed 49972K, reserved 51200K
}
2019-07-17T04:25:09.851-0400: 65862.302: [Rescan (parallel) , 0.9303801
secs]2019-07-17T04:25:10.781-0400: 65863.233: [weak refs processing, 0.0006290
secs]2019-07-17T04:25:10.782-0400: 65863.233: [class unloading, 0.0384533
secs]2019-07-17T04:25:10.821-0400: 65863.272: [scrub symbol table, 0.0144739
secs]2019-07-17T04:25:10.835-0400: 65863.286: [scrub string table, 0.0011079
secs][1 CMS-remark: 21645299K(35389440K)] 23611379K(45219840K), 8.9261811 secs]
[Times: user=29.89 sys=3.49, real=8.93 secs]
2019-07-17T04:25:10.842-0400: 65863.293: Total time for which application
threads were stopped: 8.9277070 seconds, Stopping threads took: 0.0003798
seconds
2019-07-17T04:25:10.842-0400: 65863.294: [CMS-concurrent-sweep-start]
2019-07-17T04:25:10.907-0400: 65863.359: Total time for which application
threads were stopped: 0.0007783 seconds, Stopping threads took: 0.0001437
seconds
2019-07-17T04:25:10.912-0400: 65863.363: Total time for which application
threads were stopped: 0.0004003 seconds, Stopping threads took: 0.0001217
seconds
2019-07-17T04:25:10.945-0400: 65863.396: Total time for which application
threads were stopped: 0.0007363 seconds, Stopping threads took: 0.0002805
seconds
2019-07-17T04:25:10.946-0400: 65863.397: Total time for which application
threads were stopped: 0.0005372 seconds, Stopping threads took: 0.0002592
seconds
2019-07-17T04:25:10.946-0400: 65863.398: Total time for which application
threads were stopped: 0.0003554 seconds, Stopping threads took: 0.0001091
seconds
2019-07-17T04:25:10.947-0400: 65863.398: Total time for which application
threads were stopped: 0.0003623 seconds, Stopping threads took: 0.0000771
seconds
2019-07-17T04:25:10.949-0400: 65863.401: Total time for which application
threads were stopped: 0.0003844 seconds, Stopping threads took: 0.0001492
seconds
2019-07-17T04:25:10.952-0400: 65863.403: Total time for which application
threads were stopped: 0.0020621 seconds, Stopping threads took: 0.0001047
seconds
2019-07-17T04:25:10.953-0400: 65863.404: Total time for which application
threads were stopped: 0.0003829 seconds, Stopping threads took: 0.0001492
seconds
2019-07-17T04:25:11.154-0400: 65863.606: Total time for which application
threads were stopped: 0.0005713 seconds, Stopping threads took: 0.0001479
seconds
2019-07-17T04:25:11.163-0400: 65863.614: Total time for which application
threads were stopped: 0.0004437 seconds, Stopping threads took: 0.0000840
seconds
2019-07-17T04:25:11.163-0400: 65863.615: Total time for which application
threads were stopped: 0.0002879 seconds, Stopping threads took: 0.0000768
seconds
2019-07-17T04:25:11.164-0400: 65863.615: Total time for which application
threads were stopped: 0.0002805 seconds, Stopping threads took: 0.0000717
seconds
2019-07-17T04:25:11.182-0400: 65863.633: [CMS-concurrent-sweep: 0.331/0.339
secs] [Times: user=0.55 sys=0.19, real=0.34 secs]
2019-07-17T04:25:11.182-0400: 65863.633: [CMS-concurrent-reset-start]
2019-07-17T04:25:11.425-0400: 65863.876: [CMS-concurrent-reset: 0.243/0.243
secs] [Times: user=0.26 sys=0.22, real=0.24 secs]
2019-07-17T04:25:12.165-0400: 65864.616: Total time for which application
threads were stopped: 0.0006721 seconds, Stopping threads took: 0.0001776
seconds
2019-07-17T04:25:13.426-0400: 65865.877: [GC (CMS Initial Mark) [1
CMS-initial-mark: 21645299K(35389440K)] 24654256K(45219840K), 0.1905752 secs]
[Times: user=0.73 sys=0.01, real=0.19 secs]
2019-07-17T04:25:13.616-0400: 65866.067: Total time for which application
threads were stopped: 0.1913003 seconds, Stopping threads took: 0.0001714
seconds
2019-07-17T04:25:13.617-0400: 65866.068: [CMS-concurrent-mark-start]
2019-07-17T04:25:22.233-0400: 65874.684: [CMS-concurrent-mark: 8.617/8.617
secs] [Times: user=38.97 sys=3.22, real=8.62 secs]
2019-07-17T04:25:22.233-0400: 65874.685: [CMS-concurrent-preclean-start]
2019-07-17T04:25:23.957-0400: 65876.408: [CMS-concurrent-preclean: 1.665/1.723
secs] [Times: user=2.56 sys=0.89, real=1.72 secs]
2019-07-17T04:25:23.957-0400: 65876.408:
[CMS-concurrent-abortable-preclean-start]
CMS: abort preclean due to time 2019-07-17T04:25:35.298-0400: 65887.749:
[CMS-concurrent-abortable-preclean: 11.311/11.341 secs] [Times: user=19.98
sys=2.80, real=11.34 secs]
2019-07-17T04:25:35.300-0400: 65887.751: [GC (CMS Final Remark) [YG occupancy:
6895549 K (9830400 K)]{Heap before GC invocations=6 (full 5):
par new generation total 9830400K, used 6895549K [0x00007fd060000000,
0x00007fd330000000, 0x00007fd330000000)
eden space 7864320K, 62% used [0x00007fd060000000, 0x00007fd18cdef700,
0x00007fd240000000)
from space 1966080K, 100% used [0x00007fd240000000, 0x00007fd2b8000000,
0x00007fd2b8000000)
to space 1966080K, 0% used [0x00007fd2b8000000, 0x00007fd2b8000000,
0x00007fd330000000)
concurrent mark-sweep generation total 35389440K, used 21645299K
[0x00007fd330000000, 0x00007fdba0000000, 0x00007fdba0000000)
Metaspace used 47790K, capacity 49718K, committed 49972K, reserved 51200K
2019-07-17T04:25:35.300-0400: 65887.752: [GC (CMS Final Remark)
2019-07-17T04:25:35.301-0400: 65887.752: [ParNew
Desired survivor size 1811939328 bytes, new threshold 2 (max 8)
- age 1: 1300738672 bytes, 1300738672 total
- age 2: 618382040 bytes, 1919120712 total
- age 3: 94144976 bytes, 2013265688 total
: 6895549K->1966080K(9830400K), 15.4072264 secs]
28540849K->27679974K(45219840K), 15.4076337 secs] [Times: user=53.46 sys=4.54,
real=15.41 secs]
Heap after GC invocations=7 (full 5):
par new generation total 9830400K, used 1966080K [0x00007fd060000000,
0x00007fd330000000, 0x00007fd330000000)
eden space 7864320K, 0% used [0x00007fd060000000, 0x00007fd060000000,
0x00007fd240000000)
from space 1966080K, 100% used [0x00007fd2b8000000, 0x00007fd330000000,
0x00007fd330000000)
to space 1966080K, 0% used [0x00007fd240000000, 0x00007fd240000000,
0x00007fd2b8000000)
concurrent mark-sweep generation total 35389440K, used 25713894K
[0x00007fd330000000, 0x00007fdba0000000, 0x00007fdba0000000)
Metaspace used 47790K, capacity 49718K, committed 49972K, reserved 51200K
}
2019-07-17T04:25:50.708-0400: 65903.159: [Rescan (parallel) , 1.8441958
secs]2019-07-17T04:25:52.552-0400: 65905.004: [weak refs processing, 0.0007312
secs]2019-07-17T04:25:52.553-0400: 65905.004: [class unloading, 0.0405475
secs]2019-07-17T04:25:52.594-0400: 65905.045: [scrub symbol table, 0.0138251
secs]2019-07-17T04:25:52.608-0400: 65905.059: [scrub string table, 0.0013725
secs][1 CMS-remark: 25713894K(35389440K)] 27679974K(45219840K), 17.3145086
secs] [Times: user=59.43 sys=6.01, real=17.32 secs]
2019-07-17T04:25:52.615-0400: 65905.066: Total time for which application
threads were stopped: 17.3165459 seconds, Stopping threads took: 0.0005692
seconds
2019-07-17T04:25:52.615-0400: 65905.066: [CMS-concurrent-sweep-start]
2019-07-17T04:25:52.732-0400: 65905.183: Total time for which application
threads were stopped: 0.0007164 seconds, Stopping threads took: 0.0001596
seconds
2019-07-17T04:25:52.891-0400: 65905.342: Total time for which application
threads were stopped: 0.0006033 seconds, Stopping threads took: 0.0001392
seconds
2019-07-17T04:25:52.891-0400: 65905.343: Total time for which application
threads were stopped: 0.0003773 seconds, Stopping threads took: 0.0000899
seconds
2019-07-17T04:25:52.892-0400: 65905.343: Total time for which application
threads were stopped: 0.0003796 seconds, Stopping threads took: 0.0000785
seconds
2019-07-17T04:25:52.900-0400: 65905.351: Total time for which application
threads were stopped: 0.0003977 seconds, Stopping threads took: 0.0000936
seconds
2019-07-17T04:25:53.138-0400: 65905.589: Total time for which application
threads were stopped: 0.0007846 seconds, Stopping threads took: 0.0002831
seconds
2019-07-17T04:25:53.139-0400: 65905.590: Total time for which application
threads were stopped: 0.0008578 seconds, Stopping threads took: 0.0005474
seconds
2019-07-17T04:25:53.139-0400: 65905.590: Total time for which application
threads were stopped: 0.0004470 seconds, Stopping threads took: 0.0001674
seconds
2019-07-17T04:25:53.148-0400: 65905.599: Total time for which application
threads were stopped: 0.0003956 seconds, Stopping threads took: 0.0000997
seconds
2019-07-17T04:25:53.154-0400: 65905.605: Total time for which application
threads were stopped: 0.0004002 seconds, Stopping threads took: 0.0000811
seconds
2019-07-17T04:25:54.040-0400: 65906.491: Total time for which application
threads were stopped: 0.0006663 seconds, Stopping threads took: 0.0001182
seconds
2019-07-17T04:25:54.490-0400: 65906.941: [CMS-concurrent-sweep: 1.868/1.874
secs] [Times: user=3.33 sys=0.55, real=1.87 secs]
2019-07-17T04:25:54.490-0400: 65906.941: [CMS-concurrent-reset-start]
2019-07-17T04:25:54.673-0400: 65907.124: [CMS-concurrent-reset: 0.183/0.183
secs] [Times: user=0.24 sys=0.13, real=0.19 secs]
2019-07-17T04:25:55.041-0400: 65907.492: Total time for which application
threads were stopped: 0.0006208 seconds, Stopping threads took: 0.0001262
seconds
2019-07-17T04:25:55.044-0400: 65907.495: Total time for which application
threads were stopped: 0.0003771 seconds, Stopping threads took: 0.0000761
seconds
2019-07-17T04:25:55.052-0400: 65907.503: Total time for which application
threads were stopped: 0.0004139 seconds, Stopping threads took: 0.0000973
seconds
2019-07-17T04:25:55.053-0400: 65907.504: Total time for which application
threads were stopped: 0.0003032 seconds, Stopping threads took: 0.0000557
seconds
2019-07-17T04:25:56.674-0400: 65909.125: [GC (CMS Initial Mark) [1
CMS-initial-mark: 25713894K(35389440K)] 29027537K(45219840K), 0.7663142 secs]
[Times: user=1.22 sys=1.35, real=0.76 secs]
2019-07-17T04:25:57.440-0400: 65909.891: Total time for which application
threads were stopped: 0.7670981 seconds, Stopping threads took: 0.0001151
seconds
2019-07-17T04:25:57.440-0400: 65909.891: [CMS-concurrent-mark-start]
2019-07-17T04:26:04.442-0400: 65916.893: Total time for which application
threads were stopped: 0.0005925 seconds, Stopping threads took: 0.0001420
seconds
2019-07-17T04:26:07.449-0400: 65919.901: Total time for which application
threads were stopped: 0.0006222 seconds, Stopping threads took: 0.0001309
seconds
2019-07-17T04:26:07.459-0400: 65919.910: Total time for which application
threads were stopped: 0.0004247 seconds, Stopping threads took: 0.0001262
seconds
2019-07-17T04:26:07.461-0400: 65919.912: Total time for which application
threads were stopped: 0.0004319 seconds, Stopping threads took: 0.0001575
seconds
2019-07-17T04:26:07.468-0400: 65919.919: Total time for which application
threads were stopped: 0.0004331 seconds, Stopping threads took: 0.0001005
seconds
2019-07-17T04:26:08.046-0400: 65920.497: Total time for which application
threads were stopped: 0.0020486 seconds, Stopping threads took: 0.0005648
seconds
2019-07-17T04:26:08.072-0400: 65920.523: Total time for which application
threads were stopped: 0.0063436 seconds, Stopping threads took: 0.0000911
seconds
2019-07-17T04:26:08.076-0400: 65920.527: Total time for which application
threads were stopped: 0.0004344 seconds, Stopping threads took: 0.0000712
seconds
2019-07-17T04:26:08.080-0400: 65920.531: Total time for which application
threads were stopped: 0.0003917 seconds, Stopping threads took: 0.0000889
seconds
2019-07-17T04:26:08.471-0400: 65920.922: Total time for which application
threads were stopped: 0.0006511 seconds, Stopping threads took: 0.0001257
seconds
2019-07-17T04:26:08.483-0400: 65920.934: Total time for which application
threads were stopped: 0.0003696 seconds, Stopping threads took: 0.0000889
seconds
2019-07-17T04:26:08.498-0400: 65920.949: Total time for which application
threads were stopped: 0.0004476 seconds, Stopping threads took: 0.0001055
seconds
2019-07-17T04:26:08.499-0400: 65920.950: Total time for which application
threads were stopped: 0.0003499 seconds, Stopping threads took: 0.0000707
seconds
2019-07-17T04:26:08.503-0400: 65920.954: Total time for which application
threads were stopped: 0.0004135 seconds, Stopping threads took: 0.0001103
seconds
2019-07-17T04:26:08.514-0400: 65920.966: Total time for which application
threads were stopped: 0.0003443 seconds, Stopping threads took: 0.0000619
seconds
2019-07-17T04:26:09.526-0400: 65921.977: Total time for which application
threads were stopped: 0.0006916 seconds, Stopping threads took: 0.0001273
seconds
2019-07-17T04:26:12.048-0400: 65924.499: Total time for which application
threads were stopped: 0.0005852 seconds, Stopping threads took: 0.0001273
seconds
2019-07-17T04:26:12.211-0400: 65924.662: Total time for which application
threads were stopped: 0.0006580 seconds, Stopping threads took: 0.0001196
seconds
2019-07-17T04:26:12.211-0400: 65924.663: Total time for which application
threads were stopped: 0.0004730 seconds, Stopping threads took: 0.0001387
seconds
2019-07-17T04:26:12.219-0400: 65924.670: Total time for which application
threads were stopped: 0.0003358 seconds, Stopping threads took: 0.0000745
seconds
2019-07-17T04:26:12.219-0400: 65924.670: Total time for which application
threads were stopped: 0.0003246 seconds, Stopping threads took: 0.0000685
seconds
2019-07-17T04:26:21.221-0400: 65933.672: Total time for which application
threads were stopped: 0.0005698 seconds, Stopping threads took: 0.0001292
seconds
{Heap before GC invocations=7 (full 6):
par new generation total 9830400K, used 9830400K [0x00007fd060000000,
0x00007fd330000000, 0x00007fd330000000)
eden space 7864320K, 100% used [0x00007fd060000000, 0x00007fd240000000,
0x00007fd240000000)
from space 1966080K, 100% used [0x00007fd2b8000000, 0x00007fd330000000,
0x00007fd330000000)
to space 1966080K, 0% used [0x00007fd240000000, 0x00007fd240000000,
0x00007fd2b8000000)
concurrent mark-sweep generation total 35389440K, used 25713894K
[0x00007fd330000000, 0x00007fdba0000000, 0x00007fdba0000000)
Metaspace used 48066K, capacity 50103K, committed 50228K, reserved 51200K
2019-07-17T04:26:25.643-0400: 65938.094: [GC (Allocation Failure)
2019-07-17T04:26:25.644-0400: 65938.095: [ParNew
Desired survivor size 1811939328 bytes, new threshold 2 (max 8)
- age 1: 1190351856 bytes, 1190351856 total
- age 2: 822913904 bytes, 2013265760 total
: 9830400K->1966080K(9830400K), 24.6565675 secs]
35544294K->34913549K(45219840K), 24.6569958 secs] [Times: user=84.84 sys=7.83,
real=24.65 secs]
Heap after GC invocations=8 (full 6):
par new generation total 9830400K, used 1966080K [0x00007fd060000000,
0x00007fd330000000, 0x00007fd330000000)
eden space 7864320K, 0% used [0x00007fd060000000, 0x00007fd060000000,
0x00007fd240000000)
from space 1966080K, 100% used [0x00007fd240000000, 0x00007fd2b8000000,
0x00007fd2b8000000)
to space 1966080K, 0% used [0x00007fd2b8000000, 0x00007fd2b8000000,
0x00007fd330000000)
concurrent mark-sweep generation total 35389440K, used 32947469K
[0x00007fd330000000, 0x00007fdba0000000, 0x00007fdba0000000)
Metaspace used 48066K, capacity 50103K, committed 50228K, reserved 51200K
}
2019-07-17T04:26:50.301-0400: 65962.752: Total time for which application
threads were stopped: 24.6590132 seconds, Stopping threads took: 0.0004347
seconds
2019-07-17T04:26:50.309-0400: 65962.760: Total time for which application
threads were stopped: 0.0018485 seconds, Stopping threads took: 0.0002773
seconds
2019-07-17T04:26:50.411-0400: 65962.862: Total time for which application
threads were stopped: 0.0004705 seconds, Stopping threads took: 0.0001434
seconds
2019-07-17T04:26:51.146-0400: 65963.597: Total time for which application
threads were stopped: 0.0006941 seconds, Stopping threads took: 0.0001609
seconds
2019-07-17T04:26:51.147-0400: 65963.598: Total time for which application
threads were stopped: 0.0003938 seconds, Stopping threads took: 0.0000903
seconds
2019-07-17T04:26:51.148-0400: 65963.599: Total time for which application
threads were stopped: 0.0003504 seconds, Stopping threads took: 0.0000820
seconds
2019-07-17T04:26:51.157-0400: 65963.608: Total time for which application
threads were stopped: 0.0004219 seconds, Stopping threads took: 0.0000957
seconds
2019-07-17T04:26:51.289-0400: 65963.740: Total time for which application
threads were stopped: 0.0007436 seconds, Stopping threads took: 0.0001343
seconds
2019-07-17T04:26:51.289-0400: 65963.740: Total time for which application
threads were stopped: 0.0004040 seconds, Stopping threads took: 0.0000866
seconds
2019-07-17T04:26:51.291-0400: 65963.742: Total time for which application
threads were stopped: 0.0006039 seconds, Stopping threads took: 0.0002549
seconds
2019-07-17T04:26:51.295-0400: 65963.746: Total time for which application
threads were stopped: 0.0004503 seconds, Stopping threads took: 0.0001076
seconds
2019-07-17T04:26:51.296-0400: 65963.747: Total time for which application
threads were stopped: 0.0003773 seconds, Stopping threads took: 0.0000924
seconds
2019-07-17T04:26:51.299-0400: 65963.751: Total time for which application
threads were stopped: 0.0004197 seconds, Stopping threads took: 0.0000807
seconds
2019-07-17T04:26:51.369-0400: 65963.820: Total time for which application
threads were stopped: 0.0004646 seconds, Stopping threads took: 0.0000890
seconds
2019-07-17T04:26:51.375-0400: 65963.826: Total time for which application
threads were stopped: 0.0003935 seconds, Stopping threads took: 0.0000910
seconds
2019-07-17T04:26:52.370-0400: 65964.821: Total time for which application
threads were stopped: 0.0006283 seconds, Stopping threads took: 0.0001303
seconds
2019-07-17T04:26:52.371-0400: 65964.822: Total time for which application
threads were stopped: 0.0003737 seconds, Stopping threads took: 0.0000804
seconds
2019-07-17T04:26:52.383-0400: 65964.835: Total time for which application
threads were stopped: 0.0004302 seconds, Stopping threads took: 0.0001088
seconds
2019-07-17T04:26:52.384-0400: 65964.835: Total time for which application
threads were stopped: 0.0003855 seconds, Stopping threads took: 0.0001013
seconds
2019-07-17T04:27:00.387-0400: 65972.838: Total time for which application
threads were stopped: 0.0019082 seconds, Stopping threads took: 0.0005338
seconds
2019-07-17T04:27:02.389-0400: 65974.840: Total time for which application
threads were stopped: 0.0006313 seconds, Stopping threads took: 0.0001304
seconds
2019-07-17T04:27:08.391-0400: 65980.842: Total time for which application
threads were stopped: 0.0007376 seconds, Stopping threads took: 0.0001527
seconds
2019-07-17T04:27:16.395-0400: 65988.846: Total time for which application
threads were stopped: 0.0016547 seconds, Stopping threads took: 0.0003892
seconds
2019-07-17T04:27:16.442-0400: 65988.893: Total time for which application
threads were stopped: 0.0004927 seconds, Stopping threads took: 0.0001326
seconds
2019-07-17T04:27:16.452-0400: 65988.903: Total time for which application
threads were stopped: 0.0004287 seconds, Stopping threads took: 0.0000822
seconds
2019-07-17T04:27:16.503-0400: 65988.954: [CMS-concurrent-mark: 54.353/79.063
secs] [Times: user=323.52 sys=33.04, real=79.07 secs]
2019-07-17T04:27:16.503-0400: 65988.954: [CMS-concurrent-preclean-start]
2019-07-17T04:27:17.135-0400: 65989.586: Total time for which application
threads were stopped: 0.0006349 seconds, Stopping threads took: 0.0001279
seconds
2019-07-17T04:27:17.146-0400: 65989.597: Total time for which application
threads were stopped: 0.0005024 seconds, Stopping threads took: 0.0001007
seconds
2019-07-17T04:27:19.147-0400: 65991.598: Total time for which application
threads were stopped: 0.0006171 seconds, Stopping threads took: 0.0001226
seconds
{Heap before GC invocations=8 (full 6):
par new generation total 9830400K, used 9830400K [0x00007fd060000000,
0x00007fd330000000, 0x00007fd330000000)
eden space 7864320K, 100% used [0x00007fd060000000, 0x00007fd240000000,
0x00007fd240000000)
from space 1966080K, 100% used [0x00007fd240000000, 0x00007fd2b8000000,
0x00007fd2b8000000)
to space 1966080K, 0% used [0x00007fd2b8000000, 0x00007fd2b8000000,
0x00007fd330000000)
concurrent mark-sweep generation total 35389440K, used 32947469K
[0x00007fd330000000, 0x00007fdba0000000, 0x00007fdba0000000)
Metaspace used 48080K, capacity 50103K, committed 50228K, reserved 51200K
2019-07-17T04:27:20.340-0400: 65992.791: [GC (Allocation Failure)
2019-07-17T04:27:20.340-0400: 65992.791: [ParNew: 9830400K->9830400K(9830400K),
0.0000497 secs]2019-07-17T04:27:20.340-0400: 65992.791:
[CMS2019-07-17T04:27:29.047-0400: 66001.498: [CMS-concurrent-preclean:
12.517/12.544 secs] [Times: user=14.71 sys=1.75, real=12.54 secs]
(concurrent mode failure): 32947469K->35389439K(35389440K), 66.3560524 secs]
42777869K->42476768K(45219840K), [Metaspace: 48080K->48080K(51200K)],
66.3566319 secs] [Times: user=53.36 sys=13.07, real=66.36 secs]
Heap after GC invocations=9 (full 7):
par new generation total 9830400K, used 7087328K [0x00007fd060000000,
0x00007fd330000000, 0x00007fd330000000)
eden space 7864320K, 90% used [0x00007fd060000000, 0x00007fd210938158,
0x00007fd240000000)
from space 1966080K, 0% used [0x00007fd240000000, 0x00007fd240000000,
0x00007fd2b8000000)
to space 1966080K, 0% used [0x00007fd2b8000000, 0x00007fd2b8000000,
0x00007fd330000000)
concurrent mark-sweep generation total 35389440K, used 35389439K
[0x00007fd330000000, 0x00007fdba0000000, 0x00007fdba0000000)
Metaspace used 47971K, capacity 49913K, committed 50228K, reserved 51200K
}
2019-07-17T04:28:26.697-0400: 66059.148: Total time for which application
threads were stopped: 66.3574720 seconds, Stopping threads took: 0.0001053
seconds
{Heap before GC invocations=9 (full 7):
par new generation total 9830400K, used 7408920K [0x00007fd060000000,
0x00007fd330000000, 0x00007fd330000000)
eden space 7864320K, 94% used [0x00007fd060000000, 0x00007fd224346348,
0x00007fd240000000)
from space 1966080K, 0% used [0x00007fd240000000, 0x00007fd240000000,
0x00007fd2b8000000)
to space 1966080K, 0% used [0x00007fd2b8000000, 0x00007fd2b8000000,
0x00007fd330000000)
concurrent mark-sweep generation total 35389440K, used 35389439K
[0x00007fd330000000, 0x00007fdba0000000, 0x00007fdba0000000)
Metaspace used 47972K, capacity 49913K, committed 50228K, reserved 51200K
2019-07-17T04:28:26.707-0400: 66059.158: [Full GC (Allocation Failure)
2019-07-17T04:28:26.707-0400: 66059.158: [CMS: 35389439K->35389440K(35389440K),
42.8817886 secs] 42798360K->42477782K(45219840K), [Metaspace:
47972K->47972K(51200K)], 42.8819764 secs] [Times: user=38.80 sys=4.12,
real=42.88 secs]
Heap after GC invocations=10 (full 8):
par new generation total 9830400K, used 7088342K [0x00007fd060000000,
0x00007fd330000000, 0x00007fd330000000)
eden space 7864320K, 90% used [0x00007fd060000000, 0x00007fd210a35b00,
0x00007fd240000000)
from space 1966080K, 0% used [0x00007fd240000000, 0x00007fd240000000,
0x00007fd2b8000000)
to space 1966080K, 0% used [0x00007fd2b8000000, 0x00007fd2b8000000,
0x00007fd330000000)
concurrent mark-sweep generation total 35389440K, used 35389440K
[0x00007fd330000000, 0x00007fdba0000000, 0x00007fdba0000000)
Metaspace used 47972K, capacity 49913K, committed 50228K, reserved 51200K
}
{Heap before GC invocations=10 (full 8):
par new generation total 9830400K, used 7088342K [0x00007fd060000000,
0x00007fd330000000, 0x00007fd330000000)
eden space 7864320K, 90% used [0x00007fd060000000, 0x00007fd210a35b00,
0x00007fd240000000)
from space 1966080K, 0% used [0x00007fd240000000, 0x00007fd240000000,
0x00007fd2b8000000)
to space 1966080K, 0% used [0x00007fd2b8000000, 0x00007fd2b8000000,
0x00007fd330000000)
concurrent mark-sweep generation total 35389440K, used 35389440K
[0x00007fd330000000, 0x00007fdba0000000, 0x00007fdba0000000)
Metaspace used 47972K, capacity 49913K, committed 50228K, reserved 51200K
2019-07-17T04:29:09.589-0400: 66102.040: [Full GC (Allocation Failure)
2019-07-17T04:29:09.589-0400: 66102.040: [CMS: 35389440K->35389439K(35389440K),
41.6544542 secs] 42477782K->42477709K(45219840K), [Metaspace:
47972K->47972K(51200K)], 41.6546671 secs] [Times: user=38.60 sys=3.10,
real=41.66 secs]
Heap after GC invocations=11 (full 9):
par new generation total 9830400K, used 7088269K [0x00007fd060000000,
0x00007fd330000000, 0x00007fd330000000)
eden space 7864320K, 90% used [0x00007fd060000000, 0x00007fd210a23628,
0x00007fd240000000)
from space 1966080K, 0% used [0x00007fd240000000, 0x00007fd240000000,
0x00007fd2b8000000)
to space 1966080K, 0% used [0x00007fd2b8000000, 0x00007fd2b8000000,
0x00007fd330000000)
concurrent mark-sweep generation total 35389440K, used 35389439K
[0x00007fd330000000, 0x00007fdba0000000, 0x00007fdba0000000)
Metaspace used 47968K, capacity 49908K, committed 50228K, reserved 51200K
}
2019-07-17T04:29:51.244-0400: 66143.695: Total time for which application
threads were stopped: 84.5378246 seconds, Stopping threads took: 0.0001610
seconds
2019-07-17T04:29:51.245-0400: 66143.696: [GC (CMS Initial Mark) [1
CMS-initial-mark: 35389439K(35389440K)] 42478093K(45219840K), 1.6201370 secs]
[Times: user=1.62 sys=0.02, real=1.62 secs]
2019-07-17T04:29:52.865-0400: 66145.316: Total time for which application
threads were stopped: 1.6210197 seconds, Stopping threads took: 0.0002394
seconds
2019-07-17T04:29:52.865-0400: 66145.316: [CMS-concurrent-mark-start]
2019-07-17T04:29:52.899-0400: 66145.350: Total time for which application
threads were stopped: 0.0048659 seconds, Stopping threads took: 0.0002143
seconds
2019-07-17T04:29:53.075-0400: 66145.526: Total time for which application
threads were stopped: 0.0006478 seconds, Stopping threads took: 0.0001208
seconds
2019-07-17T04:29:53.076-0400: 66145.527: Total time for which application
threads were stopped: 0.0003673 seconds, Stopping threads took: 0.0000694
seconds
2019-07-17T04:29:53.076-0400: 66145.527: Total time for which application
threads were stopped: 0.0003728 seconds, Stopping threads took: 0.0000673
seconds
2019-07-17T04:29:53.739-0400: 66146.190: Total time for which application
threads were stopped: 0.0006386 seconds, Stopping threads took: 0.0001145
seconds
2019-07-17T04:29:53.740-0400: 66146.191: Total time for which application
threads were stopped: 0.0004395 seconds, Stopping threads took: 0.0001023
seconds
2019-07-17T04:29:53.742-0400: 66146.193: Total time for which application
threads were stopped: 0.0003994 seconds, Stopping threads took: 0.0000747
seconds
2019-07-17T04:29:53.742-0400: 66146.193: Total time for which application
threads were stopped: 0.0003744 seconds, Stopping threads took: 0.0000562
seconds
2019-07-17T04:29:53.755-0400: 66146.206: Total time for which application
threads were stopped: 0.0004387 seconds, Stopping threads took: 0.0000813
seconds
2019-07-17T04:29:53.803-0400: 66146.254: Total time for which application
threads were stopped: 0.0019793 seconds, Stopping threads took: 0.0001046
seconds
2019-07-17T04:29:54.801-0400: 66147.252: Total time for which application
threads were stopped: 0.0007200 seconds, Stopping threads took: 0.0001496
seconds
2019-07-17T04:29:54.801-0400: 66147.252: Total time for which application
threads were stopped: 0.0004148 seconds, Stopping threads took: 0.0000645
seconds
2019-07-17T04:29:54.808-0400: 66147.259: Total time for which application
threads were stopped: 0.0004059 seconds, Stopping threads took: 0.0000907
seconds
2019-07-17T04:29:54.808-0400: 66147.260: Total time for which application
threads were stopped: 0.0003804 seconds, Stopping threads took: 0.0000857
seconds
2019-07-17T04:30:04.811-0400: 66157.262: Total time for which application
threads were stopped: 0.0006678 seconds, Stopping threads took: 0.0001312
seconds
2019-07-17T04:30:05.813-0400: 66158.265: Total time for which application
threads were stopped: 0.0006312 seconds, Stopping threads took: 0.0001155
seconds
2019-07-17T04:30:06.815-0400: 66159.266: Total time for which application
threads were stopped: 0.0007393 seconds, Stopping threads took: 0.0001385
seconds
2019-07-17T04:30:09.821-0400: 66162.272: Total time for which application
threads were stopped: 0.0021837 seconds, Stopping threads took: 0.0005936
seconds
2019-07-17T04:30:11.824-0400: 66164.275: Total time for which application
threads were stopped: 0.0006556 seconds, Stopping threads took: 0.0001196
seconds
2019-07-17T04:30:14.826-0400: 66167.278: Total time for which application
threads were stopped: 0.0007517 seconds, Stopping threads took: 0.0001096
seconds
2019-07-17T04:30:14.841-0400: 66167.292: Total time for which application
threads were stopped: 0.0004859 seconds, Stopping threads took: 0.0000609
seconds
2019-07-17T04:30:14.841-0400: 66167.292: Total time for which application
threads were stopped: 0.0004192 seconds, Stopping threads took: 0.0000510
seconds
2019-07-17T04:30:14.844-0400: 66167.295: Total time for which application
threads were stopped: 0.0005006 seconds, Stopping threads took: 0.0001103
seconds
2019-07-17T04:30:14.845-0400: 66167.296: Total time for which application
threads were stopped: 0.0003899 seconds, Stopping threads took: 0.0000696
seconds
2019-07-17T04:30:17.846-0400: 66170.297: Total time for which application
threads were stopped: 0.0006552 seconds, Stopping threads took: 0.0001180
seconds
{Heap before GC invocations=11 (full 10):
par new generation total 9830400K, used 9830399K [0x00007fd060000000,
0x00007fd330000000, 0x00007fd330000000)
eden space 7864320K, 100% used [0x00007fd060000000, 0x00007fd240000000,
0x00007fd240000000)
from space 1966080K, 99% used [0x00007fd240000000, 0x00007fd2b7ffffe0,
0x00007fd2b8000000)
to space 1966080K, 0% used [0x00007fd2b8000000, 0x00007fd2b8000000,
0x00007fd330000000)
concurrent mark-sweep generation total 35389440K, used 35389439K
[0x00007fd330000000, 0x00007fdba0000000, 0x00007fdba0000000)
Metaspace used 47988K, capacity 49933K, committed 50228K, reserved 51200K
2019-07-17T04:30:21.483-0400: 66173.934: [Full GC (Allocation Failure)
2019-07-17T04:30:21.483-0400: 66173.934: [CMS2019-07-17T04:32:31.747-0400:
66304.198: [CMS-concurrent-mark: 158.853/158.882 secs] [Times: user=567.15
sys=57.88, real=158.88 secs]
(concurrent mode failure): 35389439K->35389440K(35389440K), 190.5823226 secs]
45219839K->45002736K(45219840K), [Metaspace: 47988K->47988K(51200K)],
190.5827140 secs] [Times: user=483.80 sys=60.36, real=190.58 secs]
Heap after GC invocations=12 (full 11):
par new generation total 9830400K, used 9613296K [0x00007fd060000000,
0x00007fd330000000, 0x00007fd330000000)
eden space 7864320K, 100% used [0x00007fd060000000, 0x00007fd240000000,
0x00007fd240000000)
from space 1966080K, 88% used [0x00007fd240000000, 0x00007fd2aabfc350,
0x00007fd2b8000000)
to space 1966080K, 0% used [0x00007fd2b8000000, 0x00007fd2b8000000,
0x00007fd330000000)
concurrent mark-sweep generation total 35389440K, used 35389440K
[0x00007fd330000000, 0x00007fdba0000000, 0x00007fdba0000000)
Metaspace used 47988K, capacity 49933K, committed 50228K, reserved 51200K
}
2019-07-17T04:33:32.066-0400: 66364.517: Total time for which application
threads were stopped: 190.5837101 seconds, Stopping threads took: 0.0001034
seconds
2019-07-17T04:33:32.174-0400: 66364.626: Total time for which application
threads were stopped: 0.0010834 seconds, Stopping threads took: 0.0001768
seconds
2019-07-17T04:33:32.175-0400: 66364.626: Total time for which application
threads were stopped: 0.0005792 seconds, Stopping threads took: 0.0001218
seconds
2019-07-17T04:33:32.176-0400: 66364.627: Total time for which application
threads were stopped: 0.0005754 seconds, Stopping threads took: 0.0001473
seconds
2019-07-17T04:33:32.369-0400: 66364.820: Total time for which application
threads were stopped: 0.0007392 seconds, Stopping threads took: 0.0001103
seconds
2019-07-17T04:33:32.370-0400: 66364.821: Total time for which application
threads were stopped: 0.0004083 seconds, Stopping threads took: 0.0000604
seconds
2019-07-17T04:33:32.370-0400: 66364.821: Total time for which application
threads were stopped: 0.0003862 seconds, Stopping threads took: 0.0000472
seconds
2019-07-17T04:33:32.409-0400: 66364.860: Total time for which application
threads were stopped: 0.0006280 seconds, Stopping threads took: 0.0001734
seconds
2019-07-17T04:33:32.412-0400: 66364.863: Total time for which application
threads were stopped: 0.0005459 seconds, Stopping threads took: 0.0001032
seconds
2019-07-17T04:33:32.412-0400: 66364.864: Total time for which application
threads were stopped: 0.0005235 seconds, Stopping threads took: 0.0001132
seconds
2019-07-17T04:33:32.417-0400: 66364.868: Total time for which application
threads were stopped: 0.0005104 seconds, Stopping threads took: 0.0000669
seconds
2019-07-17T04:33:32.418-0400: 66364.869: Total time for which application
threads were stopped: 0.0004730 seconds, Stopping threads took: 0.0000751
seconds
2019-07-17T04:33:32.418-0400: 66364.870: Total time for which application
threads were stopped: 0.0004380 seconds, Stopping threads took: 0.0000628
seconds
2019-07-17T04:33:32.433-0400: 66364.884: Total time for which application
threads were stopped: 0.0004782 seconds, Stopping threads took: 0.0000772
seconds
2019-07-17T04:33:32.493-0400: 66364.944: Total time for which application
threads were stopped: 0.0005892 seconds, Stopping threads took: 0.0001184
seconds
2019-07-17T04:33:32.494-0400: 66364.945: Total time for which application
threads were stopped: 0.0006326 seconds, Stopping threads took: 0.0001670
seconds
2019-07-17T04:33:32.499-0400: 66364.950: Total time for which application
threads were stopped: 0.0006094 seconds, Stopping threads took: 0.0001385
seconds
2019-07-17T04:33:32.501-0400: 66364.952: Total time for which application
threads were stopped: 0.0005162 seconds, Stopping threads took: 0.0000712
seconds
2019-07-17T04:33:32.665-0400: 66365.116: Total time for which application
threads were stopped: 0.0008147 seconds, Stopping threads took: 0.0001013
seconds
2019-07-17T04:33:33.666-0400: 66366.117: Total time for which application
threads were stopped: 0.0008408 seconds, Stopping threads took: 0.0001141
seconds
2019-07-17T04:33:34.067-0400: 66366.518: [GC (CMS Initial Mark) [1
CMS-initial-mark: 35389440K(35389440K)] 45181209K(45219840K), 2.3252878 secs]
[Times: user=2.94 sys=0.02, real=2.33 secs]
2019-07-17T04:33:36.392-0400: 66368.843: Total time for which application
threads were stopped: 2.3261517 seconds, Stopping threads took: 0.0000560
seconds
2019-07-17T04:33:36.392-0400: 66368.843: [CMS-concurrent-mark-start]
2019-07-17T04:33:36.415-0400: 66368.866: Total time for which application
threads were stopped: 0.0053783 seconds, Stopping threads took: 0.0001528
seconds
2019-07-17T04:33:36.431-0400: 66368.882: Total time for which application
threads were stopped: 0.0057907 seconds, Stopping threads took: 0.0001685
seconds
{Heap before GC invocations=12 (full 12):
par new generation total 9830400K, used 9830399K [0x00007fd060000000,
0x00007fd330000000, 0x00007fd330000000)
eden space 7864320K, 100% used [0x00007fd060000000, 0x00007fd240000000,
0x00007fd240000000)
from space 1966080K, 99% used [0x00007fd240000000, 0x00007fd2b7ffffe8,
0x00007fd2b8000000)
to space 1966080K, 0% used [0x00007fd2b8000000, 0x00007fd2b8000000,
0x00007fd330000000)
concurrent mark-sweep generation total 35389440K, used 35389440K
[0x00007fd330000000, 0x00007fdba0000000, 0x00007fdba0000000)
Metaspace used 48443K, capacity 50618K, committed 50740K, reserved 51200K
2019-07-17T04:33:36.886-0400: 66369.337: [Full GC (Allocation Failure)
2019-07-17T04:33:36.886-0400: 66369.337: [CMS2019-07-17T04:36:28.607-0400:
66541.058: [CMS-concurrent-mark: 172.199/172.215 secs] [Times: user=585.22
sys=57.30, real=172.21 secs]
(concurrent mode failure): 35389440K->35389439K(35389440K), 229.6309350 secs]
45219839K->45207154K(45219840K), [Metaspace: 48443K->48443K(51200K)],
229.6312205 secs] [Times: user=632.19 sys=65.62, real=229.63 secs]
Heap after GC invocations=13 (full 13):
par new generation total 9830400K, used 9817714K [0x00007fd060000000,
0x00007fd330000000, 0x00007fd330000000)
eden space 7864320K, 99% used [0x00007fd060000000, 0x00007fd23fffffe8,
0x00007fd240000000)
from space 1966080K, 99% used [0x00007fd240000000, 0x00007fd2b739cb88,
0x00007fd2b8000000)
to space 1966080K, 0% used [0x00007fd2b8000000, 0x00007fd2b8000000,
0x00007fd330000000)
concurrent mark-sweep generation total 35389440K, used 35389439K
[0x00007fd330000000, 0x00007fdba0000000, 0x00007fdba0000000)
Metaspace used 48436K, capacity 50608K, committed 50740K, reserved 51200K
}
2019-07-17T04:37:26.518-0400: 66598.969: Total time for which application
threads were stopped: 229.6325102 seconds, Stopping threads took: 0.0001308
seconds
2019-07-17T04:37:26.559-0400: 66599.010: Total time for which application
threads were stopped: 0.0048519 seconds, Stopping threads took: 0.0001304
seconds
2019-07-17T04:37:26.575-0400: 66599.026: Total time for which application
threads were stopped: 0.0042055 seconds, Stopping threads took: 0.0001027
seconds
{Heap before GC invocations=13 (full 13):
par new generation total 9830400K, used 9830389K [0x00007fd060000000,
0x00007fd330000000, 0x00007fd330000000)
eden space 7864320K, 100% used [0x00007fd060000000, 0x00007fd240000000,
0x00007fd240000000)
from space 1966080K, 99% used [0x00007fd240000000, 0x00007fd2b7ffd658,
0x00007fd2b8000000)
to space 1966080K, 0% used [0x00007fd2b8000000, 0x00007fd2b8000000,
0x00007fd330000000)
concurrent mark-sweep generation total 35389440K, used 35389439K
[0x00007fd330000000, 0x00007fdba0000000, 0x00007fdba0000000)
Metaspace used 48451K, capacity 50682K, committed 50740K, reserved 51200K
2019-07-17T04:37:26.616-0400: 66599.067: [Full GC (Allocation Failure)
2019-07-17T04:37:26.616-0400: 66599.067: [CMS: 35389439K->35389440K(35389440K),
47.8298671 secs] 45219829K->45208059K(45219840K), [Metaspace:
48451K->48451K(51200K)], 47.8300960 secs] [Times: user=43.40 sys=4.48,
real=47.83 secs]
Heap after GC invocations=14 (full 14):
par new generation total 9830400K, used 9818619K [0x00007fd060000000,
0x00007fd330000000, 0x00007fd330000000)
eden space 7864320K, 99% used [0x00007fd060000000, 0x00007fd23fffffe8,
0x00007fd240000000)
from space 1966080K, 99% used [0x00007fd240000000, 0x00007fd2b747ecb8,
0x00007fd2b8000000)
to space 1966080K, 0% used [0x00007fd2b8000000, 0x00007fd2b8000000,
0x00007fd330000000)
concurrent mark-sweep generation total 35389440K, used 35389440K
[0x00007fd330000000, 0x00007fdba0000000, 0x00007fdba0000000)
Metaspace used 48451K, capacity 50682K, committed 50740K, reserved 51200K
}
2019-07-17T04:38:14.446-0400: 66646.897: Total time for which application
threads were stopped: 47.8312247 seconds, Stopping threads took: 0.0000784
seconds
