Oh, the query just stopped showing up in the profiles webui, completely gone like it never happened. Seems to be responding a bit better, the sqlline is still hung though.
(Yes this is all related to my CTAS of the parquet data, at this point I am just looking for ways to handle the data and not make drill really unhappy. ) On Tue, May 31, 2016 at 10:51 AM, John Omernik <j...@omernik.com> wrote: > Also: Doing a CTAS using the new reader and dictionary encoding is > producing this, everything is hung at this point. The query in sqlline is > not returning, the web UI is running extremely slowly, and when it does > return, shows the running query, however, when I click on it, the profile > shows an error saying profile not found. The Full GCs are happening quite > a bit, and take a long time (>10 seconds) And (this is my tailed gcclog, > it's actually writing part of the the "allocation error" message and then > waits a before anything else happens. This is "the scary" state my cluster > can get into, and I am trying to avoid this :) Any tips on what may be > happening here would be appreciated. > > (24 GB of Heap, 5 nodes at this point) > > > > > > 912.895: [Full GC (Allocation Failure) 23G->20G(24G), 11.7923015 secs] > > 2924.692: [GC concurrent-mark-abort] > > 2925.099: [GC pause (G1 Evacuation Pause) (young) 22G->21G(24G), 0.0540177 > secs] > > 2925.401: [GC pause (G1 Evacuation Pause) (young) (initial-mark) > 22G->21G(24G), 0.0638409 secs] > > 2925.465: [GC concurrent-root-region-scan-start] > > 2925.475: [GC concurrent-root-region-scan-end, 0.0097528 secs] > > 2925.475: [GC concurrent-mark-start] > > 2925.846: [GC pause (G1 Evacuation Pause) (young) 22G->21G(24G), 0.0454322 > secs] > > 2926.252: [GC pause (G1 Evacuation Pause) (young) 22G->21G(24G), 0.0543209 > secs] > > 2926.604: [GC pause (G1 Evacuation Pause) (young) 22G->21G(24G), 0.0525408 > secs] > > 2926.986: [GC pause (G1 Evacuation Pause) (young) 22G->21G(24G), 0.0534530 > secs] > > 2927.389: [GC concurrent-mark-end, 1.9133249 secs] > > 2927.405: [GC remark, 0.0446448 secs] > > 2927.462: [GC cleanup 22G->22G(24G), 0.0290235 secs] > > 2927.494: [GC concurrent-cleanup-start] > > 2927.494: [GC concurrent-cleanup-end, 0.0000190 secs] > > 2927.530: [GC pause (G1 Evacuation Pause) (young) 22G->21G(24G), 0.0500267 > secs] > > 2927.828: [GC pause (G1 Evacuation Pause) (young) 22G->21G(24G), 0.0462845 > secs] > > 2928.184: [GC pause (G1 Evacuation Pause) (young) (initial-mark) > 22G->21G(24G), 0.0749704 secs] > > 2928.259: [GC concurrent-root-region-scan-start] > > 2928.268: [GC concurrent-root-region-scan-end, 0.0093531 secs] > > 2928.268: [GC concurrent-mark-start] > > 2928.568: [GC pause (G1 Evacuation Pause) (young) 22G->22G(24G), 0.0555025 > secs] > > 2928.952: [GC pause (G1 Evacuation Pause) (young) 23G->22G(24G), 0.0489993 > secs] > > 2929.333: [GC pause (G1 Evacuation Pause) (young)-- 23G->22G(24G), > 0.0676159 secs] > > 2929.693: [GC pause (G1 Evacuation Pause) (young)-- 23G->23G(24G), > 0.2088768 secs] > > 2929.914: [Full GC (Allocation Failure) 23G->20G(24G), 11.6264600 secs] > > 2941.544: [GC concurrent-mark-abort] > > 2941.836: [GC pause (G1 Evacuation Pause) (young) 22G->20G(24G), 0.0416962 > secs] > > 2942.127: [GC pause (G1 Evacuation Pause) (young) (initial-mark) > 22G->21G(24G), 0.0627406 secs] > > 2942.190: [GC concurrent-root-region-scan-start] > > 2942.193: [GC concurrent-root-region-scan-end, 0.0029795 secs] > > 2942.193: [GC concurrent-mark-start] > > 2942.548: [GC pause (G1 Evacuation Pause) (young) 22G->21G(24G), 0.0591030 > secs] > > 2942.934: [GC pause (G1 Evacuation Pause) (young) 22G->21G(24G), 0.0589163 > secs] > > 2943.304: [GC pause (G1 Evacuation Pause) (young) 22G->21G(24G), 0.0459117 > secs] > > 2943.743: [GC pause (G1 Evacuation Pause) (young) 22G->21G(24G), 0.0461640 > secs] > > 2943.941: [GC concurrent-mark-end, 1.7476855 secs] > > 2943.953: [GC remark, 0.0356995 secs] > > 2944.000: [GC cleanup 22G->22G(24G), 0.0307393 secs] > > 2944.034: [GC concurrent-cleanup-start] > > 2944.034: [GC concurrent-cleanup-end, 0.0000281 secs] > > 2944.162: [GC pause (G1 Evacuation Pause) (young) 22G->21G(24G), 0.0558067 > secs] > > 2944.510: [GC pause (G1 Evacuation Pause) (young) 22G->21G(24G), 0.0497960 > secs] > > 2944.837: [GC pause (G1 Evacuation Pause) (young) (initial-mark) > 22G->21G(24G), 0.0719856 secs] > > 2944.909: [GC concurrent-root-region-scan-start] > > 2944.917: [GC concurrent-root-region-scan-end, 0.0076375 secs] > > 2944.917: [GC concurrent-mark-start] > > 2945.204: [GC pause (G1 Evacuation Pause) (young) 22G->21G(24G), 0.0476954 > secs] > > 2945.604: [GC pause (G1 Evacuation Pause) (young) 22G->21G(24G), 0.0438138 > secs] > > 2945.940: [GC pause (G1 Evacuation Pause) (young) 23G->22G(24G), 0.0554554 > secs] > > 2946.358: [GC pause (G1 Evacuation Pause) (young) 23G->22G(24G), 0.0502923 > secs] > > 2946.708: [GC pause (G1 Evacuation Pause) (young)-- 23G->22G(24G), > 0.0728342 secs] > > 2947.021: [GC pause (G1 Evacuation Pause) (young)-- 23G->23G(24G), > 0.1938188 secs] > > 2947.227: [Full GC (Allocation Failure) > > On Tue, May 31, 2016 at 10:20 AM, John Omernik <j...@omernik.com> wrote: > >> I am looking at the GC logs for some big queries (now that I know how to >> enabled them, thanks Paul!) and found the item below, it worries, "it >> says, failure, it too 8 seconds) Should I be worried about that? I know my >> HEAP is set fairly high (24GB) how should I interpret this? >> >> John >> >> >> 1924.138: [GC concurrent-root-region-scan-end, 0.0019536 secs] >> >> 1924.138: [GC concurrent-mark-start] >> >> 1924.505: [GC pause (G1 Evacuation Pause) (young) 22G->21G(24G), >> 0.0314046 secs] >> >> 1924.853: [GC pause (G1 Evacuation Pause) (young) 22G->21G(24G), >> 0.0466492 secs] >> >> 1925.110: [GC concurrent-mark-end, 0.9724279 secs] >> >> 1925.126: [GC remark, 0.0403063 secs] >> >> 1925.182: [GC cleanup 22G->22G(24G), 0.0117316 secs] >> >> 1925.306: [GC pause (G1 Evacuation Pause) (young) 22G->21G(24G), >> 0.0434484 secs] >> >> 1925.623: [GC pause (G1 Evacuation Pause) (mixed)-- 22G->23G(24G), >> 0.2346237 secs] >> >> 1926.001: [GC pause (G1 Evacuation Pause) (mixed)-- 23G->23G(24G), >> 0.6835194 secs] >> >> 1926.702: [GC pause (G1 Evacuation Pause) (young) 23G->23G(24G), >> 0.0388452 secs] >> >> 1926.756: [GC pause (G1 Evacuation Pause) (young) (initial-mark) >> 23G->23G(24G), 0.0265543 secs] >> >> 1926.783: [GC concurrent-root-region-scan-start] >> >> 1926.783: [GC concurrent-root-region-scan-end, 0.0000162 secs] >> >> 1926.783: [GC concurrent-mark-start] >> >> 1926.798: [GC pause (G1 Evacuation Pause) (young) 23G->23G(24G), >> 0.0397901 secs] >> >> 1926.850: [GC pause (G1 Evacuation Pause) (young) 23G->23G(24G), >> 0.0370946 secs] >> >> 1926.902: [Full GC (Allocation Failure) 23G->18G(24G), 8.3348025 secs] >> >> 1935.243: [GC concurrent-mark-abort] >> >> 1935.967: [GC pause (G1 Evacuation Pause) (young) 20G->18G(24G), >> 0.0479378 secs] >> >> 1936.733: [GC pause (G1 Evacuation Pause) (young) (initial-mark) >> 20G->18G(24G), 0.0556650 secs] >> >> 1936.789: [GC concurrent-root-region-scan-start] >> >> 1936.794: [GC concurrent-root-region-scan-end, 0.0049816 secs] >> >> 1936.794: [GC concurrent-mark-start] >> >> 1937.526: [GC pause (G1 Evacuation Pause) (young) 20G->18G(24G), >> 0.0528627 secs] >> >> 1937.793: [GC concurrent-mark-end, 0.9993197 secs] >> >> 1937.811: [GC remark, 0.0503934 secs] >> >> 1937.878: [GC cleanup 19G->19G(24G), 0.0124116 secs] >> >> 1938.225: [GC pause (G1 Evacuation Pause) (young) 20G->18G(24G), >> 0.0500627 secs] >> >> 1938.827: [GC pause (G1 Evacuation Pause) (young) 20G->18G(24G), >> 0.0434986 secs] >> >> 1939.352: [GC pause (G1 Evacuation Pause) (young) (initial-mark) >> 20G->18G(24G), 0.0595028 secs] >> >> 1939.412: [GC concurrent-root-region-scan-start] >> >> 1939.415: [GC concurrent-root-region-scan-end, 0.0038086 secs] >> >> 1939.415: [GC concurrent-mark-start] >> >> 1939.911: [GC pause (G1 Evacuation Pause) (young) 20G->19G(24G), >> 0.0394493 secs] >> > >
