Jacek Furmankiewicz created CASSANDRA-7361:
----------------------------------------------
Summary: Cassandra locks up in full GC, unable to free memory
Key: CASSANDRA-7361
URL: https://issues.apache.org/jira/browse/CASSANDRA-7361
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: Ubuntu, RedHat, JDK 1.7
Reporter: Jacek Furmankiewicz
Priority: Blocker
We have a long running batch load process, which runs for many hours.
Massive amount of writes, in large mutation batches (we increase the thrift
frame size to 45 MB).
Everything goes well, but after about 3 hrs of processing everything locks up.
We start getting NoHostsAvailable exceptions on the Java application side (with
Astyanax as our driver), eventually socket timeouts.
Looking at Cassandra, we can see that it is using nearly the full 8GB of heap
and unable to free it. It spends most of its time in full GC, but the amount of
memory does not go down.
Here is a long sample from jstat to show this over an extended time period
e.g.
{quote}
-bash-4.1$ $JAVA_HOME/bin/jstat -gc 3728 5s
S0C S1C S0U S1U EC EU OC OU PC PU
YGC YGCT FGC FGCT GCT
81920.0 81920.0 0.0 4097.0 655360.0 150419.0 7569408.0 7202990.3 43280.0
25855.8 5271 268.378 4940 9877.726 10146.104
81920.0 81920.0 0.0 4097.0 655360.0 150657.2 7569408.0 7202990.3 43280.0
25855.8 5271 268.378 4940 9877.726 10146.104
81920.0 81920.0 0.0 4097.0 655360.0 150708.5 7569408.0 7202990.3 43280.0
25855.8 5271 268.378 4941 9877.950 10146.328
81920.0 81920.0 0.0 4097.0 655360.0 150920.4 7569408.0 7202990.2 43280.0
25855.8 5271 268.378 4942 9878.085 10146.463
81920.0 81920.0 0.0 4097.0 655360.0 150930.7 7569408.0 7202990.2 43280.0
25855.8 5271 268.378 4942 9878.085 10146.463
81920.0 81920.0 0.0 4097.0 655360.0 150998.2 7569408.0 7202990.2 43280.0
25855.8 5271 268.378 4942 9878.085 10146.463
81920.0 81920.0 0.0 4097.0 655360.0 151335.6 7569408.0 7202990.2 43280.0
25855.8 5271 268.378 4943 9878.253 10146.630
81920.0 81920.0 0.0 4097.0 655360.0 151483.3 7569408.0 7202990.2 43280.0
25855.8 5271 268.378 4944 9878.389 10146.766
81920.0 81920.0 0.0 4097.0 655360.0 151750.4 7569408.0 7202990.2 43280.0
25855.8 5271 268.378 4944 9878.389 10146.766
81920.0 81920.0 0.0 4097.0 655360.0 151752.5 7569408.0 7202990.2 43280.0
25855.8 5271 268.378 4944 9878.389 10146.766
81920.0 81920.0 0.0 4097.0 655360.0 151997.4 7569408.0 7202990.2 43280.0
25855.8 5271 268.378 4945 9878.593 10146.970
81920.0 81920.0 0.0 4097.0 655360.0 152063.0 7569408.0 7202988.6 43280.0
25855.8 5271 268.378 4946 9878.732 10147.110
81920.0 81920.0 0.0 4097.0 655360.0 152256.2 7569408.0 7202988.6 43280.0
25855.8 5271 268.378 4946 9878.732 10147.110
81920.0 81920.0 0.0 4097.0 655360.0 152276.2 7569408.0 7202988.6 43280.0
25855.8 5271 268.378 4946 9878.732 10147.110
81920.0 81920.0 0.0 4097.0 655360.0 152276.2 7569408.0 7202988.6 43280.0
25855.8 5271 268.378 4947 9878.968 10147.345
81920.0 81920.0 0.0 4097.0 655360.0 152525.3 7569408.0 7202857.5 43280.0
25855.8 5271 268.378 4948 9879.107 10147.484
81920.0 81920.0 0.0 4097.0 655360.0 152673.0 7569408.0 7202857.5 43280.0
25855.8 5271 268.378 4948 9879.107 10147.484
81920.0 81920.0 0.0 4097.0 655360.0 152855.5 7569408.0 7202857.5 43280.0
25855.8 5271 268.378 4948 9879.107 10147.484
81920.0 81920.0 0.0 4097.0 655360.0 152871.3 7569408.0 7202857.5 43280.0
25855.8 5271 268.378 4949 9879.275 10147.653
81920.0 81920.0 0.0 4097.0 655360.0 153116.2 7569408.0 7202857.0 43280.0
25855.8 5271 268.378 4950 9879.415 10147.793
81920.0 81920.0 0.0 4097.0 655360.0 153191.6 7569408.0 7202857.0 43280.0
25855.8 5271 268.378 4950 9879.415 10147.793
81920.0 81920.0 0.0 4097.0 655360.0 153454.3 7569408.0 7202857.0 43280.0
25855.8 5271 268.378 4950 9879.415 10147.793
81920.0 81920.0 0.0 4097.0 655360.0 153528.0 7569408.0 7202857.0 43280.0
25855.8 5271 268.378 4951 9879.583 10147.961
81920.0 81920.0 0.0 4097.0 655360.0 153766.1 7569408.0 7202857.0 43280.0
25855.8 5271 268.378 4952 9879.721 10148.099
81920.0 81920.0 0.0 4097.0 655360.0 153791.9 7569408.0 7202857.0 43280.0
25855.8 5271 268.378 4952 9879.721 10148.099
81920.0 81920.0 0.0 4097.0 655360.0 153855.1 7569408.0 7202857.0 43280.0
25855.8 5271 268.378 4952 9879.721 10148.099
81920.0 81920.0 0.0 4097.0 655360.0 154114.8 7569408.0 7202857.0 43280.0
25855.8 5271 268.378 4953 9879.953 10148.331
81920.0 81920.0 0.0 4097.0 655360.0 154118.8 7569408.0 7202856.8 43280.0
25855.8 5271 268.378 4954 9880.097 10148.474
81920.0 81920.0 0.0 4097.0 655360.0 154382.7 7569408.0 7202856.8 43280.0
25855.8 5271 268.378 4954 9880.097 10148.474
81920.0 81920.0 0.0 4097.0 655360.0 154511.8 7569408.0 7202856.8 43280.0
25855.8 5271 268.378 4954 9880.097 10148.474
81920.0 81920.0 0.0 4097.0 655360.0 154699.2 7569408.0 7202856.8 43280.0
25855.8 5271 268.378 4955 9880.273 10148.650
81920.0 81920.0 0.0 4097.0 655360.0 154707.4 7569408.0 7202856.7 43280.0
25855.8 5271 268.378 4956 9880.273 10148.650
81920.0 81920.0 0.0 4097.0 655360.0 154985.2 7569408.0 7202856.7 43280.0
25855.8 5271 268.378 4956 9880.414 10148.792
81920.0 81920.0 0.0 4097.0 655360.0 155006.1 7569408.0 7202856.7 43280.0
25855.8 5271 268.378 4956 9880.414 10148.792
81920.0 81920.0 0.0 4097.0 655360.0 155050.8 7569408.0 7202856.7 43280.0
25855.8 5271 268.378 4957 9880.585 10148.963
81920.0 81920.0 0.0 4097.0 655360.0 155233.1 7569408.0 7202856.7 43280.0
25855.8 5271 268.378 4957 9880.585 10148.963
81920.0 81920.0 0.0 4097.0 655360.0 155536.7 7569408.0 7202856.7 43280.0
25855.8 5271 268.378 4958 9880.730 10149.108
81920.0 81920.0 0.0 4097.0 655360.0 156764.1 7569408.0 7202856.7 43280.0
25855.8 5271 268.378 4958 9880.730 10149.108
81920.0 81920.0 0.0 4097.0 655360.0 156829.3 7569408.0 7202856.7 43280.0
25855.8 5271 268.378 4959 9880.963 10149.341
81920.0 81920.0 0.0 4097.0 655360.0 157015.2 7569408.0 7202856.6 43280.0
25855.8 5271 268.378 4959 9880.963 10149.341
81920.0 81920.0 0.0 4097.0 655360.0 157089.1 7569408.0 7202856.6 43280.0
25855.8 5271 268.378 4960 9881.104 10149.481
81920.0 81920.0 0.0 4097.0 655360.0 157353.1 7569408.0 7202856.6 43280.0
25855.8 5271 268.378 4960 9881.104 10149.481
81920.0 81920.0 0.0 4097.0 655360.0 157368.8 7569408.0 7202856.6 43280.0
25855.8 5271 268.378 4961 9881.279 10149.656
81920.0 81920.0 0.0 4097.0 655360.0 157608.5 7569408.0 7202856.6 43280.0
25855.8 5271 268.378 4961 9881.279 10149.656
81920.0 81920.0 0.0 4097.0 655360.0 157684.2 7569408.0 7202856.6 43280.0
25855.8 5271 268.378 4962 9881.422 10149.800
81920.0 81920.0 0.0 4097.0 655360.0 157922.4 7569408.0 7202856.6 43280.0
25855.8 5271 268.378 4962 9881.422 10149.800
81920.0 81920.0 0.0 4097.0 655360.0 157959.9 7569408.0 7202856.6 43280.0
25855.8 5271 268.378 4963 9881.600 10149.977
81920.0 81920.0 0.0 4097.0 655360.0 158004.8 7569408.0 7202856.5 43280.0
25855.8 5271 268.378 4963 9881.600 10149.977
81920.0 81920.0 0.0 4097.0 655360.0 158197.6 7569408.0 7202856.5 43280.0
25855.8 5271 268.378 4964 9881.742 10150.120
81920.0 81920.0 0.0 4097.0 655360.0 158209.8 7569408.0 7202856.5 43280.0
25855.8 5271 268.378 4964 9881.742 10150.120
81920.0 81920.0 0.0 4097.0 655360.0 158473.8 7569408.0 7202856.5 43280.0
25855.8 5271 268.378 4965 9881.920 10150.297
81920.0 81920.0 0.0 4097.0 655360.0 158473.8 7569408.0 7202856.5 43280.0
25855.8 5271 268.378 4965 9881.920 10150.297
81920.0 81920.0 0.0 4097.0 655360.0 158769.7 7569408.0 7202856.5 43280.0
25855.8 5271 268.378 4966 9882.063 10150.441
81920.0 81920.0 0.0 4097.0 655360.0 158796.6 7569408.0 7202856.5 43280.0
25855.8 5271 268.378 4966 9882.063 10150.441
81920.0 81920.0 0.0 4097.0 655360.0 159076.4 7569408.0 7202856.5 43280.0
25855.8 5271 268.378 4966 9882.063 10150.441
81920.0 81920.0 0.0 4097.0 655360.0 159080.7 7569408.0 7202856.4 43280.0
25855.8 5271 268.378 4967 9882.250 10150.628
81920.0 81920.0 0.0 4097.0 655360.0 159190.8 7569408.0 7202856.4 43280.0
25855.8 5271 268.378 4968 9882.394 10150.772
81920.0 81920.0 0.0 4097.0 655360.0 159391.8 7569408.0 7202856.4 43280.0
25855.8 5271 268.378 4968 9882.394 10150.772
81920.0 81920.0 0.0 4097.0 655360.0 159697.5 7569408.0 7202856.4 43280.0
25855.8 5271 268.378 4968 9882.394 10150.772
81920.0 81920.0 0.0 4097.0 655360.0 159737.1 7569408.0 7202856.3 43280.0
25855.8 5271 268.378 4969 9882.636 10151.014
81920.0 81920.0 0.0 4097.0 655360.0 159737.1 7569408.0 7202856.3 43280.0
25855.8 5271 268.378 4970 9882.782 10151.159
81920.0 81920.0 0.0 4097.0 655360.0 159958.2 7569408.0 7202856.3 43280.0
25855.8 5271 268.378 4970 9882.782 10151.159
81920.0 81920.0 0.0 4097.0 655360.0 159985.0 7569408.0 7202856.3 43280.0
25855.8 5271 268.378 4970 9882.782 10151.159
81920.0 81920.0 0.0 4097.0 655360.0 160249.0 7569408.0 7202856.3 43280.0
25855.8 5271 268.378 4971 9882.974 10151.352
81920.0 81920.0 0.0 4097.0 655360.0 160249.0 7569408.0 7202856.3 43280.0
25855.8 5271 268.378 4972 9883.121 10151.499
81920.0 81920.0 0.0 4097.0 655360.0 160361.3 7569408.0 7202856.3 43280.0
25855.8 5271 268.378 4972 9883.121 10151.499
81920.0 81920.0 0.0 4097.0 655360.0 160638.2 7569408.0 7202856.3 43280.0
25855.8 5271 268.378 4972 9883.121 10151.499
81920.0 81920.0 0.0 4097.0 655360.0 160894.9 7569408.0 7202856.3 43280.0
25855.8 5271 268.378 4973 9883.368 10151.746
81920.0 81920.0 0.0 4097.0 655360.0 160920.7 7569408.0 7202856.3 43280.0
25855.8 5271 268.378 4974 9883.519 10151.897
81920.0 81920.0 0.0 4097.0 655360.0 160920.7 7569408.0 7202856.3 43280.0
25855.8 5271 268.378 4974 9883.519 10151.897
81920.0 81920.0 0.0 4097.0 655360.0 161139.5 7569408.0 7202856.3 43280.0
25855.8 5271 268.378 4975 9883.519 10151.897
81920.0 81920.0 0.0 4097.0 655360.0 161168.4 7569408.0 7202856.1 43280.0
25855.8 5271 268.378 4975 9883.735 10152.113
81920.0 81920.0 0.0 4097.0 655360.0 161427.2 7569408.0 7202856.1 43280.0
25855.8 5271 268.378 4976 9883.884 10152.262
81920.0 81920.0 0.0 4097.0 655360.0 161446.2 7569408.0 7202856.1 43280.0
25855.8 5271 268.378 4976 9883.884 10152.262
81920.0 81920.0 0.0 4097.0 655360.0 161490.9 7569408.0 7202856.1 43280.0
25855.8 5271 268.378 4976 9883.884 10152.262
81920.0 81920.0 0.0 4097.0 655360.0 161673.2 7569408.0 7202856.1 43280.0
25855.8 5271 268.378 4977 9884.066 10152.444
81920.0 81920.0 0.0 4097.0 655360.0 162002.3 7569408.0 7202856.1 43280.0
25855.8 5271 268.378 4978 9884.217 10152.595
81920.0 81920.0 0.0 4097.0 655360.0 162023.6 7569408.0 7202856.1 43280.0
25855.8 5271 268.378 4978 9884.217 10152.595
81920.0 81920.0 0.0 4097.0 655360.0 162086.0 7569408.0 7202856.1 43280.0
25855.8 5271 268.378 4978 9884.217 10152.595
81920.0 81920.0 0.0 4097.0 655360.0 162271.9 7569408.0 7202856.1 43280.0
25855.8 5271 268.378 4979 9884.405 10152.782
81920.0 81920.0 0.0 4097.0 655360.0 162537.0 7569408.0 7202856.1 43280.0
25855.8 5271 268.378 4980 9884.557 10152.935
81920.0 81920.0 0.0 4097.0 655360.0 162696.2 7569408.0 7202856.1 43280.0
25855.8 5271 268.378 4980 9884.557 10152.935
81920.0 81920.0 0.0 4097.0 655360.0 162696.2 7569408.0 7202856.1 43280.0
25855.8 5271 268.378 4980 9884.557 10152.935
81920.0 81920.0 0.0 4097.0 655360.0 162740.8 7569408.0 7202856.1 43280.0
25855.8 5271 268.378 4981 9884.748 10153.125
81920.0 81920.0 0.0 4097.0 655360.0 163009.5 7569408.0 7202856.1 43280.0
25855.8 5271 268.378 4982 9884.901 10153.279
81920.0 81920.0 0.0 4097.0 655360.0 163285.1 7569408.0 7202856.1 43280.0
25855.8 5271 268.378 4982 9884.901 10153.279
81920.0 81920.0 0.0 4097.0 655360.0 163285.1 7569408.0 7202856.1 43280.0
25855.8 5271 268.378 4982 9884.901 10153.279
81920.0 81920.0 0.0 4097.0 655360.0 163397.4 7569408.0 7202856.1 43280.0
25855.8 5271 268.378 4983 9885.093 10153.471
81920.0 81920.0 0.0 4097.0 655360.0 163590.2 7569408.0 7202855.9 43280.0
25855.8 5271 268.378 4984 9885.251 10153.628
81920.0 81920.0 0.0 4097.0 655360.0 165038.0 7569408.0 7202855.9 43280.0
25855.8 5271 268.378 4984 9885.251 10153.628
81920.0 81920.0 0.0 4097.0 655360.0 165142.5 7569408.0 7202855.9 43280.0
25855.8 5271 268.378 4984 9885.251 10153.628
81920.0 81920.0 0.0 4097.0 655360.0 165142.5 7569408.0 7202855.9 43280.0
25855.8 5271 268.378 4985 9885.435 10153.813
81920.0 81920.0 0.0 4097.0 655360.0 165391.7 7569408.0 7202855.8 43280.0
25855.8 5271 268.378 4986 9885.588 10153.966
81920.0 81920.0 0.0 4097.0 655360.0 165401.9 7569408.0 7202855.8 43280.0
25855.8 5271 268.378 4986 9885.588 10153.966
81920.0 81920.0 0.0 4097.0 655360.0 165665.9 7569408.0 7202855.8 43280.0
25855.8 5271 268.378 4986 9885.588 10153.966
81920.0 81920.0 0.0 4097.0 655360.0 165667.9 7569408.0 7202855.8 43280.0
25855.8 5271 268.378 4987 9885.848 10154.226
81920.0 81920.0 0.0 4097.0 655360.0 165712.6 7569408.0 7202855.8 43280.0
25855.8 5271 268.378 4988 9886.002 10154.379
81920.0 81920.0 0.0 4097.0 655360.0 165971.0 7569408.0 7202855.8 43280.0
25855.8 5271 268.378 4988 9886.002 10154.379
81920.0 81920.0 0.0 4097.0 655360.0 166254.9 7569408.0 7202855.8 43280.0
25855.8 5271 268.378 4988 9886.002 10154.379
81920.0 81920.0 0.0 4097.0 655360.0 166257.1 7569408.0 7202855.8 43280.0
25855.8 5271 268.378 4989 9886.256 10154.633
81920.0 81920.0 0.0 4097.0 655360.0 166257.1 7569408.0 7202855.7 43280.0
25855.8 5271 268.378 4989 9886.256 10154.633
81920.0 81920.0 0.0 4097.0 655360.0 166496.5 7569408.0 7202855.7 43280.0
25855.8 5271 268.378 4990 9886.412 10154.789
81920.0 81920.0 0.0 4097.0 655360.0 166843.1 7569408.0 7202855.7 43280.0
25855.8 5271 268.378 4990 9886.412 10154.789
81920.0 81920.0 0.0 4097.0 655360.0 166852.0 7569408.0 7202855.7 43280.0
25855.8 5271 268.378 4991 9886.605 10154.983
81920.0 81920.0 0.0 4097.0 655360.0 166852.0 7569408.0 7202855.6 43280.0
25855.8 5271 268.378 4991 9886.605 10154.983
81920.0 81920.0 0.0 4097.0 655360.0 166908.3 7569408.0 7202855.6 43280.0
25855.8 5271 268.378 4992 9886.762 10155.140
81920.0 81920.0 0.0 4097.0 655360.0 167122.1 7569408.0 7202855.6 43280.0
25855.8 5271 268.378 4992 9886.762 10155.140
81920.0 81920.0 0.0 4097.0 655360.0 167375.4 7569408.0 7202855.6 43280.0
25855.8 5271 268.378 4993 9887.023 10155.401
81920.0 81920.0 0.0 4097.0 655360.0 167375.4 7569408.0 7202855.6 43280.0
25855.8 5271 268.378 4993 9887.023 10155.401
81920.0 81920.0 0.0 4097.0 655360.0 167377.6 7569408.0 7202855.6 43280.0
25855.8 5271 268.378 4994 9887.179 10155.557
81920.0 81920.0 0.0 4097.0 655360.0 167683.0 7569408.0 7202855.6 43280.0
25855.8 5271 268.378 4994 9887.179 10155.557
81920.0 81920.0 0.0 4097.0 655360.0 167948.1 7569408.0 7202855.6 43280.0
25855.8 5271 268.378 4994 9887.179 10155.557
81920.0 81920.0 0.0 4097.0 655360.0 167968.7 7569408.0 7202855.6 43280.0
25855.8 5271 268.378 4995 9887.376 10155.754
81920.0 81920.0 0.0 4097.0 655360.0 168034.2 7569408.0 7202855.6 43280.0
25855.8 5271 268.378 4996 9887.534 10155.911
81920.0 81920.0 0.0 4097.0 655360.0 168094.6 7569408.0 7202855.6 43280.0
25855.8 5271 268.378 4996 9887.534 10155.911
81920.0 81920.0 0.0 4097.0 655360.0 168270.8 7569408.0 7202855.6 43280.0
25855.8 5271 268.378 4996 9887.534 10155.911
81920.0 81920.0 0.0 4097.0 655360.0 168561.5 7569408.0 7202855.6 43280.0
25855.8 5271 268.378 4997 9887.808 10156.185
81920.0 81920.0 0.0 4097.0 655360.0 168627.3 7569408.0 7202855.5 43280.0
25855.8 5271 268.378 4998 9887.968 10156.345
81920.0 81920.0 0.0 4097.0 655360.0 168641.1 7569408.0 7202855.5 43280.0
25855.8 5271 268.378 4998 9887.968 10156.345
81920.0 81920.0 0.0 4097.0 655360.0 168859.9 7569408.0 7202855.5 43280.0
25855.8 5271 268.378 4998 9887.968 10156.345
81920.0 81920.0 0.0 4097.0 655360.0 169143.9 7569408.0 7202855.5 43280.0
25855.8 5271 268.378 4999 9888.170 10156.547
81920.0 81920.0 0.0 4097.0 655360.0 169152.9 7569408.0 7202855.4 43280.0
25855.8 5271 268.378 5000 9888.330 10156.708
81920.0 81920.0 0.0 4097.0 655360.0 169152.9 7569408.0 7202855.4 43280.0
25855.8 5271 268.378 5000 9888.330 10156.708
81920.0 81920.0 0.0 4097.0 655360.0 169373.9 7569408.0 7202855.4 43280.0
25855.8 5271 268.378 5000 9888.330 10156.708
81920.0 81920.0 0.0 4097.0 655360.0 169639.0 7569408.0 7202855.4 43280.0
25855.8 5271 268.378 5001 9888.591 10156.968
81920.0 81920.0 0.0 4097.0 655360.0 169744.2 7569408.0 7202855.4 43280.0
25855.8 5271 268.378 5002 9888.754 10157.131
81920.0 81920.0 0.0 4097.0 655360.0 169822.8 7569408.0 7202855.4 43280.0
25855.8 5271 268.378 5002 9888.754 10157.131
81920.0 81920.0 0.0 4097.0 655360.0 169888.3 7569408.0 7202855.4 43280.0
25855.8 5271 268.378 5002 9888.754 10157.131
81920.0 81920.0 0.0 4097.0 655360.0 170125.7 7569408.0 7202855.4 43280.0
25855.8 5271 268.378 5003 9888.946 10157.323
81920.0 81920.0 0.0 4097.0 655360.0 170399.8 7569408.0 7202855.3 43280.0
25855.8 5271 268.378 5004 9889.106 10157.483
81920.0 81920.0 0.0 4097.0 655360.0 170399.8 7569408.0 7202855.3 43280.0
25855.8 5271 268.378 5004 9889.106 10157.483
81920.0 81920.0 0.0 4097.0 655360.0 170420.7 7569408.0 7202855.3 43280.0
25855.8 5271 268.378 5004 9889.106 10157.483
81920.0 81920.0 0.0 4097.0 655360.0 170651.4 7569408.0 7202855.3 43280.0
25855.8 5271 268.378 5005 9889.301 10157.679
81920.0 81920.0 0.0 4097.0 655360.0 170916.8 7569408.0 7202855.2 43280.0
25855.8 5271 268.378 5006 9889.464 10157.842
81920.0 81920.0 0.0 4097.0 655360.0 170923.6 7569408.0 7202855.2 43280.0
25855.8 5271 268.378 5006 9889.464 10157.842
81920.0 81920.0 0.0 4097.0 655360.0 170925.6 7569408.0 7202855.2 43280.0
25855.8 5271 268.378 5006 9889.464 10157.842
81920.0 81920.0 0.0 4097.0 655360.0 171009.8 7569408.0 7202855.2 43280.0
25855.8 5271 268.378 5007 9889.660 10158.037
81920.0 81920.0 0.0 4097.0 655360.0 171249.8 7569408.0 7202855.2 43280.0
25855.8 5271 268.378 5008 9889.823 10158.201
81920.0 81920.0 0.0 4097.0 655360.0 171518.8 7569408.0 7202855.2 43280.0
25855.8 5271 268.378 5008 9889.823 10158.201
81920.0 81920.0 0.0 4097.0 655360.0 171530.6 7569408.0 7202855.2 43280.0
25855.8 5271 268.378 5008 9889.823 10158.201
81920.0 81920.0 0.0 4097.0 655360.0 171549.2 7569408.0 7202855.2 43280.0
25855.8 5271 268.378 5009 9890.024 10158.402
81920.0 81920.0 0.0 4097.0 655360.0 173035.0 7569408.0 7202850.4 43280.0
25855.8 5271 268.378 5010 9890.190 10158.568
81920.0 81920.0 0.0 4097.0 655360.0 173367.8 7569408.0 7202850.4 43280.0
25855.8 5271 268.378 5010 9890.190 10158.568
81920.0 81920.0 0.0 4097.0 655360.0 173376.8 7569408.0 7202850.4 43280.0
25855.8 5271 268.378 5010 9890.190 10158.568
81920.0 81920.0 0.0 4097.0 655360.0 173395.5 7569408.0 7202850.4 43280.0
25855.8 5271 268.378 5011 9890.456 10158.834
81920.0 81920.0 0.0 4097.0 655360.0 173451.8 7569408.0 7202850.3 43280.0
25855.8 5271 268.378 5011 9890.456 10158.834
81920.0 81920.0 0.0 4097.0 655360.0 173933.5 7569408.0 7202850.3 43280.0
25855.8 5271 268.378 5012 9890.622 10158.999
81920.0 81920.0 0.0 4097.0 655360.0 173969.7 7569408.0 7202850.3 43280.0
25855.8 5271 268.378 5012 9890.622 10158.999
81920.0 81920.0 0.0 4097.0 655360.0 173988.3 7569408.0 7202850.3 43280.0
25855.8 5271 268.378 5013 9890.820 10159.197
81920.0 81920.0 0.0 4097.0 655360.0 174053.8 7569408.0 7202850.2 43280.0
25855.8 5271 268.378 5013 9890.820 10159.197
81920.0 81920.0 0.0 4097.0 655360.0 174286.1 7569408.0 7202850.2 43280.0
25855.8 5271 268.378 5014 9890.986 10159.363
81920.0 81920.0 0.0 4097.0 655360.0 174551.0 7569408.0 7202850.2 43280.0
25855.8 5271 268.378 5014 9890.986 10159.363
81920.0 81920.0 0.0 4097.0 655360.0 174572.0 7569408.0 7202850.2 43280.0
25855.8 5271 268.378 5015 9890.986 10159.363
81920.0 81920.0 0.0 4097.0 655360.0 174592.9 7569408.0 7202850.1 43280.0
25855.8 5271 268.378 5015 9891.185 10159.562
81920.0 81920.0 0.0 4097.0 655360.0 174637.4 7569408.0 7202850.1 43280.0
25855.8 5271 268.378 5016 9891.350 10159.728
81920.0 81920.0 0.0 4097.0 655360.0 175049.8 7569408.0 7202850.1 43280.0
25855.8 5271 268.378 5016 9891.350 10159.728
81920.0 81920.0 0.0 4097.0 655360.0 175130.7 7569408.0 7202850.1 43280.0
25855.8 5271 268.378 5016 9891.350 10159.728
81920.0 81920.0 0.0 4097.0 655360.0 175168.4 7569408.0 7202850.1 43280.0
25855.8 5271 268.378 5017 9891.554 10159.932
81920.0 81920.0 0.0 4097.0 655360.0 175182.2 7569408.0 7202850.1 43280.0
25855.8 5271 268.378 5018 9891.721 10160.098
81920.0 81920.0 0.0 4097.0 655360.0 175402.9 7569408.0 7202850.1 43280.0
25855.8 5271 268.378 5018 9891.721 10160.098
81920.0 81920.0 0.0 4097.0 655360.0 175723.5 7569408.0 7202850.1 43280.0
25855.8 5271 268.378 5018 9891.721 10160.098
81920.0 81920.0 0.0 4097.0 655360.0 175742.2 7569408.0 7202850.1 43280.0
25855.8 5271 268.378 5019 9891.921 10160.298
81920.0 81920.0 0.0 4097.0 655360.0 175778.9 7569408.0 7202850.0 43280.0
25855.8 5271 268.378 5020 9892.089 10160.466
81920.0 81920.0 0.0 4097.0 655360.0 175823.6 7569408.0 7202850.0 43280.0
25855.8 5271 268.378 5020 9892.089 10160.466
81920.0 81920.0 0.0 4097.0 655360.0 176315.6 7569408.0 7202850.0 43280.0
25855.8 5271 268.378 5020 9892.089 10160.466
81920.0 81920.0 0.0 4097.0 655360.0 176351.6 7569408.0 7202850.0 43280.0
25855.8 5271 268.378 5021 9892.291 10160.669
81920.0 81920.0 0.0 4097.0 655360.0 176351.6 7569408.0 7202849.9 43280.0
25855.8 5271 268.378 5022 9892.460 10160.838
81920.0 81920.0 0.0 4097.0 655360.0 176370.6 7569408.0 7202849.9 43280.0
25855.8 5271 268.378 5022 9892.460 10160.838
81920.0 81920.0 0.0 4097.0 655360.0 176846.1 7569408.0 7202849.9 43280.0
25855.8 5271 268.378 5022 9892.460 10160.838
81920.0 81920.0 0.0 4097.0 655360.0 176865.2 7569408.0 7202849.9 43280.0
25855.8 5271 268.378 5023 9892.665 10161.043
81920.0 81920.0 0.0 4097.0 655360.0 176865.2 7569408.0 7202849.9 43280.0
25855.8 5271 268.378 5024 9892.834 10161.212
81920.0 81920.0 0.0 4097.0 655360.0 176942.8 7569408.0 7202849.9 43280.0
25855.8 5271 268.378 5024 9892.834 10161.212
81920.0 81920.0 0.0 4097.0 655360.0 177025.1 7569408.0 7202849.9 43280.0
25855.8 5271 268.378 5024 9892.834 10161.212
81920.0 81920.0 0.0 4097.0 655360.0 177437.4 7569408.0 7202849.9 43280.0
25855.8 5271 268.378 5025 9893.039 10161.416
81920.0 81920.0 0.0 4097.0 655360.0 177468.4 7569408.0 7202849.9 43280.0
25855.8 5271 268.378 5026 9893.213 10161.590
81920.0 81920.0 0.0 4097.0 655360.0 177472.6 7569408.0 7202849.9 43280.0
25855.8 5271 268.378 5026 9893.213 10161.590
81920.0 81920.0 0.0 4097.0 655360.0 177556.8 7569408.0 7202849.9 43280.0
25855.8 5271 268.378 5026 9893.213 10161.590
81920.0 81920.0 0.0 4097.0 655360.0 178032.6 7569408.0 7202849.9 43280.0
25855.8 5271 268.378 5027 9893.454 10161.832
81920.0 81920.0 0.0 4097.0 655360.0 178059.4 7569408.0 7202849.8 43280.0
25855.8 5271 268.378 5028 9893.624 10162.001
81920.0 81920.0 0.0 4097.0 655360.0 178059.4 7569408.0 7202849.8 43280.0
25855.8 5271 268.378 5028 9893.624 10162.001
81920.0 81920.0 0.0 4097.0 655360.0 178059.4 7569408.0 7202849.8 43280.0
25855.8 5271 268.378 5028 9893.624 10162.001
81920.0 81920.0 0.0 4097.0 655360.0 178300.8 7569408.0 7202849.8 43280.0
25855.8 5271 268.378 5029 9893.901 10162.279
81920.0 81920.0 0.0 4097.0 655360.0 178630.4 7569408.0 7202849.7 43280.0
25855.8 5271 268.378 5030 9894.075 10162.453
81920.0 81920.0 0.0 4097.0 655360.0 178638.6 7569408.0 7202849.7 43280.0
25855.8 5271 268.378 5030 9894.075 10162.453
81920.0 81920.0 0.0 4097.0 655360.0 178638.6 7569408.0 7202849.7 43280.0
25855.8 5271 268.378 5030 9894.075 10162.453
81920.0 81920.0 0.0 4097.0 655360.0 178734.5 7569408.0 7202849.7 43280.0
25855.8 5271 268.378 5031 9894.294 10162.672
81920.0 81920.0 0.0 4097.0 655360.0 179272.2 7569408.0 7202849.7 43280.0
25855.8 5271 268.378 5032 9894.468 10162.845
81920.0 81920.0 0.0 4097.0 655360.0 179308.5 7569408.0 7202849.7 43280.0
25855.8 5271 268.378 5032 9894.468 10162.845
81920.0 81920.0 0.0 4097.0 655360.0 179308.5 7569408.0 7202849.7 43280.0
25855.8 5271 268.378 5032 9894.468 10162.845
81920.0 81920.0 0.0 4097.0 655360.0 179308.5 7569408.0 7202849.7 43280.0
25855.8 5271 268.378 5033 9894.681 10163.058
81920.0 81920.0 0.0 4097.0 655360.0 179784.0 7569408.0 7202849.6 43280.0
25855.8 5271 268.378 5033 9894.681 10163.058
81920.0 81920.0 0.0 4097.0 655360.0 179804.6 7569408.0 7202849.6 43280.0
25855.8 5271 268.378 5034 9894.854 10163.231
81920.0 81920.0 0.0 4097.0 655360.0 181033.4 7569408.0 7202849.6 43280.0
25855.8 5271 268.378 5034 9894.854 10163.231
81920.0 81920.0 0.0 4097.0 655360.0 181033.4 7569408.0 7202849.6 43280.0
25855.8 5271 268.378 5035 9895.135 10163.513
81920.0 81920.0 0.0 4097.0 655360.0 181052.0 7569408.0 7202849.5 43280.0
25855.8 5271 268.378 5035 9895.135 10163.513
81920.0 81920.0 0.0 4097.0 655360.0 181509.0 7569408.0 7202849.5 43280.0
25855.8 5271 268.378 5036 9895.311 10163.689
81920.0 81920.0 0.0 4097.0 655360.0 181608.5 7569408.0 7202849.5 43280.0
25855.8 5271 268.378 5036 9895.311 10163.689
81920.0 81920.0 0.0 4097.0 655360.0 181610.7 7569408.0 7202849.5 43280.0
25855.8 5271 268.378 5037 9895.311 10163.689
81920.0 81920.0 0.0 4097.0 655360.0 181640.3 7569408.0 7202849.5 43280.0
25855.8 5271 268.378 5037 9895.539 10163.917
81920.0 81920.0 0.0 4097.0 655360.0 182116.0 7569408.0 7202849.5 43280.0
25855.8 5271 268.378 5038 9895.716 10164.093
81920.0 81920.0 0.0 4097.0 655360.0 182207.3 7569408.0 7202849.5 43280.0
25855.8 5271 268.378 5038 9895.716 10164.093
81920.0 81920.0 0.0 4097.0 655360.0 182281.0 7569408.0 7202849.5 43280.0
25855.8 5271 268.378 5038 9895.716 10164.093
81920.0 81920.0 0.0 4097.0 655360.0 182281.0 7569408.0 7202849.5 43280.0
25855.8 5271 268.378 5039 9896.000 10164.378
81920.0 81920.0 0.0 4097.0 655360.0 182285.1 7569408.0 7202849.5 43280.0
25855.8 5271 268.378 5040 9896.180 10164.558
81920.0 81920.0 0.0 4097.0 655360.0 182844.7 7569408.0 7202849.5 43280.0
25855.8 5271 268.378 5040 9896.180 10164.558
81920.0 81920.0 0.0 4097.0 655360.0 182872.0 7569408.0 7202849.5 43280.0
25855.8 5271 268.378 5040 9896.180 10164.558
81920.0 81920.0 0.0 4097.0 655360.0 182872.0 7569408.0 7202849.5 43280.0
25855.8 5271 268.378 5041 9896.463 10164.841
81920.0 81920.0 0.0 4097.0 655360.0 182872.0 7569408.0 7202849.3 43280.0
25855.8 5271 268.378 5042 9896.641 10165.019
81920.0 81920.0 0.0 4097.0 655360.0 183413.3 7569408.0 7202849.3 43280.0
25855.8 5271 268.378 5042 9896.641 10165.019
81920.0 81920.0 0.0 4097.0 655360.0 183443.4 7569408.0 7202849.3 43280.0
25855.8 5271 268.378 5042 9896.641 10165.019
81920.0 81920.0 0.0 4097.0 655360.0 183464.8 7569408.0 7202849.3 43280.0
25855.8 5271 268.378 5043 9896.857 10165.235
81920.0 81920.0 0.0 4097.0 655360.0 183476.6 7569408.0 7202849.2 43280.0
25855.8 5271 268.378 5044 9897.033 10165.411
81920.0 81920.0 0.0 4097.0 655360.0 183733.3 7569408.0 7202849.2 43280.0
25855.8 5271 268.378 5044 9897.033 10165.411
81920.0 81920.0 0.0 4097.0 655360.0 184045.6 7569408.0 7202849.2 43280.0
25855.8 5271 268.378 5044 9897.033 10165.411
81920.0 81920.0 0.0 4097.0 655360.0 184055.9 7569408.0 7202849.2 43280.0
25855.8 5271 268.378 5045 9897.314 10165.691
81920.0 81920.0 0.0 4097.0 655360.0 184057.9 7569408.0 7202849.2 43280.0
25855.8 5271 268.378 5046 9897.492 10165.869
81920.0 81920.0 0.0 4097.0 655360.0 184057.9 7569408.0 7202849.2 43280.0
25855.8 5271 268.378 5046 9897.492 10165.869
81920.0 81920.0 0.0 4097.0 655360.0 184618.0 7569408.0 7202849.2 43280.0
25855.8 5271 268.378 5046 9897.492 10165.869
81920.0 81920.0 0.0 4097.0 655360.0 184644.8 7569408.0 7202849.2 43280.0
25855.8 5271 268.378 5047 9897.706 10166.084
81920.0 81920.0 0.0 4097.0 655360.0 184647.0 7569408.0 7202849.1 43280.0
25855.8 5271 268.378 5048 9897.886 10166.264
81920.0 81920.0 0.0 4097.0 655360.0 184647.0 7569408.0 7202849.1 43280.0
25855.8 5271 268.378 5048 9897.886 10166.264
81920.0 81920.0 0.0 4097.0 655360.0 185105.9 7569408.0 7202849.1 43280.0
25855.8 5271 268.378 5048 9897.886 10166.264
81920.0 81920.0 0.0 4097.0 655360.0 185231.6 7569408.0 7202849.1 43280.0
25855.8 5271 268.378 5049 9898.095 10166.472
81920.0 81920.0 0.0 4097.0 655360.0 185239.8 7569408.0 7202849.1 43280.0
25855.8 5271 268.378 5050 9898.276 10166.654
81920.0 81920.0 0.0 4097.0 655360.0 185251.5 7569408.0 7202849.1 43280.0
25855.8 5271 268.378 5050 9898.276 10166.654
81920.0 81920.0 0.0 4097.0 655360.0 194817.6 7569408.0 7202849.1 43280.0
25855.8 5271 268.378 5050 9898.276 10166.654
81920.0 81920.0 0.0 4097.0 655360.0 195326.2 7569408.0 7202849.1 43280.0
25855.8 5271 268.378 5051 9898.570 10166.948
81920.0 81920.0 0.0 4097.0 655360.0 195333.0 7569408.0 7202849.0 43280.0
25855.8 5271 268.378 5052 9898.752 10167.130
81920.0 81920.0 0.0 4097.0 655360.0 195341.3 7569408.0 7202849.0 43280.0
25855.8 5271 268.378 5052 9898.752 10167.130
81920.0 81920.0 0.0 4097.0 655360.0 195341.3 7569408.0 7202849.0 43280.0
25855.8 5271 268.378 5052 9898.752 10167.130
81920.0 81920.0 0.0 4097.0 655360.0 196751.1 7569408.0 7202849.0 43280.0
25855.8 5271 268.378 5053 9899.045 10167.423
81920.0 81920.0 0.0 4097.0 655360.0 196934.4 7569408.0 7202848.8 43280.0
25855.8 5271 268.378 5053 9899.045 10167.423
81920.0 81920.0 0.0 4097.0 655360.0 196944.7 7569408.0 7202848.8 43280.0
25855.8 5271 268.378 5054 9899.230 10167.608
81920.0 81920.0 0.0 4097.0 655360.0 197021.9 7569408.0 7202848.8 43280.0
25855.8 5271 268.378 5054 9899.230 10167.608
81920.0 81920.0 0.0 4097.0 655360.0 197037.8 7569408.0 7202848.8 43280.0
25855.8 5271 268.378 5055 9899.451 10167.829
81920.0 81920.0 0.0 4097.0 655360.0 197541.2 7569408.0 7202848.7 43280.0
25855.8 5271 268.378 5055 9899.451 10167.829
81920.0 81920.0 0.0 4097.0 655360.0 197541.2 7569408.0 7202848.7 43280.0
25855.8 5271 268.378 5056 9899.638 10168.015
81920.0 81920.0 0.0 4097.0 655360.0 197614.9 7569408.0 7202848.7 43280.0
25855.8 5271 268.378 5056 9899.638 10168.015
81920.0 81920.0 0.0 4097.0 655360.0 197617.0 7569408.0 7202848.7 43280.0
25855.8 5271 268.378 5057 9899.858 10168.235
81920.0 81920.0 0.0 4097.0 655360.0 198123.4 7569408.0 7202848.7 43280.0
25855.8 5271 268.378 5057 9899.858 10168.235
81920.0 81920.0 0.0 4097.0 655360.0 198130.2 7569408.0 7202848.7 43280.0
25855.8 5271 268.378 5058 9900.043 10168.420
81920.0 81920.0 0.0 4097.0 655360.0 198140.7 7569408.0 7202848.7 43280.0
25855.8 5271 268.378 5058 9900.043 10168.420
81920.0 81920.0 0.0 4097.0 655360.0 199340.2 7569408.0 7202848.7 43280.0
25855.8 5271 268.378 5059 9900.340 10168.718
81920.0 81920.0 0.0 4097.0 655360.0 199580.3 7569408.0 7202848.6 43280.0
25855.8 5271 268.378 5059 9900.340 10168.718
81920.0 81920.0 0.0 4097.0 655360.0 199843.5 7569408.0 7202848.6 43280.0
25855.8 5271 268.378 5060 9900.528 10168.905
81920.0 81920.0 0.0 4097.0 655360.0 199917.3 7569408.0 7202848.6 43280.0
25855.8 5271 268.378 5060 9900.528 10168.905
81920.0 81920.0 0.0 4097.0 655360.0 200007.7 7569408.0 7202848.6 43280.0
25855.8 5271 268.378 5061 9900.528 10168.905
81920.0 81920.0 0.0 4097.0 655360.0 200009.8 7569408.0 7202848.6 43280.0
25855.8 5271 268.378 5061 9900.818 10169.196
81920.0 81920.0 0.0 4097.0 655360.0 200466.7 7569408.0 7202848.6 43280.0
25855.8 5271 268.378 5062 9901.005 10169.383
81920.0 81920.0 0.0 4097.0 655360.0 200513.2 7569408.0 7202848.6 43280.0
25855.8 5271 268.378 5062 9901.005 10169.383
81920.0 81920.0 0.0 4097.0 655360.0 200521.4 7569408.0 7202848.6 43280.0
25855.8 5271 268.378 5062 9901.005 10169.383
81920.0 81920.0 0.0 4097.0 655360.0 200521.4 7569408.0 7202848.5 43280.0
25855.8 5271 268.378 5063 9901.302 10169.680
81920.0 81920.0 0.0 4097.0 655360.0 200759.4 7569408.0 7202848.5 43280.0
25855.8 5271 268.378 5064 9901.490 10169.868
81920.0 81920.0 0.0 4097.0 655360.0 201018.1 7569408.0 7202848.5 43280.0
25855.8 5271 268.378 5064 9901.490 10169.868
81920.0 81920.0 0.0 4097.0 655360.0 201045.0 7569408.0 7202848.5 43280.0
25855.8 5271 268.378 5064 9901.490 10169.868
81920.0 81920.0 0.0 4097.0 655360.0 201112.5 7569408.0 7202848.5 43280.0
25855.8 5271 268.378 5065 9901.710 10170.087
81920.0 81920.0 0.0 4097.0 655360.0 201112.5 7569408.0 7202848.5 43280.0
25855.8 5271 268.378 5066 9901.900 10170.277
81920.0 81920.0 0.0 4097.0 655360.0 201590.6 7569408.0 7202848.5 43280.0
25855.8 5271 268.378 5066 9901.900 10170.277
81920.0 81920.0 0.0 4097.0 655360.0 201613.2 7569408.0 7202848.5 43280.0
25855.8 5271 268.378 5066 9901.900 10170.277
81920.0 81920.0 0.0 4097.0 655360.0 201640.1 7569408.0 7202848.5 43280.0
25855.8 5271 268.378 5067 9902.200 10170.578
81920.0 81920.0 0.0 4097.0 655360.0 201717.4 7569408.0 7202848.4 43280.0
25855.8 5271 268.378 5068 9902.390 10170.768
81920.0 81920.0 0.0 4097.0 655360.0 202023.1 7569408.0 7202848.4 43280.0
25855.8 5271 268.378 5068 9902.390 10170.768
81920.0 81920.0 0.0 4097.0 655360.0 202267.7 7569408.0 7202848.4 43280.0
25855.8 5271 268.378 5068 9902.390 10170.768
81920.0 81920.0 0.0 4097.0 655360.0 202278.1 7569408.0 7202848.4 43280.0
25855.8 5271 268.378 5069 9902.693 10171.071
81920.0 81920.0 0.0 4097.0 655360.0 202362.5 7569408.0 7202848.4 43280.0
25855.8 5271 268.378 5070 9902.883 10171.261
81920.0 81920.0 0.0 4097.0 655360.0 202600.7 7569408.0 7202848.4 43280.0
25855.8 5271 268.378 5070 9902.883 10171.261
81920.0 81920.0 0.0 4097.0 655360.0 202854.2 7569408.0 7202848.4 43280.0
25855.8 5271 268.378 5070 9902.883 10171.261
81920.0 81920.0 0.0 4097.0 655360.0 202926.5 7569408.0 7202848.3 43280.0
25855.8 5271 268.378 5071 9903.113 10171.491
81920.0 81920.0 0.0 4097.0 655360.0 202934.8 7569408.0 7202848.2 43280.0
25855.8 5271 268.378 5072 9903.303 10171.681
81920.0 81920.0 0.0 4097.0 655360.0 202955.5 7569408.0 7202848.2 43280.0
25855.8 5271 268.378 5072 9903.303 10171.681
81920.0 81920.0 0.0 4097.0 655360.0 203291.3 7569408.0 7202848.2 43280.0
25855.8 5271 268.378 5072 9903.303 10171.681
81920.0 81920.0 0.0 4097.0 655360.0 203517.0 7569408.0 7202848.2 43280.0
25855.8 5271 268.378 5073 9903.607 10171.985
81920.0 81920.0 0.0 4097.0 655360.0 203539.0 7569408.0 7202848.0 43280.0
25855.8 5271 268.378 5074 9903.799 10172.177
81920.0 81920.0 0.0 4097.0 655360.0 203557.6 7569408.0 7202848.0 43280.0
25855.8 5271 268.378 5074 9903.799 10172.177
81920.0 81920.0 0.0 4097.0 655360.0 203795.7 7569408.0 7202848.0 43280.0
25855.8 5271 268.378 5074 9903.799 10172.177
81920.0 81920.0 0.0 4097.0 655360.0 204042.6 7569408.0 7202848.0 43280.0
25855.8 5271 268.378 5075 9904.025 10172.403
81920.0 81920.0 0.0 4097.0 655360.0 204042.6 7569408.0 7202848.0 43280.0
25855.8 5271 268.378 5076 9904.220 10172.598
81920.0 81920.0 0.0 4097.0 655360.0 204135.0 7569408.0 7202848.0 43280.0
25855.8 5271 268.378 5076 9904.220 10172.598
81920.0 81920.0 0.0 4097.0 655360.0 204137.0 7569408.0 7202848.0 43280.0
25855.8 5271 268.378 5076 9904.220 10172.598
81920.0 81920.0 0.0 4097.0 655360.0 204582.2 7569408.0 7202848.0 43280.0
25855.8 5271 268.378 5077 9904.451 10172.829
81920.0 81920.0 0.0 4097.0 655360.0 204633.6 7569408.0 7202847.9 43280.0
25855.8 5271 268.378 5078 9904.644 10173.022
81920.0 81920.0 0.0 4097.0 655360.0 204660.4 7569408.0 7202847.9 43280.0
25855.8 5271 268.378 5078 9904.644 10173.022
81920.0 81920.0 0.0 4097.0 655360.0 204741.6 7569408.0 7202847.9 43280.0
25855.8 5271 268.378 5078 9904.644 10173.022
81920.0 81920.0 0.0 4097.0 655360.0 205000.7 7569408.0 7202847.9 43280.0
25855.8 5271 268.378 5079 9904.952 10173.329
81920.0 81920.0 0.0 4097.0 655360.0 205226.3 7569408.0 7202847.8 43280.0
25855.8 5271 268.378 5080 9904.952 10173.329
81920.0 81920.0 0.0 4097.0 655360.0 205228.5 7569408.0 7202847.8 43280.0
25855.8 5271 268.378 5080 9905.144 10173.521
81920.0 81920.0 0.0 4097.0 655360.0 205267.2 7569408.0 7202847.8 43280.0
25855.8 5271 268.378 5080 9905.144 10173.521
81920.0 81920.0 0.0 4097.0 655360.0 205267.2 7569408.0 7202847.8 43280.0
25855.8 5271 268.378 5081 9905.369 10173.746
81920.0 81920.0 0.0 4097.0 655360.0 205698.6 7569408.0 7202847.8 43280.0
25855.8 5271 268.378 5081 9905.369 10173.746
81920.0 81920.0 0.0 4097.0 655360.0 205752.3 7569408.0 7202847.8 43280.0
25855.8 5271 268.378 5082 9905.563 10173.940
81920.0 81920.0 0.0 4097.0 655360.0 205844.6 7569408.0 7202847.8 43280.0
25855.8 5271 268.378 5082 9905.563 10173.940
81920.0 81920.0 0.0 4097.0 655360.0 205846.6 7569408.0 7202847.8 43280.0
25855.8 5271 268.378 5083 9905.797 10174.174
81920.0 81920.0 0.0 4097.0 655360.0 206115.6 7569408.0 7202847.8 43280.0
25855.8 5271 268.378 5083 9905.797 10174.174
81920.0 81920.0 0.0 4097.0 655360.0 206406.6 7569408.0 7202847.8 43280.0
25855.8 5271 268.378 5084 9905.992 10174.369
81920.0 81920.0 0.0 4097.0 655360.0 207640.6 7569408.0 7202847.8 43280.0
25855.8 5271 268.378 5084 9905.992 10174.369
81920.0 81920.0 0.0 4097.0 655360.0 207650.8 7569408.0 7202847.8 43280.0
25855.8 5271 268.378 5085 9906.220 10174.598
81920.0 81920.0 0.0 4097.0 655360.0 207908.0 7569408.0 7202847.7 43280.0
25855.8 5271 268.378 5085 9906.220 10174.598
81920.0 81920.0 0.0 4097.0 655360.0 208149.9 7569408.0 7202847.7 43280.0
25855.8 5271 268.378 5086 9906.417 10174.794
81920.0 81920.0 0.0 4097.0 655360.0 208201.3 7569408.0 7202847.7 43280.0
25855.8 5271 268.378 5086 9906.417 10174.794
81920.0 81920.0 0.0 4097.0 655360.0 208240.0 7569408.0 7202847.7 43280.0
25855.8 5271 268.378 5087 9906.417 10174.794
81920.0 81920.0 0.0 4097.0 655360.0 208240.0 7569408.0 7202847.6 43280.0
25855.8 5271 268.378 5087 9906.654 10175.031
81920.0 81920.0 0.0 4097.0 655360.0 208501.2 7569408.0 7202847.6 43280.0
25855.8 5271 268.378 5088 9906.853 10175.230
81920.0 81920.0 0.0 4097.0 655360.0 208726.8 7569408.0 7202847.6 43280.0
25855.8 5271 268.378 5088 9906.853 10175.230
81920.0 81920.0 0.0 4097.0 655360.0 208726.8 7569408.0 7202847.6 43280.0
25855.8 5271 268.378 5088 9906.853 10175.230
81920.0 81920.0 0.0 4097.0 655360.0 208821.1 7569408.0 7202847.6 43280.0
25855.8 5271 268.378 5089 9907.164 10175.542
81920.0 81920.0 0.0 4097.0 655360.0 209059.2 7569408.0 7202847.6 43280.0
25855.8 5271 268.378 5090 9907.365 10175.743
81920.0 81920.0 0.0 4097.0 655360.0 209398.1 7569408.0 7202847.6 43280.0
25855.8 5271 268.378 5090 9907.365 10175.743
81920.0 81920.0 0.0 4097.0 655360.0 209400.4 7569408.0 7202847.6 43280.0
25855.8 5271 268.378 5090 9907.365 10175.743
81920.0 81920.0 0.0 4097.0 655360.0 209408.6 7569408.0 7202847.6 43280.0
25855.8 5271 268.378 5091 9907.596 10175.973
81920.0 81920.0 0.0 4097.0 655360.0 209427.3 7569408.0 7202847.5 43280.0
25855.8 5271 268.378 5092 9907.793 10176.171
81920.0 81920.0 0.0 4097.0 655360.0 209684.5 7569408.0 7202847.5 43280.0
25855.8 5271 268.378 5092 9907.793 10176.171
81920.0 81920.0 0.0 4097.0 655360.0 209910.1 7569408.0 7202847.5 43280.0
25855.8 5271 268.378 5092 9907.793 10176.171
81920.0 81920.0 0.0 4097.0 655360.0 209932.1 7569408.0 7202847.5 43280.0
25855.8 5271 268.378 5093 9908.028 10176.405
81920.0 81920.0 0.0 4097.0 655360.0 209952.8 7569408.0 7202847.4 43280.0
25855.8 5271 268.378 5094 9908.225 10176.603
81920.0 81920.0 0.0 4097.0 655360.0 210190.9 7569408.0 7202847.4 43280.0
25855.8 5271 268.378 5094 9908.225 10176.603
81920.0 81920.0 0.0 4097.0 655360.0 210500.9 7569408.0 7202847.4 43280.0
25855.8 5271 268.378 5094 9908.225 10176.603
81920.0 81920.0 0.0 4097.0 655360.0 210500.9 7569408.0 7202847.4 43280.0
25855.8 5271 268.378 5095 9908.470 10176.848
81920.0 81920.0 0.0 4097.0 655360.0 210509.1 7569408.0 7202847.4 43280.0
25855.8 5271 268.378 5096 9908.668 10177.046
81920.0 81920.0 0.0 4097.0 655360.0 210785.8 7569408.0 7202847.4 43280.0
25855.8 5271 268.378 5096 9908.668 10177.046
81920.0 81920.0 0.0 4097.0 655360.0 210818.8 7569408.0 7202847.4 43280.0
25855.8 5271 268.378 5096 9908.668 10177.046
81920.0 81920.0 0.0 4097.0 655360.0 211109.8 7569408.0 7202847.4 43280.0
25855.8 5271 268.378 5097 9908.902 10177.279
81920.0 81920.0 0.0 4097.0 655360.0 211175.3 7569408.0 7202847.4 43280.0
25855.8 5271 268.378 5098 9909.103 10177.481
81920.0 81920.0 0.0 4097.0 655360.0 211267.7 7569408.0 7202847.4 43280.0
25855.8 5271 268.378 5098 9909.103 10177.481
81920.0 81920.0 0.0 4097.0 655360.0 211527.0 7569408.0 7202847.4 43280.0
25855.8 5271 268.378 5098 9909.103 10177.481
81920.0 81920.0 0.0 4097.0 655360.0 211754.7 7569408.0 7202847.4 43280.0
25855.8 5271 268.378 5099 9909.341 10177.718
81920.0 81920.0 0.0 4097.0 655360.0 211766.4 7569408.0 7202847.2 43280.0
25855.8 5271 268.378 5100 9909.542 10177.920
81920.0 81920.0 0.0 4097.0 655360.0 211840.2 7569408.0 7202847.2 43280.0
25855.8 5271 268.378 5100 9909.542 10177.920
81920.0 81920.0 0.0 4097.0 655360.0 212116.0 7569408.0 7202847.2 43280.0
25855.8 5271 268.378 5100 9909.542 10177.920
81920.0 81920.0 0.0 4097.0 655360.0 212122.8 7569408.0 7202847.2 43280.0
25855.8 5271 268.378 5101 9909.783 10178.161
81920.0 81920.0 0.0 4097.0 655360.0 212411.4 7569408.0 7202847.2 43280.0
25855.8 5271 268.378 5102 9909.783 10178.161
81920.0 81920.0 0.0 4097.0 655360.0 212424.6 7569408.0 7202847.2 43280.0
25855.8 5271 268.378 5102 9909.987 10178.365
81920.0 81920.0 0.0 4097.0 655360.0 212432.8 7569408.0 7202847.2 43280.0
25855.8 5271 268.378 5102 9909.987 10178.365
81920.0 81920.0 0.0 4097.0 655360.0 212720.2 7569408.0 7202847.2 43280.0
25855.8 5271 268.378 5103 9910.313 10178.690
81920.0 81920.0 0.0 4097.0 655360.0 212945.8 7569408.0 7202847.1 43280.0
25855.8 5271 268.378 5103 9910.313 10178.690
81920.0 81920.0 0.0 4097.0 655360.0 212945.8 7569408.0 7202847.1 43280.0
25855.8 5271 268.378 5104 9910.516 10178.893
81920.0 81920.0 0.0 4097.0 655360.0 212958.1 7569408.0 7202847.1 43280.0
25855.8 5271 268.378 5104 9910.516 10178.893
81920.0 81920.0 0.0 4097.0 655360.0 213301.3 7569408.0 7202847.1 43280.0
25855.8 5271 268.378 5105 9910.851 10179.229
81920.0 81920.0 0.0 4097.0 655360.0 213482.3 7569408.0 7202847.0 43280.0
25855.8 5271 268.378 5105 9910.851 10179.229
81920.0 81920.0 0.0 4097.0 655360.0 213527.0 7569408.0 7202847.0 43280.0
25855.8 5271 268.378 5106 9911.056 10179.433
81920.0 81920.0 0.0 4097.0 655360.0 213547.0 7569408.0 7202847.0 43280.0
25855.8 5271 268.378 5106 9911.056 10179.433
81920.0 81920.0 0.0 4097.0 655360.0 213612.5 7569408.0 7202847.0 43280.0
25855.8 5271 268.378 5107 9911.307 10179.684
81920.0 81920.0 0.0 4097.0 655360.0 213892.6 7569408.0 7202847.0 43280.0
25855.8 5271 268.378 5107 9911.307 10179.684
81920.0 81920.0 0.0 4097.0 655360.0 214133.8 7569408.0 7202847.0 43280.0
25855.8 5271 268.378 5108 9911.513 10179.891
81920.0 81920.0 0.0 4097.0 655360.0 214133.8 7569408.0 7202847.0 43280.0
25855.8 5271 268.378 5108 9911.513 10179.891
81920.0 81920.0 0.0 4097.0 655360.0 215407.3 7569408.0 7202847.0 43280.0
25855.8 5271 268.378 5108 9911.513 10179.891
81920.0 81920.0 0.0 4097.0 655360.0 215677.9 7569408.0 7202846.9 43280.0
25855.8 5271 268.378 5109 9911.822 10180.199
81920.0 81920.0 0.0 4097.0 655360.0 215879.9 7569408.0 7202846.9 43280.0
25855.8 5271 268.378 5110 9912.026 10180.404
81920.0 81920.0 0.0 4097.0 655360.0 215924.5 7569408.0 7202846.9 43280.0
25855.8 5271 268.378 5110 9912.026 10180.404
81920.0 81920.0 0.0 4097.0 655360.0 215932.7 7569408.0 7202846.9 43280.0
25855.8 5271 268.378 5110 9912.026 10180.404
81920.0 81920.0 0.0 4097.0 655360.0 216189.6 7569408.0 7202846.9 43280.0
25855.8 5271 268.378 5111 9912.347 10180.725
81920.0 81920.0 0.0 4097.0 655360.0 216208.6 7569408.0 7202846.9 43280.0
25855.8 5271 268.378 5112 9912.556 10180.934
81920.0 81920.0 0.0 4097.0 655360.0 216504.1 7569408.0 7202846.9 43280.0
25855.8 5271 268.378 5112 9912.556 10180.934
81920.0 81920.0 0.0 4097.0 655360.0 216517.9 7569408.0 7202846.9 43280.0
25855.8 5271 268.378 5112 9912.556 10180.934
{quote}
This continues even after we shut down our app. Nothing is connected to
Cassandra any more, yet it is still stuck in full GC and cannot free up memory.
Running nodetool tpstats shows that nothing is pending, all seems OK:
{quote}
Pool Name Active Pending Completed Blocked All
time blocked
ReadStage 0 0 69555935 0
0
RequestResponseStage 0 0 0 0
0
MutationStage 0 0 73123690 0
0
ReadRepairStage 0 0 0 0
0
ReplicateOnWriteStage 0 0 0 0
0
GossipStage 0 0 0 0
0
CacheCleanupExecutor 0 0 0 0
0
MigrationStage 0 0 46 0
0
MemoryMeter 0 0 1125 0
0
FlushWriter 0 0 824 0
30
ValidationExecutor 0 0 0 0
0
InternalResponseStage 0 0 23 0
0
AntiEntropyStage 0 0 0 0
0
MemtablePostFlusher 0 0 1783 0
0
MiscStage 0 0 0 0
0
PendingRangeCalculator 0 0 1 0
0
CompactionExecutor 0 0 74330 0
0
commitlog_archiver 0 0 0 0
0
HintedHandoff 0 0 0 0
0
Message type Dropped
RANGE_SLICE 0
READ_REPAIR 0
PAGED_RANGE 0
BINARY 0
READ 585
MUTATION 75775
_TRACE 0
REQUEST_RESPONSE 0
COUNTER_MUTATION 0
{quote}
We had this happen on 2 separate boxes, one with 2.0.6, the other with 2.0.8.
Right now this is a total blocker for us. We are unable to process the customer
data and have to abort in the middle of large processing.
This is a new customer, so we did not have a chance to see if this occurred
with 1.1 or 1.2 in the past (we moved to 2.0 recently).
We have the Cassandra process still running, pls let us know if there is
anything else we could run to give you more insight.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
