Hi Kevin,
Could you please re-run your case and attach Ignite logs and GC logs from
all participating servers? I would expect that this is either a kind of
out-of-memory problem, or network saturation. Also please explain how
exactly do you load data to Ignite? Do you use *DataStreamer*, or may be
*IgniteCache.loadlCache*, or *IgniteCache.put*?
Vladimir.
On Tue, Apr 26, 2016 at 3:01 PM, Zhengqingzheng <zhengqingzh...@huawei.com>
wrote:
> Hi Vladimir,
>
> Thank you for your help.
>
> I tried to load 1Million records, and calculate each object’s size
> (including key and value object, where key is string type), summarized
> together and found the total memory consumption is 130Mb.
>
>
>
> Because ./ignitevisor.sh command only shows number of records, and no
> data allocation information, I don’t know how much memory has been consumed
> for each type of cache.
>
>
>
> My result is as follows:
>
> Log:
>
> Found big object: [Ljava.util.HashMap$Node;@24833164 size:
> 30.888206481933594Mb
>
> Found big object: java.util.HashMap@15819383 size: 30.88824462890625Mb
>
> Found big object: java.util.HashSet@10236420 size: 30.888259887695312Mb
>
> key size: 32388688 human readable data: 30.888259887695312Mb
>
> Found big object: [Lorg.jsr166.ConcurrentHashMap8$Node;@29556439 size:
> 129.99818420410156Mb
>
> Found big object: org.jsr166.ConcurrentHashMap8@19238297 size:
> *129.99822235107422Mb*
>
> value size: 136313016 human readable data: 129.99822235107422Mb
>
> *the whole number of record is 47Million, so the data size inside the
> cache should be 130*47=**6110Mb(around 6Gb).*
>
>
>
> However, when I try to load the whole data into the cache, I still get
> exceptions:
>
> The exception information is listed as follows:
>
> 1. ---------------------------------- exception info from client
> ----------------------------------------
>
> Before exception occurred, I have 10 nodes on two servers, server1 (48G
> ram) has 6nodes, each node is assigned with 7gb jvm heap; server2 (32gb
> ram) has 4 nodes with the same jvm settings as previous one.
>
> After exception, client stopped, and 8 node left (server1’s node remain,
> no exception occurred on this server), server2 ( 2 nodes remain, two nodes
> droped)
>
> The total number of records loaded is 37Million.
>
>
>
> [19:01:47] Topology snapshot [ver=77, servers=9, clients=1, CPUs=20,
> heap=64.0GB]
>
> [19:01:47,463][SEVERE][pub-#46%null%][GridTaskWorker] Failed to obtain
> remote job result policy for result from ComputeTask.result(..) method
> (will fail the whole task): GridJobResultImpl [job=C2 [],
> sib=GridJobSiblingImpl
> [sesId=0dbc0f15451-880a2dd1-bc95-4084-a705-4effcec5d2cd,
> jobId=6dbc0f15451-832bed3e-dc5d-4743-9853-127e3b516924,
> nodeId=832bed3e-dc5d-4743-9853-127e3b516924, isJobDone=false],
> jobCtx=GridJobContextImpl
> [jobId=6dbc0f15451-832bed3e-dc5d-4743-9853-127e3b516924, timeoutObj=null,
> attrs={}], node=TcpDiscoveryNode [id=832bed3e-dc5d-4743-9853-127e3b516924,
> addrs=[0:0:0:0:0:0:0:1%lo, 10.120.70.122, 127.0.0.1],
> sockAddrs=[/0:0:0:0:0:0:0:1%lo:47500, /0:0:0:0:0:0:0:1%lo:47500, /
> 10.120.70.122:47500, /127.0.0.1:47500], discPort=47500, order=7,
> intOrder=7, lastExchangeTime=1461663619161, loc=false,
> ver=1.5.0#20151229-sha1:f1f8cda2, isClient=false], ex=class
> o.a.i.cluster.ClusterTopologyException: Node has left grid:
> 832bed3e-dc5d-4743-9853-127e3b516924, hasRes=true, isCancelled=false,
> isOccupied=true]
>
> class org.apache.ignite.cluster.ClusterTopologyException: Node has left
> grid: 832bed3e-dc5d-4743-9853-127e3b516924
>
> at
> org.apache.ignite.internal.processors.task.GridTaskWorker.onNodeLeft(GridTaskWorker.java:1315)
>
> at
> org.apache.ignite.internal.processors.task.GridTaskProcessor$TaskDiscoveryListener$1.run(GridTaskProcessor.java:1246)
>
> at
> org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6453)
>
> at
> org.apache.ignite.internal.processors.closure.GridClosureProcessor$1.body(GridClosureProcessor.java:788)
>
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
>
> [19:02:16] Ignite node stopped OK [uptime=01:21:52:963]
>
>
>
> 2 ------------------------------ also, I use the recommended settings for
> the jvm and get the following log information
> ------------------------------------------------
>
> Java HotSpot(TM) Client VM (25.66-b18) for windows-x86 JRE (1.8.0_66-b18),
> built on Nov 9 2015 10:58:29 by "java_re" with MS VC++ 10.0 (VS2010)
>
> Memory: 4k page, physical 6291064k(1912832k free), swap 12580288k(6996288k
> free)
>
> CommandLine flags: -XX:GCLogFileSize=104857600
> -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824
> -XX:NumberOfGCLogFiles=10 -XX:+PrintGC -XX:+PrintGCDateStamps
> -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation
> -XX:-UseLargePagesIndividualAllocation
>
> 2016-04-26T17:40:15.609+0800: 4.839: [Full GC (Metadata GC Threshold)
> 2016-04-26T17:40:15.610+0800: 4.839: [Tenured: 0K->10330K(699072K),
> 0.0656011 secs] 162451K->10330K(1013632K), [Metaspace:
> 12085K->12085K(12672K)], 0.0659450 secs] [Times: user=0.06 sys=0.00,
> real=0.07 secs]
>
> 2016-04-26T17:40:24.837+0800: 14.044: [Full GC (Metadata GC Threshold)
> 2016-04-26T17:40:24.837+0800: 14.044: [Tenured: 10330K->32179K(699072K),
> 0.1810716 secs] 282679K->32179K(1013632K), [Metaspace:
> 20221K->20221K(20864K)], 0.1812825 secs] [Times: user=0.17 sys=0.00,
> real=0.18 secs]
>
> 2016-04-26T17:40:53.699+0800: 42.937: [GC (Allocation Failure)
> 2016-04-26T17:40:53.699+0800: 42.937: [DefNew: 279616K->8837K(314560K),
> 0.0242718 secs] 311795K->41016K(1013632K), 0.0244783 secs] [Times:
> user=0.03 sys=0.00, real=0.03 secs]
>
> 2016-04-26T17:41:26.411+0800: 75.516: [GC (Allocation Failure)
> 2016-04-26T17:41:26.412+0800: 75.516: [DefNew: 288453K->2685K(314560K),
> 0.0116993 secs] 320632K->34864K(1013632K), 0.0119035 secs] [Times:
> user=0.01 sys=0.00, real=0.01 secs]
>
> 2016-04-26T17:42:01.895+0800: 110.834: [GC (Allocation Failure)
> 2016-04-26T17:42:01.895+0800: 110.834: [DefNew: 282301K->2689K(314560K),
> 0.0111799 secs] 314480K->34868K(1013632K), 0.0113629 secs] [Times:
> user=0.01 sys=0.00, real=0.01 secs]
>
> 2016-04-26T17:42:37.354+0800: 145.818: [GC (Allocation Failure)
> 2016-04-26T17:42:37.354+0800: 145.818: [DefNew: 282305K->2695K(314560K),
> 0.0102415 secs] 314484K->34874K(1013632K), 0.0104181 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T17:43:13.549+0800: 181.641: [GC (Allocation Failure)
> 2016-04-26T17:43:13.549+0800: 181.641: [DefNew: 282311K->2893K(314560K),
> 0.0134864 secs] 314490K->35072K(1013632K), 0.0136970 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T17:43:50.453+0800: 218.129: [GC (Allocation Failure)
> 2016-04-26T17:43:50.453+0800: 218.129: [DefNew: 282509K->2708K(314560K),
> 0.0160308 secs] 314688K->34887K(1013632K), 0.0162459 secs] [Times:
> user=0.03 sys=0.00, real=0.03 secs]
>
> 2016-04-26T17:44:27.317+0800: 254.604: [GC (Allocation Failure)
> 2016-04-26T17:44:27.317+0800: 254.605: [DefNew: 282324K->2715K(314560K),
> 0.0128055 secs] 314503K->34894K(1013632K), 0.0130086 secs] [Times:
> user=0.01 sys=0.00, real=0.01 secs]
>
> 2016-04-26T17:45:03.944+0800: 291.040: [GC (Allocation Failure)
> 2016-04-26T17:45:03.944+0800: 291.040: [DefNew: 282331K->2724K(314560K),
> 0.0109078 secs] 314510K->34903K(1013632K), 0.0110838 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T17:45:41.069+0800: 327.816: [GC (Allocation Failure)
> 2016-04-26T17:45:41.069+0800: 327.816: [DefNew: 282340K->2732K(314560K),
> 0.0128223 secs] 314519K->34912K(1013632K), 0.0130151 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T17:46:18.108+0800: 364.538: [GC (Allocation Failure)
> 2016-04-26T17:46:18.108+0800: 364.538: [DefNew: 282348K->2738K(314560K),
> 0.0123320 secs] 314528K->34917K(1013632K), 0.0125273 secs] [Times:
> user=0.00 sys=0.00, real=0.01 secs]
>
> 2016-04-26T17:46:54.492+0800: 400.975: [GC (Allocation Failure)
> 2016-04-26T17:46:54.492+0800: 400.975: [DefNew: 282354K->2747K(314560K),
> 0.0126284 secs] 314533K->34926K(1013632K), 0.0128229 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T17:47:36.805+0800: 443.099: [GC (Allocation Failure)
> 2016-04-26T17:47:36.805+0800: 443.099: [DefNew: 282363K->2755K(314560K),
> 0.0137048 secs] 314542K->34934K(1013632K), 0.0139138 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T17:48:13.500+0800: 480.054: [GC (Allocation Failure)
> 2016-04-26T17:48:13.500+0800: 480.054: [DefNew: 282371K->2764K(314560K),
> 0.0134777 secs] 314550K->34943K(1013632K), 0.0137571 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T17:48:50.568+0800: 517.242: [GC (Allocation Failure)
> 2016-04-26T17:48:50.568+0800: 517.242: [DefNew: 282380K->2769K(314560K),
> 0.0121474 secs] 314559K->34948K(1013632K), 0.0125373 secs] [Times:
> user=0.00 sys=0.02, real=0.02 secs]
>
> 2016-04-26T17:49:28.506+0800: 555.103: [GC (Allocation Failure)
> 2016-04-26T17:49:28.507+0800: 555.103: [DefNew: 282385K->2777K(314560K),
> 0.0133794 secs] 314564K->34957K(1013632K), 0.0135732 secs] [Times:
> user=0.01 sys=0.00, real=0.01 secs]
>
> 2016-04-26T17:50:05.869+0800: 592.355: [GC (Allocation Failure)
> 2016-04-26T17:50:05.869+0800: 592.355: [DefNew: 282393K->1713K(314560K),
> 0.0129486 secs] 314573K->34965K(1013632K), 0.0131366 secs] [Times:
> user=0.01 sys=0.00, real=0.01 secs]
>
> 2016-04-26T17:50:43.336+0800: 629.663: [GC (Allocation Failure)
> 2016-04-26T17:50:43.336+0800: 629.663: [DefNew: 281329K->1688K(314560K),
> 0.0107519 secs] 314581K->34971K(1013632K), 0.0109511 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T17:51:20.603+0800: 666.805: [GC (Allocation Failure)
> 2016-04-26T17:51:20.604+0800: 666.806: [DefNew: 281304K->2043K(314560K),
> 0.0125329 secs] 314587K->35333K(1013632K), 0.0127661 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T17:51:57.981+0800: 704.195: [GC (Allocation Failure)
> 2016-04-26T17:51:57.981+0800: 704.195: [DefNew: 281659K->1688K(314560K),
> 0.0083589 secs] 314949K->34985K(1013632K), 0.0085578 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T17:52:35.234+0800: 741.413: [GC (Allocation Failure)
> 2016-04-26T17:52:35.234+0800: 741.413: [DefNew: 281304K->1686K(314560K),
> 0.0080466 secs] 314601K->34990K(1013632K), 0.0082145 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T17:53:11.734+0800: 778.046: [GC (Allocation Failure)
> 2016-04-26T17:53:11.734+0800: 778.046: [DefNew: 281302K->1689K(314560K),
> 0.0106016 secs] 314606K->35000K(1013632K), 0.0108028 secs] [Times:
> user=0.00 sys=0.00, real=0.01 secs]
>
> 2016-04-26T17:53:48.679+0800: 814.992: [GC (Allocation Failure)
> 2016-04-26T17:53:48.679+0800: 814.992: [DefNew: 281305K->1689K(314560K),
> 0.0085276 secs] 314616K->35007K(1013632K), 0.0086721 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T17:54:24.734+0800: 851.078: [GC (Allocation Failure)
> 2016-04-26T17:54:24.735+0800: 851.078: [DefNew: 281305K->1685K(314560K),
> 0.0130525 secs] 314623K->35012K(1013632K), 0.0132933 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T17:55:01.614+0800: 888.090: [GC (Allocation Failure)
> 2016-04-26T17:55:01.614+0800: 888.090: [DefNew: 281301K->1686K(314560K),
> 0.0084343 secs] 314628K->35021K(1013632K), 0.0086352 secs] [Times:
> user=0.00 sys=0.00, real=0.01 secs]
>
> 2016-04-26T17:55:38.728+0800: 925.129: [GC (Allocation Failure)
> 2016-04-26T17:55:38.728+0800: 925.129: [DefNew: 281302K->1684K(314560K),
> 0.0081086 secs] 314637K->35026K(1013632K), 0.0082835 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T17:56:15.257+0800: 961.859: [GC (Allocation Failure)
> 2016-04-26T17:56:15.257+0800: 961.859: [DefNew: 281300K->1683K(314560K),
> 0.0097599 secs] 314642K->35032K(1013632K), 0.0099144 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T17:56:51.417+0800: 998.123: [GC (Allocation Failure)
> 2016-04-26T17:56:51.417+0800: 998.123: [DefNew: 281299K->1681K(314560K),
> 0.0095046 secs] 314648K->35038K(1013632K), 0.0096970 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T17:57:27.838+0800: 1034.605: [GC (Allocation Failure)
> 2016-04-26T17:57:27.838+0800: 1034.605: [DefNew: 281297K->1681K(314560K),
> 0.0104843 secs] 314654K->35045K(1013632K), 0.0106871 secs] [Times:
> user=0.01 sys=0.00, real=0.01 secs]
>
> 2016-04-26T17:58:04.008+0800: 1070.815: [GC (Allocation Failure)
> 2016-04-26T17:58:04.008+0800: 1070.815: [DefNew: 281297K->1680K(314560K),
> 0.0087810 secs] 314661K->35051K(1013632K), 0.0089486 secs] [Times:
> user=0.00 sys=0.00, real=0.01 secs]
>
> 2016-04-26T17:58:40.916+0800: 1107.629: [GC (Allocation Failure)
> 2016-04-26T17:58:40.916+0800: 1107.629: [DefNew: 281296K->1679K(314560K),
> 0.0123532 secs] 314667K->35058K(1013632K), 0.0125745 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T17:59:17.297+0800: 1143.843: [GC (Allocation Failure)
> 2016-04-26T17:59:17.297+0800: 1143.843: [DefNew: 281295K->1682K(314560K),
> 0.0099356 secs] 314674K->35069K(1013632K), 0.0101206 secs] [Times:
> user=0.01 sys=0.00, real=0.01 secs]
>
> 2016-04-26T17:59:54.265+0800: 1180.679: [GC (Allocation Failure)
> 2016-04-26T17:59:54.265+0800: 1180.680: [DefNew: 281298K->1678K(314560K),
> 0.0089453 secs] 314685K->35071K(1013632K), 0.0091503 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:00:31.910+0800: 1218.043: [GC (Allocation Failure)
> 2016-04-26T18:00:31.910+0800: 1218.043: [DefNew: 281294K->1677K(314560K),
> 0.0082765 secs] 314687K->35078K(1013632K), 0.0084452 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:01:07.853+0800: 1254.048: [GC (Allocation Failure)
> 2016-04-26T18:01:07.854+0800: 1254.048: [DefNew: 281293K->1678K(314560K),
> 0.0089447 secs] 314694K->35085K(1013632K), 0.0091405 secs] [Times:
> user=0.01 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:01:44.497+0800: 1290.474: [GC (Allocation Failure)
> 2016-04-26T18:01:44.497+0800: 1290.474: [DefNew: 281294K->1678K(314560K),
> 0.0076934 secs] 314701K->35091K(1013632K), 0.0078666 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:02:27.863+0800: 1333.673: [GC (Allocation Failure)
> 2016-04-26T18:02:27.863+0800: 1333.674: [DefNew: 281294K->1674K(314560K),
> 0.0084192 secs] 314707K->35098K(1013632K), 0.0085597 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:03:04.588+0800: 1370.424: [GC (Allocation Failure)
> 2016-04-26T18:03:04.589+0800: 1370.424: [DefNew: 281290K->1677K(314560K),
> 0.0086385 secs] 314714K->35107K(1013632K), 0.0087911 secs] [Times:
> user=0.00 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:03:41.646+0800: 1407.396: [GC (Allocation Failure)
> 2016-04-26T18:03:41.646+0800: 1407.396: [DefNew: 281293K->1674K(314560K),
> 0.0079049 secs] 314723K->35111K(1013632K), 0.0080862 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:04:18.618+0800: 1444.458: [GC (Allocation Failure)
> 2016-04-26T18:04:18.618+0800: 1444.458: [DefNew: 281290K->1674K(314560K),
> 0.0089752 secs] 314727K->35118K(1013632K), 0.0091841 secs] [Times:
> user=0.00 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:04:55.359+0800: 1481.281: [GC (Allocation Failure)
> 2016-04-26T18:04:55.359+0800: 1481.281: [DefNew: 281290K->1674K(314560K),
> 0.0091816 secs] 314734K->35124K(1013632K), 0.0093663 secs] [Times:
> user=0.01 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:05:31.696+0800: 1517.690: [GC (Allocation Failure)
> 2016-04-26T18:05:31.696+0800: 1517.690: [DefNew: 281290K->1680K(314560K),
> 0.0097599 secs] 314740K->35136K(1013632K), 0.0099613 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:06:08.293+0800: 1554.357: [GC (Allocation Failure)
> 2016-04-26T18:06:08.293+0800: 1554.357: [DefNew: 281296K->1681K(314560K),
> 0.0102206 secs] 314752K->35144K(1013632K), 0.0104262 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:06:44.886+0800: 1590.882: [GC (Allocation Failure)
> 2016-04-26T18:06:44.886+0800: 1590.882: [DefNew: 281297K->1680K(314560K),
> 0.0090525 secs] 314760K->35150K(1013632K), 0.0092456 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:07:20.852+0800: 1626.984: [GC (Allocation Failure)
> 2016-04-26T18:07:20.852+0800: 1626.984: [DefNew: 281296K->1682K(314560K),
> 0.0095817 secs] 314766K->35158K(1013632K), 0.0097445 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:07:57.290+0800: 1663.543: [GC (Allocation Failure)
> 2016-04-26T18:07:57.290+0800: 1663.543: [DefNew: 281298K->1682K(314560K),
> 0.0076180 secs] 314774K->35164K(1013632K), 0.0077856 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:08:34.599+0800: 1700.805: [GC (Allocation Failure)
> 2016-04-26T18:08:34.599+0800: 1700.805: [DefNew: 281298K->1680K(314560K),
> 0.0136244 secs] 314780K->35170K(1013632K), 0.0138406 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:09:10.999+0800: 1737.438: [GC (Allocation Failure)
> 2016-04-26T18:09:10.999+0800: 1737.438: [DefNew: 281296K->1681K(314560K),
> 0.0086762 secs] 314786K->35177K(1013632K), 0.0088701 secs] [Times:
> user=0.00 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:09:47.695+0800: 1774.086: [GC (Allocation Failure)
> 2016-04-26T18:09:47.695+0800: 1774.086: [DefNew: 281297K->1679K(314560K),
> 0.0079063 secs] 314793K->35182K(1013632K), 0.0080798 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:10:24.630+0800: 1811.127: [GC (Allocation Failure)
> 2016-04-26T18:10:24.630+0800: 1811.127: [DefNew: 281295K->1680K(314560K),
> 0.0100454 secs] 314798K->35189K(1013632K), 0.0101801 secs] [Times:
> user=0.00 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:11:00.972+0800: 1847.581: [GC (Allocation Failure)
> 2016-04-26T18:11:00.972+0800: 1847.581: [DefNew: 281296K->1679K(314560K),
> 0.0083639 secs] 314805K->35195K(1013632K), 0.0085452 secs] [Times:
> user=0.00 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:11:37.280+0800: 1883.810: [GC (Allocation Failure)
> 2016-04-26T18:11:37.280+0800: 1883.810: [DefNew: 281295K->1679K(314560K),
> 0.0084648 secs] 314811K->35202K(1013632K), 0.0086475 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:12:13.694+0800: 1920.386: [GC (Allocation Failure)
> 2016-04-26T18:12:13.694+0800: 1920.386: [DefNew: 281295K->1677K(314560K),
> 0.0080820 secs] 314818K->35207K(1013632K), 0.0082119 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:12:49.894+0800: 1956.802: [GC (Allocation Failure)
> 2016-04-26T18:12:49.894+0800: 1956.803: [DefNew: 281293K->1677K(314560K),
> 0.0146823 secs] 314823K->35213K(1013632K), 0.0149620 secs] [Times:
> user=0.00 sys=0.02, real=0.01 secs]
>
> 2016-04-26T18:13:26.430+0800: 1993.317: [GC (Allocation Failure)
> 2016-04-26T18:13:26.430+0800: 1993.317: [DefNew: 281293K->1677K(314560K),
> 0.0088413 secs] 314829K->35220K(1013632K), 0.0090009 secs] [Times:
> user=0.01 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:14:03.233+0800: 2030.321: [GC (Allocation Failure)
> 2016-04-26T18:14:03.233+0800: 2030.321: [DefNew: 281293K->1676K(314560K),
> 0.0078476 secs] 314836K->35225K(1013632K), 0.0079887 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:14:40.063+0800: 2067.102: [GC (Allocation Failure)
> 2016-04-26T18:14:40.063+0800: 2067.102: [DefNew: 281292K->1677K(314560K),
> 0.0092786 secs] 314841K->35232K(1013632K), 0.0094601 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:15:16.165+0800: 2103.376: [GC (Allocation Failure)
> 2016-04-26T18:15:16.165+0800: 2103.376: [DefNew: 281293K->2106K(314560K),
> 0.0135981 secs] 314848K->35675K(1013632K), 0.0138093 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:15:52.922+0800: 2139.884: [GC (Allocation Failure)
> 2016-04-26T18:15:52.922+0800: 2139.884: [DefNew: 281722K->1669K(314560K),
> 0.0123189 secs] 315291K->35244K(1013632K), 0.0125281 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:16:30.496+0800: 2177.184: [GC (Allocation Failure)
> 2016-04-26T18:16:30.496+0800: 2177.184: [DefNew: 281285K->1669K(314560K),
> 0.0089190 secs] 314860K->35251K(1013632K), 0.0090788 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:17:06.767+0800: 2213.365: [GC (Allocation Failure)
> 2016-04-26T18:17:06.767+0800: 2213.365: [DefNew: 281285K->1826K(314560K),
> 0.0079870 secs] 314867K->35415K(1013632K), 0.0081401 secs] [Times:
> user=0.01 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:17:43.991+0800: 2250.422: [GC (Allocation Failure)
> 2016-04-26T18:17:43.991+0800: 2250.422: [DefNew: 281442K->1669K(314560K),
> 0.0115439 secs] 315031K->35264K(1013632K), 0.0117428 secs] [Times:
> user=0.01 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:18:20.528+0800: 2286.733: [GC (Allocation Failure)
> 2016-04-26T18:18:20.528+0800: 2286.733: [DefNew: 281285K->1668K(314560K),
> 0.0082332 secs] 314880K->35270K(1013632K), 0.0084053 secs] [Times:
> user=0.00 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:18:57.215+0800: 2323.271: [GC (Allocation Failure)
> 2016-04-26T18:18:57.215+0800: 2323.271: [DefNew: 281284K->1671K(314560K),
> 0.0077934 secs] 314886K->35277K(1013632K), 0.0079697 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:19:34.151+0800: 2359.977: [GC (Allocation Failure)
> 2016-04-26T18:19:34.151+0800: 2359.977: [DefNew: 281287K->1669K(314560K),
> 0.0078809 secs] 314893K->35283K(1013632K), 0.0080410 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:20:11.106+0800: 2396.706: [GC (Allocation Failure)
> 2016-04-26T18:20:11.106+0800: 2396.706: [DefNew: 281285K->1673K(314560K),
> 0.0092928 secs] 314899K->35293K(1013632K), 0.0094864 secs] [Times:
> user=0.00 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:20:47.129+0800: 2432.801: [GC (Allocation Failure)
> 2016-04-26T18:20:47.129+0800: 2432.801: [DefNew: 281289K->1672K(314560K),
> 0.0089699 secs] 314909K->35298K(1013632K), 0.0091397 secs] [Times:
> user=0.01 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:21:23.028+0800: 2468.633: [GC (Allocation Failure)
> 2016-04-26T18:21:23.028+0800: 2468.633: [DefNew: 281288K->1671K(314560K),
> 0.0124404 secs] 314914K->35303K(1013632K), 0.0128810 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:21:59.745+0800: 2505.506: [GC (Allocation Failure)
> 2016-04-26T18:21:59.745+0800: 2505.506: [DefNew: 281287K->1671K(314560K),
> 0.0080019 secs] 314919K->35309K(1013632K), 0.0082167 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:22:36.639+0800: 2542.570: [GC (Allocation Failure)
> 2016-04-26T18:22:36.639+0800: 2542.570: [DefNew: 281287K->1670K(314560K),
> 0.0082147 secs] 314925K->35315K(1013632K), 0.0083938 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:23:12.390+0800: 2578.421: [GC (Allocation Failure)
> 2016-04-26T18:23:12.390+0800: 2578.421: [DefNew: 281286K->2207K(314560K),
> 0.0114121 secs] 314931K->35857K(1013632K), 0.0116115 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:23:48.693+0800: 2615.071: [GC (Allocation Failure)
> 2016-04-26T18:23:48.693+0800: 2615.071: [DefNew: 281823K->1798K(314560K),
> 0.0104047 secs] 315473K->35455K(1013632K), 0.0105935 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:24:25.698+0800: 2652.227: [GC (Allocation Failure)
> 2016-04-26T18:24:25.698+0800: 2652.228: [DefNew: 281414K->1795K(314560K),
> 0.0102968 secs] 315071K->35459K(1013632K), 0.0104854 secs] [Times:
> user=0.00 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:25:01.801+0800: 2688.457: [GC (Allocation Failure)
> 2016-04-26T18:25:01.801+0800: 2688.458: [DefNew: 281411K->2126K(314560K),
> 0.0090964 secs] 315075K->35795K(1013632K), 0.0093529 secs] [Times:
> user=0.00 sys=0.02, real=0.01 secs]
>
> 2016-04-26T18:25:39.055+0800: 2725.732: [GC (Allocation Failure)
> 2016-04-26T18:25:39.055+0800: 2725.732: [DefNew: 281742K->1799K(314560K),
> 0.0085785 secs] 315411K->35474K(1013632K), 0.0087168 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:26:27.232+0800: 2773.853: [GC (Allocation Failure)
> 2016-04-26T18:26:27.232+0800: 2773.853: [DefNew: 281415K->1800K(314560K),
> 0.0091224 secs] 315090K->35482K(1013632K), 0.0094059 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:27:05.583+0800: 2812.453: [GC (Allocation Failure)
> 2016-04-26T18:27:05.584+0800: 2812.453: [DefNew: 281416K->1801K(314560K),
> 0.0083206 secs] 315098K->35489K(1013632K), 0.0084969 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:27:43.412+0800: 2850.382: [GC (Allocation Failure)
> 2016-04-26T18:27:43.412+0800: 2850.382: [DefNew: 281417K->1801K(314560K),
> 0.0095576 secs] 315105K->35495K(1013632K), 0.0097616 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:28:20.067+0800: 2887.191: [GC (Allocation Failure)
> 2016-04-26T18:28:20.067+0800: 2887.191: [DefNew: 281417K->1799K(314560K),
> 0.0121244 secs] 315111K->35501K(1013632K), 0.0123463 secs] [Times:
> user=0.01 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:28:55.431+0800: 2923.064: [GC (Allocation Failure)
> 2016-04-26T18:28:55.431+0800: 2923.064: [DefNew: 281415K->1798K(314560K),
> 0.0088000 secs] 315117K->35506K(1013632K), 0.0089771 secs] [Times:
> user=0.01 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:29:30.761+0800: 2958.335: [GC (Allocation Failure)
> 2016-04-26T18:29:30.761+0800: 2958.335: [DefNew: 281414K->1798K(314560K),
> 0.0113492 secs] 315122K->35512K(1013632K), 0.0115526 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:30:05.795+0800: 2993.528: [GC (Allocation Failure)
> 2016-04-26T18:30:05.795+0800: 2993.528: [DefNew: 281414K->1797K(314560K),
> 0.0112305 secs] 315128K->35518K(1013632K), 0.0114227 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:30:41.648+0800: 3029.322: [GC (Allocation Failure)
> 2016-04-26T18:30:41.648+0800: 3029.322: [DefNew: 281413K->1797K(314560K),
> 0.0174930 secs] 315134K->35524K(1013632K), 0.0177282 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:31:34.467+0800: 3082.237: [GC (Allocation Failure)
> 2016-04-26T18:31:34.467+0800: 3082.237: [DefNew: 281413K->1800K(314560K),
> 0.0095434 secs] 315140K->35534K(1013632K), 0.0097186 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:32:14.021+0800: 3121.810: [GC (Allocation Failure)
> 2016-04-26T18:32:14.021+0800: 3121.810: [DefNew: 281416K->1800K(314560K),
> 0.0096049 secs] 315150K->35540K(1013632K), 0.0097856 secs] [Times:
> user=0.00 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:33:07.874+0800: 3175.537: [GC (Allocation Failure)
> 2016-04-26T18:33:07.874+0800: 3175.537: [DefNew: 281416K->1697K(314560K),
> 0.0135850 secs] 315156K->35550K(1013632K), 0.0138194 secs] [Times:
> user=0.00 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:33:47.621+0800: 3215.029: [GC (Allocation Failure)
> 2016-04-26T18:33:47.621+0800: 3215.029: [DefNew: 281313K->1694K(314560K),
> 0.0080256 secs] 315166K->35566K(1013632K), 0.0081854 secs] [Times:
> user=0.01 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:34:24.160+0800: 3251.370: [GC (Allocation Failure)
> 2016-04-26T18:34:24.160+0800: 3251.371: [DefNew: 281310K->1691K(314560K),
> 0.0086257 secs] 315182K->35570K(1013632K), 0.0087810 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:34:59.507+0800: 3286.713: [GC (Allocation Failure)
> 2016-04-26T18:34:59.507+0800: 3286.713: [DefNew: 281307K->1690K(314560K),
> 0.0082689 secs] 315186K->35576K(1013632K), 0.0084583 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:35:34.974+0800: 3322.008: [GC (Allocation Failure)
> 2016-04-26T18:35:34.974+0800: 3322.008: [DefNew: 281306K->1690K(314560K),
> 0.0134400 secs] 315192K->35582K(1013632K), 0.0136509 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:36:11.007+0800: 3357.907: [GC (Allocation Failure)
> 2016-04-26T18:36:11.007+0800: 3357.907: [DefNew: 281306K->1689K(314560K),
> 0.0118568 secs] 315198K->35590K(1013632K), 0.0120744 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:36:48.227+0800: 3394.956: [GC (Allocation Failure)
> 2016-04-26T18:36:48.227+0800: 3394.956: [DefNew: 281305K->1688K(314560K),
> 0.0083156 secs] 315206K->35596K(1013632K), 0.0084603 secs] [Times:
> user=0.03 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:37:24.551+0800: 3430.991: [GC (Allocation Failure)
> 2016-04-26T18:37:24.551+0800: 3430.991: [DefNew: 281304K->1688K(314560K),
> 0.0091559 secs] 315212K->35602K(1013632K), 0.0093369 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:38:01.307+0800: 3467.736: [GC (Allocation Failure)
> 2016-04-26T18:38:01.307+0800: 3467.736: [DefNew: 281304K->1685K(314560K),
> 0.0085799 secs] 315218K->35609K(1013632K), 0.0087592 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:38:37.590+0800: 3503.935: [GC (Allocation Failure)
> 2016-04-26T18:38:37.590+0800: 3503.935: [DefNew: 281301K->1685K(314560K),
> 0.0103468 secs] 315225K->35616K(1013632K), 0.0105709 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:39:14.026+0800: 3540.397: [GC (Allocation Failure)
> 2016-04-26T18:39:14.026+0800: 3540.397: [DefNew: 281301K->1685K(314560K),
> 0.0119247 secs] 315232K->35622K(1013632K), 0.0121515 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:39:50.866+0800: 3577.377: [GC (Allocation Failure)
> 2016-04-26T18:39:50.866+0800: 3577.377: [DefNew: 281301K->1687K(314560K),
> 0.0134316 secs] 315238K->35630K(1013632K), 0.0136453 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:40:28.204+0800: 3614.598: [GC (Allocation Failure)
> 2016-04-26T18:40:28.204+0800: 3614.598: [DefNew: 281303K->1687K(314560K),
> 0.0093311 secs] 315246K->35635K(1013632K), 0.0095685 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:41:04.400+0800: 3650.792: [GC (Allocation Failure)
> 2016-04-26T18:41:04.400+0800: 3650.792: [DefNew: 281303K->1684K(314560K),
> 0.0114953 secs] 315251K->35642K(1013632K), 0.0116892 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:41:41.313+0800: 3687.569: [GC (Allocation Failure)
> 2016-04-26T18:41:41.313+0800: 3687.569: [DefNew: 281300K->1683K(314560K),
> 0.0078890 secs] 315258K->35648K(1013632K), 0.0080298 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:42:16.550+0800: 3722.774: [GC (Allocation Failure)
> 2016-04-26T18:42:16.550+0800: 3722.774: [DefNew: 281299K->1679K(314560K),
> 0.0124507 secs] 315264K->35654K(1013632K), 0.0126969 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:42:52.776+0800: 3759.034: [GC (Allocation Failure)
> 2016-04-26T18:42:52.776+0800: 3759.034: [DefNew: 281295K->1673K(314560K),
> 0.0105041 secs] 315270K->35661K(1013632K), 0.0106768 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:43:28.412+0800: 3794.638: [GC (Allocation Failure)
> 2016-04-26T18:43:28.412+0800: 3794.638: [DefNew: 281289K->1674K(314560K),
> 0.0103795 secs] 315277K->35669K(1013632K), 0.0105525 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:44:04.536+0800: 3830.831: [GC (Allocation Failure)
> 2016-04-26T18:44:04.536+0800: 3830.831: [DefNew: 281290K->1674K(314560K),
> 0.0079088 secs] 315285K->35675K(1013632K), 0.0080555 secs] [Times:
> user=0.00 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:44:40.190+0800: 3866.449: [GC (Allocation Failure)
> 2016-04-26T18:44:40.190+0800: 3866.449: [DefNew: 281290K->1673K(314560K),
> 0.0095096 secs] 315291K->35681K(1013632K), 0.0096861 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:45:16.528+0800: 3902.765: [GC (Allocation Failure)
> 2016-04-26T18:45:16.528+0800: 3902.765: [DefNew: 281289K->1673K(314560K),
> 0.0130461 secs] 315297K->35687K(1013632K), 0.0132673 secs] [Times:
> user=0.01 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:45:52.574+0800: 3939.001: [GC (Allocation Failure)
> 2016-04-26T18:45:52.574+0800: 3939.001: [DefNew: 281289K->1680K(314560K),
> 0.0101569 secs] 315303K->35700K(1013632K), 0.0103066 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:46:28.413+0800: 3975.223: [GC (Allocation Failure)
> 2016-04-26T18:46:28.413+0800: 3975.223: [DefNew: 281296K->1679K(314560K),
> 0.0080063 secs] 315316K->35706K(1013632K), 0.0081820 secs] [Times:
> user=0.00 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:47:04.190+0800: 4011.234: [GC (Allocation Failure)
> 2016-04-26T18:47:04.190+0800: 4011.234: [DefNew: 281295K->1678K(314560K),
> 0.0102572 secs] 315322K->35712K(1013632K), 0.0104334 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:47:40.161+0800: 4047.120: [GC (Allocation Failure)
> 2016-04-26T18:47:40.161+0800: 4047.120: [DefNew: 281294K->1677K(314560K),
> 0.0109581 secs] 315328K->35718K(1013632K), 0.0111609 secs] [Times:
> user=0.01 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:48:15.352+0800: 4082.428: [GC (Allocation Failure)
> 2016-04-26T18:48:15.352+0800: 4082.428: [DefNew: 281293K->1677K(314560K),
> 0.0087567 secs] 315334K->35724K(1013632K), 0.0089472 secs] [Times:
> user=0.00 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:48:51.254+0800: 4118.407: [GC (Allocation Failure)
> 2016-04-26T18:48:51.254+0800: 4118.407: [DefNew: 281293K->1677K(314560K),
> 0.0115241 secs] 315340K->35731K(1013632K), 0.0117504 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:49:26.547+0800: 4153.705: [GC (Allocation Failure)
> 2016-04-26T18:49:26.547+0800: 4153.705: [DefNew: 281293K->1967K(314560K),
> 0.0095867 secs] 315347K->36027K(1013632K), 0.0097683 secs] [Times:
> user=0.01 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:50:02.043+0800: 4189.206: [GC (Allocation Failure)
> 2016-04-26T18:50:02.043+0800: 4189.206: [DefNew: 281583K->1676K(314560K),
> 0.0096753 secs] 315643K->35742K(1013632K), 0.0098599 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:50:37.663+0800: 4224.505: [GC (Allocation Failure)
> 2016-04-26T18:50:37.663+0800: 4224.505: [DefNew: 281292K->1675K(314560K),
> 0.0077731 secs] 315358K->35749K(1013632K), 0.0079524 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:51:13.058+0800: 4259.780: [GC (Allocation Failure)
> 2016-04-26T18:51:13.058+0800: 4259.780: [DefNew: 281291K->1676K(314560K),
> 0.0086972 secs] 315365K->35756K(1013632K), 0.0088771 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:51:48.677+0800: 4295.269: [GC (Allocation Failure)
> 2016-04-26T18:51:48.677+0800: 4295.269: [DefNew: 281292K->1931K(314560K),
> 0.0099264 secs] 315372K->36016K(1013632K), 0.0101337 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:52:23.956+0800: 4330.493: [GC (Allocation Failure)
> 2016-04-26T18:52:23.956+0800: 4330.493: [DefNew: 281547K->1673K(314560K),
> 0.0079896 secs] 315632K->35766K(1013632K), 0.0081558 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:52:59.459+0800: 4366.025: [GC (Allocation Failure)
> 2016-04-26T18:52:59.459+0800: 4366.025: [DefNew: 281289K->1942K(314560K),
> 0.0099881 secs] 315382K->36041K(1013632K), 0.0101999 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:53:35.248+0800: 4401.571: [GC (Allocation Failure)
> 2016-04-26T18:53:35.248+0800: 4401.571: [DefNew: 281558K->1675K(314560K),
> 0.0080323 secs] 315657K->35779K(1013632K), 0.0081949 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:54:11.231+0800: 4437.434: [GC (Allocation Failure)
> 2016-04-26T18:54:11.231+0800: 4437.434: [DefNew: 281291K->1672K(314560K),
> 0.0107709 secs] 315395K->35785K(1013632K), 0.0109682 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:54:47.360+0800: 4473.741: [GC (Allocation Failure)
> 2016-04-26T18:54:47.360+0800: 4473.741: [DefNew: 281288K->1675K(314560K),
> 0.0077845 secs] 315401K->35796K(1013632K), 0.0079611 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:55:22.937+0800: 4509.361: [GC (Allocation Failure)
> 2016-04-26T18:55:22.937+0800: 4509.361: [DefNew: 281291K->1675K(314560K),
> 0.0082094 secs] 315412K->35802K(1013632K), 0.0083421 secs] [Times:
> user=0.01 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:55:58.273+0800: 4544.708: [GC (Allocation Failure)
> 2016-04-26T18:55:58.273+0800: 4544.708: [DefNew: 281291K->1674K(314560K),
> 0.0100806 secs] 315418K->35807K(1013632K), 0.0102764 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:56:33.184+0800: 4579.506: [GC (Allocation Failure)
> 2016-04-26T18:56:33.184+0800: 4579.507: [DefNew: 281290K->1674K(314560K),
> 0.0111419 secs] 315423K->35814K(1013632K), 0.0113590 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:57:09.091+0800: 4615.482: [GC (Allocation Failure)
> 2016-04-26T18:57:09.091+0800: 4615.482: [DefNew: 281290K->1672K(314560K),
> 0.0139571 secs] 315430K->35818K(1013632K), 0.0141755 secs] [Times:
> user=0.00 sys=0.00, real=0.01 secs]
>
> 2016-04-26T18:57:45.582+0800: 4651.989: [GC (Allocation Failure)
> 2016-04-26T18:57:45.582+0800: 4651.989: [DefNew: 281288K->1672K(314560K),
> 0.0089576 secs] 315434K->35824K(1013632K), 0.0091738 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:58:21.593+0800: 4688.001: [GC (Allocation Failure)
> 2016-04-26T18:58:21.593+0800: 4688.002: [DefNew: 281288K->1676K(314560K),
> 0.0158752 secs] 315440K->35835K(1013632K), 0.0160361 secs] [Times:
> user=0.00 sys=0.00, real=0.02 secs]
>
> 2016-04-26T18:58:56.204+0800: 4722.662: [GC (Allocation Failure)
> 2016-04-26T18:58:56.204+0800: 4722.662: [DefNew: 281292K->1675K(314560K),
> 0.0076781 secs] 315451K->35841K(1013632K), 0.0078622 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T18:59:31.386+0800: 4757.760: [GC (Allocation Failure)
> 2016-04-26T18:59:31.386+0800: 4757.760: [DefNew: 281291K->1678K(314560K),
> 0.0079359 secs] 315457K->35849K(1013632K), 0.0080991 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T19:00:07.775+0800: 4794.332: [GC (Allocation Failure)
> 2016-04-26T19:00:07.775+0800: 4794.332: [DefNew: 281294K->1676K(314560K),
> 0.0087637 secs] 315465K->35854K(1013632K), 0.0089416 secs] [Times:
> user=0.02 sys=0.00, real=0.01 secs]
>
> 2016-04-26T19:00:44.243+0800: 4830.820: [GC (Allocation Failure)
> 2016-04-26T19:00:44.243+0800: 4830.820: [DefNew: 281292K->1674K(314560K),
> 0.0098652 secs] 315470K->35859K(1013632K), 0.0101359 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T19:01:19.025+0800: 4865.773: [GC (Allocation Failure)
> 2016-04-26T19:01:19.025+0800: 4865.773: [DefNew: 281290K->1686K(314560K),
> 0.0077619 secs] 315475K->35877K(1013632K), 0.0079340 secs] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
>
> 2016-04-26T19:01:47.622+0800: 4894.541: [GC (Allocation Failure)
> 2016-04-26T19:01:47.622+0800: 4894.541: [DefNew: 281302K->2540K(314560K),
> 0.0189695 secs] 315493K->36737K(1013632K), 0.0192055 secs] [Times:
> user=0.02 sys=0.00, real=0.02 secs]
>
> Heap
>
> def new generation total 314560K, used 268425K [0x04600000, 0x19b50000,
> 0x19b50000)
>
> eden space 279616K, 95% used [0x04600000, 0x149a73e0, 0x15710000)
>
> from space 34944K, 7% used [0x17930000, 0x17bab258, 0x19b50000)
>
> to space 34944K, 0% used [0x15710000, 0x15710000, 0x17930000)
>
> tenured generation total 699072K, used 34196K [0x19b50000, 0x44600000,
> 0x44600000)
>
> the space 699072K, 4% used [0x19b50000, 0x1bcb5300, 0x1bcb5400,
> 0x44600000)
>
> Metaspace used 21496K, capacity 21761K, committed 21888K, reserved
> 21888K
>
>
>
>
>
> *发件人:* Vladimir Ozerov [mailto:voze...@gridgain.com]
> *发送时间:* 2016年4月25日 20:21
> *收件人:* user@ignite.apache.org
> *主题:* Re: Ignite cache data size problem.
>
>
>
> Hi Kevin,
>
>
>
> I performed several experiments. Essentially, I put 1M entries of the
> class you provided with fields initialized as follows:
>
>
>
> *for *(*int *i = 0; i < 1_000_000; i++) {
> UniqueField field = *new *UniqueField();
>
> field.setDate(*new *Date());
> field.setGuid(UUID.*randomUUID*().toString());
> field.setMsg(String.*valueOf*(i));
>
> field.setNum(BigDecimal.*valueOf*(ThreadLocalRandom.*current*().nextDouble()));
> field.setOId(String.*valueOf*(i));
> field.setOrgId(String.*valueOf*(i));
>
> cache.put(i, field);
> }
>
>
>
> My results are:
>
> 1) Onheap, no indexes - about 400Mb is required to store 1M objects, or
> ~20Gb for 47M objects.
>
> 2) Onheap, with indexes - about 650Mb, or ~30Gb for 47M objects.
>
> 3) Offheap, with indexes - about 400Mb offheap memory is required, or
> ~20Gb for all your objects.
>
>
>
> Could you please provide more information on the error you receive? Also,
> you could try load entries in a batches of a well-defined size (say, 1M),
> and see what happens to the system. I expect you should see similar numbers.
>
>
>
> Vladimir.
>
>
>
> On Fri, Apr 22, 2016 at 3:26 PM, kevin.zheng <zhengqingzh...@huawei.com>
> wrote:
>
> BTW, I created 4 + 3 nodes on two servers.
> each node I called command like this ./ignite.sh -J-Xmx8g -J-Xms8g
>
> kind regards,
> Kevin
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/Ignite-cache-data-size-problem-tp4449p4454.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
>
>
