I did observe the same thing with ignite and was not able to resolve it at all. So I am very interested on a resolution for this too.
> On Feb 25, 2019, at 9:30 AM, Antonio Conforti <antonio.confo...@sia.eu> wrote: > > Hello, > > first of all thanks for your advice. > > Before reading your post I runned another session (at 15:45 about) of test > with the same rate before changing the frequency of checkpoint you suggested > but with data already loaded. > > > I observed that performance degrades suddenly and not after a while (about > 45 minutes) as observed in my first session when the cache was empty. > Below you can see the statistics: > > 2019-02-22 15:49:03.992 INFO 5271 --- [oint-thread-#67] > i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished > [cpId=ab04c732-a72c-4398-8a09-31f05243a7ad, pages=129721, > markPos=FileWALPointer [idx=470, fileOff=35838781, len=79426], > walSegmentsCleared=11, walSegmentsCovered=[460 - 469], markDuration=387ms, > pagesWrite=1149ms, fsync=227229ms, total=228765ms] > 2019-02-22 15:53:25.681 INFO 5271 --- [oint-thread-#67] > i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished > [cpId=cf6e6d68-ac4d-4445-becb-b7bcbb5aee32, pages=178943, > markPos=FileWALPointer [idx=484, fileOff=61358243, len=79426], > walSegmentsCleared=14, walSegmentsCovered=[470 - 483], markDuration=1316ms, > pagesWrite=1043ms, fsync=259329ms, total=261688ms] > 2019-02-22 15:56:07.878 INFO 5271 --- [oint-thread-#67] > i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished > [cpId=86154f6f-907e-4972-b0f1-f86be06c3a40, pages=44514, > markPos=FileWALPointer [idx=487, fileOff=66267301, len=79426], > walSegmentsCleared=0, walSegmentsCovered=[484 - 486], markDuration=888ms, > pagesWrite=239ms, fsync=161070ms, total=162197ms] > > What looks strange in this second session is that I see a delay between 228 > and 260 sec to write 129721 / 178943 pages. > I don't observe this behaviour when I start from scratch (with no data in > cache): it usually takes about 45 minutes before observing this performance > degradation. > Any clues on what can be the cause? > > > I started today another test session with the suggested frequency > (checkpointingFrequency=600000) and I also configured direct IO and set the > WAL mode to NONE as suggested in tuning performance but the result observed > is quite the same > > > > 2019-02-25 09:34:02.284 INFO 1208 --- [oint-thread-#66] > i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished > [cpId=a9423ba9-9d93-40a3-bedb-7379a617f716, pages=78, markPos=FileWALPointer > [idx=0, fileOff=0, len=0], walSegmentsCleared=0, walSegmentsCovered=[], > markDuration=13ms, pagesWrite=15ms, fsync=11ms, total=39ms] > 2019-02-25 10:04:04.082 INFO 1208 --- [oint-thread-#66] > i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished > [cpId=bd76a704-e125-42e1-b2a8-d285cf15246d, pages=52252, > markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0, > walSegmentsCovered=[], markDuration=144ms, pagesWrite=1560ms, fsync=123ms, > total=1827ms] > 2019-02-25 10:14:27.275 INFO 1208 --- [oint-thread-#66] > i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished > [cpId=a10f5483-050c-4678-98bd-deca7695e782, pages=368810, > markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0, > walSegmentsCovered=[], markDuration=323ms, pagesWrite=24669ms, fsync=21ms, > total=25013ms] > 2019-02-25 10:24:55.492 INFO 1208 --- [oint-thread-#66] > i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished > [cpId=bfab59c6-a1a0-4391-92e5-a23b6c416625, pages=382555, > markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0, > walSegmentsCovered=[], markDuration=225ms, pagesWrite=52912ms, fsync=93ms, > total=53230ms] > 2019-02-25 10:35:22.512 INFO 1208 --- [oint-thread-#66] > i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished > [cpId=feff29e3-db80-4002-82f8-2011e4eca12e, pages=436129, > markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0, > walSegmentsCovered=[], markDuration=70ms, pagesWrite=80164ms, fsync=7ms, > total=80241ms] > 2019-02-25 10:45:34.524 INFO 1208 --- [oint-thread-#66] > i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished > [cpId=7e3f181b-21d8-4dd1-a018-09711142caf4, pages=384415, > markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0, > walSegmentsCovered=[], markDuration=56ms, pagesWrite=92186ms, fsync=3ms, > total=92245ms] > 2019-02-25 10:55:43.139 INFO 1208 --- [oint-thread-#66] > i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished > [cpId=a77d48d8-f806-4a48-bfbf-d1afac7d16e3, pages=161643, > markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0, > walSegmentsCovered=[], markDuration=73ms, pagesWrite=100765ms, fsync=16ms, > total=100854ms] > 2019-02-25 11:06:04.437 INFO 1208 --- [oint-thread-#66] > i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished > [cpId=dbf6a8ab-1c47-4cdd-96f4-fec852dc4fe0, pages=129477, > markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0, > walSegmentsCovered=[], markDuration=60ms, pagesWrite=121933ms, fsync=154ms, > total=122147ms] > 2019-02-25 11:16:08.333 INFO 1208 --- [oint-thread-#66] > i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished > [cpId=74924992-3dc4-42db-971c-b9030f41d0c5, pages=97511, > markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0, > walSegmentsCovered=[], markDuration=1676ms, pagesWrite=124336ms, fsync=25ms, > total=126037ms] > 2019-02-25 11:18:53.622 INFO 1208 --- [oint-thread-#66] > i.i.p.c.p.GridCacheDatabaseSharedManager : Checkpoint finished > [cpId=7c60efc2-4007-4be2-b3c8-3912d2cbd651, pages=19277, > markPos=FileWALPointer [idx=0, fileOff=0, len=0], walSegmentsCleared=0, > walSegmentsCovered=[], markDuration=12ms, pagesWrite=47883ms, fsync=344ms, > total=48239ms] > > > Do you have any other suggestions considering also the behaviour described > above? > > Thanks, > Antonio > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/
