Hello! My recommendation is to increase time between checkpoints (checkpointFrequency) and, maybe, also increase checkpoint page buffer.
If you do data loading in bursts, it may make sense to turn off WAL why you do data loading. You may still expect some data degradation as time goes on (what's your total allocated after the last 2.5 million?) Regards, -- Ilya Kasnacheev пн, 27 апр. 2020 г. в 00:32, Shubham Agrawal <agrawalshubham1...@gmail.com>: > Hi Team, > > I am using ignite 2.7.5. > > I am trying to insert 25 million records in the ignite persistent table. > > Observations: > I am observing that initial 2.5 million records took 5 mins to insert, > next 2.5 million records took 8 mins to insert, following 2.5 records took > 12 mins to insert and it kept on increasing . The last 2.5 million records > took over 40 minutes to insert. The CPU of the client and server is around > 30%, so nothing on that front. Heap usage looks normal. I have tried > inserting to the table with and without indexes but nothing seems to make a > difference. > > Configurations: > I am running a 3 node cluster, 16 Cores 32 GB machines. Ignite Version: > 2.7.5 > > Things I have tried: > JDBI Implementation - > 1. @SQLBatch does not work for ignite giving the below error > org.skife.jdbi.v2.exceptions.TransactionException: Failed to start > transaction at > org.skife.jdbi.v2.tweak.transactions.LocalTransactionHandler.begin(LocalTransactionHandler.java:57) > at org.skife.jdbi.v2.BasicHandle.begin(BasicHandle.java:159) > > 2. Tried ?streaming=true, but with no effect > > 3. Increasing threads to insert, but not much effect > > JDBC Implementation > 1. Tried ?streaming=true, but with no effect > 2. Tried batching, but the performance actually degraded > 3. Tried ?streaming=true with and without batching but with no effect. > > Want to achieve a common pattern in insertions, like 5 mins or 8 mins > constantly for 25 million insertions > > Please let me know your thoughts on the same. Your inputs could help me a > lot. Thanks a lot. > > Regards, > Shubham Agrawal >
