Hello! Your understanding is correct.
Yes, if you have more RAM and larger data region, the process should slow down less rapidly. Regards, -- Ilya Kasnacheev пн, 27 апр. 2020 г. в 17:56, Shubham Agrawal <agrawalshubham1...@gmail.com>: > Hi Ilya, > > This data loading is not a one time activity for our scenario. So, I guess > turning off the WAL would not be a solution since turning off and on every > time is not feasible every time the data loading happens. Please let me > know if my understanding is correct. > > I'll try to increase time between checkpoints and also increase checkpoint > page buffer and let you know. > > In the meantime, just wanted to check if the Disk I/O and memory also has > an effect? Currently, I am using 32 GB machine and increasing the memory > will make any difference? > > Thanks & Regards, > Shubham Agrawal > > > On Mon, Apr 27, 2020 at 7:31 PM Ilya Kasnacheev <ilya.kasnach...@gmail.com> > wrote: > >> Hello! >> >> My recommendation is to increase time between checkpoints >> (checkpointFrequency) and, maybe, also increase checkpoint page buffer. >> >> If you do data loading in bursts, it may make sense to turn off WAL why >> you do data loading. >> >> You may still expect some data degradation as time goes on (what's your >> total allocated after the last 2.5 million?) >> >> Regards, >> -- >> Ilya Kasnacheev >> >> >> пн, 27 апр. 2020 г. в 00:32, Shubham Agrawal < >> agrawalshubham1...@gmail.com>: >> >>> Hi Team, >>> >>> I am using ignite 2.7.5. >>> >>> I am trying to insert 25 million records in the ignite persistent table. >>> >>> Observations: >>> I am observing that initial 2.5 million records took 5 mins to insert, >>> next 2.5 million records took 8 mins to insert, following 2.5 records took >>> 12 mins to insert and it kept on increasing . The last 2.5 million records >>> took over 40 minutes to insert. The CPU of the client and server is around >>> 30%, so nothing on that front. Heap usage looks normal. I have tried >>> inserting to the table with and without indexes but nothing seems to make a >>> difference. >>> >>> Configurations: >>> I am running a 3 node cluster, 16 Cores 32 GB machines. Ignite Version: >>> 2.7.5 >>> >>> Things I have tried: >>> JDBI Implementation - >>> 1. @SQLBatch does not work for ignite giving the below error >>> org.skife.jdbi.v2.exceptions.TransactionException: Failed to start >>> transaction at >>> org.skife.jdbi.v2.tweak.transactions.LocalTransactionHandler.begin(LocalTransactionHandler.java:57) >>> at org.skife.jdbi.v2.BasicHandle.begin(BasicHandle.java:159) >>> >>> 2. Tried ?streaming=true, but with no effect >>> >>> 3. Increasing threads to insert, but not much effect >>> >>> JDBC Implementation >>> 1. Tried ?streaming=true, but with no effect >>> 2. Tried batching, but the performance actually degraded >>> 3. Tried ?streaming=true with and without batching but with no effect. >>> >>> Want to achieve a common pattern in insertions, like 5 mins or 8 mins >>> constantly for 25 million insertions >>> >>> Please let me know your thoughts on the same. Your inputs could help me >>> a lot. Thanks a lot. >>> >>> Regards, >>> Shubham Agrawal >>> >>
