Re: Performance of persistent store too low when bulb-loading
I also had the same problem here. I'm using ignite-spark's saveParis method(which is also a IgniteDataStreamer) to ingest 550 million entries of data into Ignite, and the ingesting speed got slow down after first few minutes and seems stuck for now with persistent store enabled. My setup is 4 nodes with 16GB heap size and 32GB off heap size. Here's the dstat result for about a minute when the ingesting is stuck. [7ltotal-cpu-usage --memory-usage- -dsk/total- ---paging-- swap--- --filesystem- usr sys idl wai hiq siq| used buff cach free| read writ| in out | used free|files inodes 1 0 99 0 0 0|89.9G 3970M 157G 507G| 30k 967k| 0 0 | 0 0 | 8176453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 012M| 0 0 | 0 0 | 8176453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 0 9548k| 0 0 | 0 0 | 8176453k 0 0 96 3 0 0|89.9G 3970M 157G 507G| 018M| 0 0 | 0 0 | 8176453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 017M| 0 0 | 0 0 | 8176453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 017M| 0 0 | 0 0 | 8176453k 0 0 96 3 0 0|89.9G 3970M 157G 507G| 016M| 0 0 | 0 0 | 8176453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 015M| 0 0 | 0 0 | 8176453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 017M| 0 0 | 0 0 | 8176453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 016M| 0 0 | 0 0 | 8176453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 018M| 0 0 | 0 0 | 8176453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 017M| 0 0 | 0 0 | 8176453k 2 1 94 3 0 0|89.9G 3970M 157G 507G| 015M| 0 0 | 0 0 | 8176453k 1 0 96 3 0 0|89.9G 3970M 157G 507G| 024M| 0 0 | 0 0 | 8176453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 025M| 0 0 | 0 0 | 8176453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 020M| 0 0 | 0 0 | 8176453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 026M| 0 0 | 0 0 | 8176453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 016M| 0 0 | 0 0 | 8176453k 0 0 96 3 0 0|89.9G 3970M 157G 507G| 010M| 0 0 | 0 0 | 8176453k 1 0 96 3 0 0|89.9G 3970M 157G 507G| 024M| 0 0 | 0 0 | 8176453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 020M| 0 0 | 0 0 | 8176453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 025M| 0 0 | 0 0 | 8176453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 031M| 0 0 | 0 0 | 8176453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 023M| 0 0 | 0 0 | 8176453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 021M| 0 0 | 0 0 | 8176453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 021M| 0 0 | 0 0 | 8176453k 0 0 96 3 0 0|89.9G 3970M 157G 507G| 027M| 0 0 | 0 0 | 8176453k 0 0 96 3 0 0|89.9G 3970M 157G 507G| 024M| 0 0 | 0 0 | 8176453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 027M| 0 0 | 0 0 | 8176453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 016M| 0 0 | 0 0 | 8176453k 0 0 96 3 0 0|89.9G 3970M 157G 507G| 020M| 0 0 | 0 0 | 8176453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 028M| 0 0 | 0 0 | 8176453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 022M| 0 0 | 0 0 | 8176453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 024M| 0 0 | 0 0 | 8176453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 027M| 0 0 | 0 0 | 8176453k 1 0 96 3 0 0|89.9G 3970M 157G 507G| 027M| 0 0 | 0 0 | 8176453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 024M| 0 0 | 0 0 | 8176453k 1 0 96 3 0 0|89.9G 3970M 157G 507G| 029M| 0 0 | 0 0 | 8176453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 027M| 0 0 | 0 0 | 8176453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 023M| 0 0 | 0 0 | 8176453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 017M| 0 0 | 0 0 | 8176453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 062M| 0 0 | 0 0 | 8176453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 016M| 0 0 | 0 0 | 8176453k 0 0 96 3 0 0|89.9G 3970M 157G 507G| 017M| 0 0 | 0 0 | 8176453k 0 0 97 3 0 0|89.9G 3970M 157G 507G| 020M| 0
Re: Performance of persistent store too low when bulb-loading
Hi, Had you any chance to get GC and dstat logs? Thanks! -Dmitry. -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Performance-of-persistent-store-too-low-when-bulb-loading-tp16247p16417.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Performance of persistent store too low when bulb-loading
Use this command for dstat: dstat -cmdgs --fs > dstat.txt Thanks! -Dmitry. -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Performance-of-persistent-store-too-low-when-bulb-loading-tp16247p16345.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Performance of persistent store too low when bulb-loading
Hi, Could you please take dstat, Ignite and GC logs and few thread dumps when application is experiencing slowdown? Thanks! -Dmitry. -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Performance-of-persistent-store-too-low-when-bulb-loading-tp16247p16342.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Performance of persistent store too low when bulb-loading
Hi, The reason of slowdown might be the fact, that Ignite by default consumes 80% of available memory, which forces OS to start a lot of swapping, so all memory operations limited by disk speed. Try to reduce memory size [1]. [1] https://apacheignite.readme.io/v2.1/docs/memory-configuration#memory-policies Thanks! -Dmitry. -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Performance-of-persistent-store-too-low-when-bulb-loading-tp16247p16335.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
