Hi, I am evaluating Apache Ignite for a use case where I know we may not have enough RAM to store the entire database. For example we might store 30 days of data, but only have enough memory for 15, and in fact most of the queries will be against the last 5 days of data. I am doing some tests loading the 30 days of data (roughly 2 billion objects) into a test environment.
· The test environment has 6 servers each with 128GB RAM, which I am allocating roughly 70% to Ignite to allow the OS enough memory. · This is using Native Persistence in Ignite 2.8.1. · The disks I am using are not as fast as production would be (they are not SSD) but they are still reasonably fast. · The object I am storing is reasonably simple, it has 25 properties which are integers (2 of these have indexes on) and 4 which are floats, but there are a lot of them (roughly 2 billion). · The cache the data is being loaded into is partitioned. · Data is loaded using IDataStreamer – there is a single loader process which has 10 threads reading files, then submitting into a single DataStreamer. What I am observing is that: · Whilst the default region has RAM available, the inserts are quite quick – even though the data is being written to disk (the WAL is enabled). I get on average 150K objects inserted per second. · Once the default region runs out of RAM, I see an error message “WARN|org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl|Page replacements started, pages will be rotated with disk, this will affect storage performance (consider increasing DataRegionConfiguration#setMaxSize).” · After this, inserts are much slower, going down to about 19K objects per second. Now I kind of understand what is happening here is that for every new page in memory that needs to be allocated, Ignite is having to purge another page from memory (this is implied by the error message). But what I don’t understand is why the performance impact is so great (150K objects / sec when memory is available vs 19K per sec when memory is exhausted). Before and after the region ran out of RAM, the data was being written to disk. After the region ran out of RAM, I would expect the only difference to be that it needs to purge some data from RAM to make space for the data that is loading – I wouldn’t have thought there should be any more disk activity since through Ignite Persistence the data was already being written to disk. Maybe I have something mis-configured, or I have misunderstood something? -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
