Joel Lang created IGNITE-8359: --------------------------------- Summary: Severe performance degradation with persistence and data streaming on HDD Key: IGNITE-8359 URL: https://issues.apache.org/jira/browse/IGNITE-8359 Project: Ignite Issue Type: Bug Components: cache, persistence, sql, streaming Affects Versions: 2.4, 2.5 Environment: Linux CentOS 7 VM using Ignite DirectIO plugin with HDD. Reporter: Joel Lang
I am testing the use of Ignite's native persistence to store a data set long term. This is on a 2.5 nightly build. To do this I am using Ignite's data streamers to stream in 6,000,000 entries into cache A, and 12,000,000 entries into cache B to simulate the upper limit for 2 years worth of data. The test ran smoothly on my personal machine which has a SSD running Windows, but ran into tremendous issues on a development test machine which is a Linux VM using a HDD. I realize when looking at Ignite documentation that it specifically excludes HDD's as something to base a persistent store on, but perhaps my experience could yield improvements for SSD performance too. The root issue is that cache updates over time become severely bottlenecked by reading SQL index pages from disk in order to update the index. If I had to guess this would be related to BPlusTree.findInsertionPoint() and it having to load pages from disk if they've been evicted. I used a 2.5 nightly build because 2.3 and 2.4 have the same issue where this whole process was further bottlenecked by a lock behind held by Ignite while it read the page from disk in PageMemoryImpl.acquirePage(). 2.5 fixed this. The performance issue was much more severe in the previously mentioned cache B, which contains user comments on entries in cache A. The key for each comment entry is a Java class containing the creation timestamp and the string key of the owning entry in cache A. This owning entry key is indexed so comments can be queried by their owner. In this test case there were two comments in cache B for every entry in cache A. I found that even 25% of the way through streaming data into cache B, it would take anywhere from 15 to 35 seconds to insert a batch of 2000 comments. This slowed streaming to a crawl and ensures that streaming would need to continue overnight to have any hope of finishing. This also brings up concerns about data rebalancing which will have the same performance penalty and similarly take a day at least to rebalance both caches. I am worried about the dependency on a large amount of disk reads being done to update the index, even though it is considerably faster with an SSD than without. -- This message was sent by Atlassian JIRA (v7.6.3#76005)