[ https://issues.apache.org/jira/browse/IGNITE-8359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16459546#comment-16459546 ]
Joel Lang commented on IGNITE-8359: ----------------------------------- So using the 2.5 nightly build from April 27th, I ran the code to generate the 6,000,000 entries for cache A then 12,000,000 entries for cache B using the data streamer for each. This again in a linux VM operating on a HDD. This was started on Friday before I left work. When I came into the office I found that it had not even finished the operation. It was about 95-97% done. The fact that it didn't finish over such a long period of time is a bit obscene. > Severe performance degradation with persistence and data streaming on HDD > ------------------------------------------------------------------------- > > Key: IGNITE-8359 > URL: https://issues.apache.org/jira/browse/IGNITE-8359 > Project: Ignite > Issue Type: Bug > Components: cache, persistence, sql, streaming > Affects Versions: 2.4, 2.5 > Environment: Linux CentOS 7 VM using Ignite DirectIO plugin with HDD. > Reporter: Joel Lang > Priority: Major > > I am testing the use of Ignite's native persistence to store a data set long > term. This is on a 2.5 nightly build. To do this I am using Ignite's data > streamers to stream in 6,000,000 entries into cache A, and 12,000,000 entries > into cache B to simulate the upper limit for 2 years worth of data. > The test ran smoothly on my personal machine which has a SSD running Windows, > but ran into tremendous issues on a development test machine which is a Linux > VM using a HDD. I realize when looking at Ignite documentation that it > specifically excludes HDD's as something to base a persistent store on, but > perhaps my experience could yield improvements for SSD performance too. > The root issue is that cache updates over time become severely bottlenecked > by reading SQL index pages from disk in order to update the index. If I had > to guess this would be related to BPlusTree.findInsertionPoint() and it > having to load pages from disk if they've been evicted. > I used a 2.5 nightly build because 2.3 and 2.4 have the same issue where this > whole process was further bottlenecked by a lock behind held by Ignite while > it read the page from disk in PageMemoryImpl.acquirePage(). 2.5 fixed this. > The performance issue was much more severe in the previously mentioned cache > B, which contains user comments on entries in cache A. The key for each > comment entry is a Java class containing the creation timestamp and the > string key of the owning entry in cache A. This owning entry key is indexed > so comments can be queried by their owner. In this test case there were two > comments in cache B for every entry in cache A. > I found that even 25% of the way through streaming data into cache B, it > would take anywhere from 15 to 35 seconds to insert a batch of 2000 comments. > This slowed streaming to a crawl and ensures that streaming would need to > continue overnight to have any hope of finishing. > This also brings up concerns about data rebalancing which will have the same > performance penalty and similarly take a day at least to rebalance both > caches. > I am worried about the dependency on a large amount of disk reads being done > to update the index, even though it is considerably faster with an SSD than > without. I've also not been able to test whether performance for an SSD will > be different when running in a VM, which is another worry. -- This message was sent by Atlassian JIRA (v7.6.3#76005)