[jira] [Comment Edited] (IGNITE-14341) Reduce contention in the PendingEntriesTree when cleaning up expired entries.
[
https://issues.apache.org/jira/browse/IGNITE-14341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17539666#comment-17539666
]
Pavel Pereslegin edited comment on IGNITE-14341 at 5/20/22 10:17 AM:
-
Currently to clean up expired entries we are getting the cursor and deleting
rows one by one.
Thus, for each row, we perform a search from the root, lock the page for
writing, and delete single row.
!expire1.png!
Instead of deleting a single row, we can delete all rows from a specified range
on the page.
This reduces the number of page write locks.
!expire2.png!
The proposed changes extend the {{remove}} (rather than the {{cursor}})
operation.
Those. after deleting rows from the page, the removal operation (search) is
repeated from the root.
This is also why this operation does not support tree mode, in which the inner
page may contain a key that is not present in the leaf page
({{canGetRowFromInner = false}}).
Benchmark results on local machine (i7-8700 CPU @ 3.20GHz, 6 cores, 12 threads).
!bench_diagram.png!
Since this patch only speeds up pendingtree clearing, it would be more correct
to compare the speed of inserting into a cache with an expiry policy but
with/and without pendingtree cleanup (with excluding code that removes entries
from the page store).
!bench3.png!
was (Author: xtern):
Currently to clean up expired entries we are getting the cursor and deleting
rows one by one.
Thus, for each row, we perform a search from the root, lock the page for
writing, and delete single row.
!expire1.png!
Instead of deleting a single row, we can delete all rows from a specified range
on the page.
This reduces the number of page write locks.
!expire2.png!
The proposed changes extend the {{remove}} (rather than the {{cursor}})
operation.
Those. after deleting rows from the page, the removal operation (search) is
repeated from the root.
This is also why this operation does not support tree mode, in which the inner
page may contain a key that is not present in the leaf page
({{canGetRowFromInner = false}}).
Benchmark results on local machine (i7-8700 CPU @ 3.20GHz, 6 cores, 12 threads).
!bench_diagram.png!
> Reduce contention in the PendingEntriesTree when cleaning up expired entries.
> -
>
> Key: IGNITE-14341
> URL: https://issues.apache.org/jira/browse/IGNITE-14341
> Project: Ignite
> Issue Type: Improvement
>Reporter: Aleksey Plekhanov
>Assignee: Pavel Pereslegin
>Priority: Major
> Labels: ise
> Attachments: JmhCacheExpireBenchmark.java, bench3.png,
> bench_diagram.png, expire1.png, expire2.png
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Currently, there is a significant performance drop when expired entries
> concurrently evicted by threads that perform some actions with cache (see
> attached reproducer):
> {noformat}
> Benchmark Mode Cnt Score Error
> Units
> JmhCacheExpireBenchmark.putWithExpire thrpt3 100,132 ± 21,025
> ops/ms
> JmhCacheExpireBenchmark.putWithoutExpire thrpt3 2133,122 ± 559,694
> ops/ms{noformat}
> Root cause: pending entries tree (offheap BPlusTree) is used to track expired
> entries, after each cache operation (and by timeout thread) there is an
> attempt to evict some amount of expired entries. these entries looked up from
> the start of the pending entries tree and there is a contention on the first
> leaf page of that tree.
> All threads waiting for the same page lock:
> {noformat}
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> org.apache.ignite.internal.util.OffheapReadWriteLock.waitAcquireWriteLock(OffheapReadWriteLock.java:503)
> at
> org.apache.ignite.internal.util.OffheapReadWriteLock.writeLock(OffheapReadWriteLock.java:244)
> at
> org.apache.ignite.internal.pagemem.impl.PageMemoryNoStoreImpl.writeLock(PageMemoryNoStoreImpl.java:528)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.writeLock(PageHandler.java:422)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.writePage(PageHandler.java:350)
> at
> org.apache.ignite.internal.processors.cache.persistence.DataStructure.write(DataStructure.java:325)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$13200(BPlusTree.java:100)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Remove.doRemoveFromLeaf(BPlusTree.java:4588)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Remove.removeFromLeaf(BPlusTree.java:4567)
> at
> org.apache.ignite.intern
[jira] [Comment Edited] (IGNITE-14341) Reduce contention in the PendingEntriesTree when cleaning up expired entries.
[
https://issues.apache.org/jira/browse/IGNITE-14341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17539666#comment-17539666
]
Pavel Pereslegin edited comment on IGNITE-14341 at 5/20/22 9:16 AM:
Currently to clean up expired entries we are getting the cursor and deleting
rows one by one.
Thus, for each row, we perform a search from the root, lock the page for
writing, and delete single row.
!expire1.png!
Instead of deleting a single row, we can delete all rows from a specified range
on the page.
This reduces the number of page write locks.
!expire2.png!
The proposed changes extend the {{remove}} (rather than the {{cursor}})
operation.
Those. after deleting rows from the page, the removal operation (search) is
repeated from the root.
This is also why this operation does not support tree mode, in which the inner
page may contain a key that is not present in the leaf page
({{canGetRowFromInner = false}}).
Benchmark results on local machine (i7-8700 CPU @ 3.20GHz, 6 cores, 12 threads).
!bench_diagram.png!
was (Author: xtern):
Currently to clean up expired entries we are getting the cursor and deleting
rows one by one.
Thus, for each row, we perform a search from the root, lock the page for
writing, and delete single row.
!expire1.png!
Instead of deleting a single row, we can delete all rows from a specified range
on the page.
This reduces the number of page write locks.
!expire2.png!
The proposed changes extend the {{remove}} (rather than the {{cursor}})
operation.
Those. after deleting rows from the page, the removal operation (search) is
repeated from the root.
This is also why this operation does not support tree mode, in which the inner
page may contain a key that is not present in the leaf page
({{canGetRowFromInner = false}}).
> Reduce contention in the PendingEntriesTree when cleaning up expired entries.
> -
>
> Key: IGNITE-14341
> URL: https://issues.apache.org/jira/browse/IGNITE-14341
> Project: Ignite
> Issue Type: Improvement
>Reporter: Aleksey Plekhanov
>Assignee: Pavel Pereslegin
>Priority: Major
> Labels: ise
> Attachments: JmhCacheExpireBenchmark.java, bench_diagram.png,
> expire1.png, expire2.png
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Currently, there is a significant performance drop when expired entries
> concurrently evicted by threads that perform some actions with cache (see
> attached reproducer):
> {noformat}
> Benchmark Mode Cnt Score Error
> Units
> JmhCacheExpireBenchmark.putWithExpire thrpt3 100,132 ± 21,025
> ops/ms
> JmhCacheExpireBenchmark.putWithoutExpire thrpt3 2133,122 ± 559,694
> ops/ms{noformat}
> Root cause: pending entries tree (offheap BPlusTree) is used to track expired
> entries, after each cache operation (and by timeout thread) there is an
> attempt to evict some amount of expired entries. these entries looked up from
> the start of the pending entries tree and there is a contention on the first
> leaf page of that tree.
> All threads waiting for the same page lock:
> {noformat}
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> org.apache.ignite.internal.util.OffheapReadWriteLock.waitAcquireWriteLock(OffheapReadWriteLock.java:503)
> at
> org.apache.ignite.internal.util.OffheapReadWriteLock.writeLock(OffheapReadWriteLock.java:244)
> at
> org.apache.ignite.internal.pagemem.impl.PageMemoryNoStoreImpl.writeLock(PageMemoryNoStoreImpl.java:528)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.writeLock(PageHandler.java:422)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.writePage(PageHandler.java:350)
> at
> org.apache.ignite.internal.processors.cache.persistence.DataStructure.write(DataStructure.java:325)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$13200(BPlusTree.java:100)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Remove.doRemoveFromLeaf(BPlusTree.java:4588)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Remove.removeFromLeaf(BPlusTree.java:4567)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Remove.tryRemoveFromLeaf(BPlusTree.java:5196)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Remove.access$6800(BPlusTree.java:4209)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.removeDown(BPlusTree.java:2189)
> at
> org.apache.ignite.internal.processors.cache.persistenc
[jira] [Comment Edited] (IGNITE-14341) Reduce contention in the PendingEntriesTree when cleaning up expired entries.
[
https://issues.apache.org/jira/browse/IGNITE-14341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17539666#comment-17539666
]
Pavel Pereslegin edited comment on IGNITE-14341 at 5/19/22 5:32 PM:
Currently to clean up expired entries we are getting the cursor and deleting
rows one by one.
Thus, for each row, we perform a search from the root, lock the page for
writing, and delete single row.
!expire1.png!
Instead of deleting a single row, we can delete all rows from a specified range
on the page.
This reduces the number of page write locks.
!expire2.png!
The proposed changes extend the {{remove}} (rather than the {{cursor}})
operation.
Those. after deleting rows from the page, the removal operation (search) is
repeated from the root.
This is also why this operation does not support tree mode, in which the inner
page may contain a key that is not present in the leaf page
({{canGetRowFromInner = false}}).
was (Author: xtern):
Currently to clean up expired entries we are getting the cursor and deleting
rows one by one.
Thus, for each row, we perform a search from the root, lock the page for
writing, and delete single row.
!expire1.png!
Instead of deleting a single row, we can delete all rows from a specified range
on the page.
This reduces the number of page write locks.
!expire2.png!
> Reduce contention in the PendingEntriesTree when cleaning up expired entries.
> -
>
> Key: IGNITE-14341
> URL: https://issues.apache.org/jira/browse/IGNITE-14341
> Project: Ignite
> Issue Type: Improvement
>Reporter: Aleksey Plekhanov
>Assignee: Pavel Pereslegin
>Priority: Major
> Labels: ise
> Attachments: JmhCacheExpireBenchmark.java, expire1.png, expire2.png
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Currently, there is a significant performance drop when expired entries
> concurrently evicted by threads that perform some actions with cache (see
> attached reproducer):
> {noformat}
> Benchmark Mode Cnt Score Error
> Units
> JmhCacheExpireBenchmark.putWithExpire thrpt3 100,132 ± 21,025
> ops/ms
> JmhCacheExpireBenchmark.putWithoutExpire thrpt3 2133,122 ± 559,694
> ops/ms{noformat}
> Root cause: pending entries tree (offheap BPlusTree) is used to track expired
> entries, after each cache operation (and by timeout thread) there is an
> attempt to evict some amount of expired entries. these entries looked up from
> the start of the pending entries tree and there is a contention on the first
> leaf page of that tree.
> All threads waiting for the same page lock:
> {noformat}
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> org.apache.ignite.internal.util.OffheapReadWriteLock.waitAcquireWriteLock(OffheapReadWriteLock.java:503)
> at
> org.apache.ignite.internal.util.OffheapReadWriteLock.writeLock(OffheapReadWriteLock.java:244)
> at
> org.apache.ignite.internal.pagemem.impl.PageMemoryNoStoreImpl.writeLock(PageMemoryNoStoreImpl.java:528)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.writeLock(PageHandler.java:422)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.writePage(PageHandler.java:350)
> at
> org.apache.ignite.internal.processors.cache.persistence.DataStructure.write(DataStructure.java:325)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$13200(BPlusTree.java:100)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Remove.doRemoveFromLeaf(BPlusTree.java:4588)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Remove.removeFromLeaf(BPlusTree.java:4567)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Remove.tryRemoveFromLeaf(BPlusTree.java:5196)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Remove.access$6800(BPlusTree.java:4209)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.removeDown(BPlusTree.java:2189)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.removeDown(BPlusTree.java:2165)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.removeDown(BPlusTree.java:2165)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doRemove(BPlusTree.java:2076)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.removex(BPlusTree.java:1905)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.expireInternal(Ignite
