[jira] [Commented] (IGNITE-14076) Quadratic putAll performance degradation in transactional cache
[ https://issues.apache.org/jira/browse/IGNITE-14076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17310842#comment-17310842 ] Vyacheslav Koptilin commented on IGNITE-14076: -- Hello [~zstan] Well, the proposed patch looks good to me. However, we need to think about a proper fix for the case of using near caches. Merged into the master branch. Thank you for your contribution! > Quadratic putAll performance degradation in transactional cache > --- > > Key: IGNITE-14076 > URL: https://issues.apache.org/jira/browse/IGNITE-14076 > Project: Ignite > Issue Type: Improvement > Components: cache >Affects Versions: 2.10 >Reporter: Pavel Tupitsyn >Assignee: Stanilovsky Evgeny >Priority: Critical > Time Spent: 50m > Remaining Estimate: 0h > > {{putAll}} execution time grows almost exponentially while the number of keys > grows linearly in the following test: > {code:java} > public class PutAllTxTest extends GridCommonAbstractTest { > @Test > public void testPutAll() throws Exception { > Ignition.start(getConfiguration("server1")); > Ignition.start(getConfiguration("server2")); > Ignite ignite = > Ignition.start(getConfiguration("client").setClientMode(true)); > IgniteCache cache = ignite.createCache( > new CacheConfiguration("c") > .setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL)); > int count = 5; > Map data = new TreeMap<>(); > for (int i = 0; i < count; i++) > data.put(i, i); > long begin = System.nanoTime(); > cache.putAll(data); > long dur = System.nanoTime() - begin; > System.out.println("> " + dur / 100); > } > } > {code} > ||Entries||Seconds|| > |1000|0.4| > |5000|1.9| > |1|3.8| > |2|10.7| > |3|23.5| > |4|41| > |5|64| > |6|90| > |10|254| > This does not reproduce with 1 server node, and does not reproduce on > {{ATOMIC}} caches with any number of nodes. > *Observations:* > - Not a GC issue (it barely runs) > - Not a memory issue (heap is under 1GB) > - GridDhtTxPrepareFuture#localDhtPendingVersions -> > GridCacheMapEntry.localCandidates is the bottleneck. For 1K keys, > localCandidates gets called 123K times, 2K keys - 484K times, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-14076) Quadratic putAll performance degradation in transactional cache
[ https://issues.apache.org/jira/browse/IGNITE-14076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17309175#comment-17309175 ] Stanilovsky Evgeny commented on IGNITE-14076: - [~sk0x50] are you plan to merge it ? > Quadratic putAll performance degradation in transactional cache > --- > > Key: IGNITE-14076 > URL: https://issues.apache.org/jira/browse/IGNITE-14076 > Project: Ignite > Issue Type: Improvement > Components: cache >Affects Versions: 2.10 >Reporter: Pavel Tupitsyn >Assignee: Stanilovsky Evgeny >Priority: Critical > Time Spent: 40m > Remaining Estimate: 0h > > {{putAll}} execution time grows almost exponentially while the number of keys > grows linearly in the following test: > {code:java} > public class PutAllTxTest extends GridCommonAbstractTest { > @Test > public void testPutAll() throws Exception { > Ignition.start(getConfiguration("server1")); > Ignition.start(getConfiguration("server2")); > Ignite ignite = > Ignition.start(getConfiguration("client").setClientMode(true)); > IgniteCache cache = ignite.createCache( > new CacheConfiguration("c") > .setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL)); > int count = 5; > Map data = new TreeMap<>(); > for (int i = 0; i < count; i++) > data.put(i, i); > long begin = System.nanoTime(); > cache.putAll(data); > long dur = System.nanoTime() - begin; > System.out.println("> " + dur / 100); > } > } > {code} > ||Entries||Seconds|| > |1000|0.4| > |5000|1.9| > |1|3.8| > |2|10.7| > |3|23.5| > |4|41| > |5|64| > |6|90| > |10|254| > This does not reproduce with 1 server node, and does not reproduce on > {{ATOMIC}} caches with any number of nodes. > *Observations:* > - Not a GC issue (it barely runs) > - Not a memory issue (heap is under 1GB) > - GridDhtTxPrepareFuture#localDhtPendingVersions -> > GridCacheMapEntry.localCandidates is the bottleneck. For 1K keys, > localCandidates gets called 123K times, 2K keys - 484K times, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-14076) Quadratic putAll performance degradation in transactional cache
[ https://issues.apache.org/jira/browse/IGNITE-14076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17308520#comment-17308520 ] Vladislav Pyatkov commented on IGNITE-14076: Now it looks good. > Quadratic putAll performance degradation in transactional cache > --- > > Key: IGNITE-14076 > URL: https://issues.apache.org/jira/browse/IGNITE-14076 > Project: Ignite > Issue Type: Improvement > Components: cache >Affects Versions: 2.10 >Reporter: Pavel Tupitsyn >Assignee: Stanilovsky Evgeny >Priority: Critical > Time Spent: 40m > Remaining Estimate: 0h > > {{putAll}} execution time grows almost exponentially while the number of keys > grows linearly in the following test: > {code:java} > public class PutAllTxTest extends GridCommonAbstractTest { > @Test > public void testPutAll() throws Exception { > Ignition.start(getConfiguration("server1")); > Ignition.start(getConfiguration("server2")); > Ignite ignite = > Ignition.start(getConfiguration("client").setClientMode(true)); > IgniteCache cache = ignite.createCache( > new CacheConfiguration("c") > .setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL)); > int count = 5; > Map data = new TreeMap<>(); > for (int i = 0; i < count; i++) > data.put(i, i); > long begin = System.nanoTime(); > cache.putAll(data); > long dur = System.nanoTime() - begin; > System.out.println("> " + dur / 100); > } > } > {code} > ||Entries||Seconds|| > |1000|0.4| > |5000|1.9| > |1|3.8| > |2|10.7| > |3|23.5| > |4|41| > |5|64| > |6|90| > |10|254| > This does not reproduce with 1 server node, and does not reproduce on > {{ATOMIC}} caches with any number of nodes. > *Observations:* > - Not a GC issue (it barely runs) > - Not a memory issue (heap is under 1GB) > - GridDhtTxPrepareFuture#localDhtPendingVersions -> > GridCacheMapEntry.localCandidates is the bottleneck. For 1K keys, > localCandidates gets called 123K times, 2K keys - 484K times, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-14076) Quadratic putAll performance degradation in transactional cache
[ https://issues.apache.org/jira/browse/IGNITE-14076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17308513#comment-17308513 ] Stanilovsky Evgeny commented on IGNITE-14076: - [~v.pyatkov] thanks for review, i revert GridNearOptimisticTxPrepareFuture, is it ok now ? > Quadratic putAll performance degradation in transactional cache > --- > > Key: IGNITE-14076 > URL: https://issues.apache.org/jira/browse/IGNITE-14076 > Project: Ignite > Issue Type: Improvement > Components: cache >Affects Versions: 2.10 >Reporter: Pavel Tupitsyn >Assignee: Stanilovsky Evgeny >Priority: Critical > Time Spent: 40m > Remaining Estimate: 0h > > {{putAll}} execution time grows almost exponentially while the number of keys > grows linearly in the following test: > {code:java} > public class PutAllTxTest extends GridCommonAbstractTest { > @Test > public void testPutAll() throws Exception { > Ignition.start(getConfiguration("server1")); > Ignition.start(getConfiguration("server2")); > Ignite ignite = > Ignition.start(getConfiguration("client").setClientMode(true)); > IgniteCache cache = ignite.createCache( > new CacheConfiguration("c") > .setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL)); > int count = 5; > Map data = new TreeMap<>(); > for (int i = 0; i < count; i++) > data.put(i, i); > long begin = System.nanoTime(); > cache.putAll(data); > long dur = System.nanoTime() - begin; > System.out.println("> " + dur / 100); > } > } > {code} > ||Entries||Seconds|| > |1000|0.4| > |5000|1.9| > |1|3.8| > |2|10.7| > |3|23.5| > |4|41| > |5|64| > |6|90| > |10|254| > This does not reproduce with 1 server node, and does not reproduce on > {{ATOMIC}} caches with any number of nodes. > *Observations:* > - Not a GC issue (it barely runs) > - Not a memory issue (heap is under 1GB) > - GridDhtTxPrepareFuture#localDhtPendingVersions -> > GridCacheMapEntry.localCandidates is the bottleneck. For 1K keys, > localCandidates gets called 123K times, 2K keys - 484K times, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-14076) Quadratic putAll performance degradation in transactional cache
[ https://issues.apache.org/jira/browse/IGNITE-14076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17308391#comment-17308391 ] Stanilovsky Evgeny commented on IGNITE-14076: - i believe it improve code reading and clearing. > Quadratic putAll performance degradation in transactional cache > --- > > Key: IGNITE-14076 > URL: https://issues.apache.org/jira/browse/IGNITE-14076 > Project: Ignite > Issue Type: Improvement > Components: cache >Affects Versions: 2.10 >Reporter: Pavel Tupitsyn >Assignee: Stanilovsky Evgeny >Priority: Critical > Time Spent: 40m > Remaining Estimate: 0h > > {{putAll}} execution time grows almost exponentially while the number of keys > grows linearly in the following test: > {code:java} > public class PutAllTxTest extends GridCommonAbstractTest { > @Test > public void testPutAll() throws Exception { > Ignition.start(getConfiguration("server1")); > Ignition.start(getConfiguration("server2")); > Ignite ignite = > Ignition.start(getConfiguration("client").setClientMode(true)); > IgniteCache cache = ignite.createCache( > new CacheConfiguration("c") > .setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL)); > int count = 5; > Map data = new TreeMap<>(); > for (int i = 0; i < count; i++) > data.put(i, i); > long begin = System.nanoTime(); > cache.putAll(data); > long dur = System.nanoTime() - begin; > System.out.println("> " + dur / 100); > } > } > {code} > ||Entries||Seconds|| > |1000|0.4| > |5000|1.9| > |1|3.8| > |2|10.7| > |3|23.5| > |4|41| > |5|64| > |6|90| > |10|254| > This does not reproduce with 1 server node, and does not reproduce on > {{ATOMIC}} caches with any number of nodes. > *Observations:* > - Not a GC issue (it barely runs) > - Not a memory issue (heap is under 1GB) > - GridDhtTxPrepareFuture#localDhtPendingVersions -> > GridCacheMapEntry.localCandidates is the bottleneck. For 1K keys, > localCandidates gets called 123K times, 2K keys - 484K times, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-14076) Quadratic putAll performance degradation in transactional cache
[ https://issues.apache.org/jira/browse/IGNITE-14076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17308114#comment-17308114 ] Vladislav Pyatkov commented on IGNITE-14076: [~zstan] I have watched your patch and didn't understand what the purpose of change in GridNearOptimisticTxPrepareFuture.java. > Quadratic putAll performance degradation in transactional cache > --- > > Key: IGNITE-14076 > URL: https://issues.apache.org/jira/browse/IGNITE-14076 > Project: Ignite > Issue Type: Improvement > Components: cache >Affects Versions: 2.10 >Reporter: Pavel Tupitsyn >Assignee: Stanilovsky Evgeny >Priority: Critical > Time Spent: 40m > Remaining Estimate: 0h > > {{putAll}} execution time grows almost exponentially while the number of keys > grows linearly in the following test: > {code:java} > public class PutAllTxTest extends GridCommonAbstractTest { > @Test > public void testPutAll() throws Exception { > Ignition.start(getConfiguration("server1")); > Ignition.start(getConfiguration("server2")); > Ignite ignite = > Ignition.start(getConfiguration("client").setClientMode(true)); > IgniteCache cache = ignite.createCache( > new CacheConfiguration("c") > .setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL)); > int count = 5; > Map data = new TreeMap<>(); > for (int i = 0; i < count; i++) > data.put(i, i); > long begin = System.nanoTime(); > cache.putAll(data); > long dur = System.nanoTime() - begin; > System.out.println("> " + dur / 100); > } > } > {code} > ||Entries||Seconds|| > |1000|0.4| > |5000|1.9| > |1|3.8| > |2|10.7| > |3|23.5| > |4|41| > |5|64| > |6|90| > |10|254| > This does not reproduce with 1 server node, and does not reproduce on > {{ATOMIC}} caches with any number of nodes. > *Observations:* > - Not a GC issue (it barely runs) > - Not a memory issue (heap is under 1GB) > - GridDhtTxPrepareFuture#localDhtPendingVersions -> > GridCacheMapEntry.localCandidates is the bottleneck. For 1K keys, > localCandidates gets called 123K times, 2K keys - 484K times, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-14076) Quadratic putAll performance degradation in transactional cache
[ https://issues.apache.org/jira/browse/IGNITE-14076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17303169#comment-17303169 ] Pavel Tupitsyn commented on IGNITE-14076: - [~zstan] looks good to me > Quadratic putAll performance degradation in transactional cache > --- > > Key: IGNITE-14076 > URL: https://issues.apache.org/jira/browse/IGNITE-14076 > Project: Ignite > Issue Type: Improvement > Components: cache >Affects Versions: 2.10 >Reporter: Pavel Tupitsyn >Assignee: Stanilovsky Evgeny >Priority: Critical > Time Spent: 0.5h > Remaining Estimate: 0h > > {{putAll}} execution time grows almost exponentially while the number of keys > grows linearly in the following test: > {code:java} > public class PutAllTxTest extends GridCommonAbstractTest { > @Test > public void testPutAll() throws Exception { > Ignition.start(getConfiguration("server1")); > Ignition.start(getConfiguration("server2")); > Ignite ignite = > Ignition.start(getConfiguration("client").setClientMode(true)); > IgniteCache cache = ignite.createCache( > new CacheConfiguration("c") > .setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL)); > int count = 5; > Map data = new TreeMap<>(); > for (int i = 0; i < count; i++) > data.put(i, i); > long begin = System.nanoTime(); > cache.putAll(data); > long dur = System.nanoTime() - begin; > System.out.println("> " + dur / 100); > } > } > {code} > ||Entries||Seconds|| > |1000|0.4| > |5000|1.9| > |1|3.8| > |2|10.7| > |3|23.5| > |4|41| > |5|64| > |6|90| > |10|254| > This does not reproduce with 1 server node, and does not reproduce on > {{ATOMIC}} caches with any number of nodes. > *Observations:* > - Not a GC issue (it barely runs) > - Not a memory issue (heap is under 1GB) > - GridDhtTxPrepareFuture#localDhtPendingVersions -> > GridCacheMapEntry.localCandidates is the bottleneck. For 1K keys, > localCandidates gets called 123K times, 2K keys - 484K times, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-14076) Quadratic putAll performance degradation in transactional cache
[ https://issues.apache.org/jira/browse/IGNITE-14076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17303166#comment-17303166 ] Stanilovsky Evgeny commented on IGNITE-14076: - [~ptupitsyn] [~sk0x50] guys can you check plz ? > Quadratic putAll performance degradation in transactional cache > --- > > Key: IGNITE-14076 > URL: https://issues.apache.org/jira/browse/IGNITE-14076 > Project: Ignite > Issue Type: Improvement > Components: cache >Affects Versions: 2.9.1 >Reporter: Pavel Tupitsyn >Assignee: Stanilovsky Evgeny >Priority: Critical > Fix For: 2.11 > > Time Spent: 0.5h > Remaining Estimate: 0h > > {{putAll}} execution time grows almost exponentially while the number of keys > grows linearly in the following test: > {code:java} > public class PutAllTxTest extends GridCommonAbstractTest { > @Test > public void testPutAll() throws Exception { > Ignition.start(getConfiguration("server1")); > Ignition.start(getConfiguration("server2")); > Ignite ignite = > Ignition.start(getConfiguration("client").setClientMode(true)); > IgniteCache cache = ignite.createCache( > new CacheConfiguration("c") > .setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL)); > int count = 5; > Map data = new TreeMap<>(); > for (int i = 0; i < count; i++) > data.put(i, i); > long begin = System.nanoTime(); > cache.putAll(data); > long dur = System.nanoTime() - begin; > System.out.println("> " + dur / 100); > } > } > {code} > ||Entries||Seconds|| > |1000|0.4| > |5000|1.9| > |1|3.8| > |2|10.7| > |3|23.5| > |4|41| > |5|64| > |6|90| > |10|254| > This does not reproduce with 1 server node, and does not reproduce on > {{ATOMIC}} caches with any number of nodes. > *Observations:* > - Not a GC issue (it barely runs) > - Not a memory issue (heap is under 1GB) > - GridDhtTxPrepareFuture#localDhtPendingVersions -> > GridCacheMapEntry.localCandidates is the bottleneck. For 1K keys, > localCandidates gets called 123K times, 2K keys - 484K times, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-14076) Quadratic putAll performance degradation in transactional cache
[ https://issues.apache.org/jira/browse/IGNITE-14076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17303165#comment-17303165 ] Stanilovsky Evgeny commented on IGNITE-14076: - I hope that no additional tests need to be here, putAll throughput speedup can be checked locally, for example, by running: IgnitePutAllLargeBatchSelfTest#testPutAllOptimisticOneBackupPartitioned with appropriate change in _int keyCnt = 200;_ for example by 20_000 > Quadratic putAll performance degradation in transactional cache > --- > > Key: IGNITE-14076 > URL: https://issues.apache.org/jira/browse/IGNITE-14076 > Project: Ignite > Issue Type: Improvement > Components: cache >Affects Versions: 2.9.1 >Reporter: Pavel Tupitsyn >Assignee: Stanilovsky Evgeny >Priority: Critical > Fix For: 2.11 > > Time Spent: 0.5h > Remaining Estimate: 0h > > {{putAll}} execution time grows almost exponentially while the number of keys > grows linearly in the following test: > {code:java} > public class PutAllTxTest extends GridCommonAbstractTest { > @Test > public void testPutAll() throws Exception { > Ignition.start(getConfiguration("server1")); > Ignition.start(getConfiguration("server2")); > Ignite ignite = > Ignition.start(getConfiguration("client").setClientMode(true)); > IgniteCache cache = ignite.createCache( > new CacheConfiguration("c") > .setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL)); > int count = 5; > Map data = new TreeMap<>(); > for (int i = 0; i < count; i++) > data.put(i, i); > long begin = System.nanoTime(); > cache.putAll(data); > long dur = System.nanoTime() - begin; > System.out.println("> " + dur / 100); > } > } > {code} > ||Entries||Seconds|| > |1000|0.4| > |5000|1.9| > |1|3.8| > |2|10.7| > |3|23.5| > |4|41| > |5|64| > |6|90| > |10|254| > This does not reproduce with 1 server node, and does not reproduce on > {{ATOMIC}} caches with any number of nodes. > *Observations:* > - Not a GC issue (it barely runs) > - Not a memory issue (heap is under 1GB) > - GridDhtTxPrepareFuture#localDhtPendingVersions -> > GridCacheMapEntry.localCandidates is the bottleneck. For 1K keys, > localCandidates gets called 123K times, 2K keys - 484K times, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-14076) Quadratic putAll performance degradation in transactional cache
[ https://issues.apache.org/jira/browse/IGNITE-14076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17303163#comment-17303163 ] Ignite TC Bot commented on IGNITE-14076: {panel:title=Branch: [pull/8885/head] Base: [master] : No blockers found!|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}{panel} {panel:title=Branch: [pull/8885/head] Base: [master] : No new tests found!|borderStyle=dashed|borderColor=#ccc|titleBGColor=#F7D6C1}{panel} [TeamCity *-- Run :: All* Results|https://ci.ignite.apache.org/viewLog.html?buildId=5918009buildTypeId=IgniteTests24Java8_RunAll] > Quadratic putAll performance degradation in transactional cache > --- > > Key: IGNITE-14076 > URL: https://issues.apache.org/jira/browse/IGNITE-14076 > Project: Ignite > Issue Type: Improvement > Components: cache >Affects Versions: 2.9.1 >Reporter: Pavel Tupitsyn >Assignee: Stanilovsky Evgeny >Priority: Critical > Fix For: 2.11 > > Time Spent: 0.5h > Remaining Estimate: 0h > > {{putAll}} execution time grows almost exponentially while the number of keys > grows linearly in the following test: > {code:java} > public class PutAllTxTest extends GridCommonAbstractTest { > @Test > public void testPutAll() throws Exception { > Ignition.start(getConfiguration("server1")); > Ignition.start(getConfiguration("server2")); > Ignite ignite = > Ignition.start(getConfiguration("client").setClientMode(true)); > IgniteCache cache = ignite.createCache( > new CacheConfiguration("c") > .setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL)); > int count = 5; > Map data = new TreeMap<>(); > for (int i = 0; i < count; i++) > data.put(i, i); > long begin = System.nanoTime(); > cache.putAll(data); > long dur = System.nanoTime() - begin; > System.out.println("> " + dur / 100); > } > } > {code} > ||Entries||Seconds|| > |1000|0.4| > |5000|1.9| > |1|3.8| > |2|10.7| > |3|23.5| > |4|41| > |5|64| > |6|90| > |10|254| > This does not reproduce with 1 server node, and does not reproduce on > {{ATOMIC}} caches with any number of nodes. > *Observations:* > - Not a GC issue (it barely runs) > - Not a memory issue (heap is under 1GB) > - GridDhtTxPrepareFuture#localDhtPendingVersions -> > GridCacheMapEntry.localCandidates is the bottleneck. For 1K keys, > localCandidates gets called 123K times, 2K keys - 484K times, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)