[jira] [Commented] (IGNITE-13558) GridCacheProcessor should implement better parallelization when restoring partition states on startup
[ https://issues.apache.org/jira/browse/IGNITE-13558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17445110#comment-17445110 ] Denis Chudov commented on IGNITE-13558: --- [~sergeychugunov] seems we are ready to merge. > GridCacheProcessor should implement better parallelization when restoring > partition states on startup > - > > Key: IGNITE-13558 > URL: https://issues.apache.org/jira/browse/IGNITE-13558 > Project: Ignite > Issue Type: Improvement > Components: persistence >Reporter: Sergey Chugunov >Assignee: Denis Chudov >Priority: Major > Time Spent: 4h 10m > Remaining Estimate: 0h > > GridCacheProcessor#restorePartitionStates method tries to employ striped pool > to restore partition states in parallel but level of parallelization is down > only to cache group per thread. > It is not enough and not utilizes resources effectively in case of one cache > group much bigger than the others. > We need to parallel restore process down to individual partitions to get the > most from the available resources and speed up node startup. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (IGNITE-13558) GridCacheProcessor should implement better parallelization when restoring partition states on startup
[ https://issues.apache.org/jira/browse/IGNITE-13558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17445109#comment-17445109 ] Denis Chudov commented on IGNITE-13558: --- PR 9565 is almost equal to PR 9327 except of last merge commit. > GridCacheProcessor should implement better parallelization when restoring > partition states on startup > - > > Key: IGNITE-13558 > URL: https://issues.apache.org/jira/browse/IGNITE-13558 > Project: Ignite > Issue Type: Improvement > Components: persistence >Reporter: Sergey Chugunov >Assignee: Denis Chudov >Priority: Major > Time Spent: 4h 10m > Remaining Estimate: 0h > > GridCacheProcessor#restorePartitionStates method tries to employ striped pool > to restore partition states in parallel but level of parallelization is down > only to cache group per thread. > It is not enough and not utilizes resources effectively in case of one cache > group much bigger than the others. > We need to parallel restore process down to individual partitions to get the > most from the available resources and speed up node startup. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (IGNITE-13558) GridCacheProcessor should implement better parallelization when restoring partition states on startup
[
https://issues.apache.org/jira/browse/IGNITE-13558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17445024#comment-17445024
]
Ignite TC Bot commented on IGNITE-13558:
{panel:title=Branch: [pull/9565/head] Base: [master] : No blockers
found!|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}{panel}
{panel:title=Branch: [pull/9565/head] Base: [master] : New Tests
(1023)|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}
{color:#8b}Platform C++ CMake (Win x64 / Release){color} [[tests
1023|https://ci.ignite.apache.org/viewLog.html?buildId=6271258]]
* {color:#013220}IgniteThinClientTest: SqlFieldsQueryTestSuite: SelectTwoValues
- PASSED{color}
* {color:#013220}IgniteOdbcTest: SqlNumericFunctionTestSuite:
TestNumericFunctionFloor - PASSED{color}
* {color:#013220}IgniteThinClientTest: SqlFieldsQueryTestSuite:
SelectSingleValue - PASSED{color}
* {color:#013220}IgniteOdbcTest: SqlNumericFunctionTestSuite:
TestNumericFunctionLog - PASSED{color}
* {color:#013220}IgniteThinClientTest: SqlFieldsQueryTestSuite:
CreateTableInsertSelect - PASSED{color}
* {color:#013220}IgniteOdbcTest: SqlDateTimeFunctionTestSuite: TestCurrentDate
- PASSED{color}
* {color:#013220}IgniteThinClientTest: SqlFieldsQueryTestSuite:
SelectTwoValuesInDifferentOrder - PASSED{color}
* {color:#013220}IgniteCoreTest: ClusterTestSuite: IgniteForCacheNodes -
PASSED{color}
* {color:#013220}IgniteCoreTest: ClusterTestSuite: IgniteForClientNodes -
PASSED{color}
* {color:#013220}IgniteCoreTest: CacheQueryTestSuite:
TestFieldsQueryByteArrayInsertSelect - PASSED{color}
* {color:#013220}IgniteCoreTest: ClusterTestSuite: IgniteForDaemons -
PASSED{color}
... and 1012 new tests
{panel}
[TeamCity *-- Run :: All*
Results|https://ci.ignite.apache.org/viewLog.html?buildId=6271351buildTypeId=IgniteTests24Java8_RunAll]
> GridCacheProcessor should implement better parallelization when restoring
> partition states on startup
> -
>
> Key: IGNITE-13558
> URL: https://issues.apache.org/jira/browse/IGNITE-13558
> Project: Ignite
> Issue Type: Improvement
> Components: persistence
>Reporter: Sergey Chugunov
>Assignee: Denis Chudov
>Priority: Major
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> GridCacheProcessor#restorePartitionStates method tries to employ striped pool
> to restore partition states in parallel but level of parallelization is down
> only to cache group per thread.
> It is not enough and not utilizes resources effectively in case of one cache
> group much bigger than the others.
> We need to parallel restore process down to individual partitions to get the
> most from the available resources and speed up node startup.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (IGNITE-13558) GridCacheProcessor should implement better parallelization when restoring partition states on startup
[ https://issues.apache.org/jira/browse/IGNITE-13558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444381#comment-17444381 ] Denis Chudov commented on IGNITE-13558: --- Thank you! I'm going to get a new visa and then we will be ready to merge. > GridCacheProcessor should implement better parallelization when restoring > partition states on startup > - > > Key: IGNITE-13558 > URL: https://issues.apache.org/jira/browse/IGNITE-13558 > Project: Ignite > Issue Type: Improvement > Components: persistence >Reporter: Sergey Chugunov >Assignee: Denis Chudov >Priority: Major > Time Spent: 4h > Remaining Estimate: 0h > > GridCacheProcessor#restorePartitionStates method tries to employ striped pool > to restore partition states in parallel but level of parallelization is down > only to cache group per thread. > It is not enough and not utilizes resources effectively in case of one cache > group much bigger than the others. > We need to parallel restore process down to individual partitions to get the > most from the available resources and speed up node startup. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (IGNITE-13558) GridCacheProcessor should implement better parallelization when restoring partition states on startup
[ https://issues.apache.org/jira/browse/IGNITE-13558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17443587#comment-17443587 ] Ivan Bessonov commented on IGNITE-13558: [~Denis Chudov] I reviewed the code, looks good to me, thank you! > GridCacheProcessor should implement better parallelization when restoring > partition states on startup > - > > Key: IGNITE-13558 > URL: https://issues.apache.org/jira/browse/IGNITE-13558 > Project: Ignite > Issue Type: Improvement > Components: persistence >Reporter: Sergey Chugunov >Assignee: Denis Chudov >Priority: Major > Time Spent: 4h > Remaining Estimate: 0h > > GridCacheProcessor#restorePartitionStates method tries to employ striped pool > to restore partition states in parallel but level of parallelization is down > only to cache group per thread. > It is not enough and not utilizes resources effectively in case of one cache > group much bigger than the others. > We need to parallel restore process down to individual partitions to get the > most from the available resources and speed up node startup. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (IGNITE-13558) GridCacheProcessor should implement better parallelization when restoring partition states on startup
[ https://issues.apache.org/jira/browse/IGNITE-13558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17426324#comment-17426324 ] Denis Chudov commented on IGNITE-13558: --- [~sergeychugunov] could you pls review this patch? > GridCacheProcessor should implement better parallelization when restoring > partition states on startup > - > > Key: IGNITE-13558 > URL: https://issues.apache.org/jira/browse/IGNITE-13558 > Project: Ignite > Issue Type: Improvement > Components: persistence >Reporter: Sergey Chugunov >Assignee: Denis Chudov >Priority: Major > Time Spent: 4h > Remaining Estimate: 0h > > GridCacheProcessor#restorePartitionStates method tries to employ striped pool > to restore partition states in parallel but level of parallelization is down > only to cache group per thread. > It is not enough and not utilizes resources effectively in case of one cache > group much bigger than the others. > We need to parallel restore process down to individual partitions to get the > most from the available resources and speed up node startup. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-13558) GridCacheProcessor should implement better parallelization when restoring partition states on startup
[
https://issues.apache.org/jira/browse/IGNITE-13558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17426323#comment-17426323
]
Ignite TC Bot commented on IGNITE-13558:
{panel:title=Branch: [pull/9327/head] Base: [master] : Possible Blockers
(2)|borderStyle=dashed|borderColor=#ccc|titleBGColor=#F7D6C1}
{color:#d04437}Platform C++ CMake (Win x64 / Release){color} [[tests 0 TIMEOUT
, Exit Code |https://ci.ignite.apache.org/viewLog.html?buildId=6215958]]
{color:#d04437}Platform C++ (Win x64 / Release){color} [[tests 0 TIMEOUT , Exit
Code |https://ci.ignite.apache.org/viewLog.html?buildId=6215960]]
{panel}
{panel:title=Branch: [pull/9327/head] Base: [master] : No new tests
found!|borderStyle=dashed|borderColor=#ccc|titleBGColor=#F7D6C1}{panel}
[TeamCity *-- Run :: All*
Results|https://ci.ignite.apache.org/viewLog.html?buildId=6214735buildTypeId=IgniteTests24Java8_RunAll]
> GridCacheProcessor should implement better parallelization when restoring
> partition states on startup
> -
>
> Key: IGNITE-13558
> URL: https://issues.apache.org/jira/browse/IGNITE-13558
> Project: Ignite
> Issue Type: Improvement
> Components: persistence
>Reporter: Sergey Chugunov
>Assignee: Denis Chudov
>Priority: Major
> Time Spent: 4h
> Remaining Estimate: 0h
>
> GridCacheProcessor#restorePartitionStates method tries to employ striped pool
> to restore partition states in parallel but level of parallelization is down
> only to cache group per thread.
> It is not enough and not utilizes resources effectively in case of one cache
> group much bigger than the others.
> We need to parallel restore process down to individual partitions to get the
> most from the available resources and speed up node startup.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (IGNITE-13558) GridCacheProcessor should implement better parallelization when restoring partition states on startup
[ https://issues.apache.org/jira/browse/IGNITE-13558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17425386#comment-17425386 ] Kirill Tkalenko commented on IGNITE-13558: -- [~Denis Chudov]Looks good to me, please merge the last master and get a visa, you can also get rid of *RestorePartitionStateBenchmark*. > GridCacheProcessor should implement better parallelization when restoring > partition states on startup > - > > Key: IGNITE-13558 > URL: https://issues.apache.org/jira/browse/IGNITE-13558 > Project: Ignite > Issue Type: Improvement > Components: persistence >Reporter: Sergey Chugunov >Assignee: Denis Chudov >Priority: Major > Time Spent: 4h > Remaining Estimate: 0h > > GridCacheProcessor#restorePartitionStates method tries to employ striped pool > to restore partition states in parallel but level of parallelization is down > only to cache group per thread. > It is not enough and not utilizes resources effectively in case of one cache > group much bigger than the others. > We need to parallel restore process down to individual partitions to get the > most from the available resources and speed up node startup. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-13558) GridCacheProcessor should implement better parallelization when restoring partition states on startup
[ https://issues.apache.org/jira/browse/IGNITE-13558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17411181#comment-17411181 ] Denis Chudov commented on IGNITE-13558: --- [~ktkale...@gridgain.com] pls check PR 9327 where worker per partition was implemented. > GridCacheProcessor should implement better parallelization when restoring > partition states on startup > - > > Key: IGNITE-13558 > URL: https://issues.apache.org/jira/browse/IGNITE-13558 > Project: Ignite > Issue Type: Improvement > Components: persistence >Reporter: Sergey Chugunov >Assignee: Denis Chudov >Priority: Major > Time Spent: 3h > Remaining Estimate: 0h > > GridCacheProcessor#restorePartitionStates method tries to employ striped pool > to restore partition states in parallel but level of parallelization is down > only to cache group per thread. > It is not enough and not utilizes resources effectively in case of one cache > group much bigger than the others. > We need to parallel restore process down to individual partitions to get the > most from the available resources and speed up node startup. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-13558) GridCacheProcessor should implement better parallelization when restoring partition states on startup
[
https://issues.apache.org/jira/browse/IGNITE-13558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17411179#comment-17411179
]
Ignite TC Bot commented on IGNITE-13558:
{panel:title=Branch: [pull/9327/head] Base: [master] : No blockers
found!|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}{panel}
{panel:title=Branch: [pull/9327/head] Base: [master] : No new tests
found!|borderStyle=dashed|borderColor=#ccc|titleBGColor=#F7D6C1}{panel}
[TeamCity *-- Run :: All*
Results|https://ci.ignite.apache.org/viewLog.html?buildId=6167836buildTypeId=IgniteTests24Java8_RunAll]
> GridCacheProcessor should implement better parallelization when restoring
> partition states on startup
> -
>
> Key: IGNITE-13558
> URL: https://issues.apache.org/jira/browse/IGNITE-13558
> Project: Ignite
> Issue Type: Improvement
> Components: persistence
>Reporter: Sergey Chugunov
>Assignee: Denis Chudov
>Priority: Major
> Time Spent: 3h
> Remaining Estimate: 0h
>
> GridCacheProcessor#restorePartitionStates method tries to employ striped pool
> to restore partition states in parallel but level of parallelization is down
> only to cache group per thread.
> It is not enough and not utilizes resources effectively in case of one cache
> group much bigger than the others.
> We need to parallel restore process down to individual partitions to get the
> most from the available resources and speed up node startup.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (IGNITE-13558) GridCacheProcessor should implement better parallelization when restoring partition states on startup
[ https://issues.apache.org/jira/browse/IGNITE-13558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17407983#comment-17407983 ] Kirill Tkalenko commented on IGNITE-13558: -- Results of the run on my laptop (groups: 59, total partitions: 105_790): ||PR||Run 1||Run 2||Run 3|| |9243|17 501 ms|20_812 ms|19_157 ms| |9334|15_054 ms|18_500 ms|17_930 ms| > GridCacheProcessor should implement better parallelization when restoring > partition states on startup > - > > Key: IGNITE-13558 > URL: https://issues.apache.org/jira/browse/IGNITE-13558 > Project: Ignite > Issue Type: Improvement > Components: persistence >Reporter: Sergey Chugunov >Assignee: Denis Chudov >Priority: Major > Time Spent: 3h > Remaining Estimate: 0h > > GridCacheProcessor#restorePartitionStates method tries to employ striped pool > to restore partition states in parallel but level of parallelization is down > only to cache group per thread. > It is not enough and not utilizes resources effectively in case of one cache > group much bigger than the others. > We need to parallel restore process down to individual partitions to get the > most from the available resources and speed up node startup. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-13558) GridCacheProcessor should implement better parallelization when restoring partition states on startup
[ https://issues.apache.org/jira/browse/IGNITE-13558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17404400#comment-17404400 ] Evgeny Stanilovsky commented on IGNITE-13558: - [~ktkale...@gridgain.com] 100ms is not a case for such a tests, if this is one case for rejecting Denis approach - it`s not valid i hope. > GridCacheProcessor should implement better parallelization when restoring > partition states on startup > - > > Key: IGNITE-13558 > URL: https://issues.apache.org/jira/browse/IGNITE-13558 > Project: Ignite > Issue Type: Improvement > Components: persistence >Reporter: Sergey Chugunov >Assignee: Denis Chudov >Priority: Major > Time Spent: 3h > Remaining Estimate: 0h > > GridCacheProcessor#restorePartitionStates method tries to employ striped pool > to restore partition states in parallel but level of parallelization is down > only to cache group per thread. > It is not enough and not utilizes resources effectively in case of one cache > group much bigger than the others. > We need to parallel restore process down to individual partitions to get the > most from the available resources and speed up node startup. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-13558) GridCacheProcessor should implement better parallelization when restoring partition states on startup
[ https://issues.apache.org/jira/browse/IGNITE-13558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17404396#comment-17404396 ] Denis Chudov commented on IGNITE-13558: --- [~ktkale...@gridgain.com] difference of average values of both approaches is below 1% and is statistically insignificant. > GridCacheProcessor should implement better parallelization when restoring > partition states on startup > - > > Key: IGNITE-13558 > URL: https://issues.apache.org/jira/browse/IGNITE-13558 > Project: Ignite > Issue Type: Improvement > Components: persistence >Reporter: Sergey Chugunov >Assignee: Denis Chudov >Priority: Major > Time Spent: 3h > Remaining Estimate: 0h > > GridCacheProcessor#restorePartitionStates method tries to employ striped pool > to restore partition states in parallel but level of parallelization is down > only to cache group per thread. > It is not enough and not utilizes resources effectively in case of one cache > group much bigger than the others. > We need to parallel restore process down to individual partitions to get the > most from the available resources and speed up node startup. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-13558) GridCacheProcessor should implement better parallelization when restoring partition states on startup
[ https://issues.apache.org/jira/browse/IGNITE-13558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17404194#comment-17404194 ] Kirill Tkalenko commented on IGNITE-13558: -- [~Denis Chudov] I see that *9327* is better in two tests. > GridCacheProcessor should implement better parallelization when restoring > partition states on startup > - > > Key: IGNITE-13558 > URL: https://issues.apache.org/jira/browse/IGNITE-13558 > Project: Ignite > Issue Type: Improvement > Components: persistence >Reporter: Sergey Chugunov >Assignee: Denis Chudov >Priority: Major > Time Spent: 3h > Remaining Estimate: 0h > > GridCacheProcessor#restorePartitionStates method tries to employ striped pool > to restore partition states in parallel but level of parallelization is down > only to cache group per thread. > It is not enough and not utilizes resources effectively in case of one cache > group much bigger than the others. > We need to parallel restore process down to individual partitions to get the > most from the available resources and speed up node startup. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-13558) GridCacheProcessor should implement better parallelization when restoring partition states on startup
[ https://issues.apache.org/jira/browse/IGNITE-13558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17403177#comment-17403177 ] Denis Chudov commented on IGNITE-13558: --- [~ktkale...@gridgain.com] ^ > GridCacheProcessor should implement better parallelization when restoring > partition states on startup > - > > Key: IGNITE-13558 > URL: https://issues.apache.org/jira/browse/IGNITE-13558 > Project: Ignite > Issue Type: Improvement > Components: persistence >Reporter: Sergey Chugunov >Assignee: Denis Chudov >Priority: Major > Time Spent: 3h > Remaining Estimate: 0h > > GridCacheProcessor#restorePartitionStates method tries to employ striped pool > to restore partition states in parallel but level of parallelization is down > only to cache group per thread. > It is not enough and not utilizes resources effectively in case of one cache > group much bigger than the others. > We need to parallel restore process down to individual partitions to get the > most from the available resources and speed up node startup. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-13558) GridCacheProcessor should implement better parallelization when restoring partition states on startup
[ https://issues.apache.org/jira/browse/IGNITE-13558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17403175#comment-17403175 ] Denis Chudov commented on IGNITE-13558: --- Example for more partitions and data (groups: 60, total partitions: 137790) ||PR||Run 1||Run 2||Run 3|| |9243|4s628ms|4s597ms|4s444ms| |9327|4s520ms|4s475ms|4s567ms| |9334|7s795ms|7s565ms|7s442ms| > GridCacheProcessor should implement better parallelization when restoring > partition states on startup > - > > Key: IGNITE-13558 > URL: https://issues.apache.org/jira/browse/IGNITE-13558 > Project: Ignite > Issue Type: Improvement > Components: persistence >Reporter: Sergey Chugunov >Assignee: Denis Chudov >Priority: Major > Time Spent: 3h > Remaining Estimate: 0h > > GridCacheProcessor#restorePartitionStates method tries to employ striped pool > to restore partition states in parallel but level of parallelization is down > only to cache group per thread. > It is not enough and not utilizes resources effectively in case of one cache > group much bigger than the others. > We need to parallel restore process down to individual partitions to get the > most from the available resources and speed up node startup. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-13558) GridCacheProcessor should implement better parallelization when restoring partition states on startup
[ https://issues.apache.org/jira/browse/IGNITE-13558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17402159#comment-17402159 ] Denis Chudov commented on IGNITE-13558: --- [~ktkale...@gridgain.com] could you continue the review? > GridCacheProcessor should implement better parallelization when restoring > partition states on startup > - > > Key: IGNITE-13558 > URL: https://issues.apache.org/jira/browse/IGNITE-13558 > Project: Ignite > Issue Type: Improvement > Components: persistence >Reporter: Sergey Chugunov >Assignee: Denis Chudov >Priority: Major > Time Spent: 3h > Remaining Estimate: 0h > > GridCacheProcessor#restorePartitionStates method tries to employ striped pool > to restore partition states in parallel but level of parallelization is down > only to cache group per thread. > It is not enough and not utilizes resources effectively in case of one cache > group much bigger than the others. > We need to parallel restore process down to individual partitions to get the > most from the available resources and speed up node startup. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-13558) GridCacheProcessor should implement better parallelization when restoring partition states on startup
[ https://issues.apache.org/jira/browse/IGNITE-13558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17402133#comment-17402133 ] Denis Chudov commented on IGNITE-13558: --- I added RestorePartitionStateBenchmark into ignite-benchmarks module. It shows significant difference on a node with non-uniform distribution of cache group sizes between current version and this optimization (both approaches). In the same time, none of approaches has an advantage, as I expected. PR with my implementation: [https://github.com/apache/ignite/pull/9243] PR with alternative approach, task per partition: [https://github.com/apache/ignite/pull/9327] PR for tests of current version: [https://github.com/apache/ignite/pull/9334] Partition restore times on teamcity: ||PR||Run 1||Run 2||Run 3|| |9243|_795ms_|_804ms_|_935ms_| |9327|_941ms_|_750ms_|_769ms_| |9334|_1s515ms_|_1s590ms_|_1s471ms_| > GridCacheProcessor should implement better parallelization when restoring > partition states on startup > - > > Key: IGNITE-13558 > URL: https://issues.apache.org/jira/browse/IGNITE-13558 > Project: Ignite > Issue Type: Improvement > Components: persistence >Reporter: Sergey Chugunov >Assignee: Denis Chudov >Priority: Major > Time Spent: 2h 50m > Remaining Estimate: 0h > > GridCacheProcessor#restorePartitionStates method tries to employ striped pool > to restore partition states in parallel but level of parallelization is down > only to cache group per thread. > It is not enough and not utilizes resources effectively in case of one cache > group much bigger than the others. > We need to parallel restore process down to individual partitions to get the > most from the available resources and speed up node startup. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-13558) GridCacheProcessor should implement better parallelization when restoring partition states on startup
[ https://issues.apache.org/jira/browse/IGNITE-13558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17397096#comment-17397096 ] Kirill Tkalenko commented on IGNITE-13558: -- [~Denis Chudov] Put new comments in the PR, please comment on the proposals above. > GridCacheProcessor should implement better parallelization when restoring > partition states on startup > - > > Key: IGNITE-13558 > URL: https://issues.apache.org/jira/browse/IGNITE-13558 > Project: Ignite > Issue Type: Improvement > Components: persistence >Reporter: Sergey Chugunov >Assignee: Denis Chudov >Priority: Major > Time Spent: 2h 20m > Remaining Estimate: 0h > > GridCacheProcessor#restorePartitionStates method tries to employ striped pool > to restore partition states in parallel but level of parallelization is down > only to cache group per thread. > It is not enough and not utilizes resources effectively in case of one cache > group much bigger than the others. > We need to parallel restore process down to individual partitions to get the > most from the available resources and speed up node startup. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-13558) GridCacheProcessor should implement better parallelization when restoring partition states on startup
[
https://issues.apache.org/jira/browse/IGNITE-13558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17393411#comment-17393411
]
Ignite TC Bot commented on IGNITE-13558:
{panel:title=Branch: [pull/9243/head] Base: [master] : No blockers
found!|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}{panel}
{panel:title=Branch: [pull/9243/head] Base: [master] : No new tests
found!|borderStyle=dashed|borderColor=#ccc|titleBGColor=#F7D6C1}{panel}
[TeamCity *-- Run :: All*
Results|https://ci.ignite.apache.org/viewLog.html?buildId=6116602buildTypeId=IgniteTests24Java8_RunAll]
> GridCacheProcessor should implement better parallelization when restoring
> partition states on startup
> -
>
> Key: IGNITE-13558
> URL: https://issues.apache.org/jira/browse/IGNITE-13558
> Project: Ignite
> Issue Type: Improvement
> Components: persistence
>Reporter: Sergey Chugunov
>Assignee: Denis Chudov
>Priority: Major
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> GridCacheProcessor#restorePartitionStates method tries to employ striped pool
> to restore partition states in parallel but level of parallelization is down
> only to cache group per thread.
> It is not enough and not utilizes resources effectively in case of one cache
> group much bigger than the others.
> We need to parallel restore process down to individual partitions to get the
> most from the available resources and speed up node startup.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (IGNITE-13558) GridCacheProcessor should implement better parallelization when restoring partition states on startup
[ https://issues.apache.org/jira/browse/IGNITE-13558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17381927#comment-17381927 ] Denis Chudov commented on IGNITE-13558: --- [~ktkale...@gridgain.com] could you pls review PR [https://github.com/apache/ignite/pull/9243] ? > GridCacheProcessor should implement better parallelization when restoring > partition states on startup > - > > Key: IGNITE-13558 > URL: https://issues.apache.org/jira/browse/IGNITE-13558 > Project: Ignite > Issue Type: Improvement > Components: persistence >Reporter: Sergey Chugunov >Assignee: Denis Chudov >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > > GridCacheProcessor#restorePartitionStates method tries to employ striped pool > to restore partition states in parallel but level of parallelization is down > only to cache group per thread. > It is not enough and not utilizes resources effectively in case of one cache > group much bigger than the others. > We need to parallel restore process down to individual partitions to get the > most from the available resources and speed up node startup. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-13558) GridCacheProcessor should implement better parallelization when restoring partition states on startup
[
https://issues.apache.org/jira/browse/IGNITE-13558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17381921#comment-17381921
]
Denis Chudov commented on IGNITE-13558:
---
There was a deadlock in
{noformat}
org.apache.ignite.internal.processors.cache.persistence.IgnitePdsCacheStartStopWithFreqCheckpointTest#testCheckpointDeadlock{noformat}
on CheckpointReadWriteLock. I tried to fix it, now seems to be ok, after
multiple runs of PDS suites.
> GridCacheProcessor should implement better parallelization when restoring
> partition states on startup
> -
>
> Key: IGNITE-13558
> URL: https://issues.apache.org/jira/browse/IGNITE-13558
> Project: Ignite
> Issue Type: Improvement
> Components: persistence
>Reporter: Sergey Chugunov
>Assignee: Denis Chudov
>Priority: Major
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> GridCacheProcessor#restorePartitionStates method tries to employ striped pool
> to restore partition states in parallel but level of parallelization is down
> only to cache group per thread.
> It is not enough and not utilizes resources effectively in case of one cache
> group much bigger than the others.
> We need to parallel restore process down to individual partitions to get the
> most from the available resources and speed up node startup.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (IGNITE-13558) GridCacheProcessor should implement better parallelization when restoring partition states on startup
[
https://issues.apache.org/jira/browse/IGNITE-13558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17381922#comment-17381922
]
Ignite TC Bot commented on IGNITE-13558:
{panel:title=Branch: [pull/9243/head] Base: [master] : No blockers
found!|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}{panel}
{panel:title=Branch: [pull/9243/head] Base: [master] : No new tests
found!|borderStyle=dashed|borderColor=#ccc|titleBGColor=#F7D6C1}{panel}
[TeamCity *-- Run :: All*
Results|https://ci.ignite.apache.org/viewLog.html?buildId=6087409buildTypeId=IgniteTests24Java8_RunAll]
> GridCacheProcessor should implement better parallelization when restoring
> partition states on startup
> -
>
> Key: IGNITE-13558
> URL: https://issues.apache.org/jira/browse/IGNITE-13558
> Project: Ignite
> Issue Type: Improvement
> Components: persistence
>Reporter: Sergey Chugunov
>Assignee: Denis Chudov
>Priority: Major
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> GridCacheProcessor#restorePartitionStates method tries to employ striped pool
> to restore partition states in parallel but level of parallelization is down
> only to cache group per thread.
> It is not enough and not utilizes resources effectively in case of one cache
> group much bigger than the others.
> We need to parallel restore process down to individual partitions to get the
> most from the available resources and speed up node startup.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (IGNITE-13558) GridCacheProcessor should implement better parallelization when restoring partition states on startup
[
https://issues.apache.org/jira/browse/IGNITE-13558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17380498#comment-17380498
]
Ignite TC Bot commented on IGNITE-13558:
{panel:title=Branch: [pull/9243/head] Base: [master] : No blockers
found!|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}{panel}
{panel:title=Branch: [pull/9243/head] Base: [master] : No new tests
found!|borderStyle=dashed|borderColor=#ccc|titleBGColor=#F7D6C1}{panel}
[TeamCity *-- Run :: All*
Results|https://ci.ignite.apache.org/viewLog.html?buildId=6079240buildTypeId=IgniteTests24Java8_RunAll]
> GridCacheProcessor should implement better parallelization when restoring
> partition states on startup
> -
>
> Key: IGNITE-13558
> URL: https://issues.apache.org/jira/browse/IGNITE-13558
> Project: Ignite
> Issue Type: Improvement
> Components: persistence
>Reporter: Sergey Chugunov
>Assignee: Denis Chudov
>Priority: Major
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> GridCacheProcessor#restorePartitionStates method tries to employ striped pool
> to restore partition states in parallel but level of parallelization is down
> only to cache group per thread.
> It is not enough and not utilizes resources effectively in case of one cache
> group much bigger than the others.
> We need to parallel restore process down to individual partitions to get the
> most from the available resources and speed up node startup.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (IGNITE-13558) GridCacheProcessor should implement better parallelization when restoring partition states on startup
[ https://issues.apache.org/jira/browse/IGNITE-13558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17218164#comment-17218164 ] Sergey Chugunov commented on IGNITE-13558: -- [~ibessonov], It turned out this change introduced a possible deadlock, so we need to approach it one more time. Reverted the commit from master for now. > GridCacheProcessor should implement better parallelization when restoring > partition states on startup > - > > Key: IGNITE-13558 > URL: https://issues.apache.org/jira/browse/IGNITE-13558 > Project: Ignite > Issue Type: Improvement > Components: persistence >Reporter: Sergey Chugunov >Assignee: Ivan Bessonov >Priority: Major > Fix For: 2.10 > > Time Spent: 1h > Remaining Estimate: 0h > > GridCacheProcessor#restorePartitionStates method tries to employ striped pool > to restore partition states in parallel but level of parallelization is down > only to cache group per thread. > It is not enough and not utilizes resources effectively in case of one cache > group much bigger than the others. > We need to parallel restore process down to individual partitions to get the > most from the available resources and speed up node startup. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-13558) GridCacheProcessor should implement better parallelization when restoring partition states on startup
[
https://issues.apache.org/jira/browse/IGNITE-13558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17215266#comment-17215266
]
Ignite TC Bot commented on IGNITE-13558:
{panel:title=Branch: [pull/8333/head] Base: [master] : Possible Blockers
(6)|borderStyle=dashed|borderColor=#ccc|titleBGColor=#F7D6C1}
{color:#d04437}MVCC Cache 7{color} [[tests 0 TIMEOUT , Exit Code
|https://ci.ignite.apache.org/viewLog.html?buildId=5659448]]
{color:#d04437}Cache 6{color} [[tests 0 Exit Code
|https://ci.ignite.apache.org/viewLog.html?buildId=5659412]]
{color:#d04437}MVCC Cache 1{color} [[tests 0 TIMEOUT , Exit Code
|https://ci.ignite.apache.org/viewLog.html?buildId=5659442]]
{color:#d04437}MVCC PDS 2{color} [[tests 0 TIMEOUT , Exit Code
|https://ci.ignite.apache.org/viewLog.html?buildId=5659452]]
{color:#d04437}Thin Client: Java{color} [[tests 0 Exit Code
|https://ci.ignite.apache.org/viewLog.html?buildId=5659367]]
{color:#d04437}MVCC Cache 2{color} [[tests 0 Exit Code
|https://ci.ignite.apache.org/viewLog.html?buildId=5659443]]
{panel}
{panel:title=Branch: [pull/8333/head] Base: [master] : No new tests
found!|borderStyle=dashed|borderColor=#ccc|titleBGColor=#F7D6C1}{panel}
[TeamCity *-- Run :: All*
Results|https://ci.ignite.apache.org/viewLog.html?buildId=5659464buildTypeId=IgniteTests24Java8_RunAll]
> GridCacheProcessor should implement better parallelization when restoring
> partition states on startup
> -
>
> Key: IGNITE-13558
> URL: https://issues.apache.org/jira/browse/IGNITE-13558
> Project: Ignite
> Issue Type: Improvement
> Components: persistence
>Reporter: Sergey Chugunov
>Assignee: Ivan Bessonov
>Priority: Major
> Fix For: 2.10
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> GridCacheProcessor#restorePartitionStates method tries to employ striped pool
> to restore partition states in parallel but level of parallelization is down
> only to cache group per thread.
> It is not enough and not utilizes resources effectively in case of one cache
> group much bigger than the others.
> We need to parallel restore process down to individual partitions to get the
> most from the available resources and speed up node startup.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (IGNITE-13558) GridCacheProcessor should implement better parallelization when restoring partition states on startup
[ https://issues.apache.org/jira/browse/IGNITE-13558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17213020#comment-17213020 ] Semyon Danilov commented on IGNITE-13558: - Changes look good to me > GridCacheProcessor should implement better parallelization when restoring > partition states on startup > - > > Key: IGNITE-13558 > URL: https://issues.apache.org/jira/browse/IGNITE-13558 > Project: Ignite > Issue Type: Improvement > Components: persistence >Reporter: Sergey Chugunov >Assignee: Ivan Bessonov >Priority: Major > Fix For: 2.10 > > Time Spent: 40m > Remaining Estimate: 0h > > GridCacheProcessor#restorePartitionStates method tries to employ striped pool > to restore partition states in parallel but level of parallelization is down > only to cache group per thread. > It is not enough and not utilizes resources effectively in case of one cache > group much bigger than the others. > We need to parallel restore process down to individual partitions to get the > most from the available resources and speed up node startup. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (IGNITE-13558) GridCacheProcessor should implement better parallelization when restoring partition states on startup
[
https://issues.apache.org/jira/browse/IGNITE-13558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17212332#comment-17212332
]
Ignite TC Bot commented on IGNITE-13558:
{panel:title=Branch: [pull/8333/head] Base: [master] : Possible Blockers
(6)|borderStyle=dashed|borderColor=#ccc|titleBGColor=#F7D6C1}
{color:#d04437}MVCC Cache 7{color} [[tests 0 TIMEOUT , Exit Code
|https://ci.ignite.apache.org/viewLog.html?buildId=5659448]]
{color:#d04437}Cache 6{color} [[tests 0 Exit Code
|https://ci.ignite.apache.org/viewLog.html?buildId=5659412]]
{color:#d04437}MVCC Cache 1{color} [[tests 0 TIMEOUT , Exit Code
|https://ci.ignite.apache.org/viewLog.html?buildId=5659442]]
{color:#d04437}MVCC PDS 2{color} [[tests 0 TIMEOUT , Exit Code
|https://ci.ignite.apache.org/viewLog.html?buildId=5659452]]
{color:#d04437}Thin Client: Java{color} [[tests 0 Exit Code
|https://ci.ignite.apache.org/viewLog.html?buildId=5659367]]
{color:#d04437}MVCC Cache 2{color} [[tests 0 Exit Code
|https://ci.ignite.apache.org/viewLog.html?buildId=5659443]]
{panel}
{panel:title=Branch: [pull/8333/head] Base: [master] : No new tests
found!|borderStyle=dashed|borderColor=#ccc|titleBGColor=#F7D6C1}{panel}
[TeamCity *-- Run :: All*
Results|https://ci.ignite.apache.org/viewLog.html?buildId=5659464buildTypeId=IgniteTests24Java8_RunAll]
> GridCacheProcessor should implement better parallelization when restoring
> partition states on startup
> -
>
> Key: IGNITE-13558
> URL: https://issues.apache.org/jira/browse/IGNITE-13558
> Project: Ignite
> Issue Type: Improvement
> Components: persistence
>Reporter: Sergey Chugunov
>Assignee: Ivan Bessonov
>Priority: Major
> Fix For: 2.10
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> GridCacheProcessor#restorePartitionStates method tries to employ striped pool
> to restore partition states in parallel but level of parallelization is down
> only to cache group per thread.
> It is not enough and not utilizes resources effectively in case of one cache
> group much bigger than the others.
> We need to parallel restore process down to individual partitions to get the
> most from the available resources and speed up node startup.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
