[jira] [Commented] (SOLR-9264) Optimize ZkController.publishAndWaitForDownStates
[ https://issues.apache.org/jira/browse/SOLR-9264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15357029#comment-15357029 ] ASF subversion and git services commented on SOLR-9264: --- Commit 1ce09b482e9370649e9a7421b4961a67e744e46f in lucene-solr's branch refs/heads/branch_6x from [~shalinmangar] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=1ce09b4 ] SOLR-9264: Remove unused imports (cherry picked from commit 74c8606) > Optimize ZkController.publishAndWaitForDownStates > - > > Key: SOLR-9264 > URL: https://issues.apache.org/jira/browse/SOLR-9264 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Reporter: Shalin Shekhar Mangar >Assignee: Shalin Shekhar Mangar > Fix For: 6.2, master (7.0) > > Attachments: SOLR-9264.patch, SOLR-9264.patch, SOLR-9264.patch > > > ZkController.publishAndWaitForDownStates keeps looping over all collections > in the cluster state to ensure that every replica hosted on the current node > has been marked as down. This is wasteful when you have a large number of > collections because each access to a non-watched collection gets data from > ZK. Instead, we can watch the interesting collections (i.e. which have > replicas hosted locally) and wait till we see the required state. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9264) Optimize ZkController.publishAndWaitForDownStates
[ https://issues.apache.org/jira/browse/SOLR-9264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15357028#comment-15357028 ] ASF subversion and git services commented on SOLR-9264: --- Commit 74c86063cf94dcc4dc022776bba31ae278686b42 in lucene-solr's branch refs/heads/master from [~shalinmangar] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=74c8606 ] SOLR-9264: Remove unused imports > Optimize ZkController.publishAndWaitForDownStates > - > > Key: SOLR-9264 > URL: https://issues.apache.org/jira/browse/SOLR-9264 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Reporter: Shalin Shekhar Mangar >Assignee: Shalin Shekhar Mangar > Fix For: 6.2, master (7.0) > > Attachments: SOLR-9264.patch, SOLR-9264.patch, SOLR-9264.patch > > > ZkController.publishAndWaitForDownStates keeps looping over all collections > in the cluster state to ensure that every replica hosted on the current node > has been marked as down. This is wasteful when you have a large number of > collections because each access to a non-watched collection gets data from > ZK. Instead, we can watch the interesting collections (i.e. which have > replicas hosted locally) and wait till we see the required state. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9264) Optimize ZkController.publishAndWaitForDownStates
[ https://issues.apache.org/jira/browse/SOLR-9264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15356837#comment-15356837 ] ASF subversion and git services commented on SOLR-9264: --- Commit 8cb37842ec531a84469607971024336b68c6ed50 in lucene-solr's branch refs/heads/branch_6x from [~shalinmangar] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=8cb3784 ] SOLR-9264: Optimize ZkController.publishAndWaitForDownStates to not read all collection states and watch relevant collections instead (cherry picked from commit 015e0fc) > Optimize ZkController.publishAndWaitForDownStates > - > > Key: SOLR-9264 > URL: https://issues.apache.org/jira/browse/SOLR-9264 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Reporter: Shalin Shekhar Mangar >Assignee: Shalin Shekhar Mangar > Fix For: 6.2, master (7.0) > > Attachments: SOLR-9264.patch, SOLR-9264.patch, SOLR-9264.patch > > > ZkController.publishAndWaitForDownStates keeps looping over all collections > in the cluster state to ensure that every replica hosted on the current node > has been marked as down. This is wasteful when you have a large number of > collections because each access to a non-watched collection gets data from > ZK. Instead, we can watch the interesting collections (i.e. which have > replicas hosted locally) and wait till we see the required state. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9264) Optimize ZkController.publishAndWaitForDownStates
[ https://issues.apache.org/jira/browse/SOLR-9264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15356835#comment-15356835 ] ASF subversion and git services commented on SOLR-9264: --- Commit 015e0fc1cf1d581c9657cd8f5588062c02588793 in lucene-solr's branch refs/heads/master from [~shalinmangar] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=015e0fc ] SOLR-9264: Optimize ZkController.publishAndWaitForDownStates to not read all collection states and watch relevant collections instead > Optimize ZkController.publishAndWaitForDownStates > - > > Key: SOLR-9264 > URL: https://issues.apache.org/jira/browse/SOLR-9264 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Reporter: Shalin Shekhar Mangar >Assignee: Shalin Shekhar Mangar > Fix For: 6.2, master (7.0) > > Attachments: SOLR-9264.patch, SOLR-9264.patch, SOLR-9264.patch > > > ZkController.publishAndWaitForDownStates keeps looping over all collections > in the cluster state to ensure that every replica hosted on the current node > has been marked as down. This is wasteful when you have a large number of > collections because each access to a non-watched collection gets data from > ZK. Instead, we can watch the interesting collections (i.e. which have > replicas hosted locally) and wait till we see the required state. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9264) Optimize ZkController.publishAndWaitForDownStates
[ https://issues.apache.org/jira/browse/SOLR-9264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15356600#comment-15356600 ] Hrishikesh Gadre commented on SOLR-9264: bq. since this variable is in the local scope. OK my bad. I see that the AtomicBoolean is indeed outside the scope of lambda expression. Now I don't quite understand how does this fix the concurrency issue. Don't we need separate AtomicBoolean per collection ? > Optimize ZkController.publishAndWaitForDownStates > - > > Key: SOLR-9264 > URL: https://issues.apache.org/jira/browse/SOLR-9264 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Reporter: Shalin Shekhar Mangar >Assignee: Shalin Shekhar Mangar > Fix For: 6.2, master (7.0) > > Attachments: SOLR-9264.patch, SOLR-9264.patch > > > ZkController.publishAndWaitForDownStates keeps looping over all collections > in the cluster state to ensure that every replica hosted on the current node > has been marked as down. This is wasteful when you have a large number of > collections because each access to a non-watched collection gets data from > ZK. Instead, we can watch the interesting collections (i.e. which have > replicas hosted locally) and wait till we see the required state. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9264) Optimize ZkController.publishAndWaitForDownStates
[ https://issues.apache.org/jira/browse/SOLR-9264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15356595#comment-15356595 ] Hrishikesh Gadre commented on SOLR-9264: [~ shalinmangar] Thanks for the updated patch. It seems to me that the logic based on AtomicBoolean is probably not sufficient if the callback is invoked multiple times *sequentially* for the same collection since this variable is in the local scope. Is it a possibility? I think instead of AtomicBoolean we should use a concurrent hashmap (outside the scope of the lambda expression). This map should be pre-populated with the collection names before registering the callback. We can even reuse the collectionsWithLocalReplica variable for this purpose (i.e. instead of Set, we will use ConcurrentHashMap type). Inside the callback we can use the remove method in a similar fashion to compareAndSet. i.e. instead of counted.compareAndSet(false, true) do collectionsWithLocalReplica.remove(collectionName) != null What do you think? > Optimize ZkController.publishAndWaitForDownStates > - > > Key: SOLR-9264 > URL: https://issues.apache.org/jira/browse/SOLR-9264 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Reporter: Shalin Shekhar Mangar >Assignee: Shalin Shekhar Mangar > Fix For: 6.2, master (7.0) > > Attachments: SOLR-9264.patch, SOLR-9264.patch > > > ZkController.publishAndWaitForDownStates keeps looping over all collections > in the cluster state to ensure that every replica hosted on the current node > has been marked as down. This is wasteful when you have a large number of > collections because each access to a non-watched collection gets data from > ZK. Instead, we can watch the interesting collections (i.e. which have > replicas hosted locally) and wait till we see the required state. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9264) Optimize ZkController.publishAndWaitForDownStates
[ https://issues.apache.org/jira/browse/SOLR-9264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15355865#comment-15355865 ] Hrishikesh Gadre commented on SOLR-9264: [~ shalinmangar] I think the patch looks good. Only couple of minor comments, - Can we rename the "interestingCollections" and "interestingCollection" variable to something like "collectionsWithLocalReplica" and "collectionWithLocalReplica" ? It is a little difficult to understand what "interesting" means in this context without reading the JIRA description... - Is it possible for the callback to be delivered more than once? If yes then we should probably add some defensive check before invoking the countDown method on the latch. > Optimize ZkController.publishAndWaitForDownStates > - > > Key: SOLR-9264 > URL: https://issues.apache.org/jira/browse/SOLR-9264 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Reporter: Shalin Shekhar Mangar >Assignee: Shalin Shekhar Mangar > Fix For: 6.2, master (7.0) > > Attachments: SOLR-9264.patch > > > ZkController.publishAndWaitForDownStates keeps looping over all collections > in the cluster state to ensure that every replica hosted on the current node > has been marked as down. This is wasteful when you have a large number of > collections because each access to a non-watched collection gets data from > ZK. Instead, we can watch the interesting collections (i.e. which have > replicas hosted locally) and wait till we see the required state. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
