[jira] [Commented] (SOLR-17650) Determine executor for UpdateLog reading
[ https://issues.apache.org/jira/browse/SOLR-17650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17933686#comment-17933686 ] ASF subversion and git services commented on SOLR-17650: Commit 0a50edab23350947157731ab973a1f07ec7cf69a in solr's branch refs/heads/deprecations from Houston Putman [ https://gitbox.apache.org/repos/asf?p=solr.git;h=0a50edab233 ] SOLR-17650: Fix tests for unordered buffered updates (#3197) > Determine executor for UpdateLog reading > > > Key: SOLR-17650 > URL: https://issues.apache.org/jira/browse/SOLR-17650 > Project: Solr > Issue Type: Improvement >Reporter: Houston Putman >Assignee: Houston Putman >Priority: Major > Labels: pull-request-available > Fix For: 9.9 > > Time Spent: 1h > Remaining Estimate: 0h > > Currently most operations that read from the updateLog use the > OrderedExecutor that is setup for that purpose. However, TLOG replicas, when > taking leadership, read from the updateLog without the executor (using the > inSortedOrder = true parameter). > SOLR-17391 changed the default executor for updateLog reading to use a fixed > coreSize thread pool, which seems like started to enable parallel reading of > the updateLog (which was supposed to be true beforehand, but it looks like > our understanding of the executor was wrong). > This ticket's purpose is to: > # Investigate the correct way of reading the updateLog using an executor > # Determine if there needs to actually be multiple ways > (inSortedOrder=true/false) > # Make sure the tests pass with whatever we decide. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-17650) Determine executor for UpdateLog reading
[ https://issues.apache.org/jira/browse/SOLR-17650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17930453#comment-17930453 ] ASF subversion and git services commented on SOLR-17650: Commit 76fdb86b3ee321bb43ef9b1de2d19d04be517922 in solr's branch refs/heads/branch_9x from Houston Putman [ https://gitbox.apache.org/repos/asf?p=solr.git;h=76fdb86b3ee ] SOLR-17650: Add in missing gradle dependency > Determine executor for UpdateLog reading > > > Key: SOLR-17650 > URL: https://issues.apache.org/jira/browse/SOLR-17650 > Project: Solr > Issue Type: Improvement >Reporter: Houston Putman >Assignee: Houston Putman >Priority: Major > Labels: pull-request-available > Fix For: 9.9 > > Time Spent: 1h > Remaining Estimate: 0h > > Currently most operations that read from the updateLog use the > OrderedExecutor that is setup for that purpose. However, TLOG replicas, when > taking leadership, read from the updateLog without the executor (using the > inSortedOrder = true parameter). > SOLR-17391 changed the default executor for updateLog reading to use a fixed > coreSize thread pool, which seems like started to enable parallel reading of > the updateLog (which was supposed to be true beforehand, but it looks like > our understanding of the executor was wrong). > This ticket's purpose is to: > # Investigate the correct way of reading the updateLog using an executor > # Determine if there needs to actually be multiple ways > (inSortedOrder=true/false) > # Make sure the tests pass with whatever we decide. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-17650) Determine executor for UpdateLog reading
[ https://issues.apache.org/jira/browse/SOLR-17650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17930388#comment-17930388 ] ASF subversion and git services commented on SOLR-17650: Commit e620879f5e1acba4e60ca78adcfd548bc511dc26 in solr's branch refs/heads/branch_9x from Houston Putman [ https://gitbox.apache.org/repos/asf?p=solr.git;h=e620879f5e1 ] SOLR-17650: Fix tests for unordered buffered updates (#3197) (cherry picked from commit 0a50edab23350947157731ab973a1f07ec7cf69a) > Determine executor for UpdateLog reading > > > Key: SOLR-17650 > URL: https://issues.apache.org/jira/browse/SOLR-17650 > Project: Solr > Issue Type: Improvement >Reporter: Houston Putman >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > Currently most operations that read from the updateLog use the > OrderedExecutor that is setup for that purpose. However, TLOG replicas, when > taking leadership, read from the updateLog without the executor (using the > inSortedOrder = true parameter). > SOLR-17391 changed the default executor for updateLog reading to use a fixed > coreSize thread pool, which seems like started to enable parallel reading of > the updateLog (which was supposed to be true beforehand, but it looks like > our understanding of the executor was wrong). > This ticket's purpose is to: > # Investigate the correct way of reading the updateLog using an executor > # Determine if there needs to actually be multiple ways > (inSortedOrder=true/false) > # Make sure the tests pass with whatever we decide. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-17650) Determine executor for UpdateLog reading
[ https://issues.apache.org/jira/browse/SOLR-17650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17930372#comment-17930372 ] ASF subversion and git services commented on SOLR-17650: Commit 0a50edab23350947157731ab973a1f07ec7cf69a in solr's branch refs/heads/main from Houston Putman [ https://gitbox.apache.org/repos/asf?p=solr.git;h=0a50edab233 ] SOLR-17650: Fix tests for unordered buffered updates (#3197) > Determine executor for UpdateLog reading > > > Key: SOLR-17650 > URL: https://issues.apache.org/jira/browse/SOLR-17650 > Project: Solr > Issue Type: Improvement >Reporter: Houston Putman >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > Currently most operations that read from the updateLog use the > OrderedExecutor that is setup for that purpose. However, TLOG replicas, when > taking leadership, read from the updateLog without the executor (using the > inSortedOrder = true parameter). > SOLR-17391 changed the default executor for updateLog reading to use a fixed > coreSize thread pool, which seems like started to enable parallel reading of > the updateLog (which was supposed to be true beforehand, but it looks like > our understanding of the executor was wrong). > This ticket's purpose is to: > # Investigate the correct way of reading the updateLog using an executor > # Determine if there needs to actually be multiple ways > (inSortedOrder=true/false) > # Make sure the tests pass with whatever we decide. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-17650) Determine executor for UpdateLog reading
[ https://issues.apache.org/jira/browse/SOLR-17650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17925714#comment-17925714 ] Mark Robert Miller commented on SOLR-17650: --- I took a look at this. In regards to multiple ways to read the updateLog: We started with no parallel reply. You need to be able to read in sorted order because updates might be reordered, say going from leader to replica, and you don’t drop based on versions at the tlog level and you have things like dependent updates. So sorted order read will make sure you read in logical order from the tlog with a sorted reader. The reason we wanted to add parallel replay is because if you have buffered and are trying to catch up while updates are continuously coming in, maybe it takes forever to catch up or a long time, because you are buffering as fast or faster than you are replaying. So OrderedExecutor was added, in this case you don’t have to read in sorted order, the OrderedExecutor handles reorders of updates to the same ID and delete by query is handled by waiting for the executor to drain before replaying it. I think Yonik didn’t always use parallel replay because typically you wouldn’t expect to be replaying a lot, and so the coordination overhead is often probably not worth it in the normal case. As to the fails, Houston's analysis looks correct to me, before SOLR-17391, only one thread appears to have been used, and now thread scheduling can changed the order of applied buffered updates. > Determine executor for UpdateLog reading > > > Key: SOLR-17650 > URL: https://issues.apache.org/jira/browse/SOLR-17650 > Project: Solr > Issue Type: Improvement >Reporter: Houston Putman >Priority: Major > > Currently most operations that read from the updateLog use the > OrderedExecutor that is setup for that purpose. However, TLOG replicas, when > taking leadership, read from the updateLog without the executor (using the > inSortedOrder = true parameter). > SOLR-17391 changed the default executor for updateLog reading to use a fixed > coreSize thread pool, which seems like started to enable parallel reading of > the updateLog (which was supposed to be true beforehand, but it looks like > our understanding of the executor was wrong). > This ticket's purpose is to: > # Investigate the correct way of reading the updateLog using an executor > # Determine if there needs to actually be multiple ways > (inSortedOrder=true/false) > # Make sure the tests pass with whatever we decide. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-17650) Determine executor for UpdateLog reading
[
https://issues.apache.org/jira/browse/SOLR-17650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17923488#comment-17923488
]
Houston Putman commented on SOLR-17650:
---
Currently the buffered update tests (
{{TestRecovery.testBufferedMultipleCalls}} , {{
TestRecovery.testBufferedMultipleCalls}} , {{TestRecovery.testDropBuffered}} )
are failing because of this executor change.
> Determine executor for UpdateLog reading
>
>
> Key: SOLR-17650
> URL: https://issues.apache.org/jira/browse/SOLR-17650
> Project: Solr
> Issue Type: Improvement
>Reporter: Houston Putman
>Priority: Major
>
> Currently most operations that read from the updateLog use the
> OrderedExecutor that is setup for that purpose. However, TLOG replicas, when
> taking leadership, read from the updateLog without the executor (using the
> inSortedOrder = true parameter).
> SOLR-17391 changed the default executor for updateLog reading to use a fixed
> coreSize thread pool, which seems like started to enable parallel reading of
> the updateLog (which was supposed to be true beforehand, but it looks like
> our understanding of the executor was wrong).
> This ticket's purpose is to:
> # Investigate the correct way of reading the updateLog using an executor
> # Determine if there needs to actually be multiple ways
> (inSortedOrder=true/false)
> # Make sure the tests pass with whatever we decide.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
