[jira] [Commented] (SOLR-14413) allow timeAllowed and cursorMark parameters
[ https://issues.apache.org/jira/browse/SOLR-14413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17262847#comment-17262847 ] ASF subversion and git services commented on SOLR-14413: Commit 87ed3439e88b664fe9ee935152fef700a47182de in lucene-solr's branch refs/heads/branch_8x from Mike Drob [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=87ed343 ] SOLR-14413 fix unit test to use delayed handler (#2189) > allow timeAllowed and cursorMark parameters > --- > > Key: SOLR-14413 > URL: https://issues.apache.org/jira/browse/SOLR-14413 > Project: Solr > Issue Type: Improvement > Components: search >Reporter: John Gallagher >Assignee: Mike Drob >Priority: Minor > Fix For: 8.8, master (9.0) > > Attachments: SOLR-14413-bram.patch, SOLR-14413-jg-update1.patch, > SOLR-14413-jg-update2.patch, SOLR-14413-jg-update3.patch, SOLR-14413.patch, > SOLR-14413.testfix.patch, Screen Shot 2020-10-23 at 10.08.26 PM.png, Screen > Shot 2020-10-23 at 10.09.11 PM.png, image-2020-08-18-16-56-41-736.png, > image-2020-08-18-16-56-59-178.png, image-2020-08-21-14-18-36-229.png, > timeallowed_cursormarks_results.txt > > Time Spent: 3h 10m > Remaining Estimate: 0h > > Ever since cursorMarks were introduced in SOLR-5463 in 2014, cursorMark and > timeAllowed parameters were not allowed in combination ("Can not search using > both cursorMark and timeAllowed") > , from [QueryComponent.java|#L359]]: > > {code:java} > > if (null != rb.getCursorMark() && 0 < timeAllowed) { > // fundamentally incompatible > throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Can not > search using both " + CursorMarkParams.CURSOR_MARK_PARAM + " and " + > CommonParams.TIME_ALLOWED); > } {code} > While theoretically impure to use them in combination, it is often desirable > to support cursormarks-style deep paging and attempt to protect Solr nodes > from runaway queries using timeAllowed, in the hopes that most of the time, > the query completes in the allotted time, and there is no conflict. > > However if the query takes too long, it may be preferable to end the query > and protect the Solr node and provide the user with a somewhat inaccurate > sorted list. As noted in SOLR-6930, SOLR-5986 and others, timeAllowed is > frequently used to prevent runaway load. In fact, cursorMark and > shards.tolerant are allowed in combination, so any argument in favor of > purity would be a bit muddied in my opinion. > > This was discussed once in the mailing list that I can find: > [https://mail-archives.apache.org/mod_mbox/lucene-solr-user/201506.mbox/%3c5591740b.4080...@elyograg.org%3E] > It did not look like there was strong support for preventing the combination. > > I have tested cursorMark and timeAllowed combination together, and even when > partial results are returned because the timeAllowed is exceeded, the > cursorMark response value is still valid and reasonable. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14413) allow timeAllowed and cursorMark parameters
[ https://issues.apache.org/jira/browse/SOLR-14413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17262841#comment-17262841 ] ASF subversion and git services commented on SOLR-14413: Commit a429b969d87090c9cc1ec787553528bece0d809e in lucene-solr's branch refs/heads/master from Mike Drob [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=a429b96 ] SOLR-14413 fix unit test to use delayed handler (#2189) > allow timeAllowed and cursorMark parameters > --- > > Key: SOLR-14413 > URL: https://issues.apache.org/jira/browse/SOLR-14413 > Project: Solr > Issue Type: Improvement > Components: search >Reporter: John Gallagher >Assignee: Mike Drob >Priority: Minor > Fix For: 8.8, master (9.0) > > Attachments: SOLR-14413-bram.patch, SOLR-14413-jg-update1.patch, > SOLR-14413-jg-update2.patch, SOLR-14413-jg-update3.patch, SOLR-14413.patch, > SOLR-14413.testfix.patch, Screen Shot 2020-10-23 at 10.08.26 PM.png, Screen > Shot 2020-10-23 at 10.09.11 PM.png, image-2020-08-18-16-56-41-736.png, > image-2020-08-18-16-56-59-178.png, image-2020-08-21-14-18-36-229.png, > timeallowed_cursormarks_results.txt > > Time Spent: 3h 10m > Remaining Estimate: 0h > > Ever since cursorMarks were introduced in SOLR-5463 in 2014, cursorMark and > timeAllowed parameters were not allowed in combination ("Can not search using > both cursorMark and timeAllowed") > , from [QueryComponent.java|#L359]]: > > {code:java} > > if (null != rb.getCursorMark() && 0 < timeAllowed) { > // fundamentally incompatible > throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Can not > search using both " + CursorMarkParams.CURSOR_MARK_PARAM + " and " + > CommonParams.TIME_ALLOWED); > } {code} > While theoretically impure to use them in combination, it is often desirable > to support cursormarks-style deep paging and attempt to protect Solr nodes > from runaway queries using timeAllowed, in the hopes that most of the time, > the query completes in the allotted time, and there is no conflict. > > However if the query takes too long, it may be preferable to end the query > and protect the Solr node and provide the user with a somewhat inaccurate > sorted list. As noted in SOLR-6930, SOLR-5986 and others, timeAllowed is > frequently used to prevent runaway load. In fact, cursorMark and > shards.tolerant are allowed in combination, so any argument in favor of > purity would be a bit muddied in my opinion. > > This was discussed once in the mailing list that I can find: > [https://mail-archives.apache.org/mod_mbox/lucene-solr-user/201506.mbox/%3c5591740b.4080...@elyograg.org%3E] > It did not look like there was strong support for preventing the combination. > > I have tested cursorMark and timeAllowed combination together, and even when > partial results are returned because the timeAllowed is exceeded, the > cursorMark response value is still valid and reasonable. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14413) allow timeAllowed and cursorMark parameters
[ https://issues.apache.org/jira/browse/SOLR-14413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17261546#comment-17261546 ] Mike Drob commented on SOLR-14413: -- There's something strange going on with the logging, I'll revert it for now in the PR but it needs to be investigated further. We currently log a stack trace from SolrIndexSearcher (L#212), but if I suppress that then we log the same message in SearchHandler (L#380). But I don't see the second one if the first is around. Something odd going on with the timeouts. I also don't think the full stack trace is interesting in most contexts - if your users setting a time allowed on a query then hitting that time allowed is more likely an expected occurrence and not necessary to dump a full trace for. Agree on the surprising nature if you switch to debug and the warn disappears. Will think about the appropriate pattern for discovering more information. > allow timeAllowed and cursorMark parameters > --- > > Key: SOLR-14413 > URL: https://issues.apache.org/jira/browse/SOLR-14413 > Project: Solr > Issue Type: Improvement > Components: search >Reporter: John Gallagher >Assignee: Mike Drob >Priority: Minor > Fix For: 8.8, master (9.0) > > Attachments: SOLR-14413-bram.patch, SOLR-14413-jg-update1.patch, > SOLR-14413-jg-update2.patch, SOLR-14413-jg-update3.patch, SOLR-14413.patch, > SOLR-14413.testfix.patch, Screen Shot 2020-10-23 at 10.08.26 PM.png, Screen > Shot 2020-10-23 at 10.09.11 PM.png, image-2020-08-18-16-56-41-736.png, > image-2020-08-18-16-56-59-178.png, image-2020-08-21-14-18-36-229.png, > timeallowed_cursormarks_results.txt > > Time Spent: 2h 10m > Remaining Estimate: 0h > > Ever since cursorMarks were introduced in SOLR-5463 in 2014, cursorMark and > timeAllowed parameters were not allowed in combination ("Can not search using > both cursorMark and timeAllowed") > , from [QueryComponent.java|#L359]]: > > {code:java} > > if (null != rb.getCursorMark() && 0 < timeAllowed) { > // fundamentally incompatible > throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Can not > search using both " + CursorMarkParams.CURSOR_MARK_PARAM + " and " + > CommonParams.TIME_ALLOWED); > } {code} > While theoretically impure to use them in combination, it is often desirable > to support cursormarks-style deep paging and attempt to protect Solr nodes > from runaway queries using timeAllowed, in the hopes that most of the time, > the query completes in the allotted time, and there is no conflict. > > However if the query takes too long, it may be preferable to end the query > and protect the Solr node and provide the user with a somewhat inaccurate > sorted list. As noted in SOLR-6930, SOLR-5986 and others, timeAllowed is > frequently used to prevent runaway load. In fact, cursorMark and > shards.tolerant are allowed in combination, so any argument in favor of > purity would be a bit muddied in my opinion. > > This was discussed once in the mailing list that I can find: > [https://mail-archives.apache.org/mod_mbox/lucene-solr-user/201506.mbox/%3c5591740b.4080...@elyograg.org%3E] > It did not look like there was strong support for preventing the combination. > > I have tested cursorMark and timeAllowed combination together, and even when > partial results are returned because the timeAllowed is exceeded, the > cursorMark response value is still valid and reasonable. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14413) allow timeAllowed and cursorMark parameters
[ https://issues.apache.org/jira/browse/SOLR-14413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17261518#comment-17261518 ] Chris M. Hostetter commented on SOLR-14413: --- Hrmmm... I don't think the loggingchange is a good idea ... it does not seem sane that enabling DEBUG logging in a production solr install will now _prevent_ WARN logging ... what if i'm seeing WARNings i want to investigate further so i turn on debugging? If your concern is how much logging this test produces, then use the \@LogLevel test annotation to supress the logging just in this test. > allow timeAllowed and cursorMark parameters > --- > > Key: SOLR-14413 > URL: https://issues.apache.org/jira/browse/SOLR-14413 > Project: Solr > Issue Type: Improvement > Components: search >Reporter: John Gallagher >Assignee: Mike Drob >Priority: Minor > Fix For: 8.8, master (9.0) > > Attachments: SOLR-14413-bram.patch, SOLR-14413-jg-update1.patch, > SOLR-14413-jg-update2.patch, SOLR-14413-jg-update3.patch, SOLR-14413.patch, > SOLR-14413.testfix.patch, Screen Shot 2020-10-23 at 10.08.26 PM.png, Screen > Shot 2020-10-23 at 10.09.11 PM.png, image-2020-08-18-16-56-41-736.png, > image-2020-08-18-16-56-59-178.png, image-2020-08-21-14-18-36-229.png, > timeallowed_cursormarks_results.txt > > Time Spent: 2h 10m > Remaining Estimate: 0h > > Ever since cursorMarks were introduced in SOLR-5463 in 2014, cursorMark and > timeAllowed parameters were not allowed in combination ("Can not search using > both cursorMark and timeAllowed") > , from [QueryComponent.java|#L359]]: > > {code:java} > > if (null != rb.getCursorMark() && 0 < timeAllowed) { > // fundamentally incompatible > throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Can not > search using both " + CursorMarkParams.CURSOR_MARK_PARAM + " and " + > CommonParams.TIME_ALLOWED); > } {code} > While theoretically impure to use them in combination, it is often desirable > to support cursormarks-style deep paging and attempt to protect Solr nodes > from runaway queries using timeAllowed, in the hopes that most of the time, > the query completes in the allotted time, and there is no conflict. > > However if the query takes too long, it may be preferable to end the query > and protect the Solr node and provide the user with a somewhat inaccurate > sorted list. As noted in SOLR-6930, SOLR-5986 and others, timeAllowed is > frequently used to prevent runaway load. In fact, cursorMark and > shards.tolerant are allowed in combination, so any argument in favor of > purity would be a bit muddied in my opinion. > > This was discussed once in the mailing list that I can find: > [https://mail-archives.apache.org/mod_mbox/lucene-solr-user/201506.mbox/%3c5591740b.4080...@elyograg.org%3E] > It did not look like there was strong support for preventing the combination. > > I have tested cursorMark and timeAllowed combination together, and even when > partial results are returned because the timeAllowed is exceeded, the > cursorMark response value is still valid and reasonable. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14413) allow timeAllowed and cursorMark parameters
[ https://issues.apache.org/jira/browse/SOLR-14413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17261513#comment-17261513 ] Mike Drob commented on SOLR-14413: -- I put up a PR with your pointers, [~hossman]. Also had to tune down the verbosity of the logging that this uncovered. [~slackhappy] - I wasn't able to tag you as a reviewer, but would appreciate you taking a look as well. > allow timeAllowed and cursorMark parameters > --- > > Key: SOLR-14413 > URL: https://issues.apache.org/jira/browse/SOLR-14413 > Project: Solr > Issue Type: Improvement > Components: search >Reporter: John Gallagher >Assignee: Mike Drob >Priority: Minor > Fix For: 8.8, master (9.0) > > Attachments: SOLR-14413-bram.patch, SOLR-14413-jg-update1.patch, > SOLR-14413-jg-update2.patch, SOLR-14413-jg-update3.patch, SOLR-14413.patch, > SOLR-14413.testfix.patch, Screen Shot 2020-10-23 at 10.08.26 PM.png, Screen > Shot 2020-10-23 at 10.09.11 PM.png, image-2020-08-18-16-56-41-736.png, > image-2020-08-18-16-56-59-178.png, image-2020-08-21-14-18-36-229.png, > timeallowed_cursormarks_results.txt > > Time Spent: 2h 10m > Remaining Estimate: 0h > > Ever since cursorMarks were introduced in SOLR-5463 in 2014, cursorMark and > timeAllowed parameters were not allowed in combination ("Can not search using > both cursorMark and timeAllowed") > , from [QueryComponent.java|#L359]]: > > {code:java} > > if (null != rb.getCursorMark() && 0 < timeAllowed) { > // fundamentally incompatible > throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Can not > search using both " + CursorMarkParams.CURSOR_MARK_PARAM + " and " + > CommonParams.TIME_ALLOWED); > } {code} > While theoretically impure to use them in combination, it is often desirable > to support cursormarks-style deep paging and attempt to protect Solr nodes > from runaway queries using timeAllowed, in the hopes that most of the time, > the query completes in the allotted time, and there is no conflict. > > However if the query takes too long, it may be preferable to end the query > and protect the Solr node and provide the user with a somewhat inaccurate > sorted list. As noted in SOLR-6930, SOLR-5986 and others, timeAllowed is > frequently used to prevent runaway load. In fact, cursorMark and > shards.tolerant are allowed in combination, so any argument in favor of > purity would be a bit muddied in my opinion. > > This was discussed once in the mailing list that I can find: > [https://mail-archives.apache.org/mod_mbox/lucene-solr-user/201506.mbox/%3c5591740b.4080...@elyograg.org%3E] > It did not look like there was strong support for preventing the combination. > > I have tested cursorMark and timeAllowed combination together, and even when > partial results are returned because the timeAllowed is exceeded, the > cursorMark response value is still valid and reasonable. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14413) allow timeAllowed and cursorMark parameters
[ https://issues.apache.org/jira/browse/SOLR-14413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17261473#comment-17261473 ] Chris M. Hostetter commented on SOLR-14413: --- {quote}I don't believe we need to switch the handler name, because it should not affect other tests unless they pass the sleep parameter explicitly. {quote} My reason for suggesting that you change the name and refer to it explicitly in the test was so the handler name would intrinsicly be self documenting and the test would fail if anyone ever removed it. imagine someone comes along in N years and refactors this test or changes it's config in some way that results in effectively the same problem as we have on master: {{/select}} doesn't have the delay component anymore, ergo the test starts to _sometimes_ fail. If however you explicitly use a new name like {{/delayingHandleToTestTimeAllowed}} then it will be less likely that someone removes that declaration or modifies it (because it says right in the name why it's there) and if anyone _does_ inadvertently remove it (hypotheitcally: someone refactors the test to be cloud based and uses the {{_default}} config set w/o realising it has a custom config) then you will get a *HARD* failure that will draw their attention to the mistake) Writing good tests isn't just about testing for the change you are making today, it's about protecting your tests assumptions against the changes that might be made tomorrow. > allow timeAllowed and cursorMark parameters > --- > > Key: SOLR-14413 > URL: https://issues.apache.org/jira/browse/SOLR-14413 > Project: Solr > Issue Type: Improvement > Components: search >Reporter: John Gallagher >Assignee: Mike Drob >Priority: Minor > Fix For: 8.8, master (9.0) > > Attachments: SOLR-14413-bram.patch, SOLR-14413-jg-update1.patch, > SOLR-14413-jg-update2.patch, SOLR-14413-jg-update3.patch, SOLR-14413.patch, > SOLR-14413.testfix.patch, Screen Shot 2020-10-23 at 10.08.26 PM.png, Screen > Shot 2020-10-23 at 10.09.11 PM.png, image-2020-08-18-16-56-41-736.png, > image-2020-08-18-16-56-59-178.png, image-2020-08-21-14-18-36-229.png, > timeallowed_cursormarks_results.txt > > Time Spent: 2h > Remaining Estimate: 0h > > Ever since cursorMarks were introduced in SOLR-5463 in 2014, cursorMark and > timeAllowed parameters were not allowed in combination ("Can not search using > both cursorMark and timeAllowed") > , from [QueryComponent.java|#L359]]: > > {code:java} > > if (null != rb.getCursorMark() && 0 < timeAllowed) { > // fundamentally incompatible > throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Can not > search using both " + CursorMarkParams.CURSOR_MARK_PARAM + " and " + > CommonParams.TIME_ALLOWED); > } {code} > While theoretically impure to use them in combination, it is often desirable > to support cursormarks-style deep paging and attempt to protect Solr nodes > from runaway queries using timeAllowed, in the hopes that most of the time, > the query completes in the allotted time, and there is no conflict. > > However if the query takes too long, it may be preferable to end the query > and protect the Solr node and provide the user with a somewhat inaccurate > sorted list. As noted in SOLR-6930, SOLR-5986 and others, timeAllowed is > frequently used to prevent runaway load. In fact, cursorMark and > shards.tolerant are allowed in combination, so any argument in favor of > purity would be a bit muddied in my opinion. > > This was discussed once in the mailing list that I can find: > [https://mail-archives.apache.org/mod_mbox/lucene-solr-user/201506.mbox/%3c5591740b.4080...@elyograg.org%3E] > It did not look like there was strong support for preventing the combination. > > I have tested cursorMark and timeAllowed combination together, and even when > partial results are returned because the timeAllowed is exceeded, the > cursorMark response value is still valid and reasonable. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14413) allow timeAllowed and cursorMark parameters
[ https://issues.apache.org/jira/browse/SOLR-14413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17261464#comment-17261464 ] John Gallagher commented on SOLR-14413: --- Thanks for reporting [~hossman]. I didn't know about the "last definition wins" semantics. I removed the last line of that configuration: [^SOLR-14413.testfix.patch] I don't believe we need to switch the handler name, because it should not affect other tests unless they pass the sleep parameter explicitly. I reran the test using the parameters in your log, and it passed. {code:java} ./gradlew test --tests CursorPagingTest.testTimeAllowed -Dtests.seed=4D63CDC87865C2E0 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=en-MO -Dtests.timezone=Africa/Asmara -Dtests.asserts=true -Dtests.file.encoding=UTF-8 > Task :solr:core:test :solr:core:test (SUCCESS): 1 test(s) The slowest tests (exceeding 500 ms) during this run: 2.12s CursorPagingTest.testTimeAllowed (:solr:core) BUILD SUCCESSFUL in 4m 13s 263 actionable tasks: 263 executed {code} The test failure was this line: [https://github.com/apache/lucene-solr/blob/a7391fb73ef9169c58bc20291fdcefcbd47fa0a8/solr/core/src/test/org/apache/solr/CursorPagingTest.java#L571] {{assertTrue("Should have experienced at least one partialResult", partialCount > 0);}} I had added DelayingSearchComponent to make partial results _more_ likely (well, 100% likely, I was hoping), not less. I guess my own machine was too slow to complete an initial query without partial results. It didn't need the aid of the artificially injected delay. Apologies for the flakiness you experienced John > allow timeAllowed and cursorMark parameters > --- > > Key: SOLR-14413 > URL: https://issues.apache.org/jira/browse/SOLR-14413 > Project: Solr > Issue Type: Improvement > Components: search >Reporter: John Gallagher >Assignee: Mike Drob >Priority: Minor > Fix For: 8.8, master (9.0) > > Attachments: SOLR-14413-bram.patch, SOLR-14413-jg-update1.patch, > SOLR-14413-jg-update2.patch, SOLR-14413-jg-update3.patch, SOLR-14413.patch, > SOLR-14413.testfix.patch, Screen Shot 2020-10-23 at 10.08.26 PM.png, Screen > Shot 2020-10-23 at 10.09.11 PM.png, image-2020-08-18-16-56-41-736.png, > image-2020-08-18-16-56-59-178.png, image-2020-08-21-14-18-36-229.png, > timeallowed_cursormarks_results.txt > > Time Spent: 2h > Remaining Estimate: 0h > > Ever since cursorMarks were introduced in SOLR-5463 in 2014, cursorMark and > timeAllowed parameters were not allowed in combination ("Can not search using > both cursorMark and timeAllowed") > , from [QueryComponent.java|#L359]]: > > {code:java} > > if (null != rb.getCursorMark() && 0 < timeAllowed) { > // fundamentally incompatible > throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Can not > search using both " + CursorMarkParams.CURSOR_MARK_PARAM + " and " + > CommonParams.TIME_ALLOWED); > } {code} > While theoretically impure to use them in combination, it is often desirable > to support cursormarks-style deep paging and attempt to protect Solr nodes > from runaway queries using timeAllowed, in the hopes that most of the time, > the query completes in the allotted time, and there is no conflict. > > However if the query takes too long, it may be preferable to end the query > and protect the Solr node and provide the user with a somewhat inaccurate > sorted list. As noted in SOLR-6930, SOLR-5986 and others, timeAllowed is > frequently used to prevent runaway load. In fact, cursorMark and > shards.tolerant are allowed in combination, so any argument in favor of > purity would be a bit muddied in my opinion. > > This was discussed once in the mailing list that I can find: > [https://mail-archives.apache.org/mod_mbox/lucene-solr-user/201506.mbox/%3c5591740b.4080...@elyograg.org%3E] > It did not look like there was strong support for preventing the combination. > > I have tested cursorMark and timeAllowed combination together, and even when > partial results are returned because the timeAllowed is exceeded, the > cursorMark response value is still valid and reasonable. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14413) allow timeAllowed and cursorMark parameters
[ https://issues.apache.org/jira/browse/SOLR-14413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17261393#comment-17261393 ] Mike Drob commented on SOLR-14413: -- Looking. > allow timeAllowed and cursorMark parameters > --- > > Key: SOLR-14413 > URL: https://issues.apache.org/jira/browse/SOLR-14413 > Project: Solr > Issue Type: Improvement > Components: search >Reporter: John Gallagher >Assignee: Mike Drob >Priority: Minor > Fix For: 8.8, master (9.0) > > Attachments: SOLR-14413-bram.patch, SOLR-14413-jg-update1.patch, > SOLR-14413-jg-update2.patch, SOLR-14413-jg-update3.patch, SOLR-14413.patch, > Screen Shot 2020-10-23 at 10.08.26 PM.png, Screen Shot 2020-10-23 at 10.09.11 > PM.png, image-2020-08-18-16-56-41-736.png, image-2020-08-18-16-56-59-178.png, > image-2020-08-21-14-18-36-229.png, timeallowed_cursormarks_results.txt > > Time Spent: 2h > Remaining Estimate: 0h > > Ever since cursorMarks were introduced in SOLR-5463 in 2014, cursorMark and > timeAllowed parameters were not allowed in combination ("Can not search using > both cursorMark and timeAllowed") > , from [QueryComponent.java|#L359]]: > > {code:java} > > if (null != rb.getCursorMark() && 0 < timeAllowed) { > // fundamentally incompatible > throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Can not > search using both " + CursorMarkParams.CURSOR_MARK_PARAM + " and " + > CommonParams.TIME_ALLOWED); > } {code} > While theoretically impure to use them in combination, it is often desirable > to support cursormarks-style deep paging and attempt to protect Solr nodes > from runaway queries using timeAllowed, in the hopes that most of the time, > the query completes in the allotted time, and there is no conflict. > > However if the query takes too long, it may be preferable to end the query > and protect the Solr node and provide the user with a somewhat inaccurate > sorted list. As noted in SOLR-6930, SOLR-5986 and others, timeAllowed is > frequently used to prevent runaway load. In fact, cursorMark and > shards.tolerant are allowed in combination, so any argument in favor of > purity would be a bit muddied in my opinion. > > This was discussed once in the mailing list that I can find: > [https://mail-archives.apache.org/mod_mbox/lucene-solr-user/201506.mbox/%3c5591740b.4080...@elyograg.org%3E] > It did not look like there was strong support for preventing the combination. > > I have tested cursorMark and timeAllowed combination together, and even when > partial results are returned because the timeAllowed is exceeded, the > cursorMark response value is still valid and reasonable. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14413) allow timeAllowed and cursorMark parameters
[ https://issues.apache.org/jira/browse/SOLR-14413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17254199#comment-17254199 ] ASF subversion and git services commented on SOLR-14413: Commit b13011e97aeddf7c8af1e0bd851c167665187f36 in lucene-solr's branch refs/heads/branch_8x from John Gallagher [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=b13011e ] SOLR-14413 allow timeAllowed and cursorMark parameters closes #1436 > allow timeAllowed and cursorMark parameters > --- > > Key: SOLR-14413 > URL: https://issues.apache.org/jira/browse/SOLR-14413 > Project: Solr > Issue Type: Improvement > Components: search >Reporter: John Gallagher >Priority: Minor > Attachments: SOLR-14413-bram.patch, SOLR-14413-jg-update1.patch, > SOLR-14413-jg-update2.patch, SOLR-14413-jg-update3.patch, SOLR-14413.patch, > Screen Shot 2020-10-23 at 10.08.26 PM.png, Screen Shot 2020-10-23 at 10.09.11 > PM.png, image-2020-08-18-16-56-41-736.png, image-2020-08-18-16-56-59-178.png, > image-2020-08-21-14-18-36-229.png, timeallowed_cursormarks_results.txt > > Time Spent: 2h > Remaining Estimate: 0h > > Ever since cursorMarks were introduced in SOLR-5463 in 2014, cursorMark and > timeAllowed parameters were not allowed in combination ("Can not search using > both cursorMark and timeAllowed") > , from [QueryComponent.java|#L359]]: > > {code:java} > > if (null != rb.getCursorMark() && 0 < timeAllowed) { > // fundamentally incompatible > throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Can not > search using both " + CursorMarkParams.CURSOR_MARK_PARAM + " and " + > CommonParams.TIME_ALLOWED); > } {code} > While theoretically impure to use them in combination, it is often desirable > to support cursormarks-style deep paging and attempt to protect Solr nodes > from runaway queries using timeAllowed, in the hopes that most of the time, > the query completes in the allotted time, and there is no conflict. > > However if the query takes too long, it may be preferable to end the query > and protect the Solr node and provide the user with a somewhat inaccurate > sorted list. As noted in SOLR-6930, SOLR-5986 and others, timeAllowed is > frequently used to prevent runaway load. In fact, cursorMark and > shards.tolerant are allowed in combination, so any argument in favor of > purity would be a bit muddied in my opinion. > > This was discussed once in the mailing list that I can find: > [https://mail-archives.apache.org/mod_mbox/lucene-solr-user/201506.mbox/%3c5591740b.4080...@elyograg.org%3E] > It did not look like there was strong support for preventing the combination. > > I have tested cursorMark and timeAllowed combination together, and even when > partial results are returned because the timeAllowed is exceeded, the > cursorMark response value is still valid and reasonable. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14413) allow timeAllowed and cursorMark parameters
[ https://issues.apache.org/jira/browse/SOLR-14413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17254200#comment-17254200 ] ASF subversion and git services commented on SOLR-14413: Commit 70f461ee453afd59f971a3ec5c431181aa1edd10 in lucene-solr's branch refs/heads/master from John Gallagher [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=70f461e ] SOLR-14413 allow timeAllowed and cursorMark parameters closes #1436 > allow timeAllowed and cursorMark parameters > --- > > Key: SOLR-14413 > URL: https://issues.apache.org/jira/browse/SOLR-14413 > Project: Solr > Issue Type: Improvement > Components: search >Reporter: John Gallagher >Priority: Minor > Attachments: SOLR-14413-bram.patch, SOLR-14413-jg-update1.patch, > SOLR-14413-jg-update2.patch, SOLR-14413-jg-update3.patch, SOLR-14413.patch, > Screen Shot 2020-10-23 at 10.08.26 PM.png, Screen Shot 2020-10-23 at 10.09.11 > PM.png, image-2020-08-18-16-56-41-736.png, image-2020-08-18-16-56-59-178.png, > image-2020-08-21-14-18-36-229.png, timeallowed_cursormarks_results.txt > > Time Spent: 2h > Remaining Estimate: 0h > > Ever since cursorMarks were introduced in SOLR-5463 in 2014, cursorMark and > timeAllowed parameters were not allowed in combination ("Can not search using > both cursorMark and timeAllowed") > , from [QueryComponent.java|#L359]]: > > {code:java} > > if (null != rb.getCursorMark() && 0 < timeAllowed) { > // fundamentally incompatible > throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Can not > search using both " + CursorMarkParams.CURSOR_MARK_PARAM + " and " + > CommonParams.TIME_ALLOWED); > } {code} > While theoretically impure to use them in combination, it is often desirable > to support cursormarks-style deep paging and attempt to protect Solr nodes > from runaway queries using timeAllowed, in the hopes that most of the time, > the query completes in the allotted time, and there is no conflict. > > However if the query takes too long, it may be preferable to end the query > and protect the Solr node and provide the user with a somewhat inaccurate > sorted list. As noted in SOLR-6930, SOLR-5986 and others, timeAllowed is > frequently used to prevent runaway load. In fact, cursorMark and > shards.tolerant are allowed in combination, so any argument in favor of > purity would be a bit muddied in my opinion. > > This was discussed once in the mailing list that I can find: > [https://mail-archives.apache.org/mod_mbox/lucene-solr-user/201506.mbox/%3c5591740b.4080...@elyograg.org%3E] > It did not look like there was strong support for preventing the combination. > > I have tested cursorMark and timeAllowed combination together, and even when > partial results are returned because the timeAllowed is exceeded, the > cursorMark response value is still valid and reasonable. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14413) allow timeAllowed and cursorMark parameters
[ https://issues.apache.org/jira/browse/SOLR-14413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17237423#comment-17237423 ] Bram Van Dam commented on SOLR-14413: - [~slackhappy] [~mdrob] Sounds good to me. When partialResults is set, you know it's possible that some items are missing from the result set. If that's not acceptable, you should retry with a larger timeout, or inform the user that their query is unacceptable. This is definitely a tradeoff I can live with. Thanks for the effort! I hope someone will be kind enough to merge this <3 > allow timeAllowed and cursorMark parameters > --- > > Key: SOLR-14413 > URL: https://issues.apache.org/jira/browse/SOLR-14413 > Project: Solr > Issue Type: Improvement > Components: search >Reporter: John Gallagher >Priority: Minor > Attachments: SOLR-14413-bram.patch, SOLR-14413-jg-update1.patch, > SOLR-14413-jg-update2.patch, SOLR-14413-jg-update3.patch, SOLR-14413.patch, > Screen Shot 2020-10-23 at 10.08.26 PM.png, Screen Shot 2020-10-23 at 10.09.11 > PM.png, image-2020-08-18-16-56-41-736.png, image-2020-08-18-16-56-59-178.png, > image-2020-08-21-14-18-36-229.png, timeallowed_cursormarks_results.txt > > Time Spent: 1h 50m > Remaining Estimate: 0h > > Ever since cursorMarks were introduced in SOLR-5463 in 2014, cursorMark and > timeAllowed parameters were not allowed in combination ("Can not search using > both cursorMark and timeAllowed") > , from [QueryComponent.java|#L359]]: > > {code:java} > > if (null != rb.getCursorMark() && 0 < timeAllowed) { > // fundamentally incompatible > throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Can not > search using both " + CursorMarkParams.CURSOR_MARK_PARAM + " and " + > CommonParams.TIME_ALLOWED); > } {code} > While theoretically impure to use them in combination, it is often desirable > to support cursormarks-style deep paging and attempt to protect Solr nodes > from runaway queries using timeAllowed, in the hopes that most of the time, > the query completes in the allotted time, and there is no conflict. > > However if the query takes too long, it may be preferable to end the query > and protect the Solr node and provide the user with a somewhat inaccurate > sorted list. As noted in SOLR-6930, SOLR-5986 and others, timeAllowed is > frequently used to prevent runaway load. In fact, cursorMark and > shards.tolerant are allowed in combination, so any argument in favor of > purity would be a bit muddied in my opinion. > > This was discussed once in the mailing list that I can find: > [https://mail-archives.apache.org/mod_mbox/lucene-solr-user/201506.mbox/%3c5591740b.4080...@elyograg.org%3E] > It did not look like there was strong support for preventing the combination. > > I have tested cursorMark and timeAllowed combination together, and even when > partial results are returned because the timeAllowed is exceeded, the > cursorMark response value is still valid and reasonable. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14413) allow timeAllowed and cursorMark parameters
[ https://issues.apache.org/jira/browse/SOLR-14413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17224785#comment-17224785 ] John Gallagher commented on SOLR-14413: --- [~mdrob], [~bvd], any thoughts on the updated test and documentation? > allow timeAllowed and cursorMark parameters > --- > > Key: SOLR-14413 > URL: https://issues.apache.org/jira/browse/SOLR-14413 > Project: Solr > Issue Type: Improvement > Components: search >Reporter: John Gallagher >Priority: Minor > Attachments: SOLR-14413-bram.patch, SOLR-14413-jg-update1.patch, > SOLR-14413-jg-update2.patch, SOLR-14413-jg-update3.patch, SOLR-14413.patch, > Screen Shot 2020-10-23 at 10.08.26 PM.png, Screen Shot 2020-10-23 at 10.09.11 > PM.png, image-2020-08-18-16-56-41-736.png, image-2020-08-18-16-56-59-178.png, > image-2020-08-21-14-18-36-229.png, timeallowed_cursormarks_results.txt > > Time Spent: 1h 50m > Remaining Estimate: 0h > > Ever since cursorMarks were introduced in SOLR-5463 in 2014, cursorMark and > timeAllowed parameters were not allowed in combination ("Can not search using > both cursorMark and timeAllowed") > , from [QueryComponent.java|#L359]]: > > {code:java} > > if (null != rb.getCursorMark() && 0 < timeAllowed) { > // fundamentally incompatible > throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Can not > search using both " + CursorMarkParams.CURSOR_MARK_PARAM + " and " + > CommonParams.TIME_ALLOWED); > } {code} > While theoretically impure to use them in combination, it is often desirable > to support cursormarks-style deep paging and attempt to protect Solr nodes > from runaway queries using timeAllowed, in the hopes that most of the time, > the query completes in the allotted time, and there is no conflict. > > However if the query takes too long, it may be preferable to end the query > and protect the Solr node and provide the user with a somewhat inaccurate > sorted list. As noted in SOLR-6930, SOLR-5986 and others, timeAllowed is > frequently used to prevent runaway load. In fact, cursorMark and > shards.tolerant are allowed in combination, so any argument in favor of > purity would be a bit muddied in my opinion. > > This was discussed once in the mailing list that I can find: > [https://mail-archives.apache.org/mod_mbox/lucene-solr-user/201506.mbox/%3c5591740b.4080...@elyograg.org%3E] > It did not look like there was strong support for preventing the combination. > > I have tested cursorMark and timeAllowed combination together, and even when > partial results are returned because the timeAllowed is exceeded, the > cursorMark response value is still valid and reasonable. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14413) allow timeAllowed and cursorMark parameters
[ https://issues.apache.org/jira/browse/SOLR-14413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17224345#comment-17224345 ] Yevhen Tienkaiev commented on SOLR-14413: - Sounds good to me, since *partialResults* will show when I get partial results or not. Thank you! > allow timeAllowed and cursorMark parameters > --- > > Key: SOLR-14413 > URL: https://issues.apache.org/jira/browse/SOLR-14413 > Project: Solr > Issue Type: Improvement > Components: search >Reporter: John Gallagher >Priority: Minor > Attachments: SOLR-14413-bram.patch, SOLR-14413-jg-update1.patch, > SOLR-14413-jg-update2.patch, SOLR-14413-jg-update3.patch, SOLR-14413.patch, > Screen Shot 2020-10-23 at 10.08.26 PM.png, Screen Shot 2020-10-23 at 10.09.11 > PM.png, image-2020-08-18-16-56-41-736.png, image-2020-08-18-16-56-59-178.png, > image-2020-08-21-14-18-36-229.png, timeallowed_cursormarks_results.txt > > Time Spent: 1h 50m > Remaining Estimate: 0h > > Ever since cursorMarks were introduced in SOLR-5463 in 2014, cursorMark and > timeAllowed parameters were not allowed in combination ("Can not search using > both cursorMark and timeAllowed") > , from [QueryComponent.java|#L359]]: > > {code:java} > > if (null != rb.getCursorMark() && 0 < timeAllowed) { > // fundamentally incompatible > throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Can not > search using both " + CursorMarkParams.CURSOR_MARK_PARAM + " and " + > CommonParams.TIME_ALLOWED); > } {code} > While theoretically impure to use them in combination, it is often desirable > to support cursormarks-style deep paging and attempt to protect Solr nodes > from runaway queries using timeAllowed, in the hopes that most of the time, > the query completes in the allotted time, and there is no conflict. > > However if the query takes too long, it may be preferable to end the query > and protect the Solr node and provide the user with a somewhat inaccurate > sorted list. As noted in SOLR-6930, SOLR-5986 and others, timeAllowed is > frequently used to prevent runaway load. In fact, cursorMark and > shards.tolerant are allowed in combination, so any argument in favor of > purity would be a bit muddied in my opinion. > > This was discussed once in the mailing list that I can find: > [https://mail-archives.apache.org/mod_mbox/lucene-solr-user/201506.mbox/%3c5591740b.4080...@elyograg.org%3E] > It did not look like there was strong support for preventing the combination. > > I have tested cursorMark and timeAllowed combination together, and even when > partial results are returned because the timeAllowed is exceeded, the > cursorMark response value is still valid and reasonable. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14413) allow timeAllowed and cursorMark parameters
[ https://issues.apache.org/jira/browse/SOLR-14413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17220001#comment-17220001 ] John Gallagher commented on SOLR-14413: --- [~bvd] the issue was with an assumption that the test was making. It is not always true that every document will be found when using timeAllowed and cursorMark in combination. There may be holes in the result sets, but at least the ordering between and within result sets will be correct with respect to the sort. This is what I had suspected was the case when I proposed allowing the combination, but I didn't have an example at the time. I think it is still a good idea to allow these parameters in combination (its something that you could encounter when using shards.tolerant and cursorMark in combination, and that combination is allowed). When using timeAllowed and cursorMark in combination, and there are multiple segments in the index, it is possible that a query may terminate before visiting the matching documents in every segment. The hint for this is in the warning message's stack trace associated with the failing seed you found in the previous revision: [https://gist.github.com/slackhappy/1a48d56e10679404cea3441f87a0fecc#file-gistfile1-txt-L6] . "The request took too long to iterate over terms." occurs while in a specific segment, which prevents iterating on to the next segment. I have updated my pull request: [https://github.com/apache/lucene-solr/pull/1436] And I have updated my proposed documentation changes to include a mention that results may be missing if partialResults is true: !Screen Shot 2020-10-23 at 10.08.26 PM.png|width=545,height=114! !Screen Shot 2020-10-23 at 10.09.11 PM.png|width=577,height=161! > allow timeAllowed and cursorMark parameters > --- > > Key: SOLR-14413 > URL: https://issues.apache.org/jira/browse/SOLR-14413 > Project: Solr > Issue Type: Improvement > Components: search >Reporter: John Gallagher >Priority: Minor > Attachments: SOLR-14413-bram.patch, SOLR-14413-jg-update1.patch, > SOLR-14413-jg-update2.patch, SOLR-14413.patch, Screen Shot 2020-10-23 at > 10.08.26 PM.png, Screen Shot 2020-10-23 at 10.09.11 PM.png, > image-2020-08-18-16-56-41-736.png, image-2020-08-18-16-56-59-178.png, > image-2020-08-21-14-18-36-229.png, timeallowed_cursormarks_results.txt > > Time Spent: 1h 50m > Remaining Estimate: 0h > > Ever since cursorMarks were introduced in SOLR-5463 in 2014, cursorMark and > timeAllowed parameters were not allowed in combination ("Can not search using > both cursorMark and timeAllowed") > , from [QueryComponent.java|#L359]]: > > {code:java} > > if (null != rb.getCursorMark() && 0 < timeAllowed) { > // fundamentally incompatible > throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Can not > search using both " + CursorMarkParams.CURSOR_MARK_PARAM + " and " + > CommonParams.TIME_ALLOWED); > } {code} > While theoretically impure to use them in combination, it is often desirable > to support cursormarks-style deep paging and attempt to protect Solr nodes > from runaway queries using timeAllowed, in the hopes that most of the time, > the query completes in the allotted time, and there is no conflict. > > However if the query takes too long, it may be preferable to end the query > and protect the Solr node and provide the user with a somewhat inaccurate > sorted list. As noted in SOLR-6930, SOLR-5986 and others, timeAllowed is > frequently used to prevent runaway load. In fact, cursorMark and > shards.tolerant are allowed in combination, so any argument in favor of > purity would be a bit muddied in my opinion. > > This was discussed once in the mailing list that I can find: > [https://mail-archives.apache.org/mod_mbox/lucene-solr-user/201506.mbox/%3c5591740b.4080...@elyograg.org%3E] > It did not look like there was strong support for preventing the combination. > > I have tested cursorMark and timeAllowed combination together, and even when > partial results are returned because the timeAllowed is exceeded, the > cursorMark response value is still valid and reasonable. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14413) allow timeAllowed and cursorMark parameters
[ https://issues.apache.org/jira/browse/SOLR-14413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17219927#comment-17219927 ] Yevhen Tienkaiev commented on SOLR-14413: - Hello, is there any status on this? This is pretty critical, please someone can push this forward? > allow timeAllowed and cursorMark parameters > --- > > Key: SOLR-14413 > URL: https://issues.apache.org/jira/browse/SOLR-14413 > Project: Solr > Issue Type: Improvement > Components: search >Reporter: John Gallagher >Priority: Minor > Attachments: SOLR-14413-bram.patch, SOLR-14413-jg-update1.patch, > SOLR-14413-jg-update2.patch, SOLR-14413.patch, > image-2020-08-18-16-56-41-736.png, image-2020-08-18-16-56-59-178.png, > image-2020-08-21-14-18-36-229.png, timeallowed_cursormarks_results.txt > > Time Spent: 1h 50m > Remaining Estimate: 0h > > Ever since cursorMarks were introduced in SOLR-5463 in 2014, cursorMark and > timeAllowed parameters were not allowed in combination ("Can not search using > both cursorMark and timeAllowed") > , from [QueryComponent.java|#L359]]: > > {code:java} > > if (null != rb.getCursorMark() && 0 < timeAllowed) { > // fundamentally incompatible > throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Can not > search using both " + CursorMarkParams.CURSOR_MARK_PARAM + " and " + > CommonParams.TIME_ALLOWED); > } {code} > While theoretically impure to use them in combination, it is often desirable > to support cursormarks-style deep paging and attempt to protect Solr nodes > from runaway queries using timeAllowed, in the hopes that most of the time, > the query completes in the allotted time, and there is no conflict. > > However if the query takes too long, it may be preferable to end the query > and protect the Solr node and provide the user with a somewhat inaccurate > sorted list. As noted in SOLR-6930, SOLR-5986 and others, timeAllowed is > frequently used to prevent runaway load. In fact, cursorMark and > shards.tolerant are allowed in combination, so any argument in favor of > purity would be a bit muddied in my opinion. > > This was discussed once in the mailing list that I can find: > [https://mail-archives.apache.org/mod_mbox/lucene-solr-user/201506.mbox/%3c5591740b.4080...@elyograg.org%3E] > It did not look like there was strong support for preventing the combination. > > I have tested cursorMark and timeAllowed combination together, and even when > partial results are returned because the timeAllowed is exceeded, the > cursorMark response value is still valid and reasonable. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14413) allow timeAllowed and cursorMark parameters
[ https://issues.apache.org/jira/browse/SOLR-14413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17185188#comment-17185188 ] Bram Van Dam commented on SOLR-14413: - [~slackhappy] Thanks for adding that test. On master, it seems to fail roughly 1 times out of 5. I'm pasting sample output below. But basically anywhere between 10 and 300 results are missing. I'm not sure whether this is a bug in the test or in the cursor implementation. Any thoughts? {noformat} java.lang.AssertionError: Should have found all documents eventually expected:<[0, 1, 10, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 11, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 12, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 13, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 14, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 15, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 16, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 17, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 18, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 19, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 2, 20, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 21, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 22, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 23, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 24, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 25, 250, 251, 252, 253, 254, 255, 256, 257, 258, 259, 26, 260, 261, 262, 263, 264, 265, 266, 267, 268, 269, 27, 270, 271, 272, 273, 274, 275, 276, 277, 278, 279, 28, 280, 281, 282, 283, 284, 285, 286, 287, 288, 289, 29, 290, 291, 292, 293, 294, 295, 296, 297, 298, 299, 3, 30, 300, 301, 302, 303, 304, 305, 306, 307, 308, 309, 31, 310, 311, 312, 313, 314, 315, 316, 317, 318, 319, 32, 320, 321, 322, 323, 324, 325, 326, 327, 328, 329, 33, 330, 331, 332, 333, 334, 335, 336, 337, 338, 339, 34, 340, 341, 342, 343, 344, 345, 346, 347, 348, 349, 35, 350, 351, 352, 353, 354, 355, 356, 357, 358, 359, 36, 360, 361, 362, 363, 364, 365, 366, 367, 368, 369, 37, 370, 371, 372, 373, 374, 375, 376, 377, 378, 379, 38, 380, 381, 382, 383, 384, 385, 386, 387, 388, 389, 39, 390, 391, 392, 393, 394, 395, 396, 397, 398, 399, 4, 40, 400, 401, 402, 403, 404, 405, 406, 407, 408, 409, 41, 410, 411, 412, 413, 414, 415, 416, 417, 418, 419, 42, 420, 421, 422, 423, 424, 425, 426, 427, 428, 429, 43, 430, 431, 432, 433, 434, 435, 436, 437, 438, 439, 44, 440, 441, 442, 443, 444, 445, 446, 447, 448, 449, 45, 450, 451, 452, 453, 454, 455, 456, 457, 458, 459, 46, 460, 461, 462, 463, 464, 465, 466, 467, 468, 469, 47, 470, 471, 472, 473, 474, 475, 476, 477, 478, 479, 48, 480, 481, 482, 483, 484, 485, 486, 487, 488, 489, 49, 490, 491, 492, 493, 494, 495, 496, 497, 498, 499, 5, 50, 500, 501, 502, 503, 504, 505, 506, 507, 508, 509, 51, 510, 511, 512, 513, 514, 515, 516, 517, 518, 519, 52, 520, 521, 522, 523, 524, 525, 526, 527, 528, 529, 53, 530, 531, 532, 533, 534, 535, 536, 537, 538, 539, 54, 540, 541, 542, 543, 544, 545, 546, 547, 548, 549, 55, 550, 551, 552, 553, 554, 555, 556, 557, 558, 559, 56, 560, 561, 562, 563, 564, 565, 566, 567, 568, 569, 57, 570, 571, 572, 573, 574, 575, 576, 577, 578, 579, 58, 580, 581, 582, 583, 584, 585, 586, 587, 588, 589, 59, 590, 591, 592, 593, 594, 595, 596, 597, 598, 599, 6, 60, 600, 601, 602, 603, 604, 605, 606, 607, 608, 609, 61, 610, 611, 612, 613, 614, 615, 616, 617, 618, 619, 62, 620, 621, 622, 623, 624, 625, 626, 627, 628, 629, 63, 630, 631, 632, 633, 634, 635, 636, 637, 638, 639, 64, 640, 641, 642, 643, 644, 645, 646, 647, 648, 649, 65, 650, 651, 652, 653, 654, 655, 656, 657, 658, 659, 66, 660, 661, 662, 663, 664, 665, 666, 667, 668, 669, 67, 670, 671, 672, 673, 674, 675, 676, 677, 678, 679, 68, 680, 681, 682, 683, 684, 685, 686, 687, 688, 689, 69, 690, 691, 692, 693, 694, 695, 696, 697, 698, 699, 7, 70, 700, 701, 702, 703, 704, 705, 706, 707, 708, 709, 71, 710, 711, 712, 713, 714, 715, 716, 717, 718, 719, 72, 720, 721, 722, 723, 724, 725, 726, 727, 728, 729, 73, 730, 731, 732, 733, 734, 735, 736, 737, 738, 739, 74, 740, 741, 742, 743, 744, 745, 746, 747, 748, 749, 75, 750, 751, 752, 753, 754, 755, 756, 757, 758, 759, 76, 760, 761, 762, 763, 764, 765, 766, 767, 768, 769, 77, 770, 771, 772, 773, 774, 775, 776, 777, 778, 779, 78, 780, 781, 782, 783, 784, 785, 786, 787, 788, 789, 79, 790, 791, 792, 793, 794, 795, 796, 797, 798, 799, 8, 80, 800, 801, 802, 803, 804, 805, 806, 807, 808, 809, 81, 810, 811, 812, 813, 814, 815, 816, 817, 818, 819, 82, 820, 821, 822, 823, 824, 825, 826, 827, 828, 829, 83, 830, 831, 832, 833, 834, 835, 836, 837, 838, 839, 84, 840, 841, 842, 843, 844, 845, 846, 847, 848, 849, 85, 850, 851, 852, 853, 854, 855, 856, 857, 858, 859, 86, 860, 861, 862, 863, 864, 865, 866, 867, 868, 869, 87, 870, 871, 872, 873, 874, 875, 876, 877, 878, 879, 88, 880, 881, 882, 883, 884, 885, 886, 887, 888, 889, 89, 890, 891,
[jira] [Commented] (SOLR-14413) allow timeAllowed and cursorMark parameters
[ https://issues.apache.org/jira/browse/SOLR-14413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17182060#comment-17182060 ] John Gallagher commented on SOLR-14413: --- Thanks [~mdrob] for reviewing! I added new documentation for omitHeader, advising against omitting when using parameters that can lead to partial results, because the header flag informs the interpretation of the result values. !image-2020-08-21-14-18-36-229.png|width=558,height=180! I updated timeAllowed documentation as well. I incorporated [~bvd]'s test idea of generating a result set, running it to completion, and confirming all documents were found in the correct order, while also asserting that at least one partialResults event occurred. I have generated a new patch, SOLR-14413-jg-update2.patch, and updated my PR: [https://github.com/apache/lucene-solr/pull/1436] John > allow timeAllowed and cursorMark parameters > --- > > Key: SOLR-14413 > URL: https://issues.apache.org/jira/browse/SOLR-14413 > Project: Solr > Issue Type: Improvement > Components: search >Reporter: John Gallagher >Priority: Minor > Attachments: SOLR-14413-bram.patch, SOLR-14413-jg-update1.patch, > SOLR-14413.patch, image-2020-08-18-16-56-41-736.png, > image-2020-08-18-16-56-59-178.png, image-2020-08-21-14-18-36-229.png, > timeallowed_cursormarks_results.txt > > Time Spent: 1h 50m > Remaining Estimate: 0h > > Ever since cursorMarks were introduced in SOLR-5463 in 2014, cursorMark and > timeAllowed parameters were not allowed in combination ("Can not search using > both cursorMark and timeAllowed") > , from [QueryComponent.java|#L359]]: > > {code:java} > > if (null != rb.getCursorMark() && 0 < timeAllowed) { > // fundamentally incompatible > throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Can not > search using both " + CursorMarkParams.CURSOR_MARK_PARAM + " and " + > CommonParams.TIME_ALLOWED); > } {code} > While theoretically impure to use them in combination, it is often desirable > to support cursormarks-style deep paging and attempt to protect Solr nodes > from runaway queries using timeAllowed, in the hopes that most of the time, > the query completes in the allotted time, and there is no conflict. > > However if the query takes too long, it may be preferable to end the query > and protect the Solr node and provide the user with a somewhat inaccurate > sorted list. As noted in SOLR-6930, SOLR-5986 and others, timeAllowed is > frequently used to prevent runaway load. In fact, cursorMark and > shards.tolerant are allowed in combination, so any argument in favor of > purity would be a bit muddied in my opinion. > > This was discussed once in the mailing list that I can find: > [https://mail-archives.apache.org/mod_mbox/lucene-solr-user/201506.mbox/%3c5591740b.4080...@elyograg.org%3E] > It did not look like there was strong support for preventing the combination. > > I have tested cursorMark and timeAllowed combination together, and even when > partial results are returned because the timeAllowed is exceeded, the > cursorMark response value is still valid and reasonable. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14413) allow timeAllowed and cursorMark parameters
[ https://issues.apache.org/jira/browse/SOLR-14413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180102#comment-17180102 ] John Gallagher commented on SOLR-14413: --- Hi folks, sorry I stepped away from this for a bit, and thanks [~mdrob] , and [~epugh] for the comments, and [~bvd] for contributing! To Mike's earlier point: when partialResults is true, you cannot be sure whether there are more results or not. You could summarize it this way: || ||partialResults not present|| partialResults: true|| |more results?|nextCursorMark != cursorMark|unknown| After figuring out how to properly generate documentation, I took a stab at adding two notes to the documentation: 1. In timeAllowed documentation, common-query-parameters.html#timeallowed-parameter !image-2020-08-18-16-56-59-178.png|width=508,height=116! 2. In the Constraints when using Cursors documentation, pagination-of-results.html#constraints-when-using-cursors !image-2020-08-18-16-56-41-736.png|width=410,height=250! Please let me know what you think of these additions. I was also able to reproduce [~bvd]'s case where nextCursorMark was null (the field actually wasn't present in the response at all). I tracked it down to [this code in SearchHandler.java|[https://github.com/slackhappy/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/handler/component/SearchHandler.java#L349]]. When the Search times out very early, an exception is thrown before responses can be handled, which skips the calculation of the next cursor mark. In that case, the response is null, and the SearchHandler creates an empty response to return (numFound 0, docs:[]). I added code to also return the original cursorMark in that case, since we haven't progressed (it is the value you would want to pass to a subsequent search). I added a basic test to CursorPagingTest that checks that the parameters can be used in conjunction, and that the nextCursorMark returned is a valid one that can be used in subsequent requests. Perhaps that test can be combined with [~bvd]'s. I struggled a bit to come up with a test that would produce reliable results when using timeAllowed. I borrowed a bit from ExitableDirectoryReaderTest's use of a DelayingSearchComponent, which helped somewhat. The updated patch file is SOLR-14413-jg-update1.patch, and I have updated the [corresponding PR|[https://github.com/apache/lucene-solr/pull/1436]]. John > allow timeAllowed and cursorMark parameters > --- > > Key: SOLR-14413 > URL: https://issues.apache.org/jira/browse/SOLR-14413 > Project: Solr > Issue Type: Improvement > Components: search >Reporter: John Gallagher >Priority: Minor > Attachments: SOLR-14413-bram.patch, SOLR-14413-jg-update1.patch, > SOLR-14413.patch, image-2020-08-18-16-56-41-736.png, > image-2020-08-18-16-56-59-178.png, timeallowed_cursormarks_results.txt > > Time Spent: 10m > Remaining Estimate: 0h > > Ever since cursorMarks were introduced in SOLR-5463 in 2014, cursorMark and > timeAllowed parameters were not allowed in combination ("Can not search using > both cursorMark and timeAllowed") > , from [QueryComponent.java|#L359]]: > > {code:java} > > if (null != rb.getCursorMark() && 0 < timeAllowed) { > // fundamentally incompatible > throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Can not > search using both " + CursorMarkParams.CURSOR_MARK_PARAM + " and " + > CommonParams.TIME_ALLOWED); > } {code} > While theoretically impure to use them in combination, it is often desirable > to support cursormarks-style deep paging and attempt to protect Solr nodes > from runaway queries using timeAllowed, in the hopes that most of the time, > the query completes in the allotted time, and there is no conflict. > > However if the query takes too long, it may be preferable to end the query > and protect the Solr node and provide the user with a somewhat inaccurate > sorted list. As noted in SOLR-6930, SOLR-5986 and others, timeAllowed is > frequently used to prevent runaway load. In fact, cursorMark and > shards.tolerant are allowed in combination, so any argument in favor of > purity would be a bit muddied in my opinion. > > This was discussed once in the mailing list that I can find: > [https://mail-archives.apache.org/mod_mbox/lucene-solr-user/201506.mbox/%3c5591740b.4080...@elyograg.org%3E] > It did not look like there was strong support for preventing the combination. > > I have tested cursorMark and timeAllowed combination together, and even when > partial results are returned because the timeAllowed is exceeded, the > cursorMark response value is still valid and reasonable. -- This message was sent by Atlassian Jira (v8.3.4#803005) ---
[jira] [Commented] (SOLR-14413) allow timeAllowed and cursorMark parameters
[ https://issues.apache.org/jira/browse/SOLR-14413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17179702#comment-17179702 ] Bram Van Dam commented on SOLR-14413: - [~epugh] I created a little test case to demonstrate the two parameters working together, but in doing so I appear to have done the opposite. The test fails roughly 2/3 of the time. There appear to be three failure modes: # nextCursorMark is null (when it should be the same as the previous cursor mark, or possibly a new mark) # partialResults header is missing (this only seems to happen when the response does not contain any results) # not all documents are hit even after the cursor has reached its end (this one occurs much less frequently) When the assertions checking failure mode 1 and 2 are disabled, the test only fails 1 time out of 4, when failure more 3 is hit. IMO only the third failure mode is unacceptable, because it produces incorrect results. The test never fails when minimum timeAllowed is set to 50ms or above. Values that low are obviously ridiculous. Perhaps the documentation (or the implementation?) should be updated to discourage unreasonably low values. I'll attach an updated version of [~slackhappy]'s patch which includes my test case. And I'll paste the test case here for good measure. Apologies for the mess, I'm not familiar with Solr's test harness, so I'm sure there cleaner ways of doing this. {code:java} /** * - check whether the cursor can be advanced when using timeAllowed * - ensure partial results are advertised as such * - check the correctness of those results */ @SuppressWarnings({"unchecked", "rawtypes"}) public void testTimeAllowedAdvancesCursor() throws Exception { String cursorMark; ModifiableSolrParams params = null; // Add 1000 docs, anything less and the requests complete too quickly to be interesting for(int i=1;i<=100;i++) { // don't add in order of any field to ensure we aren't inadvertently // counting on internal docid ordering assertU(adoc("id", i+ "9", "str", "c", "float", "-3.2", "int", "42" + i)); assertU(adoc("id", i+ "7", "str", "c", "float", "-3.2", "int", "-1976" + i)); assertU(adoc("id", i+ "2", "str", "c", "float", "-3.2", "int", "666" + i)); assertU(adoc("id", i+ "0", "str", "b", "float", "64.5", "int", "-42" + i)); assertU(adoc("id", i+ "5", "str", "b", "float", "64.5", "int", "2001" + i)); assertU(adoc("id", i+ "8", "str", "b", "float", "64.5", "int", "4055" + i)); assertU(adoc("id", i+ "6", "str", "a", "float", "64.5", "int", "7" + i)); assertU(adoc("id", i+ "1", "str", "a", "float", "64.5", "int", "7" + i)); assertU(adoc("id", i+ "4", "str", "a", "float", "11.1", "int", "6" + i)); assertU(adoc("id", i+ "3", "str", "a", "float", "11.1")); // int is missing } assertU(commit()); // Prepare a list (sorted) of docIds we expect to find, populated without using a cursor or timeAllowed List expectedDocIds = new ArrayList<>(); Map docResponse = (Map) fromJSONString(h.query(req(params("q", "-str:b", "rows", "700", "fl", "id", "sort", "id desc"; expectedDocIds.addAll((Collection) (((Map)docResponse.get("response")).get("docs"))); cursorMark = CURSOR_MARK_START; params = params("q", "-str:b", "rows","40", "fl", "id", "sort", "id desc"); List foundDocIds = new ArrayList<>(); boolean cursorAdvanced = false; do { cursorAdvanced = false; // If the cursor does not advance, we increase timeAllowed until it's long enough to return a result for(int timeAllowed=1; timeAllowed<=100; timeAllowed++) { // Keep timeAllowed between 1 and 100ms String json = assertJQ(req(params, CURSOR_MARK_PARAM, cursorMark, CommonParams.TIME_ALLOWED, "" + timeAllowed)); Map response = (Map) fromJSONString(json); String next = (String)response.get(CURSOR_MARK_NEXT); assertNotNull(CURSOR_MARK_NEXT + " is null", next); if(null != next && !cursorMark.equals(next)) { // Cursor advanced, record foundDocs and move on foundDocIds.addAll((Collection) (((Map)response.get("response")).get("docs"))); cursorMark = next; cursorAdvanced = true; break; } else if(foundDocIds.size() != 700) { // Unless we've found all documents, the result must be partial assertNull("Response is missing partialResults header for: " + "old cursor " + cursorMark + " new cursor: " + next + " timeAllowed " + timeAllowed + " foundDocIds: " + foundDocIds.size(), JSONTestUtil.match(json, "/responseHeader/partialResults==true", JSONTestUtil.DEFAULT_DELTA)); } } } while(cursorAdvanced); assertEquals("Should have found 700 documents eventually", expe
[jira] [Commented] (SOLR-14413) allow timeAllowed and cursorMark parameters
[ https://issues.apache.org/jira/browse/SOLR-14413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17179204#comment-17179204 ] David Eric Pugh commented on SOLR-14413: [~bvd] emailed the mailing list about this ticket. Reading through the feedback, it appears what we need to do is: 1) update the tests to demonstrate the two parameters working together, and what happens when a timeAllowed is exceeded. Looking at it now, the tests that demonstrate they don’t work together have been removed, but no new ones added. 2) Updating the ref guide to explain what behavior occurs when the timeAllowed is exceeded. > allow timeAllowed and cursorMark parameters > --- > > Key: SOLR-14413 > URL: https://issues.apache.org/jira/browse/SOLR-14413 > Project: Solr > Issue Type: Improvement > Components: search >Reporter: John Gallagher >Priority: Minor > Attachments: SOLR-14413.patch, timeallowed_cursormarks_results.txt > > Time Spent: 10m > Remaining Estimate: 0h > > Ever since cursorMarks were introduced in SOLR-5463 in 2014, cursorMark and > timeAllowed parameters were not allowed in combination ("Can not search using > both cursorMark and timeAllowed") > , from [QueryComponent.java|#L359]]: > > {code:java} > > if (null != rb.getCursorMark() && 0 < timeAllowed) { > // fundamentally incompatible > throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Can not > search using both " + CursorMarkParams.CURSOR_MARK_PARAM + " and " + > CommonParams.TIME_ALLOWED); > } {code} > While theoretically impure to use them in combination, it is often desirable > to support cursormarks-style deep paging and attempt to protect Solr nodes > from runaway queries using timeAllowed, in the hopes that most of the time, > the query completes in the allotted time, and there is no conflict. > > However if the query takes too long, it may be preferable to end the query > and protect the Solr node and provide the user with a somewhat inaccurate > sorted list. As noted in SOLR-6930, SOLR-5986 and others, timeAllowed is > frequently used to prevent runaway load. In fact, cursorMark and > shards.tolerant are allowed in combination, so any argument in favor of > purity would be a bit muddied in my opinion. > > This was discussed once in the mailing list that I can find: > [https://mail-archives.apache.org/mod_mbox/lucene-solr-user/201506.mbox/%3c5591740b.4080...@elyograg.org%3E] > It did not look like there was strong support for preventing the combination. > > I have tested cursorMark and timeAllowed combination together, and even when > partial results are returned because the timeAllowed is exceeded, the > cursorMark response value is still valid and reasonable. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14413) allow timeAllowed and cursorMark parameters
[ https://issues.apache.org/jira/browse/SOLR-14413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17090672#comment-17090672 ] Mike Drob commented on SOLR-14413: -- That's awesome, thanks for verifying! Definitely update the docs with that info, please! The one concern I have at this point is about the return of zero results. Typically returning the same cursor indicates that we've reached the end of the results. Is there a way to distinguish the real end of the results from the case where we do not get any results in the time allowed? I know that we have the {{partialResults}} header there, but could there be a case where the opposite is true? We return partialResults:true, but there actually are no more results? Again, probably documenting the permutations here is sufficient. Also, can we add tests that explicitly demonstrate a partial cursor mark working? > allow timeAllowed and cursorMark parameters > --- > > Key: SOLR-14413 > URL: https://issues.apache.org/jira/browse/SOLR-14413 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: search >Reporter: John Gallagher >Priority: Minor > Attachments: SOLR-14413.patch, timeallowed_cursormarks_results.txt > > Time Spent: 10m > Remaining Estimate: 0h > > Ever since cursorMarks were introduced in SOLR-5463 in 2014, cursorMark and > timeAllowed parameters were not allowed in combination ("Can not search using > both cursorMark and timeAllowed") > , from [QueryComponent.java|#L359]]: > > {code:java} > > if (null != rb.getCursorMark() && 0 < timeAllowed) { > // fundamentally incompatible > throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Can not > search using both " + CursorMarkParams.CURSOR_MARK_PARAM + " and " + > CommonParams.TIME_ALLOWED); > } {code} > While theoretically impure to use them in combination, it is often desirable > to support cursormarks-style deep paging and attempt to protect Solr nodes > from runaway queries using timeAllowed, in the hopes that most of the time, > the query completes in the allotted time, and there is no conflict. > > However if the query takes too long, it may be preferable to end the query > and protect the Solr node and provide the user with a somewhat inaccurate > sorted list. As noted in SOLR-6930, SOLR-5986 and others, timeAllowed is > frequently used to prevent runaway load. In fact, cursorMark and > shards.tolerant are allowed in combination, so any argument in favor of > purity would be a bit muddied in my opinion. > > This was discussed once in the mailing list that I can find: > [https://mail-archives.apache.org/mod_mbox/lucene-solr-user/201506.mbox/%3c5591740b.4080...@elyograg.org%3E] > It did not look like there was strong support for preventing the combination. > > I have tested cursorMark and timeAllowed combination together, and even when > partial results are returned because the timeAllowed is exceeded, the > cursorMark response value is still valid and reasonable. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14413) allow timeAllowed and cursorMark parameters
[ https://issues.apache.org/jira/browse/SOLR-14413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17090169#comment-17090169 ] John Gallagher commented on SOLR-14413: --- Hi [~mdrob]. I've attached a session that I ran where I created 100 docs, restarted solr, and used a short timeAllowed=1 to get partial results. In my tests, the nextCursorMark works as you would want: it returns a nextCursorMark value corresponding to how far it got in the partial results (even of it returns 0 results, it will just return that cursor value again) The session has the commands I ran and the comments, but the TL;DR is here: I created 100 docs, from id:100 to id:199. I then restarted solr so that it would run my query cold and slowly: {code:java} # try to run the query and show all the rows: [johng solr (SOLR-14413)]$ curl -sq 'localhost:8983/solr/test_timeallowed_cursormark/query?q=*:*&cursorMark=*&timeAllowed=1&sort=id+asc&rows=100' { "responseHeader":{ "zkConnected":true, "partialResults":true, "status":0, "QTime":7, "params":{ "q":"*:*", "cursorMark":"*", "timeAllowed":"1", "sort":"id asc", "rows":"100"}}, "response":{"numFound":45,"start":0,"docs":[ { "id":"100", "_version_":1664726406826819584}, { "id":"101", "_version_":1664726406874005504}, { "id":"102", "_version_":1664726406902317056}, { "id":"103", "_version_":1664726406926434304}, { "id":"104", "_version_":1664726406952648704}, { "id":"105", "_version_":1664726406977814528}, { "id":"106", "_version_":1664726407000883200}, { "id":"107", "_version_":1664726407023951872}, { "id":"108", "_version_":1664726407047020544}, { "id":"109", "_version_":1664726407070089216}, { "id":"110", "_version_":1664726407094206464}, { "id":"111", "_version_":1664726407119372288}, { "id":"112", "_version_":1664726407142440960}, { "id":"113", "_version_":1664726407165509632}, { "id":"114", "_version_":1664726407188578304}, { "id":"115", "_version_":1664726407211646976}, { "id":"116", "_version_":1664726407235764224}, { "id":"117", "_version_":1664726407258832896}, { "id":"118", "_version_":1664726407280852992}, { "id":"119", "_version_":1664726407303921664}, { "id":"120", "_version_":1664726407328038912}, { "id":"121", "_version_":1664726407355301888}, { "id":"122", "_version_":1664726407378370560}, { "id":"123", "_version_":1664726407401439232}, { "id":"124", "_version_":1664726407425556480}, { "id":"125", "_version_":1664726407449673728}, { "id":"126", "_version_":1664726407474839552}, { "id":"127", "_version_":166472640755376}, { "id":"128", "_version_":1664726407522025472}, { "id":"129", "_version_":1664726407549288448}, { "id":"130", "_version_":1664726407572357120}, { "id":"131", "_version_":1664726407596474368}, { "id":"132", "_version_":1664726407619543040}, { "id":"133", "_version_":1664726407643660288}, { "id":"134", "_version_":1664726407668826112}, { "id":"135", "_version_":1664726407691894784}, { "id":"136", "_version_":1664726407716012032}, { "id":"137", "_version_":1664726407739080704}, { "id":"138", "_version_":1664726407760052224}, { "id":"139", "_version_":1664726407783120896}, { "id":"140", "_version_":1664726407805140992}, { "id":"141", "_version_":1664726407830306816}, { "id":"142", "_version_":1664726407854424064}, { "id":"143", "_version_":1664726407877492736}] }, "nextCursorMark":"AoEjMTQz"} # lets decode that cursormark - is it pointing at id:143 ?? YES! [johng solr (SOLR-14413)]$ python3 -c 'import base64; print(base64.b64decode("AoEjMTQz"))' b'\x02\x81#143' # using it as the next cursormark works as expected: [johng solr (SOLR-14413)]$ curl -sq 'localhost:8983/solr/test_timeallowed_cursormark/query?q=*:*&cursorMark=AoEjMTQz&timeAllowed=1&sort=id+asc&rows=100' { "responseHeader":{ "zkConnected":true, "partialResults":true, "status":0, "QTime":6, "params":{ "q":"*:*", "cursorMark":"AoEjMTQz", "timeAllowed":"1",
[jira] [Commented] (SOLR-14413) allow timeAllowed and cursorMark parameters
[ https://issues.apache.org/jira/browse/SOLR-14413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17089877#comment-17089877 ] Mike Drob commented on SOLR-14413: -- What is the behavior when timeAllowed is exceeded? Do we get a nextCursor value that corresponds to the partial results? Can we add documentation somewhere to explain what users should expect to see? > allow timeAllowed and cursorMark parameters > --- > > Key: SOLR-14413 > URL: https://issues.apache.org/jira/browse/SOLR-14413 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: search >Reporter: John Gallagher >Priority: Minor > Attachments: SOLR-14413.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Ever since cursorMarks were introduced in SOLR-5463 in 2014, cursorMark and > timeAllowed parameters were not allowed in combination ("Can not search using > both cursorMark and timeAllowed") > , from [QueryComponent.java|#L359]]: > > {code:java} > > if (null != rb.getCursorMark() && 0 < timeAllowed) { > // fundamentally incompatible > throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Can not > search using both " + CursorMarkParams.CURSOR_MARK_PARAM + " and " + > CommonParams.TIME_ALLOWED); > } {code} > While theoretically impure to use them in combination, it is often desirable > to support cursormarks-style deep paging and attempt to protect Solr nodes > from runaway queries using timeAllowed, in the hopes that most of the time, > the query completes in the allotted time, and there is no conflict. > > However if the query takes too long, it may be preferable to end the query > and protect the Solr node and provide the user with a somewhat inaccurate > sorted list. As noted in SOLR-6930, SOLR-5986 and others, timeAllowed is > frequently used to prevent runaway load. In fact, cursorMark and > shards.tolerant are allowed in combination, so any argument in favor of > purity would be a bit muddied in my opinion. > > This was discussed once in the mailing list that I can find: > [https://mail-archives.apache.org/mod_mbox/lucene-solr-user/201506.mbox/%3c5591740b.4080...@elyograg.org%3E] > It did not look like there was strong support for preventing the combination. > > I have tested cursorMark and timeAllowed combination together, and even when > partial results are returned because the timeAllowed is exceeded, the > cursorMark response value is still valid and reasonable. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14413) allow timeAllowed and cursorMark parameters
[ https://issues.apache.org/jira/browse/SOLR-14413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17086300#comment-17086300 ] Lucene/Solr QA commented on SOLR-14413: --- | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 40s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 30s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 30s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} Release audit (RAT) {color} | {color:green} 1m 30s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} Check forbidden APIs {color} | {color:green} 1m 30s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} Validate source patterns {color} | {color:green} 1m 30s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 74m 38s{color} | {color:green} core in the patch passed. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 82m 35s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Issue | SOLR-14413 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/13000318/SOLR-14413.patch | | Optional Tests | compile javac unit ratsources checkforbiddenapis validatesourcepatterns | | uname | Linux lucene2-us-west.apache.org 4.4.0-170-generic #199-Ubuntu SMP Thu Nov 14 01:45:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | ant | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-SOLR-Build/sourcedir/dev-tools/test-patch/lucene-solr-yetus-personality.sh | | git revision | master / 3af165b | | ant | version: Apache Ant(TM) version 1.9.6 compiled on July 20 2018 | | Default Java | LTS | | Test Results | https://builds.apache.org/job/PreCommit-SOLR-Build/737/testReport/ | | modules | C: solr/core U: solr/core | | Console output | https://builds.apache.org/job/PreCommit-SOLR-Build/737/console | | Powered by | Apache Yetus 0.7.0 http://yetus.apache.org | This message was automatically generated. > allow timeAllowed and cursorMark parameters > --- > > Key: SOLR-14413 > URL: https://issues.apache.org/jira/browse/SOLR-14413 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: search >Reporter: John Gallagher >Priority: Minor > Attachments: SOLR-14413.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Ever since cursorMarks were introduced in SOLR-5463 in 2014, cursorMark and > timeAllowed parameters were not allowed in combination ("Can not search using > both cursorMark and timeAllowed") > , from [QueryComponent.java|#L359]]: > > {code:java} > > if (null != rb.getCursorMark() && 0 < timeAllowed) { > // fundamentally incompatible > throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Can not > search using both " + CursorMarkParams.CURSOR_MARK_PARAM + " and " + > CommonParams.TIME_ALLOWED); > } {code} > While theoretically impure to use them in combination, it is often desirable > to support cursormarks-style deep paging and attempt to protect Solr nodes > from runaway queries using timeAllowed, in the hopes that most of the time, > the query completes in the allotted time, and there is no conflict. > > However if the query takes too long, it may be preferable to end the query > and protect the Solr node and provide the user with a somewhat inaccurate > sorted list. As noted in SOLR-6930, SOLR-5986 and others, timeAllowed is > frequently used to prevent runaway load. In fact, cursorMark and > shards.tolerant are allowed in combination, so any argument in favor of > purity would be a bit muddied in my opinion. > > This was discussed once in the mailing list that I can find: > [https://mail-archives.apache.org/mod_mbox/lucene-solr-user/201506.mbox/%3c5591740b.4080...@elyograg.org%3E] > It did not look like there was strong support for preventing the combination. > > I have tested cursorMark and timeAllowed combination together, and even when > partial results are r
[jira] [Commented] (SOLR-14413) allow timeAllowed and cursorMark parameters
[ https://issues.apache.org/jira/browse/SOLR-14413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17085971#comment-17085971 ] John Gallagher commented on SOLR-14413: --- Hi [~ichattopadhyaya], thanks for having a look! I've attached a patch, and created a PR here: [https://github.com/apache/lucene-solr/pull/1436] > allow timeAllowed and cursorMark parameters > --- > > Key: SOLR-14413 > URL: https://issues.apache.org/jira/browse/SOLR-14413 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: search >Reporter: John Gallagher >Priority: Minor > Attachments: SOLR-14413.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Ever since cursorMarks were introduced in SOLR-5463 in 2014, cursorMark and > timeAllowed parameters were not allowed in combination ("Can not search using > both cursorMark and timeAllowed") > , from [QueryComponent.java|#L359]]: > > {code:java} > > if (null != rb.getCursorMark() && 0 < timeAllowed) { > // fundamentally incompatible > throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Can not > search using both " + CursorMarkParams.CURSOR_MARK_PARAM + " and " + > CommonParams.TIME_ALLOWED); > } {code} > While theoretically impure to use them in combination, it is often desirable > to support cursormarks-style deep paging and attempt to protect Solr nodes > from runaway queries using timeAllowed, in the hopes that most of the time, > the query completes in the allotted time, and there is no conflict. > > However if the query takes too long, it may be preferable to end the query > and protect the Solr node and provide the user with a somewhat inaccurate > sorted list. As noted in SOLR-6930, SOLR-5986 and others, timeAllowed is > frequently used to prevent runaway load. In fact, cursorMark and > shards.tolerant are allowed in combination, so any argument in favor of > purity would be a bit muddied in my opinion. > > This was discussed once in the mailing list that I can find: > [https://mail-archives.apache.org/mod_mbox/lucene-solr-user/201506.mbox/%3c5591740b.4080...@elyograg.org%3E] > It did not look like there was strong support for preventing the combination. > > I have tested cursorMark and timeAllowed combination together, and even when > partial results are returned because the timeAllowed is exceeded, the > cursorMark response value is still valid and reasonable. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14413) allow timeAllowed and cursorMark parameters
[ https://issues.apache.org/jira/browse/SOLR-14413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17085965#comment-17085965 ] Ishan Chattopadhyaya commented on SOLR-14413: - Please raise a PR. > allow timeAllowed and cursorMark parameters > --- > > Key: SOLR-14413 > URL: https://issues.apache.org/jira/browse/SOLR-14413 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: search >Reporter: John Gallagher >Priority: Minor > > Ever since cursorMarks were introduced in SOLR-5463 in 2014, cursorMark and > timeAllowed parameters were not allowed in combination ("Can not search using > both cursorMark and timeAllowed") > , from [QueryComponent.java|#L359]]: > > {code:java} > > if (null != rb.getCursorMark() && 0 < timeAllowed) { > // fundamentally incompatible > throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Can not > search using both " + CursorMarkParams.CURSOR_MARK_PARAM + " and " + > CommonParams.TIME_ALLOWED); > } {code} > While theoretically impure to use them in combination, it is often desirable > to support cursormarks-style deep paging and attempt to protect Solr nodes > from runaway queries using timeAllowed, in the hopes that most of the time, > the query completes in the allotted time, and there is no conflict. > > However if the query takes too long, it may be preferable to end the query > and protect the Solr node and provide the user with a somewhat inaccurate > sorted list. As noted in SOLR-6930, SOLR-5986 and others, timeAllowed is > frequently used to prevent runaway load. In fact, cursorMark and > shards.tolerant are allowed in combination, so any argument in favor of > purity would be a bit muddied in my opinion. > > This was discussed once in the mailing list that I can find: > [https://mail-archives.apache.org/mod_mbox/lucene-solr-user/201506.mbox/%3c5591740b.4080...@elyograg.org%3E] > It did not look like there was strong support for preventing the combination. > > I have tested cursorMark and timeAllowed combination together, and even when > partial results are returned because the timeAllowed is exceeded, the > cursorMark response value is still valid and reasonable. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14413) allow timeAllowed and cursorMark parameters
[ https://issues.apache.org/jira/browse/SOLR-14413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17085964#comment-17085964 ] Ishan Chattopadhyaya commented on SOLR-14413: - Good catch. This combination should be supported. > allow timeAllowed and cursorMark parameters > --- > > Key: SOLR-14413 > URL: https://issues.apache.org/jira/browse/SOLR-14413 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: search >Reporter: John Gallagher >Priority: Minor > > Ever since cursorMarks were introduced in SOLR-5463 in 2014, cursorMark and > timeAllowed parameters were not allowed in combination ("Can not search using > both cursorMark and timeAllowed") > , from [QueryComponent.java|#L359]]: > > {code:java} > > if (null != rb.getCursorMark() && 0 < timeAllowed) { > // fundamentally incompatible > throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Can not > search using both " + CursorMarkParams.CURSOR_MARK_PARAM + " and " + > CommonParams.TIME_ALLOWED); > } {code} > While theoretically impure to use them in combination, it is often desirable > to support cursormarks-style deep paging and attempt to protect Solr nodes > from runaway queries using timeAllowed, in the hopes that most of the time, > the query completes in the allotted time, and there is no conflict. > > However if the query takes too long, it may be preferable to end the query > and protect the Solr node and provide the user with a somewhat inaccurate > sorted list. As noted in SOLR-6930, SOLR-5986 and others, timeAllowed is > frequently used to prevent runaway load. In fact, cursorMark and > shards.tolerant are allowed in combination, so any argument in favor of > purity would be a bit muddied in my opinion. > > This was discussed once in the mailing list that I can find: > [https://mail-archives.apache.org/mod_mbox/lucene-solr-user/201506.mbox/%3c5591740b.4080...@elyograg.org%3E] > It did not look like there was strong support for preventing the combination. > > I have tested cursorMark and timeAllowed combination together, and even when > partial results are returned because the timeAllowed is exceeded, the > cursorMark response value is still valid and reasonable. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org