[jira] [Commented] (LUCENE-6114) Remove bw compat cruft from packedints
[ https://issues.apache.org/jira/browse/LUCENE-6114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14247928#comment-14247928 ] Adrien Grand commented on LUCENE-6114: -- +1 Remove bw compat cruft from packedints -- Key: LUCENE-6114 URL: https://issues.apache.org/jira/browse/LUCENE-6114 Project: Lucene - Core Issue Type: Task Reporter: Robert Muir Fix For: Trunk Attachments: LUCENE-6114.patch In trunk we have some old logic that is not needed (versions 0 and 1). So we can remove support for structures that aren't byte-aligned, zigzag-encoded monotonics, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6813) distrib.singlePass does not work for expand-request - start/rows included
[
https://issues.apache.org/jira/browse/SOLR-6813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14247963#comment-14247963
]
Per Steffensen commented on SOLR-6813:
--
I would prefer making it work instead of saying that single-pass in not
supported for expand-queries. But I guess that is easy for me to say :-)
If I understand you explanations correctly, expand-queries ought to (almost)
always fail for distributed. Also when rows/start is not set (default start=0
and rows=10). Can you explain why it does not fail in those cases
{code}
query(q, *:*, fq, {!collapse field=+group+}, defType,
edismax, bf, field(test_ti), expand, true, fl,*,score,
ShardParams.DISTRIB_SINGLE_PASS, true);
query(q, *:*, fq, {!collapse field=+group+}, defType,
edismax, bf, field(test_ti), expand, true, expand.sort, test_tl
desc, fl,*,score, ShardParams.DISTRIB_SINGLE_PASS, true);
query(q, *:*, fq, {!collapse field=+group+}, defType,
edismax, bf, field(test_ti), expand, true, expand.sort, test_tl
desc, expand.rows, 1, fl,*,score, ShardParams.DISTRIB_SINGLE_PASS,
true);
//Test no expand results
query(q, test_ti:5, fq, {!collapse field=+group+}, defType,
edismax, bf, field(test_ti), expand, true, expand.sort, test_tl
desc, expand.rows, 1, fl,*,score, ShardParams.DISTRIB_SINGLE_PASS,
true);
//Test zero results
query(q, test_ti:5434343, fq, {!collapse field=+group+},
defType, edismax, bf, field(test_ti), expand, true, expand.sort,
test_tl desc, expand.rows, 1, fl,*,score,
ShardParams.DISTRIB_SINGLE_PASS, true);
//Test page 2
query(q, *:*, start,1, rows, 1, fq, {!collapse
field=+group+}, defType, edismax, bf, field(test_ti), expand,
true, fl,*,score, ShardParams.DISTRIB_SINGLE_PASS, true);
{code}
Only the last one above fails?
distrib.singlePass does not work for expand-request - start/rows included
-
Key: SOLR-6813
URL: https://issues.apache.org/jira/browse/SOLR-6813
Project: Solr
Issue Type: Bug
Components: multicore, search
Reporter: Per Steffensen
Assignee: Joel Bernstein
Labels: distributed_search, search
Attachments: test_that_reveals_the_problem.patch
Using distrib.singlePass does not work for expand-requests. Even after the
fix provided to SOLR-6812, it does not work for requests where you add start
and/or rows.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[JENKINS] Lucene-Solr-Tests-5.x-Java7 - Build # 2334 - Still Failing
Build: https://builds.apache.org/job/Lucene-Solr-Tests-5.x-Java7/2334/ 1 tests failed. FAILED: org.apache.solr.cloud.TestModifyConfFiles.testDistribSearch Error Message: expected:[Error from server at https://127.0.0.1:14720/collection1: ]No file name specifi... but was:[]No file name specifi... Stack Trace: org.junit.ComparisonFailure: expected:[Error from server at https://127.0.0.1:14720/collection1: ]No file name specifi... but was:[]No file name specifi... at __randomizedtesting.SeedInfo.seed([6B69D83959C21E61:EA8F56212E9D7E5D]:0) at org.junit.Assert.assertEquals(Assert.java:125) at org.junit.Assert.assertEquals(Assert.java:147) at org.apache.solr.cloud.TestModifyConfFiles.doTest(TestModifyConfFiles.java:65) at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869) at sun.reflect.GeneratedMethodAccessor44.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836) at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738) at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) at
[JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.8.0_40-ea-b09) - Build # 4493 - Still Failing!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Windows/4493/ Java: 64bit/jdk1.8.0_40-ea-b09 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC (asserts: false) 1 tests failed. FAILED: org.apache.solr.cloud.TestModifyConfFiles.testDistribSearch Error Message: expected:[Error from server at https://127.0.0.1:61762/collection1: ]No file name specifi... but was:[]No file name specifi... Stack Trace: org.junit.ComparisonFailure: expected:[Error from server at https://127.0.0.1:61762/collection1: ]No file name specifi... but was:[]No file name specifi... at __randomizedtesting.SeedInfo.seed([DA45BB2D0638DF7:8C42D5AAA73CEDCB]:0) at org.junit.Assert.assertEquals(Assert.java:125) at org.junit.Assert.assertEquals(Assert.java:147) at org.apache.solr.cloud.TestModifyConfFiles.doTest(TestModifyConfFiles.java:65) at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869) at sun.reflect.GeneratedMethodAccessor44.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836) at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738) at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at
[jira] [Commented] (SOLR-6849) RemoteSolrExceptions thrown from HttpSolrServer should include the URL of the remote host
[ https://issues.apache.org/jira/browse/SOLR-6849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248079#comment-14248079 ] ASF subversion and git services commented on SOLR-6849: --- Commit 1645895 from [~romseygeek] in branch 'dev/trunk' [ https://svn.apache.org/r1645895 ] SOLR-6849: And really fix it this time RemoteSolrExceptions thrown from HttpSolrServer should include the URL of the remote host - Key: SOLR-6849 URL: https://issues.apache.org/jira/browse/SOLR-6849 Project: Solr Issue Type: Improvement Reporter: Alan Woodward Assignee: Alan Woodward Priority: Minor Fix For: 5.0, Trunk Attachments: SOLR-6849.patch All very well telling me there was an error on a remote host, but it's difficult to work out what's wrong if it doesn't tell me *which* host the error was on... -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6849) RemoteSolrExceptions thrown from HttpSolrServer should include the URL of the remote host
[ https://issues.apache.org/jira/browse/SOLR-6849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248080#comment-14248080 ] ASF subversion and git services commented on SOLR-6849: --- Commit 1645896 from [~romseygeek] in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1645896 ] SOLR-6849: And really fix it this time RemoteSolrExceptions thrown from HttpSolrServer should include the URL of the remote host - Key: SOLR-6849 URL: https://issues.apache.org/jira/browse/SOLR-6849 Project: Solr Issue Type: Improvement Reporter: Alan Woodward Assignee: Alan Woodward Priority: Minor Fix For: 5.0, Trunk Attachments: SOLR-6849.patch All very well telling me there was an error on a remote host, but it's difficult to work out what's wrong if it doesn't tell me *which* host the error was on... -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (LUCENE-6115) Add getMergeInstance to CompressingStoredFieldsReader
Adrien Grand created LUCENE-6115: Summary: Add getMergeInstance to CompressingStoredFieldsReader Key: LUCENE-6115 URL: https://issues.apache.org/jira/browse/LUCENE-6115 Project: Lucene - Core Issue Type: Improvement Reporter: Adrien Grand Assignee: Adrien Grand Priority: Minor CompressingStoredFieldsReader is currently terrible at merging with different codecs or wrapped readers since it does not keep state. So if you want to get 5 documents that come from the same block, it means that you will have to decode the block header and decompress 5 times. It has some optimizations so that if you want to get the 2nd doc of the block then it will stop decompressing soon after the 2nd document, but it doesn't help much with merging since we want all documents. We should implement getMergeInstance and have a different behaviour when merging by decompressing everything up-front and then reusing for all documents of the block. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-6115) Add getMergeInstance to CompressingStoredFieldsReader
[ https://issues.apache.org/jira/browse/LUCENE-6115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248096#comment-14248096 ] Robert Muir commented on LUCENE-6115: - +1, Can we make a Buffering instance or something similar? Maybe it would be simple and reusable. Add getMergeInstance to CompressingStoredFieldsReader - Key: LUCENE-6115 URL: https://issues.apache.org/jira/browse/LUCENE-6115 Project: Lucene - Core Issue Type: Improvement Reporter: Adrien Grand Assignee: Adrien Grand Priority: Minor CompressingStoredFieldsReader is currently terrible at merging with different codecs or wrapped readers since it does not keep state. So if you want to get 5 documents that come from the same block, it means that you will have to decode the block header and decompress 5 times. It has some optimizations so that if you want to get the 2nd doc of the block then it will stop decompressing soon after the 2nd document, but it doesn't help much with merging since we want all documents. We should implement getMergeInstance and have a different behaviour when merging by decompressing everything up-front and then reusing for all documents of the block. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-6115) Add getMergeInstance to CompressingStoredFieldsReader
[ https://issues.apache.org/jira/browse/LUCENE-6115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adrien Grand updated LUCENE-6115: - Attachment: LUCENE-6115.patch Here is a patch. Here are the differences with today: - when doing random access, the header is always completely decoded (while we previously sometimes stopped eagerly) - when merging, we make sure to only decompress a given block of documents a single time - checksums are verified up-front instead of on-the-fly (like most other formats actually) - I made the logic for large blocks more robust. Up to now, if you were storing a document that was so large that it would grow larger than 2x the chunk size, then it would be splitted into 16KB (exact this time) slices in order to make decompressing a bit more memory-efficient. The reader used to duplicate the writer logic (if block_len = 2 * chunk_size) but this info is now encoded in the stream. I did some benchmarking and there were no significant differences in terms of indexing speed or read speed, so having to read the whole header every time does not seem to hurt (since the bottleneck is likely decompressing documents). I tried to remove the specialized merging to see if it was still needed and unfortunately it seems so, I got merging times that were about 20% slower without specialized merging. (In that case specialized merging still decompresses and recompresses all the time, it only saves some decoding and reuses directly the serialized bytes of each document.) Add getMergeInstance to CompressingStoredFieldsReader - Key: LUCENE-6115 URL: https://issues.apache.org/jira/browse/LUCENE-6115 Project: Lucene - Core Issue Type: Improvement Reporter: Adrien Grand Assignee: Adrien Grand Priority: Minor Attachments: LUCENE-6115.patch CompressingStoredFieldsReader is currently terrible at merging with different codecs or wrapped readers since it does not keep state. So if you want to get 5 documents that come from the same block, it means that you will have to decode the block header and decompress 5 times. It has some optimizations so that if you want to get the 2nd doc of the block then it will stop decompressing soon after the 2nd document, but it doesn't help much with merging since we want all documents. We should implement getMergeInstance and have a different behaviour when merging by decompressing everything up-front and then reusing for all documents of the block. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-6115) Add getMergeInstance to CompressingStoredFieldsReader
[ https://issues.apache.org/jira/browse/LUCENE-6115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248133#comment-14248133 ] Robert Muir commented on LUCENE-6115: - At a glance the patch looks great. About testing, does/can our BaseXXXStoredFieldsTest exercise merging in the different ways? Stored fields are easy to write without flushing and merging. we should be able to do things like call addIndexes(bogusWrapper) for now and really exercise the different cases. Add getMergeInstance to CompressingStoredFieldsReader - Key: LUCENE-6115 URL: https://issues.apache.org/jira/browse/LUCENE-6115 Project: Lucene - Core Issue Type: Improvement Reporter: Adrien Grand Assignee: Adrien Grand Priority: Minor Attachments: LUCENE-6115.patch CompressingStoredFieldsReader is currently terrible at merging with different codecs or wrapped readers since it does not keep state. So if you want to get 5 documents that come from the same block, it means that you will have to decode the block header and decompress 5 times. It has some optimizations so that if you want to get the 2nd doc of the block then it will stop decompressing soon after the 2nd document, but it doesn't help much with merging since we want all documents. We should implement getMergeInstance and have a different behaviour when merging by decompressing everything up-front and then reusing for all documents of the block. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-2878) Allow Scorer to expose positions and payloads aka. nuke spans
[ https://issues.apache.org/jira/browse/LUCENE-2878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248162#comment-14248162 ] ASF subversion and git services commented on LUCENE-2878: - Commit 1645925 from [~romseygeek] in branch 'dev/branches/lucene2878' [ https://svn.apache.org/r1645925 ] LUCENE-2878: merge conflicts Allow Scorer to expose positions and payloads aka. nuke spans -- Key: LUCENE-2878 URL: https://issues.apache.org/jira/browse/LUCENE-2878 Project: Lucene - Core Issue Type: Improvement Components: core/search Affects Versions: Positions Branch Reporter: Simon Willnauer Assignee: Robert Muir Labels: gsoc2014 Fix For: Positions Branch Attachments: LUCENE-2878-OR.patch, LUCENE-2878-vs-trunk.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878_trunk.patch, LUCENE-2878_trunk.patch, PosHighlighter.patch, PosHighlighter.patch Currently we have two somewhat separate types of queries, the one which can make use of positions (mainly spans) and payloads (spans). Yet Span*Query doesn't really do scoring comparable to what other queries do and at the end of the day they are duplicating lot of code all over lucene. Span*Queries are also limited to other Span*Query instances such that you can not use a TermQuery or a BooleanQuery with SpanNear or anthing like that. Beside of the Span*Query limitation other queries lacking a quiet interesting feature since they can not score based on term proximity since scores doesn't expose any positional information. All those problems bugged me for a while now so I stared working on that using the bulkpostings API. I would have done that first cut on trunk but TermScorer is working on BlockReader that do not expose positions while the one in this branch does. I started adding a new Positions class which users can pull from a scorer, to prevent unnecessary positions enums I added ScorerContext#needsPositions and eventually Scorere#needsPayloads to create the corresponding enum on demand. Yet, currently only TermQuery / TermScorer implements this API and other simply return null instead. To show that the API really works and our BulkPostings work fine too with positions I cut over TermSpanQuery to use a TermScorer under the hood and nuked TermSpans entirely. A nice sideeffect of this was that the Position BulkReading implementation got some exercise which now :) work all with positions while Payloads for bulkreading are kind of experimental in the patch and those only work with Standard codec. So all spans now work on top of TermScorer ( I truly hate spans since today ) including the ones that need Payloads (StandardCodec ONLY)!! I didn't bother to implement the other codecs yet since I want to get feedback on the API and on this first cut before I go one with it. I will upload the corresponding patch in a minute. I also had to cut over SpanQuery.getSpans(IR) to SpanQuery.getSpans(AtomicReaderContext) which I should probably do on trunk first but after that pain today I need a break first :). The patch passes all core tests (org.apache.lucene.search.highlight.HighlighterTest still fails but I didn't look into the MemoryIndex BulkPostings API yet) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-2878) Allow Scorer to expose positions and payloads aka. nuke spans
[ https://issues.apache.org/jira/browse/LUCENE-2878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Woodward updated LUCENE-2878: -- Attachment: LUCENE-2878.patch nocommits either resolved or (mostly!) changed to TODOs. I think this is ready. Allow Scorer to expose positions and payloads aka. nuke spans -- Key: LUCENE-2878 URL: https://issues.apache.org/jira/browse/LUCENE-2878 Project: Lucene - Core Issue Type: Improvement Components: core/search Affects Versions: Positions Branch Reporter: Simon Willnauer Assignee: Robert Muir Labels: gsoc2014 Fix For: Positions Branch Attachments: LUCENE-2878-OR.patch, LUCENE-2878-vs-trunk.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878_trunk.patch, LUCENE-2878_trunk.patch, PosHighlighter.patch, PosHighlighter.patch Currently we have two somewhat separate types of queries, the one which can make use of positions (mainly spans) and payloads (spans). Yet Span*Query doesn't really do scoring comparable to what other queries do and at the end of the day they are duplicating lot of code all over lucene. Span*Queries are also limited to other Span*Query instances such that you can not use a TermQuery or a BooleanQuery with SpanNear or anthing like that. Beside of the Span*Query limitation other queries lacking a quiet interesting feature since they can not score based on term proximity since scores doesn't expose any positional information. All those problems bugged me for a while now so I stared working on that using the bulkpostings API. I would have done that first cut on trunk but TermScorer is working on BlockReader that do not expose positions while the one in this branch does. I started adding a new Positions class which users can pull from a scorer, to prevent unnecessary positions enums I added ScorerContext#needsPositions and eventually Scorere#needsPayloads to create the corresponding enum on demand. Yet, currently only TermQuery / TermScorer implements this API and other simply return null instead. To show that the API really works and our BulkPostings work fine too with positions I cut over TermSpanQuery to use a TermScorer under the hood and nuked TermSpans entirely. A nice sideeffect of this was that the Position BulkReading implementation got some exercise which now :) work all with positions while Payloads for bulkreading are kind of experimental in the patch and those only work with Standard codec. So all spans now work on top of TermScorer ( I truly hate spans since today ) including the ones that need Payloads (StandardCodec ONLY)!! I didn't bother to implement the other codecs yet since I want to get feedback on the API and on this first cut before I go one with it. I will upload the corresponding patch in a minute. I also had to cut over SpanQuery.getSpans(IR) to SpanQuery.getSpans(AtomicReaderContext) which I should probably do on trunk first but after that pain today I need a break first :). The patch passes all core tests (org.apache.lucene.search.highlight.HighlighterTest still fails but I didn't look into the MemoryIndex BulkPostings API yet) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Interesting blog on G1 GC improvemnts u25 - u60
Hi Shawn, Could you start a discussion on this at the Hotspot GC usage-mailing list ? http://mail.openjdk.java.net/mailman/listinfo/hotspot-gc-use Thanks in advance, Rory On 16/12/2014 06:07, Shawn Heisey wrote: On 12/6/2014 3:00 PM, Shawn Heisey wrote: On 12/5/2014 2:42 PM, Erick Erickson wrote: Saw this on the Cloudera website: http://blog.cloudera.com/blog/2014/12/tuning-java-garbage-collection-for-hbase/ Original post here: https://software.intel.com/en-us/blogs/2014/06/18/part-1-tuning-java-garbage-collection-for-hbase Although it's for hbase, I thought the presentation went into enough detail about what improvements they'd seen that I can see it being useful for Solr folks. And we have some people on this list who are interested in this sort of thing Very interesting. My own experiences with G1 and Solr (which I haven't repeated since early Java 7 releases, something like 7u10 or 7u13) would show even worse spikes compared to the blue lines on those graphs ... and my heap isn't anywhere even CLOSE to 100GB. Solr probably has different garbage creation characteristics than hbase. Followup with graphs. I've cc'd Rory at Oracle too, with hopes that this info will ultimately reach those who work on G1. I can provide the actual GC logs as well. Here's a graph of a GC log lasting over two weeks with a tuned CMS collector and Oracle Java 7u25 and a 6GB heap. https://www.dropbox.com/s/mygjeviyybqqnqd/cms-7u25.png?dl=0 CMS was tuned using these settings: http://wiki.apache.org/solr/ShawnHeisey#GC_Tuning This graph shows that virtually all collection pauses were a little under half a second. There were exactly three full garbage collections, and each one took around six seconds. While that is a significant pause, having only three such collections over a period of 16 days sounds pretty good to me. Here's about half as much runtime (8 days) on the same server running G1 with Oracle 7u72 and the same 6GB heap. G1 is untuned, because I do not know how: https://www.dropbox.com/s/2kgx60gj988rflj/g1-7u72.png?dl=0 Most of these collections were around a tenth of a second ... which is certainly better than nearly half a second ... but there are a LOT of collections that take longer than a second, and a fair number of them that took between 3 and 5 seconds. It's difficult to say which of these graphs is actually better. The CMS graph is certainly more consistent, and does a LOT fewer full GCs ... but is the 4 to 1 improvement in a typical GC enough to reveal significantly better performance? My instinct says that it would NOT be enough for that, especially with so many collections taking 1-3 seconds. If the server was really busy (mine isn't), I wonder whether the GC graph would look similar, or whether it would be really different. A busy server would need to collect a lot more garbage, so I fear that the yellow and black parts of the G1 graph would dominate more than they do in my graph, which would be overall a bad thing. Only real testing on busy servers can tell us that. I can tell you for sure that the G1 graph looks a lot better than it did in early Java 7 releases, but additional work by Oracle (and perhaps some G1 tuning options) might significantly improve it. Thanks, Shawn -- Rgds,Rory O'Donnell Quality Engineering Manager Oracle EMEA , Dublin, Ireland - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-2878) Allow Scorer to expose positions and payloads aka. nuke spans
[
https://issues.apache.org/jira/browse/LUCENE-2878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248188#comment-14248188
]
Robert Muir commented on LUCENE-2878:
-
Hi Alan,
Why are there now additional branches in nextPosition() in posting readers? Can
we avoid these? If it is to support calling nextPosition() after it has already
been exhausted, I am unsure we should be doing this. nextDoc() does not support
this, for example.
Can we improve this logic in reuse code? Reuse code is notorious for sneaky
bugs, and I don't like the comparison done here. Why can't we just check if the
bit is set?
{code}
if ((flags DocsEnum.FLAG_POSITIONS) = DocsEnum.FLAG_POSITIONS)
{code}
This check also is confusing to me in the first place. Why do we have both
docsEnum and docsAndPositionsEnum methods, both returning DocsEnum, with the
former checking for a FLAG_POSITIONS and calling the other one in that case. I
think there should just be one method instead if we want both to share the same
api, thats ok.
TermScorer introduces outdated dead code that is unused.
What are the Span scoring classes still doing here?
Do we have tests for any of this new code? I see no added tests files.
What about benchmarks? We should try to compare against trunk, since there are
a lot of change here.
In general, I can try to make a patch for smaller things, to make things
faster. Iterating this way may take a long time.
Allow Scorer to expose positions and payloads aka. nuke spans
--
Key: LUCENE-2878
URL: https://issues.apache.org/jira/browse/LUCENE-2878
Project: Lucene - Core
Issue Type: Improvement
Components: core/search
Affects Versions: Positions Branch
Reporter: Simon Willnauer
Assignee: Robert Muir
Labels: gsoc2014
Fix For: Positions Branch
Attachments: LUCENE-2878-OR.patch, LUCENE-2878-vs-trunk.patch,
LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch,
LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch,
LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch,
LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch,
LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch,
LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch,
LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch,
LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878_trunk.patch,
LUCENE-2878_trunk.patch, PosHighlighter.patch, PosHighlighter.patch
Currently we have two somewhat separate types of queries, the one which can
make use of positions (mainly spans) and payloads (spans). Yet Span*Query
doesn't really do scoring comparable to what other queries do and at the end
of the day they are duplicating lot of code all over lucene. Span*Queries are
also limited to other Span*Query instances such that you can not use a
TermQuery or a BooleanQuery with SpanNear or anthing like that.
Beside of the Span*Query limitation other queries lacking a quiet interesting
feature since they can not score based on term proximity since scores doesn't
expose any positional information. All those problems bugged me for a while
now so I stared working on that using the bulkpostings API. I would have done
that first cut on trunk but TermScorer is working on BlockReader that do not
expose positions while the one in this branch does. I started adding a new
Positions class which users can pull from a scorer, to prevent unnecessary
positions enums I added ScorerContext#needsPositions and eventually
Scorere#needsPayloads to create the corresponding enum on demand. Yet,
currently only TermQuery / TermScorer implements this API and other simply
return null instead.
To show that the API really works and our BulkPostings work fine too with
positions I cut over TermSpanQuery to use a TermScorer under the hood and
nuked TermSpans entirely. A nice sideeffect of this was that the Position
BulkReading implementation got some exercise which now :) work all with
positions while Payloads for bulkreading are kind of experimental in the
patch and those only work with Standard codec.
So all spans now work on top of TermScorer ( I truly hate spans since today )
including the ones that need Payloads (StandardCodec ONLY)!! I didn't bother
to implement the other codecs yet since I want to get feedback on the API and
on this first cut before I go one with it. I will upload the corresponding
patch in a minute.
I also had to cut over SpanQuery.getSpans(IR) to
SpanQuery.getSpans(AtomicReaderContext) which I should probably do on trunk
first but after that pain today I need a break first :).
The
[jira] [Commented] (SOLR-6813) distrib.singlePass does not work for expand-request - start/rows included
[ https://issues.apache.org/jira/browse/SOLR-6813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248200#comment-14248200 ] Joel Bernstein commented on SOLR-6813: -- The test will fail when the number of collapsed groups returned by the query is larger then the page size. For example with start=1 and rows=1 the controlRsp will have one expanded group, while the rsp will have multiple expanded groups. With default page size of 10, both the controlRsp and rsp will have four expanded groups so the tests pass. We can solve the test failures easily with a change to handleResults to remove groups that are not in the final docList. The distributed deep paging problem will continue to exist. The only way I see to resolve that problem is with the two pass model. distrib.singlePass is not the most efficient approach for every situation, so we can pick and choose when to apply it. [~shalinmangar], how does distrib.singlePass work with standard grouping? I suspect grouping still does multi-pass because it has a different distributed flow. distrib.singlePass does not work for expand-request - start/rows included - Key: SOLR-6813 URL: https://issues.apache.org/jira/browse/SOLR-6813 Project: Solr Issue Type: Bug Components: multicore, search Reporter: Per Steffensen Assignee: Joel Bernstein Labels: distributed_search, search Attachments: test_that_reveals_the_problem.patch Using distrib.singlePass does not work for expand-requests. Even after the fix provided to SOLR-6812, it does not work for requests where you add start and/or rows. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-6813) distrib.singlePass does not work for expand-request - start/rows included
[ https://issues.apache.org/jira/browse/SOLR-6813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248200#comment-14248200 ] Joel Bernstein edited comment on SOLR-6813 at 12/16/14 12:37 PM: - The test will fail when the number of collapsed groups returned by the query is larger then the page size. For example with start=1 and rows=1 the controlRsp will have one expanded group, while the rsp will have multiple expanded groups. With default page size of 10, both the controlRsp and rsp will have four expanded groups so the tests pass. We can solve the test failures easily with a change to handleResults to remove groups that are not in the final docList. The distributed deep paging problem will continue to exist. The only way I see to resolve that problem is with the two pass model. distrib.singlePass is not the most efficient approach for every situation, so we can pick and choose when to apply it. [~shalinmangar], how does distrib.singlePass work with standard grouping? I suspect grouping still does multi-pass because it has a different distributed flow. was (Author: joel.bernstein): The test will fail when the number of collapsed groups returned by the query is larger then the page size. For example with start=1 and rows=1 the controlRsp will have one expanded group, while the rsp will have multiple expanded groups. With default page size of 10, both the controlRsp and rsp will have four expanded groups so the tests pass. We can solve the test failures easily with a change to handleResults to remove groups that are not in the final docList. The distributed deep paging problem will continue to exist. The only way I see to resolve that problem is with the two pass model. distrib.singlePass is not the most efficient approach for every situation, so we can pick and choose when to apply it. [~shalinmangar], how does distrib.singlePass work with standard grouping? I suspect grouping still does multi-pass because it has a different distributed flow. distrib.singlePass does not work for expand-request - start/rows included - Key: SOLR-6813 URL: https://issues.apache.org/jira/browse/SOLR-6813 Project: Solr Issue Type: Bug Components: multicore, search Reporter: Per Steffensen Assignee: Joel Bernstein Labels: distributed_search, search Attachments: test_that_reveals_the_problem.patch Using distrib.singlePass does not work for expand-requests. Even after the fix provided to SOLR-6812, it does not work for requests where you add start and/or rows. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Early Access builds for JDK 9 b42, JDK 8 b18 JDK 7 b03 are available on java.net
Hi Uwe Dawid, Now that JDK 9 Early Access build images are modular [1], there is a fresh Early Access build for JDK 9 b42 https://jdk9.java.net/download/ available on java.net. The summary of changes are listed here http://www.java.net/download/jdk9/changes/jdk9-b42.html In addition, there are new Early Access builds for the ongoing update releases. The Early Access build for JDK 8u40 b18 http://jdk8.java.net/download.html is available on java.net, with the summary of changes listed here. http://www.java.net/download/jdk8u40/changes/jdk8u40-b18.html Finally, the Early Access build for JDK 7u80 b03 https://jdk7.java.net/download.htmlis available on java.net, with the summary of changes listed here. http://download.java.net/jdk7u80/changes/jdk7u80-b03.html As we enter the later phases of development for JDK 7u80 JDK 8u40, please log any show stoppers as soon as possible. Rgds,Rory [1] http://mreinhold.org/blog/jigsaw-modular-images -- Rgds,Rory O'Donnell Quality Engineering Manager Oracle EMEA , Dublin, Ireland
[JENKINS] Lucene-Solr-5.x-Windows (32bit/jdk1.8.0_20) - Build # 4388 - Still Failing!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-5.x-Windows/4388/ Java: 32bit/jdk1.8.0_20 -client -XX:+UseParallelGC (asserts: true) 1 tests failed. FAILED: org.apache.solr.cloud.TestModifyConfFiles.testDistribSearch Error Message: expected:[Error from server at https://127.0.0.1:59420/collection1: ]No file name specifi... but was:[]No file name specifi... Stack Trace: org.junit.ComparisonFailure: expected:[Error from server at https://127.0.0.1:59420/collection1: ]No file name specifi... but was:[]No file name specifi... at __randomizedtesting.SeedInfo.seed([E279001F7561224A:639F8E07023E4276]:0) at org.junit.Assert.assertEquals(Assert.java:125) at org.junit.Assert.assertEquals(Assert.java:147) at org.apache.solr.cloud.TestModifyConfFiles.doTest(TestModifyConfFiles.java:65) at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869) at sun.reflect.GeneratedMethodAccessor82.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836) at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738) at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at
[jira] [Commented] (LUCENE-6115) Add getMergeInstance to CompressingStoredFieldsReader
[ https://issues.apache.org/jira/browse/LUCENE-6115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248222#comment-14248222 ] Adrien Grand commented on LUCENE-6115: -- We already have testWriteReadMerge which tests merging across codecs, I'll add another test with a filter reader. Add getMergeInstance to CompressingStoredFieldsReader - Key: LUCENE-6115 URL: https://issues.apache.org/jira/browse/LUCENE-6115 Project: Lucene - Core Issue Type: Improvement Reporter: Adrien Grand Assignee: Adrien Grand Priority: Minor Attachments: LUCENE-6115.patch CompressingStoredFieldsReader is currently terrible at merging with different codecs or wrapped readers since it does not keep state. So if you want to get 5 documents that come from the same block, it means that you will have to decode the block header and decompress 5 times. It has some optimizations so that if you want to get the 2nd doc of the block then it will stop decompressing soon after the 2nd document, but it doesn't help much with merging since we want all documents. We should implement getMergeInstance and have a different behaviour when merging by decompressing everything up-front and then reusing for all documents of the block. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-6853) solr.ManagedSynonymFilterFactory/ManagedStopwordFilterFactory: URLEncoding - Not able to delete Synonyms/Stopwords with special characters
Tomasz Sulkowski created SOLR-6853:
--
Summary:
solr.ManagedSynonymFilterFactory/ManagedStopwordFilterFactory: URLEncoding -
Not able to delete Synonyms/Stopwords with special characters
Key: SOLR-6853
URL: https://issues.apache.org/jira/browse/SOLR-6853
Project: Solr
Issue Type: Bug
Components: Schema and Analysis
Affects Versions: 4.10.2
Environment: Solr 4.10.2 running @ Win7
Reporter: Tomasz Sulkowski
Hi Guys,
We're using the SOLR Rest API in order to manage synonyms and stopwords with
solr.Managed*FilterFactory.
{_emphasis_}The same applies to stopwords. I am going to explain the synonym
case only from this point on.{_emphasis_}
Let us consider the following _schema_analysis_synonyms_en.json managedMap: {
xxx#xxx:[xxx#xxx],
xxx%xxx:[xxx%xxx],
xxx/xxx:[xxx/xxx],
xxx:xxx:[xxx:xxx],
xxx;xxx:[xxx;xxx],
xx :[xx ]
}
I can add such synonym to keyword relations using REST API. The problem is that
I cannot remove/list them as
http://localhost:8983/solr/collection1/schema/analysis/synonyms/en/entryname
where entryname is one of the map's key throws 404, or 500 (in case of
xxx%25xxx):
java.lang.NullPointerException at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:367)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:368) at
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
at
org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
at
org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942)
at
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640) at
org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) at
org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
at
org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Unknown Source)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
Transactions multiplexing IndexWriter
Hey devs,
I am looking at making use of the two-phase commit approach available
in IndexWriter but the current architecture there does not quite fit
with what we want to achieve. It seems that I could build this atop
IndexWriter, however I wonder if there is either an existing
alternative that I have not discovered or whether it would be better
to contribute a patch to the Lucene project itself?
In my system I have many concurrent transactions and each of them
needs to make modifications to a single Lucene index in an atomic and
consistent manner.
If I had a single-thread (i.e. one concurrent transaction) for
example, I could access the IndexWriter instance, call addDocument or
whatever as many times as I like, call prepareForCommit and then
commit.
The main issue that I have is that if I let all concurrent
transactions use the same IndexWriter then I loose isolation, as a
commit of one transaction may write the partial pending updates of
another transaction.
Now I can see a naive solution for my application where I could add
all updates that I want to make to the index to a `pending list`, I
could then take an exclusive lock for the index writer, apply my
pending list to the index writer and then commit, finally releasing
the exclusive lock. Whilst I could get this working, the down-side is
that I have to implement and manage this `pending list` and applying
it to the index myself, and it comes at the cost of memory (or even
paging it to disk).
It seems likely to me that others before me have also had such a
requirement, does anything like this already exist in Lucene or would
it be desirable for me to contribute something? At a rough guess I
would imagine separating the IndexWriter from transaction content,
something like:
try(Transaction txn = indexWriter.beginTransaction()) {
indexWriter.addDocument(txn, doc1);
indexWriter.addDocument(txn, doc2);
indexWriter.addDocument(txn, doc3);
indexWriter.commit(txn);
}
The transaction could be automatically rolled-back in the close()
method called by try-with-resources if it has not been committed,
which would allow any exceptions to be cleanly handled by the caller.
Does that make any sense, or am I way off?
Cheers Adam.
--
Adam Retter
skype: adam.retter
tweet: adamretter
http://www.adamretter.org.uk
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3393) Implement an optimized LFUCache
[ https://issues.apache.org/jira/browse/SOLR-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248265#comment-14248265 ] Shawn Heisey commented on SOLR-3393: I really need to finish this and see if I can get it into 5.0. Implement an optimized LFUCache --- Key: SOLR-3393 URL: https://issues.apache.org/jira/browse/SOLR-3393 Project: Solr Issue Type: Improvement Components: search Affects Versions: 3.6, 4.0-ALPHA Reporter: Shawn Heisey Assignee: Shawn Heisey Priority: Minor Fix For: 4.9, Trunk Attachments: SOLR-3393-4x-withdecay.patch, SOLR-3393-trunk-withdecay.patch, SOLR-3393.patch, SOLR-3393.patch, SOLR-3393.patch, SOLR-3393.patch, SOLR-3393.patch, SOLR-3393.patch, SOLR-3393.patch SOLR-2906 gave us an inefficient LFU cache modeled on FastLRUCache/ConcurrentLRUCache. It could use some serious improvement. The following project includes an Apache 2.0 licensed O(1) implementation. The second link is the paper (PDF warning) it was based on: https://github.com/chirino/hawtdb http://dhruvbird.com/lfu.pdf Using this project and paper, I will attempt to make a new O(1) cache called FastLFUCache that is modeled on LRUCache.java. This will (for now) leave the existing LFUCache/ConcurrentLFUCache implementation in place. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-6031) TokenSources optimization, avoid sort
[ https://issues.apache.org/jira/browse/LUCENE-6031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Smiley updated LUCENE-6031: - Attachment: LUCENE-6031_lazy_init_at_incrementToken,_and_optimize_payloads.patch The attached patch further improves this by: * delaying lazy-init to first incrementToken() call instead of at reset(). It seems people call addAttributes before and after reset() and so the only way to know if consumers wants payloads is to wait till incrementToken(). ** a consequence of the implementation is that reset() is no longer required, which is how it used to be for this TokenStream. Obviously it *should* be called generally, it's just that this one will still work if you don't. * Payload data is managed in a BytesRefArray which reduces overall memory use and object count (reduces GC burden). TokenSources optimization, avoid sort - Key: LUCENE-6031 URL: https://issues.apache.org/jira/browse/LUCENE-6031 Project: Lucene - Core Issue Type: Improvement Components: modules/highlighter Reporter: David Smiley Assignee: David Smiley Fix For: 5.0, Trunk Attachments: LUCENE-6031.patch, LUCENE-6031.patch, LUCENE-6031_lazy_init_at_incrementToken,_and_optimize_payloads.patch TokenSources.java, in the highlight module, is a facade that returns a TokenStream for a field by either un-inverting converting the TermVector Terms, or by text re-analysis if TermVectors are unavailable or don't have the right options. TokenSources is used by the default highlighter, which is the most accurate highlighter we've got. When documents are large (say hundreds of kilobytes on up), I found that most of the highlighter's activity was up-front spent un-inverting converting the term vector to a TokenStream, not on the actual/real highlighting that follows. Much of that time was on a huge sort of hundreds of thousands of Tokens. Time was also spent doing lots of String conversion and char copying, and it used a lot of memory, too. In this patch, I overhauled TokenStreamFromTermPositionVector.java, and I removed similar logic in TokenSources that was used in circumstances when positions weren't available but offsets were. This class can un-invert term vectors that have positions *and/or* offsets (at least one). It doesn't sort. It places Tokens _directly_ into an array of tokens directly indexed by position. When positions aren't available, the startOffset/8 is a substitute. I've got a more light-weight Token inner class used in place of the former and deprecated Token that ultimately forms a linked-list when the process is done. There is no string conversion; character copying is minimized. The Token array is GC'ed after initialization, it's only needed during construction. Misc: * It implements reset() efficiently so it need not be wrapped in CachingTokenFilter (I'll supply a patch later on this). * It only fetches payloads if you ask for them by adding the attribute (the default highlighter won't add the attribute). * It exposes the underlying TermVector terms via a getter too, which is needed by another patch to follow later. A key assumption is that the position increment gap or first position isn't gigantic, as that will create wasted space and the linked-list formation ultimately has to visit all the slots. We also assume that there aren't a ton of tokens at the same position, since inserting new tokens in sorted order is O(N^2) where 'N' is the average co-occurring token length. My performance testing using Lucene's benchmark module on a megabyte document showed 5x speedup, in conjunction with some other patches to be posted separately. This patch made the most difference. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-6115) Add getMergeInstance to CompressingStoredFieldsReader
[ https://issues.apache.org/jira/browse/LUCENE-6115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adrien Grand updated LUCENE-6115: - Attachment: LUCENE-6115.patch New patch which tests addIndexes. Add getMergeInstance to CompressingStoredFieldsReader - Key: LUCENE-6115 URL: https://issues.apache.org/jira/browse/LUCENE-6115 Project: Lucene - Core Issue Type: Improvement Reporter: Adrien Grand Assignee: Adrien Grand Priority: Minor Attachments: LUCENE-6115.patch, LUCENE-6115.patch CompressingStoredFieldsReader is currently terrible at merging with different codecs or wrapped readers since it does not keep state. So if you want to get 5 documents that come from the same block, it means that you will have to decode the block header and decompress 5 times. It has some optimizations so that if you want to get the 2nd doc of the block then it will stop decompressing soon after the 2nd document, but it doesn't help much with merging since we want all documents. We should implement getMergeInstance and have a different behaviour when merging by decompressing everything up-front and then reusing for all documents of the block. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-5813) Directory should implement Accountable
[ https://issues.apache.org/jira/browse/LUCENE-5813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adrien Grand resolved LUCENE-5813. -- Resolution: Won't Fix Fix Version/s: (was: 4.10) Directory should implement Accountable -- Key: LUCENE-5813 URL: https://issues.apache.org/jira/browse/LUCENE-5813 Project: Lucene - Core Issue Type: Improvement Reporter: Adrien Grand Assignee: Adrien Grand Priority: Minor Attachments: LUCENE-5813.patch Follow-up of LUCENE-5812. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-6107) Add statistics to LRUFilterCache
[ https://issues.apache.org/jira/browse/LUCENE-6107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248329#comment-14248329 ] ASF subversion and git services commented on LUCENE-6107: - Commit 1645958 from [~jpountz] in branch 'dev/trunk' [ https://svn.apache.org/r1645958 ] LUCENE-6107: Add stats to LRUFilterCache. Add statistics to LRUFilterCache Key: LUCENE-6107 URL: https://issues.apache.org/jira/browse/LUCENE-6107 Project: Lucene - Core Issue Type: Improvement Reporter: Adrien Grand Assignee: Adrien Grand Priority: Minor Attachments: LUCENE-6107.patch, LUCENE-6107.patch, LUCENE-6107.patch It would be useful to have statistics about the usage of the filter cache to figure out whether the cache is useful at all and to help tune filter caching policies. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-6107) Add statistics to LRUFilterCache
[ https://issues.apache.org/jira/browse/LUCENE-6107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248333#comment-14248333 ] ASF subversion and git services commented on LUCENE-6107: - Commit 1645961 from [~jpountz] in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1645961 ] LUCENE-6107: Add stats to LRUFilterCache. Add statistics to LRUFilterCache Key: LUCENE-6107 URL: https://issues.apache.org/jira/browse/LUCENE-6107 Project: Lucene - Core Issue Type: Improvement Reporter: Adrien Grand Assignee: Adrien Grand Priority: Minor Attachments: LUCENE-6107.patch, LUCENE-6107.patch, LUCENE-6107.patch It would be useful to have statistics about the usage of the filter cache to figure out whether the cache is useful at all and to help tune filter caching policies. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-6107) Add statistics to LRUFilterCache
[ https://issues.apache.org/jira/browse/LUCENE-6107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adrien Grand resolved LUCENE-6107. -- Resolution: Fixed Fix Version/s: Trunk 5.0 Add statistics to LRUFilterCache Key: LUCENE-6107 URL: https://issues.apache.org/jira/browse/LUCENE-6107 Project: Lucene - Core Issue Type: Improvement Reporter: Adrien Grand Assignee: Adrien Grand Priority: Minor Fix For: 5.0, Trunk Attachments: LUCENE-6107.patch, LUCENE-6107.patch, LUCENE-6107.patch It would be useful to have statistics about the usage of the filter cache to figure out whether the cache is useful at all and to help tune filter caching policies. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4509) Disable HttpClient stale check for performance and fewer spurious connection errors.
[ https://issues.apache.org/jira/browse/SOLR-4509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248342#comment-14248342 ] Mark Miller commented on SOLR-4509: --- I've been pushing this along off and on for a while now. This had a rather large impact on a variety of tests - especially with SSL. I think I'm pretty close though - one more flaky test to investigate I believe. There is a rather interesting affect from this change - the 'close idle' thread will trump the socket timeout. So we are looking at lower socket timeouts or longer idle timeouts or some compromise in between. Disable HttpClient stale check for performance and fewer spurious connection errors. Key: SOLR-4509 URL: https://issues.apache.org/jira/browse/SOLR-4509 Project: Solr Issue Type: Improvement Components: search Environment: 5 node SmartOS cluster (all nodes living in same global zone - i.e. same physical machine) Reporter: Ryan Zezeski Assignee: Mark Miller Priority: Minor Fix For: 5.0, Trunk Attachments: IsStaleTime.java, SOLR-4509-4_4_0.patch, SOLR-4509.patch, SOLR-4509.patch, SOLR-4509.patch, SOLR-4509.patch, SOLR-4509.patch, baremetal-stale-nostale-med-latency.dat, baremetal-stale-nostale-med-latency.svg, baremetal-stale-nostale-throughput.dat, baremetal-stale-nostale-throughput.svg By disabling the Apache HTTP Client stale check I've witnessed a 2-4x increase in throughput and reduction of over 100ms. This patch was made in the context of a project I'm leading, called Yokozuna, which relies on distributed search. Here's the patch on Yokozuna: https://github.com/rzezeski/yokozuna/pull/26 Here's a write-up I did on my findings: http://www.zinascii.com/2013/solr-distributed-search-and-the-stale-check.html I'm happy to answer any questions or make changes to the patch to make it acceptable. ReviewBoard: https://reviews.apache.org/r/28393/ -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-6810) Faster searching limited but high rows across many shards all with many hits
[ https://issues.apache.org/jira/browse/SOLR-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Per Steffensen updated SOLR-6810: - Attachment: branch_5x_rev1645549.patch New patch where the my solution to SOLR-6812 has been removed. Patch now matches revision 1645549 where the final solution to SOLR-6812 is included Faster searching limited but high rows across many shards all with many hits Key: SOLR-6810 URL: https://issues.apache.org/jira/browse/SOLR-6810 Project: Solr Issue Type: Improvement Components: search Reporter: Per Steffensen Assignee: Shalin Shekhar Mangar Labels: distributed_search, performance Attachments: branch_5x_rev1642874.patch, branch_5x_rev1642874.patch, branch_5x_rev1645549.patch Searching limited but high rows across many shards all with many hits is slow E.g. * Query from outside client: q=somethingrows=1000 * Resulting in sub-requests to each shard something a-la this ** 1) q=somethingrows=1000fl=id,score ** 2) Request the full documents with ids in the global-top-1000 found among the top-1000 from each shard What does the subject mean * limited but high rows means 1000 in the example above * many shards means 200-1000 in our case * all with many hits means that each of the shards have a significant number of hits on the query The problem grows on all three factors above Doing such a query on our system takes between 5 min to 1 hour - depending on a lot of things. It ought to be much faster, so lets make it. Profiling show that the problem is that it takes lots of time to access the store to get id’s for (up to) 1000 docs (value of rows parameter) per shard. Having 1000 shards its up to 1 mio ids that has to be fetched. There is really no good reason to ever read information from store for more than the overall top-1000 documents, that has to be returned to the client. For further detail see mail-thread Slow searching limited but high rows across many shards all with high hits started 13/11-2014 on dev@lucene.apache.org -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6813) distrib.singlePass does not work for expand-request - start/rows included
[ https://issues.apache.org/jira/browse/SOLR-6813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248410#comment-14248410 ] Per Steffensen commented on SOLR-6813: -- bq. The distributed deep paging problem will continue to exist But isnt that a performance problem only. Functionality-wise it will work doing deep paging, right? If it is just a performance problem, basically you should not use distrib.singlePass if you are deep paging, I guess we should not disallow using it together with deep paging. It is just a bad idea performance-wise to use distrib.singlePass together with deep paging. distrib.singlePass does not work for expand-request - start/rows included - Key: SOLR-6813 URL: https://issues.apache.org/jira/browse/SOLR-6813 Project: Solr Issue Type: Bug Components: multicore, search Reporter: Per Steffensen Assignee: Joel Bernstein Labels: distributed_search, search Attachments: test_that_reveals_the_problem.patch Using distrib.singlePass does not work for expand-requests. Even after the fix provided to SOLR-6812, it does not work for requests where you add start and/or rows. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6816) Review SolrCloud Indexing Performance.
[ https://issues.apache.org/jira/browse/SOLR-6816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248447#comment-14248447 ] Timothy Potter commented on SOLR-6816: -- bq. Yep. Just like on a single node, if overwrite=false, we could skip version checking. [~ysee...@gmail.com] I'm curious if we shouldn't have a different flag here, something like bulkAdd=true to disable the version lookup to the ulog on the replicas for bulk adds vs. trying to overload the overwrite=false behavior as it seems like setting overwrite=false would by-pass a lot of checking around the unique doc ID. This is how I think it should work: * client sends batch of new docs with bulkAdd=true to leader * leader does versionAdd * leader sends docs with version set to replica(s) and bulkAdd=true param * replica checks bulkAdd and by-passes the lookup to the ulog for that doc I guess what I need to hear is that we think it's safe and a valid approach to allow an indexing client to tell Solr this request contains a bunch of new docs, so optimize for that. Thanks in advance for feedback ;-) Review SolrCloud Indexing Performance. -- Key: SOLR-6816 URL: https://issues.apache.org/jira/browse/SOLR-6816 Project: Solr Issue Type: Task Components: SolrCloud Reporter: Mark Miller Priority: Critical Attachments: SolrBench.pdf We have never really focused on indexing performance, just correctness and low hanging fruit. We need to vet the performance and try to address any holes. Note: A common report is that adding any replication is very slow. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-6640) ChaosMonkeySafeLeaderTest failure with CorruptIndexException
[
https://issues.apache.org/jira/browse/SOLR-6640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Thacker updated SOLR-6640:
Attachment: SOLR-6640.patch
This test passes with the patch - {{ant test
-Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.method=testDistribSearch
-Dtests.seed=EDA082CD42EB33E3 -Dtests.slow=true -Dtests.locale=hi_IN
-Dtests.timezone=Asia/KuchingDtests.asserts=true
-Dtests.file.encoding=ISO-8859-1}}
The patch does something very simple - Before we begin to download segment
files, check against the current commit point which files are extra and remove
them.
For example -
{code}
-001/jetty3/index.20141216162501726
[junit4] 2 22194 T113 C7 P59775 oash.SnapPuller.fetchLatestIndex
SOLR-6640:: indexDir.listAll() pre remove _0.cfe _0.cfs _0.si _0_1.liv _1.fdt
_1.fdx segments_1
[junit4] 2 22195 T79 C6 P59766 oasup.LogUpdateProcessor.finish
[collection1] webapp=/_ path=/update params={wt=javabinversion=2} {add=[0-19
(1487664160941539328)]} 0 1
[junit4] 2 22196 T113 C7 P59775 oash.SnapPuller.fetchLatestIndex
SOLR-6640:: indexDir.listAll() post remove segments_1
{code}
So it's these files which are not getting removed when we do IW.rollback that
were causing the problem -
{{_0.cfe _0.cfs _0.si _0_1.liv _1.fdt _1.fdx}}
I am yet to figure out whether these files should have been removed by
IW.rollback() or not?
ChaosMonkeySafeLeaderTest failure with CorruptIndexException
Key: SOLR-6640
URL: https://issues.apache.org/jira/browse/SOLR-6640
Project: Solr
Issue Type: Bug
Components: replication (java)
Affects Versions: 5.0
Reporter: Shalin Shekhar Mangar
Fix For: 5.0
Attachments: Lucene-Solr-5.x-Linux-64bit-jdk1.8.0_20-Build-11333.txt,
SOLR-6640.patch, SOLR-6640.patch
Test failure found on jenkins:
http://jenkins.thetaphi.de/job/Lucene-Solr-5.x-Linux/11333/
{code}
1 tests failed.
REGRESSION: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
Error Message:
shard2 is not consistent. Got 62 from
http://127.0.0.1:57436/collection1lastClient and got 24 from
http://127.0.0.1:53065/collection1
Stack Trace:
java.lang.AssertionError: shard2 is not consistent. Got 62 from
http://127.0.0.1:57436/collection1lastClient and got 24 from
http://127.0.0.1:53065/collection1
at
__randomizedtesting.SeedInfo.seed([F4B371D421E391CD:7555FFCC56BCF1F1]:0)
at org.junit.Assert.fail(Assert.java:93)
at
org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1255)
at
org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1234)
at
org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:162)
at
org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
{code}
Cause of inconsistency is:
{code}
Caused by: org.apache.lucene.index.CorruptIndexException: file mismatch,
expected segment id=yhq3vokoe1den2av9jbd3yp8, got=yhq3vokoe1den2av9jbd3yp7
(resource=BufferedChecksumIndexInput(MMapIndexInput(path=/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-F4B371D421E391CD-001/tempDir-001/jetty3/index/_1_2.liv)))
[junit4] 2 at
org.apache.lucene.codecs.CodecUtil.checkSegmentHeader(CodecUtil.java:259)
[junit4] 2 at
org.apache.lucene.codecs.lucene50.Lucene50LiveDocsFormat.readLiveDocs(Lucene50LiveDocsFormat.java:88)
[junit4] 2 at
org.apache.lucene.codecs.asserting.AssertingLiveDocsFormat.readLiveDocs(AssertingLiveDocsFormat.java:64)
[junit4] 2 at
org.apache.lucene.index.SegmentReader.init(SegmentReader.java:102)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[JENKINS-MAVEN] Lucene-Solr-Maven-5.x #789: POMs out of sync
Build: https://builds.apache.org/job/Lucene-Solr-Maven-5.x/789/ 4 tests failed. FAILED: org.apache.solr.hadoop.MapReduceIndexerToolArgumentParserTest.org.apache.solr.hadoop.MapReduceIndexerToolArgumentParserTest Error Message: null Stack Trace: java.lang.AssertionError: null at __randomizedtesting.SeedInfo.seed([DF6426CC7C129381]:0) at org.apache.lucene.util.TestRuleTemporaryFilesCleanup.before(TestRuleTemporaryFilesCleanup.java:105) at com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.before(TestRuleAdapter.java:26) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:35) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) at java.lang.Thread.run(Thread.java:745) REGRESSION: org.apache.solr.cloud.TestModifyConfFiles.testDistribSearch Error Message: expected:[Error from server at http://127.0.0.1:44393/collection1: ]No file name specifi... but was:[]No file name specifi... Stack Trace: org.junit.ComparisonFailure: expected:[Error from server at http://127.0.0.1:44393/collection1: ]No file name specifi... but was:[]No file name specifi... at __randomizedtesting.SeedInfo.seed([9B246DFB81BCD32B:1AC2E3E3F6E3B317]:0) at org.junit.Assert.assertEquals(Assert.java:125) at org.junit.Assert.assertEquals(Assert.java:147) at org.apache.solr.cloud.TestModifyConfFiles.doTest(TestModifyConfFiles.java:65) FAILED: org.apache.solr.hadoop.MorphlineBasicMiniMRTest.testPathParts Error Message: Test abandoned because suite timeout was reached. Stack Trace: java.lang.Exception: Test abandoned because suite timeout was reached. at __randomizedtesting.SeedInfo.seed([84BC0E473655606B]:0) FAILED: org.apache.solr.hadoop.MorphlineBasicMiniMRTest.org.apache.solr.hadoop.MorphlineBasicMiniMRTest Error Message: Suite timeout exceeded (= 720 msec). Stack Trace: java.lang.Exception: Suite timeout exceeded (= 720 msec). at __randomizedtesting.SeedInfo.seed([84BC0E473655606B]:0) Build Log: [...truncated 53983 lines...] BUILD FAILED /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Maven-5.x/build.xml:552: The following error occurred while executing this line: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Maven-5.x/build.xml:204: The following error occurred while executing this line: : Java returned: 1 Total time: 394 minutes 34 seconds Build step 'Invoke Ant' marked build as failure Recording test results Email was triggered for: Failure Sending email for trigger: Failure - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (LUCENE-6116) Simplify RoaringDocIdSetConstructor
Adrien Grand created LUCENE-6116: Summary: Simplify RoaringDocIdSetConstructor Key: LUCENE-6116 URL: https://issues.apache.org/jira/browse/LUCENE-6116 Project: Lucene - Core Issue Type: Improvement Reporter: Adrien Grand Assignee: Adrien Grand Priority: Minor The constructor duplicates the logic from firstDocFromNextBlock whichs looks for the next block that contains at least one document. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-6116) Simplify RoaringDocIdSetConstructor
[ https://issues.apache.org/jira/browse/LUCENE-6116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adrien Grand updated LUCENE-6116: - Attachment: LUCENE-6116.patch Simple patch. A good side-effect is that pulling an iterator is now a cheap operation. Simplify RoaringDocIdSetConstructor --- Key: LUCENE-6116 URL: https://issues.apache.org/jira/browse/LUCENE-6116 Project: Lucene - Core Issue Type: Improvement Reporter: Adrien Grand Assignee: Adrien Grand Priority: Minor Attachments: LUCENE-6116.patch The constructor duplicates the logic from firstDocFromNextBlock whichs looks for the next block that contains at least one document. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-6116) Simplify RoaringDocIdSet constructor
[ https://issues.apache.org/jira/browse/LUCENE-6116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adrien Grand updated LUCENE-6116: - Summary: Simplify RoaringDocIdSet constructor (was: Simplify RoaringDocIdSetConstructor) Simplify RoaringDocIdSet constructor Key: LUCENE-6116 URL: https://issues.apache.org/jira/browse/LUCENE-6116 Project: Lucene - Core Issue Type: Improvement Reporter: Adrien Grand Assignee: Adrien Grand Priority: Minor Attachments: LUCENE-6116.patch The constructor duplicates the logic from firstDocFromNextBlock whichs looks for the next block that contains at least one document. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: svn commit: r1645869 - in /lucene/dev/branches/branch_5x: ./ solr/ solr/CHANGES.txt
anshum: there are probably enough people using post.jar in scripts that we should call this out in the top upgrading section of CHANGES.txt for 5.0, with an example of how existing calls need to be modified. : Date: Tue, 16 Dec 2014 07:25:17 - : From: ans...@apache.org : Reply-To: dev@lucene.apache.org : To: comm...@lucene.apache.org : Subject: svn commit: r1645869 - in /lucene/dev/branches/branch_5x: ./ solr/ : solr/CHANGES.txt : : Author: anshum : Date: Tue Dec 16 07:25:16 2014 : New Revision: 1645869 : : URL: http://svn.apache.org/r1645869 : Log: : SOLR-6852: Adding the CHANGES.txt entry (Merging from trunk) : : Modified: : lucene/dev/branches/branch_5x/ (props changed) : lucene/dev/branches/branch_5x/solr/ (props changed) : lucene/dev/branches/branch_5x/solr/CHANGES.txt (contents, props changed) : : Modified: lucene/dev/branches/branch_5x/solr/CHANGES.txt : URL: http://svn.apache.org/viewvc/lucene/dev/branches/branch_5x/solr/CHANGES.txt?rev=1645869r1=1645868r2=1645869view=diff : == : --- lucene/dev/branches/branch_5x/solr/CHANGES.txt (original) : +++ lucene/dev/branches/branch_5x/solr/CHANGES.txt Tue Dec 16 07:25:16 2014 : @@ -453,6 +453,9 @@ Other Changes : * SOLR-6849: HttpSolrServer.RemoteSolrException reports the URL of the remote :host where the exception occurred. (Alan Woodward) : : +* SOLR-6852: SimplePostTool no longer defaults to collection1 making core/collection/update URL : + mandatory. (Anshum Gupta) : + : == 4.10.3 == : : Bug Fixes : : : -Hoss http://www.lucidworks.com/ - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
ANNOUNCE: CFP and Travel Assistance now open for ApacheCon North America 2015
(NOTE: cross posted to several lucene lists, if you have replies, please confine them to general@lucene) -- Forwarded message -- In case you've missed it: - ApacheCon North America returns to Austin, Texas, 13-17 April 2015 http://apachecon.com/ - Call for Papers open until 1 February --submissions and presentation guidelines http://events.linuxfoundation.org/events/apachecon-north-america/program/cfp - Become involved with the program selection process --check out http://s.apache.org/60N - Applications accepted for Apache Travel Assistance --deadline is 6 February! http://www.apache.org/travel/ We look forward to seeing you in Austin! - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[JENKINS] Lucene-Solr-Tests-5.x-Java7 - Build # 2335 - Still Failing
Build: https://builds.apache.org/job/Lucene-Solr-Tests-5.x-Java7/2335/ 1 tests failed. REGRESSION: org.apache.solr.cloud.DistribDocExpirationUpdateProcessorTest.testDistribSearch Error Message: There are still nodes recoverying - waited for 30 seconds Stack Trace: java.lang.AssertionError: There are still nodes recoverying - waited for 30 seconds at __randomizedtesting.SeedInfo.seed([469A7A859AA4A9B9:C77CF49DEDFBC985]:0) at org.junit.Assert.fail(Assert.java:93) at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:178) at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:840) at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForThingsToLevelOut(AbstractFullDistribZkTestBase.java:1459) at org.apache.solr.cloud.DistribDocExpirationUpdateProcessorTest.doTest(DistribDocExpirationUpdateProcessorTest.java:79) at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836) at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738) at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at
[jira] [Commented] (SOLR-6852) SimplePostTool should no longer default to collection1
[ https://issues.apache.org/jira/browse/SOLR-6852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248590#comment-14248590 ] ASF subversion and git services commented on SOLR-6852: --- Commit 1646032 from [~anshumg] in branch 'dev/trunk' [ https://svn.apache.org/r1646032 ] SOLR-6852: Updating the CHANGES.txt entry to the 'Upgrading from..' section SimplePostTool should no longer default to collection1 -- Key: SOLR-6852 URL: https://issues.apache.org/jira/browse/SOLR-6852 Project: Solr Issue Type: Improvement Reporter: Anshum Gupta Assignee: Anshum Gupta Fix For: 5.0 Attachments: SOLR-6852.patch, SOLR-6852.patch Solr no longer would be bootstrapped with collection1 and so it no longer makes sense for the SimplePostTool to default to collection1 either. Without an explicit collection/core/url value, the call should just fail fast. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6852) SimplePostTool should no longer default to collection1
[ https://issues.apache.org/jira/browse/SOLR-6852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248592#comment-14248592 ] ASF subversion and git services commented on SOLR-6852: --- Commit 1646033 from [~anshumg] in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1646033 ] SOLR-6852: Updating the CHANGES.txt entry to the 'Upgrading from..' section (merge from trunk) SimplePostTool should no longer default to collection1 -- Key: SOLR-6852 URL: https://issues.apache.org/jira/browse/SOLR-6852 Project: Solr Issue Type: Improvement Reporter: Anshum Gupta Assignee: Anshum Gupta Fix For: 5.0 Attachments: SOLR-6852.patch, SOLR-6852.patch Solr no longer would be bootstrapped with collection1 and so it no longer makes sense for the SimplePostTool to default to collection1 either. Without an explicit collection/core/url value, the call should just fail fast. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: svn commit: r1645869 - in /lucene/dev/branches/branch_5x: ./ solr/ solr/CHANGES.txt
Thanks for pointing that out Hoss. I've committed that suggestion. On Tue, Dec 16, 2014 at 9:27 AM, Chris Hostetter hossman_luc...@fucit.org wrote: anshum: there are probably enough people using post.jar in scripts that we should call this out in the top upgrading section of CHANGES.txt for 5.0, with an example of how existing calls need to be modified. : Date: Tue, 16 Dec 2014 07:25:17 - : From: ans...@apache.org : Reply-To: dev@lucene.apache.org : To: comm...@lucene.apache.org : Subject: svn commit: r1645869 - in /lucene/dev/branches/branch_5x: ./ solr/ : solr/CHANGES.txt : : Author: anshum : Date: Tue Dec 16 07:25:16 2014 : New Revision: 1645869 : : URL: http://svn.apache.org/r1645869 : Log: : SOLR-6852: Adding the CHANGES.txt entry (Merging from trunk) : : Modified: : lucene/dev/branches/branch_5x/ (props changed) : lucene/dev/branches/branch_5x/solr/ (props changed) : lucene/dev/branches/branch_5x/solr/CHANGES.txt (contents, props changed) : : Modified: lucene/dev/branches/branch_5x/solr/CHANGES.txt : URL: http://svn.apache.org/viewvc/lucene/dev/branches/branch_5x/solr/CHANGES.txt?rev=1645869r1=1645868r2=1645869view=diff : == : --- lucene/dev/branches/branch_5x/solr/CHANGES.txt (original) : +++ lucene/dev/branches/branch_5x/solr/CHANGES.txt Tue Dec 16 07:25:16 2014 : @@ -453,6 +453,9 @@ Other Changes : * SOLR-6849: HttpSolrServer.RemoteSolrException reports the URL of the remote :host where the exception occurred. (Alan Woodward) : : +* SOLR-6852: SimplePostTool no longer defaults to collection1 making core/collection/update URL : + mandatory. (Anshum Gupta) : + : == 4.10.3 == : : Bug Fixes : : : -Hoss http://www.lucidworks.com/ - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org -- Anshum Gupta http://about.me/anshumgupta
[jira] [Commented] (LUCENE-6116) Simplify RoaringDocIdSet constructor
[ https://issues.apache.org/jira/browse/LUCENE-6116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248613#comment-14248613 ] Robert Muir commented on LUCENE-6116: - +1 Simplify RoaringDocIdSet constructor Key: LUCENE-6116 URL: https://issues.apache.org/jira/browse/LUCENE-6116 Project: Lucene - Core Issue Type: Improvement Reporter: Adrien Grand Assignee: Adrien Grand Priority: Minor Attachments: LUCENE-6116.patch The constructor duplicates the logic from firstDocFromNextBlock whichs looks for the next block that contains at least one document. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-6116) Simplify RoaringDocIdSet.Iterator constructor
[ https://issues.apache.org/jira/browse/LUCENE-6116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adrien Grand updated LUCENE-6116: - Summary: Simplify RoaringDocIdSet.Iterator constructor (was: Simplify RoaringDocIdSet constructor) Simplify RoaringDocIdSet.Iterator constructor - Key: LUCENE-6116 URL: https://issues.apache.org/jira/browse/LUCENE-6116 Project: Lucene - Core Issue Type: Improvement Reporter: Adrien Grand Assignee: Adrien Grand Priority: Minor Attachments: LUCENE-6116.patch The constructor duplicates the logic from firstDocFromNextBlock whichs looks for the next block that contains at least one document. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-6116) Simplify RoaringDocIdSet constructor
[ https://issues.apache.org/jira/browse/LUCENE-6116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248664#comment-14248664 ] ASF subversion and git services commented on LUCENE-6116: - Commit 1646041 from [~jpountz] in branch 'dev/trunk' [ https://svn.apache.org/r1646041 ] LUCENE-6116: Simplify RoaringDocIdSet.Iterator constructor. Simplify RoaringDocIdSet constructor Key: LUCENE-6116 URL: https://issues.apache.org/jira/browse/LUCENE-6116 Project: Lucene - Core Issue Type: Improvement Reporter: Adrien Grand Assignee: Adrien Grand Priority: Minor Attachments: LUCENE-6116.patch The constructor duplicates the logic from firstDocFromNextBlock whichs looks for the next block that contains at least one document. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-6116) Simplify RoaringDocIdSet.Iterator constructor
[ https://issues.apache.org/jira/browse/LUCENE-6116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adrien Grand resolved LUCENE-6116. -- Resolution: Fixed Fix Version/s: Trunk 5.0 Simplify RoaringDocIdSet.Iterator constructor - Key: LUCENE-6116 URL: https://issues.apache.org/jira/browse/LUCENE-6116 Project: Lucene - Core Issue Type: Improvement Reporter: Adrien Grand Assignee: Adrien Grand Priority: Minor Fix For: 5.0, Trunk Attachments: LUCENE-6116.patch The constructor duplicates the logic from firstDocFromNextBlock whichs looks for the next block that contains at least one document. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-6116) Simplify RoaringDocIdSet.Iterator constructor
[ https://issues.apache.org/jira/browse/LUCENE-6116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248666#comment-14248666 ] ASF subversion and git services commented on LUCENE-6116: - Commit 1646042 from [~jpountz] in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1646042 ] LUCENE-6116: Simplify RoaringDocIdSet.Iterator constructor. Simplify RoaringDocIdSet.Iterator constructor - Key: LUCENE-6116 URL: https://issues.apache.org/jira/browse/LUCENE-6116 Project: Lucene - Core Issue Type: Improvement Reporter: Adrien Grand Assignee: Adrien Grand Priority: Minor Fix For: 5.0, Trunk Attachments: LUCENE-6116.patch The constructor duplicates the logic from firstDocFromNextBlock whichs looks for the next block that contains at least one document. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-6761) Ability to ignore commit and optimize requests from clients when running in SolrCloud mode.
[ https://issues.apache.org/jira/browse/SOLR-6761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timothy Potter updated SOLR-6761: - Attachment: SOLR-6761.patch Here's a patch that shows the custom UpdateRequestProcessor approach suggested by [~hossman]. The only concern I have is that it needs to be wired into solrconfig.xml. Going with the idea that this feature is for a system administrator, it might make more sense to set this at a more global level, esp. if admins give the ability for other groups to upload their own custom configs and create their own collections. So I'm thinking maybe just a system property (or solr.xml level property) that can be set that affects the DistributedUpateRequestProcessor? Ability to ignore commit and optimize requests from clients when running in SolrCloud mode. --- Key: SOLR-6761 URL: https://issues.apache.org/jira/browse/SOLR-6761 Project: Solr Issue Type: New Feature Components: SolrCloud, SolrJ Reporter: Timothy Potter Attachments: SOLR-6761.patch In most SolrCloud environments, it's advisable to only rely on auto-commits (soft and hard) configured in solrconfig.xml and not send explicit commit requests from client applications. In fact, I've seen cases where improperly coded client applications can send commit requests too frequently, which can lead to harming the cluster's health. As a system administrator, I'd like the ability to disallow commit requests from client applications. Ideally, I could configure the updateHandler to ignore the requests and return an HTTP response code of my choosing as I may not want to break existing client applications by returning an error. In other words, I may want to just return 200 vs. 405. The same goes for optimize requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-6115) Add getMergeInstance to CompressingStoredFieldsReader
[ https://issues.apache.org/jira/browse/LUCENE-6115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248696#comment-14248696 ] Michael McCandless commented on LUCENE-6115: TestDemoParallelLeafReader should get faster with this! (It iterates all stored docs in a segment, to create a parallel index). Add getMergeInstance to CompressingStoredFieldsReader - Key: LUCENE-6115 URL: https://issues.apache.org/jira/browse/LUCENE-6115 Project: Lucene - Core Issue Type: Improvement Reporter: Adrien Grand Assignee: Adrien Grand Priority: Minor Attachments: LUCENE-6115.patch, LUCENE-6115.patch CompressingStoredFieldsReader is currently terrible at merging with different codecs or wrapped readers since it does not keep state. So if you want to get 5 documents that come from the same block, it means that you will have to decode the block header and decompress 5 times. It has some optimizations so that if you want to get the 2nd doc of the block then it will stop decompressing soon after the 2nd document, but it doesn't help much with merging since we want all documents. We should implement getMergeInstance and have a different behaviour when merging by decompressing everything up-front and then reusing for all documents of the block. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Transactions multiplexing IndexWriter
Lucene's IndexWriter only allows one transaction at a time. Fixing
this would be challenging I think.
One workaround might be to let your separate transactions write into
private directories, and then when complete, use
IndexWriter.addIndexes (on the main writer) to fold those changes it.
That part would still be single-transaction, but the addIndexes call
should be faster than doing N separate indexing ops.
Mike McCandless
http://blog.mikemccandless.com
On Tue, Dec 16, 2014 at 8:54 AM, Adam Retter adam.ret...@googlemail.com wrote:
Hey devs,
I am looking at making use of the two-phase commit approach available
in IndexWriter but the current architecture there does not quite fit
with what we want to achieve. It seems that I could build this atop
IndexWriter, however I wonder if there is either an existing
alternative that I have not discovered or whether it would be better
to contribute a patch to the Lucene project itself?
In my system I have many concurrent transactions and each of them
needs to make modifications to a single Lucene index in an atomic and
consistent manner.
If I had a single-thread (i.e. one concurrent transaction) for
example, I could access the IndexWriter instance, call addDocument or
whatever as many times as I like, call prepareForCommit and then
commit.
The main issue that I have is that if I let all concurrent
transactions use the same IndexWriter then I loose isolation, as a
commit of one transaction may write the partial pending updates of
another transaction.
Now I can see a naive solution for my application where I could add
all updates that I want to make to the index to a `pending list`, I
could then take an exclusive lock for the index writer, apply my
pending list to the index writer and then commit, finally releasing
the exclusive lock. Whilst I could get this working, the down-side is
that I have to implement and manage this `pending list` and applying
it to the index myself, and it comes at the cost of memory (or even
paging it to disk).
It seems likely to me that others before me have also had such a
requirement, does anything like this already exist in Lucene or would
it be desirable for me to contribute something? At a rough guess I
would imagine separating the IndexWriter from transaction content,
something like:
try(Transaction txn = indexWriter.beginTransaction()) {
indexWriter.addDocument(txn, doc1);
indexWriter.addDocument(txn, doc2);
indexWriter.addDocument(txn, doc3);
indexWriter.commit(txn);
}
The transaction could be automatically rolled-back in the close()
method called by try-with-resources if it has not been committed,
which would allow any exceptions to be cleanly handled by the caller.
Does that make any sense, or am I way off?
Cheers Adam.
--
Adam Retter
skype: adam.retter
tweet: adamretter
http://www.adamretter.org.uk
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-6113) ReferenceManager.release uses assertion to expect argument not null, also expects argument to be not null
[
https://issues.apache.org/jira/browse/LUCENE-6113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248716#comment-14248716
]
Michael McCandless commented on LUCENE-6113:
I agree this is a hassle, but I think it'd be dangerous to accept null in
release and ignore it? It could mask a real bug (reader ref count leak) in the
application.
We could simple drop the assert: the app will see NPE which should be self
explanatory.
ReferenceManager.release uses assertion to expect argument not null, also
expects argument to be not null
-
Key: LUCENE-6113
URL: https://issues.apache.org/jira/browse/LUCENE-6113
Project: Lucene - Core
Issue Type: Bug
Affects Versions: 4.10.1
Reporter: ryan rawson
A common use pattern for the Reference Manager looks like so:
{code}
IndexSearcher searcher = null;
try {
searcher = searcherManager.acquire();
// do real work
} finally {
searcherManager.release(searcher);
}
{code}
The problem with this code is if 'acquire' throws an exception, the finally
block is called with a null reference for 'searcher'. There are two issues,
one is this call release() uses assertion to check for argument validity,
which is not recommended
(http://docs.oracle.com/javase/8/docs/technotes/guides/language/assert.html)
and secondly to fix this, we need to guard all calls to release with an if
clause.
Why not have release() be a noop if it is passed null, instead of triggering
an NPE? It would support this API usage pattern w/o any changes on the
behalf of users.
Looking at the code, it appears that it is very unlikely that the acquire()
call throws an exception.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-6115) Add getMergeInstance to CompressingStoredFieldsReader
[ https://issues.apache.org/jira/browse/LUCENE-6115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248718#comment-14248718 ] Robert Muir commented on LUCENE-6115: - We need to make it clear how bad the problem is in trunk: if you use a leafreader or are upgrading/changing codec today, with my test data merging is 7x slower merging with LZ4, and 64x slower merging with deflate. This means indexing data that ordinarily takes a minute takes an hour instead. Unfortunately, this patch won't solve the performance issues with FilterReaders at all. The problem is: nobody will ever call getMergeInstance in that case. See LUCENE-6065 for some details (I am not happy with any solution yet, i need help). However, the patch will work in the case of a user upgrading or changing codec, just as long as IW is passed a segmentreader. So its a great step. It is separate from LeafReader's brokenness. Add getMergeInstance to CompressingStoredFieldsReader - Key: LUCENE-6115 URL: https://issues.apache.org/jira/browse/LUCENE-6115 Project: Lucene - Core Issue Type: Improvement Reporter: Adrien Grand Assignee: Adrien Grand Priority: Minor Attachments: LUCENE-6115.patch, LUCENE-6115.patch CompressingStoredFieldsReader is currently terrible at merging with different codecs or wrapped readers since it does not keep state. So if you want to get 5 documents that come from the same block, it means that you will have to decode the block header and decompress 5 times. It has some optimizations so that if you want to get the 2nd doc of the block then it will stop decompressing soon after the 2nd document, but it doesn't help much with merging since we want all documents. We should implement getMergeInstance and have a different behaviour when merging by decompressing everything up-front and then reusing for all documents of the block. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6761) Ability to ignore commit and optimize requests from clients when running in SolrCloud mode.
[
https://issues.apache.org/jira/browse/SOLR-6761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248748#comment-14248748
]
Hoss Man commented on SOLR-6761:
i don't see how it's really any differnet then anything else we expect solr
admins to edit in their solrconfig.xml (like enableStreaming, whether the
/update should have any defaults on it, etc...)
but if you really want it to be sysprop driven, that can still be done via the
enable property...
{code}
processor class=solrIgnoreCommitUpdateProcessorFactory
enable=${solr.ignore.explicit.commit:false}
...
/processor
{code}
Ability to ignore commit and optimize requests from clients when running in
SolrCloud mode.
---
Key: SOLR-6761
URL: https://issues.apache.org/jira/browse/SOLR-6761
Project: Solr
Issue Type: New Feature
Components: SolrCloud, SolrJ
Reporter: Timothy Potter
Attachments: SOLR-6761.patch
In most SolrCloud environments, it's advisable to only rely on auto-commits
(soft and hard) configured in solrconfig.xml and not send explicit commit
requests from client applications. In fact, I've seen cases where improperly
coded client applications can send commit requests too frequently, which can
lead to harming the cluster's health.
As a system administrator, I'd like the ability to disallow commit requests
from client applications. Ideally, I could configure the updateHandler to
ignore the requests and return an HTTP response code of my choosing as I may
not want to break existing client applications by returning an error. In
other words, I may want to just return 200 vs. 405. The same goes for
optimize requests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-6854) Stale cached state in CloudSolrServer
Jessica Cheng Mallet created SOLR-6854:
--
Summary: Stale cached state in CloudSolrServer
Key: SOLR-6854
URL: https://issues.apache.org/jira/browse/SOLR-6854
Project: Solr
Issue Type: Bug
Components: SolrCloud, SolrJ
Reporter: Jessica Cheng Mallet
CloudSolrServer’s cached state is not being updated for a newly created
collection if we started polling for the collection state too early and a
down state is cached. Requests to the newly created collection continues to
fail with No live SolrServers available to handle this request until the
cache is invalidated by time.
Logging on the client side reveals that while the state in ZkStateReader is
updated to active, the cached state in CloudSolrServer remains in down.
{quote}
CloudSolrServer cached state:
DocCollection(collection-1418250319268)={
shards:{shard1:{
range:8000-7fff,
state:active,
replicas:{core_node1:{
state:down,
base_url:http://localhost:8983/solr;,
core:collection-1418250319268_shard1_replica1,
node_name:localhost:8983_solr,
maxShardsPerNode:1,
external:true,
router:{name:compositeId},
replicationFactor:1”}
ZkStateReader state:
DocCollection(collection-1418250319268)={
shards:{shard1:{
range:8000-7fff,
state:active,
replicas:{core_node1:{
state:active,
base_url:http://localhost:8983/solr;,
core:collection-1418250319268_shard1_replica1,
node_name:localhost:8983_solr,
leader:true,
maxShardsPerNode:1,
router:{name:compositeId},
external:true,
replicationFactor:1”}
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-6854) Stale cached state in CloudSolrServer
[
https://issues.apache.org/jira/browse/SOLR-6854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jessica Cheng Mallet updated SOLR-6854:
---
Description:
CloudSolrServer’s cached state is not being updated for a newly created
collection if we started polling for the collection state too early and a
down state is cached. Requests to the newly created collection continues to
fail with No live SolrServers available to handle this request until the
cache is invalidated by time.
Logging on the client side reveals that while the state in ZkStateReader is
updated to active, the cached state in CloudSolrServer remains in down.
{quote}
CloudSolrServer cached state:
DocCollection(collection-1418250319268)={
shards:{shard1:{
range:8000-7fff,
state:active,
replicas:{core_node1:{
state:down,
base_url:http://localhost:8983/solr;,
core:collection-1418250319268_shard1_replica1,
node_name:localhost:8983_solr,
maxShardsPerNode:1,
external:true,
router:{ name:compositeId},
replicationFactor:1”}
ZkStateReader state:
DocCollection(collection-1418250319268)={
shards:{shard1:{
range:8000-7fff,
state:active,
replicas:{core_node1:{
state:active,
base_url:http://localhost:8983/solr;,
core:collection-1418250319268_shard1_replica1,
node_name:localhost:8983_solr,
leader:true,
maxShardsPerNode:1,
router:{ name:compositeId},
external:true,
replicationFactor:1”}
{quote}
was:
CloudSolrServer’s cached state is not being updated for a newly created
collection if we started polling for the collection state too early and a
down state is cached. Requests to the newly created collection continues to
fail with No live SolrServers available to handle this request until the
cache is invalidated by time.
Logging on the client side reveals that while the state in ZkStateReader is
updated to active, the cached state in CloudSolrServer remains in down.
{quote}
CloudSolrServer cached state:
DocCollection(collection-1418250319268)={
shards:{shard1:{
range:8000-7fff,
state:active,
replicas:{core_node1:{
state:down,
base_url:http://localhost:8983/solr;,
core:collection-1418250319268_shard1_replica1,
node_name:localhost:8983_solr,
maxShardsPerNode:1,
external:true,
router:{name:compositeId},
replicationFactor:1”}
ZkStateReader state:
DocCollection(collection-1418250319268)={
shards:{shard1:{
range:8000-7fff,
state:active,
replicas:{core_node1:{
state:active,
base_url:http://localhost:8983/solr;,
core:collection-1418250319268_shard1_replica1,
node_name:localhost:8983_solr,
leader:true,
maxShardsPerNode:1,
router:{name:compositeId},
external:true,
replicationFactor:1”}
{quote}
Stale cached state in CloudSolrServer
-
Key: SOLR-6854
URL: https://issues.apache.org/jira/browse/SOLR-6854
Project: Solr
Issue Type: Bug
Components: SolrCloud, SolrJ
Reporter: Jessica Cheng Mallet
Labels: cache, solrcloud, solrj
CloudSolrServer’s cached state is not being updated for a newly created
collection if we started polling for the collection state too early and a
down state is cached. Requests to the newly created collection continues to
fail with No live SolrServers available to handle this request until the
cache is invalidated by time.
Logging on the client side reveals that while the state in ZkStateReader is
updated to active, the cached state in CloudSolrServer remains in down.
{quote}
CloudSolrServer cached state:
DocCollection(collection-1418250319268)={
shards:{shard1:{
range:8000-7fff,
state:active,
replicas:{core_node1:{
state:down,
base_url:http://localhost:8983/solr;,
core:collection-1418250319268_shard1_replica1,
node_name:localhost:8983_solr,
maxShardsPerNode:1,
external:true,
router:{ name:compositeId},
replicationFactor:1”}
ZkStateReader state:
DocCollection(collection-1418250319268)={
shards:{shard1:{
range:8000-7fff,
state:active,
replicas:{core_node1:{
state:active,
base_url:http://localhost:8983/solr;,
core:collection-1418250319268_shard1_replica1,
node_name:localhost:8983_solr,
leader:true,
maxShardsPerNode:1,
router:{ name:compositeId},
external:true,
replicationFactor:1”}
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional
[JENKINS] Lucene-Solr-5.x-MacOSX (64bit/jdk1.7.0) - Build # 1954 - Still Failing!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-5.x-MacOSX/1954/ Java: 64bit/jdk1.7.0 -XX:+UseCompressedOops -XX:+UseSerialGC (asserts: true) 1 tests failed. FAILED: org.apache.solr.handler.dataimport.TestSolrEntityProcessorEndToEnd.testFullImportFqParam Error Message: Timeout occured while waiting response from server at: https://127.0.0.1:61993/solr Stack Trace: java.lang.AssertionError: Timeout occured while waiting response from server at: https://127.0.0.1:61993/solr at __randomizedtesting.SeedInfo.seed([9C3A0A47AA3E9DE3:6D71B4B43A7EEAD6]:0) at org.junit.Assert.fail(Assert.java:93) at org.apache.solr.handler.dataimport.TestSolrEntityProcessorEndToEnd.testFullImportFqParam(TestSolrEntityProcessorEndToEnd.java:169) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836) at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738) at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) at java.lang.Thread.run(Thread.java:745)
[jira] [Comment Edited] (SOLR-6761) Ability to ignore commit and optimize requests from clients when running in SolrCloud mode.
[ https://issues.apache.org/jira/browse/SOLR-6761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248811#comment-14248811 ] Erick Erickson edited comment on SOLR-6761 at 12/16/14 8:01 PM: Does Noble's work with an API to alter solrconfig.xml apply at all here? was (Author: erickerickson): Does Noble's work with an API to alter solrconfig.xml applhy at all here? Ability to ignore commit and optimize requests from clients when running in SolrCloud mode. --- Key: SOLR-6761 URL: https://issues.apache.org/jira/browse/SOLR-6761 Project: Solr Issue Type: New Feature Components: SolrCloud, SolrJ Reporter: Timothy Potter Attachments: SOLR-6761.patch In most SolrCloud environments, it's advisable to only rely on auto-commits (soft and hard) configured in solrconfig.xml and not send explicit commit requests from client applications. In fact, I've seen cases where improperly coded client applications can send commit requests too frequently, which can lead to harming the cluster's health. As a system administrator, I'd like the ability to disallow commit requests from client applications. Ideally, I could configure the updateHandler to ignore the requests and return an HTTP response code of my choosing as I may not want to break existing client applications by returning an error. In other words, I may want to just return 200 vs. 405. The same goes for optimize requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6761) Ability to ignore commit and optimize requests from clients when running in SolrCloud mode.
[ https://issues.apache.org/jira/browse/SOLR-6761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248811#comment-14248811 ] Erick Erickson commented on SOLR-6761: -- Does Noble's work with an API to alter solrconfig.xml applhy at all here? Ability to ignore commit and optimize requests from clients when running in SolrCloud mode. --- Key: SOLR-6761 URL: https://issues.apache.org/jira/browse/SOLR-6761 Project: Solr Issue Type: New Feature Components: SolrCloud, SolrJ Reporter: Timothy Potter Attachments: SOLR-6761.patch In most SolrCloud environments, it's advisable to only rely on auto-commits (soft and hard) configured in solrconfig.xml and not send explicit commit requests from client applications. In fact, I've seen cases where improperly coded client applications can send commit requests too frequently, which can lead to harming the cluster's health. As a system administrator, I'd like the ability to disallow commit requests from client applications. Ideally, I could configure the updateHandler to ignore the requests and return an HTTP response code of my choosing as I may not want to break existing client applications by returning an error. In other words, I may want to just return 200 vs. 405. The same goes for optimize requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-6113) ReferenceManager.release uses assertion to expect argument not null, also expects argument to be not null
[
https://issues.apache.org/jira/browse/LUCENE-6113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248848#comment-14248848
]
ryan rawson commented on LUCENE-6113:
-
most people should be seeing the NPE because assertions are not always enabled.
The bigger question to me is API design. as it stands, people MUST guard their
call in to this method call since there is no way to ensure that it will always
be not-null (since the place you get one may fail).
As an illustrative example, in the Cocoa API, the 'release' refcount calls will
do nothing to a null reference for similar reasons (dont spread null checking
everywhere else in the code).
So finally, what's the code scenario whereby a 'reader ref count leak' could be
made by doing the ignore a null release? The app code already has to make that
if guard, so why not absorb it in to the API?
ReferenceManager.release uses assertion to expect argument not null, also
expects argument to be not null
-
Key: LUCENE-6113
URL: https://issues.apache.org/jira/browse/LUCENE-6113
Project: Lucene - Core
Issue Type: Bug
Affects Versions: 4.10.1
Reporter: ryan rawson
A common use pattern for the Reference Manager looks like so:
{code}
IndexSearcher searcher = null;
try {
searcher = searcherManager.acquire();
// do real work
} finally {
searcherManager.release(searcher);
}
{code}
The problem with this code is if 'acquire' throws an exception, the finally
block is called with a null reference for 'searcher'. There are two issues,
one is this call release() uses assertion to check for argument validity,
which is not recommended
(http://docs.oracle.com/javase/8/docs/technotes/guides/language/assert.html)
and secondly to fix this, we need to guard all calls to release with an if
clause.
Why not have release() be a noop if it is passed null, instead of triggering
an NPE? It would support this API usage pattern w/o any changes on the
behalf of users.
Looking at the code, it appears that it is very unlikely that the acquire()
call throws an exception.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
Re: svn commit: r1646083 - in /lucene/dev/trunk: lucene/suggest/src/java/org/apache/lucene/search/suggest/analyzing/AnalyzingInfixSuggester.java solr/server/start.jar
Shai,
start.jar is included in solr/server/’s svn:ignore list - I’m assuming you
accidentally committed it.
Steve
On Dec 16, 2014, at 4:21 PM, sh...@apache.org wrote:
Author: shaie
Date: Tue Dec 16 21:21:19 2014
New Revision: 1646083
URL: http://svn.apache.org/r1646083
Log:
Fix typo
Added:
lucene/dev/trunk/solr/server/start.jar
Modified:
lucene/dev/trunk/lucene/suggest/src/java/org/apache/lucene/search/suggest/analyzing/AnalyzingInfixSuggester.java
Modified:
lucene/dev/trunk/lucene/suggest/src/java/org/apache/lucene/search/suggest/analyzing/AnalyzingInfixSuggester.java
URL:
http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/suggest/src/java/org/apache/lucene/search/suggest/analyzing/AnalyzingInfixSuggester.java?rev=1646083r1=1646082r2=1646083view=diff
==
---
lucene/dev/trunk/lucene/suggest/src/java/org/apache/lucene/search/suggest/analyzing/AnalyzingInfixSuggester.java
(original)
+++
lucene/dev/trunk/lucene/suggest/src/java/org/apache/lucene/search/suggest/analyzing/AnalyzingInfixSuggester.java
Tue Dec 16 21:21:19 2014
@@ -308,7 +308,7 @@ public class AnalyzingInfixSuggester ext
/** Updates a previous suggestion, matching the exact same
* text as before. Use this to change the weight or
- * payload of an already added suggstion. If you know
+ * payload of an already added suggestion. If you know
* this text is not already present you can use {@link
* #add} instead. After adding or updating a batch of
* new suggestions, you must call {@link #refresh} in the
Added: lucene/dev/trunk/solr/server/start.jar
URL:
http://svn.apache.org/viewvc/lucene/dev/trunk/solr/server/start.jar?rev=1646083view=auto
==
Binary files lucene/dev/trunk/solr/server/start.jar (added) and
lucene/dev/trunk/solr/server/start.jar Tue Dec 16 21:21:19 2014 differ
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-6113) ReferenceManager.release uses assertion to expect argument not null, also expects argument to be not null
[
https://issues.apache.org/jira/browse/LUCENE-6113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14249018#comment-14249018
]
Michael McCandless commented on LUCENE-6113:
OK I think I agree; such a code case that would fall into this trap is ... poor
code anyway, and we shouldn't design our APIs on the abusers.
I wish we could somehow use AutoCloseable here so you could just use
try-with-resources.
ReferenceManager.release uses assertion to expect argument not null, also
expects argument to be not null
-
Key: LUCENE-6113
URL: https://issues.apache.org/jira/browse/LUCENE-6113
Project: Lucene - Core
Issue Type: Bug
Affects Versions: 4.10.1
Reporter: ryan rawson
A common use pattern for the Reference Manager looks like so:
{code}
IndexSearcher searcher = null;
try {
searcher = searcherManager.acquire();
// do real work
} finally {
searcherManager.release(searcher);
}
{code}
The problem with this code is if 'acquire' throws an exception, the finally
block is called with a null reference for 'searcher'. There are two issues,
one is this call release() uses assertion to check for argument validity,
which is not recommended
(http://docs.oracle.com/javase/8/docs/technotes/guides/language/assert.html)
and secondly to fix this, we need to guard all calls to release with an if
clause.
Why not have release() be a noop if it is passed null, instead of triggering
an NPE? It would support this API usage pattern w/o any changes on the
behalf of users.
Looking at the code, it appears that it is very unlikely that the acquire()
call throws an exception.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-6113) ReferenceManager.release uses assertion to expect argument not null, also expects argument to be not null
[
https://issues.apache.org/jira/browse/LUCENE-6113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14249030#comment-14249030
]
Shai Erera commented on LUCENE-6113:
Why do you call searcher.acquire() inside the try? I usually do it like that:
IndexSearcher searcher = manager.acquire();
try {
// do some work
} finally {
manager.release(searcher);
}
The jdocs of SearcherManager also suggest to do the same thing.
if you're calling it inside the try, you should definitely check that it's not
null before attempting to pass it to SM. That's the problem of calling it
inside the try. If you call it outside and it fails, there's nothing to release.
ReferenceManager.release uses assertion to expect argument not null, also
expects argument to be not null
-
Key: LUCENE-6113
URL: https://issues.apache.org/jira/browse/LUCENE-6113
Project: Lucene - Core
Issue Type: Bug
Affects Versions: 4.10.1
Reporter: ryan rawson
A common use pattern for the Reference Manager looks like so:
{code}
IndexSearcher searcher = null;
try {
searcher = searcherManager.acquire();
// do real work
} finally {
searcherManager.release(searcher);
}
{code}
The problem with this code is if 'acquire' throws an exception, the finally
block is called with a null reference for 'searcher'. There are two issues,
one is this call release() uses assertion to check for argument validity,
which is not recommended
(http://docs.oracle.com/javase/8/docs/technotes/guides/language/assert.html)
and secondly to fix this, we need to guard all calls to release with an if
clause.
Why not have release() be a noop if it is passed null, instead of triggering
an NPE? It would support this API usage pattern w/o any changes on the
behalf of users.
Looking at the code, it appears that it is very unlikely that the acquire()
call throws an exception.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-6113) ReferenceManager.release uses assertion to expect argument not null, also expects argument to be not null
[
https://issues.apache.org/jira/browse/LUCENE-6113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14249030#comment-14249030
]
Shai Erera edited comment on LUCENE-6113 at 12/16/14 9:54 PM:
--
Why do you call manager.acquire() inside the try? I usually do it like that:
{code}
IndexSearcher searcher = manager.acquire();
try {
// do some work
} finally {
manager.release(searcher);
}
{code}
The jdocs of SearcherManager also suggest to do the same thing.
if you're calling it inside the try, you should definitely check that it's not
null before attempting to pass it to SM. That's the problem of calling it
inside the try. If you call it outside and it fails, there's nothing to release.
was (Author: shaie):
Why do you call searcher.acquire() inside the try? I usually do it like that:
IndexSearcher searcher = manager.acquire();
try {
// do some work
} finally {
manager.release(searcher);
}
The jdocs of SearcherManager also suggest to do the same thing.
if you're calling it inside the try, you should definitely check that it's not
null before attempting to pass it to SM. That's the problem of calling it
inside the try. If you call it outside and it fails, there's nothing to release.
ReferenceManager.release uses assertion to expect argument not null, also
expects argument to be not null
-
Key: LUCENE-6113
URL: https://issues.apache.org/jira/browse/LUCENE-6113
Project: Lucene - Core
Issue Type: Bug
Affects Versions: 4.10.1
Reporter: ryan rawson
A common use pattern for the Reference Manager looks like so:
{code}
IndexSearcher searcher = null;
try {
searcher = searcherManager.acquire();
// do real work
} finally {
searcherManager.release(searcher);
}
{code}
The problem with this code is if 'acquire' throws an exception, the finally
block is called with a null reference for 'searcher'. There are two issues,
one is this call release() uses assertion to check for argument validity,
which is not recommended
(http://docs.oracle.com/javase/8/docs/technotes/guides/language/assert.html)
and secondly to fix this, we need to guard all calls to release with an if
clause.
Why not have release() be a noop if it is passed null, instead of triggering
an NPE? It would support this API usage pattern w/o any changes on the
behalf of users.
Looking at the code, it appears that it is very unlikely that the acquire()
call throws an exception.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
Re: [CI] Lucene Linux Java 8 64 Test Only - Build # 18333 - Failure!
This is a scary failure: an assert trip deep inside IndexWriter. I haven't seen this before. It was G1GC ... could be related. Mike McCandless http://blog.mikemccandless.com On Tue, Dec 16, 2014 at 4:31 PM, bu...@elasticsearch.com wrote: *BUILD FAILURE* Build URL http://build-eu-00.elasticsearch.org/job/lucene_trunk_linux_java8_64_test_only/18333/ Project:lucene_trunk_linux_java8_64_test_only Date of build:Tue, 16 Dec 2014 22:29:11 +0100 Build duration:2 min 22 sec *CHANGES* No Changes *BUILD ARTIFACTS* checkout/lucene/build/core/test/temp/junit4-J0-20141216_222918_539.events http://build-eu-00.elasticsearch.org/job/lucene_trunk_linux_java8_64_test_only/18333/artifact/checkout/lucene/build/core/test/temp/junit4-J0-20141216_222918_539.events checkout/lucene/build/core/test/temp/junit4-J1-20141216_222918_539.events http://build-eu-00.elasticsearch.org/job/lucene_trunk_linux_java8_64_test_only/18333/artifact/checkout/lucene/build/core/test/temp/junit4-J1-20141216_222918_539.events checkout/lucene/build/core/test/temp/junit4-J2-20141216_222918_539.events http://build-eu-00.elasticsearch.org/job/lucene_trunk_linux_java8_64_test_only/18333/artifact/checkout/lucene/build/core/test/temp/junit4-J2-20141216_222918_539.events checkout/lucene/build/core/test/temp/junit4-J3-20141216_222918_539.events http://build-eu-00.elasticsearch.org/job/lucene_trunk_linux_java8_64_test_only/18333/artifact/checkout/lucene/build/core/test/temp/junit4-J3-20141216_222918_539.events checkout/lucene/build/core/test/temp/junit4-J4-20141216_222918_539.events http://build-eu-00.elasticsearch.org/job/lucene_trunk_linux_java8_64_test_only/18333/artifact/checkout/lucene/build/core/test/temp/junit4-J4-20141216_222918_539.events checkout/lucene/build/core/test/temp/junit4-J5-20141216_222918_539.events http://build-eu-00.elasticsearch.org/job/lucene_trunk_linux_java8_64_test_only/18333/artifact/checkout/lucene/build/core/test/temp/junit4-J5-20141216_222918_539.events checkout/lucene/build/core/test/temp/junit4-J6-20141216_222918_550.events http://build-eu-00.elasticsearch.org/job/lucene_trunk_linux_java8_64_test_only/18333/artifact/checkout/lucene/build/core/test/temp/junit4-J6-20141216_222918_550.events checkout/lucene/build/core/test/temp/junit4-J7-20141216_222918_550.events http://build-eu-00.elasticsearch.org/job/lucene_trunk_linux_java8_64_test_only/18333/artifact/checkout/lucene/build/core/test/temp/junit4-J7-20141216_222918_550.events *FAILED JUNIT TESTS* Name: org.apache.lucene.index Failed: 1 test(s), Passed: 732 test(s), Skipped: 19 test(s), Total: 752 test(s) *Failed: org.apache.lucene.index.TestFlushByRamOrCountsPolicy.testStallControl * *CONSOLE OUTPUT* [...truncated 1560 lines...] [junit4] [junit4] Tests with failures: [junit4] - org.apache.lucene.index.TestFlushByRamOrCountsPolicy.testStallControl [junit4] [junit4] [junit4] JVM J0: 1.03 .. 125.53 = 124.50s [junit4] JVM J1: 1.01 .. 128.07 = 127.06s [junit4] JVM J2: 1.01 .. 125.22 = 124.20s [junit4] JVM J3: 1.01 .. 124.95 = 123.94s [junit4] JVM J4: 1.25 .. 125.78 = 124.54s [junit4] JVM J5: 1.03 .. 125.43 = 124.40s [junit4] JVM J6: 1.01 .. 128.55 = 127.54s [junit4] JVM J7: 1.01 .. 124.32 = 123.31s [junit4] Execution time total: 2 minutes 8 seconds [junit4] Tests summary: 407 suites, 3095 tests, 1 error, 60 ignored (50 assumptions) BUILD FAILED /home/jenkins/workspace/lucene_trunk_linux_java8_64_test_only/checkout/lucene/build.xml:49: The following error occurred while executing this line: /home/jenkins/workspace/lucene_trunk_linux_java8_64_test_only/checkout/lucene/common-build.xml:1349: The following error occurred while executing this line: /home/jenkins/workspace/lucene_trunk_linux_java8_64_test_only/checkout/lucene/common-build.xml:956: There were test failures: 407 suites, 3095 tests, 1 error, 60 ignored (50 assumptions) Total time: 2 minutes 16 seconds Build step 'Invoke Ant' marked build as failure Archiving artifacts Recording test results Email was triggered for: Failure - 1st Trigger Failure - Any was overridden by another trigger and will not send an email. Trigger Failure - Still was overridden by another trigger and will not send an email. Sending email for trigger: Failure - 1st
[jira] [Created] (SOLR-6855) bin/solr -e dih launches, but has some path cruft issues preventing some of the imports don't work
Hoss Man created SOLR-6855:
--
Summary: bin/solr -e dih launches, but has some path cruft issues
preventing some of the imports don't work
Key: SOLR-6855
URL: https://issues.apache.org/jira/browse/SOLR-6855
Project: Solr
Issue Type: Bug
Reporter: Hoss Man
Priority: Blocker
While trying to update this ref guide page...
https://cwiki.apache.org/confluence/display/solr/Uploading+Structured+Data+Store+Data+with+the+Data+Import+Handler
I encountered a bunch of problems when running {{bin/solr -e dih}}:
# every collection in {{example/example-DIH/solr}} started getting
\_rest_managed\* and \_schema_analysis\* files created in it
#* either we should commit these empty files into the example, or pair down the
schema's in these collections not to use these fieldTypes
# a {{server/example-DIH}} directory got created containing some hsqldb logs
properties
# at least 2 of the full import commands don't seem to work
#* the DB probably isn't working because the path to the hsqldb may not be
correct anymore - hence the problem above as well (JDBC probably relative to
CWD at the moment? need sys prop to be relative to solr home?)
#* the tika import doesn't seem to work either - probably another relative path
problem
# the {{example/example-DIH/README.txt}} file still needs updated to refer to
{{bin/solr -e dih}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-6855) bin/solr -e dih launches, but has some path cruft issues preventing some of the imports don't work
[
https://issues.apache.org/jira/browse/SOLR-6855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hoss Man updated SOLR-6855:
---
Fix Version/s: 5.0
bin/solr -e dih launches, but has some path cruft issues preventing some of
the imports don't work
--
Key: SOLR-6855
URL: https://issues.apache.org/jira/browse/SOLR-6855
Project: Solr
Issue Type: Bug
Reporter: Hoss Man
Priority: Blocker
Fix For: 5.0
While trying to update this ref guide page...
https://cwiki.apache.org/confluence/display/solr/Uploading+Structured+Data+Store+Data+with+the+Data+Import+Handler
I encountered a bunch of problems when running {{bin/solr -e dih}}:
# every collection in {{example/example-DIH/solr}} started getting
\_rest_managed\* and \_schema_analysis\* files created in it
#* either we should commit these empty files into the example, or pair down
the schema's in these collections not to use these fieldTypes
# a {{server/example-DIH}} directory got created containing some hsqldb logs
properties
# at least 2 of the full import commands don't seem to work
#* the DB probably isn't working because the path to the hsqldb may not be
correct anymore - hence the problem above as well (JDBC probably relative to
CWD at the moment? need sys prop to be relative to solr home?)
#* the tika import doesn't seem to work either - probably another relative
path problem
# the {{example/example-DIH/README.txt}} file still needs updated to refer to
{{bin/solr -e dih}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-6856) regression in /update/extract ? ref guide examples of fmap xpath don't seem to be working
Hoss Man created SOLR-6856:
--
Summary: regression in /update/extract ? ref guide examples of
fmap xpath don't seem to be working
Key: SOLR-6856
URL: https://issues.apache.org/jira/browse/SOLR-6856
Project: Solr
Issue Type: Bug
Affects Versions: 5.0
Reporter: Hoss Man
Priority: Blocker
I updated this page to know about hte new bin/solr and example/exampledocs
structure/contents...
https://cwiki.apache.org/confluence/display/solr/Uploading+Data+with+Solr+Cell+using+Apache+Tika
however i noticed that several of the examples listed on that page didn't seem
to work any more -- notably...
* examples using fmap don't seem to create the fields they say they will
* examples using xpath don't seem to create any docs at all
Specific examples i had problems with...
{noformat}
curl
http://localhost:8983/solr/techproducts/update/extract?literal.id=doc2captureAttr=truedefaultField=textfmap.div=foo_tcapture=divcommit=true;
-F sample=@example/exampledocs/sample.html
curl
http://localhost:8983/solr/techproducts/update/extract?literal.id=doc3captureAttr=truedefaultField=textcapture=divfmap.div=foo_tboost.foo_t=3commit=true;
-F sample=@example/exampledocs/sample.html
curl
http://localhost:8983/solr/techproducts/update/extract?literal.id=doc4captureAttr=truedefaultField=textcapture=divfmap.div=foo_tboost.foo_t=3literal.blah_s=Bahcommit=true;
-F sample=@example/exampledocs/sample.html
curl
http://localhost:8983/solr/techproducts/update/extract?literal.id=doc5captureAttr=truedefaultField=textcapture=divfmap.div=foo_tboost.foo_t=3literal.id=idxpath=/xhtml:html/xhtml:body/xhtml:div/descendant:node()commit=true
-F sample=@example/exampledocs/sample.html
{noformat}
...none of these example commands produced an error, but they also didn't seem
to create the fields/docs they said they would (ie: no foo_t field was
created)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6852) SimplePostTool should no longer default to collection1
[ https://issues.apache.org/jira/browse/SOLR-6852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14249251#comment-14249251 ] Alexandre Rafalovitch commented on SOLR-6852: - What about post.sh? That's got the URL hard-coded in as far as I can tell. SimplePostTool should no longer default to collection1 -- Key: SOLR-6852 URL: https://issues.apache.org/jira/browse/SOLR-6852 Project: Solr Issue Type: Improvement Reporter: Anshum Gupta Assignee: Anshum Gupta Fix For: 5.0 Attachments: SOLR-6852.patch, SOLR-6852.patch Solr no longer would be bootstrapped with collection1 and so it no longer makes sense for the SimplePostTool to default to collection1 either. Without an explicit collection/core/url value, the call should just fail fast. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (LUCENE-6117) infostream is currently unusable out of box
Robert Muir created LUCENE-6117:
---
Summary: infostream is currently unusable out of box
Key: LUCENE-6117
URL: https://issues.apache.org/jira/browse/LUCENE-6117
Project: Lucene - Core
Issue Type: Bug
Reporter: Robert Muir
testpoints used to only be emitted by assertions (still sketchy), but now are
emitted always. I assume this is due to the change to support running tests
with assertions disabled.
we should try to clean this up, simple stuff like this is now useless:
{code}
indexWriterConfig.setInfoStream(System.out);
// causes massive flooding like this:
// TP 0 [Tue Dec 16 20:19:37 EST 2014; Thread-0]: DocumentsWriterPerThread
addDocument start
// TP 0 [Tue Dec 16 20:19:37 EST 2014; Thread-0]: DocumentsWriterPerThread
addDocument start
// TP 0 [Tue Dec 16 20:19:37 EST 2014; Thread-0]: DocumentsWriterPerThread
addDocument start
{code}
I hit this several times today just trying to do benchmarks and debugging.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6852) SimplePostTool should no longer default to collection1
[ https://issues.apache.org/jira/browse/SOLR-6852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14249295#comment-14249295 ] Anshum Gupta commented on SOLR-6852: I'll create another JIRA to perhaps just (re)move post.sh as it doesn't accept anything but a list of files. We need a bin/post script that does more than what example/exampledocs/post.sh does. SimplePostTool should no longer default to collection1 -- Key: SOLR-6852 URL: https://issues.apache.org/jira/browse/SOLR-6852 Project: Solr Issue Type: Improvement Reporter: Anshum Gupta Assignee: Anshum Gupta Fix For: 5.0 Attachments: SOLR-6852.patch, SOLR-6852.patch Solr no longer would be bootstrapped with collection1 and so it no longer makes sense for the SimplePostTool to default to collection1 either. Without an explicit collection/core/url value, the call should just fail fast. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6852) SimplePostTool should no longer default to collection1
[ https://issues.apache.org/jira/browse/SOLR-6852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14249298#comment-14249298 ] Anshum Gupta commented on SOLR-6852: Also, post.sh doesn't do anything using the SimplePostTool and just uses curl and default URL to post files. SOLR-6435 should be a replacement for this but anyways, it's a different issue. SimplePostTool should no longer default to collection1 -- Key: SOLR-6852 URL: https://issues.apache.org/jira/browse/SOLR-6852 Project: Solr Issue Type: Improvement Reporter: Anshum Gupta Assignee: Anshum Gupta Fix For: 5.0 Attachments: SOLR-6852.patch, SOLR-6852.patch Solr no longer would be bootstrapped with collection1 and so it no longer makes sense for the SimplePostTool to default to collection1 either. Without an explicit collection/core/url value, the call should just fail fast. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-2878) Allow Scorer to expose positions and payloads aka. nuke spans
[
https://issues.apache.org/jira/browse/LUCENE-2878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14249360#comment-14249360
]
Robert Muir commented on LUCENE-2878:
-
I ran the benchmark:
{noformat}
Task QPS trunk StdDev QPS patch StdDev
Pct diff
HighSloppyPhrase 13.40 (10.4%) 10.31 (5.5%)
-23.1% ( -35% - -7%)
HighPhrase 17.98 (5.6%) 13.92 (3.1%)
-22.6% ( -29% - -14%)
MedPhrase 257.86 (7.1%) 213.38 (3.6%)
-17.2% ( -26% - -7%)
LowPhrase 35.68 (1.8%) 33.32 (1.7%)
-6.6% ( -9% - -3%)
MedSloppyPhrase 15.79 (4.2%) 14.92 (3.6%)
-5.5% ( -12% -2%)
LowSloppyPhrase 118.09 (2.4%) 112.14 (2.0%)
-5.0% ( -9% -0%)
HighTerm 138.18 (10.2%) 136.72 (6.7%)
-1.1% ( -16% - 17%)
MedTerm 202.67 (9.6%) 200.94 (6.3%)
-0.9% ( -15% - 16%)
HighSpanNear 144.67 (4.3%) 144.35 (4.3%)
-0.2% ( -8% -8%)
MedSpanNear 143.52 (3.9%) 143.30 (4.0%)
-0.2% ( -7% -8%)
Respell 85.33 (1.8%) 85.32 (2.6%)
-0.0% ( -4% -4%)
LowTerm 1052.81 (8.5%) 1053.59 (5.3%)
0.1% ( -12% - 15%)
LowSpanNear 27.81 (2.9%) 27.83 (2.9%)
0.1% ( -5% -6%)
Prefix3 232.97 (4.6%) 233.55 (4.5%)
0.3% ( -8% -9%)
AndHighHigh 90.67 (1.7%) 91.01 (1.1%)
0.4% ( -2% -3%)
Fuzzy1 102.98 (2.1%) 103.38 (3.5%)
0.4% ( -5% -6%)
AndHighLow 1121.50 (4.8%) 1126.02 (3.9%)
0.4% ( -7% -9%)
AndHighMed 127.28 (2.0%) 127.88 (1.1%)
0.5% ( -2% -3%)
Fuzzy2 68.39 (2.1%) 68.77 (3.1%)
0.5% ( -4% -5%)
Wildcard 48.08 (2.4%) 48.43 (4.2%)
0.7% ( -5% -7%)
IntNRQ9.69 (5.8%)9.79 (7.2%)
1.1% ( -11% - 15%)
OrNotHighLow 67.55 (8.1%) 68.88 (7.8%)
2.0% ( -12% - 19%)
OrNotHighMed 61.00 (8.3%) 62.38 (8.0%)
2.3% ( -12% - 20%)
OrNotHighHigh 35.44 (9.5%) 36.50 (9.5%)
3.0% ( -14% - 24%)
OrHighNotHigh 25.97 (9.6%) 26.80 (9.7%)
3.2% ( -14% - 24%)
OrHighNotMed 82.14 (10.1%) 84.84 (10.2%)
3.3% ( -15% - 26%)
OrHighHigh 29.25 (10.3%) 30.27 (10.5%)
3.5% ( -15% - 27%)
OrHighNotLow 104.15 (10.3%) 107.82 (10.5%)
3.5% ( -15% - 27%)
OrHighMed 65.67 (10.4%) 68.01 (10.7%)
3.6% ( -15% - 27%)
OrHighLow 63.61 (10.6%) 65.91 (10.7%)
3.6% ( -16% - 27%)
{noformat}
We should look into the regressions for phrases. But first I need to work on
LUCENE-6117, it is killing me :)
Allow Scorer to expose positions and payloads aka. nuke spans
--
Key: LUCENE-2878
URL: https://issues.apache.org/jira/browse/LUCENE-2878
Project: Lucene - Core
Issue Type: Improvement
Components: core/search
Affects Versions: Positions Branch
Reporter: Simon Willnauer
Assignee: Robert Muir
Labels: gsoc2014
Fix For: Positions Branch
Attachments: LUCENE-2878-OR.patch, LUCENE-2878-vs-trunk.patch,
LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch,
LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch,
LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch,
LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch,
LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch,
LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch,
LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878.patch,
LUCENE-2878.patch, LUCENE-2878.patch, LUCENE-2878_trunk.patch,
LUCENE-2878_trunk.patch, PosHighlighter.patch, PosHighlighter.patch
Currently we have two somewhat separate types of queries, the one which can
make use of positions (mainly spans) and payloads (spans). Yet Span*Query
doesn't really do scoring comparable to what other queries do and at the end
of the day they are duplicating lot of
[jira] [Updated] (SOLR-6691) REBALANCELEADERS needs to change the leader election queue.
[
https://issues.apache.org/jira/browse/SOLR-6691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erick Erickson updated SOLR-6691:
-
Attachment: BalanceLeaderTester.java
SOLR-6691.patch
OK, I think this is finally working as I expect. The attached java file is a
stand-alone program that stresses the heck out of shard leader election. The
idea is that you fire it up against a collection and it
1 takes the initial state
2 tries to issue the preferred leader command to a randome replica on each
shard.
3 issues the rebalanceleaders comand
4 verifies that all the shard leader election queues have one entry for all
the nodes that were there originally.
5 verifies that the actual leader is the preferred leader
6 goes to 2.
Note that the guts of this test are in the new unit test.
I had to change the leader election code to get all this predictable, and that
makes me a little nervous given how difficult that all was to get working in
the first place so this makes me a little nervous, but the external test code
beats _all_ the leader election code up pretty fiercely which gives me hope.
So I have a couple of options here:
1 go ahead and check it in. 5.0 appears to be receding here so it has some
time to bake before release
2 check it in to trunk and let it bake there for a while, perhaps until after
5.0 is cut, then merge and bake.
Opinions?
REBALANCELEADERS needs to change the leader election queue.
---
Key: SOLR-6691
URL: https://issues.apache.org/jira/browse/SOLR-6691
Project: Solr
Issue Type: Bug
Reporter: Erick Erickson
Assignee: Erick Erickson
Attachments: BalanceLeaderTester.java, SOLR-6691.patch
The original code (SOLR-6517) assumed that changes in the clusterstate after
issuing a command to the overseer to change the leader indicated that the
leader was successfully changed. Fortunately, Noble clued me in that this
isn't the case and that the potential leader needs to insert itself in the
leader election queue before trigging the change leader command.
Inserting themselves in the front of the queue should probably happen in
BALANCESHARDUNIQUE when the preferredLeader property is assigned as well.
[~noble.paul] Do evil things happen if a node joins at the head but it's
_already_ in the queue? These ephemeral nodes in the queue are watching each
other. So if node1 is the leader you have
node1 - node2 - node3 - node4
where - means watches.
Now, if node3 puts itself at the head of the list, you have
{code}
node1 - node2
- node3 - node4
{code}
I _think_ when I was looking at this it all just worked.
1 node 1 goes down. Nodes 2 and 3 duke it out but there's code to insure
that node3 becomes the leader and node2 inserts itself at then end so it's
watching node 4.
2 node 2 goes down, nobody gets notified and it doesn't matter.
3 node 3 goes down, node 4 gets notified and starts watching node 2 by
inserting itself at the end of the list.
4 node 4 goes down, nobody gets notified and it doesn't matter.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-6832) Queries be served locally rather than being forwarded to another replica
[ https://issues.apache.org/jira/browse/SOLR-6832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sachin Goyal updated SOLR-6832: --- Attachment: SOLR-6832.patch Attaching a patch for this ticket. The patch creates an option *preferLocalShards* in solrconfig.xml and in the query request params (giving more preference to the one in the query). If this option is set, *HttpShardHandler.preferCurrentHostForDistributedReq()* tries to find a local URL and puts that URL as the first one in the list of URLs sent to LBHttpSolrServer. This ensures that the current host's cores will be given preference for distributed queries. Other important function is *ResponseBuilder.findCurrentHostAddress()* which tries to find the current host's URL by searching for current core's name in the list of shards. The URL found by this function is used by the HttpShardHandle's function above. Default value of the option is kept as 'false' to ensure normal behavior. Queries be served locally rather than being forwarded to another replica Key: SOLR-6832 URL: https://issues.apache.org/jira/browse/SOLR-6832 Project: Solr Issue Type: Bug Components: SolrCloud Affects Versions: 4.10.2 Reporter: Sachin Goyal Attachments: SOLR-6832.patch Currently, I see that code flow for a query in SolrCloud is as follows: For distributed query: SolrCore - SearchHandler.handleRequestBody() - HttpShardHandler.submit() For non-distributed query: SolrCore - SearchHandler.handleRequestBody() - QueryComponent.process() \\ \\ \\ For a distributed query, the request is always sent to all the shards even if the originating SolrCore (handling the original distributed query) is a replica of one of the shards. If the original Solr-Core can check itself before sending http requests for any shard, we can probably save some network hopping and gain some performance. \\ \\ We can change SearchHandler.handleRequestBody() or HttpShardHandler.submit() to fix this behavior (most likely the former and not the latter). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6845) figure out why suggester causes slow startup - even when not used
[
https://issues.apache.org/jira/browse/SOLR-6845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14249431#comment-14249431
]
Tomás Fernández Löbbe commented on SOLR-6845:
-
bq. why the hell is this suggester so damn slow to build it's dictionary even
when the fields aren't used at all in the index?
I understand the reason of this is that the suggester still iterates across all
docs in the index, trying to get the stored content of those fields. Even if
the field is never present, this needs to be done. Most of the time in my tests
is spent in {{InputIterator.next()}}
bq. why does this suggester auto-register a firstSearcher/newSearcher event
listener to build the dict w/o there being any sort of configuration option
indicating that the solr-admin has requested it to build on firstSearcher (or
on every searcher open if that's what/why this is happening)
That's right, the suggester is building on startup and there is no way to
disable this. We should add an option to enable/disable this, maybe a
buildOnStartup conf option that could be false by default. I think it
should still load the stored suggesters when present.
figure out why suggester causes slow startup - even when not used
-
Key: SOLR-6845
URL: https://issues.apache.org/jira/browse/SOLR-6845
Project: Solr
Issue Type: Bug
Reporter: Hoss Man
SOLR-6679 was filed to track the investigation into the following problem...
{panel}
The stock solrconfig provides a bad experience with a large index... start up
Solr and it will spin at 100% CPU for minutes, unresponsive, while it
apparently builds a suggester index.
...
This is what I did:
1) indexed 10M very small docs (only takes a few minutes).
2) shut down Solr
3) start up Solr and watch it be unresponsive for over 4 minutes!
I didn't even use any of the fields specified in the suggester config and I
never called the suggest request handler.
{panel}
..but ultimately focused on removing/disabling the suggester from the sample
configs.
Opening this new issue to focus on actually trying to identify the root
problem fix it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-6857) Idea modules missing dependencies
James Strassburg created SOLR-6857: -- Summary: Idea modules missing dependencies Key: SOLR-6857 URL: https://issues.apache.org/jira/browse/SOLR-6857 Project: Solr Issue Type: Bug Components: Build Affects Versions: Trunk Environment: IntelliJ IDEA Reporter: James Strassburg Priority: Trivial The IDEA dev-tools configuration doesn't build in IDEA after running ant idea because the following modules are missing a dependency to analysis-common module: * velocity * extraction * map-reduce * dataimporthandler-extras To reproduce, run ant clean-idea followed by ant idea. Open the project in IDEA, configure the JDK, and make the project. The modules listed above will fail with an error finding org.apache.lucene.analysis.util.ResourceLoader. Adding analysis-common as a module dependency fixes this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6691) REBALANCELEADERS needs to change the leader election queue.
[
https://issues.apache.org/jira/browse/SOLR-6691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14249455#comment-14249455
]
Noble Paul commented on SOLR-6691:
--
bq.So I have a couple of options here:
5.0 is veru near .This touches the guts of SolrCloud and any bug will have huge
impact on the stability of the system. I would like this feature to be off from
5.0 and just keep in trunk till we have enough confidence
REBALANCELEADERS needs to change the leader election queue.
---
Key: SOLR-6691
URL: https://issues.apache.org/jira/browse/SOLR-6691
Project: Solr
Issue Type: Bug
Reporter: Erick Erickson
Assignee: Erick Erickson
Attachments: BalanceLeaderTester.java, SOLR-6691.patch
The original code (SOLR-6517) assumed that changes in the clusterstate after
issuing a command to the overseer to change the leader indicated that the
leader was successfully changed. Fortunately, Noble clued me in that this
isn't the case and that the potential leader needs to insert itself in the
leader election queue before trigging the change leader command.
Inserting themselves in the front of the queue should probably happen in
BALANCESHARDUNIQUE when the preferredLeader property is assigned as well.
[~noble.paul] Do evil things happen if a node joins at the head but it's
_already_ in the queue? These ephemeral nodes in the queue are watching each
other. So if node1 is the leader you have
node1 - node2 - node3 - node4
where - means watches.
Now, if node3 puts itself at the head of the list, you have
{code}
node1 - node2
- node3 - node4
{code}
I _think_ when I was looking at this it all just worked.
1 node 1 goes down. Nodes 2 and 3 duke it out but there's code to insure
that node3 becomes the leader and node2 inserts itself at then end so it's
watching node 4.
2 node 2 goes down, nobody gets notified and it doesn't matter.
3 node 3 goes down, node 4 gets notified and starts watching node 2 by
inserting itself at the end of the list.
4 node 4 goes down, nobody gets notified and it doesn't matter.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-6857) Idea modules missing dependencies
[ https://issues.apache.org/jira/browse/SOLR-6857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Strassburg updated SOLR-6857: --- Attachment: SOLR-6857.patch After applying this patch run ant clean-idea followed by ant idea and the project will compile successfully in IDEA. Idea modules missing dependencies - Key: SOLR-6857 URL: https://issues.apache.org/jira/browse/SOLR-6857 Project: Solr Issue Type: Bug Components: Build Affects Versions: Trunk Environment: IntelliJ IDEA Reporter: James Strassburg Priority: Trivial Attachments: SOLR-6857.patch The IDEA dev-tools configuration doesn't build in IDEA after running ant idea because the following modules are missing a dependency to analysis-common module: * velocity * extraction * map-reduce * dataimporthandler-extras To reproduce, run ant clean-idea followed by ant idea. Open the project in IDEA, configure the JDK, and make the project. The modules listed above will fail with an error finding org.apache.lucene.analysis.util.ResourceLoader. Adding analysis-common as a module dependency fixes this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[JENKINS] Lucene-Solr-trunk-MacOSX (64bit/jdk1.8.0) - Build # 1999 - Failure!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/1999/ Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseParallelGC (asserts: true) 1 tests failed. FAILED: org.apache.solr.update.SoftAutoCommitTest.testSoftAndHardCommitMaxTimeMixedAdds Error Message: soft529 wasn't fast enough Stack Trace: java.lang.AssertionError: soft529 wasn't fast enough at __randomizedtesting.SeedInfo.seed([C41328C9E0A78A21:95C7D14951D4BA86]:0) at org.junit.Assert.fail(Assert.java:93) at org.junit.Assert.assertTrue(Assert.java:43) at org.junit.Assert.assertNotNull(Assert.java:526) at org.apache.solr.update.SoftAutoCommitTest.testSoftAndHardCommitMaxTimeMixedAdds(SoftAutoCommitTest.java:111) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836) at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738) at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) at java.lang.Thread.run(Thread.java:745) Build Log: [...truncated
[JENKINS] Lucene-Solr-Tests-5.x-Java7 - Build # 2338 - Failure
Build: https://builds.apache.org/job/Lucene-Solr-Tests-5.x-Java7/2338/ 1 tests failed. REGRESSION: org.apache.solr.client.solrj.TestLBHttpSolrServer.testReliability Error Message: No live SolrServers available to handle this request Stack Trace: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request at __randomizedtesting.SeedInfo.seed([89891CBF8320CE1:C9504C8D5954DD48]:0) at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:539) at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91) at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301) at org.apache.solr.client.solrj.TestLBHttpSolrServer.testReliability(TestLBHttpSolrServer.java:223) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836) at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738) at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at
[jira] [Updated] (LUCENE-6117) infostream is currently unusable out of box
[
https://issues.apache.org/jira/browse/LUCENE-6117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Muir updated LUCENE-6117:
Attachment: LUCENE-6117.patch
Here is a patch.
This adds a package private boolean 'enableTestPoints' to IndexWriter. We don't
base it on assertion status or any of that, you only want test points if you
are one of the few special tests asking for them.
I refactored all tests to use the RandomIndexWriter.mockIndexWriter for
registering listeners at test points, and this sets the necessary boolean for
you on IW.
I also added simple tests ensuring that test points are only output when we ask
for them.
infostream is currently unusable out of box
---
Key: LUCENE-6117
URL: https://issues.apache.org/jira/browse/LUCENE-6117
Project: Lucene - Core
Issue Type: Bug
Reporter: Robert Muir
Attachments: LUCENE-6117.patch
testpoints used to only be emitted by assertions (still sketchy), but now are
emitted always. I assume this is due to the change to support running tests
with assertions disabled.
we should try to clean this up, simple stuff like this is now useless:
{code}
indexWriterConfig.setInfoStream(System.out);
// causes massive flooding like this:
// TP 0 [Tue Dec 16 20:19:37 EST 2014; Thread-0]: DocumentsWriterPerThread
addDocument start
// TP 0 [Tue Dec 16 20:19:37 EST 2014; Thread-0]: DocumentsWriterPerThread
addDocument start
// TP 0 [Tue Dec 16 20:19:37 EST 2014; Thread-0]: DocumentsWriterPerThread
addDocument start
{code}
I hit this several times today just trying to do benchmarks and debugging.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
