[JENKINS] Lucene-Solr-5.x-Linux (32bit/jdk1.8.0_40-ea-b04) - Build # 11201 - Still Failing!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-5.x-Linux/11201/ Java: 32bit/jdk1.8.0_40-ea-b04 -server -XX:+UseG1GC 2 tests failed. REGRESSION: org.apache.solr.cloud.CloudExitableDirectoryReaderTest.testDistribSearch Error Message: no exception matching expected: 400: Request took too long during query expansion. Terminating request. Stack Trace: java.lang.AssertionError: no exception matching expected: 400: Request took too long during query expansion. Terminating request. at __randomizedtesting.SeedInfo.seed([7178A0348F4483B2:F09E2E2CF81BE38E]:0) at org.junit.Assert.fail(Assert.java:93) at org.apache.solr.cloud.CloudExitableDirectoryReaderTest.assertFail(CloudExitableDirectoryReaderTest.java:101) at org.apache.solr.cloud.CloudExitableDirectoryReaderTest.doTimeoutTests(CloudExitableDirectoryReaderTest.java:81) at org.apache.solr.cloud.CloudExitableDirectoryReaderTest.doTest(CloudExitableDirectoryReaderTest.java:54) at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869) at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836) at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738) at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55) at com.carr
[JENKINS] Lucene-Solr-NightlyTests-5.x - Build # 636 - Still Failing
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.x/636/ 1 tests failed. FAILED: org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch Error Message: Error CREATEing SolrCore 'halfcollection_shard1_replica1': Unable to create core [halfcollection_shard1_replica1] Caused by: null Stack Trace: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Error CREATEing SolrCore 'halfcollection_shard1_replica1': Unable to create core [halfcollection_shard1_replica1] Caused by: null at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:568) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211) at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testErrorHandling(CollectionsAPIDistributedZkTest.java:583) at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:205) at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836) at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738) at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate
[JENKINS] Lucene-Solr-5.x-Linux (32bit/jdk1.7.0_67) - Build # 11200 - Failure!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-5.x-Linux/11200/ Java: 32bit/jdk1.7.0_67 -server -XX:+UseConcMarkSweepGC 6 tests failed. REGRESSION: org.apache.solr.client.solrj.SolrExampleBinaryTest.testChildDoctransformer Error Message: Expected mime type application/octet-stream but got text/html. Error 500 Server Error HTTP ERROR: 500 Problem accessing /solr/collection1/select. Reason: Server Error Powered by Jetty:// Stack Trace: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Expected mime type application/octet-stream but got text/html. Error 500 Server Error HTTP ERROR: 500 Problem accessing /solr/collection1/select. Reason: Server Error Powered by Jetty:// at __randomizedtesting.SeedInfo.seed([DBB6A0751A5A2765:A86CBFEF96425063]:0) at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:530) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211) at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91) at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301) at org.apache.solr.client.solrj.SolrExampleTests.testChildDoctransformer(SolrExampleTests.java:1373) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com
[jira] [Updated] (SOLR-6512) Add a collections API call to add/delete arbitrary properties to a specific replica
[ https://issues.apache.org/jira/browse/SOLR-6512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erick Erickson updated SOLR-6512: - Attachment: SOLR-6512.patch Reworked patch for arbitrary property assignment rather than roles. Also on review board, see: https://reviews.apache.org/r/26161/ the "preferredLeader" role is special in that there is a list of known properties that we enforce the "only one per slice" rule. This list may be added to in the future if necessary. There is an optional parameter "sliceUnique" that can be specified with arbitrary properties to enforce this rule without changing the code. sliceUnique defaults to "false", in which case properties can be added however desired. I'll probably commit this Wednesday unless there are objections. > Add a collections API call to add/delete arbitrary properties to a specific > replica > --- > > Key: SOLR-6512 > URL: https://issues.apache.org/jira/browse/SOLR-6512 > Project: Solr > Issue Type: Improvement >Reporter: Erick Erickson >Assignee: Erick Erickson > Attachments: SOLR-6512.patch, SOLR-6512.patch, SOLR-6512.patch > > > This is a sub-task for SOLR-6491, but seems generally useful. > Since this is in support of the "preferredLeader" functionality, I've run > into some considerations that I wanted some feedback on how to handle. > "preferredLeader" has the restriction that there should only be one per > slice, so setting this for a particular node means removing the property for > all the other replicas on the slice. Not a problem to do, my question is more > whether this is something reasonable to enforce on an arbitrary property > based on what that property is? Perfectly do-able, but "semantically > challenged". Currently, this is never a node with "preferedLeader" set to > "false", it is forcibly removed from other nodes in the slice when this > property is assigned. > The problem here is that there's nothing about assigning an arbitrary > property to a node that would reasonably imply this kind of behavior. One > could always control this with secondary flags on the command, e.g. > "shardExclusive=true|false" for instance, perhaps with safety checks in for > known one-per-shard properties like "preferredLeader". > "preferredLeader" seems to fit more naturally into a "role", but currently > ADDROLE and DELTEROLE have nothing to do with the notion of setting a role > for a particular node relative to a collection/shard. Easy enough to add, but > enforcing the "only one node per slice may have this role" rule there is > similarly arbitrary and overloads the ADDROLE/DELETEROLE in a way that seems > equally confusing. Plus, checking whether the required collection/shard/node > params are present becomes based on the value of the property being set, > which is all equally arbitrary. > The other interesting thing is that setting an arbitrary property on a node > would allow one to mess things up royally by, say, changing properties like > "core", or "base_url" or node_name at will. Actually this is potentially > useful, but very, very dangerous and I'm not particularly interested in > supporting it ;). I suppose we could require a prefix, say the only settable > properties are "property.whatever". > We could also add something specific to nodes, something like > ADDREPLICAROLE/DELETEREPLICAROLE, perhaps with sub-params like > "onlyOneAllowedPerShard", but this gets messy and relies on the users "doing > the right thing". I prefer enforcing rules like this based on the role I > think. Or at least enforcing these kinds of requirements on the > "preferredLeader" role if we go that way. > What are people's thoughts here? I think I'm tending towards the > ADDREPLICAROLE/DELETEREPLICAROLE way of doing this, but it's not set in > stone. I have code locally for arbitrary properties that I can modify for the > role bits. > So, if I'm going to summarize the points I'd like feedback on: > 1> Is setting arbitrary properties on a node desirable? If so, should we > require a prefix like "property" to prevent resetting values SolrCloud > depends on? > 2> Is it better to piggyback on ADDROLE/DELETEROLE? Personally I'm not in > favor of this one. Too messy with requiring additional parameters to work > right in this case > 3> Is the best option to create new collections API calls for > ADDREPLICAROLE/DELETEREPLICAROLE that > 3.1> require collection/slice/node parameters > 3.2> enforces the "onlyOnePerShard" rule for certain known roles > 3.3 v1> allows users to specify arbitrary roles something like > "onlyOnePerShard" as an optional T|F parameter, otherwise is totally open. > -or- > 3.3 v2> No support other than "preferredLeader", only roles that are > pre-defined are allowed, in w
Review Request 26161: SOLR-6512: Add a collections API call to add/delete arbitrary properties to a specific replica
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26161/ --- Review request for lucene. Repository: lucene Description --- This allows us to add properties to a replica, it takes a collection/slice/replica triplet along with a property. There is exactly one known "sliceUnique" property at present, preferredLeader. Arbitrary properties can be assigned a "sliceUnique" attribute though. If sliceUnique==false, then there are no restrictions on how many replicas can have an property/value pair. Diffs - trunk/solr/core/src/java/org/apache/solr/cloud/Overseer.java 1628219 trunk/solr/core/src/java/org/apache/solr/cloud/OverseerCollectionProcessor.java 1628219 trunk/solr/core/src/java/org/apache/solr/handler/admin/CollectionsHandler.java 1628219 trunk/solr/core/src/test/org/apache/solr/cloud/DeleteReplicaTest.java 1628219 trunk/solr/core/src/test/org/apache/solr/cloud/TestCollectionAPI.java 1628219 trunk/solr/solrj/src/java/org/apache/solr/common/cloud/ZkStateReader.java 1628219 trunk/solr/solrj/src/java/org/apache/solr/common/params/CollectionParams.java 1628219 Diff: https://reviews.apache.org/r/26161/diff/ Testing --- Unit tests are in place. Thanks, Erick Erickson
[jira] [Updated] (SOLR-6512) Add a collections API call to add/delete arbitrary properties to a specific replica
[ https://issues.apache.org/jira/browse/SOLR-6512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erick Erickson updated SOLR-6512: - Summary: Add a collections API call to add/delete arbitrary properties to a specific replica (was: Add a collections API call to add/delete arbitrary roles to a specific replica) > Add a collections API call to add/delete arbitrary properties to a specific > replica > --- > > Key: SOLR-6512 > URL: https://issues.apache.org/jira/browse/SOLR-6512 > Project: Solr > Issue Type: Improvement >Reporter: Erick Erickson >Assignee: Erick Erickson > Attachments: SOLR-6512.patch, SOLR-6512.patch > > > This is a sub-task for SOLR-6491, but seems generally useful. > Since this is in support of the "preferredLeader" functionality, I've run > into some considerations that I wanted some feedback on how to handle. > "preferredLeader" has the restriction that there should only be one per > slice, so setting this for a particular node means removing the property for > all the other replicas on the slice. Not a problem to do, my question is more > whether this is something reasonable to enforce on an arbitrary property > based on what that property is? Perfectly do-able, but "semantically > challenged". Currently, this is never a node with "preferedLeader" set to > "false", it is forcibly removed from other nodes in the slice when this > property is assigned. > The problem here is that there's nothing about assigning an arbitrary > property to a node that would reasonably imply this kind of behavior. One > could always control this with secondary flags on the command, e.g. > "shardExclusive=true|false" for instance, perhaps with safety checks in for > known one-per-shard properties like "preferredLeader". > "preferredLeader" seems to fit more naturally into a "role", but currently > ADDROLE and DELTEROLE have nothing to do with the notion of setting a role > for a particular node relative to a collection/shard. Easy enough to add, but > enforcing the "only one node per slice may have this role" rule there is > similarly arbitrary and overloads the ADDROLE/DELETEROLE in a way that seems > equally confusing. Plus, checking whether the required collection/shard/node > params are present becomes based on the value of the property being set, > which is all equally arbitrary. > The other interesting thing is that setting an arbitrary property on a node > would allow one to mess things up royally by, say, changing properties like > "core", or "base_url" or node_name at will. Actually this is potentially > useful, but very, very dangerous and I'm not particularly interested in > supporting it ;). I suppose we could require a prefix, say the only settable > properties are "property.whatever". > We could also add something specific to nodes, something like > ADDREPLICAROLE/DELETEREPLICAROLE, perhaps with sub-params like > "onlyOneAllowedPerShard", but this gets messy and relies on the users "doing > the right thing". I prefer enforcing rules like this based on the role I > think. Or at least enforcing these kinds of requirements on the > "preferredLeader" role if we go that way. > What are people's thoughts here? I think I'm tending towards the > ADDREPLICAROLE/DELETEREPLICAROLE way of doing this, but it's not set in > stone. I have code locally for arbitrary properties that I can modify for the > role bits. > So, if I'm going to summarize the points I'd like feedback on: > 1> Is setting arbitrary properties on a node desirable? If so, should we > require a prefix like "property" to prevent resetting values SolrCloud > depends on? > 2> Is it better to piggyback on ADDROLE/DELETEROLE? Personally I'm not in > favor of this one. Too messy with requiring additional parameters to work > right in this case > 3> Is the best option to create new collections API calls for > ADDREPLICAROLE/DELETEREPLICAROLE that > 3.1> require collection/slice/node parameters > 3.2> enforces the "onlyOnePerShard" rule for certain known roles > 3.3 v1> allows users to specify arbitrary roles something like > "onlyOnePerShard" as an optional T|F parameter, otherwise is totally open. > -or- > 3.3 v2> No support other than "preferredLeader", only roles that are > pre-defined are allowed, in which case the "onlyOnePerShard" is implicit in > the role. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-6278) add admin/collections?action=DELETEREPLICA&core=... support, make collection=... and shard=... parameters optional
[
https://issues.apache.org/jira/browse/SOLR-6278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erick Erickson resolved SOLR-6278.
--
Resolution: Won't Fix
See discussion. There's still room for being able to do node-type operations,
think of a physical machine going down and you want to remove all traces of it
from the cluster state, but that's for another ticket.
> add admin/collections?action=DELETEREPLICA&core=... support, make
> collection=... and shard=... parameters optional
> --
>
> Key: SOLR-6278
> URL: https://issues.apache.org/jira/browse/SOLR-6278
> Project: Solr
> Issue Type: Improvement
>Reporter: Christine Poerschke
>Assignee: Erick Erickson
> Attachments: SOLR-6278.patch
>
>
> To add {{core=...}} as an alternative to {{replica=...}} way of identifying
> what is to be deleted, {{collection=...}} and {{shard=...}} to be optional
> provided the other parameters uniquely identify exactly one deletion target.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-5978) don't write a norm of infinity when analyzer returns no tokens
[ https://issues.apache.org/jira/browse/LUCENE-5978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14152473#comment-14152473 ] Adrien Grand commented on LUCENE-5978: -- +1 > don't write a norm of infinity when analyzer returns no tokens > -- > > Key: LUCENE-5978 > URL: https://issues.apache.org/jira/browse/LUCENE-5978 > Project: Lucene - Core > Issue Type: Bug >Reporter: Robert Muir > Attachments: LUCENE-5978.patch > > > When a document doesn't have the field, we fill with zero. when a segment > doesn't have the field, we also fill with zero. > however, when the analyzer doesn't return any terms for the field, we still > call similarity.computeNorm(0)... with the default similarity this encodes > infinity... -1 > in such a case, it doesnt really matter what the norm is, since it has no > terms. But its more efficient for e.g. compression if we consistently use > zero. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-5938) New DocIdSet implementation with random write access
[
https://issues.apache.org/jira/browse/LUCENE-5938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adrien Grand updated LUCENE-5938:
-
Attachment: LUCENE-5938.patch
Thanks for the review Mike, here is a new patch that tries to address your
concerns.
bq. Maybe fix "word" to be "long" instead
Done.
bq. do we need to guard against zeroWords being 0?
I added a comment as well as a test as you suggested.
{quote}
Maybe add some more comments around tricky parts of SparseFixedBitSet.
E.g., the different branches inside set? And, it looks strange doing
1L << i, but in fact the JVM/processor make that 1L << (i % 64). And
Iterator.currentOrNextDoc is scary looking... do we have enough tests
here?
{quote}
I added more comments, hopefully they make sense. Please let me know if there
are still things that are not clear. currentOrNextDoc is a bit complicated
because of the different cases that need to be handled but the structure is
actually quite simple so at least get and set should be easy to understand. I
extracted the insertion of a new long to a separate method, this should make
set easier to read?
Regarding the shift, indeed it relies on the fact that shifts are mod 64
(FixedBitSet does the same). I added some comments about it.
Regarding the tests, BaseDocIdSetTestCase.testAgainstBitSet tests various
densities and assertEquals checks nextDoc(), docId(), interleaved calls to
nextDoc() and advance(), and that the oal.util.Bits representation is
consistent with the iterator. I think that is good?
bq. I hit this test failure, which reproduces with the patch
I dug that one, and the reason is that the searcher is created with threads, so
the exception is indeed wrapped into an ExecutionException, which is in turn
wrapped into a RuntimeException to by-pass the fact that ExecutionException is
checked. It doesn't reproduce on trunk because the default rewrite method reads
data from the index in MultiTermQuery.rewrite (collectTerms) which does not run
in a thread pool. You can reproduce the issue on trunk by setting the rewrite
method of the term range query to {{CONSTANT_SCORE_FILTER_REWRITE}}. I fixed
the test in the patch to walk down the causes of the exception that is thrown.
> New DocIdSet implementation with random write access
>
>
> Key: LUCENE-5938
> URL: https://issues.apache.org/jira/browse/LUCENE-5938
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Adrien Grand
>Assignee: Adrien Grand
> Attachments: LUCENE-5938.patch, LUCENE-5938.patch, LUCENE-5938.patch,
> LUCENE-5938.patch, LUCENE-5938.patch, low_freq.tasks
>
>
> We have a great cost API that is supposed to help make decisions about how to
> best execute queries. However, due to the fact that several of our filter
> implementations (eg. TermsFilter and BooleanFilter) return FixedBitSets,
> either we use the cost API and make bad decisions, or need to fall back to
> heuristics which are not as good such as
> RandomAccessFilterStrategy.useRandomAccess which decides that random access
> should be used if the first doc in the set is less than 100.
> On the other hand, we also have some nice compressed and cacheable DocIdSet
> implementation but we cannot make use of them because TermsFilter requires a
> DocIdSet that has random write access, and FixedBitSet is the only DocIdSet
> that we have that supports random access.
> I think it would be nice to replace FixedBitSet in those filters with another
> DocIdSet that would also support random write access but would have a better
> cost?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-5978) don't write a norm of infinity when analyzer returns no tokens
[ https://issues.apache.org/jira/browse/LUCENE-5978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Muir updated LUCENE-5978: Attachment: LUCENE-5978.patch here's a patch with a test. > don't write a norm of infinity when analyzer returns no tokens > -- > > Key: LUCENE-5978 > URL: https://issues.apache.org/jira/browse/LUCENE-5978 > Project: Lucene - Core > Issue Type: Bug >Reporter: Robert Muir > Attachments: LUCENE-5978.patch > > > When a document doesn't have the field, we fill with zero. when a segment > doesn't have the field, we also fill with zero. > however, when the analyzer doesn't return any terms for the field, we still > call similarity.computeNorm(0)... with the default similarity this encodes > infinity... -1 > in such a case, it doesnt really matter what the norm is, since it has no > terms. But its more efficient for e.g. compression if we consistently use > zero. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5814) CoreContainer reports incorrect & missleading path for solrconfig.xml when there are loading problems
[
https://issues.apache.org/jira/browse/SOLR-5814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jan Høydahl updated SOLR-5814:
--
Fix Version/s: 4.10.1
> CoreContainer reports incorrect & missleading path for solrconfig.xml when
> there are loading problems
> -
>
> Key: SOLR-5814
> URL: https://issues.apache.org/jira/browse/SOLR-5814
> Project: Solr
> Issue Type: Bug
>Reporter: Hoss Man
>Assignee: Hoss Man
> Fix For: 4.10.1, 5.0, Trunk
>
> Attachments: SOLR-5814.patch, SOLR-5814.patch
>
>
> The error messages thrown by CoreContainer when there is a problem loading
> solrconfig.xml refer to the wrong path (leaves out "conf/").
> This is missleading users (who may not notice the root cause) into thinking
> they need to move their solrconfig.xml from
> {{collection_name/conf/solrconfig.xml}} to {{collection_name/solrconfig.xml}}
> at which point they still get the same error message because solr still can't
> find the file in the conf dir
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-5978) don't write a norm of infinity when analyzer returns no tokens
[ https://issues.apache.org/jira/browse/LUCENE-5978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14152412#comment-14152412 ] Adrien Grand commented on LUCENE-5978: -- +1 as well > don't write a norm of infinity when analyzer returns no tokens > -- > > Key: LUCENE-5978 > URL: https://issues.apache.org/jira/browse/LUCENE-5978 > Project: Lucene - Core > Issue Type: Bug >Reporter: Robert Muir > > When a document doesn't have the field, we fill with zero. when a segment > doesn't have the field, we also fill with zero. > however, when the analyzer doesn't return any terms for the field, we still > call similarity.computeNorm(0)... with the default similarity this encodes > infinity... -1 > in such a case, it doesnt really matter what the norm is, since it has no > terms. But its more efficient for e.g. compression if we consistently use > zero. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-5978) don't write a norm of infinity when analyzer returns no tokens
[ https://issues.apache.org/jira/browse/LUCENE-5978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14152408#comment-14152408 ] Michael McCandless commented on LUCENE-5978: +1 > don't write a norm of infinity when analyzer returns no tokens > -- > > Key: LUCENE-5978 > URL: https://issues.apache.org/jira/browse/LUCENE-5978 > Project: Lucene - Core > Issue Type: Bug >Reporter: Robert Muir > > When a document doesn't have the field, we fill with zero. when a segment > doesn't have the field, we also fill with zero. > however, when the analyzer doesn't return any terms for the field, we still > call similarity.computeNorm(0)... with the default similarity this encodes > infinity... -1 > in such a case, it doesnt really matter what the norm is, since it has no > terms. But its more efficient for e.g. compression if we consistently use > zero. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (LUCENE-5978) don't write a norm of infinity when analyzer returns no tokens
Robert Muir created LUCENE-5978: --- Summary: don't write a norm of infinity when analyzer returns no tokens Key: LUCENE-5978 URL: https://issues.apache.org/jira/browse/LUCENE-5978 Project: Lucene - Core Issue Type: Bug Reporter: Robert Muir When a document doesn't have the field, we fill with zero. when a segment doesn't have the field, we also fill with zero. however, when the analyzer doesn't return any terms for the field, we still call similarity.computeNorm(0)... with the default similarity this encodes infinity... -1 in such a case, it doesnt really matter what the norm is, since it has no terms. But its more efficient for e.g. compression if we consistently use zero. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6476) Create a bulk mode for schema API
[
https://issues.apache.org/jira/browse/SOLR-6476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14152356#comment-14152356
]
Steve Rowe commented on SOLR-6476:
--
{quote}
bq. AFAICT, a bulk Schema API request with a new field using a new field type
in the same request can fail depending on the order of the specified operations
Yes it fails. Works as designed. This is exactly the same behavior you will see
in an RDBMS as well. Order is important
{quote}
Order is not important in {{schema.xml}}, and in plenty of other contexts.
This order dependence will need to be explicitly documented.
> Create a bulk mode for schema API
> -
>
> Key: SOLR-6476
> URL: https://issues.apache.org/jira/browse/SOLR-6476
> Project: Solr
> Issue Type: Bug
> Components: Schema and Analysis
>Reporter: Noble Paul
>Assignee: Noble Paul
> Labels: managedResource
> Attachments: SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch,
> SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch
>
>
> The current schema API does one operation at a time and the normal usecase is
> that users add multiple fields/fieldtypes/copyFields etc in one shot.
> example
> {code:javascript}
> curl http://localhost:8983/solr/collection1/schema -H
> 'Content-type:application/json' -d '{
> "add-field": {
> "name":"sell-by",
> "type":"tdate",
> "stored":true
> },
> "add-field":{
> "name":"catchall",
> "type":"text_general",
> "stored":false
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6476) Create a bulk mode for schema API
[
https://issues.apache.org/jira/browse/SOLR-6476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14152339#comment-14152339
]
Steve Rowe commented on SOLR-6476:
--
{quote}
bq. One of the tests should have a non-trivial fieldtype definition, like at
least one analyzer.
It is not supported yet. Pls correct me if I am wrong. does the current REST
API support it?
{quote}
Yes. See {{TestManagedSchemaFieldTypeResource}}.
{quote}
bq. Most of the add*() javadocs in IndexSchema say that persistence always
happens, but it doesn't if persist=false
It says "* @param persist to persist or not" . Isn't it right?
{quote}
This is what I'm talking about: "Copies this schema, adds the given fields to
the copy, then persists the new schema."
{quote}
bq. {{SchemaManager}} has zero javadocs. More would be good.
It is not a class for others to use . But , it will be added
{quote}
Thanks. Javadocs (or rather comments of any kind) are for maintainers too, not
just users. Here's an example where javadocs/comments would help a maintainer:
{{SchemaManager.Operation.getMetaWithout()}}. What does that thing do? (Hint:
the name of the method doesn't tell you.)
> Create a bulk mode for schema API
> -
>
> Key: SOLR-6476
> URL: https://issues.apache.org/jira/browse/SOLR-6476
> Project: Solr
> Issue Type: Bug
> Components: Schema and Analysis
>Reporter: Noble Paul
>Assignee: Noble Paul
> Labels: managedResource
> Attachments: SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch,
> SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch
>
>
> The current schema API does one operation at a time and the normal usecase is
> that users add multiple fields/fieldtypes/copyFields etc in one shot.
> example
> {code:javascript}
> curl http://localhost:8983/solr/collection1/schema -H
> 'Content-type:application/json' -d '{
> "add-field": {
> "name":"sell-by",
> "type":"tdate",
> "stored":true
> },
> "add-field":{
> "name":"catchall",
> "type":"text_general",
> "stored":false
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1632) Distributed IDF
[ https://issues.apache.org/jira/browse/SOLR-1632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14152329#comment-14152329 ] Anshum Gupta commented on SOLR-1632: Thanks for updating the patch [~vzhovtiuk]. The tests pass now. I'm looking at the updated patch. > Distributed IDF > --- > > Key: SOLR-1632 > URL: https://issues.apache.org/jira/browse/SOLR-1632 > Project: Solr > Issue Type: New Feature > Components: search >Affects Versions: 1.5 >Reporter: Andrzej Bialecki >Assignee: Mark Miller > Fix For: 4.9, Trunk > > Attachments: 3x_SOLR-1632_doesntwork.patch, SOLR-1632.patch, > SOLR-1632.patch, SOLR-1632.patch, SOLR-1632.patch, SOLR-1632.patch, > SOLR-1632.patch, SOLR-1632.patch, SOLR-1632.patch, SOLR-1632.patch, > SOLR-1632.patch, SOLR-1632.patch, SOLR-1632.patch, SOLR-1632.patch, > distrib-2.patch, distrib.patch > > > Distributed IDF is a valuable enhancement for distributed search across > non-uniform shards. This issue tracks the proposed implementation of an API > to support this functionality in Solr. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: svn commit: r1628194 - /lucene/dev/branches/branch_5x/lucene/memory/src/test/org/apache/lucene/index/memory/TestMemoryIndexAgainstRAMDir.java
Oh, Rob already fixed it. Thanks!
Alan Woodward
www.flax.co.uk
On 29 Sep 2014, at 15:37, rm...@apache.org wrote:
> Author: rmuir
> Date: Mon Sep 29 14:37:55 2014
> New Revision: 1628194
>
> URL: http://svn.apache.org/r1628194
> Log:
> LUCENE-5911: fix compile
>
> Modified:
>
> lucene/dev/branches/branch_5x/lucene/memory/src/test/org/apache/lucene/index/memory/TestMemoryIndexAgainstRAMDir.java
>
> Modified:
> lucene/dev/branches/branch_5x/lucene/memory/src/test/org/apache/lucene/index/memory/TestMemoryIndexAgainstRAMDir.java
> URL:
> http://svn.apache.org/viewvc/lucene/dev/branches/branch_5x/lucene/memory/src/test/org/apache/lucene/index/memory/TestMemoryIndexAgainstRAMDir.java?rev=1628194&r1=1628193&r2=1628194&view=diff
> ==
> ---
> lucene/dev/branches/branch_5x/lucene/memory/src/test/org/apache/lucene/index/memory/TestMemoryIndexAgainstRAMDir.java
> (original)
> +++
> lucene/dev/branches/branch_5x/lucene/memory/src/test/org/apache/lucene/index/memory/TestMemoryIndexAgainstRAMDir.java
> Mon Sep 29 14:37:55 2014
> @@ -431,7 +431,7 @@ public class TestMemoryIndexAgainstRAMDi
> IndexWriter writer = new IndexWriter(dir,
> newIndexWriterConfig(random(), mockAnalyzer));
> Document nextDoc = lineFileDocs.nextDoc();
> Document doc = new Document();
> - for (Field field : nextDoc.getFields()) {
> + for (IndexableField field : nextDoc.getFields()) {
> if (field.fieldType().indexed()) {
> doc.add(field);
> if (random().nextInt(3) == 0) {
> @@ -442,7 +442,7 @@ public class TestMemoryIndexAgainstRAMDi
>
> writer.addDocument(doc);
> writer.close();
> - for (IndexableField field : doc.indexableFields()) {
> + for (IndexableField field : doc.getFields()) {
> memory.addField(field.name(), ((Field)field).stringValue(),
> mockAnalyzer);
> }
> DirectoryReader competitor = DirectoryReader.open(dir);
>
>
Re: [JENKINS] Lucene-Solr-5.x-Linux (32bit/jdk1.8.0_40-ea-b04) - Build # 11197 - Still Failing!
Bah, I'll fix
Alan Woodward
www.flax.co.uk
On 29 Sep 2014, at 16:07, Policeman Jenkins Server wrote:
> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-5.x-Linux/11197/
> Java: 32bit/jdk1.8.0_40-ea-b04 -server -XX:+UseG1GC
>
> All tests passed
>
> Build Log:
> [...truncated 4413 lines...]
>[javac] Compiling 2 source files to
> /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/build/memory/classes/test
>[javac]
> /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/memory/src/test/org/apache/lucene/index/memory/TestMemoryIndexAgainstRAMDir.java:434:
> error: incompatible types: IndexableField cannot be converted to Field
>[javac] for (Field field : nextDoc.getFields()) {
>[javac] ^
>[javac]
> /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/memory/src/test/org/apache/lucene/index/memory/TestMemoryIndexAgainstRAMDir.java:445:
> error: cannot find symbol
>[javac] for (IndexableField field : doc.indexableFields()) {
>[javac] ^
>[javac] symbol: method indexableFields()
>[javac] location: variable doc of type Document
>[javac] Note:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/memory/src/test/org/apache/lucene/index/memory/TestMemoryIndexAgainstRAMDir.java
> uses or overrides a deprecated API.
>[javac] Note: Recompile with -Xlint:deprecation for details.
>[javac] 2 errors
>
> BUILD FAILED
> /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/build.xml:524: The following
> error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/build.xml:472: The following
> error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/build.xml:61: The following
> error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/extra-targets.xml:39: The
> following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/build.xml:447: The
> following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/common-build.xml:2142:
> The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/module-build.xml:55:
> The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/common-build.xml:785:
> The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/common-build.xml:799:
> The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/common-build.xml:1867:
> Compile failed; see the compiler error output for details.
>
> Total time: 32 minutes 32 seconds
> Build step 'Invoke Ant' marked build as failure
> [description-setter] Description set: Java: 32bit/jdk1.8.0_40-ea-b04 -server
> -XX:+UseG1GC
> Archiving artifacts
> Recording test results
> Email was triggered for: Failure - Any
> Sending email for trigger: Failure - Any
>
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
Re: queryResultMaxDocsCached vs. queryResultWindowSize
Thanks for your help Yonik and Tomas, I had several mistaken assumptions about how caching worked which were resolved by walking through the code in the debugger after reading your replies. Tom On Fri, Sep 26, 2014 at 4:55 PM, Yonik Seeley wrote: > On Fri, Sep 26, 2014 at 4:38 PM, Tom Burton-West > wrote: > > Hi Yonik, > > > > I'm still confused. > > > > suspect don't understand how paging and caching interact. I probably > need > > to walk through the code. Is there a unit test that exercises > > SolrIndexSearcher.getDocListC or a good unit test to use as a base to > write > > one? > > > > > > Part of what confuses me is whether what gets cached always starts at > row 1 > > of results. > > Yes, we always cache from the first row. > Asking for rows 91-100 requires collecting 1-100 (and it's the latter > we cache - ignoring deep paging). > It's also just ids (and optionally scores) that are cached... so > either 4 bytes or 8 bytes per document cached, depending on if you ask > for scores back. > > queryWindowSize just rounds up the upper bound. > > > I'll try to explain my confusion. > > Using the defaults in the solrconfig example: > > 20 > > 200 > > > > If I query for start=0, rows =10 Solr fetches 20 results and caches > them. > > If I query for start =11 rows =10 Solr read rows 11-20 from cache > > Correct. > > > What happens when I query for start = 21 rows= 10? > > I thought that Solr would then fetch rows 21-40 into the > queryResultCache. > > Is this wrong? > > It will result in a cache miss and we'll collect 0-40 and cache that. > > > If I query for start =195 rows =10 does Solr cache rows 195-200 but go > to > > disk for rows over 200 (queryResultMaxDocsCached=200)? Or does Solr > skip > > caching altogether for rows over 200 > > Probably the latter... it's an edge case so I'd have to check the code > to know for sure if the check is pre or post rounding up. > > -Yonik > http://heliosearch.org - native code faceting, facet functions, > sub-facets, off-heap data > > - > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > >
[JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 642 - Still Failing
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/642/ 1 tests failed. REGRESSION: org.apache.solr.handler.component.DistributedFacetPivotSmallTest.testDistribSearch Error Message: Captured an uncaught exception in thread: Thread[id=11082, name=Thread-4297, state=RUNNABLE, group=TGRP-DistributedFacetPivotSmallTest] Stack Trace: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=11082, name=Thread-4297, state=RUNNABLE, group=TGRP-DistributedFacetPivotSmallTest] at __randomizedtesting.SeedInfo.seed([3DD00CB100ACFBAF:BC3682A977F39B93]:0) Caused by: java.lang.RuntimeException: org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: https://127.0.0.1:13614/dk_tb/bq at __randomizedtesting.SeedInfo.seed([3DD00CB100ACFBAF]:0) at org.apache.solr.BaseDistributedSearchTestCase$5.run(BaseDistributedSearchTestCase.java:580) Caused by: org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: https://127.0.0.1:13614/dk_tb/bq at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:580) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211) at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91) at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301) at org.apache.solr.BaseDistributedSearchTestCase$5.run(BaseDistributedSearchTestCase.java:575) Caused by: java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:152) at java.net.SocketInputStream.read(SocketInputStream.java:122) at sun.security.ssl.InputRecord.readFully(InputRecord.java:442) at sun.security.ssl.InputRecord.read(InputRecord.java:480) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927) at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884) at sun.security.ssl.AppInputStream.read(AppInputStream.java:102) at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160) at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84) at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260) at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283) at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251) at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:682) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:486) at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57) at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:466) ... 5 more Build Log: [...truncated 12549 lines...] [junit4] Suite: org.apache.solr.handler.component.DistributedFacetPivotSmallTest [junit4] 2> Creating dataDir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.handler.component.DistributedFacetPivotSmallTest-3DD00CB100ACFBAF-001/init-core-data-001 [junit4] 2> 2668501 T10193 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true) [junit4] 2> 2668502 T10193 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /dk_tb/bq [junit4] 2> 2668506 T10193 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch [junit4] 2> 2668509 T10193 oejs.Server.doStart je
[jira] [Commented] (SOLR-6571) Need a znode watcher support class
[
https://issues.apache.org/jira/browse/SOLR-6571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14152232#comment-14152232
]
Mark Miller commented on SOLR-6571:
---
One of my favorite subtleties was that some Watcher events don't require that
you re-create the Watcher. That's a bid insidious given the API ;)
> Need a znode watcher support class
> --
>
> Key: SOLR-6571
> URL: https://issues.apache.org/jira/browse/SOLR-6571
> Project: Solr
> Issue Type: Improvement
> Components: SolrCloud
>Reporter: Timothy Potter
>
> When implementing SOLR-6249, [~noble.paul] points out that
> ZkIndexSchemaReader doesn't watch the managed schema znode correctly and
> should use a strategy similar to what ZkStateReader does so that watchers
> persist across zk client connection failures:
> The correct example is the constructor of ZkStateReader
>zkClient = new SolrZkClient(zkServerAddress, zkClientTimeout,
> zkClientConnectTimeout,
> // on reconnect, reload cloud info
> new OnReconnect() {
> //implement stuff here
> }
> });
> this ensures that the watchers are persisted across reconnect.
> We need a watch support class to help developers implement watchers correctly
> instead of having some code do it correctly and other code not.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6571) Need a znode watcher support class
[
https://issues.apache.org/jira/browse/SOLR-6571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14152223#comment-14152223
]
Mark Miller commented on SOLR-6571:
---
To note, nothing I said in the issue related to this is meant to mean there are
not improvements or helper classes or documentation improvements to be made.
It's also not meant to mean this stuff is easy - it's not. You usually learn it
properly by burning your fingers off one by one. We have some code that try's
to make things better. Things like handling an expired ZooKeeper instance for
you (you can't keep using the same one). Handling the common ConnectionLoss
retry for you. Etc. There is still a lot we can probably do. Still, to properly
code for ZooKeeper you have to understand ConnectionLoss and SessionExpiration
pretty well - you can't really count on the code you are writing otherwise - or
test it properly. Perhaps there are more javadocs that can be beefed up. The
tiny corners around this stuff was not easy to glean from the ZK doc back when
I tackled it either.
> Need a znode watcher support class
> --
>
> Key: SOLR-6571
> URL: https://issues.apache.org/jira/browse/SOLR-6571
> Project: Solr
> Issue Type: Improvement
> Components: SolrCloud
>Reporter: Timothy Potter
>
> When implementing SOLR-6249, [~noble.paul] points out that
> ZkIndexSchemaReader doesn't watch the managed schema znode correctly and
> should use a strategy similar to what ZkStateReader does so that watchers
> persist across zk client connection failures:
> The correct example is the constructor of ZkStateReader
>zkClient = new SolrZkClient(zkServerAddress, zkClientTimeout,
> zkClientConnectTimeout,
> // on reconnect, reload cloud info
> new OnReconnect() {
> //implement stuff here
> }
> });
> this ensures that the watchers are persisted across reconnect.
> We need a watch support class to help developers implement watchers correctly
> instead of having some code do it correctly and other code not.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[ANNOUNCE] Apache Solr 4.10.1 released
September 2014, Apache Solr™ 4.10.1 available The Lucene PMC is pleased to announce the release of Apache Solr 4.10.1 Solr is the popular, blazing fast, open source NoSQL search platform from the Apache Lucene project. Its major features include powerful full-text search, hit highlighting, faceted search, dynamic clustering, database integration, rich document (e.g., Word, PDF) handling, and geospatial search. Solr is highly scalable, providing fault tolerant distributed search and indexing, and powers the search and navigation features of many of the world's largest internet sites. Solr 4.10.1 is available for immediate download at: http://lucene.apache.org/solr/mirrors-solr-latest-redir.html Solr 4.10.1 includes 6 bug fixes, as well as Lucene 4.10.1 and its 7 bug fixes. See the CHANGES.txt file included with the release for a full list of changes and further details. Please report any feedback to the mailing lists (http://lucene.apache.org/solr/discussion.html) Note: The Apache Software Foundation uses an extensive mirroring network for distributing releases. It is possible that the mirror you are using may not have replicated the release yet. If that is the case, please try another mirror. This also goes for Maven access. Mike McCandless http://blog.mikemccandless.com - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[ANNOUNCE] Apache Lucene 4.10.1 released
September 2014, Apache Lucene™ 4.10.1 available The Lucene PMC is pleased to announce the release of Apache Lucene 4.10.1 Apache Lucene is a high-performance, full-featured text search engine library written entirely in Java. It is a technology suitable for nearly any application that requires full-text search, especially cross-platform. The release is available for immediate download at: http://lucene.apache.org/core/mirrors-core-latest-redir.html Lucene 4.10.1 includes 7 bug fixes. See the CHANGES.txt file included with the release for a full list of changes and further details. Please report any feedback to the mailing lists (http://lucene.apache.org/core/discussion.html) Note: The Apache Software Foundation uses an extensive mirroring network for distributing releases. It is possible that the mirror you are using may not have replicated the release yet. If that is the case, please try another mirror. This also goes for Maven access. Mike McCandless http://blog.mikemccandless.com - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6571) Need a znode watcher support class
[
https://issues.apache.org/jira/browse/SOLR-6571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14152173#comment-14152173
]
Ramkumar Aiyengar commented on SOLR-6571:
-
FWIW, I still think this is an interesting idea. SolrZkClient already wraps all
watchers, so probably that should store and fire all watchers on expiry. Most
things (except when you use it to do things like trigger recovery) shouldn't
really worry about expiry, having such code exposes us to rare edge cases which
happen only when some zk misconfiguration or issue happens.
> Need a znode watcher support class
> --
>
> Key: SOLR-6571
> URL: https://issues.apache.org/jira/browse/SOLR-6571
> Project: Solr
> Issue Type: Improvement
> Components: SolrCloud
>Reporter: Timothy Potter
>
> When implementing SOLR-6249, [~noble.paul] points out that
> ZkIndexSchemaReader doesn't watch the managed schema znode correctly and
> should use a strategy similar to what ZkStateReader does so that watchers
> persist across zk client connection failures:
> The correct example is the constructor of ZkStateReader
>zkClient = new SolrZkClient(zkServerAddress, zkClientTimeout,
> zkClientConnectTimeout,
> // on reconnect, reload cloud info
> new OnReconnect() {
> //implement stuff here
> }
> });
> this ensures that the watchers are persisted across reconnect.
> We need a watch support class to help developers implement watchers correctly
> instead of having some code do it correctly and other code not.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6476) Create a bulk mode for schema API
[
https://issues.apache.org/jira/browse/SOLR-6476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14152117#comment-14152117
]
Noble Paul commented on SOLR-6476:
--
bq.One of the tests should have a non-trivial fieldtype definition, like at
least one analyzer.
It is not supported yet. Pls correct me if I am wrong. does the current REST
API support it?
bq.Most of the add*() javadocs in IndexSchema say that persistence always
happens, but it doesn't if persist=false
It says "* @param persist to persist or not" . Isn't it right?
bq.SchemaManager has zero javadocs. More would be good.
It is not a class for others to use . But , it will be added
> Create a bulk mode for schema API
> -
>
> Key: SOLR-6476
> URL: https://issues.apache.org/jira/browse/SOLR-6476
> Project: Solr
> Issue Type: Bug
> Components: Schema and Analysis
>Reporter: Noble Paul
>Assignee: Noble Paul
> Labels: managedResource
> Attachments: SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch,
> SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch
>
>
> The current schema API does one operation at a time and the normal usecase is
> that users add multiple fields/fieldtypes/copyFields etc in one shot.
> example
> {code:javascript}
> curl http://localhost:8983/solr/collection1/schema -H
> 'Content-type:application/json' -d '{
> "add-field": {
> "name":"sell-by",
> "type":"tdate",
> "stored":true
> },
> "add-field":{
> "name":"catchall",
> "type":"text_general",
> "stored":false
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-6476) Create a bulk mode for schema API
[
https://issues.apache.org/jira/browse/SOLR-6476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14152063#comment-14152063
]
Noble Paul edited comment on SOLR-6476 at 9/29/14 6:55 PM:
---
bq.AFAICT, a bulk Schema API request with a new field using a new field type in
the same request can fail depending on the order of the specified operations
Yes it fails. Works as designed. This is exactly the same behavior you will see
in an RDBMS as well. Order is important
was (Author: noble.paul):
bq.AFAICT, a bulk Schema API request with a new field using a new field type in
the same request can fail depending on the order of the specified operations
Yes it fails. Works as designed. This is exactly the same behavior you will see
in an RDBMS as well
> Create a bulk mode for schema API
> -
>
> Key: SOLR-6476
> URL: https://issues.apache.org/jira/browse/SOLR-6476
> Project: Solr
> Issue Type: Bug
> Components: Schema and Analysis
>Reporter: Noble Paul
>Assignee: Noble Paul
> Labels: managedResource
> Attachments: SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch,
> SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch
>
>
> The current schema API does one operation at a time and the normal usecase is
> that users add multiple fields/fieldtypes/copyFields etc in one shot.
> example
> {code:javascript}
> curl http://localhost:8983/solr/collection1/schema -H
> 'Content-type:application/json' -d '{
> "add-field": {
> "name":"sell-by",
> "type":"tdate",
> "stored":true
> },
> "add-field":{
> "name":"catchall",
> "type":"text_general",
> "stored":false
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-4212) Support for facet pivot query for filtered count
[ https://issues.apache.org/jira/browse/SOLR-4212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Molloy updated SOLR-4212: --- Attachment: SOLR-4212-multiple-q.patch Add support for multiple queries per field. > Support for facet pivot query for filtered count > > > Key: SOLR-4212 > URL: https://issues.apache.org/jira/browse/SOLR-4212 > Project: Solr > Issue Type: Improvement > Components: search >Affects Versions: 4.0 >Reporter: Steve Molloy > Fix For: 4.9, Trunk > > Attachments: SOLR-4212-multiple-q.patch, SOLR-4212.patch, > SOLR-4212.patch, SOLR-4212.patch, patch-4212.txt > > > Facet pivot provide hierarchical support for computing data used to populate > a treemap or similar visualization. TreeMaps usually offer users extra > information by applying an overlay color on top of the existing square sizes > based on hierarchical counts. This second count is based on user choices, > representing, usually with gradient, the proportion of the square that fits > the user's choices. > The proposition is to add a facet.pivot.q parameter that would allow to > specify one or more queries (per field) that would be intersected with DocSet > used to calculate pivot count, stored in separate qcounts list, each entry > keyed by the query. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6476) Create a bulk mode for schema API
[
https://issues.apache.org/jira/browse/SOLR-6476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14152063#comment-14152063
]
Noble Paul commented on SOLR-6476:
--
bq.AFAICT, a bulk Schema API request with a new field using a new field type in
the same request can fail depending on the order of the specified operations
Yes it fails. Works as designed. This is exactly the same behavior you will see
in an RDBMS as well
> Create a bulk mode for schema API
> -
>
> Key: SOLR-6476
> URL: https://issues.apache.org/jira/browse/SOLR-6476
> Project: Solr
> Issue Type: Bug
> Components: Schema and Analysis
>Reporter: Noble Paul
>Assignee: Noble Paul
> Labels: managedResource
> Attachments: SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch,
> SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch
>
>
> The current schema API does one operation at a time and the normal usecase is
> that users add multiple fields/fieldtypes/copyFields etc in one shot.
> example
> {code:javascript}
> curl http://localhost:8983/solr/collection1/schema -H
> 'Content-type:application/json' -d '{
> "add-field": {
> "name":"sell-by",
> "type":"tdate",
> "stored":true
> },
> "add-field":{
> "name":"catchall",
> "type":"text_general",
> "stored":false
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-5938) New DocIdSet implementation with random write access
[
https://issues.apache.org/jira/browse/LUCENE-5938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14152040#comment-14152040
]
Michael McCandless commented on LUCENE-5938:
This is a nice change; I like simplifying MTQ's rewrite methods, to
push "sparse/dense" handling "lower". It's hacky now how the auto
method tries to switch from Query to FixedBitSet backed filter
depending on term/doc count...
Maybe fix "word" to be "long" instead? (In javadocs, variable names,
etc.). "word" is kind of low-level, platform dependent term ... I
found SparseFixedBitSet somewhat hard to understand :) Maybe rename
wordCount to nonZeroLongCount or something?
approximateCardinality / linear counting algorithm is cool ... do we
need to guard against zeroWords being 0? I guess this is allowed with
doubles in Java? At least add a comment explaining this corner case
"works", and I think add an explicit test case that sets a bit in
every long?
Spelling: in TestSparseBitSet.copyOf, change sensible -> sensitive
Maybe add some more comments around tricky parts of SparseFixedBitSet.
E.g., the different branches inside set? And, it looks strange doing
1L << i, but in fact the JVM/processor make that 1L << (i % 64). And
Iterator.currentOrNextDoc is scary looking... do we have enough tests
here?
I hit this test failure, which reproduces with the patch, but not on
trunk ... not sure if it's somehow related ... but the test case seems
buggy (it doesn't try to unwrap an ExecutionException to get the ACE
root cause ... yet I can't get it to fail on trunk w/ beasting):
{noformat}
NOTE: reproduce with: ant test -Dtestcase=TestReaderClosed
-Dtests.method=testReaderChaining -Dtests.seed=89DF4A597D3C8CB1
-Dtests.slow=true
-Dtests.linedocsfile=/lucenedata/hudson.enwiki.random.lines.txt
-Dtests.locale=sk -Dtests.timezone=America/Scoresbysund
-Dtests.file.encoding=UTF-8
NOTE: test params are:
codec=HighCompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=HIGH_COMPRESSION,
chunkSize=248),
termVectorsFormat=CompressingTermVectorsFormat(compressionMode=HIGH_COMPRESSION,
chunkSize=248)), sim=RandomSimilarityProvider(queryNorm=true,coord=yes): {},
locale=sk, timezone=America/Scoresbysund
NOTE: Linux 3.13.0-32-generic amd64/Oracle Corporation 1.7.0_55
(64-bit)/cpus=8,threads=1,free=453126896,total=518979584
NOTE: All tests run in this JVM: [TestReaderClosed]
Time: 0.485
There was 1 failure:
1) testReaderChaining(org.apache.lucene.index.TestReaderClosed)
java.lang.RuntimeException: java.util.concurrent.ExecutionException:
org.apache.lucene.store.AlreadyClosedException: this IndexReader cannot be used
anymore as one of its child readers was closed
at
__randomizedtesting.SeedInfo.seed([89DF4A597D3C8CB1:1EE91FD6CAA6CE7C]:0)
at
org.apache.lucene.search.IndexSearcher$ExecutionHelper.next(IndexSearcher.java:836)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:452)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:273)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:261)
at
org.apache.lucene.index.TestReaderClosed.testReaderChaining(TestReaderClosed.java:83)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at
com.carrotsearch.randomizedtesting.Threa
[jira] [Commented] (SOLR-6476) Create a bulk mode for schema API
[
https://issues.apache.org/jira/browse/SOLR-6476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14152025#comment-14152025
]
Steve Rowe commented on SOLR-6476:
--
Previously mentioned outstanding issues:
# One of the tests should have a non-trivial fieldtype definition, like at
least one analyzer.
# SchemaManager has lots of non-generic collections - I looked at a couple, and
they could be generified - maybe they all could?
# Most of the {{add\*()}} javadocs in {{IndexSchema}} say that persistence
always happens, but it doesn't if persist=false
New things I noticed today:
# {{SchemaManager}} has zero javadocs. More would be good.
# AFAICT, a bulk Schema API request with a new field using a new field type in
the same request can fail depending on the order of the specified operations,
e.g. this will fail because "newtype" won't exist when {{SchemaManager}} tries
to add "newfield" (caveat: untested):
\\
{code:javascript}
{
"add-field" : { "name":"newfield", "type":"newtype" },
"add-field-type" : { "name":"newtype", "class":"solr.StrField" }
}
{code}
> Create a bulk mode for schema API
> -
>
> Key: SOLR-6476
> URL: https://issues.apache.org/jira/browse/SOLR-6476
> Project: Solr
> Issue Type: Bug
> Components: Schema and Analysis
>Reporter: Noble Paul
>Assignee: Noble Paul
> Labels: managedResource
> Attachments: SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch,
> SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch
>
>
> The current schema API does one operation at a time and the normal usecase is
> that users add multiple fields/fieldtypes/copyFields etc in one shot.
> example
> {code:javascript}
> curl http://localhost:8983/solr/collection1/schema -H
> 'Content-type:application/json' -d '{
> "add-field": {
> "name":"sell-by",
> "type":"tdate",
> "stored":true
> },
> "add-field":{
> "name":"catchall",
> "type":"text_general",
> "stored":false
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[JENKINS] Lucene-Solr-5.x-Linux (64bit/jdk1.8.0_20) - Build # 11198 - Still Failing!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-5.x-Linux/11198/ Java: 64bit/jdk1.8.0_20 -XX:-UseCompressedOops -XX:+UseG1GC 2 tests failed. FAILED: org.apache.solr.cloud.CloudExitableDirectoryReaderTest.testDistribSearch Error Message: no exception matching expected: 400: Request took too long during query expansion. Terminating request. Stack Trace: java.lang.AssertionError: no exception matching expected: 400: Request took too long during query expansion. Terminating request. at __randomizedtesting.SeedInfo.seed([370D36ADAF140588:B6EBB8B5D84B65B4]:0) at org.junit.Assert.fail(Assert.java:93) at org.apache.solr.cloud.CloudExitableDirectoryReaderTest.assertFail(CloudExitableDirectoryReaderTest.java:101) at org.apache.solr.cloud.CloudExitableDirectoryReaderTest.doTimeoutTests(CloudExitableDirectoryReaderTest.java:81) at org.apache.solr.cloud.CloudExitableDirectoryReaderTest.doTest(CloudExitableDirectoryReaderTest.java:54) at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836) at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738) at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.ut
[jira] [Commented] (SOLR-6476) Create a bulk mode for schema API
[
https://issues.apache.org/jira/browse/SOLR-6476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14151974#comment-14151974
]
Steve Rowe commented on SOLR-6476:
--
bq. The only outstanding issue AFAIK is the no:of reties. whether it should be
finite or infinite.
(There are other outstanding issues - I'll list them in another comment after
this one.)
I still think continuous retrying when there are competing updates is the right
thing to do.
How about this: in SOLR-6249, [~thelabdude] added request param
{{updateTimeoutSecs}} to fail Schema API requests unless they succeed within
the given time. We could add checking of this timeout to the update-retry loop
if it's provided, but when it's not, we allow the update-retry loop to continue
ad infinitum.
In any case, the patch on this issue needs to be changed to make bulk Schema
API requests aware of the new {{updateTimeoutSecs}} param and perform the same
all-replicas-in-sync check that the other Schema API methods now have.
> Create a bulk mode for schema API
> -
>
> Key: SOLR-6476
> URL: https://issues.apache.org/jira/browse/SOLR-6476
> Project: Solr
> Issue Type: Bug
> Components: Schema and Analysis
>Reporter: Noble Paul
>Assignee: Noble Paul
> Labels: managedResource
> Attachments: SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch,
> SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch
>
>
> The current schema API does one operation at a time and the normal usecase is
> that users add multiple fields/fieldtypes/copyFields etc in one shot.
> example
> {code:javascript}
> curl http://localhost:8983/solr/collection1/schema -H
> 'Content-type:application/json' -d '{
> "add-field": {
> "name":"sell-by",
> "type":"tdate",
> "stored":true
> },
> "add-field":{
> "name":"catchall",
> "type":"text_general",
> "stored":false
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[JENKINS] Lucene-Solr-Tests-5.x-Java7 - Build # 2142 - Still Failing
Build: https://builds.apache.org/job/Lucene-Solr-Tests-5.x-Java7/2142/ 1 tests failed. REGRESSION: org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.testDistribSearch Error Message: No live SolrServers available to handle this request:[https://127.0.0.1:30539/yec/na, https://127.0.0.1:30475/yec/na, https://127.0.0.1:30484/yec/na] Stack Trace: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://127.0.0.1:30539/yec/na, https://127.0.0.1:30475/yec/na, https://127.0.0.1:30484/yec/na] at __randomizedtesting.SeedInfo.seed([18C89EF967239DBA:992E10E1107CFD86]:0) at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:322) at org.apache.solr.client.solrj.impl.CloudSolrServer.sendRequest(CloudSolrServer.java:880) at org.apache.solr.client.solrj.impl.CloudSolrServer.requestWithRetryOnStaleState(CloudSolrServer.java:658) at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:601) at org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.removeAndWaitForLastReplicaGone(DeleteLastCustomShardedReplicaTest.java:117) at org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.doTest(DeleteLastCustomShardedReplicaTest.java:107) at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836) at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738) at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.a
[jira] [Commented] (SOLR-6249) Schema API changes return success before all cores are updated
[ https://issues.apache.org/jira/browse/SOLR-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14151926#comment-14151926 ] Timothy Potter commented on SOLR-6249: -- I'll commit another fix to address the watcher re-connect issue for ZkIndexSchemaReader. > Schema API changes return success before all cores are updated > -- > > Key: SOLR-6249 > URL: https://issues.apache.org/jira/browse/SOLR-6249 > Project: Solr > Issue Type: Improvement > Components: Schema and Analysis, SolrCloud >Reporter: Gregory Chanan >Assignee: Timothy Potter > Attachments: SOLR-6249.patch, SOLR-6249.patch, SOLR-6249.patch > > > See SOLR-6137 for more details. > The basic issue is that Schema API changes return success when the first core > is updated, but other cores asynchronously read the updated schema from > ZooKeeper. > So a client application could make a Schema API change and then index some > documents based on the new schema that may fail on other nodes. > Possible fixes: > 1) Make the Schema API calls synchronous > 2) Give the client some ability to track the state of the schema. They can > already do this to a certain extent by checking the Schema API on all the > replicas and verifying that the field has been added, though this is pretty > cumbersome. Maybe it makes more sense to do this sort of thing on the > collection level, i.e. Schema API changes return the zk version to the > client. We add an API to return the current zk version. On a replica, if > the zk version is >= the version the client has, the client knows that > replica has at least seen the schema change. We could also provide an API to > do the distribution and checking across the different replicas of the > collection so that clients don't need ot do that themselves. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Closed] (SOLR-6571) Need a znode watcher support class
[
https://issues.apache.org/jira/browse/SOLR-6571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timothy Potter closed SOLR-6571.
Resolution: Won't Fix
Looks like ZkIndexSchemaReader is the only place where this isn't done
correctly so there's not really any general problem here. Will fix the schema
reader as part of SOLR-6249
> Need a znode watcher support class
> --
>
> Key: SOLR-6571
> URL: https://issues.apache.org/jira/browse/SOLR-6571
> Project: Solr
> Issue Type: Improvement
> Components: SolrCloud
>Reporter: Timothy Potter
>
> When implementing SOLR-6249, [~noble.paul] points out that
> ZkIndexSchemaReader doesn't watch the managed schema znode correctly and
> should use a strategy similar to what ZkStateReader does so that watchers
> persist across zk client connection failures:
> The correct example is the constructor of ZkStateReader
>zkClient = new SolrZkClient(zkServerAddress, zkClientTimeout,
> zkClientConnectTimeout,
> // on reconnect, reload cloud info
> new OnReconnect() {
> //implement stuff here
> }
> });
> this ensures that the watchers are persisted across reconnect.
> We need a watch support class to help developers implement watchers correctly
> instead of having some code do it correctly and other code not.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.9.0-ea-b28) - Build # 11350 - Still Failing!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11350/ Java: 64bit/jdk1.9.0-ea-b28 -XX:+UseCompressedOops -XX:+UseParallelGC 3 tests failed. REGRESSION: org.apache.solr.cloud.DeleteReplicaTest.testDistribSearch Error Message: No live SolrServers available to handle this request:[http://127.0.0.1:47512/vqqpo, http://127.0.0.1:59264/vqqpo, http://127.0.0.1:59549/vqqpo, http://127.0.0.1:46014/vqqpo, http://127.0.0.1:56419/vqqpo] Stack Trace: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:47512/vqqpo, http://127.0.0.1:59264/vqqpo, http://127.0.0.1:59549/vqqpo, http://127.0.0.1:46014/vqqpo, http://127.0.0.1:56419/vqqpo] at __randomizedtesting.SeedInfo.seed([48532332BF07798A:C9B5AD2AC85819B6]:0) at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:322) at org.apache.solr.client.solrj.impl.CloudSolrServer.sendRequest(CloudSolrServer.java:880) at org.apache.solr.client.solrj.impl.CloudSolrServer.requestWithRetryOnStaleState(CloudSolrServer.java:658) at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:601) at org.apache.solr.cloud.DeleteReplicaTest.removeAndWaitForReplicaGone(DeleteReplicaTest.java:171) at org.apache.solr.cloud.DeleteReplicaTest.deleteLiveReplicaTest(DeleteReplicaTest.java:144) at org.apache.solr.cloud.DeleteReplicaTest.doTest(DeleteReplicaTest.java:88) at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869) at sun.reflect.GeneratedMethodAccessor39.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:484) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836) at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738) at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrot
[jira] [Commented] (SOLR-6249) Schema API changes return success before all cores are updated
[ https://issues.apache.org/jira/browse/SOLR-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14151854#comment-14151854 ] Timothy Potter commented on SOLR-6249: -- Ok fair enough clearly the implementer of ZkIndexSchemaReader (not me) missed that day. > Schema API changes return success before all cores are updated > -- > > Key: SOLR-6249 > URL: https://issues.apache.org/jira/browse/SOLR-6249 > Project: Solr > Issue Type: Improvement > Components: Schema and Analysis, SolrCloud >Reporter: Gregory Chanan >Assignee: Timothy Potter > Attachments: SOLR-6249.patch, SOLR-6249.patch, SOLR-6249.patch > > > See SOLR-6137 for more details. > The basic issue is that Schema API changes return success when the first core > is updated, but other cores asynchronously read the updated schema from > ZooKeeper. > So a client application could make a Schema API change and then index some > documents based on the new schema that may fail on other nodes. > Possible fixes: > 1) Make the Schema API calls synchronous > 2) Give the client some ability to track the state of the schema. They can > already do this to a certain extent by checking the Schema API on all the > replicas and verifying that the field has been added, though this is pretty > cumbersome. Maybe it makes more sense to do this sort of thing on the > collection level, i.e. Schema API changes return the zk version to the > client. We add an API to return the current zk version. On a replica, if > the zk version is >= the version the client has, the client knows that > replica has at least seen the schema change. We could also provide an API to > do the distribution and checking across the different replicas of the > collection so that clients don't need ot do that themselves. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6249) Schema API changes return success before all cores are updated
[ https://issues.apache.org/jira/browse/SOLR-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14151861#comment-14151861 ] Mark Miller commented on SOLR-6249: --- bq. The last param says to retry on connection loss. Do you think we need to do more than this? Again, ZooKeeper 101. You can lose the ZK connection generally in two ways: * ConnectionLoss * SessionExpiration When you lose the connection due to ConnectionLoss, all the Watches remain. You have a couple choices - you can retry and hope the connection is reestablished, or do something else (see the leader election algorithm for an example of needing to do something else). Usually, you want to retry until you get a session expiration. That is what passing true as the final param does for you - it handles ConnectionLoss in the way you want to handle it 99% of the time. SessionExpiration means the connection was lost for too long. Watches do not span sessions. In this case, when you reconnect to ZooKeeper, it's pretty use case specific how you need to handle things, but at a minimum, it usually means re-creating any watches. > Schema API changes return success before all cores are updated > -- > > Key: SOLR-6249 > URL: https://issues.apache.org/jira/browse/SOLR-6249 > Project: Solr > Issue Type: Improvement > Components: Schema and Analysis, SolrCloud >Reporter: Gregory Chanan >Assignee: Timothy Potter > Attachments: SOLR-6249.patch, SOLR-6249.patch, SOLR-6249.patch > > > See SOLR-6137 for more details. > The basic issue is that Schema API changes return success when the first core > is updated, but other cores asynchronously read the updated schema from > ZooKeeper. > So a client application could make a Schema API change and then index some > documents based on the new schema that may fail on other nodes. > Possible fixes: > 1) Make the Schema API calls synchronous > 2) Give the client some ability to track the state of the schema. They can > already do this to a certain extent by checking the Schema API on all the > replicas and verifying that the field has been added, though this is pretty > cumbersome. Maybe it makes more sense to do this sort of thing on the > collection level, i.e. Schema API changes return the zk version to the > client. We add an API to return the current zk version. On a replica, if > the zk version is >= the version the client has, the client knows that > replica has at least seen the schema change. We could also provide an API to > do the distribution and checking across the different replicas of the > collection so that clients don't need ot do that themselves. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6249) Schema API changes return success before all cores are updated
[ https://issues.apache.org/jira/browse/SOLR-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14151860#comment-14151860 ] Steve Rowe commented on SOLR-6249: -- The fault is all mine. > Schema API changes return success before all cores are updated > -- > > Key: SOLR-6249 > URL: https://issues.apache.org/jira/browse/SOLR-6249 > Project: Solr > Issue Type: Improvement > Components: Schema and Analysis, SolrCloud >Reporter: Gregory Chanan >Assignee: Timothy Potter > Attachments: SOLR-6249.patch, SOLR-6249.patch, SOLR-6249.patch > > > See SOLR-6137 for more details. > The basic issue is that Schema API changes return success when the first core > is updated, but other cores asynchronously read the updated schema from > ZooKeeper. > So a client application could make a Schema API change and then index some > documents based on the new schema that may fail on other nodes. > Possible fixes: > 1) Make the Schema API calls synchronous > 2) Give the client some ability to track the state of the schema. They can > already do this to a certain extent by checking the Schema API on all the > replicas and verifying that the field has been added, though this is pretty > cumbersome. Maybe it makes more sense to do this sort of thing on the > collection level, i.e. Schema API changes return the zk version to the > client. We add an API to return the current zk version. On a replica, if > the zk version is >= the version the client has, the client knows that > replica has at least seen the schema change. We could also provide an API to > do the distribution and checking across the different replicas of the > collection so that clients don't need ot do that themselves. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6278) add admin/collections?action=DELETEREPLICA&core=... support, make collection=... and shard=... parameters optional
[
https://issues.apache.org/jira/browse/SOLR-6278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14151857#comment-14151857
]
Christine Poerschke commented on SOLR-6278:
---
Yes, the use of
{{admin/cores?action=UNLOAD&deleteInstanceDir=true&core=eoe1_shard8_replica1}}
would work for us as equivalent of
{{admin/collections?action=DELETEREPLICA&core=eoe1_shard8_replica1}}
The latter would merely hide the action=UNLOAD&deleteInstanceDir=true
implementation detail but not exposing core names at collections API level also
makes sense.
Our typical use case is to decommission all replicas under a given solr home or
another way to describe it would be everything living in a particular host/port
solr instance. Collection name, shard name, replica name, core name, they are
strictly speaking unnecessary detail and just a
{{host:port/solr/admin/something?action=DELETE_ALL_REPLICAS}}
could do the job or if one wanted to send admin commands not directly to the
solr instance being wiped out then a
{{host:port/solr/admin/something?action=DELETE_ALL_REPLICAS&host=...&port=...}}
So, in short, yes, a "won't fix" JIRA resolution would be fine for our use case.
> add admin/collections?action=DELETEREPLICA&core=... support, make
> collection=... and shard=... parameters optional
> --
>
> Key: SOLR-6278
> URL: https://issues.apache.org/jira/browse/SOLR-6278
> Project: Solr
> Issue Type: Improvement
>Reporter: Christine Poerschke
>Assignee: Erick Erickson
> Attachments: SOLR-6278.patch
>
>
> To add {{core=...}} as an alternative to {{replica=...}} way of identifying
> what is to be deleted, {{collection=...}} and {{shard=...}} to be optional
> provided the other parameters uniquely identify exactly one deletion target.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6249) Schema API changes return success before all cores are updated
[ https://issues.apache.org/jira/browse/SOLR-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14151850#comment-14151850 ] Mark Miller commented on SOLR-6249: --- This is not really tribal knowledge - it's ZooKeeper 101. Watchers do no persist across session timeouts and you need to re-create any watchers on reconnecting after a session timeout. > Schema API changes return success before all cores are updated > -- > > Key: SOLR-6249 > URL: https://issues.apache.org/jira/browse/SOLR-6249 > Project: Solr > Issue Type: Improvement > Components: Schema and Analysis, SolrCloud >Reporter: Gregory Chanan >Assignee: Timothy Potter > Attachments: SOLR-6249.patch, SOLR-6249.patch, SOLR-6249.patch > > > See SOLR-6137 for more details. > The basic issue is that Schema API changes return success when the first core > is updated, but other cores asynchronously read the updated schema from > ZooKeeper. > So a client application could make a Schema API change and then index some > documents based on the new schema that may fail on other nodes. > Possible fixes: > 1) Make the Schema API calls synchronous > 2) Give the client some ability to track the state of the schema. They can > already do this to a certain extent by checking the Schema API on all the > replicas and verifying that the field has been added, though this is pretty > cumbersome. Maybe it makes more sense to do this sort of thing on the > collection level, i.e. Schema API changes return the zk version to the > client. We add an API to return the current zk version. On a replica, if > the zk version is >= the version the client has, the client knows that > replica has at least seen the schema change. We could also provide an API to > do the distribution and checking across the different replicas of the > collection so that clients don't need ot do that themselves. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6249) Schema API changes return success before all cores are updated
[ https://issues.apache.org/jira/browse/SOLR-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14151842#comment-14151842 ] Timothy Potter commented on SOLR-6249: -- Sure but I'm tired of all this tribal knowledge around how to do something right in the code. We need a clear path for future coders to follow and it seems like a watcher support class is the right solution. I've opened SOLR-6571 please make comments there. > Schema API changes return success before all cores are updated > -- > > Key: SOLR-6249 > URL: https://issues.apache.org/jira/browse/SOLR-6249 > Project: Solr > Issue Type: Improvement > Components: Schema and Analysis, SolrCloud >Reporter: Gregory Chanan >Assignee: Timothy Potter > Attachments: SOLR-6249.patch, SOLR-6249.patch, SOLR-6249.patch > > > See SOLR-6137 for more details. > The basic issue is that Schema API changes return success when the first core > is updated, but other cores asynchronously read the updated schema from > ZooKeeper. > So a client application could make a Schema API change and then index some > documents based on the new schema that may fail on other nodes. > Possible fixes: > 1) Make the Schema API calls synchronous > 2) Give the client some ability to track the state of the schema. They can > already do this to a certain extent by checking the Schema API on all the > replicas and verifying that the field has been added, though this is pretty > cumbersome. Maybe it makes more sense to do this sort of thing on the > collection level, i.e. Schema API changes return the zk version to the > client. We add an API to return the current zk version. On a replica, if > the zk version is >= the version the client has, the client knows that > replica has at least seen the schema change. We could also provide an API to > do the distribution and checking across the different replicas of the > collection so that clients don't need ot do that themselves. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-6571) Need a znode watcher support class
Timothy Potter created SOLR-6571:
Summary: Need a znode watcher support class
Key: SOLR-6571
URL: https://issues.apache.org/jira/browse/SOLR-6571
Project: Solr
Issue Type: Improvement
Components: SolrCloud
Reporter: Timothy Potter
When implementing SOLR-6249, [~noble.paul] points out that ZkIndexSchemaReader
doesn't watch the managed schema znode correctly and should use a strategy
similar to what ZkStateReader does so that watchers persist across zk client
connection failures:
The correct example is the constructor of ZkStateReader
zkClient = new SolrZkClient(zkServerAddress, zkClientTimeout,
zkClientConnectTimeout,
// on reconnect, reload cloud info
new OnReconnect() {
//implement stuff here
}
});
this ensures that the watchers are persisted across reconnect.
We need a watch support class to help developers implement watchers correctly
instead of having some code do it correctly and other code not.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6249) Schema API changes return success before all cores are updated
[ https://issues.apache.org/jira/browse/SOLR-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14151818#comment-14151818 ] Noble Paul commented on SOLR-6249: -- I was just giving out an easy solution.This logic can be pushed to SolrZkClient and ZkStateReader and everyone else can use it from there. > Schema API changes return success before all cores are updated > -- > > Key: SOLR-6249 > URL: https://issues.apache.org/jira/browse/SOLR-6249 > Project: Solr > Issue Type: Improvement > Components: Schema and Analysis, SolrCloud >Reporter: Gregory Chanan >Assignee: Timothy Potter > Attachments: SOLR-6249.patch, SOLR-6249.patch, SOLR-6249.patch > > > See SOLR-6137 for more details. > The basic issue is that Schema API changes return success when the first core > is updated, but other cores asynchronously read the updated schema from > ZooKeeper. > So a client application could make a Schema API change and then index some > documents based on the new schema that may fail on other nodes. > Possible fixes: > 1) Make the Schema API calls synchronous > 2) Give the client some ability to track the state of the schema. They can > already do this to a certain extent by checking the Schema API on all the > replicas and verifying that the field has been added, though this is pretty > cumbersome. Maybe it makes more sense to do this sort of thing on the > collection level, i.e. Schema API changes return the zk version to the > client. We add an API to return the current zk version. On a replica, if > the zk version is >= the version the client has, the client knows that > replica has at least seen the schema change. We could also provide an API to > do the distribution and checking across the different replicas of the > collection so that clients don't need ot do that themselves. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6249) Schema API changes return success before all cores are updated
[ https://issues.apache.org/jira/browse/SOLR-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14151809#comment-14151809 ] Timothy Potter commented on SOLR-6249: -- Ok, that makes sense but seems like a generic issue vs. specific to this class. We could solve this specifically right now doing what you suggest but rather than polluting ZkStateReader with utility methods for setting watchers, I think we should have a generic watcher support class. ZkStateReader's job is to handle collection state right? It's not some generic ZK utility class that the rest of the code should use. There should be a ticket for refactoring any code that sets a watcher to do it correctly, including ZkIndexSchemaReader, or is this the only place in the code that doesn't set a watcher correctly? > Schema API changes return success before all cores are updated > -- > > Key: SOLR-6249 > URL: https://issues.apache.org/jira/browse/SOLR-6249 > Project: Solr > Issue Type: Improvement > Components: Schema and Analysis, SolrCloud >Reporter: Gregory Chanan >Assignee: Timothy Potter > Attachments: SOLR-6249.patch, SOLR-6249.patch, SOLR-6249.patch > > > See SOLR-6137 for more details. > The basic issue is that Schema API changes return success when the first core > is updated, but other cores asynchronously read the updated schema from > ZooKeeper. > So a client application could make a Schema API change and then index some > documents based on the new schema that may fail on other nodes. > Possible fixes: > 1) Make the Schema API calls synchronous > 2) Give the client some ability to track the state of the schema. They can > already do this to a certain extent by checking the Schema API on all the > replicas and verifying that the field has been added, though this is pretty > cumbersome. Maybe it makes more sense to do this sort of thing on the > collection level, i.e. Schema API changes return the zk version to the > client. We add an API to return the current zk version. On a replica, if > the zk version is >= the version the client has, the client knows that > replica has at least seen the schema change. We could also provide an API to > do the distribution and checking across the different replicas of the > collection so that clients don't need ot do that themselves. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[JENKINS] Lucene-Solr-trunk-MacOSX (64bit/jdk1.8.0) - Build # 1857 - Still Failing!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/1857/ Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC 1 tests failed. REGRESSION: org.apache.solr.TestDistributedMissingSort.testDistribSearch Error Message: Timeout occured while waiting response from server at: https://127.0.0.1:51348 Stack Trace: org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: https://127.0.0.1:51348 at __randomizedtesting.SeedInfo.seed([ABDE7FF6831C9F0F:2A38F1EEF443FF33]:0) at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:582) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211) at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:124) at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:116) at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:102) at org.apache.solr.BaseDistributedSearchTestCase.indexDoc(BaseDistributedSearchTestCase.java:438) at org.apache.solr.BaseDistributedSearchTestCase.indexr(BaseDistributedSearchTestCase.java:420) at org.apache.solr.TestDistributedMissingSort.index(TestDistributedMissingSort.java:90) at org.apache.solr.TestDistributedMissingSort.doTest(TestDistributedMissingSort.java:42) at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:875) at sun.reflect.GeneratedMethodAccessor61.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836) at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738) at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsea
[jira] [Commented] (SOLR-6261) Run ZK watch event callbacks in parallel to the event thread
[ https://issues.apache.org/jira/browse/SOLR-6261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14151800#comment-14151800 ] Ramkumar Aiyengar commented on SOLR-6261: - np, I have now opened SOLR-6570 for this, this is for pull #78 on GitHub. > Run ZK watch event callbacks in parallel to the event thread > > > Key: SOLR-6261 > URL: https://issues.apache.org/jira/browse/SOLR-6261 > Project: Solr > Issue Type: Improvement > Components: SolrCloud >Affects Versions: 4.9 >Reporter: Ramkumar Aiyengar >Assignee: Mark Miller >Priority: Minor > Fix For: 4.10, Trunk > > Attachments: thread-compare.jpg > > > Currently checking for leadership (due to the leader's ephemeral node going > away) happens in ZK's event thread. If there are many cores and all of them > are due leadership, then they would have to serially go through the two-way > sync and leadership takeover. > For tens of cores, this could mean 30-40s without leadership before the last > in the list even gets to start the leadership process. If the leadership > process happens in a separate thread, then the cores could all take over in > parallel. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6570) Run SolrZkClient session watch asynchronously
[
https://issues.apache.org/jira/browse/SOLR-6570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14151799#comment-14151799
]
ASF GitHub Bot commented on SOLR-6570:
--
Github user andyetitmoves commented on the pull request:
https://github.com/apache/lucene-solr/pull/78#issuecomment-57178936
Moved to SOLR-6570
> Run SolrZkClient session watch asynchronously
> -
>
> Key: SOLR-6570
> URL: https://issues.apache.org/jira/browse/SOLR-6570
> Project: Solr
> Issue Type: Improvement
> Components: SolrCloud
>Reporter: Ramkumar Aiyengar
>Priority: Minor
>
> Spin off from SOLR-6261.
> This kind of already happens because the only session watcher in
> {{ConnectionManager}} does it's processing async (changed in SOLR-5615), but
> this is more consistent and avoids the possibility that a second session
> watcher or a change to that code re-surfaces the issue again.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request: Wrap connectionManager in SolrZkClient c...
Github user andyetitmoves commented on the pull request: https://github.com/apache/lucene-solr/pull/78#issuecomment-57178936 Moved to SOLR-6570 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-6570) Run SolrZkClient session watch asynchronously
Ramkumar Aiyengar created SOLR-6570:
---
Summary: Run SolrZkClient session watch asynchronously
Key: SOLR-6570
URL: https://issues.apache.org/jira/browse/SOLR-6570
Project: Solr
Issue Type: Improvement
Components: SolrCloud
Reporter: Ramkumar Aiyengar
Priority: Minor
Spin off from SOLR-6261.
This kind of already happens because the only session watcher in
{{ConnectionManager}} does it's processing async (changed in SOLR-5615), but
this is more consistent and avoids the possibility that a second session
watcher or a change to that code re-surfaces the issue again.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6476) Create a bulk mode for schema API
[
https://issues.apache.org/jira/browse/SOLR-6476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14151790#comment-14151790
]
Noble Paul commented on SOLR-6476:
--
The only outstanding issue AFAIK is the no:of reties. whether it should be
finite or infinite. I believe all user interacting APIs should be coded to fail
gracefully. If we can resolve that I can commit this
> Create a bulk mode for schema API
> -
>
> Key: SOLR-6476
> URL: https://issues.apache.org/jira/browse/SOLR-6476
> Project: Solr
> Issue Type: Bug
> Components: Schema and Analysis
>Reporter: Noble Paul
>Assignee: Noble Paul
> Labels: managedResource
> Attachments: SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch,
> SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch, SOLR-6476.patch
>
>
> The current schema API does one operation at a time and the normal usecase is
> that users add multiple fields/fieldtypes/copyFields etc in one shot.
> example
> {code:javascript}
> curl http://localhost:8983/solr/collection1/schema -H
> 'Content-type:application/json' -d '{
> "add-field": {
> "name":"sell-by",
> "type":"tdate",
> "stored":true
> },
> "add-field":{
> "name":"catchall",
> "type":"text_general",
> "stored":false
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-6249) Schema API changes return success before all cores are updated
[
https://issues.apache.org/jira/browse/SOLR-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14151786#comment-14151786
]
Noble Paul edited comment on SOLR-6249 at 9/29/14 3:25 PM:
---
I'm afraid , it is not sufficient. It just retries if the connection is lost in
the getData() call. It does not take care of the case where the watcher is lost
because of a zkclient failure.
The correct example is the constructor of ZkStateReader
{code:java}
zkClient = new SolrZkClient(zkServerAddress, zkClientTimeout,
zkClientConnectTimeout,
// on reconnect, reload cloud info
new OnReconnect() {
//implement stuff here
}
});
{code}
this ensures that the watchers are persisted across reconnect.
This is why I suggested the ZkStateReader is the right place to set watchers
was (Author: noble.paul):
I'm afraid , it is not sufficient. It just retries if the connection is lost in
the getData() call. It does not take care of the case where the watcher is lost
because of a zkclient failure.
The correct example is the constructor of ZkStateReader
{code:java}
zkClient = new SolrZkClient(zkServerAddress, zkClientTimeout,
zkClientConnectTimeout,
// on reconnect, reload cloud info
new OnReconnect() {
//implement stuff here
}
});
{code}
this ensures that the watchers are persisted across reconnect.
> Schema API changes return success before all cores are updated
> --
>
> Key: SOLR-6249
> URL: https://issues.apache.org/jira/browse/SOLR-6249
> Project: Solr
> Issue Type: Improvement
> Components: Schema and Analysis, SolrCloud
>Reporter: Gregory Chanan
>Assignee: Timothy Potter
> Attachments: SOLR-6249.patch, SOLR-6249.patch, SOLR-6249.patch
>
>
> See SOLR-6137 for more details.
> The basic issue is that Schema API changes return success when the first core
> is updated, but other cores asynchronously read the updated schema from
> ZooKeeper.
> So a client application could make a Schema API change and then index some
> documents based on the new schema that may fail on other nodes.
> Possible fixes:
> 1) Make the Schema API calls synchronous
> 2) Give the client some ability to track the state of the schema. They can
> already do this to a certain extent by checking the Schema API on all the
> replicas and verifying that the field has been added, though this is pretty
> cumbersome. Maybe it makes more sense to do this sort of thing on the
> collection level, i.e. Schema API changes return the zk version to the
> client. We add an API to return the current zk version. On a replica, if
> the zk version is >= the version the client has, the client knows that
> replica has at least seen the schema change. We could also provide an API to
> do the distribution and checking across the different replicas of the
> collection so that clients don't need ot do that themselves.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6249) Schema API changes return success before all cores are updated
[
https://issues.apache.org/jira/browse/SOLR-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14151786#comment-14151786
]
Noble Paul commented on SOLR-6249:
--
I'm afraid , it is not sufficient. It just retries if the connection is lost in
the getData() call. It does not take care of the case where the watcher is lost
because of a zkclient failure.
The correct example is the constructor of ZkStateReader
{code:java}
zkClient = new SolrZkClient(zkServerAddress, zkClientTimeout,
zkClientConnectTimeout,
// on reconnect, reload cloud info
new OnReconnect() {
//implement stuff here
}
});
{code}
this ensures that the watchers are persisted across reconnect.
> Schema API changes return success before all cores are updated
> --
>
> Key: SOLR-6249
> URL: https://issues.apache.org/jira/browse/SOLR-6249
> Project: Solr
> Issue Type: Improvement
> Components: Schema and Analysis, SolrCloud
>Reporter: Gregory Chanan
>Assignee: Timothy Potter
> Attachments: SOLR-6249.patch, SOLR-6249.patch, SOLR-6249.patch
>
>
> See SOLR-6137 for more details.
> The basic issue is that Schema API changes return success when the first core
> is updated, but other cores asynchronously read the updated schema from
> ZooKeeper.
> So a client application could make a Schema API change and then index some
> documents based on the new schema that may fail on other nodes.
> Possible fixes:
> 1) Make the Schema API calls synchronous
> 2) Give the client some ability to track the state of the schema. They can
> already do this to a certain extent by checking the Schema API on all the
> replicas and verifying that the field has been added, though this is pretty
> cumbersome. Maybe it makes more sense to do this sort of thing on the
> collection level, i.e. Schema API changes return the zk version to the
> client. We add an API to return the current zk version. On a replica, if
> the zk version is >= the version the client has, the client knows that
> replica has at least seen the schema change. We could also provide an API to
> do the distribution and checking across the different replicas of the
> collection so that clients don't need ot do that themselves.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6511) Fencepost error in LeaderInitiatedRecoveryThread
[
https://issues.apache.org/jira/browse/SOLR-6511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14151781#comment-14151781
]
ASF subversion and git services commented on SOLR-6511:
---
Commit 1628203 from [~thelabdude] in branch 'dev/trunk'
[ https://svn.apache.org/r1628203 ]
SOLR-6511: adjust test logic to account for timing issues in zk session
expiration scenario.
> Fencepost error in LeaderInitiatedRecoveryThread
>
>
> Key: SOLR-6511
> URL: https://issues.apache.org/jira/browse/SOLR-6511
> Project: Solr
> Issue Type: Bug
>Reporter: Alan Woodward
>Assignee: Timothy Potter
> Attachments: SOLR-6511.patch, SOLR-6511.patch
>
>
> At line 106:
> {code}
> while (continueTrying && ++tries < maxTries) {
> {code}
> should be
> {code}
> while (continueTrying && ++tries <= maxTries) {
> {code}
> This is only a problem when called from DistributedUpdateProcessor, as it can
> have maxTries set to 1, which means the loop is never actually run.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6249) Schema API changes return success before all cores are updated
[
https://issues.apache.org/jira/browse/SOLR-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14151776#comment-14151776
]
Timothy Potter commented on SOLR-6249:
--
It uses ZkIndexSchemaReader, which under the covers calls:
{{byte[] data = zkClient.getData(managedSchemaPath, watcher, stat,
true); }}
The last param says to retry on connection loss. Do you think we need to do
more than this?
> Schema API changes return success before all cores are updated
> --
>
> Key: SOLR-6249
> URL: https://issues.apache.org/jira/browse/SOLR-6249
> Project: Solr
> Issue Type: Improvement
> Components: Schema and Analysis, SolrCloud
>Reporter: Gregory Chanan
>Assignee: Timothy Potter
> Attachments: SOLR-6249.patch, SOLR-6249.patch, SOLR-6249.patch
>
>
> See SOLR-6137 for more details.
> The basic issue is that Schema API changes return success when the first core
> is updated, but other cores asynchronously read the updated schema from
> ZooKeeper.
> So a client application could make a Schema API change and then index some
> documents based on the new schema that may fail on other nodes.
> Possible fixes:
> 1) Make the Schema API calls synchronous
> 2) Give the client some ability to track the state of the schema. They can
> already do this to a certain extent by checking the Schema API on all the
> replicas and verifying that the field has been added, though this is pretty
> cumbersome. Maybe it makes more sense to do this sort of thing on the
> collection level, i.e. Schema API changes return the zk version to the
> client. We add an API to return the current zk version. On a replica, if
> the zk version is >= the version the client has, the client knows that
> replica has at least seen the schema change. We could also provide an API to
> do the distribution and checking across the different replicas of the
> collection so that clients don't need ot do that themselves.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[JENKINS] Lucene-Solr-5.x-Linux (32bit/jdk1.8.0_40-ea-b04) - Build # 11197 - Still Failing!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-5.x-Linux/11197/
Java: 32bit/jdk1.8.0_40-ea-b04 -server -XX:+UseG1GC
All tests passed
Build Log:
[...truncated 4413 lines...]
[javac] Compiling 2 source files to
/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/build/memory/classes/test
[javac]
/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/memory/src/test/org/apache/lucene/index/memory/TestMemoryIndexAgainstRAMDir.java:434:
error: incompatible types: IndexableField cannot be converted to Field
[javac] for (Field field : nextDoc.getFields()) {
[javac] ^
[javac]
/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/memory/src/test/org/apache/lucene/index/memory/TestMemoryIndexAgainstRAMDir.java:445:
error: cannot find symbol
[javac] for (IndexableField field : doc.indexableFields()) {
[javac] ^
[javac] symbol: method indexableFields()
[javac] location: variable doc of type Document
[javac] Note:
/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/memory/src/test/org/apache/lucene/index/memory/TestMemoryIndexAgainstRAMDir.java
uses or overrides a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] 2 errors
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/build.xml:524: The following
error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/build.xml:472: The following
error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/build.xml:61: The following
error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/extra-targets.xml:39: The
following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/build.xml:447: The
following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/common-build.xml:2142:
The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/module-build.xml:55:
The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/common-build.xml:785:
The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/common-build.xml:799:
The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/common-build.xml:1867:
Compile failed; see the compiler error output for details.
Total time: 32 minutes 32 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.8.0_40-ea-b04 -server
-XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-5977) IW should safeguard against token streams returning invalid offsets for multi-valued fields
[
https://issues.apache.org/jira/browse/LUCENE-5977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14151763#comment-14151763
]
ASF subversion and git services commented on LUCENE-5977:
-
Commit 1628200 from [~rcmuir] in branch 'dev/branches/lucene_solr_4_10'
[ https://svn.apache.org/r1628200 ]
LUCENE-5977: Fix tokenstream safety checks in IndexWriter to work across
multi-valued fields
> IW should safeguard against token streams returning invalid offsets for
> multi-valued fields
> ---
>
> Key: LUCENE-5977
> URL: https://issues.apache.org/jira/browse/LUCENE-5977
> Project: Lucene - Core
> Issue Type: Bug
>Affects Versions: 4.9, 4.9.1, 4.10, 4.10.1
>Reporter: Dawid Weiss
>Priority: Minor
> Fix For: 4.10.2
>
> Attachments: LUCENE-5977.patch
>
>
> We have a custom token stream that emits information about offsets of each
> token. My (wrong) assumption was that for multi-valued fields a token
> stream's offset information is magically shifted, much like this is the case
> with positions. It's not the case -- offsets should be increasing and
> monotonic across all instances of a field, even if it has custom token
> streams. So, something like this:
> {code}
> doc.add(new Field("field-foo", new CannedTokenStream(token("bar", 1,
> 150, 160)), ftype));
> doc.add(new Field("field-foo", new CannedTokenStream(token("bar", 1,
> 50, 60)), ftype));
> {code}
> where the token function is defined as:
> {code}
> token(String image, int positionIncrement, int startOffset, int endOffset)
> {code}
> will result in either a cryptic assertion thrown from IW:
> {code}
> Exception in thread "main" java.lang.AssertionError
> at
> org.apache.lucene.index.FreqProxTermsWriterPerField.writeOffsets(FreqProxTermsWriterPerField.java:99)
> {code}
> or nothing (or a codec error) if run without assertions.
> Obviously returning non-shifted offsets from subsequent token streams makes
> little sense but I wonder if it could be made more explicit (or asserted)
> that offsets need to be increasing between multiple-values. The minimum is to
> add some documentation to OffsetAttribute. I don't know if offsets should be
> shifted automatically, as it's the case with positions -- this would change
> the semantics of existing tokenizers and filters which implement such
> shifting internally already.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-5977) IW should safeguard against token streams returning invalid offsets for multi-valued fields
[
https://issues.apache.org/jira/browse/LUCENE-5977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Muir resolved LUCENE-5977.
-
Resolution: Fixed
Fix Version/s: Trunk
5.0
Thank you for debugging this Dawid!
> IW should safeguard against token streams returning invalid offsets for
> multi-valued fields
> ---
>
> Key: LUCENE-5977
> URL: https://issues.apache.org/jira/browse/LUCENE-5977
> Project: Lucene - Core
> Issue Type: Bug
>Affects Versions: 4.9, 4.9.1, 4.10, 4.10.1
>Reporter: Dawid Weiss
>Priority: Minor
> Fix For: 4.10.2, 5.0, Trunk
>
> Attachments: LUCENE-5977.patch
>
>
> We have a custom token stream that emits information about offsets of each
> token. My (wrong) assumption was that for multi-valued fields a token
> stream's offset information is magically shifted, much like this is the case
> with positions. It's not the case -- offsets should be increasing and
> monotonic across all instances of a field, even if it has custom token
> streams. So, something like this:
> {code}
> doc.add(new Field("field-foo", new CannedTokenStream(token("bar", 1,
> 150, 160)), ftype));
> doc.add(new Field("field-foo", new CannedTokenStream(token("bar", 1,
> 50, 60)), ftype));
> {code}
> where the token function is defined as:
> {code}
> token(String image, int positionIncrement, int startOffset, int endOffset)
> {code}
> will result in either a cryptic assertion thrown from IW:
> {code}
> Exception in thread "main" java.lang.AssertionError
> at
> org.apache.lucene.index.FreqProxTermsWriterPerField.writeOffsets(FreqProxTermsWriterPerField.java:99)
> {code}
> or nothing (or a codec error) if run without assertions.
> Obviously returning non-shifted offsets from subsequent token streams makes
> little sense but I wonder if it could be made more explicit (or asserted)
> that offsets need to be increasing between multiple-values. The minimum is to
> add some documentation to OffsetAttribute. I don't know if offsets should be
> shifted automatically, as it's the case with positions -- this would change
> the semantics of existing tokenizers and filters which implement such
> shifting internally already.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[JENKINS-MAVEN] Lucene-Solr-Maven-trunk #1222: POMs out of sync
Build: https://builds.apache.org/job/Lucene-Solr-Maven-trunk/1222/ 1 tests failed. REGRESSION: org.apache.solr.cloud.CloudExitableDirectoryReaderTest.testDistribSearch Error Message: no exception matching expected: 400: Request took too long during query expansion. Terminating request. Stack Trace: java.lang.AssertionError: no exception matching expected: 400: Request took too long during query expansion. Terminating request. at __randomizedtesting.SeedInfo.seed([D39863050D5490D9:527EED1D7A0BF0E5]:0) at org.junit.Assert.fail(Assert.java:93) at org.apache.solr.cloud.CloudExitableDirectoryReaderTest.assertFail(CloudExitableDirectoryReaderTest.java:101) at org.apache.solr.cloud.CloudExitableDirectoryReaderTest.doTimeoutTests(CloudExitableDirectoryReaderTest.java:81) at org.apache.solr.cloud.CloudExitableDirectoryReaderTest.doTest(CloudExitableDirectoryReaderTest.java:54) Build Log: [...truncated 52905 lines...] BUILD FAILED /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Maven-trunk/build.xml:547: The following error occurred while executing this line: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Maven-trunk/build.xml:199: The following error occurred while executing this line: : Java returned: 1 Total time: 229 minutes 20 seconds Build step 'Invoke Ant' marked build as failure Recording test results Email was triggered for: Failure Sending email for trigger: Failure - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-4212) Support for facet pivot query for filtered count
[ https://issues.apache.org/jira/browse/SOLR-4212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Molloy updated SOLR-4212: --- Description: Facet pivot provide hierarchical support for computing data used to populate a treemap or similar visualization. TreeMaps usually offer users extra information by applying an overlay color on top of the existing square sizes based on hierarchical counts. This second count is based on user choices, representing, usually with gradient, the proportion of the square that fits the user's choices. The proposition is to add a facet.pivot.q parameter that would allow to specify one or more queries (per field) that would be intersected with DocSet used to calculate pivot count, stored in separate qcounts list, each entry keyed by the query. was: Facet pivot provide hierarchical support for computing data used to populate a treemap or similar visualization. TreeMaps usually offer users extra information by applying an overlay color on top of the existing square sizes based on hierarchical counts. This second count is based on user choices, representing, usually with gradient, the proportion of the square that fits the user's choices. The proposition is to add a facet.pivot.q parameter that would allow to specify a query (per field) that would be intersected with DocSet used to calculate pivot count, stored in separate q-count. > Support for facet pivot query for filtered count > > > Key: SOLR-4212 > URL: https://issues.apache.org/jira/browse/SOLR-4212 > Project: Solr > Issue Type: Improvement > Components: search >Affects Versions: 4.0 >Reporter: Steve Molloy > Fix For: 4.9, Trunk > > Attachments: SOLR-4212.patch, SOLR-4212.patch, SOLR-4212.patch, > patch-4212.txt > > > Facet pivot provide hierarchical support for computing data used to populate > a treemap or similar visualization. TreeMaps usually offer users extra > information by applying an overlay color on top of the existing square sizes > based on hierarchical counts. This second count is based on user choices, > representing, usually with gradient, the proportion of the square that fits > the user's choices. > The proposition is to add a facet.pivot.q parameter that would allow to > specify one or more queries (per field) that would be intersected with DocSet > used to calculate pivot count, stored in separate qcounts list, each entry > keyed by the query. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-5977) IW should safeguard against token streams returning invalid offsets for multi-valued fields
[
https://issues.apache.org/jira/browse/LUCENE-5977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14151743#comment-14151743
]
ASF subversion and git services commented on LUCENE-5977:
-
Commit 1628196 from [~rcmuir] in branch 'dev/branches/branch_5x'
[ https://svn.apache.org/r1628196 ]
LUCENE-5977: Fix tokenstream safety checks in IndexWriter to work across
multi-valued fields
> IW should safeguard against token streams returning invalid offsets for
> multi-valued fields
> ---
>
> Key: LUCENE-5977
> URL: https://issues.apache.org/jira/browse/LUCENE-5977
> Project: Lucene - Core
> Issue Type: Bug
>Affects Versions: 4.9, 4.9.1, 4.10, 4.10.1
>Reporter: Dawid Weiss
>Priority: Minor
> Fix For: 4.10.2
>
> Attachments: LUCENE-5977.patch
>
>
> We have a custom token stream that emits information about offsets of each
> token. My (wrong) assumption was that for multi-valued fields a token
> stream's offset information is magically shifted, much like this is the case
> with positions. It's not the case -- offsets should be increasing and
> monotonic across all instances of a field, even if it has custom token
> streams. So, something like this:
> {code}
> doc.add(new Field("field-foo", new CannedTokenStream(token("bar", 1,
> 150, 160)), ftype));
> doc.add(new Field("field-foo", new CannedTokenStream(token("bar", 1,
> 50, 60)), ftype));
> {code}
> where the token function is defined as:
> {code}
> token(String image, int positionIncrement, int startOffset, int endOffset)
> {code}
> will result in either a cryptic assertion thrown from IW:
> {code}
> Exception in thread "main" java.lang.AssertionError
> at
> org.apache.lucene.index.FreqProxTermsWriterPerField.writeOffsets(FreqProxTermsWriterPerField.java:99)
> {code}
> or nothing (or a codec error) if run without assertions.
> Obviously returning non-shifted offsets from subsequent token streams makes
> little sense but I wonder if it could be made more explicit (or asserted)
> that offsets need to be increasing between multiple-values. The minimum is to
> add some documentation to OffsetAttribute. I don't know if offsets should be
> shifted automatically, as it's the case with positions -- this would change
> the semantics of existing tokenizers and filters which implement such
> shifting internally already.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-2907) java.lang.IllegalArgumentException: deltaQuery has no column to resolve to declared primary key pk='ITEM_ID, CATEGORY_ID'
[
https://issues.apache.org/jira/browse/SOLR-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14151734#comment-14151734
]
Aleksandr Ivanov edited comment on SOLR-2907 at 9/29/14 2:37 PM:
-
Hi to everyone. I've had this problem. But I've solved it by adding name of
field (ITEM_ID and CATEGORY_ID) in schema.xml file on server to (just after
):
ITEM_ID
CATEGORY_ID
But I think the main problem is using wrong value for pk= in data-config.xml
file.
You should use TemplateTransformer, in root entity. Something like this:
...
was (Author: aleksandr.ivanov):
Hi to everyone. I've had this problem. But I've solved it by adding name of
field (ITEM_ID and CATEGORY_ID) in schema.xml file on server to (just after
):
ITEM_ID
CATEGORY_ID
But I think the main problem is using wrong value for pk= in data-config.xml
file.
You should use TemplateTransformer, in root entity. Something like this:
...
> java.lang.IllegalArgumentException: deltaQuery has no column to resolve to
> declared primary key pk='ITEM_ID, CATEGORY_ID'
> -
>
> Key: SOLR-2907
> URL: https://issues.apache.org/jira/browse/SOLR-2907
> Project: Solr
> Issue Type: Bug
> Components: contrib - DataImportHandler, Schema and Analysis
>Affects Versions: 3.4
>Reporter: Alan Baker
>
> We are using solr for our site and ran into this error in our own schema and
> I was able to reproduce it using the dataimport example code in the solr
> project. We do not get this error in SOLR 1.4 only started seeing it as we
> are working to upgrade to 3.4.0. It fails when delta-importing linked tables.
> Complete trace:
> Nov 18, 2011 5:21:02 PM org.apache.solr.handler.dataimport.DataImporter
> doDeltaImport
> SEVERE: Delta Import Failed
> java.lang.IllegalArgumentException: deltaQuery has no column to resolve to
> declared primary key pk='ITEM_ID, CATEGORY_ID'
> at
> org.apache.solr.handler.dataimport.DocBuilder.findMatchingPkColumn(DocBuilder.java:849)
> at
> org.apache.solr.handler.dataimport.DocBuilder.collectDelta(DocBuilder.java:900)
> at
> org.apache.solr.handler.dataimport.DocBuilder.collectDelta(DocBuilder.java:879)
> at
> org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:285)
> at
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:179)
> at
> org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:390)
> at
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:429)
> at
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:408)
> I used this dataConfig from the wiki on the data import:
>
> url="jdbc:hsqldb:./example-DIH/hsqldb/ex" user="sa" />
>
> query="select * from item"
> deltaImportQuery="select * from item where
> ID=='${dataimporter.delta.id}'"
> deltaQuery="select id from item where last_modified >
> '${dataimporter.last_index_time}'">
> query="select CATEGORY_ID from item_category where
> ITEM_ID='${item.ID}'"
> deltaQuery="select ITEM_ID, CATEGORY_ID from
> item_category where last_modified > '${dataimporter.last_index_time}'"
> parentDeltaQuery="select ID from item where
> ID=${item_category.ITEM_ID}">
>
>query="select DESCRIPTION as cat from category where
> ID = '${item_category.CATEGORY_ID}'"
> deltaQuery="select ID from category where
> last_modified > '${dataimporter.last_index_time}'"
> parentDeltaQuery="select ITEM_ID, CATEGORY_ID from
> item_category where CATEGORY_ID=${category.ID}"/>
>
>
>
>
>
> To reproduce use the data config from above and set the dataimport.properties
> last update times to before the last_modifed date in the example data. I my
> case I had to set the year to 1969. Then run a delta-import and the
> exception occurs. Thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-5911) Make MemoryIndex thread-safe for queries
[ https://issues.apache.org/jira/browse/LUCENE-5911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14151738#comment-14151738 ] ASF subversion and git services commented on LUCENE-5911: - Commit 1628194 from [~rcmuir] in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1628194 ] LUCENE-5911: fix compile > Make MemoryIndex thread-safe for queries > > > Key: LUCENE-5911 > URL: https://issues.apache.org/jira/browse/LUCENE-5911 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Alan Woodward >Priority: Minor > Fix For: 5.0, Trunk > > Attachments: LUCENE-5911.patch, LUCENE-5911.patch > > > We want to be able to run multiple queries at once over a MemoryIndex in > luwak (see > https://github.com/flaxsearch/luwak/commit/49a8fba5764020c2f0e4dc29d80d93abb0231191), > but this isn't possible with the current implementation. However, looking > at the code, it seems that it would be relatively simple to make MemoryIndex > thread-safe for reads/queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-2907) java.lang.IllegalArgumentException: deltaQuery has no column to resolve to declared primary key pk='ITEM_ID, CATEGORY_ID'
[
https://issues.apache.org/jira/browse/SOLR-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14151734#comment-14151734
]
Aleksandr Ivanov edited comment on SOLR-2907 at 9/29/14 2:36 PM:
-
Hi to everyone. I've had this problem. But I've solved it by adding name of
field (ITEM_ID and CATEGORY_ID) in schema.xml file on server to (just after
):
ITEM_ID
CATEGORY_ID
But I think the main problem is using wrong value for pk= in data-config.xml
file.
You should use TemplateTransformer, in root entity. Something like this:
...
was (Author: aleksandr.ivanov):
Hi to everyone. I've had this problem. But I've solved it by adding name of
field (ITEM_ID and CATEGORY_ID) in schema.xml file on server to (just after
):
ITEM_ID
CATEGORY_ID
But I think the main problem is using wrong value for pk= in data-config.xml
file.
You should use TemplateTransformer, in root entity. Something like this:
...
> java.lang.IllegalArgumentException: deltaQuery has no column to resolve to
> declared primary key pk='ITEM_ID, CATEGORY_ID'
> -
>
> Key: SOLR-2907
> URL: https://issues.apache.org/jira/browse/SOLR-2907
> Project: Solr
> Issue Type: Bug
> Components: contrib - DataImportHandler, Schema and Analysis
>Affects Versions: 3.4
>Reporter: Alan Baker
>
> We are using solr for our site and ran into this error in our own schema and
> I was able to reproduce it using the dataimport example code in the solr
> project. We do not get this error in SOLR 1.4 only started seeing it as we
> are working to upgrade to 3.4.0. It fails when delta-importing linked tables.
> Complete trace:
> Nov 18, 2011 5:21:02 PM org.apache.solr.handler.dataimport.DataImporter
> doDeltaImport
> SEVERE: Delta Import Failed
> java.lang.IllegalArgumentException: deltaQuery has no column to resolve to
> declared primary key pk='ITEM_ID, CATEGORY_ID'
> at
> org.apache.solr.handler.dataimport.DocBuilder.findMatchingPkColumn(DocBuilder.java:849)
> at
> org.apache.solr.handler.dataimport.DocBuilder.collectDelta(DocBuilder.java:900)
> at
> org.apache.solr.handler.dataimport.DocBuilder.collectDelta(DocBuilder.java:879)
> at
> org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:285)
> at
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:179)
> at
> org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:390)
> at
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:429)
> at
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:408)
> I used this dataConfig from the wiki on the data import:
>
> url="jdbc:hsqldb:./example-DIH/hsqldb/ex" user="sa" />
>
> query="select * from item"
> deltaImportQuery="select * from item where
> ID=='${dataimporter.delta.id}'"
> deltaQuery="select id from item where last_modified >
> '${dataimporter.last_index_time}'">
> query="select CATEGORY_ID from item_category where
> ITEM_ID='${item.ID}'"
> deltaQuery="select ITEM_ID, CATEGORY_ID from
> item_category where last_modified > '${dataimporter.last_index_time}'"
> parentDeltaQuery="select ID from item where
> ID=${item_category.ITEM_ID}">
>
>query="select DESCRIPTION as cat from category where
> ID = '${item_category.CATEGORY_ID}'"
> deltaQuery="select ID from category where
> last_modified > '${dataimporter.last_index_time}'"
> parentDeltaQuery="select ITEM_ID, CATEGORY_ID from
> item_category where CATEGORY_ID=${category.ID}"/>
>
>
>
>
>
> To reproduce use the data config from above and set the dataimport.properties
> last update times to before the last_modifed date in the example data. I my
> case I had to set the year to 1969. Then run a delta-import and the
> exception occurs. Thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_67) - Build # 11349 - Still Failing!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11349/ Java: 64bit/jdk1.7.0_67 -XX:+UseCompressedOops -XX:+UseSerialGC 1 tests failed. REGRESSION: org.apache.solr.cloud.CloudExitableDirectoryReaderTest.testDistribSearch Error Message: no exception matching expected: 400: Request took too long during query expansion. Terminating request. Stack Trace: java.lang.AssertionError: no exception matching expected: 400: Request took too long during query expansion. Terminating request. at __randomizedtesting.SeedInfo.seed([19D3E66939E34CC4:983568714EBC2CF8]:0) at org.junit.Assert.fail(Assert.java:93) at org.apache.solr.cloud.CloudExitableDirectoryReaderTest.assertFail(CloudExitableDirectoryReaderTest.java:101) at org.apache.solr.cloud.CloudExitableDirectoryReaderTest.doTimeoutTests(CloudExitableDirectoryReaderTest.java:75) at org.apache.solr.cloud.CloudExitableDirectoryReaderTest.doTest(CloudExitableDirectoryReaderTest.java:54) at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869) at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836) at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738) at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
[jira] [Commented] (SOLR-2907) java.lang.IllegalArgumentException: deltaQuery has no column to resolve to declared primary key pk='ITEM_ID, CATEGORY_ID'
[
https://issues.apache.org/jira/browse/SOLR-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14151734#comment-14151734
]
Aleksandr Ivanov commented on SOLR-2907:
Hi to everyone. I've had this problem. But I've solved it by adding name of
field (ITEM_ID and CATEGORY_ID) in schema.xml file on server to (just after
):
ITEM_ID
CATEGORY_ID
But I think the main problem is using wrong value for pk= in data-config.xml
file.
You should use TemplateTransformer, in root entity. Something like this:
...
> java.lang.IllegalArgumentException: deltaQuery has no column to resolve to
> declared primary key pk='ITEM_ID, CATEGORY_ID'
> -
>
> Key: SOLR-2907
> URL: https://issues.apache.org/jira/browse/SOLR-2907
> Project: Solr
> Issue Type: Bug
> Components: contrib - DataImportHandler, Schema and Analysis
>Affects Versions: 3.4
>Reporter: Alan Baker
>
> We are using solr for our site and ran into this error in our own schema and
> I was able to reproduce it using the dataimport example code in the solr
> project. We do not get this error in SOLR 1.4 only started seeing it as we
> are working to upgrade to 3.4.0. It fails when delta-importing linked tables.
> Complete trace:
> Nov 18, 2011 5:21:02 PM org.apache.solr.handler.dataimport.DataImporter
> doDeltaImport
> SEVERE: Delta Import Failed
> java.lang.IllegalArgumentException: deltaQuery has no column to resolve to
> declared primary key pk='ITEM_ID, CATEGORY_ID'
> at
> org.apache.solr.handler.dataimport.DocBuilder.findMatchingPkColumn(DocBuilder.java:849)
> at
> org.apache.solr.handler.dataimport.DocBuilder.collectDelta(DocBuilder.java:900)
> at
> org.apache.solr.handler.dataimport.DocBuilder.collectDelta(DocBuilder.java:879)
> at
> org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:285)
> at
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:179)
> at
> org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:390)
> at
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:429)
> at
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:408)
> I used this dataConfig from the wiki on the data import:
>
> url="jdbc:hsqldb:./example-DIH/hsqldb/ex" user="sa" />
>
> query="select * from item"
> deltaImportQuery="select * from item where
> ID=='${dataimporter.delta.id}'"
> deltaQuery="select id from item where last_modified >
> '${dataimporter.last_index_time}'">
> query="select CATEGORY_ID from item_category where
> ITEM_ID='${item.ID}'"
> deltaQuery="select ITEM_ID, CATEGORY_ID from
> item_category where last_modified > '${dataimporter.last_index_time}'"
> parentDeltaQuery="select ID from item where
> ID=${item_category.ITEM_ID}">
>
>query="select DESCRIPTION as cat from category where
> ID = '${item_category.CATEGORY_ID}'"
> deltaQuery="select ID from category where
> last_modified > '${dataimporter.last_index_time}'"
> parentDeltaQuery="select ITEM_ID, CATEGORY_ID from
> item_category where CATEGORY_ID=${category.ID}"/>
>
>
>
>
>
> To reproduce use the data config from above and set the dataimport.properties
> last update times to before the last_modifed date in the example data. I my
> case I had to set the year to 1969. Then run a delta-import and the
> exception occurs. Thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-5977) IW should safeguard against token streams returning invalid offsets for multi-valued fields
[
https://issues.apache.org/jira/browse/LUCENE-5977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14151730#comment-14151730
]
ASF subversion and git services commented on LUCENE-5977:
-
Commit 1628192 from [~rcmuir] in branch 'dev/trunk'
[ https://svn.apache.org/r1628192 ]
LUCENE-5977: Fix tokenstream safety checks in IndexWriter to work across
multi-valued fields
> IW should safeguard against token streams returning invalid offsets for
> multi-valued fields
> ---
>
> Key: LUCENE-5977
> URL: https://issues.apache.org/jira/browse/LUCENE-5977
> Project: Lucene - Core
> Issue Type: Bug
>Affects Versions: 4.9, 4.9.1, 4.10, 4.10.1
>Reporter: Dawid Weiss
>Priority: Minor
> Fix For: 4.10.2
>
> Attachments: LUCENE-5977.patch
>
>
> We have a custom token stream that emits information about offsets of each
> token. My (wrong) assumption was that for multi-valued fields a token
> stream's offset information is magically shifted, much like this is the case
> with positions. It's not the case -- offsets should be increasing and
> monotonic across all instances of a field, even if it has custom token
> streams. So, something like this:
> {code}
> doc.add(new Field("field-foo", new CannedTokenStream(token("bar", 1,
> 150, 160)), ftype));
> doc.add(new Field("field-foo", new CannedTokenStream(token("bar", 1,
> 50, 60)), ftype));
> {code}
> where the token function is defined as:
> {code}
> token(String image, int positionIncrement, int startOffset, int endOffset)
> {code}
> will result in either a cryptic assertion thrown from IW:
> {code}
> Exception in thread "main" java.lang.AssertionError
> at
> org.apache.lucene.index.FreqProxTermsWriterPerField.writeOffsets(FreqProxTermsWriterPerField.java:99)
> {code}
> or nothing (or a codec error) if run without assertions.
> Obviously returning non-shifted offsets from subsequent token streams makes
> little sense but I wonder if it could be made more explicit (or asserted)
> that offsets need to be increasing between multiple-values. The minimum is to
> add some documentation to OffsetAttribute. I don't know if offsets should be
> shifted automatically, as it's the case with positions -- this would change
> the semantics of existing tokenizers and filters which implement such
> shifting internally already.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6249) Schema API changes return success before all cores are updated
[ https://issues.apache.org/jira/browse/SOLR-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14151702#comment-14151702 ] Noble Paul commented on SOLR-6249: -- Does it take care of zkclient losing connections? I > Schema API changes return success before all cores are updated > -- > > Key: SOLR-6249 > URL: https://issues.apache.org/jira/browse/SOLR-6249 > Project: Solr > Issue Type: Improvement > Components: Schema and Analysis, SolrCloud >Reporter: Gregory Chanan >Assignee: Timothy Potter > Attachments: SOLR-6249.patch, SOLR-6249.patch, SOLR-6249.patch > > > See SOLR-6137 for more details. > The basic issue is that Schema API changes return success when the first core > is updated, but other cores asynchronously read the updated schema from > ZooKeeper. > So a client application could make a Schema API change and then index some > documents based on the new schema that may fail on other nodes. > Possible fixes: > 1) Make the Schema API calls synchronous > 2) Give the client some ability to track the state of the schema. They can > already do this to a certain extent by checking the Schema API on all the > replicas and verifying that the field has been added, though this is pretty > cumbersome. Maybe it makes more sense to do this sort of thing on the > collection level, i.e. Schema API changes return the zk version to the > client. We add an API to return the current zk version. On a replica, if > the zk version is >= the version the client has, the client knows that > replica has at least seen the schema change. We could also provide an API to > do the distribution and checking across the different replicas of the > collection so that clients don't need ot do that themselves. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6249) Schema API changes return success before all cores are updated
[ https://issues.apache.org/jira/browse/SOLR-6249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14151690#comment-14151690 ] ASF subversion and git services commented on SOLR-6249: --- Commit 1628181 from [~thelabdude] in branch 'dev/trunk' [ https://svn.apache.org/r1628181 ] SOLR-6249: Schema API changes return success before all cores are updated > Schema API changes return success before all cores are updated > -- > > Key: SOLR-6249 > URL: https://issues.apache.org/jira/browse/SOLR-6249 > Project: Solr > Issue Type: Improvement > Components: Schema and Analysis, SolrCloud >Reporter: Gregory Chanan >Assignee: Timothy Potter > Attachments: SOLR-6249.patch, SOLR-6249.patch, SOLR-6249.patch > > > See SOLR-6137 for more details. > The basic issue is that Schema API changes return success when the first core > is updated, but other cores asynchronously read the updated schema from > ZooKeeper. > So a client application could make a Schema API change and then index some > documents based on the new schema that may fail on other nodes. > Possible fixes: > 1) Make the Schema API calls synchronous > 2) Give the client some ability to track the state of the schema. They can > already do this to a certain extent by checking the Schema API on all the > replicas and verifying that the field has been added, though this is pretty > cumbersome. Maybe it makes more sense to do this sort of thing on the > collection level, i.e. Schema API changes return the zk version to the > client. We add an API to return the current zk version. On a replica, if > the zk version is >= the version the client has, the client knows that > replica has at least seen the schema change. We could also provide an API to > do the distribution and checking across the different replicas of the > collection so that clients don't need ot do that themselves. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-5938) New DocIdSet implementation with random write access
[
https://issues.apache.org/jira/browse/LUCENE-5938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adrien Grand updated LUCENE-5938:
-
Attachment: LUCENE-5938.patch
I updated the patch to recent trunk modifications and ran the benchmark again,
I think it is ready. In summary this patch:
- introduces a new doc-id set impl similar to FixedBitSet but which is much
faster in the sparse case and a bit slower in the dense case (between 1.5x and
4x slower according to benchmarks).
- introduces a doc-id set builder that supports random write access by
starting with a sparse bit set and upgrading to a dense FixedBitSet when the
cardinality of the set increases
- changes MultiTermQueryWrapperFilter and TermsFilter to use this new builder
- removes CONSTANT_SCORE_AUTO_REWRITE and makes CONSTANT_SCORE_FILTER_REWRITE
the default
For queries that match many documents ({{wikimedium10m.tasks}}, the builder
always ends up building a FixedBitSet), this new patch can be slower or faster
depending on the cost to iterate the matching terms (since they are enumerated
only once now) vs. the cost of building the doc-id set.
For queries that match few documents ({{low_freq.tasks}}, attached to this
issue), this new patch makes things faster since it just sets a couple of bits
in random order and then iterates over them instead of merging documents coming
from several other iterators on the fly using a priority queue.
Independently of the benchmarks, I think it's a good simplification to remove
the constant-score auto rewrite mode.
{noformat}
wikimedium10m.tasks
TaskQPS baseline StdDev QPS patch StdDev
Pct diff
IntNRQ8.79 (9.6%)8.41 (3.4%)
-4.3% ( -15% -9%)
Fuzzy2 60.83 (11.1%) 58.34 (8.7%)
-4.1% ( -21% - 17%)
OrNotHighMed 98.35 (13.8%) 97.12 (10.9%)
-1.3% ( -22% - 27%)
OrHighNotHigh 18.88 (13.7%) 18.71 (11.1%)
-0.9% ( -22% - 27%)
OrNotHighHigh 17.10 (13.4%) 17.03 (11.2%)
-0.4% ( -22% - 27%)
OrNotHighLow 126.52 (13.6%) 126.85 (10.9%)
0.3% ( -21% - 28%)
OrHighMed 76.90 (14.0%) 77.25 (11.4%)
0.5% ( -21% - 30%)
OrHighNotLow 41.29 (14.3%) 41.51 (12.4%)
0.5% ( -22% - 31%)
OrHighNotMed 57.70 (13.6%) 58.03 (11.6%)
0.6% ( -21% - 29%)
OrHighLow 73.14 (14.7%) 73.74 (12.0%)
0.8% ( -22% - 32%)
LowSloppyPhrase 127.22 (8.6%) 128.43 (3.8%)
1.0% ( -10% - 14%)
OrHighHigh 29.11 (15.1%) 29.41 (12.2%)
1.0% ( -22% - 33%)
HighSloppyPhrase 12.83 (10.4%) 13.02 (5.3%)
1.4% ( -12% - 19%)
Prefix3 113.92 (9.9%) 115.71 (2.4%)
1.6% ( -9% - 15%)
PKLookup 297.13 (9.2%) 302.03 (3.5%)
1.6% ( -10% - 15%)
MedSloppyPhrase 38.60 (8.8%) 39.26 (3.7%)
1.7% ( -9% - 15%)
AndHighHigh 71.39 (6.9%) 72.67 (0.9%)
1.8% ( -5% - 10%)
HighTerm 87.17 (7.9%) 88.85 (2.1%)
1.9% ( -7% - 12%)
MedPhrase 74.60 (9.3%) 76.10 (4.3%)
2.0% ( -10% - 17%)
LowPhrase 21.67 (9.6%) 22.12 (4.0%)
2.1% ( -10% - 17%)
AndHighMed 297.13 (9.4%) 303.73 (2.1%)
2.2% ( -8% - 15%)
HighPhrase 16.65 (8.2%) 17.04 (3.7%)
2.3% ( -8% - 15%)
HighSpanNear4.56 (10.7%)4.67 (6.1%)
2.4% ( -12% - 21%)
LowTerm 769.53 (7.8%) 788.24 (2.0%)
2.4% ( -6% - 13%)
AndHighLow 726.76 (10.6%) 744.51 (4.2%)
2.4% ( -11% - 19%)
MedSpanNear 65.27 (9.3%) 67.00 (3.2%)
2.6% ( -9% - 16%)
Wildcard 114.28 (9.1%) 118.05 (7.4%)
3.3% ( -12% - 21%)
LowSpanNear 174.75 (10.3%) 180.83 (3.5%)
3.5% ( -9% - 19%)
Fuzzy1 67.63 (11.3%) 70.08 (3.2%)
3.6% ( -9% - 20%)
MedTerm 209.00 (9.3%) 216.71 (1.9%)
3.7% ( -6% - 16%)
Respell 48.30 (10.6%) 50.58 (1.7%)
4.7% ( -6% - 18%)
low_freq.tasks
TaskQPS baseline StdDev QPS patch StdDev
Pct diff
PKLookup 278.50 (8.8%) 297.48 (13.9%)
6.8% ( -14% - 32%
[jira] [Commented] (SOLR-6261) Run ZK watch event callbacks in parallel to the event thread
[ https://issues.apache.org/jira/browse/SOLR-6261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14151675#comment-14151675 ] Mark Miller commented on SOLR-6261: --- Hmm, my fault. Should have reopened the issue then. We will have to make a new JIRA it sounds. > Run ZK watch event callbacks in parallel to the event thread > > > Key: SOLR-6261 > URL: https://issues.apache.org/jira/browse/SOLR-6261 > Project: Solr > Issue Type: Improvement > Components: SolrCloud >Affects Versions: 4.9 >Reporter: Ramkumar Aiyengar >Assignee: Mark Miller >Priority: Minor > Fix For: 4.10, Trunk > > Attachments: thread-compare.jpg > > > Currently checking for leadership (due to the leader's ephemeral node going > away) happens in ZK's event thread. If there are many cores and all of them > are due leadership, then they would have to serially go through the two-way > sync and leadership takeover. > For tens of cores, this could mean 30-40s without leadership before the last > in the list even gets to start the leadership process. If the leadership > process happens in a separate thread, then the cores could all take over in > parallel. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-6261) Run ZK watch event callbacks in parallel to the event thread
[ https://issues.apache.org/jira/browse/SOLR-6261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14151666#comment-14151666 ] Ramkumar Aiyengar edited comment on SOLR-6261 at 9/29/14 1:13 PM: -- bq. Ah, confused it with the second patch on the other issue when I glanced at it. Yeah, I can commit it here. Hey [~markrmil...@gmail.com], just checking since I don't see an automated message here, was this committed? Thanks! was (Author: andyetitmoves): > Ah, confused it with the second patch on the other issue when I glanced at > it. Yeah, I can commit it here. Hey [~markrmil...@gmail.com], just checking since I don't see an automated message here, was this committed? Thanks! > Run ZK watch event callbacks in parallel to the event thread > > > Key: SOLR-6261 > URL: https://issues.apache.org/jira/browse/SOLR-6261 > Project: Solr > Issue Type: Improvement > Components: SolrCloud >Affects Versions: 4.9 >Reporter: Ramkumar Aiyengar >Assignee: Mark Miller >Priority: Minor > Fix For: 4.10, Trunk > > Attachments: thread-compare.jpg > > > Currently checking for leadership (due to the leader's ephemeral node going > away) happens in ZK's event thread. If there are many cores and all of them > are due leadership, then they would have to serially go through the two-way > sync and leadership takeover. > For tens of cores, this could mean 30-40s without leadership before the last > in the list even gets to start the leadership process. If the leadership > process happens in a separate thread, then the cores could all take over in > parallel. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6261) Run ZK watch event callbacks in parallel to the event thread
[ https://issues.apache.org/jira/browse/SOLR-6261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14151666#comment-14151666 ] Ramkumar Aiyengar commented on SOLR-6261: - > Ah, confused it with the second patch on the other issue when I glanced at > it. Yeah, I can commit it here. Hey [~markrmil...@gmail.com], just checking since I don't see an automated message here, was this committed? Thanks! > Run ZK watch event callbacks in parallel to the event thread > > > Key: SOLR-6261 > URL: https://issues.apache.org/jira/browse/SOLR-6261 > Project: Solr > Issue Type: Improvement > Components: SolrCloud >Affects Versions: 4.9 >Reporter: Ramkumar Aiyengar >Assignee: Mark Miller >Priority: Minor > Fix For: 4.10, Trunk > > Attachments: thread-compare.jpg > > > Currently checking for leadership (due to the leader's ephemeral node going > away) happens in ZK's event thread. If there are many cores and all of them > are due leadership, then they would have to serially go through the two-way > sync and leadership takeover. > For tens of cores, this could mean 30-40s without leadership before the last > in the list even gets to start the leadership process. If the leadership > process happens in a separate thread, then the cores could all take over in > parallel. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[JENKINS] Lucene-Solr-5.x-Linux (32bit/jdk1.8.0_40-ea-b04) - Build # 11196 - Still Failing!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-5.x-Linux/11196/
Java: 32bit/jdk1.8.0_40-ea-b04 -server -XX:+UseParallelGC
All tests passed
Build Log:
[...truncated 4491 lines...]
[javac] Compiling 2 source files to
/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/build/memory/classes/test
[javac]
/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/memory/src/test/org/apache/lucene/index/memory/TestMemoryIndexAgainstRAMDir.java:434:
error: incompatible types: IndexableField cannot be converted to Field
[javac] for (Field field : nextDoc.getFields()) {
[javac] ^
[javac]
/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/memory/src/test/org/apache/lucene/index/memory/TestMemoryIndexAgainstRAMDir.java:445:
error: cannot find symbol
[javac] for (IndexableField field : doc.indexableFields()) {
[javac] ^
[javac] symbol: method indexableFields()
[javac] location: variable doc of type Document
[javac] Note:
/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/memory/src/test/org/apache/lucene/index/memory/TestMemoryIndexAgainstRAMDir.java
uses or overrides a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] 2 errors
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/build.xml:524: The following
error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/build.xml:472: The following
error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/build.xml:61: The following
error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/extra-targets.xml:39: The
following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/build.xml:447: The
following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/common-build.xml:2142:
The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/module-build.xml:55:
The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/common-build.xml:785:
The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/common-build.xml:799:
The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/common-build.xml:1867:
Compile failed; see the compiler error output for details.
Total time: 29 minutes 42 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.8.0_40-ea-b04 -server
-XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-5977) IW should safeguard against token streams returning invalid offsets for multi-valued fields
[
https://issues.apache.org/jira/browse/LUCENE-5977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14151639#comment-14151639
]
Adrien Grand commented on LUCENE-5977:
--
+1
> IW should safeguard against token streams returning invalid offsets for
> multi-valued fields
> ---
>
> Key: LUCENE-5977
> URL: https://issues.apache.org/jira/browse/LUCENE-5977
> Project: Lucene - Core
> Issue Type: Bug
>Affects Versions: 4.9, 4.9.1, 4.10, 4.10.1
>Reporter: Dawid Weiss
>Priority: Minor
> Fix For: 4.10.2
>
> Attachments: LUCENE-5977.patch
>
>
> We have a custom token stream that emits information about offsets of each
> token. My (wrong) assumption was that for multi-valued fields a token
> stream's offset information is magically shifted, much like this is the case
> with positions. It's not the case -- offsets should be increasing and
> monotonic across all instances of a field, even if it has custom token
> streams. So, something like this:
> {code}
> doc.add(new Field("field-foo", new CannedTokenStream(token("bar", 1,
> 150, 160)), ftype));
> doc.add(new Field("field-foo", new CannedTokenStream(token("bar", 1,
> 50, 60)), ftype));
> {code}
> where the token function is defined as:
> {code}
> token(String image, int positionIncrement, int startOffset, int endOffset)
> {code}
> will result in either a cryptic assertion thrown from IW:
> {code}
> Exception in thread "main" java.lang.AssertionError
> at
> org.apache.lucene.index.FreqProxTermsWriterPerField.writeOffsets(FreqProxTermsWriterPerField.java:99)
> {code}
> or nothing (or a codec error) if run without assertions.
> Obviously returning non-shifted offsets from subsequent token streams makes
> little sense but I wonder if it could be made more explicit (or asserted)
> that offsets need to be increasing between multiple-values. The minimum is to
> add some documentation to OffsetAttribute. I don't know if offsets should be
> shifted automatically, as it's the case with positions -- this would change
> the semantics of existing tokenizers and filters which implement such
> shifting internally already.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0_40-ea-b04) - Build # 11348 - Still Failing!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11348/ Java: 32bit/jdk1.8.0_40-ea-b04 -server -XX:+UseG1GC 1 tests failed. FAILED: org.apache.solr.client.solrj.SolrSchemalessExampleTest.testStreamingRequest Error Message: IOException occured when talking to server at: https://127.0.0.1:38679/solr/collection1 Stack Trace: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: https://127.0.0.1:38679/solr/collection1 at __randomizedtesting.SeedInfo.seed([CD54FC8755960C32:ABD8489779DA761]:0) at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:585) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211) at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:124) at org.apache.solr.client.solrj.SolrServer.commit(SolrServer.java:168) at org.apache.solr.client.solrj.SolrServer.commit(SolrServer.java:146) at org.apache.solr.client.solrj.SolrExampleTestsBase.testStreamingRequest(SolrExampleTestsBase.java:219) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836) at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738) at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at org.
[jira] [Updated] (LUCENE-5977) IW should safeguard against token streams returning invalid offsets for multi-valued fields
[
https://issues.apache.org/jira/browse/LUCENE-5977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Muir updated LUCENE-5977:
Issue Type: Bug (was: Improvement)
> IW should safeguard against token streams returning invalid offsets for
> multi-valued fields
> ---
>
> Key: LUCENE-5977
> URL: https://issues.apache.org/jira/browse/LUCENE-5977
> Project: Lucene - Core
> Issue Type: Bug
>Affects Versions: 4.9, 4.9.1, 4.10, 4.10.1
>Reporter: Dawid Weiss
>Priority: Minor
> Fix For: 4.10.2
>
> Attachments: LUCENE-5977.patch
>
>
> We have a custom token stream that emits information about offsets of each
> token. My (wrong) assumption was that for multi-valued fields a token
> stream's offset information is magically shifted, much like this is the case
> with positions. It's not the case -- offsets should be increasing and
> monotonic across all instances of a field, even if it has custom token
> streams. So, something like this:
> {code}
> doc.add(new Field("field-foo", new CannedTokenStream(token("bar", 1,
> 150, 160)), ftype));
> doc.add(new Field("field-foo", new CannedTokenStream(token("bar", 1,
> 50, 60)), ftype));
> {code}
> where the token function is defined as:
> {code}
> token(String image, int positionIncrement, int startOffset, int endOffset)
> {code}
> will result in either a cryptic assertion thrown from IW:
> {code}
> Exception in thread "main" java.lang.AssertionError
> at
> org.apache.lucene.index.FreqProxTermsWriterPerField.writeOffsets(FreqProxTermsWriterPerField.java:99)
> {code}
> or nothing (or a codec error) if run without assertions.
> Obviously returning non-shifted offsets from subsequent token streams makes
> little sense but I wonder if it could be made more explicit (or asserted)
> that offsets need to be increasing between multiple-values. The minimum is to
> add some documentation to OffsetAttribute. I don't know if offsets should be
> shifted automatically, as it's the case with positions -- this would change
> the semantics of existing tokenizers and filters which implement such
> shifting internally already.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-5977) IW should safeguard against token streams returning invalid offsets for multi-valued fields
[
https://issues.apache.org/jira/browse/LUCENE-5977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Muir updated LUCENE-5977:
Fix Version/s: 4.10.2
> IW should safeguard against token streams returning invalid offsets for
> multi-valued fields
> ---
>
> Key: LUCENE-5977
> URL: https://issues.apache.org/jira/browse/LUCENE-5977
> Project: Lucene - Core
> Issue Type: Bug
>Affects Versions: 4.9, 4.9.1, 4.10, 4.10.1
>Reporter: Dawid Weiss
>Priority: Minor
> Fix For: 4.10.2
>
> Attachments: LUCENE-5977.patch
>
>
> We have a custom token stream that emits information about offsets of each
> token. My (wrong) assumption was that for multi-valued fields a token
> stream's offset information is magically shifted, much like this is the case
> with positions. It's not the case -- offsets should be increasing and
> monotonic across all instances of a field, even if it has custom token
> streams. So, something like this:
> {code}
> doc.add(new Field("field-foo", new CannedTokenStream(token("bar", 1,
> 150, 160)), ftype));
> doc.add(new Field("field-foo", new CannedTokenStream(token("bar", 1,
> 50, 60)), ftype));
> {code}
> where the token function is defined as:
> {code}
> token(String image, int positionIncrement, int startOffset, int endOffset)
> {code}
> will result in either a cryptic assertion thrown from IW:
> {code}
> Exception in thread "main" java.lang.AssertionError
> at
> org.apache.lucene.index.FreqProxTermsWriterPerField.writeOffsets(FreqProxTermsWriterPerField.java:99)
> {code}
> or nothing (or a codec error) if run without assertions.
> Obviously returning non-shifted offsets from subsequent token streams makes
> little sense but I wonder if it could be made more explicit (or asserted)
> that offsets need to be increasing between multiple-values. The minimum is to
> add some documentation to OffsetAttribute. I don't know if offsets should be
> shifted automatically, as it's the case with positions -- this would change
> the semantics of existing tokenizers and filters which implement such
> shifting internally already.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-5977) IW should safeguard against token streams returning invalid offsets for multi-valued fields
[
https://issues.apache.org/jira/browse/LUCENE-5977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Muir updated LUCENE-5977:
Attachment: LUCENE-5977.patch
The bug is some validation checks rely on lastStartOffset/lastPosition, but
this is currently reset to 0 on each field instance.
We should instead track it across all instances of the field. I converted
Dawid's test to a case in TestPostingsOffsets and moved these two state
variables to FieldInvertState.
> IW should safeguard against token streams returning invalid offsets for
> multi-valued fields
> ---
>
> Key: LUCENE-5977
> URL: https://issues.apache.org/jira/browse/LUCENE-5977
> Project: Lucene - Core
> Issue Type: Improvement
>Affects Versions: 4.9, 4.9.1, 4.10, 4.10.1
>Reporter: Dawid Weiss
>Priority: Minor
> Attachments: LUCENE-5977.patch
>
>
> We have a custom token stream that emits information about offsets of each
> token. My (wrong) assumption was that for multi-valued fields a token
> stream's offset information is magically shifted, much like this is the case
> with positions. It's not the case -- offsets should be increasing and
> monotonic across all instances of a field, even if it has custom token
> streams. So, something like this:
> {code}
> doc.add(new Field("field-foo", new CannedTokenStream(token("bar", 1,
> 150, 160)), ftype));
> doc.add(new Field("field-foo", new CannedTokenStream(token("bar", 1,
> 50, 60)), ftype));
> {code}
> where the token function is defined as:
> {code}
> token(String image, int positionIncrement, int startOffset, int endOffset)
> {code}
> will result in either a cryptic assertion thrown from IW:
> {code}
> Exception in thread "main" java.lang.AssertionError
> at
> org.apache.lucene.index.FreqProxTermsWriterPerField.writeOffsets(FreqProxTermsWriterPerField.java:99)
> {code}
> or nothing (or a codec error) if run without assertions.
> Obviously returning non-shifted offsets from subsequent token streams makes
> little sense but I wonder if it could be made more explicit (or asserted)
> that offsets need to be increasing between multiple-values. The minimum is to
> add some documentation to OffsetAttribute. I don't know if offsets should be
> shifted automatically, as it's the case with positions -- this would change
> the semantics of existing tokenizers and filters which implement such
> shifting internally already.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-5977) IW should safeguard against token streams returning invalid offsets for multi-valued fields
[
https://issues.apache.org/jira/browse/LUCENE-5977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14151598#comment-14151598
]
Robert Muir commented on LUCENE-5977:
-
The bug is older than 4.9 actually: it existed in the previous indexing chain,
too.
I ran the test program against 4.8, even with assertingcodec, it just silently
"succeeds".
But if you add checkindex call:
{noformat}
Segments file=segments_1 numSegments=1 version=4.8 format=
1 of 1: name=_0 docCount=1
codec=Asserting
compound=false
numFiles=12
size (MB)=0.001
diagnostics = {timestamp=1411990327375, os=Linux,
os.version=3.13.0-24-generic, source=flush, lucene.version=4.8-SNAPSHOT,
os.arch=amd64, java.version=1.7.0_55, java.vendor=Oracle Corporation}
no deletions
test: open reader.OK
test: check integrity.OK
test: check live docs.OK
test: fields..OK [1 fields]
test: field norms.OK [0 fields]
test: terms, freq, prox...ERROR: java.lang.RuntimeException: term [75 73
65]: doc 0: pos 1: startOffset -2147483597 is out of bounds
java.lang.RuntimeException: term [75 73 65]: doc 0: pos 1: startOffset
-2147483597 is out of bounds
at org.apache.lucene.index.CheckIndex.checkFields(CheckIndex.java:944)
at org.apache.lucene.index.CheckIndex.testPostings(CheckIndex.java:1278)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:626)
at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:199)
at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:191)
at org.apache.lucene.OffsetIndexingBug.main(OffsetIndexingBug.java:48)
test: stored fields...OK [0 total field count; avg 0 fields per doc]
test: term vectorsERROR [vector term=[75 73 65] field=field-foo
doc=0: startOffset=51 differs from postings startOffset=-2147483597]
java.lang.RuntimeException: vector term=[75 73 65] field=field-foo doc=0:
startOffset=51 differs from postings startOffset=-2147483597
at
org.apache.lucene.index.CheckIndex.testTermVectors(CheckIndex.java:1748)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:632)
at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:199)
at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:191)
at org.apache.lucene.OffsetIndexingBug.main(OffsetIndexingBug.java:48)
test: docvalues...OK [0 docvalues fields; 0 BINARY; 0 NUMERIC; 0
SORTED; 0 SORTED_SET]
FAILED
WARNING: fixIndex() would remove reference to this segment; full exception:
java.lang.RuntimeException: Term Index test failed
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:641)
at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:199)
at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:191)
at org.apache.lucene.OffsetIndexingBug.main(OffsetIndexingBug.java:48)
WARNING: 1 broken segments (containing 1 documents) detected
{noformat}
> IW should safeguard against token streams returning invalid offsets for
> multi-valued fields
> ---
>
> Key: LUCENE-5977
> URL: https://issues.apache.org/jira/browse/LUCENE-5977
> Project: Lucene - Core
> Issue Type: Improvement
>Affects Versions: 4.9, 4.9.1, 4.10, 4.10.1
>Reporter: Dawid Weiss
>Priority: Minor
>
> We have a custom token stream that emits information about offsets of each
> token. My (wrong) assumption was that for multi-valued fields a token
> stream's offset information is magically shifted, much like this is the case
> with positions. It's not the case -- offsets should be increasing and
> monotonic across all instances of a field, even if it has custom token
> streams. So, something like this:
> {code}
> doc.add(new Field("field-foo", new CannedTokenStream(token("bar", 1,
> 150, 160)), ftype));
> doc.add(new Field("field-foo", new CannedTokenStream(token("bar", 1,
> 50, 60)), ftype));
> {code}
> where the token function is defined as:
> {code}
> token(String image, int positionIncrement, int startOffset, int endOffset)
> {code}
> will result in either a cryptic assertion thrown from IW:
> {code}
> Exception in thread "main" java.lang.AssertionError
> at
> org.apache.lucene.index.FreqProxTermsWriterPerField.writeOffsets(FreqProxTermsWriterPerField.java:99)
> {code}
> or nothing (or a codec error) if run without assertions.
> Obviously returning non-shifted offsets from subsequent token streams makes
> little sense but I wonder if it could be made more explicit (or asserted)
> that offsets need to be increasing between multiple-values. The minimum is to
> add some documentation to OffsetAttribute. I don't know if offsets should be
> shifte
[JENKINS] Lucene-Solr-NightlyTests-5.x - Build # 635 - Still Failing
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.x/635/ 3 tests failed. REGRESSION: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch Error Message: Captured an uncaught exception in thread: Thread[id=13668, name=qtp1148022701-13668, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest] Stack Trace: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=13668, name=qtp1148022701-13668, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest] Caused by: java.lang.OutOfMemoryError: unable to create new native thread at __randomizedtesting.SeedInfo.seed([C76C029776C01205]:0) at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:714) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1047) at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1312) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1339) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1323) at org.eclipse.jetty.server.ssl.SslSocketConnector$SslConnectorEndPoint.run(SslSocketConnector.java:665) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:745) FAILED: junit.framework.TestSuite.org.apache.solr.handler.TestReplicationHandler Error Message: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=259, name=coreLoadExecutor-163-thread-1, state=RUNNABLE, group=TGRP-TestReplicationHandler], registration stack trace below. Stack Trace: com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=259, name=coreLoadExecutor-163-thread-1, state=RUNNABLE, group=TGRP-TestReplicationHandler], registration stack trace below. at __randomizedtesting.SeedInfo.seed([C76C029776C01205]:0) at java.lang.Thread.getStackTrace(Thread.java:1589) at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:166) at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:688) at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:1231) at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1122) at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:1114) at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:47) at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:350) at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:275) at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:487) at org.apache.solr.core.SolrCore.(SolrCore.java:793) at org.apache.solr.core.SolrCore.(SolrCore.java:651) at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491) at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255) at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.AssertionError: Directory not closed: MockDirectoryWrapper(NIOFSDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-C76C029776C01205-001/index-NIOFSDirectory-028 lockFactory=NativeFSLockFactory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J2/temp/solr.handler.TestReplicationHandler-C76C029776C01205-001/index-NIOFSDirectory-028) at org.junit.Assert.fail(Assert.java:93) at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47) at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:699) at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:696) at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:183) at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:712) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) ... 1 more FAILED: org.apache.solr.cloud.CollectionsAPIDistributedZkTest.
[jira] [Commented] (LUCENE-5977) IW should safeguard against token streams returning invalid offsets for multi-valued fields
[
https://issues.apache.org/jira/browse/LUCENE-5977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14151584#comment-14151584
]
Dawid Weiss commented on LUCENE-5977:
-
I looked a bit more closely at the logic of updating offsets in the
DefaultIndexingChain and I see offsets *are* propagated for multiple fields,
here:
{code}
// trigger streams to perform end-of-stream operations
stream.end();
// TODO: maybe add some safety? then again, its already checked
// when we come back around to the field...
invertState.position +=
invertState.posIncrAttribute.getPositionIncrement();
invertState.offset += invertState.offsetAttribute.endOffset();
{code}
So the problem with my implementation was that it should have set offsets
properly in end(). I still feel this should be verified / asserted cleaner
somehow, so I'll leave this issue open looking for suggestions.
> IW should safeguard against token streams returning invalid offsets for
> multi-valued fields
> ---
>
> Key: LUCENE-5977
> URL: https://issues.apache.org/jira/browse/LUCENE-5977
> Project: Lucene - Core
> Issue Type: Improvement
>Affects Versions: 4.9, 4.9.1, 4.10, 4.10.1
>Reporter: Dawid Weiss
>Priority: Minor
>
> We have a custom token stream that emits information about offsets of each
> token. My (wrong) assumption was that for multi-valued fields a token
> stream's offset information is magically shifted, much like this is the case
> with positions. It's not the case -- offsets should be increasing and
> monotonic across all instances of a field, even if it has custom token
> streams. So, something like this:
> {code}
> doc.add(new Field("field-foo", new CannedTokenStream(token("bar", 1,
> 150, 160)), ftype));
> doc.add(new Field("field-foo", new CannedTokenStream(token("bar", 1,
> 50, 60)), ftype));
> {code}
> where the token function is defined as:
> {code}
> token(String image, int positionIncrement, int startOffset, int endOffset)
> {code}
> will result in either a cryptic assertion thrown from IW:
> {code}
> Exception in thread "main" java.lang.AssertionError
> at
> org.apache.lucene.index.FreqProxTermsWriterPerField.writeOffsets(FreqProxTermsWriterPerField.java:99)
> {code}
> or nothing (or a codec error) if run without assertions.
> Obviously returning non-shifted offsets from subsequent token streams makes
> little sense but I wonder if it could be made more explicit (or asserted)
> that offsets need to be increasing between multiple-values. The minimum is to
> add some documentation to OffsetAttribute. I don't know if offsets should be
> shifted automatically, as it's the case with positions -- this would change
> the semantics of existing tokenizers and filters which implement such
> shifting internally already.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[JENKINS] Lucene-Solr-5.x-Linux (32bit/jdk1.8.0_20) - Build # 11195 - Failure!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-5.x-Linux/11195/ Java: 32bit/jdk1.8.0_20 -server -XX:+UseSerialGC 2 tests failed. REGRESSION: org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.testDistribSearch Error Message: No live SolrServers available to handle this request:[http://127.0.0.1:54590, http://127.0.0.1:38802, http://127.0.0.1:53259] Stack Trace: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:54590, http://127.0.0.1:38802, http://127.0.0.1:53259] at __randomizedtesting.SeedInfo.seed([4FE08E3460AF700C:CE06002C17F01030]:0) at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:322) at org.apache.solr.client.solrj.impl.CloudSolrServer.sendRequest(CloudSolrServer.java:880) at org.apache.solr.client.solrj.impl.CloudSolrServer.requestWithRetryOnStaleState(CloudSolrServer.java:658) at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:601) at org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.removeAndWaitForLastReplicaGone(DeleteLastCustomShardedReplicaTest.java:117) at org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.doTest(DeleteLastCustomShardedReplicaTest.java:107) at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836) at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738) at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apach
[jira] [Resolved] (LUCENE-5911) Make MemoryIndex thread-safe for queries
[ https://issues.apache.org/jira/browse/LUCENE-5911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Woodward resolved LUCENE-5911. --- Resolution: Fixed Fix Version/s: Trunk 5.0 Assignee: Alan Woodward > Make MemoryIndex thread-safe for queries > > > Key: LUCENE-5911 > URL: https://issues.apache.org/jira/browse/LUCENE-5911 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Alan Woodward >Priority: Minor > Fix For: 5.0, Trunk > > Attachments: LUCENE-5911.patch, LUCENE-5911.patch > > > We want to be able to run multiple queries at once over a MemoryIndex in > luwak (see > https://github.com/flaxsearch/luwak/commit/49a8fba5764020c2f0e4dc29d80d93abb0231191), > but this isn't possible with the current implementation. However, looking > at the code, it seems that it would be relatively simple to make MemoryIndex > thread-safe for reads/queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-5911) Make MemoryIndex thread-safe for queries
[ https://issues.apache.org/jira/browse/LUCENE-5911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14151569#comment-14151569 ] ASF subversion and git services commented on LUCENE-5911: - Commit 1628155 from [~romseygeek] in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1628155 ] LUCENE-5911: Add MemoryIndex.freeze() to allow for thread-safe searching > Make MemoryIndex thread-safe for queries > > > Key: LUCENE-5911 > URL: https://issues.apache.org/jira/browse/LUCENE-5911 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Alan Woodward >Priority: Minor > Attachments: LUCENE-5911.patch, LUCENE-5911.patch > > > We want to be able to run multiple queries at once over a MemoryIndex in > luwak (see > https://github.com/flaxsearch/luwak/commit/49a8fba5764020c2f0e4dc29d80d93abb0231191), > but this isn't possible with the current implementation. However, looking > at the code, it seems that it would be relatively simple to make MemoryIndex > thread-safe for reads/queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-5911) Make MemoryIndex thread-safe for queries
[ https://issues.apache.org/jira/browse/LUCENE-5911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14151568#comment-14151568 ] ASF subversion and git services commented on LUCENE-5911: - Commit 1628154 from [~romseygeek] in branch 'dev/trunk' [ https://svn.apache.org/r1628154 ] LUCENE-5911: Add MemoryIndex.freeze() to allow for thread-safe searching > Make MemoryIndex thread-safe for queries > > > Key: LUCENE-5911 > URL: https://issues.apache.org/jira/browse/LUCENE-5911 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Alan Woodward >Priority: Minor > Attachments: LUCENE-5911.patch, LUCENE-5911.patch > > > We want to be able to run multiple queries at once over a MemoryIndex in > luwak (see > https://github.com/flaxsearch/luwak/commit/49a8fba5764020c2f0e4dc29d80d93abb0231191), > but this isn't possible with the current implementation. However, looking > at the code, it seems that it would be relatively simple to make MemoryIndex > thread-safe for reads/queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-5977) IW should safeguard against token streams returning invalid offsets for multi-valued fields
[
https://issues.apache.org/jira/browse/LUCENE-5977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14151567#comment-14151567
]
Dawid Weiss commented on LUCENE-5977:
-
A full example which shows the problem. Run it with -ea -- you'll get the
assertion. Run it without assertions and it'll pass.
{code}
import org.apache.lucene.analysis.Token;
import org.apache.lucene.analysis.core.WhitespaceAnalyzer;
import org.apache.lucene.codecs.Codec;
import org.apache.lucene.codecs.simpletext.SimpleTextCodec;
import org.apache.lucene.document.Document;
import org.apache.lucene.document.Field;
import org.apache.lucene.document.FieldType;
import org.apache.lucene.index.FieldInfo.IndexOptions;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.index.IndexWriterConfig;
import org.apache.lucene.store.Directory;
import org.apache.lucene.store.RAMDirectory;
import org.apache.lucene.util.Version;
import org.apache.lucene.analysis.CannedTokenStream;
public class OffsetIndexingBug {
public static void main(String[] args) throws Exception {
Codec.setDefault(new SimpleTextCodec());
Version version = Version.LUCENE_CURRENT;
IndexWriterConfig conf = new IndexWriterConfig(version, new
WhitespaceAnalyzer(version));
conf.setUseCompoundFile(false);
try (Directory d = new RAMDirectory()) {
try (IndexWriter iw = new IndexWriter(d, conf)) {
iw.deleteAll();
iw.commit();
Document doc = new Document();
FieldType ftype = new FieldType();
ftype.setIndexed(true);
ftype.setStored(false);
ftype.setOmitNorms(true);
ftype.setStoreTermVectors(true);
ftype.setStoreTermVectorPositions(true);
ftype.setStoreTermVectorOffsets(true);
ftype.setTokenized(true);
ftype.setIndexOptions(IndexOptions.DOCS_AND_FREQS_AND_POSITIONS_AND_OFFSETS);
ftype.freeze();
// note "use"'s offset is negative with respect to the first field
value.
doc.add(new Field("field-foo", new CannedTokenStream(token("use", 1,
150, 160)), ftype));
doc.add(new Field("field-foo", new CannedTokenStream(token("use", 1,
50, 60)), ftype));
iw.addDocument(doc);
}
}
}
private static Token token(String image, int positionIncrement, int soffset,
int eoffset) {
Token t = new Token();
t.setPositionIncrement(positionIncrement);
t.setOffset(soffset, eoffset);
t.append(image);
return t;
}
}
{code}
> IW should safeguard against token streams returning invalid offsets for
> multi-valued fields
> ---
>
> Key: LUCENE-5977
> URL: https://issues.apache.org/jira/browse/LUCENE-5977
> Project: Lucene - Core
> Issue Type: Improvement
>Affects Versions: 4.9, 4.9.1, 4.10, 4.10.1
>Reporter: Dawid Weiss
>Priority: Minor
>
> We have a custom token stream that emits information about offsets of each
> token. My (wrong) assumption was that for multi-valued fields a token
> stream's offset information is magically shifted, much like this is the case
> with positions. It's not the case -- offsets should be increasing and
> monotonic across all instances of a field, even if it has custom token
> streams. So, something like this:
> {code}
> doc.add(new Field("field-foo", new CannedTokenStream(token("bar", 1,
> 150, 160)), ftype));
> doc.add(new Field("field-foo", new CannedTokenStream(token("bar", 1,
> 50, 60)), ftype));
> {code}
> where the token function is defined as:
> {code}
> token(String image, int positionIncrement, int startOffset, int endOffset)
> {code}
> will result in either a cryptic assertion thrown from IW:
> {code}
> Exception in thread "main" java.lang.AssertionError
> at
> org.apache.lucene.index.FreqProxTermsWriterPerField.writeOffsets(FreqProxTermsWriterPerField.java:99)
> {code}
> or nothing (or a codec error) if run without assertions.
> Obviously returning non-shifted offsets from subsequent token streams makes
> little sense but I wonder if it could be made more explicit (or asserted)
> that offsets need to be increasing between multiple-values. The minimum is to
> add some documentation to OffsetAttribute. I don't know if offsets should be
> shifted automatically, as it's the case with positions -- this would change
> the semantics of existing tokenizers and filters which implement such
> shifting internally already.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-5977) IW should safeguard against token streams returning invalid offsets for multi-valued fields
[
https://issues.apache.org/jira/browse/LUCENE-5977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14151567#comment-14151567
]
Dawid Weiss edited comment on LUCENE-5977 at 9/29/14 10:16 AM:
---
A full example which shows the problem. Run it with -ea -- you'll get the
assertion. Run it without assertions and it'll pass.
{code}
import org.apache.lucene.analysis.Token;
import org.apache.lucene.analysis.core.WhitespaceAnalyzer;
import org.apache.lucene.codecs.Codec;
import org.apache.lucene.codecs.simpletext.SimpleTextCodec;
import org.apache.lucene.document.Document;
import org.apache.lucene.document.Field;
import org.apache.lucene.document.FieldType;
import org.apache.lucene.index.FieldInfo.IndexOptions;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.index.IndexWriterConfig;
import org.apache.lucene.store.Directory;
import org.apache.lucene.store.RAMDirectory;
import org.apache.lucene.util.Version;
import org.apache.lucene.analysis.CannedTokenStream;
public class OffsetIndexingBug {
public static void main(String[] args) throws Exception {
Codec.setDefault(new SimpleTextCodec());
Version version = Version.LUCENE_CURRENT;
IndexWriterConfig conf = new IndexWriterConfig(version, new
WhitespaceAnalyzer(version));
conf.setUseCompoundFile(false);
try (Directory d = new RAMDirectory()) {
try (IndexWriter iw = new IndexWriter(d, conf)) {
Document doc = new Document();
FieldType ftype = new FieldType();
ftype.setIndexed(true);
ftype.setStored(false);
ftype.setOmitNorms(true);
ftype.setStoreTermVectors(true);
ftype.setStoreTermVectorPositions(true);
ftype.setStoreTermVectorOffsets(true);
ftype.setTokenized(true);
ftype.setIndexOptions(IndexOptions.DOCS_AND_FREQS_AND_POSITIONS_AND_OFFSETS);
ftype.freeze();
// note "use"'s offset is negative with respect to the first field
value.
doc.add(new Field("field-foo", new CannedTokenStream(token("use", 1,
150, 160)), ftype));
doc.add(new Field("field-foo", new CannedTokenStream(token("use", 1,
50, 60)), ftype));
iw.addDocument(doc);
}
}
}
private static Token token(String image, int positionIncrement, int soffset,
int eoffset) {
Token t = new Token();
t.setPositionIncrement(positionIncrement);
t.setOffset(soffset, eoffset);
t.append(image);
return t;
}
}
{code}
was (Author: dweiss):
A full example which shows the problem. Run it with -ea -- you'll get the
assertion. Run it without assertions and it'll pass.
{code}
import org.apache.lucene.analysis.Token;
import org.apache.lucene.analysis.core.WhitespaceAnalyzer;
import org.apache.lucene.codecs.Codec;
import org.apache.lucene.codecs.simpletext.SimpleTextCodec;
import org.apache.lucene.document.Document;
import org.apache.lucene.document.Field;
import org.apache.lucene.document.FieldType;
import org.apache.lucene.index.FieldInfo.IndexOptions;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.index.IndexWriterConfig;
import org.apache.lucene.store.Directory;
import org.apache.lucene.store.RAMDirectory;
import org.apache.lucene.util.Version;
import org.apache.lucene.analysis.CannedTokenStream;
public class OffsetIndexingBug {
public static void main(String[] args) throws Exception {
Codec.setDefault(new SimpleTextCodec());
Version version = Version.LUCENE_CURRENT;
IndexWriterConfig conf = new IndexWriterConfig(version, new
WhitespaceAnalyzer(version));
conf.setUseCompoundFile(false);
try (Directory d = new RAMDirectory()) {
try (IndexWriter iw = new IndexWriter(d, conf)) {
iw.deleteAll();
iw.commit();
Document doc = new Document();
FieldType ftype = new FieldType();
ftype.setIndexed(true);
ftype.setStored(false);
ftype.setOmitNorms(true);
ftype.setStoreTermVectors(true);
ftype.setStoreTermVectorPositions(true);
ftype.setStoreTermVectorOffsets(true);
ftype.setTokenized(true);
ftype.setIndexOptions(IndexOptions.DOCS_AND_FREQS_AND_POSITIONS_AND_OFFSETS);
ftype.freeze();
// note "use"'s offset is negative with respect to the first field
value.
doc.add(new Field("field-foo", new CannedTokenStream(token("use", 1,
150, 160)), ftype));
doc.add(new Field("field-foo", new CannedTokenStream(token("use", 1,
50, 60)), ftype));
iw.addDocument(doc);
}
}
}
private static Token token(String image, int positionIncrement, int soffset,
int eoffset) {
Token t = new Token();
t.setPositionIncrement(positionIncrement);
t.setOffset(soffset, eoffset);
t.append(image);
return t;
}
}
{code}
> IW should safeguard against token streams returning invalid offsets for
> multi-val
[jira] [Updated] (LUCENE-5977) IW should safeguard against token streams returning invalid offsets for multi-valued fields
[
https://issues.apache.org/jira/browse/LUCENE-5977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dawid Weiss updated LUCENE-5977:
Description:
We have a custom token stream that emits information about offsets of each
token. My (wrong) assumption was that for multi-valued fields a token stream's
offset information is magically shifted, much like this is the case with
positions. It's not the case -- offsets should be increasing and monotonic
across all instances of a field, even if it has custom token streams. So,
something like this:
{code}
doc.add(new Field("field-foo", new CannedTokenStream(token("bar", 1,
150, 160)), ftype));
doc.add(new Field("field-foo", new CannedTokenStream(token("bar", 1,
50, 60)), ftype));
{code}
where the token function is defined as:
{code}
token(String image, int positionIncrement, int startOffset, int endOffset)
{code}
will result in either a cryptic assertion thrown from IW:
{code}
Exception in thread "main" java.lang.AssertionError
at
org.apache.lucene.index.FreqProxTermsWriterPerField.writeOffsets(FreqProxTermsWriterPerField.java:99)
{code}
or nothing (or a codec error) if run without assertions.
Obviously returning non-shifted offsets from subsequent token streams makes
little sense but I wonder if it could be made more explicit (or asserted) that
offsets need to be increasing between multiple-values. The minimum is to add
some documentation to OffsetAttribute. I don't know if offsets should be
shifted automatically, as it's the case with positions -- this would change the
semantics of existing tokenizers and filters which implement such shifting
internally already.
was:
We have a custom token stream that emits information about offsets of each
token. My (wrong) assumption was that for multi-valued fields a token stream's
offset information is magically shifted, much like this is the case with
positions. It's not the case -- offsets should be increasing and monotonic
across all instances of a field, even if it has a custom token streams. So,
something like this:
{code}
doc.add(new Field("field-foo", new CannedTokenStream(token("bar", 1,
150, 160)), ftype));
doc.add(new Field("field-foo", new CannedTokenStream(token("bar", 1,
50, 60)), ftype));
{code}
where the token function is defined as:
{code}
token(String image, int positionIncrement, int startOffset, int endOffset)
{code}
will result in either a cryptic assertion thrown from IW:
{code}
Exception in thread "main" java.lang.AssertionError
at
org.apache.lucene.index.FreqProxTermsWriterPerField.writeOffsets(FreqProxTermsWriterPerField.java:99)
{code}
or nothing (or a codec error) if run without assertions.
Obviously returning non-shifted offsets from subsequent token streams makes
little sense but I wonder if it could be made more explicit (or asserted) that
offsets need to be increasing between multiple-values. The minimum is to add
some documentation to OffsetAttribute. I don't know if offsets should be
shifted automatically, as it's the case with positions -- this would change the
semantics of existing tokenizers and filters which implement such shifting
internally already.
> IW should safeguard against token streams returning invalid offsets for
> multi-valued fields
> ---
>
> Key: LUCENE-5977
> URL: https://issues.apache.org/jira/browse/LUCENE-5977
> Project: Lucene - Core
> Issue Type: Improvement
>Affects Versions: 4.9, 4.9.1, 4.10, 4.10.1
>Reporter: Dawid Weiss
>Priority: Minor
>
> We have a custom token stream that emits information about offsets of each
> token. My (wrong) assumption was that for multi-valued fields a token
> stream's offset information is magically shifted, much like this is the case
> with positions. It's not the case -- offsets should be increasing and
> monotonic across all instances of a field, even if it has custom token
> streams. So, something like this:
> {code}
> doc.add(new Field("field-foo", new CannedTokenStream(token("bar", 1,
> 150, 160)), ftype));
> doc.add(new Field("field-foo", new CannedTokenStream(token("bar", 1,
> 50, 60)), ftype));
> {code}
> where the token function is defined as:
> {code}
> token(String image, int positionIncrement, int startOffset, int endOffset)
> {code}
> will result in either a cryptic assertion thrown from IW:
> {code}
> Exception in thread "main" java.lang.AssertionError
> at
> org.apache.lucene.index.FreqProxTermsWriterPerField.writeOffsets(FreqProxTermsWriterPerField.java:99)
> {code}
> or nothing (or a codec error) if run without assertions.
> Obviously returning non-shifted offsets from subsequent token streams makes
> little sense but I wonder if it could be made more explicit (o
[jira] [Created] (LUCENE-5977) IW should safeguard against token streams returning invalid offsets for multi-valued fields
Dawid Weiss created LUCENE-5977:
---
Summary: IW should safeguard against token streams returning
invalid offsets for multi-valued fields
Key: LUCENE-5977
URL: https://issues.apache.org/jira/browse/LUCENE-5977
Project: Lucene - Core
Issue Type: Improvement
Affects Versions: 4.10.1, 4.10, 4.9.1, 4.9
Reporter: Dawid Weiss
Priority: Minor
We have a custom token stream that emits information about offsets of each
token. My (wrong) assumption was that for multi-valued fields a token stream's
offset information is magically shifted, much like this is the case with
positions. It's not the case -- offsets should be increasing and monotonic
across all instances of a field, even if it has a custom token streams. So,
something like this:
{code}
doc.add(new Field("field-foo", new CannedTokenStream(token("bar", 1,
150, 160)), ftype));
doc.add(new Field("field-foo", new CannedTokenStream(token("bar", 1,
50, 60)), ftype));
{code}
where the token function is defined as:
{code}
token(String image, int positionIncrement, int startOffset, int endOffset)
{code}
will result in either a cryptic assertion thrown from IW:
{code}
Exception in thread "main" java.lang.AssertionError
at
org.apache.lucene.index.FreqProxTermsWriterPerField.writeOffsets(FreqProxTermsWriterPerField.java:99)
{code}
or nothing (or a codec error) if run without assertions.
Obviously returning non-shifted offsets from subsequent token streams makes
little sense but I wonder if it could be made more explicit (or asserted) that
offsets need to be increasing between multiple-values. The minimum is to add
some documentation to OffsetAttribute. I don't know if offsets should be
shifted automatically, as it's the case with positions -- this would change the
semantics of existing tokenizers and filters which implement such shifting
internally already.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.8.0_40-ea-b04) - Build # 4343 - Still Failing!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Windows/4343/
Java: 64bit/jdk1.8.0_40-ea-b04 -XX:+UseCompressedOops -XX:+UseG1GC
2 tests failed.
REGRESSION: org.apache.solr.cloud.HttpPartitionTest.testDistribSearch
Error Message:
Send doc 2 to old leader core_node2 should have failed! ClusterState:
DocCollection(c8n_1x2_leader_session_loss)={ "replicationFactor":"2",
"shards":{"shard1":{ "range":"8000-7fff", "state":"active",
"replicas":{ "core_node1":{
"core":"c8n_1x2_leader_session_loss_shard1_replica1",
"base_url":"http://127.0.0.1:57338/iw_hj";,
"node_name":"127.0.0.1:57338_iw_hj", "state":"active",
"leader":"true"}, "core_node2":{
"core":"c8n_1x2_leader_session_loss_shard1_replica2",
"base_url":"http://127.0.0.1:57322/iw_hj";,
"node_name":"127.0.0.1:57322_iw_hj", "state":"down",
"router":{"name":"compositeId"}, "maxShardsPerNode":"1",
"autoAddReplicas":"false"}
Stack Trace:
java.lang.AssertionError: Send doc 2 to old leader core_node2 should have
failed! ClusterState: DocCollection(c8n_1x2_leader_session_loss)={
"replicationFactor":"2",
"shards":{"shard1":{
"range":"8000-7fff",
"state":"active",
"replicas":{
"core_node1":{
"core":"c8n_1x2_leader_session_loss_shard1_replica1",
"base_url":"http://127.0.0.1:57338/iw_hj";,
"node_name":"127.0.0.1:57338_iw_hj",
"state":"active",
"leader":"true"},
"core_node2":{
"core":"c8n_1x2_leader_session_loss_shard1_replica2",
"base_url":"http://127.0.0.1:57322/iw_hj";,
"node_name":"127.0.0.1:57322_iw_hj",
"state":"down",
"router":{"name":"compositeId"},
"maxShardsPerNode":"1",
"autoAddReplicas":"false"}
at
__randomizedtesting.SeedInfo.seed([299406B9921B386F:A87288A1E5445853]:0)
at org.junit.Assert.fail(Assert.java:93)
at
org.apache.solr.cloud.HttpPartitionTest.testLeaderZkSessionLoss(HttpPartitionTest.java:329)
at
org.apache.solr.cloud.HttpPartitionTest.doTest(HttpPartitionTest.java:119)
at
org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at
org.apache.lucene.util.TestRuleStoreClassName$1.
[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.9.0-ea-b28) - Build # 11347 - Still Failing!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11347/
Java: 32bit/jdk1.9.0-ea-b28 -server -XX:+UseConcMarkSweepGC
1 tests failed.
FAILED: org.apache.solr.cloud.HttpPartitionTest.testDistribSearch
Error Message:
Doc with id=2 not found in
http://127.0.0.1:37774/nm_uc/c8n_1x2_leader_session_loss due to: Path not
found: /id; rsp={doc=null}
Stack Trace:
java.lang.AssertionError: Doc with id=2 not found in
http://127.0.0.1:37774/nm_uc/c8n_1x2_leader_session_loss due to: Path not
found: /id; rsp={doc=null}
at
__randomizedtesting.SeedInfo.seed([FC05035FA51DFD06:7DE38D47D2429D3A]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at
org.apache.solr.cloud.HttpPartitionTest.assertDocExists(HttpPartitionTest.java:437)
at
org.apache.solr.cloud.HttpPartitionTest.testLeaderZkSessionLoss(HttpPartitionTest.java:324)
at
org.apache.solr.cloud.HttpPartitionTest.doTest(HttpPartitionTest.java:119)
at
org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:484)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
[JENKINS-MAVEN] Lucene-Solr-Maven-5.x #717: POMs out of sync
Build: https://builds.apache.org/job/Lucene-Solr-Maven-5.x/717/ 2 tests failed. REGRESSION: org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch Error Message: Error CREATEing SolrCore 'halfcollection_shard1_replica1': Unable to create core [halfcollection_shard1_replica1] Caused by: Could not get shard id for core: halfcollection_shard1_replica1 Stack Trace: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Error CREATEing SolrCore 'halfcollection_shard1_replica1': Unable to create core [halfcollection_shard1_replica1] Caused by: Could not get shard id for core: halfcollection_shard1_replica1 at __randomizedtesting.SeedInfo.seed([52360C9922C634E6:D3D08281559954DA]:0) at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:568) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211) at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testErrorHandling(CollectionsAPIDistributedZkTest.java:583) at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:205) REGRESSION: org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.testDistribSearch Error Message: No live SolrServers available to handle this request:[https://127.0.0.1:62006, https://127.0.0.1:11553, https://127.0.0.1:60126] Stack Trace: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://127.0.0.1:62006, https://127.0.0.1:11553, https://127.0.0.1:60126] at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:568) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211) at org.apache.solr.client.solrj.impl.LBHttpSolrServer.doRequest(LBHttpSolrServer.java:343) at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:304) at org.apache.solr.client.solrj.impl.CloudSolrServer.sendRequest(CloudSolrServer.java:880) at org.apache.solr.client.solrj.impl.CloudSolrServer.requestWithRetryOnStaleState(CloudSolrServer.java:658) at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:601) at org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.removeAndWaitForLastReplicaGone(DeleteLastCustomShardedReplicaTest.java:117) at org.apache.solr.cloud.DeleteLastCustomShardedReplicaTest.doTest(DeleteLastCustomShardedReplicaTest.java:107) Build Log: [...truncated 52973 lines...] BUILD FAILED /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Maven-5.x/build.xml:547: The following error occurred while executing this line: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Maven-5.x/build.xml:199: The following error occurred while executing this line: : Java returned: 1 Total time: 240 minutes 24 seconds Build step 'Invoke Ant' marked build as failure Recording test results Email was triggered for: Failure Sending email for trigger: Failure - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
