date:20190429



[ 
https://issues.apache.org/jira/browse/SOLR-13320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16829856#comment-16829856
 ] 

Noble Paul edited comment on SOLR-13320 at 4/30/19 5:33 AM:


bq.I'm just trying to avoid adding yet another way to do something that's 
already possible to do with Solr,

The reason why this has to be in DUH2 and not in a URP is because these checks 
are already done by DUH2. If a URP is introduced , it will have to do the same 
set of things done by DUH2 over and over again. Yes, I agree that an explosion 
of global parameters is not good , but this one is part of the checks that is 
already done by DUH2. Ideally, we should have done the entire version handling 
in a URP and not in DUH2. But that ship has already sailed


was (Author: noble.paul):
The version comparison logic is already handled by DUH2. Using a URP is much 
more complex than explaining people to use two URPs and I still have no idea 
how it works

> add a param ignoreVersionConflicts=true to updates to not overwrite existing 
> docs
> -
>
> Key: SOLR-13320
> URL: https://issues.apache.org/jira/browse/SOLR-13320
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Noble Paul
>Assignee: Noble Paul
>Priority: Major
> Attachments: SOLR-13320.patch
>
>
> Updates should have an option to ignore duplicate documents and drop them if 
> an option  {{ignoreDuplicates=true}} is specified



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-8.x-Linux (64bit/jdk-12) - Build # 486 - Unstable!

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-8.x-Linux/486/
Java: 64bit/jdk-12 -XX:-UseCompressedOops -XX:+UseG1GC

1 tests failed.
FAILED:  org.apache.solr.cloud.AliasIntegrationTest.testClusterStateProviderAPI

Error Message:
should be a routed alias

Stack Trace:
java.lang.AssertionError: should be a routed alias
at 
__randomizedtesting.SeedInfo.seed([8BF4C069281A5E0F:94235C455B11A744]:0)
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at 
org.apache.solr.cloud.AliasIntegrationTest.testClusterStateProviderAPI(AliasIntegrationTest.java:302)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:567)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:835)




Build Log:
[...truncated 14382 lines...]
   [junit4] Suite: org.apache.solr.cloud.AliasIntegrationTest
   [junit4]   2> 1100574 INFO  
(SUITE-AliasIntegrationTest-seed#[8BF4C069281A5E0F]-worker) [] 
o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: 
test.solr.allowed.securerandom=null & java.security.egd=file:/dev/.

[jira] [Updated] (LUCENE-8784) Nori(Korean) tokenizer removes the decimal point.

2019-04-29 Thread Munkyu Im (JIRA)



 [ 
https://issues.apache.org/jira/browse/LUCENE-8784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Munkyu Im updated LUCENE-8784:
--
Description: 
This is the same issue that I mentioned to 
[https://github.com/elastic/elasticsearch/issues/41401#event-2293189367]

unlike standard analyzer, nori analyzer removes the decimal point.

nori tokenizer removes "." character by default.
 In this case, it is difficult to index the keywords including the decimal 
point.

It would be nice if there had the option whether add a decimal point or not.

Like Japanese tokenizer does,  Nori need an option to preserve decimal point.

 

  was:
This is the same issue that I mentioned to 
[https://github.com/elastic/elasticsearch/issues/41401#event-2293189367]

unlike standard analyzer, nori analyzer removes the decimal point.

nori tokenizer removes "." character by default.
In this case, it is difficult to index the keywords including the decimal point.

It would be nice if there had the option whether add a decimal point or not 
like Japanese tokenizer

 


>  Nori(Korean) tokenizer removes the decimal point. 
> ---
>
> Key: LUCENE-8784
> URL: https://issues.apache.org/jira/browse/LUCENE-8784
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Munkyu Im
>Priority: Major
>
> This is the same issue that I mentioned to 
> [https://github.com/elastic/elasticsearch/issues/41401#event-2293189367]
> unlike standard analyzer, nori analyzer removes the decimal point.
> nori tokenizer removes "." character by default.
>  In this case, it is difficult to index the keywords including the decimal 
> point.
> It would be nice if there had the option whether add a decimal point or not.
> Like Japanese tokenizer does,  Nori need an option to preserve decimal point.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (LUCENE-8784) Nori(Korean) tokenizer removes the decimal point.

2019-04-29 Thread Munkyu Im (JIRA)

Munkyu Im created LUCENE-8784:
-

 Summary:  Nori(Korean) tokenizer removes the decimal point. 
 Key: LUCENE-8784
 URL: https://issues.apache.org/jira/browse/LUCENE-8784
 Project: Lucene - Core
  Issue Type: Improvement
Reporter: Munkyu Im


This is the same issue that I mentioned to 
[https://github.com/elastic/elasticsearch/issues/41401#event-2293189367]

unlike standard analyzer, nori analyzer removes the decimal point.

nori tokenizer removes "." character by default.
In this case, it is difficult to index the keywords including the decimal point.

It would be nice if there had the option whether add a decimal point or not 
like Japanese tokenizer

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-Tests-8.x - Build # 176 - Failure

Build: https://builds.apache.org/job/Lucene-Solr-Tests-8.x/176/

All tests passed

Build Log:
[...truncated 65660 lines...]
[asciidoctor:convert] asciidoctor: ERROR: about-this-guide.adoc: line 1: 
invalid part, must have at least one section (e.g., chapter, appendix, etc.)
[asciidoctor:convert] asciidoctor: ERROR: solr-glossary.adoc: line 1: invalid 
part, must have at least one section (e.g., chapter, appendix, etc.)
 [java] Processed 2526 links (2067 relative) to 3356 anchors in 253 files
 [echo] Validated Links & Anchors via: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-8.x/solr/build/solr-ref-guide/bare-bones-html/

-documentation-lint:
[jtidy] Checking for broken html (such as invalid tags)...
   [delete] Deleting directory 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-8.x/lucene/build/jtidy_tmp
 [echo] Checking for broken links...
 [exec] 
 [exec] Crawl/parse...
 [exec] 
 [exec] Verify...
 [echo] Checking for malformed docs...
 [exec] 
 [exec] 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-8.x/solr/build/docs/solr-core/overview-summary.html
 [exec]   missing description: org.apache.solr.cloud.autoscaling.sim
 [exec] 
 [exec] Missing javadocs were found!

BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-8.x/build.xml:634: The 
following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-8.x/build.xml:101: The 
following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-8.x/solr/build.xml:660: 
The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-8.x/solr/build.xml:676: 
The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-8.x/lucene/common-build.xml:2530:
 exec returned: 1

Total time: 112 minutes 49 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-Tests-master - Build # 3318 - Unstable

Build: https://builds.apache.org/job/Lucene-Solr-Tests-master/3318/

1 tests failed.
FAILED:  org.apache.solr.metrics.rrd.SolrRrdBackendFactoryTest.testBasic

Error Message:
{} expected:<1> but was:<0>

Stack Trace:
java.lang.AssertionError: {} expected:<1> but was:<0>
at 
__randomizedtesting.SeedInfo.seed([FF52E411B94D1197:54A8F904669197B9]:0)
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:645)
at 
org.apache.solr.metrics.rrd.SolrRrdBackendFactoryTest.testBasic(SolrRrdBackendFactoryTest.java:94)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:834)




Build Log:
[...truncated 14914 lines...]
   [junit4] Suite: org.apache.solr.metrics.rrd.SolrRrdBackendFactoryTest
   [junit4]   2> Creating dataDir: 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.metrics.rrd.SolrRrdBackendFactoryTest_FF52E411B94D1197-001/init

[JENKINS] Lucene-Solr-8.x-Linux (64bit/jdk1.8.0_201) - Build # 485 - Still Failing!

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-8.x-Linux/485/
Java: 64bit/jdk1.8.0_201 -XX:-UseCompressedOops -XX:+UseSerialGC

All tests passed

Build Log:
[...truncated 65690 lines...]
[asciidoctor:convert] asciidoctor: ERROR: about-this-guide.adoc: line 1: 
invalid part, must have at least one section (e.g., chapter, appendix, etc.)
[asciidoctor:convert] asciidoctor: ERROR: solr-glossary.adoc: line 1: invalid 
part, must have at least one section (e.g., chapter, appendix, etc.)
 [java] Processed 2526 links (2067 relative) to 3356 anchors in 253 files
 [echo] Validated Links & Anchors via: 
/home/jenkins/workspace/Lucene-Solr-8.x-Linux/solr/build/solr-ref-guide/bare-bones-html/

-documentation-lint:
[jtidy] Checking for broken html (such as invalid tags)...
   [delete] Deleting directory 
/home/jenkins/workspace/Lucene-Solr-8.x-Linux/lucene/build/jtidy_tmp
 [echo] Checking for broken links...
 [exec] 
 [exec] Crawl/parse...
 [exec] 
 [exec] Verify...
 [echo] Checking for malformed docs...
 [exec] 
 [exec] 
/home/jenkins/workspace/Lucene-Solr-8.x-Linux/solr/build/docs/solr-core/overview-summary.html
 [exec]   missing description: org.apache.solr.cloud.autoscaling.sim
 [exec] 
 [exec] Missing javadocs were found!

BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-8.x-Linux/build.xml:634: The following 
error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-8.x-Linux/build.xml:101: The following 
error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-8.x-Linux/solr/build.xml:660: The following 
error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-8.x-Linux/solr/build.xml:676: The following 
error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-8.x-Linux/lucene/common-build.xml:2530: 
exec returned: 1

Total time: 82 minutes 57 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Setting 
ANT_1_8_2_HOME=/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Setting 
ANT_1_8_2_HOME=/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
Setting 
ANT_1_8_2_HOME=/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting 
ANT_1_8_2_HOME=/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting 
ANT_1_8_2_HOME=/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting 
ANT_1_8_2_HOME=/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting 
ANT_1_8_2_HOME=/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting 
ANT_1_8_2_HOME=/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-8783) Add FST Offheap for non-default Codecs



 [ 
https://issues.apache.org/jira/browse/LUCENE-8783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ankit Jain updated LUCENE-8783:
---
Description: 
Even though, LUCENE-8635 and LUCENE-8671 adds support to keep FST offheap for 
default codec, there are many other codecs which do not support FST offheap. 
Few examples are below:

* CompletionPostingsFormat
* BlockTreeOrdsPostingsFormat
* IDVersionPostingsFormat

  was:
Even though, LUCENE-8635 and LUCENE-8671 adds support to keep FST offheap for 
default codec, there are many other codecs which do not support this. Few 
examples are below:

* CompletionPostingsFormat
* BlockTreeOrdsPostingsFormat
* IDVersionPostingsFormat


> Add FST Offheap for non-default Codecs
> --
>
> Key: LUCENE-8783
> URL: https://issues.apache.org/jira/browse/LUCENE-8783
> Project: Lucene - Core
>  Issue Type: New Feature
>  Components: core/FSTs
>Reporter: Ankit Jain
>Priority: Major
> Fix For: 8.0, 8.x, master (9.0)
>
>
> Even though, LUCENE-8635 and LUCENE-8671 adds support to keep FST offheap for 
> default codec, there are many other codecs which do not support FST offheap. 
> Few examples are below:
> * CompletionPostingsFormat
> * BlockTreeOrdsPostingsFormat
> * IDVersionPostingsFormat



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-8783) Add FST Offheap for non-default Codecs



 [ 
https://issues.apache.org/jira/browse/LUCENE-8783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ankit Jain updated LUCENE-8783:
---
Description: 
Even though, LUCENE-8635 and LUCENE-8671 adds support to keep FST offheap for 
default codec, there are many other codecs which do not support this. Few 
examples are below:

* CompletionPostingsFormat
* BlockTreeOrdsPostingsFormat
* IDVersionPostingsFormat

  was:
Even though, [~LUCENE-8635] and [~LUCENE-8671]adds support to keep FST offheap 
for default codec, there are many other codecs which do not support this. Few 
examples are below:

* CompletionPostingsFormat
* BlockTreeOrdsPostingsFormat
* IDVersionPostingsFormat


> Add FST Offheap for non-default Codecs
> --
>
> Key: LUCENE-8783
> URL: https://issues.apache.org/jira/browse/LUCENE-8783
> Project: Lucene - Core
>  Issue Type: New Feature
>  Components: core/FSTs
>Reporter: Ankit Jain
>Priority: Major
> Fix For: 8.0, 8.x, master (9.0)
>
>
> Even though, LUCENE-8635 and LUCENE-8671 adds support to keep FST offheap for 
> default codec, there are many other codecs which do not support this. Few 
> examples are below:
> * CompletionPostingsFormat
> * BlockTreeOrdsPostingsFormat
> * IDVersionPostingsFormat



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-8783) Add FST Offheap for non-default Codecs



 [ 
https://issues.apache.org/jira/browse/LUCENE-8783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ankit Jain updated LUCENE-8783:
---
Description: 
Even though, [~LUCENE-8635] and [~LUCENE-8671]adds support to keep FST offheap 
for default codec, there are many other codecs which do not support this. Few 
examples are below:

* CompletionPostingsFormat
* BlockTreeOrdsPostingsFormat
* IDVersionPostingsFormat

  was:
Even though, [^LUCENE-8635] and [^LUCENE-8671]adds support to keep FST offheap 
for default codec, there are many other codecs which do not support this. Few 
examples are below:

* CompletionPostingsFormat
* BlockTreeOrdsPostingsFormat
* IDVersionPostingsFormat


> Add FST Offheap for non-default Codecs
> --
>
> Key: LUCENE-8783
> URL: https://issues.apache.org/jira/browse/LUCENE-8783
> Project: Lucene - Core
>  Issue Type: New Feature
>  Components: core/FSTs
>Reporter: Ankit Jain
>Priority: Major
> Fix For: 8.0, 8.x, master (9.0)
>
>
> Even though, [~LUCENE-8635] and [~LUCENE-8671]adds support to keep FST 
> offheap for default codec, there are many other codecs which do not support 
> this. Few examples are below:
> * CompletionPostingsFormat
> * BlockTreeOrdsPostingsFormat
> * IDVersionPostingsFormat



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-8783) Add FST Offheap for non-default Codecs



 [ 
https://issues.apache.org/jira/browse/LUCENE-8783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ankit Jain updated LUCENE-8783:
---
Description: 
Even though, [^LUCENE-8635] and [^LUCENE-8671]adds support to keep FST offheap 
for default codec, there are many other codecs which do not support this. Few 
examples are below:

* CompletionPostingsFormat
* BlockTreeOrdsPostingsFormat
* IDVersionPostingsFormat

  was:
Even though, [LUCENE-8635](https://issues.apache.org/jira/browse/LUCENE-8635) 
and [LUCENE-8671](https://issues.apache.org/jira/browse/LUCENE-8671) adds 
support to keep FST offheap for default codec, there are many other codecs 
which do not support this. Few examples are below:

* CompletionPostingsFormat
* BlockTreeOrdsPostingsFormat
* IDVersionPostingsFormat


> Add FST Offheap for non-default Codecs
> --
>
> Key: LUCENE-8783
> URL: https://issues.apache.org/jira/browse/LUCENE-8783
> Project: Lucene - Core
>  Issue Type: New Feature
>  Components: core/FSTs
>Reporter: Ankit Jain
>Priority: Major
> Fix For: 8.0, 8.x, master (9.0)
>
>
> Even though, [^LUCENE-8635] and [^LUCENE-8671]adds support to keep FST 
> offheap for default codec, there are many other codecs which do not support 
> this. Few examples are below:
> * CompletionPostingsFormat
> * BlockTreeOrdsPostingsFormat
> * IDVersionPostingsFormat



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-13320) add a param ignoreVersionConflicts=true to updates to not overwrite existing docs



[ 
https://issues.apache.org/jira/browse/SOLR-13320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16829856#comment-16829856
 ] 

Noble Paul commented on SOLR-13320:
---

The version comparison logic is already handled by DUH2. Using a URP is much 
more complex than explaining people to use two URPs and I still have no idea 
how it works

> add a param ignoreVersionConflicts=true to updates to not overwrite existing 
> docs
> -
>
> Key: SOLR-13320
> URL: https://issues.apache.org/jira/browse/SOLR-13320
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Noble Paul
>Assignee: Noble Paul
>Priority: Major
> Attachments: SOLR-13320.patch
>
>
> Updates should have an option to ignore duplicate documents and drop them if 
> an option  {{ignoreDuplicates=true}} is specified



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-8783) Add FST Offheap for non-default Codecs



 [ 
https://issues.apache.org/jira/browse/LUCENE-8783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ankit Jain updated LUCENE-8783:
---
Description: 
Even though, [LUCENE-8635](https://issues.apache.org/jira/browse/LUCENE-8635) 
and [LUCENE-8671](https://issues.apache.org/jira/browse/LUCENE-8671) adds 
support to keep FST offheap for default codec, there are many other codecs 
which do not support this. Few examples are below:

* CompletionPostingsFormat
* BlockTreeOrdsPostingsFormat
* IDVersionPostingsFormat

  was:Even though, 
[LUCENE-8635](https://issues.apache.org/jira/browse/LUCENE-8635) and 
[LUCENE-8671](https://issues.apache.org/jira/browse/LUCENE-8671) adds sup


> Add FST Offheap for non-default Codecs
> --
>
> Key: LUCENE-8783
> URL: https://issues.apache.org/jira/browse/LUCENE-8783
> Project: Lucene - Core
>  Issue Type: New Feature
>  Components: core/FSTs
>Reporter: Ankit Jain
>Priority: Major
> Fix For: 8.0, 8.x, master (9.0)
>
>
> Even though, [LUCENE-8635](https://issues.apache.org/jira/browse/LUCENE-8635) 
> and [LUCENE-8671](https://issues.apache.org/jira/browse/LUCENE-8671) adds 
> support to keep FST offheap for default codec, there are many other codecs 
> which do not support this. Few examples are below:
> * CompletionPostingsFormat
> * BlockTreeOrdsPostingsFormat
> * IDVersionPostingsFormat



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-master-Windows (64bit/jdk-11.0.2) - Build # 7918 - Still Unstable!

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Windows/7918/
Java: 64bit/jdk-11.0.2 -XX:-UseCompressedOops -XX:+UseParallelGC

1 tests failed.
FAILED:  org.apache.solr.core.ExitableDirectoryReaderTest.testPrefixQuery

Error Message:
Path not found: /responseHeader/partialResults

Stack Trace:
java.lang.RuntimeException: Path not found: /responseHeader/partialResults
at 
__randomizedtesting.SeedInfo.seed([974F39E25BE1CA8B:24180DA03F3C2E0E]:0)
at org.apache.solr.SolrTestCaseJ4.assertJQ(SolrTestCaseJ4.java:1027)
at org.apache.solr.SolrTestCaseJ4.assertJQ(SolrTestCaseJ4.java:974)
at 
org.apache.solr.core.ExitableDirectoryReaderTest.testPrefixQuery(ExitableDirectoryReaderTest.java:60)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:834)




Build Log:
[...truncated 14481 lines...]
   [junit4] Suite: org.apache.solr.core.ExitableDirectoryReaderTest
   [junit4]   2> Creating dataDir: 
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-

[jira] [Commented] (SOLR-13320) add a param ignoreVersionConflicts=true to updates to not overwrite existing docs



[ 
https://issues.apache.org/jira/browse/SOLR-13320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16829853#comment-16829853
 ] 

Tomás Fernández Löbbe commented on SOLR-13320:
--

You can combine the {{DocBasedVersionConstraintsProcessor}} with a 
{{DefaultValueUpdateProcessor}}? I'm just trying to avoid adding yet another 
way to do something that's already possible to do with Solr, another random 
parameter that needs to be tested, maintained and documented (and explain over 
and over in the users list), Solr has already too many of those IMO. Just 
advocating for a simpler/smaller API and trying to prevent adding more 
complexity to already complex code like DistributedUpdateProcessor, or DUH2.

> add a param ignoreVersionConflicts=true to updates to not overwrite existing 
> docs
> -
>
> Key: SOLR-13320
> URL: https://issues.apache.org/jira/browse/SOLR-13320
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Noble Paul
>Assignee: Noble Paul
>Priority: Major
> Attachments: SOLR-13320.patch
>
>
> Updates should have an option to ignore duplicate documents and drop them if 
> an option  {{ignoreDuplicates=true}} is specified



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-8783) Support FST lazy loading for non-default Codecs



 [ 
https://issues.apache.org/jira/browse/LUCENE-8783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ankit Jain updated LUCENE-8783:
---
Description: Even though, 
[LUCENE-8635](https://issues.apache.org/jira/browse/LUCENE-8635) and 
[LUCENE-8671](https://issues.apache.org/jira/browse/LUCENE-8671) adds sup  
(was: Currently, FST loads all the terms into heap memory during index open. 
This causes frequent JVM OOM issues if the term size gets big. A better way of 
doing this will be to lazily load FST using mmap. That ensures only the 
required terms get loaded into memory.

 
Lucene can expose API for providing list of fields to load terms offheap. I'm 
planning to take following approach for this:
 # Add a boolean property fstOffHeap in FieldInfo
 # Pass list of offheap fields to lucene during index open (ALL can be special 
keyword for loading ALL fields offheap)
 # Initialize the fstOffHeap property during lucene index open
 # FieldReader invokes default FST constructor or OffHeap constructor based on 
fstOffHeap field

 
I created a patch (that loads all fields offheap), did some benchmarks using 
es_rally and results look good.)

> Support FST lazy loading for non-default Codecs
> ---
>
> Key: LUCENE-8783
> URL: https://issues.apache.org/jira/browse/LUCENE-8783
> Project: Lucene - Core
>  Issue Type: New Feature
>  Components: core/FSTs
>Reporter: Ankit Jain
>Priority: Major
> Fix For: 8.0, 8.x, master (9.0)
>
>
> Even though, [LUCENE-8635](https://issues.apache.org/jira/browse/LUCENE-8635) 
> and [LUCENE-8671](https://issues.apache.org/jira/browse/LUCENE-8671) adds sup



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-8783) Add FST Offheap for non-default Codecs



 [ 
https://issues.apache.org/jira/browse/LUCENE-8783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ankit Jain updated LUCENE-8783:
---
Summary: Add FST Offheap for non-default Codecs  (was: Support FST lazy 
loading for non-default Codecs)

> Add FST Offheap for non-default Codecs
> --
>
> Key: LUCENE-8783
> URL: https://issues.apache.org/jira/browse/LUCENE-8783
> Project: Lucene - Core
>  Issue Type: New Feature
>  Components: core/FSTs
>Reporter: Ankit Jain
>Priority: Major
> Fix For: 8.0, 8.x, master (9.0)
>
>
> Even though, [LUCENE-8635](https://issues.apache.org/jira/browse/LUCENE-8635) 
> and [LUCENE-8671](https://issues.apache.org/jira/browse/LUCENE-8671) adds sup



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-8783) Support FST lazy loading for non-default Codecs



 [ 
https://issues.apache.org/jira/browse/LUCENE-8783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ankit Jain updated LUCENE-8783:
---
Review Patch?:   (was: Yes)

> Support FST lazy loading for non-default Codecs
> ---
>
> Key: LUCENE-8783
> URL: https://issues.apache.org/jira/browse/LUCENE-8783
> Project: Lucene - Core
>  Issue Type: New Feature
>  Components: core/FSTs
>Reporter: Ankit Jain
>Priority: Major
> Fix For: 8.0, 8.x, master (9.0)
>
>
> Currently, FST loads all the terms into heap memory during index open. This 
> causes frequent JVM OOM issues if the term size gets big. A better way of 
> doing this will be to lazily load FST using mmap. That ensures only the 
> required terms get loaded into memory.
>  
> Lucene can expose API for providing list of fields to load terms offheap. I'm 
> planning to take following approach for this:
>  # Add a boolean property fstOffHeap in FieldInfo
>  # Pass list of offheap fields to lucene during index open (ALL can be 
> special keyword for loading ALL fields offheap)
>  # Initialize the fstOffHeap property during lucene index open
>  # FieldReader invokes default FST constructor or OffHeap constructor based 
> on fstOffHeap field
>  
> I created a patch (that loads all fields offheap), did some benchmarks using 
> es_rally and results look good.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-8783) Support FST lazy loading for non-default Codecs



 [ 
https://issues.apache.org/jira/browse/LUCENE-8783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ankit Jain updated LUCENE-8783:
---
Lucene Fields: New  (was: New,Patch Available)

> Support FST lazy loading for non-default Codecs
> ---
>
> Key: LUCENE-8783
> URL: https://issues.apache.org/jira/browse/LUCENE-8783
> Project: Lucene - Core
>  Issue Type: New Feature
>  Components: core/FSTs
>Reporter: Ankit Jain
>Priority: Major
> Fix For: 8.0, 8.x, master (9.0)
>
>
> Currently, FST loads all the terms into heap memory during index open. This 
> causes frequent JVM OOM issues if the term size gets big. A better way of 
> doing this will be to lazily load FST using mmap. That ensures only the 
> required terms get loaded into memory.
>  
> Lucene can expose API for providing list of fields to load terms offheap. I'm 
> planning to take following approach for this:
>  # Add a boolean property fstOffHeap in FieldInfo
>  # Pass list of offheap fields to lucene during index open (ALL can be 
> special keyword for loading ALL fields offheap)
>  # Initialize the fstOffHeap property during lucene index open
>  # FieldReader invokes default FST constructor or OffHeap constructor based 
> on fstOffHeap field
>  
> I created a patch (that loads all fields offheap), did some benchmarks using 
> es_rally and results look good.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (LUCENE-8783) Support FST lazy loading for non-default Codecs

Ankit Jain created LUCENE-8783:
--

 Summary: Support FST lazy loading for non-default Codecs
 Key: LUCENE-8783
 URL: https://issues.apache.org/jira/browse/LUCENE-8783
 Project: Lucene - Core
  Issue Type: New Feature
  Components: core/FSTs
 Environment: I used below setup for es_rally tests:

single node i3.xlarge running ES 6.5

es_rally was running on another i3.xlarge instance
Reporter: Ankit Jain
 Fix For: 8.0, 8.x, master (9.0)


Currently, FST loads all the terms into heap memory during index open. This 
causes frequent JVM OOM issues if the term size gets big. A better way of doing 
this will be to lazily load FST using mmap. That ensures only the required 
terms get loaded into memory.

 
Lucene can expose API for providing list of fields to load terms offheap. I'm 
planning to take following approach for this:
 # Add a boolean property fstOffHeap in FieldInfo
 # Pass list of offheap fields to lucene during index open (ALL can be special 
keyword for loading ALL fields offheap)
 # Initialize the fstOffHeap property during lucene index open
 # FieldReader invokes default FST constructor or OffHeap constructor based on 
fstOffHeap field

 
I created a patch (that loads all fields offheap), did some benchmarks using 
es_rally and results look good.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-8783) Support FST lazy loading for non-default Codecs

2019-04-29 Thread ASF subversion and git services (JIRA)



 [ 
https://issues.apache.org/jira/browse/LUCENE-8783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ankit Jain updated LUCENE-8783:
---
Environment: (was: I used below setup for es_rally tests:

single node i3.xlarge running ES 6.5

es_rally was running on another i3.xlarge instance)

> Support FST lazy loading for non-default Codecs
> ---
>
> Key: LUCENE-8783
> URL: https://issues.apache.org/jira/browse/LUCENE-8783
> Project: Lucene - Core
>  Issue Type: New Feature
>  Components: core/FSTs
>Reporter: Ankit Jain
>Priority: Major
> Fix For: 8.0, 8.x, master (9.0)
>
>
> Currently, FST loads all the terms into heap memory during index open. This 
> causes frequent JVM OOM issues if the term size gets big. A better way of 
> doing this will be to lazily load FST using mmap. That ensures only the 
> required terms get loaded into memory.
>  
> Lucene can expose API for providing list of fields to load terms offheap. I'm 
> planning to take following approach for this:
>  # Add a boolean property fstOffHeap in FieldInfo
>  # Pass list of offheap fields to lucene during index open (ALL can be 
> special keyword for loading ALL fields offheap)
>  # Initialize the fstOffHeap property during lucene index open
>  # FieldReader invokes default FST constructor or OffHeap constructor based 
> on fstOffHeap field
>  
> I created a patch (that loads all fields offheap), did some benchmarks using 
> es_rally and results look good.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12833) Use timed-out lock in DistributedUpdateProcessor

2019-04-29 Thread Andrzej Bialecki (JIRA)



[ 
https://issues.apache.org/jira/browse/SOLR-12833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16829823#comment-16829823
 ] 

Andrzej Bialecki  commented on SOLR-12833:
--

LGTM :)

[~markrmil...@gmail.com] - would you like to commit this, or should I do it?

> Use timed-out lock in DistributedUpdateProcessor
> 
>
> Key: SOLR-12833
> URL: https://issues.apache.org/jira/browse/SOLR-12833
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: update, UpdateRequestProcessors
>Affects Versions: 7.5, 8.0
>Reporter: jefferyyuan
>Assignee: Mark Miller
>Priority: Minor
> Fix For: 7.7, 8.0
>
> Attachments: SOLR-12833-noint.patch, SOLR-12833.patch, 
> SOLR-12833.patch
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> There is a synchronize block that blocks other update requests whose IDs fall 
> in the same hash bucket. The update waits forever until it gets the lock at 
> the synchronize block, this can be a problem in some cases.
>  
> Some add/update requests (for example updates with spatial/shape analysis) 
> like may take time (30+ seconds or even more), this would the request time 
> out and fail.
> Client may retry the same requests multiple times or several minutes, this 
> would make things worse.
> The server side receives all the update requests but all except one can do 
> nothing, have to wait there. This wastes precious memory and cpu resource.
> We have seen the case 2000+ threads are blocking at the synchronize lock, and 
> only a few updates are making progress. Each thread takes 3+ mb memory which 
> causes OOM.
> Also if the update can't get the lock in expected time range, its better to 
> fail fast.
>  
> We can have one configuration in solrconfig.xml: 
> updateHandler/versionLock/timeInMill, so users can specify how long they want 
> to wait the version bucket lock.
> The default value can be -1, so it behaves same - wait forever until it gets 
> the lock.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (SOLR-13427) Support simulating the execution of autoscaling suggestion

2019-04-29 Thread Andrzej Bialecki (JIRA)



 [ 
https://issues.apache.org/jira/browse/SOLR-13427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrzej Bialecki  resolved SOLR-13427.
--
Resolution: Fixed

> Support simulating the execution of autoscaling suggestion
> --
>
> Key: SOLR-13427
> URL: https://issues.apache.org/jira/browse/SOLR-13427
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: AutoScaling
>Reporter: Andrzej Bialecki 
>Assignee: Andrzej Bialecki 
>Priority: Major
> Fix For: 8.1, master (9.0)
>
> Attachments: SOLR-13427.patch
>
>
> It's not always clear what would be the final state of the cluster after 
> applying the suggested changes (obtained from {{/autoscaling/suggestions}}), 
> especially on a large and busy cluster where several autoscaling rules have 
> to be considered.
> This issue proposes to use the simulation framework for simulating the 
> effects of the suggestions.
> First, the simulator would be initialized from the current state of a real 
> cluster. Then it would run several rounds of simulated execution of 
> suggestions until there were either no more suggestions (the cluster would be 
> perfectly balanced) or the iteration count limit was reached.
> This simulation could be executed using either the deployed autoscaling 
> config or one provided by the user, which would make it easier to test the 
> effects of various configurations on the cluster layout.
> Support for this functionality would be integrated into the existing 
> {{SolrCLI autoscaling}} tool.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-13427) Support simulating the execution of autoscaling suggestion



[ 
https://issues.apache.org/jira/browse/SOLR-13427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16829813#comment-16829813
 ] 

ASF subversion and git services commented on SOLR-13427:


Commit 44efae15a948b7bb6bf78d59ba110fe8996a8194 in lucene-solr's branch 
refs/heads/branch_8x from Andrzej Bialecki
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=44efae1 ]

SOLR-13427: Support simulating the execution of autoscaling suggestions.


> Support simulating the execution of autoscaling suggestion
> --
>
> Key: SOLR-13427
> URL: https://issues.apache.org/jira/browse/SOLR-13427
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: AutoScaling
>Reporter: Andrzej Bialecki 
>Assignee: Andrzej Bialecki 
>Priority: Major
> Fix For: 8.1, master (9.0)
>
> Attachments: SOLR-13427.patch
>
>
> It's not always clear what would be the final state of the cluster after 
> applying the suggested changes (obtained from {{/autoscaling/suggestions}}), 
> especially on a large and busy cluster where several autoscaling rules have 
> to be considered.
> This issue proposes to use the simulation framework for simulating the 
> effects of the suggestions.
> First, the simulator would be initialized from the current state of a real 
> cluster. Then it would run several rounds of simulated execution of 
> suggestions until there were either no more suggestions (the cluster would be 
> perfectly balanced) or the iteration count limit was reached.
> This simulation could be executed using either the deployed autoscaling 
> config or one provided by the user, which would make it easier to test the 
> effects of various configurations on the cluster layout.
> Support for this functionality would be integrated into the existing 
> {{SolrCLI autoscaling}} tool.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: 7.7.2?

2019-04-29 Thread Noble Paul

Thanks Jan

On Tue, Apr 30, 2019 at 7:33 AM Jan Høydahl  wrote:
>
> I'll vounteer as RM for 7.7.2 and aim at first RC on Tuesday May 7th, a week 
> from now.
> Please back-port important bug- or security fixes if you are aware of any 
> that are not in.
> Who can activate Jenkins jobs for branch_7_7?
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
>
> 29. apr. 2019 kl. 17:13 skrev Bram Van Dam :
>
> Hey folks,
>
> Was anyone planning on releasing 7.7.2? It seems like there are quite a
> few bug fixes, including the one with the high CPU usage.
>
> Full list of fixes so far: SOLR-13414, SOLR-13409, SOLR-13408,
> SOLR-13392, SOLR-13355, SOLR-13349, SOLR-13344, SOLR-13285, SOLR-13281,
> SOLR-13234, SOLR-13126, SOLR-12860, SOLR-12708, SOLR-12371, SOLR-11876
>
> Feel free to let me know if there's anything I can do to help out with
> the process, though I'm not a Solr committer.
>
> Thanks,
>
> - Bram
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>


-- 
-
Noble Paul

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-13427) Support simulating the execution of autoscaling suggestion

2019-04-29 Thread ASF subversion and git services (JIRA)



[ 
https://issues.apache.org/jira/browse/SOLR-13427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16829780#comment-16829780
 ] 

ASF subversion and git services commented on SOLR-13427:


Commit 6eccf2bf53899e0b1f47c8bb67cc5bca82966cb4 in lucene-solr's branch 
refs/heads/master from Andrzej Bialecki
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=6eccf2b ]

SOLR-13427: Support simulating the execution of autoscaling suggestions.


> Support simulating the execution of autoscaling suggestion
> --
>
> Key: SOLR-13427
> URL: https://issues.apache.org/jira/browse/SOLR-13427
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: AutoScaling
>Reporter: Andrzej Bialecki 
>Assignee: Andrzej Bialecki 
>Priority: Major
> Fix For: 8.1, master (9.0)
>
> Attachments: SOLR-13427.patch
>
>
> It's not always clear what would be the final state of the cluster after 
> applying the suggested changes (obtained from {{/autoscaling/suggestions}}), 
> especially on a large and busy cluster where several autoscaling rules have 
> to be considered.
> This issue proposes to use the simulation framework for simulating the 
> effects of the suggestions.
> First, the simulator would be initialized from the current state of a real 
> cluster. Then it would run several rounds of simulated execution of 
> suggestions until there were either no more suggestions (the cluster would be 
> perfectly balanced) or the iteration count limit was reached.
> This simulation could be executed using either the deployed autoscaling 
> config or one provided by the user, which would make it easier to test the 
> effects of various configurations on the cluster layout.
> Support for this functionality would be integrated into the existing 
> {{SolrCLI autoscaling}} tool.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11959) CDCR unauthorized to replicate to a target collection that is update protected in security.json



[ 
https://issues.apache.org/jira/browse/SOLR-11959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16829770#comment-16829770
 ] 

Jan Høydahl commented on SOLR-11959:


PKI is special since it is always activated and Solr is always able to use PKI 
cross nodes no matter what plugin you have activated. JWT plugin is not like 
that, you either choose Basic or Kerberos or JWT. And if we wrote a special 
code path for using JWT cross cluster, you'd still have to manage issuing and 
distributing those keys and tokens somehow, which in most cases means an 
external IdP software. I don't think we want to require such a complex 3rd 
party software for secure CDCR. That's why I propose to extend what PKI can do.

> CDCR unauthorized to replicate to a target collection that is update 
> protected in security.json
> ---
>
> Key: SOLR-11959
> URL: https://issues.apache.org/jira/browse/SOLR-11959
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Authentication, CDCR
>Affects Versions: 7.2
>Reporter: Donny Andrews
>Priority: Major
> Attachments: SOLR-11959.patch
>
>
> Steps to reproduce: 
>  # Create a source and a target collection in their respective clusters. 
>  # Update security.json to require a non-admin role to read and write. 
>  # Index to source collection 
> Expected: 
> The target collection should receive the update
> Actual:
> {code:java}
> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
> from server at http://redacted/solr/redacted: Expected mime type 
> application/octet-stream but got text/html. 
>  
>  
>  Error 401 Unauthorized request, Response code: 401
>  
>  HTTP ERROR 401
>  Problem accessing /solr/redacted/update. Reason:
>   Unauthorized request, Response code: 401
>  
>  at 
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:607)
>  at 
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
>  at 
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
>  at 
> org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
>  at 
> org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
>  at 
> org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1103)
>  at 
> org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:883)
>  at 
> org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:816)
>  at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
>  at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
>  at 
> org.apache.solr.handler.CdcrReplicator.sendRequest(CdcrReplicator.java:140)
>  at org.apache.solr.handler.CdcrReplicator.run(CdcrReplicator.java:104)
>  at 
> org.apache.solr.handler.CdcrReplicatorScheduler.lambda$null$0(CdcrReplicatorScheduler.java:81)
>  at 
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
>  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>  at java.lang.Thread.run(Thread.java:748){code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: 7.7.2?

2019-04-29 Thread Jan Høydahl

I'll vounteer as RM for 7.7.2 and aim at first RC on Tuesday May 7th, a week 
from now.
Please back-port important bug- or security fixes if you are aware of any that 
are not in.
Who can activate Jenkins jobs for branch_7_7?

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com

> 29. apr. 2019 kl. 17:13 skrev Bram Van Dam :
> 
> Hey folks,
> 
> Was anyone planning on releasing 7.7.2? It seems like there are quite a
> few bug fixes, including the one with the high CPU usage.
> 
> Full list of fixes so far: SOLR-13414, SOLR-13409, SOLR-13408,
> SOLR-13392, SOLR-13355, SOLR-13349, SOLR-13344, SOLR-13285, SOLR-13281,
> SOLR-13234, SOLR-13126, SOLR-12860, SOLR-12708, SOLR-12371, SOLR-11876
> 
> Feel free to let me know if there's anything I can do to help out with
> the process, though I'm not a Solr committer.
> 
> Thanks,
> 
> - Bram
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>

[jira] [Commented] (LUCENE-8780) Improve ByteBufferGuard in Java 11



[ 
https://issues.apache.org/jira/browse/LUCENE-8780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16829739#comment-16829739
 ] 

Uwe Schindler commented on LUCENE-8780:
---

I did a second patch that uses AtomicBoolean instead of VarHandles. The 
underlying code is the same: {{AtomicBoolean.getOpaque()}} calls a VarHandle 
behind the scenes (BTW, AtomicInteger, AtomicBoolean,... all were rewritten and 
use the VarHandle mechanism, see e.g., 
[https://hg.openjdk.java.net/jdk/jdk11/file/1ddf9a99e4ad/src/java.base/share/classes/java/util/concurrent/atomic/AtomicBoolean.java]).

Here is this version #2: 
https://github.com/apache/lucene-solr/compare/master...uschindler:jira/LUCENE-8780-v2

The effect was worse, so it's not an option. But this brings me to the 
conclusion: The actual calls using the other memory models are not actually the 
problem. Also the VarHandles and AtomicBooleans are correctly optimized away, 
but it looks like because of the complexity of the optimizations on the lowest 
level, it takes much longer until it gets faster and some optimizations are not 
applied at all (you cannot remove opaque reads, because the memory model 
eventually makes changes from other threads visible).

Here are the results from the AtomicBoolean:

{noformat}
Report after iter 10:
TaskQPS orig  StdDev   QPS patch  StdDev
Pct diff
  IntNRQ   22.48 (12.8%)   16.86 (13.0%)  
-25.0% ( -45% -0%)
PKLookup  112.32 (14.5%)   92.37  (7.6%)  
-17.8% ( -34% -5%)
   OrHighLow  207.87 (16.6%)  185.52  (4.1%)  
-10.8% ( -26% -   11%)
OrNotHighMed  992.06  (9.3%)  914.79  (1.5%)   
-7.8% ( -16% -3%)
HighSpanNear5.22 (10.2%)4.83  (5.4%)   
-7.5% ( -20% -9%)
  Fuzzy1   44.66  (9.7%)   41.46  (2.1%)   
-7.2% ( -17% -5%)
 MedSpanNear8.24 (18.2%)7.67 (12.4%)   
-6.9% ( -31% -   28%)
 LowSloppyPhrase6.91 (19.1%)6.46 (13.4%)   
-6.6% ( -32% -   31%)
Wildcard   43.23 (13.9%)   40.47  (6.0%)   
-6.4% ( -23% -   15%)
   LowPhrase   11.89 (11.4%)   11.18  (3.7%)   
-6.0% ( -18% -   10%)
OrHighNotMed 1188.55  (6.3%) 1118.58  (1.3%)   
-5.9% ( -12% -1%)
  Fuzzy2   66.58  (1.4%)   62.85  (1.7%)   
-5.6% (  -8% -   -2%)
  HighPhrase   32.87 (11.8%)   31.15  (8.8%)   
-5.2% ( -23% -   17%)
OrNotHighLow  537.79  (2.2%)  511.56  (8.8%)   
-4.9% ( -15% -6%)
 MedSloppyPhrase   44.16 (10.0%)   42.08  (2.3%)   
-4.7% ( -15% -8%)
   OrNotHighHigh  984.54  (2.2%)  942.20  (1.7%)   
-4.3% (  -8% -0%)
 AndHighHigh6.40 (12.1%)6.15 (12.8%)   
-3.9% ( -25% -   23%)
  AndHighMed   57.29  (9.9%)   55.19  (3.4%)   
-3.7% ( -15% -   10%)
HighSloppyPhrase4.60 (14.3%)4.44  (6.0%)   
-3.5% ( -20% -   19%)
   OrHighNotHigh  853.88  (2.7%)  824.51  (3.7%)   
-3.4% (  -9% -3%)
   MedPhrase   73.25  (2.6%)   70.85  (3.4%)   
-3.3% (  -9% -2%)
 LowTerm 1130.00  (5.4%) 1093.38  (2.5%)   
-3.2% ( -10% -4%)
   OrHighMed   34.61  (2.3%)   33.58  (3.1%)   
-3.0% (  -8% -2%)
 MedTerm  994.47  (7.9%)  975.29  (7.9%)   
-1.9% ( -16% -   15%)
OrHighNotLow  762.68  (3.0%)  749.09  (5.2%)   
-1.8% (  -9% -6%)
 Respell   53.06  (6.9%)   52.44 (11.5%)   
-1.2% ( -18% -   18%)
 LowSpanNear8.29  (5.3%)8.30 (10.0%)
0.1% ( -14% -   16%)
   HighTermDayOfYearSort   22.78  (6.4%)   22.88  (7.1%)
0.4% ( -12% -   14%)
HighTerm  822.80  (3.2%)  827.84  (8.2%)
0.6% ( -10% -   12%)
  OrHighHigh8.85 (10.0%)8.92 (14.5%)
0.8% ( -21% -   28%)
  AndHighLow  258.08  (4.7%)  261.18 (10.0%)
1.2% ( -12% -   16%)
   HighTermMonthSort   13.86  (9.2%)   14.63 (22.9%)
5.6% ( -24% -   41%)
 Prefix3   27.01 (10.1%)   28.72 (25.0%)
6.3% ( -26% -   46%)
{noformat}

As said before, removing the null check does not matter at all, it just makes 
the variance on short running tests less evident, but the average is identical.

> Improve ByteBufferGuard in Java 11
> --
>
> Key: LUCENE-8780
>

[jira] [Updated] (SOLR-13427) Support simulating the execution of autoscaling suggestion

2019-04-29 Thread Andrzej Bialecki (JIRA)



 [ 
https://issues.apache.org/jira/browse/SOLR-13427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrzej Bialecki  updated SOLR-13427:
-
Fix Version/s: 8.1

> Support simulating the execution of autoscaling suggestion
> --
>
> Key: SOLR-13427
> URL: https://issues.apache.org/jira/browse/SOLR-13427
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: AutoScaling
>Reporter: Andrzej Bialecki 
>Assignee: Andrzej Bialecki 
>Priority: Major
> Fix For: 8.1, master (9.0)
>
> Attachments: SOLR-13427.patch
>
>
> It's not always clear what would be the final state of the cluster after 
> applying the suggested changes (obtained from {{/autoscaling/suggestions}}), 
> especially on a large and busy cluster where several autoscaling rules have 
> to be considered.
> This issue proposes to use the simulation framework for simulating the 
> effects of the suggestions.
> First, the simulator would be initialized from the current state of a real 
> cluster. Then it would run several rounds of simulated execution of 
> suggestions until there were either no more suggestions (the cluster would be 
> perfectly balanced) or the iteration count limit was reached.
> This simulation could be executed using either the deployed autoscaling 
> config or one provided by the user, which would make it easier to test the 
> effects of various configurations on the cluster layout.
> Support for this functionality would be integrated into the existing 
> {{SolrCLI autoscaling}} tool.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-SmokeRelease-8.x - Build # 87 - Failure

Build: https://builds.apache.org/job/Lucene-Solr-SmokeRelease-8.x/87/

No tests ran.

Build Log:
[...truncated 23878 lines...]
[asciidoctor:convert] asciidoctor: ERROR: about-this-guide.adoc: line 1: 
invalid part, must have at least one section (e.g., chapter, appendix, etc.)
[asciidoctor:convert] asciidoctor: ERROR: solr-glossary.adoc: line 1: invalid 
part, must have at least one section (e.g., chapter, appendix, etc.)
 [java] Processed 2526 links (2067 relative) to 3355 anchors in 253 files
 [echo] Validated Links & Anchors via: 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.x/solr/build/solr-ref-guide/bare-bones-html/

-dist-changes:
 [copy] Copying 4 files to 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.x/solr/package/changes

package:

-unpack-solr-tgz:

-ensure-solr-tgz-exists:
[mkdir] Created dir: 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.x/solr/build/solr.tgz.unpacked
[untar] Expanding: 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.x/solr/package/solr-8.1.0.tgz
 into 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.x/solr/build/solr.tgz.unpacked

generate-maven-artifacts:

resolve:

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.x/lucene/top-level-ivy-settings.xml

resolve:

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:co

[jira] [Commented] (SOLR-11959) CDCR unauthorized to replicate to a target collection that is update protected in security.json

2019-04-29 Thread Amrit Sarkar (JIRA)



[ 
https://issues.apache.org/jira/browse/SOLR-11959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16829715#comment-16829715
 ] 

Amrit Sarkar commented on SOLR-11959:
-

Thanks, Jan. I see the direction. 

Can we utilize JWTAuthPlugin (JWT basically) solely (written by you, and I was 
reading it to understand how auth works overall in Solr) to have secure 
communication cross clusters? And what's needed outside of it is just to have 
network access (ports open) to respective zookeeper and Solr ports (which are 
already part of instructions).

I am not yet done with understanding the Auth module of Solr, and may suggest 
something doesn't make sense.

> CDCR unauthorized to replicate to a target collection that is update 
> protected in security.json
> ---
>
> Key: SOLR-11959
> URL: https://issues.apache.org/jira/browse/SOLR-11959
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Authentication, CDCR
>Affects Versions: 7.2
>Reporter: Donny Andrews
>Priority: Major
> Attachments: SOLR-11959.patch
>
>
> Steps to reproduce: 
>  # Create a source and a target collection in their respective clusters. 
>  # Update security.json to require a non-admin role to read and write. 
>  # Index to source collection 
> Expected: 
> The target collection should receive the update
> Actual:
> {code:java}
> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
> from server at http://redacted/solr/redacted: Expected mime type 
> application/octet-stream but got text/html. 
>  
>  
>  Error 401 Unauthorized request, Response code: 401
>  
>  HTTP ERROR 401
>  Problem accessing /solr/redacted/update. Reason:
>   Unauthorized request, Response code: 401
>  
>  at 
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:607)
>  at 
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
>  at 
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
>  at 
> org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
>  at 
> org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
>  at 
> org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1103)
>  at 
> org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:883)
>  at 
> org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:816)
>  at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
>  at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
>  at 
> org.apache.solr.handler.CdcrReplicator.sendRequest(CdcrReplicator.java:140)
>  at org.apache.solr.handler.CdcrReplicator.run(CdcrReplicator.java:104)
>  at 
> org.apache.solr.handler.CdcrReplicatorScheduler.lambda$null$0(CdcrReplicatorScheduler.java:81)
>  at 
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
>  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>  at java.lang.Thread.run(Thread.java:748){code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (SOLR-10053) TestSolrCloudWithDelegationTokens failures

2019-04-29 Thread Kevin Risden (JIRA)



 [ 
https://issues.apache.org/jira/browse/SOLR-10053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kevin Risden resolved SOLR-10053.
-
Resolution: Fixed

This test isn't disabled anymore, hasn't been failing, and after upgrade to 
Hadoop 3 HADOOP-14044 would have been fixed as well.

> TestSolrCloudWithDelegationTokens failures
> --
>
> Key: SOLR-10053
> URL: https://issues.apache.org/jira/browse/SOLR-10053
> Project: Solr
>  Issue Type: Test
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Ishan Chattopadhyaya
>Assignee: Ishan Chattopadhyaya
>Priority: Major
> Attachments: fail.log, stdout, stdout, stdout, stdout
>
>
> The TestSolrCloudWithDelegationTokens tests fail often at Jenkins. I have 
> been so far unable to reproduce them using the failing seeds. However, 
> beasting these tests seem to cause failures (once after about 10-12 runs).
> Latest Jenkins failure: 
> https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.4/12/
> It wasn't apparent what caused these failures. To cut down the noise on 
> Jenkins, I propose that we disable the test with @AwaitsFix (or bad apple) 
> annotation and continue to debug and fix this test.
> WDYT, [~markrmil...@gmail.com]?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (SOLR-9586) TestSolrCloudWithDelegationTokens fails regularly on Jenkins runs

2019-04-29 Thread Kevin Risden (JIRA)



 [ 
https://issues.apache.org/jira/browse/SOLR-9586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kevin Risden resolved SOLR-9586.

Resolution: Fixed

Looks like this was fixed in SOLR-10053

> TestSolrCloudWithDelegationTokens fails regularly on Jenkins runs
> -
>
> Key: SOLR-9586
> URL: https://issues.apache.org/jira/browse/SOLR-9586
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Alan Woodward
>Priority: Major
>
> Mainly on Windows, sometimes on Solaris.  Failing seeds don't reproduce on a 
> Mac.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (SOLR-13429) HashBasedRouter logs the entire state.json when a slice is not found

2019-04-29 Thread ASF subversion and git services (JIRA)



 [ 
https://issues.apache.org/jira/browse/SOLR-13429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Noble Paul resolved SOLR-13429.
---
   Resolution: Fixed
Fix Version/s: master (9.0)
   8.1

> HashBasedRouter logs the entire state.json when a slice is not found
> 
>
> Key: SOLR-13429
> URL: https://issues.apache.org/jira/browse/SOLR-13429
> Project: Solr
>  Issue Type: Task
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Noble Paul
>Assignee: Noble Paul
>Priority: Trivial
> Fix For: 8.1, master (9.0)
>
>
> {code:java}
> protected Slice hashToSlice(int hash, DocCollection collection) {
> final Slice[] slices = collection.getActiveSlicesArr();
> for (Slice slice : slices) {
>   Range range = slice.getRange();
>   if (range != null && range.includes(hash)) return slice;
> }
> throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "No active 
> slice servicing hash code " + Integer.toHexString(hash) + " in " + 
> collection);
>   }
> {code}
> Just the collection name should be fine



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-13429) HashBasedRouter logs the entire state.json when a slice is not found



[ 
https://issues.apache.org/jira/browse/SOLR-13429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16829674#comment-16829674
 ] 

ASF subversion and git services commented on SOLR-13429:


Commit 94b9f7ed1cf3d26fed340d777b4105766dd5c45f in lucene-solr's branch 
refs/heads/branch_8x from Noble Paul
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=94b9f7e ]

SOLR-13429: HashBasedRouter logs the entire state.json when a slice is not found


> HashBasedRouter logs the entire state.json when a slice is not found
> 
>
> Key: SOLR-13429
> URL: https://issues.apache.org/jira/browse/SOLR-13429
> Project: Solr
>  Issue Type: Task
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Noble Paul
>Assignee: Noble Paul
>Priority: Trivial
>
> {code:java}
> protected Slice hashToSlice(int hash, DocCollection collection) {
> final Slice[] slices = collection.getActiveSlicesArr();
> for (Slice slice : slices) {
>   Range range = slice.getRange();
>   if (range != null && range.includes(hash)) return slice;
> }
> throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "No active 
> slice servicing hash code " + Integer.toHexString(hash) + " in " + 
> collection);
>   }
> {code}
> Just the collection name should be fine



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-13429) HashBasedRouter logs the entire state.json when a slice is not found

2019-04-29 Thread ASF subversion and git services (JIRA)



[ 
https://issues.apache.org/jira/browse/SOLR-13429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16829673#comment-16829673
 ] 

ASF subversion and git services commented on SOLR-13429:


Commit 25bd1cbb82f45aed67bca129e0c38ec300e7142d in lucene-solr's branch 
refs/heads/master from Noble Paul
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=25bd1cb ]

SOLR-13429: HashBasedRouter logs the entire state.json when a slice is not found


> HashBasedRouter logs the entire state.json when a slice is not found
> 
>
> Key: SOLR-13429
> URL: https://issues.apache.org/jira/browse/SOLR-13429
> Project: Solr
>  Issue Type: Task
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Noble Paul
>Priority: Trivial
>
> {code:java}
> protected Slice hashToSlice(int hash, DocCollection collection) {
> final Slice[] slices = collection.getActiveSlicesArr();
> for (Slice slice : slices) {
>   Range range = slice.getRange();
>   if (range != null && range.includes(hash)) return slice;
> }
> throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "No active 
> slice servicing hash code " + Integer.toHexString(hash) + " in " + 
> collection);
>   }
> {code}
> Just the collection name should be fine



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Assigned] (SOLR-13429) HashBasedRouter logs the entire state.json when a slice is not found



 [ 
https://issues.apache.org/jira/browse/SOLR-13429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Noble Paul reassigned SOLR-13429:
-

Assignee: Noble Paul

> HashBasedRouter logs the entire state.json when a slice is not found
> 
>
> Key: SOLR-13429
> URL: https://issues.apache.org/jira/browse/SOLR-13429
> Project: Solr
>  Issue Type: Task
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Noble Paul
>Assignee: Noble Paul
>Priority: Trivial
>
> {code:java}
> protected Slice hashToSlice(int hash, DocCollection collection) {
> final Slice[] slices = collection.getActiveSlicesArr();
> for (Slice slice : slices) {
>   Range range = slice.getRange();
>   if (range != null && range.includes(hash)) return slice;
> }
> throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "No active 
> slice servicing hash code " + Integer.toHexString(hash) + " in " + 
> collection);
>   }
> {code}
> Just the collection name should be fine



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-repro-Java11 - Build # 40 - Unstable

Build: https://builds.apache.org/job/Lucene-Solr-repro-Java11/40/

[...truncated 29 lines...]
[repro] Jenkins log URL: 
https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1835/consoleText

[repro] Revision: f77c56dbc636ec2305826c1734ae0885f222dd03

[repro] Ant options: -Dtests.multiplier=2 
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt
[repro] Repro line:  ant test  
-Dtestcase=HdfsTlogReplayBufferedWhileIndexingTest -Dtests.method=test 
-Dtests.seed=45362653984B5375 -Dtests.multiplier=2 -Dtests.nightly=true 
-Dtests.slow=true 
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt
 -Dtests.locale=kw-GB -Dtests.timezone=Asia/Brunei -Dtests.asserts=true 
-Dtests.file.encoding=UTF-8

[repro] Repro line:  ant test  -Dtestcase=HdfsAutoAddReplicasIntegrationTest 
-Dtests.method=testSimple -Dtests.seed=45362653984B5375 -Dtests.multiplier=2 
-Dtests.nightly=true -Dtests.slow=true 
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt
 -Dtests.locale=sk -Dtests.timezone=America/New_York -Dtests.asserts=true 
-Dtests.file.encoding=UTF-8

[repro] git rev-parse --abbrev-ref HEAD
[repro] git rev-parse HEAD
[repro] Initial local git branch/revision: 
ced0243a3eea9360993035842e556c1daf9f4bd0
[repro] git fetch
[repro] git checkout f77c56dbc636ec2305826c1734ae0885f222dd03

[...truncated 2 lines...]
[repro] git merge --ff-only

[...truncated 1 lines...]
[repro] ant clean

[...truncated 6 lines...]
[repro] Test suites by module:
[repro]solr/core
[repro]   HdfsAutoAddReplicasIntegrationTest
[repro]   HdfsTlogReplayBufferedWhileIndexingTest
[repro] ant compile-test

[...truncated 3309 lines...]
[repro] ant test-nocompile -Dtests.dups=5 -Dtests.maxfailures=10 
-Dtests.class="*.HdfsAutoAddReplicasIntegrationTest|*.HdfsTlogReplayBufferedWhileIndexingTest"
 -Dtests.showOutput=onerror -Dtests.multiplier=2 
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt
 -Dtests.seed=45362653984B5375 -Dtests.multiplier=2 -Dtests.nightly=true 
-Dtests.slow=true 
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt
 -Dtests.locale=sk -Dtests.timezone=America/New_York -Dtests.asserts=true 
-Dtests.file.encoding=UTF-8

[...truncated 5052 lines...]
[repro] Setting last failure code to 256

[repro] Failures:
[repro]   0/5 failed: 
org.apache.solr.cloud.hdfs.HdfsTlogReplayBufferedWhileIndexingTest
[repro]   2/5 failed: 
org.apache.solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest
[repro] git checkout ced0243a3eea9360993035842e556c1daf9f4bd0

[...truncated 2 lines...]
[repro] Exiting with code 256

[...truncated 6 lines...]

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-13394) Change default GC from CMS to G1

2019-04-29 Thread Shawn Heisey (JIRA)



[ 
https://issues.apache.org/jira/browse/SOLR-13394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16829606#comment-16829606
 ] 

Shawn Heisey commented on SOLR-13394:
-

bq. The default region size is very large at 16MB. G1GC recommends ~2048 
regions which would make this setting appropriate only for 32GB heaps or more. 
Can we remove this explicit setting and let G1 choose the default.

I would be curious about whether the latest Java 8 does what Oracle engineers 
promised with regard to better handling of humongous allocations.

An index with 125 million documents will have filterCache entries that are 
about 15MB each.  In order for those to not be considered humongous 
allocations, the region size must be set to its maximum -- 32MB.

I wonder if we need some minimal documentation about GC tuning, that at least 
mentions the region size parameter.

> Change default GC from CMS to G1
> 
>
> Key: SOLR-13394
> URL: https://issues.apache.org/jira/browse/SOLR-13394
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Ishan Chattopadhyaya
>Assignee: Ishan Chattopadhyaya
>Priority: Major
> Fix For: 8.1
>
> Attachments: SOLR-13394.patch
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> CMS has been deprecated in new versions of Java 
> (http://openjdk.java.net/jeps/291). This issue is to switch Solr default from 
> CMS to G1.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-13320) add a param ignoreVersionConflicts=true to updates to not overwrite existing docs



[ 
https://issues.apache.org/jira/browse/SOLR-13320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16829602#comment-16829602
 ] 

Noble Paul commented on SOLR-13320:
---

Modifying the payload is not always a good idea. Users usually already have the 
data in some format or there are tools that generate that data. This purely 
seems like a requirement added by Solr and does not belong to the data. We 
shouldn't ask the users to generate data in our format, instead we should work 
with any  format they have.

I'll check it in non cloud as well.


> add a param ignoreVersionConflicts=true to updates to not overwrite existing 
> docs
> -
>
> Key: SOLR-13320
> URL: https://issues.apache.org/jira/browse/SOLR-13320
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Noble Paul
>Assignee: Noble Paul
>Priority: Major
> Attachments: SOLR-13320.patch
>
>
> Updates should have an option to ignore duplicate documents and drop them if 
> an option  {{ignoreDuplicates=true}} is specified



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (LUCENE-4312) Index format to store position length per position

2019-04-29 Thread Michael Gibney (JIRA)



[ 
https://issues.apache.org/jira/browse/LUCENE-4312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16829599#comment-16829599
 ] 

Michael Gibney edited comment on LUCENE-4312 at 4/29/19 6:26 PM:
-

(following on discussion at 
[LUCENE-8776|https://issues.apache.org/jira/browse/LUCENE-8776?focusedCommentId=16829561&#comment-16829561]):
 [~mikemccand], agreed that for query-time cases, graphs can and should be 
handled as you say.

But functionality that relies on reconstituting _index_ time token graphs (as 
produced by synonym-/WDGF-type analysis) would require position length to be 
stored in the index (and reported back via PostingsEnum at query-time), no? And 
per [~rcmuir]'s suggestion above, the only way to do this currently would be by 
encoding position length in Payloads.

My question is a more general one about the utility of indexing position 
length, and whether anyone would be interested in considering integrating full 
support for indexed token graphs.


was (Author: mgibney):
(following on discussion at 
[LUCENE-8776|https://issues.apache.org/jira/browse/LUCENE-8776?focusedCommentId=16829561&#comment-16829561]):
 [~mikemccand], agreed that for query-time cases, graphs can and should be 
handled as you say.

But functionality that relies on reconstituting _index_-time token graphs (as 
produced by synonym-/WDGF-type analysis) would require position length to be 
stored in the index (and reported back via PostingsEnum at query-time), no? And 
per [~rcmuir]'s suggestion above, the only way to do this currently would be by 
encoding position length in Payloads.

My question is a more general one about the utility of indexing position 
length, and whether anyone would be interested in considering integrating full 
support for indexed token graphs.

> Index format to store position length per position
> --
>
> Key: LUCENE-4312
> URL: https://issues.apache.org/jira/browse/LUCENE-4312
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: core/codecs
>Affects Versions: 6.0
>Reporter: Gang Luo
>Priority: Minor
>  Labels: Suggestion
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> Mike Mccandless said:TokenStreams are actually graphs.
> Indexer ignores PositionLengthAttribute.Need change the index format (and 
> Codec APIs) to store an additional int position length per position.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-4312) Index format to store position length per position

2019-04-29 Thread Michael Gibney (JIRA)



[ 
https://issues.apache.org/jira/browse/LUCENE-4312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16829599#comment-16829599
 ] 

Michael Gibney commented on LUCENE-4312:


(following on discussion at 
[LUCENE-8776|https://issues.apache.org/jira/browse/LUCENE-8776?focusedCommentId=16829561&#comment-16829561]):
 [~mikemccand], agreed that for query-time cases, graphs can and should be 
handled as you say.

But functionality that relies on reconstituting _index_-time token graphs (as 
produced by synonym-/WDGF-type analysis) would require position length to be 
stored in the index (and reported back via PostingsEnum at query-time), no? And 
per [~rcmuir]'s suggestion above, the only way to do this currently would be by 
encoding position length in Payloads.

My question is a more general one about the utility of indexing position 
length, and whether anyone would be interested in considering integrating full 
support for indexed token graphs.

> Index format to store position length per position
> --
>
> Key: LUCENE-4312
> URL: https://issues.apache.org/jira/browse/LUCENE-4312
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: core/codecs
>Affects Versions: 6.0
>Reporter: Gang Luo
>Priority: Minor
>  Labels: Suggestion
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> Mike Mccandless said:TokenStreams are actually graphs.
> Indexer ignores PositionLengthAttribute.Need change the index format (and 
> Codec APIs) to store an additional int position length per position.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-5970) Create collection API always has status 0

2019-04-29 Thread Jason Gerlowski (JIRA)



[ 
https://issues.apache.org/jira/browse/SOLR-5970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16829569#comment-16829569
 ] 

Jason Gerlowski commented on SOLR-5970:
---

Thanks for seeing this through Ishan, and thanks for updating my patch 
Kesharee.  Sorry to leave it hanging.  I let it sit for a few hoping for 
feedback before committing and then it fell off my radar.  Glad this will make 
it into 8.1!

> Create collection API always has status 0
> -
>
> Key: SOLR-5970
> URL: https://issues.apache.org/jira/browse/SOLR-5970
> Project: Solr
>  Issue Type: Bug
>Reporter: Abraham Elmahrek
>Assignee: Ishan Chattopadhyaya
>Priority: Major
> Fix For: 8.1
>
> Attachments: SOLR-5970-test.patch, SOLR-5970.patch, SOLR-5970.patch, 
> SOLR-5970_branch_8x.patch, bad.jar, schema.xml, solrconfig.xml
>
>
> The responses below are from a successful create collection API 
> (https://cwiki.apache.org/confluence/display/solr/Collections+API#CollectionsAPI-CreateormodifyanAliasforaCollection)
>  call and an unsuccessful create collection API call. It seems the 'status' 
> is always 0.
> Success:
> {u'responseHeader': {u'status': 0, u'QTime': 4421}, u'success': {u'': 
> {u'core': u'test1_shard1_replica1', u'responseHeader': {u'status': 0, 
> u'QTime': 3449
> Failure:
> {u'failure': 
>   {u'': 
> u"org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:Error 
> CREATEing SolrCore 'test43_shard1_replica1': Unable to create core: 
> test43_shard1_replica1 Caused by: Could not find configName for collection 
> test43 found:[test1]"},
>  u'responseHeader': {u'status': 0, u'QTime': 17149}
> }
> It seems like the status should be 400 or something similar for an 
> unsuccessful attempt?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: significant lucene benchmark regression: JDK11?

2019-04-29 Thread Michael McCandless

OK I'll make this change soon and reply back.

Mike McCandless

http://blog.mikemccandless.com


On Fri, Apr 26, 2019 at 10:17 AM David Smiley 
wrote:

> +1 to choose the Parallel Collector temporarily as Uwe suggests so we can
> see distinct effects of the separate changes.
>
> I suppose the benchmarks should continue to prefer the default JVM
> settings, as it is how users will consume Lucene.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Thu, Apr 25, 2019 at 6:03 PM Uwe Schindler  wrote:
>
>> Maybe do this temporary, to not have 2 changes at the same time.
>>
>> Uwe
>>
>> Am April 25, 2019 9:48:35 PM UTC schrieb Michael McCandless <
>> luc...@mikemccandless.com>:
>>>
>>> Yeah I'm just using the JDK's default in the nightly benchmarks.
>>>
>>> Should I override back to the parallel collector?
>>>
>>> Mike McCandless
>>>
>>> http://blog.mikemccandless.com
>>>
>>>
>>> On Thu, Apr 25, 2019 at 5:44 PM Uwe Schindler  wrote:
>>>
 Hi,

 I am not sure how Mike's benchmarks are setup and if he chooses a
 specific garbage collector.

 Java 8 defaults to ParallelGC, Java 11 defaults to G1, which may slow
 down up to 10% as it is not optimized for throughput.

 So to compare you gave to be specific in your GC choices.

 Uwe

 Am April 25, 2019 5:57:16 PM UTC schrieb Nicholas Knize <
 nkn...@gmail.com>:
>
> Earlier this week I noticed a significant across the board performance
> regression on the nightly geo benchmarks
> . It appears this
> regression can also be seen on other lucene benchmarks
>  and
> appears to correspond to the upgrade to JDK 11.
>
> Any thoughts?
>
> Nicholas Knize, Ph.D., GISP
> Geospatial Software Guy  |  Elasticsearch
> Apache Lucene PMC Member and Committer
> nkn...@apache.org
>

 --
 Uwe Schindler
 Achterdiek 19, 28357 Bremen
 https://www.thetaphi.de

>>>
>> --
>> Uwe Schindler
>> Achterdiek 19, 28357 Bremen
>> https://www.thetaphi.de
>>
>

[jira] [Commented] (LUCENE-8776) Start offset going backwards has a legitimate purpose

2019-04-29 Thread Michael McCandless (JIRA)



[ 
https://issues.apache.org/jira/browse/LUCENE-8776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16829561#comment-16829561
 ] 

Michael McCandless commented on LUCENE-8776:


[~venkat11] I'm sorry this change broke your use case.

I think allowing backwards offsets was an accidental but longstanding bug in 
prior versions of Lucene.  It is unfortunate your code came to rely on that 
bug, but we need to be able to fix our bugs and move forwards.

[~mgibney] a 3rd option in your list would be for [~venkat11] to fix his query 
parser to properly consume the graph, and generate fully accurate queries, the 
way Lucene's query parsers now do.  Then you can have precisely matching 
queries, no bugs.

> Start offset going backwards has a legitimate purpose
> -
>
> Key: LUCENE-8776
> URL: https://issues.apache.org/jira/browse/LUCENE-8776
> Project: Lucene - Core
>  Issue Type: Bug
>  Components: core/search
>Affects Versions: 7.6
>Reporter: Ram Venkat
>Priority: Major
>
> Here is the use case where startOffset can go backwards:
> Say there is a line "Organic light-emitting-diode glows", and I want to run 
> span queries and highlight them properly. 
> During index time, light-emitting-diode is split into three words, which 
> allows me to search for 'light', 'emitting' and 'diode' individually. The 
> three words occupy adjacent positions in the index, as 'light' adjacent to 
> 'emitting' and 'light' at a distance of two words from 'diode' need to match 
> this word. So, the order of words after splitting are: Organic, light, 
> emitting, diode, glows. 
> But, I also want to search for 'organic' being adjacent to 
> 'light-emitting-diode' or 'light-emitting-diode' being adjacent to 'glows'. 
> The way I solved this was to also generate 'light-emitting-diode' at two 
> positions: (a) In the same position as 'light' and (b) in the same position 
> as 'glows', like below:
> ||organic||light||emitting||diode||glows||
> | |light-emitting-diode| |light-emitting-diode| |
> |0|1|2|3|4|
> The positions of the two 'light-emitting-diode' are 1 and 3, but the offsets 
> are obviously the same. This works beautifully in Lucene 5.x in both 
> searching and highlighting with span queries. 
> But when I try this in Lucene 7.6, it hits the condition "Offsets must not go 
> backwards" at DefaultIndexingChain:818. This IllegalArgumentException is 
> being thrown without any comments on why this check is needed. As I explained 
> above, startOffset going backwards is perfectly valid, to deal with word 
> splitting and span operations on these specialized use cases. On the other 
> hand, it is not clear what value is added by this check and which highlighter 
> code is affected by offsets going backwards. This same check is done at 
> BaseTokenStreamTestCase:245. 
> I see others talk about how this check found bugs in WordDelimiter etc. but 
> it also prevents legitimate use cases. Can this check be removed?  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8756) MLT queries ignore custom term frequencies

2019-04-29 Thread Michael McCandless (JIRA)



[ 
https://issues.apache.org/jira/browse/LUCENE-8756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16829556#comment-16829556
 ] 

Michael McCandless commented on LUCENE-8756:


Ahh thanks for the ping [~ollik1] I agree we need to fix this; I'll have a look 
at the PR, thanks!

> MLT queries ignore custom term frequencies
> --
>
> Key: LUCENE-8756
> URL: https://issues.apache.org/jira/browse/LUCENE-8756
> Project: Lucene - Core
>  Issue Type: Bug
>  Components: core/queryparser
>Affects Versions: 7.0, 7.0.1, 7.1, 7.2, 7.2.1, 7.3, 7.4, 7.3.1, 7.5, 7.6, 
> 7.7, 7.7.1, 8.0
>Reporter: Olli Kuonanoja
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> The MLT queries ignore any custom term frequencies for the like-texts and 
> uses a hard-coded frequency of 1 per occurrence. I have prepared a test-case 
> to demonstrate the issue and a fix proposal 
> https://github.com/ollik1/lucene-solr/commit/9dbbce2af26698cec1ac82a526d9cee60a880678
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8780) Improve ByteBufferGuard in Java 11

2019-04-29 Thread Robert Muir (JIRA)



[ 
https://issues.apache.org/jira/browse/LUCENE-8780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16829444#comment-16829444
 ] 

Robert Muir commented on LUCENE-8780:
-

i imagine any slowdown only impacts stuff doing lots of tiny reads vs reading 
big byte[].
The current code in most cases will deliver ACE in someones unit test rather 
than a crash. Maybe if you are lucky you even get an ACE rather than sigsegv in 
production too. To me it seems like the promise of this patch is that you stand 
a better chance, even after code is running for a while (10k times or 
whatever). our best effort check will no longer be optimized away? but thats 
also its downside: it wont get optimized away even if you have no bugs in your 
code. 

seems like we just decide on the correct tradeoff. personally i lean towards 
more aggressive safety: the user is using java, they dont expect sigsegv. just 
like they dont expect to run out of mmaps because its tied to gc, and they dont 
expect all their searches bottlenecked on one file handle, due to stupid 
synchronizatiom on a relative file pointer we dont even care about.

> Improve ByteBufferGuard in Java 11
> --
>
> Key: LUCENE-8780
> URL: https://issues.apache.org/jira/browse/LUCENE-8780
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: core/store
>Affects Versions: master (9.0)
>Reporter: Uwe Schindler
>Assignee: Uwe Schindler
>Priority: Major
>  Labels: Java11
> Attachments: LUCENE-8780.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> In LUCENE-7409 we added {{ByteBufferGuard}} to protect MMapDirectory from 
> crushing the JVM with SIGSEGV when you close and unmap the mmapped buffers of 
> an IndexInput, while another thread is accessing it.
> The idea was to do a volatile write access to flush the caches (to trigger a 
> full fence) and set a non-volatile boolean to true. All accesses would check 
> the boolean and stop the caller from accessing the underlying ByteBuffer. 
> This worked most of the time, until the JVM optimized away the plain read 
> access to the boolean (you can easily see this after some runtime of our 
> by-default ignored testcase).
> With master on Java 11, we can improve the whole thing. Using VarHandles you 
> can use any access type when reading or writing the boolean. After reading 
> Doug Lea's expanation  and some 
> testing, I was no longer able to crush my JDK (even after running for minutes 
> unmapping bytebuffers).
> The apraoch is the same, we do a full-fenced write (standard volatile write) 
> when we unmap, then we yield the thread (to finish in-flight reads in other 
> threads) and then unmap all byte buffers.
> On the test side (read access), instead of using a plain read, we use the new 
> "opaque read". Opaque reads are the same as plain reads, there are only 
> different order requirements. Actually the main difference is explained by 
> Doug like this: "For example in constructions in which the only modification 
> of some variable x is for one thread to write in Opaque (or stronger) mode, 
> X.setOpaque(this, 1), any other thread spinning in 
> while(X.getOpaque(this)!=1){} will eventually terminate. Note that this 
> guarantee does NOT hold in Plain mode, in which spin loops may (and usually 
> do) infinitely loop -- they are not required to notice that a write ever 
> occurred in another thread if it was not seen on first encounter." - And 
> that's waht we want to have: We don't want to do volatile reads, but we want 
> to prevent the compiler from optimizing away our read to the boolean. So we 
> want it to "eventually" see the change. By the much stronger volatile write, 
> the cache effects should be visible even faster (like in our Java 8 approach, 
> just now we improved our read side).
> The new code is much slimmer (theoretically we could also use a AtomicBoolean 
> for that and use the new method {{getOpaque()}}, but I wanted to prevent 
> extra method calls, so I used a VarHandle directly).
> It's setup like this:
> - The underlying boolean field is a private member (with unused 
> SuppressWarnings, as its unused by the java compiler), marked as volatile 
> (that's the recommendation, but in reality it does not matter at all).
> - We create a VarHandle to access this boolean, we never do this directly 
> (this is why the volatile marking does not affect us).
> - We use VarHandle.setVolatile() to change our "invalidated" boolean to 
> "true", so enforcing a full fence
> - On the read side we use VarHandle.getOpaque() instead of VarHandle.get() 
> (like in our old code for Java 8).
> I had to tune our test a bit, as the VarHandles make it take longer until it 
> actually crushes (as optimizations jump in later)

[jira] [Comment Edited] (LUCENE-8780) Improve ByteBufferGuard in Java 11

[
https://issues.apache.org/jira/browse/LUCENE-8780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16829428#comment-16829428
]

Uwe Schindler edited comment on LUCENE-8780 at 4/29/19 4:34 PM:

Hi,
I did some tests over the last night. The fluctuating results can be improved
by removing the null-check on the "cleaner". Normally this one should be
optimized away, but that seems to be too hard. Without it, you still have some
variance in the results, but it's better.

In general the new code is a bit slower, which is more visible if you have
shorter runtime, so it looks like optimizing aways the VarHandle simply takes
longer, and this degrades oaverage performance. I don't know how long the
benchmark runs a warmup.

bq. Every time I read about those memory ordering models I seem to get it and
two weeks later I'm again confused...

That's the ssme for me. But our use-case seems to be pretty good explained in
the above quotes by Doug Lea. It clearly says that a volatile write is harder
than a opaque read, the volatile write is is only visible eventually to the
opaque read (but never for a plain read, if it was optimized away).

I was already talking with Hotspot guys: In short, there is no way to make
ByteBufferGuard "safe" unless you use a volatile or with an aquire/release
fence (that can be implemented also with the varhandles). When using it, I was
unable to crush my VM, but slowdown is definitely there. Also manually putting
in fences does not help.

The main difference between plain read and opaque (as used here) is not that
the produced CPU instructions are different, the difference is only how the
optimizer may optimize code. With plain reads, a simple spin-loop will never
end, as the hotspot compiler clearly sees that the value can't change, and
because it's not volatile, opaque or whatever, he will remove it for this
thread. This does not only happen in spin loops, it also happens in our code if
it's executed often enough. This is not a bug, it's wanted.

I tuned the testcase to crush always: Just insert a loop of 10.000 reads BEFORE
you unmap, then it fails always (in Java 8). With opaque it was also doing
this, but not reproducible, so there is an improvement. IMHO, the reason why we
see a slowdown for some queries could be coming from that: Hotspot is removing
the "if (invalidated)" check, as it cannot change in the current thread. With
opaque it can't do it, so the code is slower on the long term. I will try to
get some assembly output later, just have to install hsdis.dll.

To make it bulletproof, we have to wait for official support by the ByteBuffer
API to officially unmap (https://bugs.openjdk.java.net/browse/JDK-4724038),
with our checks we cannot make it safe. Another approach would be to expand the
SIGSEGV signal handler checks in the JVM to throw an InternalError (they do it
now if the operating system truncates the file and so the mapped are changes).
I don't know why they not generally do a SIGSEGV check and if it happens inside
DirectBuffer they just throw an Exception (possibly delayed as in the
truncated file case).

So we have to decide what we want to do as a workaround:
- We can't be bulletproof
- We can catch with our current code most cases where you open an index, start
a query and then close the reader. This only works shortly after program
started and everything is not yet optimized. As soon as there were multiple
queries already running and you close the JVM later, it SIGSEGV almost always.
Not even "opaque" semantics help here. With "plain" code (current code in
branch_8x), at the time when optimizer decides to optimize, the checks are gone.
- With the current "opaque" code we can go a bit further, but we have some
slowdown, but it's still not save.

was (Author: thetaphi):
Hi,
I did some tests over the last night. The fluctuating results can be improved
by removing the null-check on the "cleaner". Normally this one should be
optimized away, but that seems to be too hard. Without it, you still have some
variance in the results, but it's better.

bq. Every time I read about those memory ordering models I seem to get it and
two weeks later I'm again confused...

I was already talking with Hotspot guys: In short, there is no way to make
ByteBufferGuard "safe" unless yo

[jira] [Comment Edited] (LUCENE-8780) Improve ByteBufferGuard in Java 11

[
https://issues.apache.org/jira/browse/LUCENE-8780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16829428#comment-16829428
]

Uwe Schindler edited comment on LUCENE-8780 at 4/29/19 4:31 PM:

bq. Every time I read about those memory ordering models I seem to get it and
two weeks later I'm again confused...

So we have to decide what we want to do as a workaround:
- We can't be bulletproof
- We can catch with our current code most cases where you open an index, start
a query and then close the reader. This only works shortly after program
started and everything is not yet optimized. As soon as there were multiple
queries already running and you close the JVM later, it SIGSEGV almost always.
Not even "opaque" semantics help here.
- With the current code we can go a bit further, but we have some slowdown, but
it's still not save.

bq. Every time I read about those memory ordering models I seem to get it and
two weeks later I'm again confused...

That's the ssme for me. But our use-case seems to be pretty good explained in
the above quotes by Doug Lea. It clearly says that a volatile write is harder
than a opaque read, but the opaque read is only visible eventually.

[JENKINS-MAVEN] Lucene-Solr-Maven-master #2550: POMs out of sync

Build: https://builds.apache.org/job/Lucene-Solr-Maven-master/2550/

No tests ran.

Build Log:
[...truncated 18175 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Maven-master/build.xml:673: The 
following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Maven-master/build.xml:209: The 
following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Maven-master/lucene/build.xml:413:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Maven-master/lucene/common-build.xml:2180:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Maven-master/lucene/analysis/build.xml:129:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Maven-master/lucene/analysis/build.xml:38:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Maven-master/lucene/common-build.xml:1648:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Maven-master/lucene/common-build.xml:581:
 Error deploying artifact 'org.apache.lucene:lucene-analyzers-common:jar': 
Error installing artifact's metadata: Error while deploying metadata: Error 
transferring file

Total time: 9 minutes 4 seconds
Build step 'Invoke Ant' marked build as failure
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8780) Improve ByteBufferGuard in Java 11



[ 
https://issues.apache.org/jira/browse/LUCENE-8780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16829428#comment-16829428
 ] 

Uwe Schindler commented on LUCENE-8780:
---

Hi,
I did some tests over the last night. The fluctuating results can be improved 
by removing the null-check on the "cleaner". Normally this one should be 
optimized away, but that seems to be too hard. Without it, you still have some 
variance in the results, but it's better.

In general the new code is a bit slower, which is more visible if you have 
shorter runtime, so it looks like optimizing aways the VarHandle simply takes 
longer, and this degrades oaverage performance. I don't know how long the 
benchmark runs a warmup.

bq. Every time I read about those memory ordering models I seem to get it and 
two weeks later I'm again confused...

That's the ssme for me. But our use-case seems to be pretty good explained in 
the above quotes by Doug Lea. It clearly says that a volatile write is harder 
than a opaque read, but the opaque read is only visible eventually.

I was already talking with Hotspot guys: In short, there is no way to make 
ByteBufferGuard "safe" unless you use a volatile or with an aquire/release 
fence (that can be implemented also with the varhandles). When using it, I was 
unable to crush my VM, but slowdown is definitely there. Also manually putting 
in fences does not help.

The main difference between plain read and opaque (as used here) is not that 
the produced CPU instructions are different, the difference is only how the 
optimizer may optimize code. With plain reads, a simple spin-loop will never 
end, as the hotspot compiler clearly sees that the value can't change, and 
because it's not volatile, opaque or whatever, he will remove it for this 
thread. This does not only happen in spin loops, it also happens in our code if 
it's executed often enough. This is not a bug, it's wanted.

I tuned the testcase to crush always: Just insert a loop of 10.000 reads BEFORE 
you unmap, then it fails always (in Java 8). With opaque it was also doing 
this, but not reproducible, so there is an improvement. IMHO, the reason why we 
see a slowdown for some queries could be coming from that: Hotspot is removing 
the "if (invalidated)" check, as it cannot change in the current thread. With 
opaque it can't do it, so the code is slower on the long term. I will try to 
get some assembly output later, just have to install hsdis.dll.

To make it bulletproof, we have to wait for official support by the ByteBuffer 
API to officially unmap (https://bugs.openjdk.java.net/browse/JDK-4724038), 
with our checks we cannot make it safe. Another approach would be to expand the 
SIGSEGV signal handler checks in the JVM to throw an InternalError (they do it 
now if the operating system truncates the file and so the mapped are changes). 
I don't know why they not generally do a SIGSEGV check and if it happens inside 
DirectBuffer they just throw an Exception (possibly delayed as in the 
truncated file case).

So we have to decide what we want to do as a workaround:
- We can't be bulletproof
- We can catch with our current code most cases where you open an index, start 
a query and then close the reader. This only works shortly after program 
started and everything is not yet optimized. As soon as there were multiple 
queries already running and you close the JVM later, it SIGSEGV almost always. 
Not even "opaque" semantics help here.
- With the current code we can go a bit further, but we have some slowdown, but 
it's still not save.

> Improve ByteBufferGuard in Java 11
> --
>
> Key: LUCENE-8780
> URL: https://issues.apache.org/jira/browse/LUCENE-8780
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: core/store
>Affects Versions: master (9.0)
>Reporter: Uwe Schindler
>Assignee: Uwe Schindler
>Priority: Major
>  Labels: Java11
> Attachments: LUCENE-8780.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> In LUCENE-7409 we added {{ByteBufferGuard}} to protect MMapDirectory from 
> crushing the JVM with SIGSEGV when you close and unmap the mmapped buffers of 
> an IndexInput, while another thread is accessing it.
> The idea was to do a volatile write access to flush the caches (to trigger a 
> full fence) and set a non-volatile boolean to true. All accesses would check 
> the boolean and stop the caller from accessing the underlying ByteBuffer. 
> This worked most of the time, until the JVM optimized away the plain read 
> access to the boolean (you can easily see this after some runtime of our 
> by-default ignored testcase).
> With master on Java 11, we can improve the whole thing. Using VarHandles you 
> can use any access type when reading or writing the boolean. After re

[JENKINS] Lucene-Solr-BadApples-Tests-8.x - Build # 89 - Failure

Build: https://builds.apache.org/job/Lucene-Solr-BadApples-Tests-8.x/89/

2 tests failed.
FAILED:  
org.apache.solr.handler.component.TestDistributedStatsComponentCardinality.test

Error Message:
.responseHeader.status:200!=0

Stack Trace:
junit.framework.AssertionFailedError: .responseHeader.status:200!=0
at 
__randomizedtesting.SeedInfo.seed([6A8B8618B29F141D:E2DFB9C21C6379E5]:0)
at junit.framework.Assert.fail(Assert.java:57)
at 
org.apache.solr.BaseDistributedSearchTestCase.compareSolrResponses(BaseDistributedSearchTestCase.java:999)
at 
org.apache.solr.BaseDistributedSearchTestCase.compareResponses(BaseDistributedSearchTestCase.java:1026)
at 
org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:680)
at 
org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:643)
at 
org.apache.solr.handler.component.TestDistributedStatsComponentCardinality.test(TestDistributedStatsComponentCardinality.java:143)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1082)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1054)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIg

[jira] [Commented] (SOLR-5970) Create collection API always has status 0



[ 
https://issues.apache.org/jira/browse/SOLR-5970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16829385#comment-16829385
 ] 

Tomás Fernández Löbbe commented on SOLR-5970:
-

+1 for this change. We should add an upgrade note, since this changes how 
people needs to handle failures (in the SolrJ case, they'll now get an 
exception where before they wouldn't have).

> Create collection API always has status 0
> -
>
> Key: SOLR-5970
> URL: https://issues.apache.org/jira/browse/SOLR-5970
> Project: Solr
>  Issue Type: Bug
>Reporter: Abraham Elmahrek
>Assignee: Ishan Chattopadhyaya
>Priority: Major
> Fix For: 8.1
>
> Attachments: SOLR-5970-test.patch, SOLR-5970.patch, SOLR-5970.patch, 
> SOLR-5970_branch_8x.patch, bad.jar, schema.xml, solrconfig.xml
>
>
> The responses below are from a successful create collection API 
> (https://cwiki.apache.org/confluence/display/solr/Collections+API#CollectionsAPI-CreateormodifyanAliasforaCollection)
>  call and an unsuccessful create collection API call. It seems the 'status' 
> is always 0.
> Success:
> {u'responseHeader': {u'status': 0, u'QTime': 4421}, u'success': {u'': 
> {u'core': u'test1_shard1_replica1', u'responseHeader': {u'status': 0, 
> u'QTime': 3449
> Failure:
> {u'failure': 
>   {u'': 
> u"org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:Error 
> CREATEing SolrCore 'test43_shard1_replica1': Unable to create core: 
> test43_shard1_replica1 Caused by: Could not find configName for collection 
> test43 found:[test1]"},
>  u'responseHeader': {u'status': 0, u'QTime': 17149}
> }
> It seems like the status should be 400 or something similar for an 
> unsuccessful attempt?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-13431) Efficient updates with shared storage

2019-04-29 Thread Yonik Seeley (JIRA)

[
https://issues.apache.org/jira/browse/SOLR-13431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Yonik Seeley updated SOLR-13431:

Description:
h2. Background & problem statement:

With shared storage support, data durability is handled by the storage layer
(e.g. S3 or HDFS) and replicas are not needed for durability. This changes the
nature of how a single update (say adding a document) must be handled. The
local transaction log does not help... a node can go down and never come back.
The implication is that *a commit must be done for any updates to be considered
durable.*

The problem is also more complex than just batching updates and adding a commit
at the end of a batch. Consider indexing documents A,B,C,D followed by a commit:
1) documents A,B sent to leader1 and indexed
2) leader1 fails, leader2 is elected
3) documents C,D sent to leader2 and indexed
4) commit
After this sequence of events, documents A,B are actually lost because a
commit was not done on leader1 before it failed.

Adding a commit for every single update would fix the problem of data loss, but
would obviously be too expensive (and each commit will be more expensive We can
still do batches if we *disable transparent failover* for a batch.
- all updates in a batch (for a specific shard) should be indexed on the *same
leader*... any change in leadership should result in a failure at the low level
instead of any transparent failover or forwarding.
- in the event of a failure, *all updates since the last commit must be
replayed* (we can't just retry the failure itself), or the failure will need to
be bubbled up to a higher layer to retry from the beginning.

h2. Indexing scenario 1: CSV upload

If SolrCloud is loading a large CSV file, The receiving Solr node will forward
updates to the correct leaders. This happens in the DistributedUpdateProcessor
via SolrCmdDistributor, which ends up using a ConcurrentUpdateHttp2SolrClient
subclass.

Fixing this scenario for shared storage in the simplest way would entail adding
a commit to every update, which would be way to slow.

The forward-to-replica use case here is quite different than the
forward-to-correct-leader (the latter has the current solr node acting much
more like an external client.). To simpliify development, we may want to
separate these cases and continue using the existing code for
forward-to-replica.

h2. Indexing scenario 2: SolrJ bulk indexing

In this scenario, a client is trying to do a large amount of indexing and can
use batches or streaming. For this scenario, we could just require that a
commit be added for each batch and then fail a batch on any leader change. This
is problematic for a couple of reasons:
- larger batches add latency to build, hurting throughput
- doesn't scale well - as a collection grows, the number of shards grow and
the chance that any shard leader goes down (or the shard is split) goes up.
Requiring that the entire batch (all shards) be replayed when this happens is
wasteful and gets worse with collection growth.

h2. Proposed Solution: a SolrJ cloud aware streaming client
- something like ConcurrentUpdateHttp2SolrClient that can stream and know
about cloud layout
- track when last commit happened for each shard leader
- buffer updates per-shard since the last commit happened
-- doesn't have to be exact... assume idempotent updates here, so overlap is
fine
-- buffering would also be triggered by the replica type of the collection (so
this class could be used for both shared storage and normal NRT replicas)
- a parameter would be passed that would disallow any forwarding (since we're
handling buffering/failover at this level)
- on a failure because of a leader going down or loss of leadership, wait
until a new leader has been elected and then replay updates since the last
commit
- insert commits where necessary to prevent buffers from growing too large
-- inserted commits should be able to proceed in parallel... we shouldn't need
to block and wait for a commit before resuming to send documents to that leader.
-- it would be nice if there was a way we could get notified if a commit
happened via some other mechanism (like an autoCommit being triggered)
--- assuming we can't get this, perhaps we should pass a flag that disables
triggering auto-commits for these batch updates?
- handle splits (not only can a shard leader change, but a shard could
split... buffered updates may need to be re-slotted)
- need to handle a leader "bounce" like a change in leadership (assuming we're
skipping using the transaction log)
- multi-threaded - all updates to a leader regardless of thread are managed as
a single update stream
-- this perhaps provides a way to coalesce incremental/realtime updates
- OPTIONAL: ability to have multiple channels to a single leader?
-- we would need to avoid reordering updates to the same ID
-- an alterna

[jira] [Commented] (SOLR-13320) add a param ignoreVersionConflicts=true to updates to not overwrite existing docs



[ 
https://issues.apache.org/jira/browse/SOLR-13320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16829375#comment-16829375
 ] 

Tomás Fernández Löbbe commented on SOLR-13320:
--

Yes, it's not a new parameter, but my point is that it achieves the use case 
Scott proposed. You just need to have a version field (which I don't remember 
if it can be \_version\_ or if it needs to be a new version field) that could 
actually be a constant number in your case, since you don't really care about 
versioning AFAICT. 
You could then, define an UpdateRequestProcessorChain where you use 
DocBasedVersionConstraintsProcessor and use that for your backfill, knowing 
that it'll skip any docs that already exist in the index. 
DocBasedVersionConstraintsProcessor also has the advantage that it works with 
deletes, by creating tombstones (though, in that case, you'd also have to use 
the DBVCP for regular updates).
I'm not sure it's a good idea to add a new global parameter to do something 
that can already be done. Also, looking at your patch, it seems to only work in 
Cloud mode?

> add a param ignoreVersionConflicts=true to updates to not overwrite existing 
> docs
> -
>
> Key: SOLR-13320
> URL: https://issues.apache.org/jira/browse/SOLR-13320
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Noble Paul
>Assignee: Noble Paul
>Priority: Major
> Attachments: SOLR-13320.patch
>
>
> Updates should have an option to ignore duplicate documents and drop them if 
> an option  {{ignoreDuplicates=true}} is specified



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] [lucene-solr] NazerkeBS opened a new pull request #660: SOLR-13047: Add facet2D Streaming Expression

NazerkeBS opened a new pull request #660: SOLR-13047: Add facet2D Streaming 
Expression
URL: https://github.com/apache/lucene-solr/pull/660
 
 
   @joel-bernstein, there are since only 2 levels, we don't call 
appendJson(...) function recursively. Also, in the fillTuples(..) I looped 
first the "x" bucket then "y" bucket and added to the tuple list which has a 
type of Tuple, but not sure.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Closed] (SOLR-12914) Solr crashes in /terms request handler

2019-04-29 Thread Vadim Miller (JIRA)



 [ 
https://issues.apache.org/jira/browse/SOLR-12914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vadim Miller closed SOLR-12914.
---

> Solr crashes in /terms request handler
> --
>
> Key: SOLR-12914
> URL: https://issues.apache.org/jira/browse/SOLR-12914
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 7.5
>Reporter: Vadim Miller
>Priority: Major
>  Labels: terms
> Attachments: terms.patch
>
>
> TermsComponent class always tries to fetch all terms from all shards for a 
> further processing. There is  {{java.lang.OutOfMemoryError}}  exception if 
> the resulting list is too long. Solr stops working on this shard after this 
> exception, only restart helps. 
> There is a very common use case when the full terms list is not required: a 
> client needs to see next N terms in alphabetically sorted list starting with 
> a given value. Usually, this is needed for some autocomplete field on a page.
> Example URL: 
>  
> {{[http://localhost:8983/solr/mycollection/terms?terms.fl=fulltext&terms.sort=index&terms.lower=cat&terms.limit=50]}}
>   
>  In this example TermsComponent needs to fetch only 50 terms from each shard 
> starting with a value provided in {{terms.lower}} URL parameter. So, it 
> should not reset TermsParams.TERMS_LIMIT parameter when generates a shard 
> query in createSmartShardQuery() method.
> The patch is attached.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (SOLR-12914) Solr crashes in /terms request handler

2019-04-29 Thread Vadim Miller (JIRA)



 [ 
https://issues.apache.org/jira/browse/SOLR-12914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vadim Miller resolved SOLR-12914.
-
Resolution: Fixed

OK. Thanks!

 

> Solr crashes in /terms request handler
> --
>
> Key: SOLR-12914
> URL: https://issues.apache.org/jira/browse/SOLR-12914
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 7.5
>Reporter: Vadim Miller
>Priority: Major
>  Labels: terms
> Attachments: terms.patch
>
>
> TermsComponent class always tries to fetch all terms from all shards for a 
> further processing. There is  {{java.lang.OutOfMemoryError}}  exception if 
> the resulting list is too long. Solr stops working on this shard after this 
> exception, only restart helps. 
> There is a very common use case when the full terms list is not required: a 
> client needs to see next N terms in alphabetically sorted list starting with 
> a given value. Usually, this is needed for some autocomplete field on a page.
> Example URL: 
>  
> {{[http://localhost:8983/solr/mycollection/terms?terms.fl=fulltext&terms.sort=index&terms.lower=cat&terms.limit=50]}}
>   
>  In this example TermsComponent needs to fetch only 50 terms from each shard 
> starting with a value provided in {{terms.lower}} URL parameter. So, it 
> should not reset TermsParams.TERMS_LIMIT parameter when generates a shard 
> query in createSmartShardQuery() method.
> The patch is attached.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] [lucene-solr] NazerkeBS closed pull request #659: SOLR-13047: Add facet2D Streaming Expression

NazerkeBS closed pull request #659: SOLR-13047: Add facet2D Streaming Expression
URL: https://github.com/apache/lucene-solr/pull/659
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] [lucene-solr] NazerkeBS opened a new pull request #659: SOLR-13047: Add facet2D Streaming Expression

NazerkeBS opened a new pull request #659: SOLR-13047: Add facet2D Streaming 
Expression
URL: https://github.com/apache/lucene-solr/pull/659
 
 
   @joel-bernstein, there are since only 2 levels, we don't call 
appendJson(...) function recursively. Also, in the fillTuples(..) I looped 
first the "x" bucket then "y" bucket and added to the tuple list which has a 
type of Tuple,  but not sure. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-8.x-MacOSX (64bit/jdk-9) - Build # 109 - Still Unstable!

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-8.x-MacOSX/109/
Java: 64bit/jdk-9 -XX:+UseCompressedOops -XX:+UseParallelGC

3 tests failed.
FAILED:  org.apache.solr.cloud.HttpPartitionTest.test

Error Message:
Error from server at http://127.0.0.1:54085: Underlying core creation failed 
while creating collection: c8n_1x2

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
from server at http://127.0.0.1:54085: Underlying core creation failed while 
creating collection: c8n_1x2
at 
__randomizedtesting.SeedInfo.seed([2329C5FCA6724B86:AB7DFA26088E267E]:0)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:649)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1274)
at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1792)
at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1813)
at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1730)
at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollectionRetry(AbstractFullDistribZkTestBase.java:2042)
at 
org.apache.solr.cloud.HttpPartitionTest.testRf2(HttpPartitionTest.java:214)
at 
org.apache.solr.cloud.HttpPartitionTest.test(HttpPartitionTest.java:135)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1082)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1054)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverri

[JENKINS] Lucene-Solr-8.x-Linux (64bit/jdk-11.0.2) - Build # 483 - Failure!

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-8.x-Linux/483/
Java: 64bit/jdk-11.0.2 -XX:-UseCompressedOops -XX:+UseG1GC

All tests passed

Build Log:
[...truncated 13944 lines...]
   [junit4] JVM J0: stdout was not empty, see: 
/home/jenkins/workspace/Lucene-Solr-8.x-Linux/solr/build/solr-core/test/temp/junit4-J0-20190429_143559_84815543786918163926765.sysout
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) 
   [junit4] java.lang.OutOfMemoryError: Java heap space
   [junit4] Dumping heap to 
/home/jenkins/workspace/Lucene-Solr-8.x-Linux/heapdumps/java_pid27117.hprof ...
   [junit4] Heap dump file created [609360626 bytes in 37.321 secs]
   [junit4] <<< JVM J0: EOF 

   [junit4] JVM J0: stderr was not empty, see: 
/home/jenkins/workspace/Lucene-Solr-8.x-Linux/solr/build/solr-core/test/temp/junit4-J0-20190429_143559_84817183038671873282715.syserr
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) 
   [junit4] 
   [junit4] Exception: java.lang.OutOfMemoryError thrown from the 
UncaughtExceptionHandler in thread "SessionTracker"
   [junit4] 
   [junit4] Exception: java.lang.OutOfMemoryError thrown from the 
UncaughtExceptionHandler in thread 
"TEST-CategoryRoutedAliasUpdateProcessorTest.testSliceRouting-seed#[FE3C8C4FB1F1AB24]-SendThread(localhost.localdomain:46233)"
   [junit4] 
   [junit4] Exception: java.lang.OutOfMemoryError thrown from the 
UncaughtExceptionHandler in thread "Connection evictor"
   [junit4] 
   [junit4] Exception: java.lang.OutOfMemoryError thrown from the 
UncaughtExceptionHandler in thread "NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0"
   [junit4] WARN: Unhandled exception in event serialization. -> 
java.lang.OutOfMemoryError: Java heap space
   [junit4] <<< JVM J0: EOF 

[...truncated 1794 lines...]
   [junit4] ERROR: JVM J0 ended with an exception, command line: 
/home/jenkins/tools/java/64bit/jdk-11.0.2/bin/java -XX:-UseCompressedOops 
-XX:+UseG1GC -XX:+HeapDumpOnOutOfMemoryError 
-XX:HeapDumpPath=/home/jenkins/workspace/Lucene-Solr-8.x-Linux/heapdumps -ea 
-esa --illegal-access=deny -Dtests.prefix=tests -Dtests.seed=FE3C8C4FB1F1AB24 
-Xmx512M -Dtests.iters= -Dtests.verbose=false -Dtests.infostream=false 
-Dtests.codec=random -Dtests.postingsformat=random 
-Dtests.docvaluesformat=random -Dtests.locale=random -Dtests.timezone=random 
-Dtests.directory=random -Dtests.linedocsfile=europarl.lines.txt.gz 
-Dtests.luceneMatchVersion=8.1.0 -Dtests.cleanthreads=perClass 
-Djava.util.logging.config.file=/home/jenkins/workspace/Lucene-Solr-8.x-Linux/lucene/tools/junit4/logging.properties
 -Dtests.nightly=false -Dtests.weekly=false -Dtests.monster=false 
-Dtests.slow=true -Dtests.asserts=true -Dtests.multiplier=3 -DtempDir=./temp 
-Djava.io.tmpdir=./temp 
-Dcommon.dir=/home/jenkins/workspace/Lucene-Solr-8.x-Linux/lucene 
-Dclover.db.dir=/home/jenkins/workspace/Lucene-Solr-8.x-Linux/lucene/build/clover/db
 
-Djava.security.policy=/home/jenkins/workspace/Lucene-Solr-8.x-Linux/lucene/tools/junit4/solr-tests.policy
 -Dtests.LUCENE_VERSION=8.1.0 -Djetty.testMode=1 -Djetty.insecurerandom=1 
-Dsolr.directoryFactory=org.apache.solr.core.MockDirectoryFactory 
-Djava.awt.headless=true -Djdk.map.althashing.threshold=0 
-Dtests.src.home=/home/jenkins/workspace/Lucene-Solr-8.x-Linux 
-Djava.security.egd=file:/dev/./urandom 
-Djunit4.childvm.cwd=/home/jenkins/workspace/Lucene-Solr-8.x-Linux/solr/build/solr-core/test/J0
 
-Djunit4.tempDir=/home/jenkins/workspace/Lucene-Solr-8.x-Linux/solr/build/solr-core/test/temp
 -Djunit4.childvm.id=0 -Djunit4.childvm.count=3 
-Djava.security.manager=org.apache.lucene.util.TestSecurityManager 
-Dtests.filterstacks=true -Dtests.leaveTemporary=false -Dtests.badapples=false 
-classpath 
/home/jenkins/workspace/Lucene-Solr-8.x-Linux/solr/build/solr-core/classes/test:/home/jenkins/workspace/Lucene-Solr-8.x-Linux/solr/build/solr-test-framework/classes/java:/home/jenkins/workspace/Lucene-Solr-8.x-Linux/solr/test-framework/lib/hamcrest-core-1.3.jar:/home/jenkins/workspace/Lucene-Solr-8.x-Linux/solr/test-framework/lib/junit4-ant-2.7.2.jar:/home/jenkins/workspace/Lucene-Solr-8.x-Linux/solr/core/src/test-files:/home/jenkins/workspace/Lucene-Solr-8.x-Linux/lucene/build/test-framework/classes/java:/home/jenkins/workspace/Lucene-Solr-8.x-Linux/lucene/build/codecs/classes/java:/home/jenkins/workspace/Lucene-Solr-8.x-Linux/solr/build/solr-solrj/classes/java:/home/jenkins/workspace/Lucene-Solr-8.x-Linux/solr/build/solr-core/classes/java:/home/jenkins/workspace/Lucene-Solr-8.x-Linux/lucene/build/analysis/common/lucene-analyzers-common-8.1.0-SNAPSHOT.jar:/home/jenkins/workspace/Lucene-Solr-8.x-Linux/lucene/build/analysis/kuromoji/lucene-analyzers-kuromoji-8.1.0-SNAPSHOT.jar:/home/jenkins/workspace/Lucene-Solr-8.x-Linux/lucene/build/analysis/nori/lucene-analyzers-nori-8.1.0-SNAPSHOT.jar:/home/jenkins/workspace/Lucene-Solr-8.x-Linux/lucene/build/analysis/phonetic/lucene-analyzers-phonetic-8.1.0-SNAPSHOT.jar:/home/jenkins/workspace/Lucene-Solr-8.x-Linux/lucene

[jira] [Updated] (SOLR-13431) Efficient updates with shared storage

2019-04-29 Thread Yonik Seeley (JIRA)

[
https://issues.apache.org/jira/browse/SOLR-13431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Yonik Seeley updated SOLR-13431:

Description:
h2. Background & problem statement:

h2. Indexing scenario 1: CSV upload

h2. Indexing scenario 2: SolrJ bulk indexing

was:
h2. Background & pro

7.7.2?

2019-04-29 Thread Bram Van Dam

Hey folks,

Was anyone planning on releasing 7.7.2? It seems like there are quite a
few bug fixes, including the one with the high CPU usage.

Full list of fixes so far: SOLR-13414, SOLR-13409, SOLR-13408,
SOLR-13392, SOLR-13355, SOLR-13349, SOLR-13344, SOLR-13285, SOLR-13281,
SOLR-13234, SOLR-13126, SOLR-12860, SOLR-12708, SOLR-12371, SOLR-11876

Feel free to let me know if there's anything I can do to help out with
the process, though I'm not a Solr committer.

Thanks,

 - Bram

[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1835 - Still Unstable

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1835/

2 tests failed.
FAILED:  
org.apache.solr.cloud.autoscaling.HdfsAutoAddReplicasIntegrationTest.testSimple

Error Message:
Waiting for collection testSimple2 Timeout waiting to see state for 
collection=testSimple2 
:DocCollection(testSimple2//collections/testSimple2/state.json/25)={   
"pullReplicas":"0",   "replicationFactor":"2",   "shards":{ "shard1":{  
 "range":"8000-",   "state":"active",   "replicas":{
 "core_node3":{   
"dataDir":"hdfs://localhost:45287/solr_hdfs_home/testSimple2/core_node3/data/", 
  "base_url":"http://127.0.0.1:38716/solr";,   
"node_name":"127.0.0.1:38716_solr",   "type":"NRT",   
"force_set_state":"false",   
"ulogDir":"hdfs://localhost:45287/solr_hdfs_home/testSimple2/core_node3/data/tlog",
   "core":"testSimple2_shard1_replica_n1",   
"shared_storage":"true",   "state":"down"}, "core_node5":{  
 
"dataDir":"hdfs://localhost:45287/solr_hdfs_home/testSimple2/core_node5/data/", 
  "base_url":"http://127.0.0.1:36739/solr";,   
"node_name":"127.0.0.1:36739_solr",   "type":"NRT",   
"force_set_state":"false",   
"ulogDir":"hdfs://localhost:45287/solr_hdfs_home/testSimple2/core_node5/data/tlog",
   "core":"testSimple2_shard1_replica_n2",   
"shared_storage":"true",   "state":"active",   
"leader":"true"}}}, "shard2":{   "range":"0-7fff",   
"state":"active",   "replicas":{ "core_node7":{   
"dataDir":"hdfs://localhost:45287/solr_hdfs_home/testSimple2/core_node7/data/", 
  "base_url":"http://127.0.0.1:38716/solr";,   
"node_name":"127.0.0.1:38716_solr",   "type":"NRT",   
"force_set_state":"false",   
"ulogDir":"hdfs://localhost:45287/solr_hdfs_home/testSimple2/core_node7/data/tlog",
   "core":"testSimple2_shard2_replica_n4",   
"shared_storage":"true",   "state":"down"}, "core_node8":{  
 
"dataDir":"hdfs://localhost:45287/solr_hdfs_home/testSimple2/core_node8/data/", 
  "base_url":"http://127.0.0.1:36739/solr";,   
"node_name":"127.0.0.1:36739_solr",   "type":"NRT",   
"force_set_state":"false",   
"ulogDir":"hdfs://localhost:45287/solr_hdfs_home/testSimple2/core_node8/data/tlog",
   "core":"testSimple2_shard2_replica_n6",   
"shared_storage":"true",   "state":"active",   
"leader":"true",   "router":{"name":"compositeId"},   
"maxShardsPerNode":"2",   "autoAddReplicas":"true",   "nrtReplicas":"2",   
"tlogReplicas":"0"} Live Nodes: [127.0.0.1:35804_solr, 127.0.0.1:36739_solr] 
Last available state: 
DocCollection(testSimple2//collections/testSimple2/state.json/25)={   
"pullReplicas":"0",   "replicationFactor":"2",   "shards":{ "shard1":{  
 "range":"8000-",   "state":"active",   "replicas":{
 "core_node3":{   
"dataDir":"hdfs://localhost:45287/solr_hdfs_home/testSimple2/core_node3/data/", 
  "base_url":"http://127.0.0.1:38716/solr";,   
"node_name":"127.0.0.1:38716_solr",   "type":"NRT",   
"force_set_state":"false",   
"ulogDir":"hdfs://localhost:45287/solr_hdfs_home/testSimple2/core_node3/data/tlog",
   "core":"testSimple2_shard1_replica_n1",   
"shared_storage":"true",   "state":"down"}, "core_node5":{  
 
"dataDir":"hdfs://localhost:45287/solr_hdfs_home/testSimple2/core_node5/data/", 
  "base_url":"http://127.0.0.1:36739/solr";,   
"node_name":"127.0.0.1:36739_solr",   "type":"NRT",   
"force_set_state":"false",   
"ulogDir":"hdfs://localhost:45287/solr_hdfs_home/testSimple2/core_node5/data/tlog",
   "core":"testSimple2_shard1_replica_n2",   
"shared_storage":"true",   "state":"active",   
"leader":"true"}}}, "shard2":{   "range":"0-7fff",   
"state":"active",   "replicas":{ "core_node7":{   
"dataDir":"hdfs://localhost:45287/solr_hdfs_home/testSimple2/core_node7/data/", 
  "base_url":"http://127.0.0.1:38716/solr";,   
"node_name":"127.0.0.1:38716_solr",   "type":"NRT",   
"force_set_state":"false",   
"ulogDir":"hdfs://localhost:45287/solr_hdfs_home/testSimple2/core_node7/data/tlog",
   "core":"testSimple2_shard2_replica_n4",   
"shared_storage":"true",   "state":"down"}, "core_node8":{  
 
"dataDir":"hdfs://localhost:45287/solr_hdfs_home/testSimple2/core_node8/data/", 
  "base_url":"http://127.0.0.1:36739/solr";,   
"node_name":"127.0.0.1:36739_solr",   "type":"NRT",   
"force_set_state":"false",   
"ulogDir":"hdfs://localhost:45287/solr_hdfs_home/testSimple2/core_node8/dat

[jira] [Updated] (SOLR-13320) add a param ignoreVersionConflicts=true to updates to not overwrite existing docs



 [ 
https://issues.apache.org/jira/browse/SOLR-13320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Noble Paul updated SOLR-13320:
--
Summary: add a param ignoreVersionConflicts=true to updates to not 
overwrite existing docs  (was: add a param ignoreDuplicates=true to updates to 
not overwrite existing docs)

> add a param ignoreVersionConflicts=true to updates to not overwrite existing 
> docs
> -
>
> Key: SOLR-13320
> URL: https://issues.apache.org/jira/browse/SOLR-13320
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Noble Paul
>Assignee: Noble Paul
>Priority: Major
> Attachments: SOLR-13320.patch
>
>
> Updates should have an option to ignore duplicate documents and drop them if 
> an option  {{ignoreDuplicates=true}} is specified



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] [lucene-solr] dsmiley commented on a change in pull request #647: SOLR-12638: Added docs in the ref-guide for nested atomic updates

dsmiley commented on a change in pull request #647: SOLR-12638: Added docs in 
the ref-guide for nested atomic updates
URL: https://github.com/apache/lucene-solr/pull/647#discussion_r279347995
 
 

 ##
 File path: solr/solr-ref-guide/src/indexing-nested-documents.adoc
 ##
 @@ -51,6 +51,13 @@ With the exception of in-place updates, the whole block 
must be updated or delet
   Consequently, the field need not be defined in the schema and probably 
shouldn't be as it would be confusing.
   There is no child document field type, at least not yet.
 
+[NOTE]
 
 Review comment:
   > Well, we probably could get away with using route to figure out root, but 
that would prevent any method of validating the user passed the right route 
param.
   
   Perhaps we can't validate it then; okay.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] [lucene-solr] moshebla commented on a change in pull request #647: SOLR-12638: Added docs in the ref-guide for nested atomic updates

moshebla commented on a change in pull request #647: SOLR-12638: Added docs in 
the ref-guide for nested atomic updates
URL: https://github.com/apache/lucene-solr/pull/647#discussion_r279347511
 
 

 ##
 File path: solr/solr-ref-guide/src/indexing-nested-documents.adoc
 ##
 @@ -51,6 +51,13 @@ With the exception of in-place updates, the whole block 
must be updated or delet
   Consequently, the field need not be defined in the schema and probably 
shouldn't be as it would be confusing.
   There is no child document field type, at least not yet.
 
+[NOTE]
 
 Review comment:
   Well, we probably could get away with using _route_ to figure out _root_, 
but that would prevent any method of validating the user passed the right 
_route_ param.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] [lucene-solr] moshebla commented on a change in pull request #647: SOLR-12638: Added docs in the ref-guide for nested atomic updates

moshebla commented on a change in pull request #647: SOLR-12638: Added docs in 
the ref-guide for nested atomic updates
URL: https://github.com/apache/lucene-solr/pull/647#discussion_r279345060
 
 

 ##
 File path: solr/solr-ref-guide/src/indexing-nested-documents.adoc
 ##
 @@ -51,6 +51,13 @@ With the exception of in-place updates, the whole block 
must be updated or delet
   Consequently, the field need not be defined in the schema and probably 
shouldn't be as it would be confusing.
   There is no child document field type, at least not yet.
 
+[NOTE]
 
 Review comment:
   Sorry,
   perhaps I should have been clearer.
   I meant for all nested atomic updates.
   Only if the document which is being updated is a nested document, the 
`_route_` param should be mandatory.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12638) Support atomic updates of nested/child documents for nested-enabled schema

2019-04-29 Thread David Smiley (JIRA)



[ 
https://issues.apache.org/jira/browse/SOLR-12638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16829222#comment-16829222
 ] 

David Smiley commented on SOLR-12638:
-

This is water under the bridge now, so to speak, but I thought I'd mention that 
this issue could have gone in smoother incrementally if the work was split up 
as follows:
 * add `[child]` support to RTG – SOLR-9006
 * atomic updates for nested docs, but only when the doc given in the update is 
a root doc.  Quite useful on its own.
 * atomic updates when the doc given is a child doc, and thus we need to pass a 
{{\_route_}} param so that it can go to the right shard.  This is more advanced.

> Support atomic updates of nested/child documents for nested-enabled schema
> --
>
> Key: SOLR-12638
> URL: https://issues.apache.org/jira/browse/SOLR-12638
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: mosh
>Assignee: David Smiley
>Priority: Major
> Fix For: 8.1
>
> Attachments: SOLR-12638-delete-old-block-no-commit.patch, 
> SOLR-12638-nocommit.patch, SOLR-12638.patch, SOLR-12638.patch
>
>  Time Spent: 15.5h
>  Remaining Estimate: 0h
>
> I have been toying with the thought of using this transformer in conjunction 
> with NestedUpdateProcessor and AtomicUpdate to allow SOLR to completely 
> re-index the entire nested structure. This is just a thought, I am still 
> thinking about implementation details. Hopefully I will be able to post a 
> more concrete proposal soon.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] [lucene-solr] dsmiley commented on a change in pull request #647: SOLR-12638: Added docs in the ref-guide for nested atomic updates

dsmiley commented on a change in pull request #647: SOLR-12638: Added docs in 
the ref-guide for nested atomic updates
URL: https://github.com/apache/lucene-solr/pull/647#discussion_r279347626
 
 

 ##
 File path: solr/solr-ref-guide/src/indexing-nested-documents.adoc
 ##
 @@ -51,6 +51,13 @@ With the exception of in-place updates, the whole block 
must be updated or delet
   Consequently, the field need not be defined in the schema and probably 
shouldn't be as it would be confusing.
   There is no child document field type, at least not yet.
 
+[NOTE]
 
 Review comment:
   > I meant for all nested atomic updates. Only if the document which is being 
updated is a nested document, the _route_ param should be mandatory.
   
   This is confusing to talk about.  I view "a nested document" as a nested 
tree of documents, both the root and child and all.  Any way, essentially, what 
I mean is that we probably shouldn't require this param if it's superfluous, 
and it superfluous if the root of an update is a root document.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] [lucene-solr] dsmiley commented on a change in pull request #647: SOLR-12638: Added docs in the ref-guide for nested atomic updates

dsmiley commented on a change in pull request #647: SOLR-12638: Added docs in 
the ref-guide for nested atomic updates
URL: https://github.com/apache/lucene-solr/pull/647#discussion_r279346419
 
 

 ##
 File path: solr/solr-ref-guide/src/indexing-nested-documents.adoc
 ##
 @@ -51,6 +51,13 @@ With the exception of in-place updates, the whole block 
must be updated or delet
   Consequently, the field need not be defined in the schema and probably 
shouldn't be as it would be confusing.
   There is no child document field type, at least not yet.
 
+[NOTE]
 
 Review comment:
   Is stored/docValues on  the`_root_` field still required when the atomic 
update provides a root doc as the root of the update?  Also, I think 
`useDocValuesAsStored` is best set to "false", not that it's a big deal.  We 
probably didn't test that.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] [lucene-solr] dsmiley commented on a change in pull request #647: SOLR-12638: Added docs in the ref-guide for nested atomic updates

dsmiley commented on a change in pull request #647: SOLR-12638: Added docs in 
the ref-guide for nested atomic updates
URL: https://github.com/apache/lucene-solr/pull/647#discussion_r279342878
 
 

 ##
 File path: solr/solr-ref-guide/src/indexing-nested-documents.adoc
 ##
 @@ -51,6 +51,13 @@ With the exception of in-place updates, the whole block 
must be updated or delet
   Consequently, the field need not be defined in the schema and probably 
shouldn't be as it would be confusing.
   There is no child document field type, at least not yet.
 
+[NOTE]
 
 Review comment:
   I think the `_route_` param should not be required for all atomic updates.  
It's kind of an edge case and I'd rather not burden the other case with 
something that's superfluous. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12167) Child documents are ignored if unknown atomic operation specified in parent doc

2019-04-29 Thread David Smiley (JIRA)



[ 
https://issues.apache.org/jira/browse/SOLR-12167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16829208#comment-16829208
 ] 

David Smiley commented on SOLR-12167:
-

(separate issue) It'd be nice if sub-JSON/maps were assumed to be child 
documents unless it is more obvious it's an atomic update.  The conundrum is 
that it's ambiguous for a one-element object.  Still... it's probably better to 
err on the side of a child document unless it looks exactly like an update (one 
element AND known operation).  We already have code that auto-assigns a child 
uniqueKey if it's absent but it's kinda dormant due to this issue.

CC [~moshebla]

> Child documents are ignored if unknown atomic operation specified in parent 
> doc
> ---
>
> Key: SOLR-12167
> URL: https://issues.apache.org/jira/browse/SOLR-12167
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: update
>Reporter: Munendra S N
>Assignee: Ishan Chattopadhyaya
>Priority: Major
> Fix For: 8.1, master (9.0)
>
> Attachments: SOLR-12167.patch
>
>
> On trying to add this nested document,
> {code:java}
> {uniqueId : book6, type_s:book, title_t : "The Way of Kings", author_s : 
> "Brandon Sanderson",
>   cat_s:fantasy, pubyear_i:2010, publisher_s:Tor, parent_unbxd:true,
>   _childDocuments_ : [
> { uniqueId: book6_c1, type_s:review, 
> review_dt:"2015-01-03T14:30:00Z",parentId : book6,
>   stars_i:5, author_s:rahul,
>   comment_t:"A great start to what looks like an epic series!"
> }
> ,
> { uniqueId: book6_c2, type_s:review, 
> review_dt:"2014-03-15T12:00:00Z",parentId : book6,
>   stars_i:3, author_s:arpan,
>   comment_t:"This book was too long."
> }
>   ],labelinfo:{label_image:"",hotdeal_type:"",apply_hotdeal:""}
>  }
> {code}
> Only parent document is getting indexed(without labelinfo field) and child 
> documents are being ingored.
> On checking the code,
> https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/update/processor/AtomicUpdateDocumentMerger.java#L94
>  
> I realized that since *labelinfo* is a Map, Solr is trying for atomic updates 
> and since label_image, hotdeal_type, apply_hotdeal are invalid operation 
> field is ignored. Unfortunately, child documents are also not getting indexed.
> h4. Problem with current behavior:
> * field is silently ignored when its value is a map instead of failing 
> document update(when present in parent)
> * In the above case, child document is also getting ignored
> * If any field value is Map in child document but not in parent then nested 
> document is indexed properly
> {code:java}
> {uniqueId : book6, type_s:book, title_t : "The Way of Kings", author_s : 
> "Brandon Sanderson",
>   cat_s:fantasy, pubyear_i:2010, publisher_s:Tor, parent_unbxd:true,
>   _childDocuments_ : [
> { uniqueId: book6_c1, type_s:review, 
> review_dt:"2015-01-03T14:30:00Z",parentId : book6,
>   stars_i:5, author_s:rahul,
>   comment_t:"A great start to what looks like an epic series!"
> ,labelinfo:{label_image:"","hotdeal_type":"","apply_hotdeal":""}
> }
> ,
> { uniqueId: book6_c2, type_s:review, 
> review_dt:"2014-03-15T12:00:00Z",parentId : book6,
>   stars_i:3, author_s:arpan,
>   comment_t:"This book was too long."
> }
>   ]
>  }
> {code}
> Here, nested document is indexed and labelinfo field value indexed in 
> book6_c1 as string(using Map.toString())
> h4. Probable solution
> * If an unknown operation is specified in update document then instead of 
> ignoring the field and field value, fail the document update(fail fast 
> approach). So, that user would know something is wrong with the document. 
> Also, this would solve the case where the parent doc is getting indexed and 
> child documents are getting ignored
> * Currently, when child document's field value is a Map even then it gets 
> indexed, instead update should fail



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-13318) JsonFacetingResponse classes should record provide access to count fields as longs

2019-04-29 Thread Jason Gerlowski (JIRA)



[ 
https://issues.apache.org/jira/browse/SOLR-13318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16829170#comment-16829170
 ] 

Jason Gerlowski commented on SOLR-13318:


Hi Munendra, thanks for the reminder.  I looked at the patches when you 
uploaded them, but haven't found time to test in more detail.  I promise to get 
this in for 8.1 though.

> JsonFacetingResponse classes should record  provide access to count fields as 
> longs
> ---
>
> Key: SOLR-13318
> URL: https://issues.apache.org/jira/browse/SOLR-13318
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrJ
>Affects Versions: 7.7.1
>Reporter: Jason Gerlowski
>Assignee: Jason Gerlowski
>Priority: Minor
> Attachments: SOLR-13318-branch_8x.patch, SOLR-13318.patch, 
> SOLR-13318.patch
>
>
> JsonFacetingResponse and its series of dependent classes hold a variety of 
> count fields for bucket counts and various optional properties 
> ({{allBuckets}}, {{numBuckets}}, etc.).  Currently, some of the code that 
> parses these values out of the originating NamedList either stores or casts 
> the values as ints.  When doc counts are low this works fine.  But when the 
> doc counts become larger and stray into "long" territory, SolrJ is liable to 
> blow up with ClassCastExceptions.
> A user on the list reported on of these with the partial stack trace:
> {code}
> Caused by: java.lang.ClassCastException: java.lang.Long cannot be cast to 
> java.lang.Integer
>   at 
> org.apache.solr.client.solrj.response.json.NestableJsonFacet.(NestableJsonFacet.java:52)
>   at 
> org.apache.solr.client.solrj.response.QueryResponse.extractJsonFacetingInfo(QueryResponse.java:200)
>   at 
> org.apache.solr.client.solrj.response.QueryResponse.getJsonFacetingResponse(QueryResponse.java:571)
> {code}
> We should fix this so that these classes can be used without incident for any 
> doc counts.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-13394) Change default GC from CMS to G1

2019-04-29 Thread Ishan Chattopadhyaya (JIRA)



[ 
https://issues.apache.org/jira/browse/SOLR-13394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16829169#comment-16829169
 ] 

Ishan Chattopadhyaya commented on SOLR-13394:
-

Thanks [~shalinmangar] for the feedback; I'll make the suggested changes.

bq. G1GC is recommended for large heaps. This patch enables G1GC regardless of 
the size of the heap.
Do you recommend that we document this somewhere clearly (maybe in Taking Solr 
to Production page)? Or are you suggesting we set the GC type dynamically based 
on -Xmx or -Xms settings?

> Change default GC from CMS to G1
> 
>
> Key: SOLR-13394
> URL: https://issues.apache.org/jira/browse/SOLR-13394
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Ishan Chattopadhyaya
>Assignee: Ishan Chattopadhyaya
>Priority: Major
> Fix For: 8.1
>
> Attachments: SOLR-13394.patch
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> CMS has been deprecated in new versions of Java 
> (http://openjdk.java.net/jeps/291). This issue is to switch Solr default from 
> CMS to G1.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-8.x-Windows (64bit/jdk-12) - Build # 231 - Unstable!

2019-04-29 Thread ASF subversion and git services (JIRA)

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-8.x-Windows/231/
Java: 64bit/jdk-12 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  org.apache.solr.core.TestDynamicLoading.testDynamicLoading

Error Message:
Could not get expected value  
'org.apache.solr.core.BlobStoreTestRequestHandler' for path 
'overlay/requestHandler/\/test1/class' full output: {   "responseHeader":{ 
"status":0, "QTime":0},   "overlay":{ "znodeVersion":0, 
"runtimeLib":{"colltest":{ "name":"colltest", "version":1,  
from server:  null

Stack Trace:
java.lang.AssertionError: Could not get expected value  
'org.apache.solr.core.BlobStoreTestRequestHandler' for path 
'overlay/requestHandler/\/test1/class' full output: {
  "responseHeader":{
"status":0,
"QTime":0},
  "overlay":{
"znodeVersion":0,
"runtimeLib":{"colltest":{
"name":"colltest",
"version":1,  from server:  null
at 
__randomizedtesting.SeedInfo.seed([14AE1563E6908BFC:CCE33834114D2E5C]:0)
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at 
org.apache.solr.core.TestSolrConfigHandler.testForResponseElement(TestSolrConfigHandler.java:590)
at 
org.apache.solr.core.TestDynamicLoading.testDynamicLoading(TestDynamicLoading.java:80)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:567)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1082)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1054)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at

[jira] [Commented] (SOLR-13407) Reject updates sent to non-routed multi collection aliases



[ 
https://issues.apache.org/jira/browse/SOLR-13407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16829147#comment-16829147
 ] 

ASF subversion and git services commented on SOLR-13407:


Commit 72230b69b3bd3ee0449e6c3ec149fa917add3ea9 in lucene-solr's branch 
refs/heads/branch_8x from Andrzej Bialecki
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=72230b6 ]

SOLR-13407: Fix NPE and be consistent about returning empty instead of null 
properties.


> Reject updates sent to non-routed multi collection aliases
> --
>
> Key: SOLR-13407
> URL: https://issues.apache.org/jira/browse/SOLR-13407
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Andrzej Bialecki 
>Assignee: Andrzej Bialecki 
>Priority: Major
> Fix For: 8.1, master (9.0)
>
> Attachments: SOLR-13407.patch
>
>
> Spin-off from SOLR-13262.
> Currently Solr uses a convention that updates sent to multi-collection 
> aliases are applied only to the first collection on the list, which is 
> nonintuitive and may hide bugs or accidental configuration changes made 
> either in Solr or in client applications.
> This issue proposes to reject all such updates with an error.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] [lucene-solr] moshebla commented on a change in pull request #647: SOLR-12638: Added docs in the ref-guide for nested atomic updates

2019-04-29 Thread ASF subversion and git services (JIRA)

moshebla commented on a change in pull request #647: SOLR-12638: Added docs in 
the ref-guide for nested atomic updates
URL: https://github.com/apache/lucene-solr/pull/647#discussion_r279317010
 
 

 ##
 File path: solr/solr-ref-guide/src/indexing-nested-documents.adoc
 ##
 @@ -51,6 +51,13 @@ With the exception of in-place updates, the whole block 
must be updated or delet
   Consequently, the field need not be defined in the schema and probably 
shouldn't be as it would be confusing.
   There is no child document field type, at least not yet.
 
+[NOTE]
 
 Review comment:
   Since by default if a doc was not found, [Solr  creates 
it](https://github.com/moshebla/lucene-solr/blob/SOLR-12638-docs/solr/core/src/java/org/apache/solr/update/processor/DistributedUpdateProcessor.java#L677),
 We can only tell if the user omitted the `_route_` param if the doc was routed 
to the correct shard.
   I think it would be a lot more simple to require `_route_` for **all** 
nested atomic updates.
   @dsmiley ,
   WDYT?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-13407) Reject updates sent to non-routed multi collection aliases



[ 
https://issues.apache.org/jira/browse/SOLR-13407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16829108#comment-16829108
 ] 

ASF subversion and git services commented on SOLR-13407:


Commit ced0243a3eea9360993035842e556c1daf9f4bd0 in lucene-solr's branch 
refs/heads/master from Andrzej Bialecki
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=ced0243 ]

SOLR-13407: Fix NPE and be consistent about returning empty instead of null 
properties.


> Reject updates sent to non-routed multi collection aliases
> --
>
> Key: SOLR-13407
> URL: https://issues.apache.org/jira/browse/SOLR-13407
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Andrzej Bialecki 
>Assignee: Andrzej Bialecki 
>Priority: Major
> Fix For: 8.1, master (9.0)
>
> Attachments: SOLR-13407.patch
>
>
> Spin-off from SOLR-13262.
> Currently Solr uses a convention that updates sent to multi-collection 
> aliases are applied only to the first collection on the list, which is 
> nonintuitive and may hide bugs or accidental configuration changes made 
> either in Solr or in client applications.
> This issue proposes to reject all such updates with an error.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-13434) OpenTracing support for Solr

2019-04-29 Thread Shalin Shekhar Mangar (JIRA)

Shalin Shekhar Mangar created SOLR-13434:


 Summary: OpenTracing support for Solr
 Key: SOLR-13434
 URL: https://issues.apache.org/jira/browse/SOLR-13434
 Project: Solr
  Issue Type: New Feature
  Security Level: Public (Default Security Level. Issues are Public)
Reporter: Shalin Shekhar Mangar
 Fix For: 8.1, master (9.0)


[OpenTracing|https://opentracing.io/] is a vendor neutral API and 
infrastructure for distributed tracing. Many OSS tracers just as Jaeger, 
OpenZipkin as well as commercial tools support OpenTracing APIs. Ideally, we 
can implement it once and have integrations for popular tracers like we have 
with metrics and prometheus.

I'm aware of SOLR-9641 but HTrace has since retired from incubator for lack of 
activity so this is a fresh attempt at solving this problem.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-Tests-8.x - Build # 174 - Still Unstable

Build: https://builds.apache.org/job/Lucene-Solr-Tests-8.x/174/

1 tests failed.
FAILED:  
org.apache.solr.client.solrj.impl.CloudSolrClientTest.preferLocalShardsTest

Error Message:
Could not load collection from ZK: localShardsTestColl

Stack Trace:
org.apache.solr.common.SolrException: Could not load collection from ZK: 
localShardsTestColl
at 
__randomizedtesting.SeedInfo.seed([4B194A4023E0CEB4:B7D4D2F9DD37594E]:0)
at 
org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1384)
at 
org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:748)
at 
org.apache.solr.common.cloud.ClusterState.getCollectionOrNull(ClusterState.java:148)
at 
org.apache.solr.common.cloud.ClusterState.getCollectionOrNull(ClusterState.java:131)
at 
org.apache.solr.common.cloud.ZkStateReader.registerCollectionStateWatcher(ZkStateReader.java:1502)
at 
org.apache.solr.common.cloud.ZkStateReader.waitForState(ZkStateReader.java:1539)
at 
org.apache.solr.client.solrj.impl.BaseCloudSolrClient.waitForState(BaseCloudSolrClient.java:380)
at 
org.apache.solr.cloud.MiniSolrCloudCluster.waitForActiveCollection(MiniSolrCloudCluster.java:750)
at 
org.apache.solr.cloud.MiniSolrCloudCluster.waitForActiveCollection(MiniSolrCloudCluster.java:764)
at 
org.apache.solr.client.solrj.impl.CloudSolrClientTest.preferLocalShardsTest(CloudSolrClientTest.java:409)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at 
org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.ut

[jira] [Updated] (SOLR-13434) OpenTracing support for Solr

2019-04-29 Thread Shalin Shekhar Mangar (JIRA)



 [ 
https://issues.apache.org/jira/browse/SOLR-13434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shalin Shekhar Mangar updated SOLR-13434:
-
Description: 
[OpenTracing|https://opentracing.io/] is a vendor neutral API and 
infrastructure for distributed tracing. Many OSS tracers just as Jaeger, 
OpenZipkin, Apache SkyWalking as well as commercial tools support OpenTracing 
APIs. Ideally, we can implement it once and have integrations for popular 
tracers like we have with metrics and prometheus.

I'm aware of SOLR-9641 but HTrace has since retired from incubator for lack of 
activity so this is a fresh attempt at solving this problem.

  was:
[OpenTracing|https://opentracing.io/] is a vendor neutral API and 
infrastructure for distributed tracing. Many OSS tracers just as Jaeger, 
OpenZipkin as well as commercial tools support OpenTracing APIs. Ideally, we 
can implement it once and have integrations for popular tracers like we have 
with metrics and prometheus.

I'm aware of SOLR-9641 but HTrace has since retired from incubator for lack of 
activity so this is a fresh attempt at solving this problem.


> OpenTracing support for Solr
> 
>
> Key: SOLR-13434
> URL: https://issues.apache.org/jira/browse/SOLR-13434
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Shalin Shekhar Mangar
>Priority: Major
> Fix For: 8.1, master (9.0)
>
>
> [OpenTracing|https://opentracing.io/] is a vendor neutral API and 
> infrastructure for distributed tracing. Many OSS tracers just as Jaeger, 
> OpenZipkin, Apache SkyWalking as well as commercial tools support OpenTracing 
> APIs. Ideally, we can implement it once and have integrations for popular 
> tracers like we have with metrics and prometheus.
> I'm aware of SOLR-9641 but HTrace has since retired from incubator for lack 
> of activity so this is a fresh attempt at solving this problem.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-master-Windows (64bit/jdk-11.0.2) - Build # 7917 - Still Unstable!

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Windows/7917/
Java: 64bit/jdk-11.0.2 -XX:+UseCompressedOops -XX:+UseSerialGC

1 tests failed.
FAILED:  
junit.framework.TestSuite.org.apache.solr.client.solrj.impl.CloudSolrClientTest

Error Message:
ObjectTracker found 5 object(s) that were not released!!! [SolrCore, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
InternalHttpClient] 
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: 
org.apache.solr.core.SolrCore  at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
  at org.apache.solr.core.SolrCore.(SolrCore.java:1063)  at 
org.apache.solr.core.SolrCore.(SolrCore.java:883)  at 
org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1227)
  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1137)  at 
org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)
  at 
org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)
  at 
org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:396)
  at 
org.apache.solr.handler.admin.CoreAdminHandler.lambda$handleRequestBody$0(CoreAdminHandler.java:188)
  at 
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181)
  at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
  at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  at 
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  at java.base/java.lang.Thread.run(Thread.java:834)  
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: 
org.apache.lucene.store.MockDirectoryWrapper  at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
  at 
org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
  at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:99)  at 
org.apache.solr.core.SolrCore.initIndex(SolrCore.java:779)  at 
org.apache.solr.core.SolrCore.(SolrCore.java:976)  at 
org.apache.solr.core.SolrCore.(SolrCore.java:883)  at 
org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1227)
  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1137)  at 
org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)
  at 
org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)
  at 
org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:396)
  at 
org.apache.solr.handler.admin.CoreAdminHandler.lambda$handleRequestBody$0(CoreAdminHandler.java:188)
  at 
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181)
  at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
  at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  at 
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  at java.base/java.lang.Thread.run(Thread.java:834)  
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: 
org.apache.lucene.store.MockDirectoryWrapper  at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
  at 
org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
  at 
org.apache.solr.core.SolrCore.initSnapshotMetaDataManager(SolrCore.java:517)  
at org.apache.solr.core.SolrCore.(SolrCore.java:968)  at 
org.apache.solr.core.SolrCore.(SolrCore.java:883)  at 
org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1227)
  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1137)  at 
org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)
  at 
org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)
  at 
org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:396)
  at 
org.apache.solr.handler.admin.CoreAdminHandler.lambda$handleRequestBody$0(CoreAdminHandler.java:188)
  at 
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181)
  at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
  at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  at 
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  at java.base/java.lang.Thread.run(Thread.java:834)  
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: 
org.apache.lucene.store.MockDirectoryWrapper  at 
org.apache.solr.common.util.Obj

[jira] [Commented] (LUCENE-8780) Improve ByteBufferGuard in Java 11

2019-04-29 Thread Dawid Weiss (JIRA)



[ 
https://issues.apache.org/jira/browse/LUCENE-8780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16829031#comment-16829031
 ] 

Dawid Weiss commented on LUCENE-8780:
-

Every time I read about those memory ordering models I seem to get it and two 
weeks later I'm again confused... So I don't know if what you propose is right 
or wrong. The volatile mode always seemed to me to be regulating relationships 
among other volatile reads/ writes (and not specifying what happens to other 
variable reads/ writes not covered by volatiles)... perhaps you can piggyback 
fences on top of that, but maybe you can't. We could fire an e-mail to 
concurrency dev list so that more knowledgeable folks take a peek and let us 
know? Just saying... 

Also: "until the JVM optimized away the plain read access to the boolean" -- 
when did that happen, do you have a reference openjdk Jira issue?

> Improve ByteBufferGuard in Java 11
> --
>
> Key: LUCENE-8780
> URL: https://issues.apache.org/jira/browse/LUCENE-8780
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: core/store
>Affects Versions: master (9.0)
>Reporter: Uwe Schindler
>Assignee: Uwe Schindler
>Priority: Major
>  Labels: Java11
> Attachments: LUCENE-8780.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> In LUCENE-7409 we added {{ByteBufferGuard}} to protect MMapDirectory from 
> crushing the JVM with SIGSEGV when you close and unmap the mmapped buffers of 
> an IndexInput, while another thread is accessing it.
> The idea was to do a volatile write access to flush the caches (to trigger a 
> full fence) and set a non-volatile boolean to true. All accesses would check 
> the boolean and stop the caller from accessing the underlying ByteBuffer. 
> This worked most of the time, until the JVM optimized away the plain read 
> access to the boolean (you can easily see this after some runtime of our 
> by-default ignored testcase).
> With master on Java 11, we can improve the whole thing. Using VarHandles you 
> can use any access type when reading or writing the boolean. After reading 
> Doug Lea's expanation  and some 
> testing, I was no longer able to crush my JDK (even after running for minutes 
> unmapping bytebuffers).
> The apraoch is the same, we do a full-fenced write (standard volatile write) 
> when we unmap, then we yield the thread (to finish in-flight reads in other 
> threads) and then unmap all byte buffers.
> On the test side (read access), instead of using a plain read, we use the new 
> "opaque read". Opaque reads are the same as plain reads, there are only 
> different order requirements. Actually the main difference is explained by 
> Doug like this: "For example in constructions in which the only modification 
> of some variable x is for one thread to write in Opaque (or stronger) mode, 
> X.setOpaque(this, 1), any other thread spinning in 
> while(X.getOpaque(this)!=1){} will eventually terminate. Note that this 
> guarantee does NOT hold in Plain mode, in which spin loops may (and usually 
> do) infinitely loop -- they are not required to notice that a write ever 
> occurred in another thread if it was not seen on first encounter." - And 
> that's waht we want to have: We don't want to do volatile reads, but we want 
> to prevent the compiler from optimizing away our read to the boolean. So we 
> want it to "eventually" see the change. By the much stronger volatile write, 
> the cache effects should be visible even faster (like in our Java 8 approach, 
> just now we improved our read side).
> The new code is much slimmer (theoretically we could also use a AtomicBoolean 
> for that and use the new method {{getOpaque()}}, but I wanted to prevent 
> extra method calls, so I used a VarHandle directly).
> It's setup like this:
> - The underlying boolean field is a private member (with unused 
> SuppressWarnings, as its unused by the java compiler), marked as volatile 
> (that's the recommendation, but in reality it does not matter at all).
> - We create a VarHandle to access this boolean, we never do this directly 
> (this is why the volatile marking does not affect us).
> - We use VarHandle.setVolatile() to change our "invalidated" boolean to 
> "true", so enforcing a full fence
> - On the read side we use VarHandle.getOpaque() instead of VarHandle.get() 
> (like in our old code for Java 8).
> I had to tune our test a bit, as the VarHandles make it take longer until it 
> actually crushes (as optimizations jump in later). I also used a random for 
> the reads to prevent the optimizer from removing all the bytebuffer reads. 
> When we commit this, we can disable the test again (it takes approx 50 secs 
> on my machine).
> I'd still like to see the diffe

[GitHub] [lucene-solr] moshebla commented on a change in pull request #647: SOLR-12638: Added docs in the ref-guide for nested atomic updates

moshebla commented on a change in pull request #647: SOLR-12638: Added docs in 
the ref-guide for nested atomic updates
URL: https://github.com/apache/lucene-solr/pull/647#discussion_r279276287
 
 

 ##
 File path: solr/solr-ref-guide/src/indexing-nested-documents.adoc
 ##
 @@ -51,6 +51,13 @@ With the exception of in-place updates, the whole block 
must be updated or delet
   Consequently, the field need not be defined in the schema and probably 
shouldn't be as it would be confusing.
   There is no child document field type, at least not yet.
 
+[NOTE]
 
 Review comment:
   > And if `_route_` is provided unnecessarily, ideally we would verify that 
its value is correct.
   > 
   > What I wonder about is what happens when `_root_` is not stored/docValues?
   
   Well, as long as `_root_` is either stored or docValues, we should be able 
to verify whether the correct `_route_` was provided.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8766) Add Luwak as a lucene module

2019-04-29 Thread Daniel Collins (JIRA)



[ 
https://issues.apache.org/jira/browse/LUCENE-8766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16829019#comment-16829019
 ] 

Daniel Collins commented on LUCENE-8766:


[~romseygeek], [https://github.com/romseygeek/lucene-solr/pull/5] adds Maven 
build support and fixes some unused imports to this PR.

> Add Luwak as a lucene module
> 
>
> Key: LUCENE-8766
> URL: https://issues.apache.org/jira/browse/LUCENE-8766
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Alan Woodward
>Assignee: Alan Woodward
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Luwak [https://github.com/flaxsearch/luwak] is a stored query engine, 
> allowing users to efficiently match a stream of documents against a large set 
> of queries.  It's only dependency is lucene, and most recent updates have 
> just been upgrading the version of lucene against which it can run.
> It's a generally useful piece of software, and already licensed as Apache 2.  
> The maintainers would like to donate it to the ASF, and merge it in to the 
> lucene-solr project.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] [lucene-solr] dweiss commented on a change in pull request #657: LUCENE-8781: FST direct arc addressing