date:20180625

[JENKINS] Lucene-Solr-repro - Build # 887 - Still Unstable

2018-06-25 Thread Apache Jenkins Server

Build: https://builds.apache.org/job/Lucene-Solr-repro/887/

[...truncated 29 lines...]
[repro] Jenkins log URL: 
https://builds.apache.org/job/Lucene-Solr-BadApples-Tests-7.x/88/consoleText

[repro] Revision: 0db26c9f6c4edf36d2e7ec679cd9a1e6bdab6570

[repro] Repro line:  ant test  -Dtestcase=IndexSizeTriggerTest 
-Dtests.method=testSplitIntegration -Dtests.seed=8FBDC81BB7DE2F32 
-Dtests.multiplier=2 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=ru 
-Dtests.timezone=VST -Dtests.asserts=true -Dtests.file.encoding=US-ASCII

[repro] Repro line:  ant test  -Dtestcase=CdcrBidirectionalTest 
-Dtests.method=testBiDir -Dtests.seed=8FBDC81BB7DE2F32 -Dtests.multiplier=2 
-Dtests.slow=true -Dtests.badapples=true -Dtests.locale=ru -Dtests.timezone=NST 
-Dtests.asserts=true -Dtests.file.encoding=US-ASCII

[repro] git rev-parse --abbrev-ref HEAD
[repro] git rev-parse HEAD
[repro] Initial local git branch/revision: 
095f9eb90db92649a0805e83ff5a0ec93763a31f
[repro] git fetch
[repro] git checkout 0db26c9f6c4edf36d2e7ec679cd9a1e6bdab6570

[...truncated 2 lines...]
[repro] git merge --ff-only

[...truncated 1 lines...]
[repro] ant clean

[...truncated 6 lines...]
[repro] Test suites by module:
[repro]solr/core
[repro]   IndexSizeTriggerTest
[repro]   CdcrBidirectionalTest
[repro] ant compile-test

[...truncated 3318 lines...]
[repro] ant test-nocompile -Dtests.dups=5 -Dtests.maxfailures=10 
-Dtests.class="*.IndexSizeTriggerTest|*.CdcrBidirectionalTest" 
-Dtests.showOutput=onerror  -Dtests.seed=8FBDC81BB7DE2F32 -Dtests.multiplier=2 
-Dtests.slow=true -Dtests.badapples=true -Dtests.locale=ru -Dtests.timezone=VST 
-Dtests.asserts=true -Dtests.file.encoding=US-ASCII

[...truncated 25724 lines...]
[repro] Setting last failure code to 256

[repro] Failures:
[repro]   1/5 failed: org.apache.solr.cloud.cdcr.CdcrBidirectionalTest
[repro]   5/5 failed: org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest

[repro] Re-testing 100% failures at the tip of branch_7x
[repro] git fetch
[repro] git checkout branch_7x

[...truncated 4 lines...]
[repro] git merge --ff-only

[...truncated 2375 lines...]
[repro] ant clean

[...truncated 8 lines...]
[repro] Test suites by module:
[repro]solr/core
[repro]   IndexSizeTriggerTest
[repro] ant compile-test

[...truncated 3318 lines...]
[repro] ant test-nocompile -Dtests.dups=5 -Dtests.maxfailures=5 
-Dtests.class="*.IndexSizeTriggerTest" -Dtests.showOutput=onerror  
-Dtests.seed=8FBDC81BB7DE2F32 -Dtests.multiplier=2 -Dtests.slow=true 
-Dtests.badapples=true -Dtests.locale=ru -Dtests.timezone=VST 
-Dtests.asserts=true -Dtests.file.encoding=US-ASCII

[...truncated 12626 lines...]
[repro] Setting last failure code to 256

[repro] Failures at the tip of branch_7x:
[repro]   5/5 failed: org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest

[repro] Re-testing 100% failures at the tip of branch_7x without a seed
[repro] ant clean

[...truncated 8 lines...]
[repro] Test suites by module:
[repro]solr/core
[repro]   IndexSizeTriggerTest
[repro] ant compile-test

[...truncated 3318 lines...]
[repro] ant test-nocompile -Dtests.dups=5 -Dtests.maxfailures=5 
-Dtests.class="*.IndexSizeTriggerTest" -Dtests.showOutput=onerror  
-Dtests.multiplier=2 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=ru 
-Dtests.timezone=VST -Dtests.asserts=true -Dtests.file.encoding=US-ASCII

[...truncated 10164 lines...]
[repro] Setting last failure code to 256

[repro] Failures at the tip of branch_7x without a seed:
[repro]   4/5 failed: org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest
[repro] git checkout 095f9eb90db92649a0805e83ff5a0ec93763a31f

[...truncated 8 lines...]
[repro] Exiting with code 256

[...truncated 5 lines...]

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-12517) Support range values for replica in autoscaling policies

2018-06-25 Thread Noble Paul (JIRA)

Noble Paul created SOLR-12517:
-

 Summary: Support range values for replica in autoscaling policies
 Key: SOLR-12517
 URL: https://issues.apache.org/jira/browse/SOLR-12517
 Project: Solr
  Issue Type: Improvement
  Security Level: Public (Default Security Level. Issues are Public)
Reporter: Noble Paul


example

{code}
{"replica" : "3 - 5", "shard" :"#EACH", "node" : "#ANY"}
{code}

means anode may have 3, 4 or 5 replicas of a shard. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-repro - Build # 886 - Unstable

2018-06-25 Thread Apache Jenkins Server

Build: https://builds.apache.org/job/Lucene-Solr-repro/886/

[...truncated 29 lines...]
[repro] Jenkins log URL: 
https://builds.apache.org/job/Lucene-Solr-SmokeRelease-7.x/247/consoleText

[repro] Revision: 82b793df56c8c9fb50c29f46f39465453a87f2b2

[repro] Ant options: -DsmokeTestRelease.java9=/home/jenkins/tools/java/latest1.9
[repro] Repro line:  ant test  -Dtestcase=TestPKIAuthenticationPlugin 
-Dtests.method=test -Dtests.seed=FD9A1A05876BCA3F -Dtests.multiplier=2 
-Dtests.locale=ar-YE -Dtests.timezone=Asia/Jerusalem -Dtests.asserts=true 
-Dtests.file.encoding=UTF-8

[repro] Repro line:  ant test  -Dtestcase=MoveReplicaTest -Dtests.method=test 
-Dtests.seed=FD9A1A05876BCA3F -Dtests.multiplier=2 -Dtests.locale=sr-Latn-RS 
-Dtests.timezone=Europe/Berlin -Dtests.asserts=true -Dtests.file.encoding=UTF-8

[repro] Repro line:  ant test  -Dtestcase=ScheduledTriggerIntegrationTest 
-Dtests.method=testScheduledTrigger -Dtests.seed=FD9A1A05876BCA3F 
-Dtests.multiplier=2 -Dtests.locale=ru -Dtests.timezone=America/Porto_Velho 
-Dtests.asserts=true -Dtests.file.encoding=UTF-8

[repro] git rev-parse --abbrev-ref HEAD
[repro] git rev-parse HEAD
[repro] Initial local git branch/revision: 
095f9eb90db92649a0805e83ff5a0ec93763a31f
[repro] git fetch
[repro] git checkout 82b793df56c8c9fb50c29f46f39465453a87f2b2

[...truncated 2 lines...]
[repro] git merge --ff-only

[...truncated 1 lines...]
[repro] ant clean

[...truncated 6 lines...]
[repro] Test suites by module:
[repro]solr/core
[repro]   ScheduledTriggerIntegrationTest
[repro]   MoveReplicaTest
[repro]   TestPKIAuthenticationPlugin
[repro] ant compile-test

[...truncated 3318 lines...]
[repro] ant test-nocompile -Dtests.dups=5 -Dtests.maxfailures=15 
-Dtests.class="*.ScheduledTriggerIntegrationTest|*.MoveReplicaTest|*.TestPKIAuthenticationPlugin"
 -Dtests.showOutput=onerror 
-DsmokeTestRelease.java9=/home/jenkins/tools/java/latest1.9 
-Dtests.seed=FD9A1A05876BCA3F -Dtests.multiplier=2 -Dtests.locale=ru 
-Dtests.timezone=America/Porto_Velho -Dtests.asserts=true 
-Dtests.file.encoding=UTF-8

[...truncated 1465 lines...]
[repro] Setting last failure code to 256

[repro] Failures:
[repro]   0/5 failed: org.apache.solr.cloud.MoveReplicaTest
[repro]   0/5 failed: org.apache.solr.security.TestPKIAuthenticationPlugin
[repro]   2/5 failed: 
org.apache.solr.cloud.autoscaling.ScheduledTriggerIntegrationTest
[repro] git checkout 095f9eb90db92649a0805e83ff5a0ec93763a31f

[...truncated 2 lines...]
[repro] Exiting with code 256

[...truncated 6 lines...]

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8367) Make per-dimension drill down optional for each facet dimension

2018-06-25 Thread Lucene/Solr QA (JIRA)



[ 
https://issues.apache.org/jira/browse/LUCENE-8367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16523219#comment-16523219
 ] 

Lucene/Solr QA commented on LUCENE-8367:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
47s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Release audit (RAT) {color} | 
{color:green}  0m 44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Check forbidden APIs {color} | 
{color:green}  0m 40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Validate source patterns {color} | 
{color:green}  0m 40s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 26m 
51s{color} | {color:green} core in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m 
34s{color} | {color:green} facet in the patch passed. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 38m 29s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | LUCENE-8367 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12929011/LUCENE-8367.patch |
| Optional Tests |  compile  javac  unit  ratsources  checkforbiddenapis  
validatesourcepatterns  |
| uname | Linux lucene2-us-west.apache.org 4.4.0-112-generic #135-Ubuntu SMP 
Fri Jan 19 11:48:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | ant |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-LUCENE-Build/sourcedir/dev-tools/test-patch/lucene-solr-yetus-personality.sh
 |
| git revision | master / 095f9eb |
| ant | version: Apache Ant(TM) version 1.9.6 compiled on July 8 2015 |
| Default Java | 1.8.0_172 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-LUCENE-Build/39/testReport/ |
| modules | C: lucene/core lucene/facet U: lucene |
| Console output | 
https://builds.apache.org/job/PreCommit-LUCENE-Build/39/console |
| Powered by | Apache Yetus 0.7.0   http://yetus.apache.org |


This message was automatically generated.



> Make per-dimension drill down optional for each facet dimension
> ---
>
> Key: LUCENE-8367
> URL: https://issues.apache.org/jira/browse/LUCENE-8367
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: modules/facet
>Reporter: Michael McCandless
>Assignee: Michael McCandless
>Priority: Major
> Attachments: LUCENE-8367.patch, LUCENE-8367.patch
>
>
> Today, when you index a {{FacetField}} with path {{foo/bar,}} we index two 
> drill down terms onto the document: {{foo}} and {{foo/bar}}.
> But I suspect some users (like me!) don't need to drilldown just on {{foo}} 
> (effectively "find all documents that have any value for this facet 
> dimension"), so I added an option to {{FacetsConfig}} to let you specify 
> per-dimension whether you need to drill down (defaults to true, matching 
> current behavior).
> I also added {{hashCode}} and {{equals}} to the {{LongRange}} and 
> {{DoubleRange}} classes in facets module, and improved {{CheckIndex}} a bit 
> to print the total %deletions across the index.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-BadApples-master-Linux (32bit/jdk1.8.0_172) - Build # 56 - Unstable!

2018-06-25 Thread Policeman Jenkins Server

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-BadApples-master-Linux/56/
Java: 32bit/jdk1.8.0_172 -client -XX:+UseParallelGC

14 tests failed.
FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates

Error Message:


Stack Trace:
java.lang.NullPointerException
at __randomizedtesting.SeedInfo.seed([3006617022AC9671]:0)
at 
org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.createMiniSolrCloudCluster(TestStressCloudBlindAtomicUpdates.java:138)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:874)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)


FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates

Error Message:


Stack Trace:
java.lang.NullPointerException
at __randomizedtesting.SeedInfo.seed([3006617022AC9671]:0)
at 
org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.afterClass(TestStressCloudBlindAtomicUpdates.java:158)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:897)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at

[jira] [Assigned] (SOLR-10243) Fix TestExtractionDateUtil.testParseDate sporadic failures

2018-06-25 Thread David Smiley (JIRA)



 [ 
https://issues.apache.org/jira/browse/SOLR-10243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Smiley reassigned SOLR-10243:
---

Assignee: David Smiley

> Fix TestExtractionDateUtil.testParseDate sporadic failures
> --
>
> Key: SOLR-10243
> URL: https://issues.apache.org/jira/browse/SOLR-10243
> Project: Solr
>  Issue Type: Task
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: David Smiley
>Assignee: David Smiley
>Priority: Major
>
> Jenkins test failure:
> {{ant test  -Dtestcase=TestExtractionDateUtil -Dtests.method=testParseDate 
> -Dtests.seed=B72AC4792F31F74B -Dtests.slow=true -Dtests.locale=lv 
> -Dtests.timezone=America/Metlakatla -Dtests.asserts=true 
> -Dtests.file.encoding=UTF-8}}   It reproduces on 6x for me but not master.
> I reviewed this briefly and there seems to be a locale assumption in the test.
> 1 tests failed.
> FAILED:  
> org.apache.solr.handler.extraction.TestExtractionDateUtil.testParseDate
> Error Message:
> Incorrect parsed timestamp: 1226583351000 != 1226579751000 (Thu Nov 13 
> 04:35:51 AKST 2008)
> Stack Trace:
> java.lang.AssertionError: Incorrect parsed timestamp: 1226583351000 != 
> 1226579751000 (Thu Nov 13 04:35:51 AKST 2008)
> at 
> __randomizedtesting.SeedInfo.seed([B72AC4792F31F74B:FD33BC4C549880FE]:0)
> at org.junit.Assert.fail(Assert.java:93)
> at org.junit.Assert.assertTrue(Assert.java:43)
> at 
> org.apache.solr.handler.extraction.TestExtractionDateUtil.assertParsedDate(TestExtractionDateUtil.java:59)
> at 
> org.apache.solr.handler.extraction.TestExtractionDateUtil.testParseDate(TestExtractionDateUtil.java:54)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-8369) Remove the spatial module as it is obsolete

2018-06-25 Thread David Smiley (JIRA)



 [ 
https://issues.apache.org/jira/browse/LUCENE-8369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Smiley updated LUCENE-8369:
-
Attachment: LUCENE-8369.patch

> Remove the spatial module as it is obsolete
> ---
>
> Key: LUCENE-8369
> URL: https://issues.apache.org/jira/browse/LUCENE-8369
> Project: Lucene - Core
>  Issue Type: Task
>  Components: modules/spatial
>Reporter: David Smiley
>Assignee: David Smiley
>Priority: Major
> Attachments: LUCENE-8369.patch
>
>
> The "spatial" module is at this juncture nearly empty with only a couple 
> utilities that aren't used by anything in the entire codebase -- 
> GeoRelationUtils, and MortonEncoder.  Perhaps it should have been removed 
> earlier in LUCENE-7664 which was the removal of GeoPointField which was 
> essentially why the module existed.  Better late than never.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-7.x-Linux (32bit/jdk1.8.0_172) - Build # 2196 - Unstable!

2018-06-25 Thread Policeman Jenkins Server

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/2196/
Java: 32bit/jdk1.8.0_172 -server -XX:+UseG1GC

4 tests failed.
FAILED:  
org.apache.lucene.analysis.core.TestRandomChains.testRandomChainsWithLargeStrings

Error Message:
finalOffset expected:<74> but was:<73>

Stack Trace:
java.lang.AssertionError: finalOffset expected:<74> but was:<73>
at 
__randomizedtesting.SeedInfo.seed([8C3CDE29C6D4A774:E66761389F9A8787]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at 
org.apache.lucene.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:305)
at 
org.apache.lucene.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:320)
at 
org.apache.lucene.analysis.BaseTokenStreamTestCase.assertTokenStreamContents(BaseTokenStreamTestCase.java:324)
at 
org.apache.lucene.analysis.BaseTokenStreamTestCase.checkAnalysisConsistency(BaseTokenStreamTestCase.java:860)
at 
org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:659)
at 
org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:561)
at 
org.apache.lucene.analysis.core.TestRandomChains.testRandomChainsWithLargeStrings(TestRandomChains.java:893)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at

[jira] [Resolved] (SOLR-12398) Make JSON Facet API support Heatmap Facet

2018-06-25 Thread David Smiley (JIRA)



 [ 
https://issues.apache.org/jira/browse/SOLR-12398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Smiley resolved SOLR-12398.
-
   Resolution: Fixed
Fix Version/s: 7.5

> Make JSON Facet API support Heatmap Facet
> -
>
> Key: SOLR-12398
> URL: https://issues.apache.org/jira/browse/SOLR-12398
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Facet Module, faceting, JSON Request API, spatial
>Reporter: Jaime Yap
>Assignee: David Smiley
>Priority: Major
>  Labels: heatmap
> Fix For: 7.5
>
> Attachments: SOLR-12398.patch, SOLR-12398.patch
>
>
> The JSON query Facet API does not support Heatmap facets. For companies that 
> have standardized around generating queries for the JSON query API, it is a 
> major wart to need to also support falling back to the param encoding API in 
> order to make use of them.
> More importantly however, given it's more natural support for nested 
> subfacets, the JSON Query facet API is be able to compute more interesting 
> Heatmap layers for each facet bucket. Without resorting to the older (and 
> much more awkward) facet pivot syntax.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12398) Make JSON Facet API support Heatmap Facet

2018-06-25 Thread ASF subversion and git services (JIRA)



[ 
https://issues.apache.org/jira/browse/SOLR-12398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16523096#comment-16523096
 ] 

ASF subversion and git services commented on SOLR-12398:


Commit d59c90e5135dc48d1e5bb58ab1a643f34739a70c in lucene-solr's branch 
refs/heads/branch_7x from [~dsmiley]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=d59c90e ]

SOLR-12398: Add Heatmap facet option to JSON Facet API.
* moved the preponderance of the implementation from SpatialHeatmapFacets (used 
by SimpleFacets) into the new API.

(cherry picked from commit 095f9eb)


> Make JSON Facet API support Heatmap Facet
> -
>
> Key: SOLR-12398
> URL: https://issues.apache.org/jira/browse/SOLR-12398
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Facet Module, faceting, JSON Request API, spatial
>Reporter: Jaime Yap
>Assignee: David Smiley
>Priority: Major
>  Labels: heatmap
> Attachments: SOLR-12398.patch, SOLR-12398.patch
>
>
> The JSON query Facet API does not support Heatmap facets. For companies that 
> have standardized around generating queries for the JSON query API, it is a 
> major wart to need to also support falling back to the param encoding API in 
> order to make use of them.
> More importantly however, given it's more natural support for nested 
> subfacets, the JSON Query facet API is be able to compute more interesting 
> Heatmap layers for each facet bucket. Without resorting to the older (and 
> much more awkward) facet pivot syntax.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12398) Make JSON Facet API support Heatmap Facet

2018-06-25 Thread ASF subversion and git services (JIRA)



[ 
https://issues.apache.org/jira/browse/SOLR-12398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16523094#comment-16523094
 ] 

ASF subversion and git services commented on SOLR-12398:


Commit 095f9eb90db92649a0805e83ff5a0ec93763a31f in lucene-solr's branch 
refs/heads/master from [~dsmiley]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=095f9eb ]

SOLR-12398: Add Heatmap facet option to JSON Facet API.
* moved the preponderance of the implementation from SpatialHeatmapFacets (used 
by SimpleFacets) into the new API.


> Make JSON Facet API support Heatmap Facet
> -
>
> Key: SOLR-12398
> URL: https://issues.apache.org/jira/browse/SOLR-12398
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Facet Module, faceting, JSON Request API, spatial
>Reporter: Jaime Yap
>Assignee: David Smiley
>Priority: Major
>  Labels: heatmap
> Attachments: SOLR-12398.patch, SOLR-12398.patch
>
>
> The JSON query Facet API does not support Heatmap facets. For companies that 
> have standardized around generating queries for the JSON query API, it is a 
> major wart to need to also support falling back to the param encoding API in 
> order to make use of them.
> More importantly however, given it's more natural support for nested 
> subfacets, the JSON Query facet API is be able to compute more interesting 
> Heatmap layers for each facet bucket. Without resorting to the older (and 
> much more awkward) facet pivot syntax.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-Tests-7.x - Build # 652 - Still Unstable

2018-06-25 Thread Apache Jenkins Server

Build: https://builds.apache.org/job/Lucene-Solr-Tests-7.x/652/

1 tests failed.
FAILED:  org.apache.solr.cloud.TestUtilizeNode.test

Error Message:
Could not load collection from ZK: utilizenodecoll

Stack Trace:
org.apache.solr.common.SolrException: Could not load collection from ZK: 
utilizenodecoll
at 
__randomizedtesting.SeedInfo.seed([4E4F809B4C9A8364:C61BBF41E266EE9C]:0)
at 
org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1316)
at 
org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:732)
at 
org.apache.solr.common.cloud.ClusterState.getCollectionOrNull(ClusterState.java:148)
at 
org.apache.solr.cloud.TestUtilizeNode.getReplicaList(TestUtilizeNode.java:170)
at 
org.apache.solr.cloud.TestUtilizeNode.assertNoReplicas(TestUtilizeNode.java:146)
at org.apache.solr.cloud.TestUtilizeNode.test(TestUtilizeNode.java:90)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at

[jira] [Commented] (LUCENE-8370) Reproducing TestLucene70DocValuesFormat.testSortedSetVariableLengthBigVsStoredFields() failure

2018-06-25 Thread Robert Muir (JIRA)



[ 
https://issues.apache.org/jira/browse/LUCENE-8370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16523034#comment-16523034
 ] 

Robert Muir commented on LUCENE-8370:
-

I mean it would be good to actually run the test with verbose+infostream to 
confirm its not a real merge policy bug, but seems like testing that merge 
policy's logic is correct belongs in the test for the specific merge policy, 
and RandomIndexWriter shouldn't do this assert.

> Reproducing 
> TestLucene70DocValuesFormat.testSortedSetVariableLengthBigVsStoredFields() 
> failure
> --
>
> Key: LUCENE-8370
> URL: https://issues.apache.org/jira/browse/LUCENE-8370
> Project: Lucene - Core
>  Issue Type: Bug
>  Components: core/index, general/test
>Reporter: Steve Rowe
>Assignee: Erick Erickson
>Priority: Major
>
> Policeman Jenkins found a reproducing seed for a 
> {{TestLucene70DocValuesFormat.testSortedSetVariableLengthBigVsStoredFields()}}
>  failure [https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/22320/]; 
> {{git bisect}} blames commit {{2519025}} on LUCENE-7976:
> {noformat}
> Checking out Revision 8c714348aeea51df19e7603905f85995bcf0371c 
> (refs/remotes/origin/master)
> [...]
>[junit4] Suite: 
> org.apache.lucene.codecs.lucene70.TestLucene70DocValuesFormat
>[junit4]   2> NOTE: reproduce with: ant test  
> -Dtestcase=TestLucene70DocValuesFormat 
> -Dtests.method=testSortedSetVariableLengthBigVsStoredFields 
> -Dtests.seed=63A61B46A6934B1A -Dtests.multiplier=3 -Dtests.slow=true 
> -Dtests.locale=sw-TZ -Dtests.timezone=Pacific/Pitcairn -Dtests.asserts=true 
> -Dtests.file.encoding=US-ASCII
>[junit4] FAILURE 23.3s J2 | 
> TestLucene70DocValuesFormat.testSortedSetVariableLengthBigVsStoredFields <<<
>[junit4]> Throwable #1: java.lang.AssertionError: limit=4 actual=5
>[junit4]>  at 
> __randomizedtesting.SeedInfo.seed([63A61B46A6934B1A:6BE93FA35E02851]:0)
>[junit4]>  at 
> org.apache.lucene.index.RandomIndexWriter.doRandomForceMerge(RandomIndexWriter.java:372)
>[junit4]>  at 
> org.apache.lucene.index.RandomIndexWriter.getReader(RandomIndexWriter.java:386)
>[junit4]>  at 
> org.apache.lucene.index.RandomIndexWriter.getReader(RandomIndexWriter.java:332)
>[junit4]>  at 
> org.apache.lucene.index.BaseDocValuesFormatTestCase.doTestSortedSetVsStoredFields(BaseDocValuesFormatTestCase.java:2155)
>[junit4]>  at 
> org.apache.lucene.codecs.lucene70.TestLucene70DocValuesFormat.testSortedSetVariableLengthBigVsStoredFields(TestLucene70DocValuesFormat.java:93)
>[junit4]>  at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>[junit4]>  at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>[junit4]>  at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>[junit4]>  at 
> java.base/java.lang.reflect.Method.invoke(Method.java:564)
>[junit4]>  at java.base/java.lang.Thread.run(Thread.java:844)
> [...]
>[junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, 
> docValues:{}, maxPointsInLeafNode=693, maxMBSortInHeap=5.078503794479895, 
> sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@20a604e6),
>  locale=sw-TZ, timezone=Pacific/Pitcairn
>[junit4]   2> NOTE: Linux 4.13.0-41-generic amd64/Oracle Corporation 9.0.4 
> (64-bit)/cpus=8,threads=1,free=352300304,total=518979584
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8370) Reproducing TestLucene70DocValuesFormat.testSortedSetVariableLengthBigVsStoredFields() failure

2018-06-25 Thread Robert Muir (JIRA)



[ 
https://issues.apache.org/jira/browse/LUCENE-8370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16523031#comment-16523031
 ] 

Robert Muir commented on LUCENE-8370:
-

The assert in RandomIndexWriter seems broken to me, given my understanding of 
the changes. Its checking that forceMerge(N) really merged to <= N segments, 
but the merge policy will no longer due that if it violates the max segment 
size, correct?

The other problem is that this only happened in a massive stress test. Seems 
like this max segment size might not being randomized properly enough in tests? 
If we lower it, more tests will engage these constraints.

> Reproducing 
> TestLucene70DocValuesFormat.testSortedSetVariableLengthBigVsStoredFields() 
> failure
> --
>
> Key: LUCENE-8370
> URL: https://issues.apache.org/jira/browse/LUCENE-8370
> Project: Lucene - Core
>  Issue Type: Bug
>  Components: core/index, general/test
>Reporter: Steve Rowe
>Assignee: Erick Erickson
>Priority: Major
>
> Policeman Jenkins found a reproducing seed for a 
> {{TestLucene70DocValuesFormat.testSortedSetVariableLengthBigVsStoredFields()}}
>  failure [https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/22320/]; 
> {{git bisect}} blames commit {{2519025}} on LUCENE-7976:
> {noformat}
> Checking out Revision 8c714348aeea51df19e7603905f85995bcf0371c 
> (refs/remotes/origin/master)
> [...]
>[junit4] Suite: 
> org.apache.lucene.codecs.lucene70.TestLucene70DocValuesFormat
>[junit4]   2> NOTE: reproduce with: ant test  
> -Dtestcase=TestLucene70DocValuesFormat 
> -Dtests.method=testSortedSetVariableLengthBigVsStoredFields 
> -Dtests.seed=63A61B46A6934B1A -Dtests.multiplier=3 -Dtests.slow=true 
> -Dtests.locale=sw-TZ -Dtests.timezone=Pacific/Pitcairn -Dtests.asserts=true 
> -Dtests.file.encoding=US-ASCII
>[junit4] FAILURE 23.3s J2 | 
> TestLucene70DocValuesFormat.testSortedSetVariableLengthBigVsStoredFields <<<
>[junit4]> Throwable #1: java.lang.AssertionError: limit=4 actual=5
>[junit4]>  at 
> __randomizedtesting.SeedInfo.seed([63A61B46A6934B1A:6BE93FA35E02851]:0)
>[junit4]>  at 
> org.apache.lucene.index.RandomIndexWriter.doRandomForceMerge(RandomIndexWriter.java:372)
>[junit4]>  at 
> org.apache.lucene.index.RandomIndexWriter.getReader(RandomIndexWriter.java:386)
>[junit4]>  at 
> org.apache.lucene.index.RandomIndexWriter.getReader(RandomIndexWriter.java:332)
>[junit4]>  at 
> org.apache.lucene.index.BaseDocValuesFormatTestCase.doTestSortedSetVsStoredFields(BaseDocValuesFormatTestCase.java:2155)
>[junit4]>  at 
> org.apache.lucene.codecs.lucene70.TestLucene70DocValuesFormat.testSortedSetVariableLengthBigVsStoredFields(TestLucene70DocValuesFormat.java:93)
>[junit4]>  at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>[junit4]>  at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>[junit4]>  at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>[junit4]>  at 
> java.base/java.lang.reflect.Method.invoke(Method.java:564)
>[junit4]>  at java.base/java.lang.Thread.run(Thread.java:844)
> [...]
>[junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, 
> docValues:{}, maxPointsInLeafNode=693, maxMBSortInHeap=5.078503794479895, 
> sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@20a604e6),
>  locale=sw-TZ, timezone=Pacific/Pitcairn
>[junit4]   2> NOTE: Linux 4.13.0-41-generic amd64/Oracle Corporation 9.0.4 
> (64-bit)/cpus=8,threads=1,free=352300304,total=518979584
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8370) Reproducing TestLucene70DocValuesFormat.testSortedSetVariableLengthBigVsStoredFields() failure

2018-06-25 Thread Erick Erickson (JIRA)



[ 
https://issues.apache.org/jira/browse/LUCENE-8370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16523021#comment-16523021
 ] 

Erick Erickson commented on LUCENE-8370:


Digging. Whimper

> Reproducing 
> TestLucene70DocValuesFormat.testSortedSetVariableLengthBigVsStoredFields() 
> failure
> --
>
> Key: LUCENE-8370
> URL: https://issues.apache.org/jira/browse/LUCENE-8370
> Project: Lucene - Core
>  Issue Type: Bug
>  Components: core/index, general/test
>Reporter: Steve Rowe
>Assignee: Erick Erickson
>Priority: Major
>
> Policeman Jenkins found a reproducing seed for a 
> {{TestLucene70DocValuesFormat.testSortedSetVariableLengthBigVsStoredFields()}}
>  failure [https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/22320/]; 
> {{git bisect}} blames commit {{2519025}} on LUCENE-7976:
> {noformat}
> Checking out Revision 8c714348aeea51df19e7603905f85995bcf0371c 
> (refs/remotes/origin/master)
> [...]
>[junit4] Suite: 
> org.apache.lucene.codecs.lucene70.TestLucene70DocValuesFormat
>[junit4]   2> NOTE: reproduce with: ant test  
> -Dtestcase=TestLucene70DocValuesFormat 
> -Dtests.method=testSortedSetVariableLengthBigVsStoredFields 
> -Dtests.seed=63A61B46A6934B1A -Dtests.multiplier=3 -Dtests.slow=true 
> -Dtests.locale=sw-TZ -Dtests.timezone=Pacific/Pitcairn -Dtests.asserts=true 
> -Dtests.file.encoding=US-ASCII
>[junit4] FAILURE 23.3s J2 | 
> TestLucene70DocValuesFormat.testSortedSetVariableLengthBigVsStoredFields <<<
>[junit4]> Throwable #1: java.lang.AssertionError: limit=4 actual=5
>[junit4]>  at 
> __randomizedtesting.SeedInfo.seed([63A61B46A6934B1A:6BE93FA35E02851]:0)
>[junit4]>  at 
> org.apache.lucene.index.RandomIndexWriter.doRandomForceMerge(RandomIndexWriter.java:372)
>[junit4]>  at 
> org.apache.lucene.index.RandomIndexWriter.getReader(RandomIndexWriter.java:386)
>[junit4]>  at 
> org.apache.lucene.index.RandomIndexWriter.getReader(RandomIndexWriter.java:332)
>[junit4]>  at 
> org.apache.lucene.index.BaseDocValuesFormatTestCase.doTestSortedSetVsStoredFields(BaseDocValuesFormatTestCase.java:2155)
>[junit4]>  at 
> org.apache.lucene.codecs.lucene70.TestLucene70DocValuesFormat.testSortedSetVariableLengthBigVsStoredFields(TestLucene70DocValuesFormat.java:93)
>[junit4]>  at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>[junit4]>  at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>[junit4]>  at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>[junit4]>  at 
> java.base/java.lang.reflect.Method.invoke(Method.java:564)
>[junit4]>  at java.base/java.lang.Thread.run(Thread.java:844)
> [...]
>[junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, 
> docValues:{}, maxPointsInLeafNode=693, maxMBSortInHeap=5.078503794479895, 
> sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@20a604e6),
>  locale=sw-TZ, timezone=Pacific/Pitcairn
>[junit4]   2> NOTE: Linux 4.13.0-41-generic amd64/Oracle Corporation 9.0.4 
> (64-bit)/cpus=8,threads=1,free=352300304,total=518979584
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Assigned] (LUCENE-8370) Reproducing TestLucene70DocValuesFormat.testSortedSetVariableLengthBigVsStoredFields() failure

2018-06-25 Thread Erick Erickson (JIRA)



 [ 
https://issues.apache.org/jira/browse/LUCENE-8370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Erick Erickson reassigned LUCENE-8370:
--

Assignee: Erick Erickson

> Reproducing 
> TestLucene70DocValuesFormat.testSortedSetVariableLengthBigVsStoredFields() 
> failure
> --
>
> Key: LUCENE-8370
> URL: https://issues.apache.org/jira/browse/LUCENE-8370
> Project: Lucene - Core
>  Issue Type: Bug
>  Components: core/index, general/test
>Reporter: Steve Rowe
>Assignee: Erick Erickson
>Priority: Major
>
> Policeman Jenkins found a reproducing seed for a 
> {{TestLucene70DocValuesFormat.testSortedSetVariableLengthBigVsStoredFields()}}
>  failure [https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/22320/]; 
> {{git bisect}} blames commit {{2519025}} on LUCENE-7976:
> {noformat}
> Checking out Revision 8c714348aeea51df19e7603905f85995bcf0371c 
> (refs/remotes/origin/master)
> [...]
>[junit4] Suite: 
> org.apache.lucene.codecs.lucene70.TestLucene70DocValuesFormat
>[junit4]   2> NOTE: reproduce with: ant test  
> -Dtestcase=TestLucene70DocValuesFormat 
> -Dtests.method=testSortedSetVariableLengthBigVsStoredFields 
> -Dtests.seed=63A61B46A6934B1A -Dtests.multiplier=3 -Dtests.slow=true 
> -Dtests.locale=sw-TZ -Dtests.timezone=Pacific/Pitcairn -Dtests.asserts=true 
> -Dtests.file.encoding=US-ASCII
>[junit4] FAILURE 23.3s J2 | 
> TestLucene70DocValuesFormat.testSortedSetVariableLengthBigVsStoredFields <<<
>[junit4]> Throwable #1: java.lang.AssertionError: limit=4 actual=5
>[junit4]>  at 
> __randomizedtesting.SeedInfo.seed([63A61B46A6934B1A:6BE93FA35E02851]:0)
>[junit4]>  at 
> org.apache.lucene.index.RandomIndexWriter.doRandomForceMerge(RandomIndexWriter.java:372)
>[junit4]>  at 
> org.apache.lucene.index.RandomIndexWriter.getReader(RandomIndexWriter.java:386)
>[junit4]>  at 
> org.apache.lucene.index.RandomIndexWriter.getReader(RandomIndexWriter.java:332)
>[junit4]>  at 
> org.apache.lucene.index.BaseDocValuesFormatTestCase.doTestSortedSetVsStoredFields(BaseDocValuesFormatTestCase.java:2155)
>[junit4]>  at 
> org.apache.lucene.codecs.lucene70.TestLucene70DocValuesFormat.testSortedSetVariableLengthBigVsStoredFields(TestLucene70DocValuesFormat.java:93)
>[junit4]>  at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>[junit4]>  at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>[junit4]>  at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>[junit4]>  at 
> java.base/java.lang.reflect.Method.invoke(Method.java:564)
>[junit4]>  at java.base/java.lang.Thread.run(Thread.java:844)
> [...]
>[junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, 
> docValues:{}, maxPointsInLeafNode=693, maxMBSortInHeap=5.078503794479895, 
> sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@20a604e6),
>  locale=sw-TZ, timezone=Pacific/Pitcairn
>[junit4]   2> NOTE: Linux 4.13.0-41-generic amd64/Oracle Corporation 9.0.4 
> (64-bit)/cpus=8,threads=1,free=352300304,total=518979584
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-BadApples-Tests-master - Build # 84 - Still Unstable

2018-06-25 Thread Apache Jenkins Server

Build: https://builds.apache.org/job/Lucene-Solr-BadApples-Tests-master/84/

2 tests failed.
FAILED:  org.apache.solr.metrics.rrd.SolrRrdBackendFactoryTest.testBasic

Error Message:
{} expected:<1> but was:<0>

Stack Trace:
java.lang.AssertionError: {} expected:<1> but was:<0>
at 
__randomizedtesting.SeedInfo.seed([60DC9B6165AE25D3:CB268674BA72A3FD]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at 
org.apache.solr.metrics.rrd.SolrRrdBackendFactoryTest.testBasic(SolrRrdBackendFactoryTest.java:90)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)


FAILED:  
org.apache.solr.client.solrj.io.stream.StreamDecoratorTest.testUpdateStream

Error Message:
Error from server at http://127.0.0.1:33216/solr: KeeperErrorCode = NoNode for

[jira] [Commented] (SOLR-12514) Rule-base Authorization plugin skips authorization if querying node does not have collection replica

2018-06-25 Thread Mahesh Kumar Vasanthu Somashekar (JIRA)



[ 
https://issues.apache.org/jira/browse/SOLR-12514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522993#comment-16522993
 ] 

Mahesh Kumar Vasanthu Somashekar commented on SOLR-12514:
-

Hi [~hgadre]

Thanks for the reply and for the patch.

If query requires hitting multiple shards distributed on multiple nodes, would 
it run one remote query for authorization and if passed trigger remote queries 
in parallel hitting all required shards?

 

Thanks for sharing the process details, I'll keep that in mind for future 
reference. Is there an official page with the process?

 

> Rule-base Authorization plugin skips authorization if querying node does not 
> have collection replica
> 
>
> Key: SOLR-12514
> URL: https://issues.apache.org/jira/browse/SOLR-12514
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: security
>Affects Versions: 7.3.1
>Reporter: Mahesh Kumar Vasanthu Somashekar
>Priority: Major
> Attachments: SOLR-12514.patch, Screen Shot 2018-06-24 at 9.36.45 
> PM.png, security.json
>
>
> Solr serves client requests going throught 3 steps - init(), authorize() and 
> handle-request ([link 
> git-link|https://github.com/apache/lucene-solr/blob/releases/lucene-solr/7.3.1/solr/core/src/java/org/apache/solr/servlet/HttpSolrCall.java#L471]).
>  init() initializes all required information to be used by authorize(). 
> init() skips initializing if request is to be served remotely, which leads to 
> skipping authorization step ([link 
> git-link|https://github.com/apache/lucene-solr/blob/releases/lucene-solr/7.3.1/solr/core/src/java/org/apache/solr/servlet/HttpSolrCall.java#L291]).
>  init() relies on 'cores' object which only has information of local node 
> (which is perfect as per design). It should actually be getting security 
> information (security.json) from zookeeper, which has global view of the 
> cluster.
>  
> Example:
>  SolrCloud setup consists of 2 nodes (solr-7.3.1):
>  live_nodes: [
>  "localhost:8983_solr",
>  "localhost:8984_solr",
>  ]
> Two collections are created - 'collection-rf-1' with RF=1 and 
> 'collection-rf-2' with RF=2.
> Two users are created - 'collection-rf-1-user' and 'collection-rf-2-user'.
> Security configuration is as below (security.json attached):
>  "authorization":{
>  "class":"solr.RuleBasedAuthorizationPlugin",
>  "permissions":[
> { "name":"read", "collection":"collection-rf-2", "role":"collection-rf-2", 
> "index":1}
> ,
> { "name":"read", "collection":"collection-rf-1", "role":"collection-rf-1", 
> "index":2}
> ,
> { "name":"read", "role":"*", "index":3}
> ,
>  ...
>  "user-role":
> { "collection-rf-1-user":[ "collection-rf-1"], "collection-rf-2-user":[ 
> "collection-rf-2"]}
> ,
>  ...
>  
> Basically, its setup to that 'collection-rf-1-user' user can only access 
> 'collection-rf-1' collection and 'collection-rf-2-user' user can only access 
> 'collection-rf-2' collection.
> Also note that 'collection-rf-1' collection replica is only on 
> 'localhost:8983_solr' node, whereas ''collection-rf-2' collection replica is 
> on both live nodes.
>  
> Authorization does not work as expected for 'collection-rf-1' collection:
> $ curl -u collection-rf-2-user:password 
> 'http://*localhost:8983*/solr/collection-rf-1/select?q=*:*'
>  
>  
>  
>  Error 403 Unauthorized request, Response code: 403
>  
>  HTTP ERROR 403
>  Problem accessing /solr/collection-rf-1/select. Reason:
>   Unauthorized request, Response code: 403
>  
>  
> $ curl -u collection-rf-2-user:password 
> 'http://*localhost:8984*/solr/collection-rf-1/select?q=*:*'
>  {
>  "responseHeader":{
>  "zkConnected":true,
>  "status":0,
>  "QTime":0,
>  "params":{
>  "q":"*:*"}},
>  "response":{"numFound":0,"start":0,"docs":[]
>  }}
>  
> Whereas authorization works perfectly for 'collection-rf-2' collection (as 
> both nodes have replica):
> $ curl -u collection-rf-1-user:password 
> 'http://*localhost:8984*/solr/collection-rf-2/select?q=*:*'
>  
>  
>  
>  Error 403 Unauthorized request, Response code: 403
>  
>  HTTP ERROR 403
>  Problem accessing /solr/collection-rf-2/select. Reason:
>   Unauthorized request, Response code: 403
>  
>  
> $ curl -u collection-rf-1-user:password 
> 'http://*localhost:8983*/solr/collection-rf-2/select?q=*:*'
>  
>  
>  
>  Error 403 Unauthorized request, Response code: 403
>  
>  HTTP ERROR 403
>  Problem accessing /solr/collection-rf-2/select. Reason:
>   Unauthorized request, Response code: 403
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail:

[jira] [Updated] (LUCENE-8370) Reproducing TestLucene70DocValuesFormat.testSortedSetVariableLengthBigVsStoredFields() failure

2018-06-25 Thread Steve Rowe (JIRA)



 [ 
https://issues.apache.org/jira/browse/LUCENE-8370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Rowe updated LUCENE-8370:
---
Summary: Reproducing 
TestLucene70DocValuesFormat.testSortedSetVariableLengthBigVsStoredFields() 
failure  (was: Reproducng 
TestLucene70DocValuesFormat.testSortedSetVariableLengthBigVsStoredFields() 
failure)

> Reproducing 
> TestLucene70DocValuesFormat.testSortedSetVariableLengthBigVsStoredFields() 
> failure
> --
>
> Key: LUCENE-8370
> URL: https://issues.apache.org/jira/browse/LUCENE-8370
> Project: Lucene - Core
>  Issue Type: Bug
>  Components: core/index, general/test
>Reporter: Steve Rowe
>Priority: Major
>
> Policeman Jenkins found a reproducing seed for a 
> {{TestLucene70DocValuesFormat.testSortedSetVariableLengthBigVsStoredFields()}}
>  failure [https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/22320/]; 
> {{git bisect}} blames commit {{2519025}} on LUCENE-7976:
> {noformat}
> Checking out Revision 8c714348aeea51df19e7603905f85995bcf0371c 
> (refs/remotes/origin/master)
> [...]
>[junit4] Suite: 
> org.apache.lucene.codecs.lucene70.TestLucene70DocValuesFormat
>[junit4]   2> NOTE: reproduce with: ant test  
> -Dtestcase=TestLucene70DocValuesFormat 
> -Dtests.method=testSortedSetVariableLengthBigVsStoredFields 
> -Dtests.seed=63A61B46A6934B1A -Dtests.multiplier=3 -Dtests.slow=true 
> -Dtests.locale=sw-TZ -Dtests.timezone=Pacific/Pitcairn -Dtests.asserts=true 
> -Dtests.file.encoding=US-ASCII
>[junit4] FAILURE 23.3s J2 | 
> TestLucene70DocValuesFormat.testSortedSetVariableLengthBigVsStoredFields <<<
>[junit4]> Throwable #1: java.lang.AssertionError: limit=4 actual=5
>[junit4]>  at 
> __randomizedtesting.SeedInfo.seed([63A61B46A6934B1A:6BE93FA35E02851]:0)
>[junit4]>  at 
> org.apache.lucene.index.RandomIndexWriter.doRandomForceMerge(RandomIndexWriter.java:372)
>[junit4]>  at 
> org.apache.lucene.index.RandomIndexWriter.getReader(RandomIndexWriter.java:386)
>[junit4]>  at 
> org.apache.lucene.index.RandomIndexWriter.getReader(RandomIndexWriter.java:332)
>[junit4]>  at 
> org.apache.lucene.index.BaseDocValuesFormatTestCase.doTestSortedSetVsStoredFields(BaseDocValuesFormatTestCase.java:2155)
>[junit4]>  at 
> org.apache.lucene.codecs.lucene70.TestLucene70DocValuesFormat.testSortedSetVariableLengthBigVsStoredFields(TestLucene70DocValuesFormat.java:93)
>[junit4]>  at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>[junit4]>  at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>[junit4]>  at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>[junit4]>  at 
> java.base/java.lang.reflect.Method.invoke(Method.java:564)
>[junit4]>  at java.base/java.lang.Thread.run(Thread.java:844)
> [...]
>[junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, 
> docValues:{}, maxPointsInLeafNode=693, maxMBSortInHeap=5.078503794479895, 
> sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@20a604e6),
>  locale=sw-TZ, timezone=Pacific/Pitcairn
>[junit4]   2> NOTE: Linux 4.13.0-41-generic amd64/Oracle Corporation 9.0.4 
> (64-bit)/cpus=8,threads=1,free=352300304,total=518979584
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (LUCENE-7976) Make TieredMergePolicy respect maxSegmentSizeMB and allow singleton merges of very large segments

2018-06-25 Thread Steve Rowe (JIRA)



[ 
https://issues.apache.org/jira/browse/LUCENE-7976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16521359#comment-16521359
 ] 

Steve Rowe edited comment on LUCENE-7976 at 6/25/18 11:20 PM:
--

{{git bisect}} points the finger at commit {{2519025}} on this issue for the 
reproducing failure noted at SOLR-12513.


was (Author: steve_rowe):
{{git blame}} points the finger at commit {{2519025}} on this issue for the 
reproducing failure noted at SOLR-12513.

> Make TieredMergePolicy respect maxSegmentSizeMB and allow singleton merges of 
> very large segments
> -
>
> Key: LUCENE-7976
> URL: https://issues.apache.org/jira/browse/LUCENE-7976
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Erick Erickson
>Assignee: Erick Erickson
>Priority: Major
> Fix For: master (8.0), 7.5
>
> Attachments: LUCENE-7976.patch, LUCENE-7976.patch, LUCENE-7976.patch, 
> LUCENE-7976.patch, LUCENE-7976.patch, LUCENE-7976.patch, LUCENE-7976.patch, 
> LUCENE-7976.patch, LUCENE-7976.patch, LUCENE-7976.patch, LUCENE-7976.patch, 
> LUCENE-7976.patch, LUCENE-7976.patch, SOLR-7976.patch
>
>
> We're seeing situations "in the wild" where there are very large indexes (on 
> disk) handled quite easily in a single Lucene index. This is particularly 
> true as features like docValues move data into MMapDirectory space. The 
> current TMP algorithm allows on the order of 50% deleted documents as per a 
> dev list conversation with Mike McCandless (and his blog here:  
> https://www.elastic.co/blog/lucenes-handling-of-deleted-documents).
> Especially in the current era of very large indexes in aggregate, (think many 
> TB) solutions like "you need to distribute your collection over more shards" 
> become very costly. Additionally, the tempting "optimize" button exacerbates 
> the issue since once you form, say, a 100G segment (by 
> optimizing/forceMerging) it is not eligible for merging until 97.5G of the 
> docs in it are deleted (current default 5G max segment size).
> The proposal here would be to add a new parameter to TMP, something like 
>  (no, that's not serious name, suggestions 
> welcome) which would default to 100 (or the same behavior we have now).
> So if I set this parameter to, say, 20%, and the max segment size stays at 
> 5G, the following would happen when segments were selected for merging:
> > any segment with > 20% deleted documents would be merged or rewritten NO 
> > MATTER HOW LARGE. There are two cases,
> >> the segment has < 5G "live" docs. In that case it would be merged with 
> >> smaller segments to bring the resulting segment up to 5G. If no smaller 
> >> segments exist, it would just be rewritten
> >> The segment has > 5G "live" docs (the result of a forceMerge or optimize). 
> >> It would be rewritten into a single segment removing all deleted docs no 
> >> matter how big it is to start. The 100G example above would be rewritten 
> >> to an 80G segment for instance.
> Of course this would lead to potentially much more I/O which is why the 
> default would be the same behavior we see now. As it stands now, though, 
> there's no way to recover from an optimize/forceMerge except to re-index from 
> scratch. We routinely see 200G-300G Lucene indexes at this point "in the 
> wild" with 10s of  shards replicated 3 or more times. And that doesn't even 
> include having these over HDFS.
> Alternatives welcome! Something like the above seems minimally invasive. A 
> new merge policy is certainly an alternative.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-7976) Make TieredMergePolicy respect maxSegmentSizeMB and allow singleton merges of very large segments

2018-06-25 Thread Steve Rowe (JIRA)



[ 
https://issues.apache.org/jira/browse/LUCENE-7976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522933#comment-16522933
 ] 

Steve Rowe commented on LUCENE-7976:


{{git bisect}} blames commit {{2519025}} on this issue for the reproducing 
failure noted at LUCENE-8370.

> Make TieredMergePolicy respect maxSegmentSizeMB and allow singleton merges of 
> very large segments
> -
>
> Key: LUCENE-7976
> URL: https://issues.apache.org/jira/browse/LUCENE-7976
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Erick Erickson
>Assignee: Erick Erickson
>Priority: Major
> Fix For: master (8.0), 7.5
>
> Attachments: LUCENE-7976.patch, LUCENE-7976.patch, LUCENE-7976.patch, 
> LUCENE-7976.patch, LUCENE-7976.patch, LUCENE-7976.patch, LUCENE-7976.patch, 
> LUCENE-7976.patch, LUCENE-7976.patch, LUCENE-7976.patch, LUCENE-7976.patch, 
> LUCENE-7976.patch, LUCENE-7976.patch, SOLR-7976.patch
>
>
> We're seeing situations "in the wild" where there are very large indexes (on 
> disk) handled quite easily in a single Lucene index. This is particularly 
> true as features like docValues move data into MMapDirectory space. The 
> current TMP algorithm allows on the order of 50% deleted documents as per a 
> dev list conversation with Mike McCandless (and his blog here:  
> https://www.elastic.co/blog/lucenes-handling-of-deleted-documents).
> Especially in the current era of very large indexes in aggregate, (think many 
> TB) solutions like "you need to distribute your collection over more shards" 
> become very costly. Additionally, the tempting "optimize" button exacerbates 
> the issue since once you form, say, a 100G segment (by 
> optimizing/forceMerging) it is not eligible for merging until 97.5G of the 
> docs in it are deleted (current default 5G max segment size).
> The proposal here would be to add a new parameter to TMP, something like 
>  (no, that's not serious name, suggestions 
> welcome) which would default to 100 (or the same behavior we have now).
> So if I set this parameter to, say, 20%, and the max segment size stays at 
> 5G, the following would happen when segments were selected for merging:
> > any segment with > 20% deleted documents would be merged or rewritten NO 
> > MATTER HOW LARGE. There are two cases,
> >> the segment has < 5G "live" docs. In that case it would be merged with 
> >> smaller segments to bring the resulting segment up to 5G. If no smaller 
> >> segments exist, it would just be rewritten
> >> The segment has > 5G "live" docs (the result of a forceMerge or optimize). 
> >> It would be rewritten into a single segment removing all deleted docs no 
> >> matter how big it is to start. The 100G example above would be rewritten 
> >> to an 80G segment for instance.
> Of course this would lead to potentially much more I/O which is why the 
> default would be the same behavior we see now. As it stands now, though, 
> there's no way to recover from an optimize/forceMerge except to re-index from 
> scratch. We routinely see 200G-300G Lucene indexes at this point "in the 
> wild" with 10s of  shards replicated 3 or more times. And that doesn't even 
> include having these over HDFS.
> Alternatives welcome! Something like the above seems minimally invasive. A 
> new merge policy is certainly an alternative.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-8370) Reproducng TestLucene70DocValuesFormat.testSortedSetVariableLengthBigVsStoredFields() failure

2018-06-25 Thread Steve Rowe (JIRA)



 [ 
https://issues.apache.org/jira/browse/LUCENE-8370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Rowe updated LUCENE-8370:
---
Component/s: general/test
 core/index

> Reproducng 
> TestLucene70DocValuesFormat.testSortedSetVariableLengthBigVsStoredFields() 
> failure
> -
>
> Key: LUCENE-8370
> URL: https://issues.apache.org/jira/browse/LUCENE-8370
> Project: Lucene - Core
>  Issue Type: Bug
>  Components: core/index, general/test
>Reporter: Steve Rowe
>Priority: Major
>
> Policeman Jenkins found a reproducing seed for a 
> {{TestLucene70DocValuesFormat.testSortedSetVariableLengthBigVsStoredFields()}}
>  failure [https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/22320/]; 
> {{git bisect}} blames commit {{2519025}} on LUCENE-7976:
> {noformat}
> Checking out Revision 8c714348aeea51df19e7603905f85995bcf0371c 
> (refs/remotes/origin/master)
> [...]
>[junit4] Suite: 
> org.apache.lucene.codecs.lucene70.TestLucene70DocValuesFormat
>[junit4]   2> NOTE: reproduce with: ant test  
> -Dtestcase=TestLucene70DocValuesFormat 
> -Dtests.method=testSortedSetVariableLengthBigVsStoredFields 
> -Dtests.seed=63A61B46A6934B1A -Dtests.multiplier=3 -Dtests.slow=true 
> -Dtests.locale=sw-TZ -Dtests.timezone=Pacific/Pitcairn -Dtests.asserts=true 
> -Dtests.file.encoding=US-ASCII
>[junit4] FAILURE 23.3s J2 | 
> TestLucene70DocValuesFormat.testSortedSetVariableLengthBigVsStoredFields <<<
>[junit4]> Throwable #1: java.lang.AssertionError: limit=4 actual=5
>[junit4]>  at 
> __randomizedtesting.SeedInfo.seed([63A61B46A6934B1A:6BE93FA35E02851]:0)
>[junit4]>  at 
> org.apache.lucene.index.RandomIndexWriter.doRandomForceMerge(RandomIndexWriter.java:372)
>[junit4]>  at 
> org.apache.lucene.index.RandomIndexWriter.getReader(RandomIndexWriter.java:386)
>[junit4]>  at 
> org.apache.lucene.index.RandomIndexWriter.getReader(RandomIndexWriter.java:332)
>[junit4]>  at 
> org.apache.lucene.index.BaseDocValuesFormatTestCase.doTestSortedSetVsStoredFields(BaseDocValuesFormatTestCase.java:2155)
>[junit4]>  at 
> org.apache.lucene.codecs.lucene70.TestLucene70DocValuesFormat.testSortedSetVariableLengthBigVsStoredFields(TestLucene70DocValuesFormat.java:93)
>[junit4]>  at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>[junit4]>  at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>[junit4]>  at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>[junit4]>  at 
> java.base/java.lang.reflect.Method.invoke(Method.java:564)
>[junit4]>  at java.base/java.lang.Thread.run(Thread.java:844)
> [...]
>[junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, 
> docValues:{}, maxPointsInLeafNode=693, maxMBSortInHeap=5.078503794479895, 
> sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@20a604e6),
>  locale=sw-TZ, timezone=Pacific/Pitcairn
>[junit4]   2> NOTE: Linux 4.13.0-41-generic amd64/Oracle Corporation 9.0.4 
> (64-bit)/cpus=8,threads=1,free=352300304,total=518979584
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (LUCENE-8370) Reproducng TestLucene70DocValuesFormat.testSortedSetVariableLengthBigVsStoredFields() failure

2018-06-25 Thread Steve Rowe (JIRA)

Steve Rowe created LUCENE-8370:
--

 Summary: Reproducng 
TestLucene70DocValuesFormat.testSortedSetVariableLengthBigVsStoredFields() 
failure
 Key: LUCENE-8370
 URL: https://issues.apache.org/jira/browse/LUCENE-8370
 Project: Lucene - Core
  Issue Type: Bug
Reporter: Steve Rowe


Policeman Jenkins found a reproducing seed for a 
{{TestLucene70DocValuesFormat.testSortedSetVariableLengthBigVsStoredFields()}} 
failure [https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/22320/]; 
{{git bisect}} blames commit {{2519025}} on LUCENE-7976:

{noformat}
Checking out Revision 8c714348aeea51df19e7603905f85995bcf0371c 
(refs/remotes/origin/master)
[...]
   [junit4] Suite: org.apache.lucene.codecs.lucene70.TestLucene70DocValuesFormat
   [junit4]   2> NOTE: reproduce with: ant test  
-Dtestcase=TestLucene70DocValuesFormat 
-Dtests.method=testSortedSetVariableLengthBigVsStoredFields 
-Dtests.seed=63A61B46A6934B1A -Dtests.multiplier=3 -Dtests.slow=true 
-Dtests.locale=sw-TZ -Dtests.timezone=Pacific/Pitcairn -Dtests.asserts=true 
-Dtests.file.encoding=US-ASCII
   [junit4] FAILURE 23.3s J2 | 
TestLucene70DocValuesFormat.testSortedSetVariableLengthBigVsStoredFields <<<
   [junit4]> Throwable #1: java.lang.AssertionError: limit=4 actual=5
   [junit4]>at 
__randomizedtesting.SeedInfo.seed([63A61B46A6934B1A:6BE93FA35E02851]:0)
   [junit4]>at 
org.apache.lucene.index.RandomIndexWriter.doRandomForceMerge(RandomIndexWriter.java:372)
   [junit4]>at 
org.apache.lucene.index.RandomIndexWriter.getReader(RandomIndexWriter.java:386)
   [junit4]>at 
org.apache.lucene.index.RandomIndexWriter.getReader(RandomIndexWriter.java:332)
   [junit4]>at 
org.apache.lucene.index.BaseDocValuesFormatTestCase.doTestSortedSetVsStoredFields(BaseDocValuesFormatTestCase.java:2155)
   [junit4]>at 
org.apache.lucene.codecs.lucene70.TestLucene70DocValuesFormat.testSortedSetVariableLengthBigVsStoredFields(TestLucene70DocValuesFormat.java:93)
   [junit4]>at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]>at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]>at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]>at 
java.base/java.lang.reflect.Method.invoke(Method.java:564)
   [junit4]>at java.base/java.lang.Thread.run(Thread.java:844)
[...]
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, 
docValues:{}, maxPointsInLeafNode=693, maxMBSortInHeap=5.078503794479895, 
sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@20a604e6),
 locale=sw-TZ, timezone=Pacific/Pitcairn
   [junit4]   2> NOTE: Linux 4.13.0-41-generic amd64/Oracle Corporation 9.0.4 
(64-bit)/cpus=8,threads=1,free=352300304,total=518979584
{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-master-Linux (64bit/jdk-9.0.4) - Build # 22320 - Unstable!

2018-06-25 Thread Policeman Jenkins Server

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/22320/
Java: 64bit/jdk-9.0.4 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

4 tests failed.
FAILED:  
org.apache.lucene.codecs.lucene70.TestLucene70DocValuesFormat.testSortedSetVariableLengthBigVsStoredFields

Error Message:
limit=4 actual=5

Stack Trace:
java.lang.AssertionError: limit=4 actual=5
at 
__randomizedtesting.SeedInfo.seed([63A61B46A6934B1A:6BE93FA35E02851]:0)
at 
org.apache.lucene.index.RandomIndexWriter.doRandomForceMerge(RandomIndexWriter.java:372)
at 
org.apache.lucene.index.RandomIndexWriter.getReader(RandomIndexWriter.java:386)
at 
org.apache.lucene.index.RandomIndexWriter.getReader(RandomIndexWriter.java:332)
at 
org.apache.lucene.index.BaseDocValuesFormatTestCase.doTestSortedSetVsStoredFields(BaseDocValuesFormatTestCase.java:2155)
at 
org.apache.lucene.codecs.lucene70.TestLucene70DocValuesFormat.testSortedSetVariableLengthBigVsStoredFields(TestLucene70DocValuesFormat.java:93)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  
org.apache.lucene.codecs.lucene70.TestLucene70DocValuesFormat.testSortedSetVariableLengthBigVsStoredFields

Error Message:
limit=4 actual=5

Stack Trace:
java.lang.AssertionError: limit=4 actual=5
at

[jira] [Resolved] (SOLR-12513) Reproducing TestCodecSupport.testMixedCompressionMode failure

2018-06-25 Thread Erick Erickson (JIRA)



 [ 
https://issues.apache.org/jira/browse/SOLR-12513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Erick Erickson resolved SOLR-12513.
---
   Resolution: Fixed
Fix Version/s: 7.5
   master (8.0)

Commit revisions:

master: 1d85cd783863f75cea133fb9c452302214165a4d
7x:53ec8224705f4f0d35751b18b3f0168517c43121

> Reproducing TestCodecSupport.testMixedCompressionMode failure
> -
>
> Key: SOLR-12513
> URL: https://issues.apache.org/jira/browse/SOLR-12513
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Steve Rowe
>Assignee: Erick Erickson
>Priority: Major
> Fix For: master (8.0), 7.5
>
> Attachments: SOLR-12513.patch, SOLR-12513.patch
>
>
> From [https://jenkins.thetaphi.de/job/Lucene-Solr-master-Windows/7375/] 
> (reproduces for me on Linux):
> {noformat}
> Checking out Revision 3b9d3a760a432b97aad2c08b2f778fa2344eb14a 
> (refs/remotes/origin/master)
> [...]
>[junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestCodecSupport 
> -Dtests.method=testMixedCompressionMode -Dtests.seed=F584376A20EC9B2D 
> -Dtests.slow=true -Dtests.locale=ha-GH -Dtests.timezone=Africa/Nairobi 
> -Dtests.asserts=true -Dtests.file.encoding=UTF-8
>[junit4] FAILURE 0.59s J1 | TestCodecSupport.testMixedCompressionMode <<<
>[junit4]> Throwable #1: org.junit.ComparisonFailure: Expecting 
> compression mode string to be BEST_SPEED but got: BEST_COMPRESSION
>[junit4]>  SegmentInfo: _5(8.0.0):c1
>[junit4]>  SegmentInfos: segments_e: _7(8.0.0):c2 _5(8.0.0):c1
>[junit4]>  Codec: Lucene70 expected: but 
> was:
>[junit4]>  at 
> __randomizedtesting.SeedInfo.seed([F584376A20EC9B2D:2BF1400E658E6E9C]:0)
>[junit4]>  at 
> org.apache.solr.core.TestCodecSupport.lambda$assertCompressionMode$0(TestCodecSupport.java:115)
>[junit4]>  at 
> org.apache.solr.core.SolrCore.withSearcher(SolrCore.java:1874)
>[junit4]>  at 
> org.apache.solr.core.TestCodecSupport.assertCompressionMode(TestCodecSupport.java:112)
>[junit4]>  at 
> org.apache.solr.core.TestCodecSupport.testMixedCompressionMode(TestCodecSupport.java:157)
>[junit4]>  at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>[junit4]>  at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>[junit4]>  at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>[junit4]>  at 
> java.base/java.lang.reflect.Method.invoke(Method.java:564)
>[junit4]>  at java.base/java.lang.Thread.run(Thread.java:844)
> [...]
>[junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, 
> docValues:{}, maxPointsInLeafNode=1658, maxMBSortInHeap=7.737749859284375, 
> sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@2da9a1c5),
>  locale=ha-GH, timezone=Africa/Nairobi
>[junit4]   2> NOTE: Windows 10 10.0 amd64/Oracle Corporation 10.0.1 
> (64-bit)/cpus=3,threads=1,free=39067800,total=97320960
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12513) Reproducing TestCodecSupport.testMixedCompressionMode failure

2018-06-25 Thread Steve Rowe (JIRA)



[ 
https://issues.apache.org/jira/browse/SOLR-12513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522901#comment-16522901
 ] 

Steve Rowe commented on SOLR-12513:
---

bq. those timings are also generated locally to an ignored file

Thanks Dawid, I had forgotten that there is both a static and a dynamic version 
of the file.

> Reproducing TestCodecSupport.testMixedCompressionMode failure
> -
>
> Key: SOLR-12513
> URL: https://issues.apache.org/jira/browse/SOLR-12513
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Steve Rowe
>Assignee: Erick Erickson
>Priority: Major
> Attachments: SOLR-12513.patch, SOLR-12513.patch
>
>
> From [https://jenkins.thetaphi.de/job/Lucene-Solr-master-Windows/7375/] 
> (reproduces for me on Linux):
> {noformat}
> Checking out Revision 3b9d3a760a432b97aad2c08b2f778fa2344eb14a 
> (refs/remotes/origin/master)
> [...]
>[junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestCodecSupport 
> -Dtests.method=testMixedCompressionMode -Dtests.seed=F584376A20EC9B2D 
> -Dtests.slow=true -Dtests.locale=ha-GH -Dtests.timezone=Africa/Nairobi 
> -Dtests.asserts=true -Dtests.file.encoding=UTF-8
>[junit4] FAILURE 0.59s J1 | TestCodecSupport.testMixedCompressionMode <<<
>[junit4]> Throwable #1: org.junit.ComparisonFailure: Expecting 
> compression mode string to be BEST_SPEED but got: BEST_COMPRESSION
>[junit4]>  SegmentInfo: _5(8.0.0):c1
>[junit4]>  SegmentInfos: segments_e: _7(8.0.0):c2 _5(8.0.0):c1
>[junit4]>  Codec: Lucene70 expected: but 
> was:
>[junit4]>  at 
> __randomizedtesting.SeedInfo.seed([F584376A20EC9B2D:2BF1400E658E6E9C]:0)
>[junit4]>  at 
> org.apache.solr.core.TestCodecSupport.lambda$assertCompressionMode$0(TestCodecSupport.java:115)
>[junit4]>  at 
> org.apache.solr.core.SolrCore.withSearcher(SolrCore.java:1874)
>[junit4]>  at 
> org.apache.solr.core.TestCodecSupport.assertCompressionMode(TestCodecSupport.java:112)
>[junit4]>  at 
> org.apache.solr.core.TestCodecSupport.testMixedCompressionMode(TestCodecSupport.java:157)
>[junit4]>  at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>[junit4]>  at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>[junit4]>  at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>[junit4]>  at 
> java.base/java.lang.reflect.Method.invoke(Method.java:564)
>[junit4]>  at java.base/java.lang.Thread.run(Thread.java:844)
> [...]
>[junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, 
> docValues:{}, maxPointsInLeafNode=1658, maxMBSortInHeap=7.737749859284375, 
> sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@2da9a1c5),
>  locale=ha-GH, timezone=Africa/Nairobi
>[junit4]   2> NOTE: Windows 10 10.0 amd64/Oracle Corporation 10.0.1 
> (64-bit)/cpus=3,threads=1,free=39067800,total=97320960
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12513) Reproducing TestCodecSupport.testMixedCompressionMode failure

2018-06-25 Thread Erick Erickson (JIRA)



[ 
https://issues.apache.org/jira/browse/SOLR-12513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522879#comment-16522879
 ] 

Erick Erickson commented on SOLR-12513:
---

Final patch, CHANGES.txt and empty hints file.

> Reproducing TestCodecSupport.testMixedCompressionMode failure
> -
>
> Key: SOLR-12513
> URL: https://issues.apache.org/jira/browse/SOLR-12513
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Steve Rowe
>Assignee: Erick Erickson
>Priority: Major
> Attachments: SOLR-12513.patch, SOLR-12513.patch
>
>
> From [https://jenkins.thetaphi.de/job/Lucene-Solr-master-Windows/7375/] 
> (reproduces for me on Linux):
> {noformat}
> Checking out Revision 3b9d3a760a432b97aad2c08b2f778fa2344eb14a 
> (refs/remotes/origin/master)
> [...]
>[junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestCodecSupport 
> -Dtests.method=testMixedCompressionMode -Dtests.seed=F584376A20EC9B2D 
> -Dtests.slow=true -Dtests.locale=ha-GH -Dtests.timezone=Africa/Nairobi 
> -Dtests.asserts=true -Dtests.file.encoding=UTF-8
>[junit4] FAILURE 0.59s J1 | TestCodecSupport.testMixedCompressionMode <<<
>[junit4]> Throwable #1: org.junit.ComparisonFailure: Expecting 
> compression mode string to be BEST_SPEED but got: BEST_COMPRESSION
>[junit4]>  SegmentInfo: _5(8.0.0):c1
>[junit4]>  SegmentInfos: segments_e: _7(8.0.0):c2 _5(8.0.0):c1
>[junit4]>  Codec: Lucene70 expected: but 
> was:
>[junit4]>  at 
> __randomizedtesting.SeedInfo.seed([F584376A20EC9B2D:2BF1400E658E6E9C]:0)
>[junit4]>  at 
> org.apache.solr.core.TestCodecSupport.lambda$assertCompressionMode$0(TestCodecSupport.java:115)
>[junit4]>  at 
> org.apache.solr.core.SolrCore.withSearcher(SolrCore.java:1874)
>[junit4]>  at 
> org.apache.solr.core.TestCodecSupport.assertCompressionMode(TestCodecSupport.java:112)
>[junit4]>  at 
> org.apache.solr.core.TestCodecSupport.testMixedCompressionMode(TestCodecSupport.java:157)
>[junit4]>  at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>[junit4]>  at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>[junit4]>  at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>[junit4]>  at 
> java.base/java.lang.reflect.Method.invoke(Method.java:564)
>[junit4]>  at java.base/java.lang.Thread.run(Thread.java:844)
> [...]
>[junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, 
> docValues:{}, maxPointsInLeafNode=1658, maxMBSortInHeap=7.737749859284375, 
> sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@2da9a1c5),
>  locale=ha-GH, timezone=Africa/Nairobi
>[junit4]   2> NOTE: Windows 10 10.0 amd64/Oracle Corporation 10.0.1 
> (64-bit)/cpus=3,threads=1,free=39067800,total=97320960
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-12513) Reproducing TestCodecSupport.testMixedCompressionMode failure

2018-06-25 Thread Erick Erickson (JIRA)



 [ 
https://issues.apache.org/jira/browse/SOLR-12513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Erick Erickson updated SOLR-12513:
--
Attachment: SOLR-12513.patch

> Reproducing TestCodecSupport.testMixedCompressionMode failure
> -
>
> Key: SOLR-12513
> URL: https://issues.apache.org/jira/browse/SOLR-12513
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Steve Rowe
>Assignee: Erick Erickson
>Priority: Major
> Attachments: SOLR-12513.patch, SOLR-12513.patch
>
>
> From [https://jenkins.thetaphi.de/job/Lucene-Solr-master-Windows/7375/] 
> (reproduces for me on Linux):
> {noformat}
> Checking out Revision 3b9d3a760a432b97aad2c08b2f778fa2344eb14a 
> (refs/remotes/origin/master)
> [...]
>[junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestCodecSupport 
> -Dtests.method=testMixedCompressionMode -Dtests.seed=F584376A20EC9B2D 
> -Dtests.slow=true -Dtests.locale=ha-GH -Dtests.timezone=Africa/Nairobi 
> -Dtests.asserts=true -Dtests.file.encoding=UTF-8
>[junit4] FAILURE 0.59s J1 | TestCodecSupport.testMixedCompressionMode <<<
>[junit4]> Throwable #1: org.junit.ComparisonFailure: Expecting 
> compression mode string to be BEST_SPEED but got: BEST_COMPRESSION
>[junit4]>  SegmentInfo: _5(8.0.0):c1
>[junit4]>  SegmentInfos: segments_e: _7(8.0.0):c2 _5(8.0.0):c1
>[junit4]>  Codec: Lucene70 expected: but 
> was:
>[junit4]>  at 
> __randomizedtesting.SeedInfo.seed([F584376A20EC9B2D:2BF1400E658E6E9C]:0)
>[junit4]>  at 
> org.apache.solr.core.TestCodecSupport.lambda$assertCompressionMode$0(TestCodecSupport.java:115)
>[junit4]>  at 
> org.apache.solr.core.SolrCore.withSearcher(SolrCore.java:1874)
>[junit4]>  at 
> org.apache.solr.core.TestCodecSupport.assertCompressionMode(TestCodecSupport.java:112)
>[junit4]>  at 
> org.apache.solr.core.TestCodecSupport.testMixedCompressionMode(TestCodecSupport.java:157)
>[junit4]>  at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>[junit4]>  at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>[junit4]>  at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>[junit4]>  at 
> java.base/java.lang.reflect.Method.invoke(Method.java:564)
>[junit4]>  at java.base/java.lang.Thread.run(Thread.java:844)
> [...]
>[junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, 
> docValues:{}, maxPointsInLeafNode=1658, maxMBSortInHeap=7.737749859284375, 
> sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@2da9a1c5),
>  locale=ha-GH, timezone=Africa/Nairobi
>[junit4]   2> NOTE: Windows 10 10.0 amd64/Oracle Corporation 10.0.1 
> (64-bit)/cpus=3,threads=1,free=39067800,total=97320960
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12513) Reproducing TestCodecSupport.testMixedCompressionMode failure

2018-06-25 Thread Erick Erickson (JIRA)



[ 
https://issues.apache.org/jira/browse/SOLR-12513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522829#comment-16522829
 ] 

Erick Erickson commented on SOLR-12513:
---

OK, I'll just remove all the content then? There's no point in keeping the 
content if it's not useful.

> Reproducing TestCodecSupport.testMixedCompressionMode failure
> -
>
> Key: SOLR-12513
> URL: https://issues.apache.org/jira/browse/SOLR-12513
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Steve Rowe
>Assignee: Erick Erickson
>Priority: Major
> Attachments: SOLR-12513.patch
>
>
> From [https://jenkins.thetaphi.de/job/Lucene-Solr-master-Windows/7375/] 
> (reproduces for me on Linux):
> {noformat}
> Checking out Revision 3b9d3a760a432b97aad2c08b2f778fa2344eb14a 
> (refs/remotes/origin/master)
> [...]
>[junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestCodecSupport 
> -Dtests.method=testMixedCompressionMode -Dtests.seed=F584376A20EC9B2D 
> -Dtests.slow=true -Dtests.locale=ha-GH -Dtests.timezone=Africa/Nairobi 
> -Dtests.asserts=true -Dtests.file.encoding=UTF-8
>[junit4] FAILURE 0.59s J1 | TestCodecSupport.testMixedCompressionMode <<<
>[junit4]> Throwable #1: org.junit.ComparisonFailure: Expecting 
> compression mode string to be BEST_SPEED but got: BEST_COMPRESSION
>[junit4]>  SegmentInfo: _5(8.0.0):c1
>[junit4]>  SegmentInfos: segments_e: _7(8.0.0):c2 _5(8.0.0):c1
>[junit4]>  Codec: Lucene70 expected: but 
> was:
>[junit4]>  at 
> __randomizedtesting.SeedInfo.seed([F584376A20EC9B2D:2BF1400E658E6E9C]:0)
>[junit4]>  at 
> org.apache.solr.core.TestCodecSupport.lambda$assertCompressionMode$0(TestCodecSupport.java:115)
>[junit4]>  at 
> org.apache.solr.core.SolrCore.withSearcher(SolrCore.java:1874)
>[junit4]>  at 
> org.apache.solr.core.TestCodecSupport.assertCompressionMode(TestCodecSupport.java:112)
>[junit4]>  at 
> org.apache.solr.core.TestCodecSupport.testMixedCompressionMode(TestCodecSupport.java:157)
>[junit4]>  at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>[junit4]>  at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>[junit4]>  at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>[junit4]>  at 
> java.base/java.lang.reflect.Method.invoke(Method.java:564)
>[junit4]>  at java.base/java.lang.Thread.run(Thread.java:844)
> [...]
>[junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, 
> docValues:{}, maxPointsInLeafNode=1658, maxMBSortInHeap=7.737749859284375, 
> sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@2da9a1c5),
>  locale=ha-GH, timezone=Africa/Nairobi
>[junit4]   2> NOTE: Windows 10 10.0 amd64/Oracle Corporation 10.0.1 
> (64-bit)/cpus=3,threads=1,free=39067800,total=97320960
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1571 - Still Unstable

2018-06-25 Thread Apache Jenkins Server

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1571/

3 tests failed.
FAILED:  org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.test_stored_idx

Error Message:
Some docs had errors -- check logs expected:<0> but was:<1>

Stack Trace:
java.lang.AssertionError: Some docs had errors -- check logs expected:<0> but 
was:<1>
at 
__randomizedtesting.SeedInfo.seed([915C419EB633AB03:813DEC67A391EE2D]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at 
org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.checkField(TestStressCloudBlindAtomicUpdates.java:342)
at 
org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.test_stored_idx(TestStressCloudBlindAtomicUpdates.java:241)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)


FAILED:

[JENKINS] Lucene-Solr-7.x-Linux (64bit/jdk1.8.0_172) - Build # 2194 - Unstable!

2018-06-25 Thread Policeman Jenkins Server

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/2194/
Java: 64bit/jdk1.8.0_172 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

2 tests failed.
FAILED:  org.apache.solr.cloud.cdcr.CdcrBidirectionalTest.testBiDir

Error Message:
Captured an uncaught exception in thread: Thread[id=237, 
name=cdcr-replicator-64-thread-1, state=RUNNABLE, 
group=TGRP-CdcrBidirectionalTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught 
exception in thread: Thread[id=237, name=cdcr-replicator-64-thread-1, 
state=RUNNABLE, group=TGRP-CdcrBidirectionalTest]
Caused by: java.lang.AssertionError
at __randomizedtesting.SeedInfo.seed([1F50649D48A0942D]:0)
at 
org.apache.solr.update.CdcrUpdateLog$CdcrLogReader.forwardSeek(CdcrUpdateLog.java:611)
at org.apache.solr.handler.CdcrReplicator.run(CdcrReplicator.java:125)
at 
org.apache.solr.handler.CdcrReplicatorScheduler.lambda$null$0(CdcrReplicatorScheduler.java:81)
at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.cloud.cdcr.CdcrBidirectionalTest.testBiDir

Error Message:
Captured an uncaught exception in thread: Thread[id=3236, 
name=cdcr-replicator-1443-thread-1, state=RUNNABLE, 
group=TGRP-CdcrBidirectionalTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught 
exception in thread: Thread[id=3236, name=cdcr-replicator-1443-thread-1, 
state=RUNNABLE, group=TGRP-CdcrBidirectionalTest]
Caused by: java.lang.AssertionError
at __randomizedtesting.SeedInfo.seed([1F50649D48A0942D]:0)
at 
org.apache.solr.update.CdcrUpdateLog$CdcrLogReader.forwardSeek(CdcrUpdateLog.java:611)
at org.apache.solr.handler.CdcrReplicator.run(CdcrReplicator.java:125)
at 
org.apache.solr.handler.CdcrReplicatorScheduler.lambda$null$0(CdcrReplicatorScheduler.java:81)
at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 12851 lines...]
   [junit4] Suite: org.apache.solr.cloud.cdcr.CdcrBidirectionalTest
   [junit4]   2> 299313 INFO  
(SUITE-CdcrBidirectionalTest-seed#[1F50649D48A0942D]-worker) [] 
o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: 
test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: 
/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.cdcr.CdcrBidirectionalTest_1F50649D48A0942D-001/init-core-data-001
   [junit4]   2> 299314 WARN  
(SUITE-CdcrBidirectionalTest-seed#[1F50649D48A0942D]-worker) [] 
o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=37 numCloses=37
   [junit4]   2> 299314 INFO  
(SUITE-CdcrBidirectionalTest-seed#[1F50649D48A0942D]-worker) [] 
o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) 
w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 299315 INFO  
(SUITE-CdcrBidirectionalTest-seed#[1F50649D48A0942D]-worker) [] 
o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: 
@org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 299316 INFO  
(TEST-CdcrBidirectionalTest.testBiDir-seed#[1F50649D48A0942D]) [] 
o.a.s.SolrTestCaseJ4 ###Starting testBiDir
   [junit4]   2> 299317 INFO  
(TEST-CdcrBidirectionalTest.testBiDir-seed#[1F50649D48A0942D]) [] 
o.a.s.c.MiniSolrCloudCluster Starting cluster of 1 servers in 
/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.cdcr.CdcrBidirectionalTest_1F50649D48A0942D-001/cdcr-cluster2-001
   [junit4]   2> 299317 INFO  
(TEST-CdcrBidirectionalTest.testBiDir-seed#[1F50649D48A0942D]) [] 
o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 299317 INFO  (Thread-719) [] o.a.s.c.ZkTestServer client 
port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 299317 INFO  (Thread-719) [] o.a.s.c.ZkTestServer Starting 
server
   [junit4]   2> 299319 ERROR (Thread-719) [] o.a.z.s.ZooKeeperServer 
ZKShutdownHandler is not registered, so ZooKeeper server won't take any action 
on ERROR or SHUTDOWN server state changes
   [junit4]   2> 299417 INFO  
(TEST-CdcrBidirectionalTest.testBiDir-seed#[1F50649D48A0942D]) [] 
o.a.s.c.ZkTestServer start zk server on port:45805
   [junit4]   2> 299419 INFO  (zkConnectionManagerCallback-662-thread-1) [] 
o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 299423

[jira] [Commented] (SOLR-12513) Reproducing TestCodecSupport.testMixedCompressionMode failure

2018-06-25 Thread Dawid Weiss (JIRA)



[ 
https://issues.apache.org/jira/browse/SOLR-12513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522773#comment-16522773
 ] 

Dawid Weiss commented on SOLR-12513:


That file is used to provide default timings for load balancing of suites 
across JVMs to minimize the overall execution time. I can't remember the last 
time I updated that file, so perhaps we should just leave it as empty (which is 
not a big deal since those timings are also generated locally to an ignored 
file and even if they're not used, there is job-stealing in place).

Try:
{code}
cd lucene
ant test-help
{code}
and see the section:
{code}
 [help] #
 [help] # Load balancing and caches. --
 [help] #
{code}

> Reproducing TestCodecSupport.testMixedCompressionMode failure
> -
>
> Key: SOLR-12513
> URL: https://issues.apache.org/jira/browse/SOLR-12513
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Steve Rowe
>Assignee: Erick Erickson
>Priority: Major
> Attachments: SOLR-12513.patch
>
>
> From [https://jenkins.thetaphi.de/job/Lucene-Solr-master-Windows/7375/] 
> (reproduces for me on Linux):
> {noformat}
> Checking out Revision 3b9d3a760a432b97aad2c08b2f778fa2344eb14a 
> (refs/remotes/origin/master)
> [...]
>[junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestCodecSupport 
> -Dtests.method=testMixedCompressionMode -Dtests.seed=F584376A20EC9B2D 
> -Dtests.slow=true -Dtests.locale=ha-GH -Dtests.timezone=Africa/Nairobi 
> -Dtests.asserts=true -Dtests.file.encoding=UTF-8
>[junit4] FAILURE 0.59s J1 | TestCodecSupport.testMixedCompressionMode <<<
>[junit4]> Throwable #1: org.junit.ComparisonFailure: Expecting 
> compression mode string to be BEST_SPEED but got: BEST_COMPRESSION
>[junit4]>  SegmentInfo: _5(8.0.0):c1
>[junit4]>  SegmentInfos: segments_e: _7(8.0.0):c2 _5(8.0.0):c1
>[junit4]>  Codec: Lucene70 expected: but 
> was:
>[junit4]>  at 
> __randomizedtesting.SeedInfo.seed([F584376A20EC9B2D:2BF1400E658E6E9C]:0)
>[junit4]>  at 
> org.apache.solr.core.TestCodecSupport.lambda$assertCompressionMode$0(TestCodecSupport.java:115)
>[junit4]>  at 
> org.apache.solr.core.SolrCore.withSearcher(SolrCore.java:1874)
>[junit4]>  at 
> org.apache.solr.core.TestCodecSupport.assertCompressionMode(TestCodecSupport.java:112)
>[junit4]>  at 
> org.apache.solr.core.TestCodecSupport.testMixedCompressionMode(TestCodecSupport.java:157)
>[junit4]>  at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>[junit4]>  at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>[junit4]>  at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>[junit4]>  at 
> java.base/java.lang.reflect.Method.invoke(Method.java:564)
>[junit4]>  at java.base/java.lang.Thread.run(Thread.java:844)
> [...]
>[junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, 
> docValues:{}, maxPointsInLeafNode=1658, maxMBSortInHeap=7.737749859284375, 
> sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@2da9a1c5),
>  locale=ha-GH, timezone=Africa/Nairobi
>[junit4]   2> NOTE: Windows 10 10.0 amd64/Oracle Corporation 10.0.1 
> (64-bit)/cpus=3,threads=1,free=39067800,total=97320960
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [DISCUSS] Geo/spatial organization in Lucene

2018-06-25 Thread Karl Wright

+1


On Mon, Jun 25, 2018 at 12:46 PM Nicholas Knize  wrote:

> +1
>
> On Mon, Jun 25, 2018, 11:32 AM Ignacio Vera Sequeiros 
> wrote:
>
>> +1
>> --
>> *From:* Alan Woodward 
>> *Sent:* Monday, June 25, 2018 5:56:16 PM
>> *To:* dev@lucene.apache.org
>>
>> *Subject:* Re: [DISCUSS] Geo/spatial organization in Lucene
>> +1 to move LatLonPoint and friends to core, and nuke the spatial module
>>
>> On 25 Jun 2018, at 16:32, David Smiley  wrote:
>>
>> Okay fine, I'm not going to block spatial stuff going into core.
>>  (celebration).  I foresee the spatial stuff there growing beyond the one
>> default impl though.
>>
>> Perhaps most of us are still not happy with seeing spatial code across so
>> many modules?  Nick and I have voiced this concern so far.  Given the
>> pittance of utility of what's in the spatial module today, can we agree to
>> simply remove it?
>>
>> I pity users trying to figure out what is where to make sense of it.  I
>> wonder how new users discover/browse to look around -- I'm too used to the
>> codebase to have any idea what newbies do.  That seems to be this:
>> http://lucene.apache.org/core/7_3_1/index.html  Each module only gets
>> one terse sentence fragment.  It'd be nice to have potentially a paragraph
>> of information?  Even without more verbage, the spatial ones could have
>> better descriptions.  I propose these changes:
>>
>> * spatial:  remove it :-)   -- see above
>> * spatial3d: Computational geometry on the surface of a sphere or
>> ellipsoid, including Lucene index & search solutions
>> * spatial-extras: Spatial code that has external dependencies like
>> Spatial4j and JTS, including Lucene index & search solutions
>>
>> perhaps "spatial-sphere" might be a more meaningful name than spatial3d?
>> Yes, it's ellipsoidal but sphere is close enough ;-)
>>
>> ~ David
>>
>> On Mon, Jun 25, 2018 at 10:42 AM Michael McCandless <
>> luc...@mikemccandless.com> wrote:
>>
>>> I also favor B: move the common case, good performing spatial
>>> implementations to core, but still bake new things in sandbox.  LatLonPoint
>>> has baked way too long already!  The addition of first class (codec
>>> support) KD trees in Lucene (dimensional points) was/is really a game
>>> changer for Lucene supporting common geo spatial applications.
>>>
>>> It would be nice to find a better name than geo3d / spatial3d: it
>>> confuses 100% of the people I explain it to, on first impression :)  But we
>>> should tackle that separately/later.
>>>
>>> Merging the 2D/3D abstractions sounds a little too ambitious at this
>>> point, so I think it's fine to leave them separate for now.
>>>
>>> Mike McCandless
>>>
>>> http://blog.mikemccandless.com
>>>
>>> On Wed, Jun 20, 2018 at 1:00 PM, Nicholas Knize 
>>> wrote:
>>>
 If I were to pick between the two, I also have a preference for B.
 I've also tried to keep this whole spatial organization rather simple:

 core - simple spatial capabilities needed by the 99% spatial use case
 (e.g., web mapping). Includes LatLonPoint, polygon & distance search
 (everything currently in sandbox). Lightweight, and no dependencies or
 complexities. If one wants simple and fast point search, all you need is
 the core module.

 spatial - dependency free. Expands on core spatial to include simple
 shape searching. Uses internal relations. Everything confined to core and
 spatial modules.

 spatial-extras - expanded spatial capabilities. Welcomes third-party
 dependencies (e.g., S3, SIS, Proj4J). Targets more advanced/expert GIS
 use-cases.

 geo3d - trades speed for accuracy. I've always struggled with the name,
 since it implies 3D shapes/point cloud support. But history has shown
 considering a name change to be a bike-shedding endeavor.

 At the end of the day I'm up for whatever makes most sense for everyone
 here. Lord knows we could use more people helping out on geo.

 - Nick



 On Wed, Jun 20, 2018 at 11:40 AM Adrien Grand 
 wrote:

> I have a slight preference for B similarly to how StandardAnalyzer is
> in core and other analyzers are in analysis, but no strong feelings. In 
> any
> case I agree that both A and B would be much better than the current
> situation.
>
>
> Le mer. 20 juin 2018 à 18:09, David Smiley 
> a écrit :
>
>> I think everyone agrees the current state of spatial code
>> organization in Lucene is not desirable.  We have a spatial module that 
>> has
>> almost nothing in it, we have mature spatial code in the sandbox that 
>> needs
>> to "graduate" somewhere, and we've got a handful of geo utilities in 
>> Lucene
>> core (mostly because I didn't notice).  No agreement has been reached on
>> what the desired state should be.
>>
>> I'd like to hear opinions on this from members of the community.  I
>> am especially interested in

[JENKINS] Lucene-Solr-7.x-Solaris (64bit/jdk1.8.0) - Build # 687 - Unstable!

2018-06-25 Thread Policeman Jenkins Server

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Solaris/687/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  org.apache.solr.cloud.TestCloudPivotFacet.test

Error Message:
Error from server at http://127.0.0.1:53443/collection1_shard1_replica_n43: 
ERROR: [doc=1] unknown field 'pivot_b1'

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
from server at http://127.0.0.1:53443/collection1_shard1_replica_n43: ERROR: 
[doc=1] unknown field 'pivot_b1'
at 
__randomizedtesting.SeedInfo.seed([3C58B24F3060F299:B40C8D959E9C9F61]:0)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:561)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1015)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:886)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:819)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:173)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:138)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:152)
at 
org.apache.solr.cloud.TestCloudPivotFacet.test(TestCloudPivotFacet.java:133)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)

[jira] [Updated] (SOLR-12483) Trend based suggestions

2018-06-25 Thread Cassandra Targett (JIRA)



 [ 
https://issues.apache.org/jira/browse/SOLR-12483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Cassandra Targett updated SOLR-12483:
-
Component/s: AutoScaling

> Trend based suggestions
> ---
>
> Key: SOLR-12483
> URL: https://issues.apache.org/jira/browse/SOLR-12483
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: AutoScaling
>Reporter: Noble Paul
>Priority: Major
>
> A cluster may be perfectly fine now and it may not be so in the future. It is 
> possible to extrapolate past trends to predict the future state certain 
> values.  Solr can make recommendations to
>  * Move replica
>  * Add node 
> based on these projections
> We can use SOLR-11779 to see the historic trend for various values such as 
> {{freedisk}} , {{queries per second}} , {{index size}} etc . it is possible 
> to extrapolate the future values and make recommendations



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12294) System collection - Lazy loading mechanism not working for custom UpdateProcessors

2018-06-25 Thread Cassandra Targett (JIRA)



[ 
https://issues.apache.org/jira/browse/SOLR-12294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522726#comment-16522726
 ] 

Cassandra Targett commented on SOLR-12294:
--

[~noble.paul] - this could be resolved, correct? Fixed for 7.4?

> System collection - Lazy loading mechanism not working for custom 
> UpdateProcessors
> --
>
> Key: SOLR-12294
> URL: https://issues.apache.org/jira/browse/SOLR-12294
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: UpdateRequestProcessors
>Affects Versions: 7.3
>Reporter: Johannes Brucher
>Assignee: Noble Paul
>Priority: Critical
> Attachments: no_active_replica_available.png, schema.xml, 
> solrconfig.xml, update-processor-0.0.1-SNAPSHOT.jar
>
>
> Hi all,
> I'm facing an issue regarding custom code inside a .system-collection and 
> starting up a Solr Cloud cluster.
> I thought, like its stated in the documentation, that in case using the 
> .system collection custom code is lazy loaded, because it can happen that a 
> collection that uses custom code is initialized before the .system collection 
> is up and running.
> I did all the necessary configuration and while debugging, I can see that the 
> custom code is wrapped via a PluginBag$LazyPluginHolder. So far its seems 
> good, but I still get Exceptions when starting the Solr Cloud with the 
> following errors:
> SolrException: Blob loading failed: .no active replica available for .system 
> collection...
> In my case I'm using custom code for a couple of UpdateProcessors. So it 
> seems, that this lazy mechanism is not working well for UpdateProcessors.
> Inside the calzz LazyPluginHolder the comment says:
> "A class that loads plugins Lazily. When the get() method is invoked the 
> Plugin is initialized and returned."
> When a core is initialized and you have a custom UpdateProcessor, the 
> get-method is invoked directly and the lazy loading mechanism tries to get 
> the custom class from the MemClassLoader, but in most scenarios the system 
> collection is not up and the above Exception is thrown...
> So maybe it’s the case that for UpdateProcessors while initializing a core, 
> the routine is not implemented optimal for the lazy loading mechanism?
>  
> Here are the steps to reproduce the issue:
>  # Unpack solr 7.3.0
>  1.1 Add SOLR_OPTS="$SOLR_OPTS -Denable.runtime.lib=true" to solr.in.sh
>  1.2 Start Solr with -c option
>  # Setup .system collection:
>  2.1 Upload custom test jar --> curl -X POST -H 'Content-Type: 
> application/octet-stream' --data-binary @ jar>/update-processor-0.0.1-SNAPSHOT.jar http:// solr>/solr/.system/blob/test_blob
>  2.2 Alter maxShardsPerNode --> 
> .../admin/collections?action=MODIFYCOLLECTION=.system=2
>  2.2 Add Replica to .system collection --> 
> .../admin/collections?action=ADDREPLICA=.system=shard1
>  # Setup test collection:
>  3.1 Upload test conf to ZK --> ./zkcli.sh -zkhost  -cmd 
> upconfig -confdir  -confname test_conf
>  3.2 Create a test1 collection with commented UP-chain inside solrconfig.xml 
> via Admin UI
>  3.3 Add blob to test collection --> curl http:// Solr>/solr/test1/config -H 'Content-type:application/json' -d 
> '\{"add-runtimelib": { "name":"test_blob", "version":1 }}'
>  3.4 Uncomment the UP-chain and upload test-conf again --> ./zkcli.sh -zkhost 
>  -cmd upconfig -confdir  -confname test_conf
>  3.5 Reload test1 collection
>  3.6 Everything should work as expected now (no erros are shown)
>  # Restart SOLR
>  4.1 Now you can see: SolrException: Blob loading failed: No active replica 
> available for .system collection
> Expected: The lazy loading mechanism should work for UpdateProcessors inside 
> core init routine, but it isn't due to above Exception.
> Sometimes you are lucky and the test1 collection will be initialize after the 
> .system collection. But in ~90% of the time this isn't the case...
> Let me know if you need further details here,
>  
> Johannes



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12389) Support nested properties in cluster props

2018-06-25 Thread Cassandra Targett (JIRA)



[ 
https://issues.apache.org/jira/browse/SOLR-12389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522724#comment-16522724
 ] 

Cassandra Targett commented on SOLR-12389:
--

[~noble.paul] - this issue could be resolved, correct? Fixed for 7.4?

> Support nested properties in cluster props
> --
>
> Key: SOLR-12389
> URL: https://issues.apache.org/jira/browse/SOLR-12389
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
> Environment: cluster props API does not support nested objects . 
>  
>  
>Reporter: Noble Paul
>Assignee: Noble Paul
>Priority: Major
>
> A new command is added to V2 endpoint to set deeply nested objects
> example 1:
> {code}
> $ curl http://localhost:8983/api/cluster -H 'Content-type: application/json' 
> -d '
> { "set-obj-property":  {
>   "collectionDefaults" :{
>  "numShards": 3, 
>  "nrtreplicas": "2 ,
>  "tlogReplicas":3,
>  "pullReplicas" : 2
> }}}'
> {code}
> example 2:
> unset the value of {{numShards}}
> {code}
> $ curl http://localhost:8983/api/cluster -H 'Content-type: application/json' 
> -d '
> { "set-obj-property":  {
>   "collectionDefaults" :{
>  "numShards": null
> }}}'
> {code}
> example 2:
> unset the value of {{numShards}}
> example 3:
> unset the entire {{collectionDefaults}} and set another value for another key 
> all in one command
> {code}
> $ curl http://localhost:8983/api/cluster -H 'Content-type: application/json' 
> -d '
> { "set-obj-property":  {
>  "autoAddReplicas" : true,
>   "collectionDefaults" :null}}'
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-12504) Leader should compute its fingerprint and retrieve recent updates in an atomic way

2018-06-25 Thread Cassandra Targett (JIRA)



 [ 
https://issues.apache.org/jira/browse/SOLR-12504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Cassandra Targett updated SOLR-12504:
-
Component/s: SolrCloud

> Leader should compute its fingerprint and retrieve recent updates in an 
> atomic way
> --
>
> Key: SOLR-12504
> URL: https://issues.apache.org/jira/browse/SOLR-12504
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrCloud
>Reporter: Cao Manh Dat
>Assignee: Cao Manh Dat
>Priority: Major
>
> SOLR-11216 solved many cases in the PeerSync (in doing recovery). But there 
> are still cases when PeerSync will fail because of mismatch in fingerprint 
> comparison. The main reason here is fingerprint and recent versions of leader 
> is not computed/retrieved in an atomic way. 
> For example: when an update made into leader's tlog after fingerprint is 
> computed but before recent versions are retrieved.
> Leader's fingerprint  : (contains updates from 1-10)
> Leader's recent versions : (contains updates from 1-12)
> Replica's fingerprint:  (contains updates from 1-12)
> --> A mismatch in fingerprint between leader and replica.
> But it seems that blocking updates in leader for {{getVersions}} operation is 
> not a good idea (it will degrade indexing performance). Still struggling on 
> finding a solution.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-12505) Streaming expressions - fetch() does not work as expected

2018-06-25 Thread Cassandra Targett (JIRA)



 [ 
https://issues.apache.org/jira/browse/SOLR-12505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Cassandra Targett updated SOLR-12505:
-
Component/s: streaming expressions

> Streaming expressions - fetch() does not work as expected
> -
>
> Key: SOLR-12505
> URL: https://issues.apache.org/jira/browse/SOLR-12505
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: streaming expressions
>Affects Versions: 7.3.1
> Environment: Windows 10, Java 10, Solr Cloud 7.3.1
>Reporter: Dariusz Wojtas
>Assignee: Joel Bernstein
>Priority: Major
> Attachments: names.zip
>
>
> The issue:
>  # when I try to use fetch() within a streaming expression, it does not 
> enrich the inner source data. The result is exactly the same as if there was 
> no surrounding fetch() function.
>  # but it works if I try to do a leftOuterJoin() function instead.
> Use the attached 'names' collection configuration.
>  SOLR works in _cloud_ mode, streaming expressions do work, ie. stream(), 
> join(), etc
> Data to be inserted:
>  ==
> {code:xml}
> 
>  
>   1
>   entity
>   Orignal Darek name
>   uk
>   
>N001
>1
>alternate
>Darek
>   
>   
>N002
>1
>alternate
>Darke
>   
>   
>N003
>1
>alternate
>   Darko
>   
>  
>  
>   2
>   entity
>   Texaco
>   de
>   
>N0011
>2
>alternate
>Texxo
>   
>   
>N0012
>2
>alternate
>Texoco
>   
>  
> 
> {code}
> ==
>  The streaming query to execute.
>  Simplified, as the mainsearch usually does more complext stuff.
>  ==
> {noformat}
>  fetch( 
>  names,
>  search(names,
>  qt="/select",
>  q="*:*",
>  fq="type:alternate",
>  fl="parentId, alias",
>  rows=10,
>  sort="parentId asc"), 
>  on="parentId=id",
>  fl="name,country"
>  )
> {noformat}
> ==
> *Result*:
>  * Collection of attributes: parentId, alias
> *Expected result*:
>  * Collection of attributes: parentId, alias, name, country



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-12509) Improve SplitShardCmd performance and reliability

2018-06-25 Thread Cassandra Targett (JIRA)



 [ 
https://issues.apache.org/jira/browse/SOLR-12509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Cassandra Targett updated SOLR-12509:
-
Component/s: SolrCloud

> Improve SplitShardCmd performance and reliability
> -
>
> Key: SOLR-12509
> URL: https://issues.apache.org/jira/browse/SOLR-12509
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrCloud
>Reporter: Andrzej Bialecki 
>Assignee: Andrzej Bialecki 
>Priority: Major
>
> {{SplitShardCmd}} is currently quite complex.
> Shard splitting occurs on active shards, which are still being updated, so 
> the splitting has to involve several carefully orchestrated steps, making 
> sure that new sub-shard placeholders are properly created and visible, and 
> then also applying buffered updates to the split leaders and performing 
> recovery on sub-shard replicas.
> This process could be simplified in cases where collections are not actively 
> being updated or can tolerate a little downtime - we could put the shard 
> "offline", ie. disable writing while the splitting is in progress (in order 
> to avoid users' confusion we should disable writing to the whole collection).
> The actual index splitting could perhaps be improved to use 
> {{HardLinkCopyDirectoryWrapper}} for creating a copy of the index by 
> hard-linking existing index segments, and then applying deletes to the 
> documents that don't belong in a sub-shard. However, the resulting index 
> slices that replicas would have to pull would be the same size as the whole 
> shard.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-12498) Deprecate some params that MODIFYCOLLECTION supports

2018-06-25 Thread Cassandra Targett (JIRA)



 [ 
https://issues.apache.org/jira/browse/SOLR-12498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Cassandra Targett updated SOLR-12498:
-
Component/s: SolrCloud

> Deprecate some params that MODIFYCOLLECTION supports
> 
>
> Key: SOLR-12498
> URL: https://issues.apache.org/jira/browse/SOLR-12498
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrCloud
>Reporter: Varun Thacker
>Priority: Major
>
> We can deprecate and then remove certain params that we support in the modify 
> collection API
>  
> This thread has the relevant discussion - 
> http://lucene.472066.n3.nabble.com/Do-we-need-the-MODIFYCOLLECTION-Api-td4394487.html



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-12501) System Collection should use the correct luceneMatchVersion

2018-06-25 Thread Cassandra Targett (JIRA)



 [ 
https://issues.apache.org/jira/browse/SOLR-12501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Cassandra Targett updated SOLR-12501:
-
Component/s: blobstore

> System Collection should use the correct luceneMatchVersion
> ---
>
> Key: SOLR-12501
> URL: https://issues.apache.org/jira/browse/SOLR-12501
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: blobstore
>Reporter: Varun Thacker
>Priority: Major
>
> Whenever we create the system collection, the solrconfig is hard-coded and 
> comes from SystemCollectionSolrConfig.java 
> Here we always set the luceneMatchVersion to the latest instead of the lucene 
> version that is being released.
> When you go to create the collection we will get this warning the logs
> {code:java}
> WARN  - 2018-06-19 14:53:00.400; [c:.system s:shard1 r:core_node2 
> x:.system_shard1_replica_n1] org.apache.solr.core.Config; You should not use 
> LATEST as luceneMatchVersion property: if you use this setting, and then Solr 
> upgrades to a newer release of Lucene, sizable changes may happen. If precise 
> back compatibility is important then you should instead explicitly specify an 
> actual Lucene version.{code}
> The downside is if a user upgrades Solr then the new docs analysis behaviour 
> could differ. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-12500) Add support for '<=' and '>=' operators in the autoscaling policy syntax

2018-06-25 Thread Cassandra Targett (JIRA)



 [ 
https://issues.apache.org/jira/browse/SOLR-12500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Cassandra Targett updated SOLR-12500:
-
Component/s: AutoScaling

> Add support for '<=' and '>=' operators in the autoscaling policy syntax
> 
>
> Key: SOLR-12500
> URL: https://issues.apache.org/jira/browse/SOLR-12500
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: AutoScaling
>Reporter: Noble Paul
>Priority: Major
>
> Add support for these commonly used operators to improve readability



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-12511) Support non integer values for replica in autoscaling rules

2018-06-25 Thread Cassandra Targett (JIRA)



 [ 
https://issues.apache.org/jira/browse/SOLR-12511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Cassandra Targett updated SOLR-12511:
-
Component/s: AutoScaling

> Support non integer values for replica in autoscaling rules
> ---
>
> Key: SOLR-12511
> URL: https://issues.apache.org/jira/browse/SOLR-12511
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: AutoScaling
>Reporter: Noble Paul
>Assignee: Noble Paul
>Priority: Major
>
> This means the user can configure a decimal value in replica
> example :
> {code:json}
> {"replica": 1.638, "node":"#ANY"}
> {code}
> This means a few things. The no:of of replicas in a node can be either 2 or 
> 1. This also means that violations are calculated as follows
>  * If the replica count is 1 or 2 there are no violations 
>  * If the replica count is 3, there is a violation and the delta is 
> *{{3-1.638 = 1.362}}*
>  * if the replica count is 0, there is a violation and the delta is *{{1.638 
> - 0 = 1.638}}*
>  * This also means that the node with zero replicas has a *more serious* 
> violation and the system would try to rectify that first before it address 
> the node with 3 replicas



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12516) JSON "range" facets don't refine sub-facets under special buckets (before,after,between)

2018-06-25 Thread Hoss Man (JIRA)



[ 
https://issues.apache.org/jira/browse/SOLR-12516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522690#comment-16522690
 ] 

Hoss Man commented on SOLR-12516:
-

One complexity in fixing this is the way the refinement requests are currently 
built – when building/parsing the {{\_l}} (and {{\_s + \_p}} ) list(s) the code 
currently assume everything it finds is a bucket value (or a pair of bucket 
value and sub-facet refinement requests) ... there's no intermediate structure 
that says "buckets" like there is in the facet response, so there is no way to 
put "before"/"between"/"after" outside of the list of bucket values to indicate 
what sub-facets need refined -- and putting them in the bucket list seems like 
a hack that could cause problems w/the existing code that expects everything in 
that list to be parsable by the {{Calc}} for this facet.
Perhaps a new {{\_other}} key should be added alongside the existing {{\_l, 
\_s, \_p}} keys that _then_ contains {{\_l, \_s, \_p}} for each of the "other" 
buckets (by name) where a sub-facet needs refinement?

Something like...

{noformat}
doTestRefine("{top:{type:range, other:all, field:R, start:0, end:1, gap:1, 
facet:{x : {type:terms, field:X, limit:2, refine:true} } } }",
"{top: {buckets:[{val:0, count:2, x:{buckets:[{val:x1, 
count:5},{val:x2, count:3}]} } ]" +
"   before:{count:0},after:{count:0}," +
"   between:{count:2,x:{buckets:[{val:x1, count:5},{val:x2, 
count:3}]} } } }",
"{top: {buckets:[{val:0, count:1, x:{buckets:[{val:x2, 
count:4},{val:x3, count:2}]} } ]" +
"   before:{count:0},after:{count:0}," +
"   between:{count:1,x:{buckets:[{val:x2, count:4},{val:x3, 
count:2}]} } } }",
null,
"=={top: {" +
"_s:[  [0 , {x:{_l:[x1]}} ]  ]" +
"_other:{_s:[  ['between' , {x:{_l:[x1]}} ]  ] }" +
"}  " +
"}"
);
{noformat}

?

> JSON "range" facets don't refine sub-facets under special buckets 
> (before,after,between)
> 
>
> Key: SOLR-12516
> URL: https://issues.apache.org/jira/browse/SOLR-12516
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Facet Module
>Reporter: Hoss Man
>Priority: Major
> Attachments: SOLR-12516.patch
>
>
> {{FacetRangeMerger extends FacetRequestSortedMerger}} ... however 
> {{FacetRangeMerger}} does not override {{getRefinement(...)}} which means 
> only {{FacetRequestSortedMerger.buckets}} is evaluated and considered for 
> refinement. The additional, special purpose, {{FacetBucket}} instances 
> tracked in {{FacetRangeMerger}} are never considered for refinement.
> In a simple range facet this doesn't cause any problems because these buckets 
> are returned by every shard on the phase#1 request -- but *if a sub-facet 
> (such as a field facet) is nested under a range facet then the buckets 
> returned by the sub-facets for the before/between/after buckets will never be 
> refined* ... the phase#1 sub-facet buckets will be merged as.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12514) Rule-base Authorization plugin skips authorization if querying node does not have collection replica

2018-06-25 Thread Hrishikesh Gadre (JIRA)



[ 
https://issues.apache.org/jira/browse/SOLR-12514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522680#comment-16522680
 ] 

Hrishikesh Gadre commented on SOLR-12514:
-

I think we should consider changing the visibility of this Jira to "private" 
and follow the regular process of fixing security vulnerability (e.g. filing 
CVE etc.). [~mahesh.kumar.vs] for future reference, such security issues should 
be reported to the security@apache mailing list instead of regular bug filing 
process.

> Rule-base Authorization plugin skips authorization if querying node does not 
> have collection replica
> 
>
> Key: SOLR-12514
> URL: https://issues.apache.org/jira/browse/SOLR-12514
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: security
>Affects Versions: 7.3.1
>Reporter: Mahesh Kumar Vasanthu Somashekar
>Priority: Major
> Attachments: SOLR-12514.patch, Screen Shot 2018-06-24 at 9.36.45 
> PM.png, security.json
>
>
> Solr serves client requests going throught 3 steps - init(), authorize() and 
> handle-request ([link 
> git-link|https://github.com/apache/lucene-solr/blob/releases/lucene-solr/7.3.1/solr/core/src/java/org/apache/solr/servlet/HttpSolrCall.java#L471]).
>  init() initializes all required information to be used by authorize(). 
> init() skips initializing if request is to be served remotely, which leads to 
> skipping authorization step ([link 
> git-link|https://github.com/apache/lucene-solr/blob/releases/lucene-solr/7.3.1/solr/core/src/java/org/apache/solr/servlet/HttpSolrCall.java#L291]).
>  init() relies on 'cores' object which only has information of local node 
> (which is perfect as per design). It should actually be getting security 
> information (security.json) from zookeeper, which has global view of the 
> cluster.
>  
> Example:
>  SolrCloud setup consists of 2 nodes (solr-7.3.1):
>  live_nodes: [
>  "localhost:8983_solr",
>  "localhost:8984_solr",
>  ]
> Two collections are created - 'collection-rf-1' with RF=1 and 
> 'collection-rf-2' with RF=2.
> Two users are created - 'collection-rf-1-user' and 'collection-rf-2-user'.
> Security configuration is as below (security.json attached):
>  "authorization":{
>  "class":"solr.RuleBasedAuthorizationPlugin",
>  "permissions":[
> { "name":"read", "collection":"collection-rf-2", "role":"collection-rf-2", 
> "index":1}
> ,
> { "name":"read", "collection":"collection-rf-1", "role":"collection-rf-1", 
> "index":2}
> ,
> { "name":"read", "role":"*", "index":3}
> ,
>  ...
>  "user-role":
> { "collection-rf-1-user":[ "collection-rf-1"], "collection-rf-2-user":[ 
> "collection-rf-2"]}
> ,
>  ...
>  
> Basically, its setup to that 'collection-rf-1-user' user can only access 
> 'collection-rf-1' collection and 'collection-rf-2-user' user can only access 
> 'collection-rf-2' collection.
> Also note that 'collection-rf-1' collection replica is only on 
> 'localhost:8983_solr' node, whereas ''collection-rf-2' collection replica is 
> on both live nodes.
>  
> Authorization does not work as expected for 'collection-rf-1' collection:
> $ curl -u collection-rf-2-user:password 
> 'http://*localhost:8983*/solr/collection-rf-1/select?q=*:*'
>  
>  
>  
>  Error 403 Unauthorized request, Response code: 403
>  
>  HTTP ERROR 403
>  Problem accessing /solr/collection-rf-1/select. Reason:
>   Unauthorized request, Response code: 403
>  
>  
> $ curl -u collection-rf-2-user:password 
> 'http://*localhost:8984*/solr/collection-rf-1/select?q=*:*'
>  {
>  "responseHeader":{
>  "zkConnected":true,
>  "status":0,
>  "QTime":0,
>  "params":{
>  "q":"*:*"}},
>  "response":{"numFound":0,"start":0,"docs":[]
>  }}
>  
> Whereas authorization works perfectly for 'collection-rf-2' collection (as 
> both nodes have replica):
> $ curl -u collection-rf-1-user:password 
> 'http://*localhost:8984*/solr/collection-rf-2/select?q=*:*'
>  
>  
>  
>  Error 403 Unauthorized request, Response code: 403
>  
>  HTTP ERROR 403
>  Problem accessing /solr/collection-rf-2/select. Reason:
>   Unauthorized request, Response code: 403
>  
>  
> $ curl -u collection-rf-1-user:password 
> 'http://*localhost:8983*/solr/collection-rf-2/select?q=*:*'
>  
>  
>  
>  Error 403 Unauthorized request, Response code: 403
>  
>  HTTP ERROR 403
>  Problem accessing /solr/collection-rf-2/select. Reason:
>   Unauthorized request, Response code: 403
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-12514) Rule-base Authorization plugin skips authorization if querying node does not have collection replica

2018-06-25 Thread Hrishikesh Gadre (JIRA)



 [ 
https://issues.apache.org/jira/browse/SOLR-12514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hrishikesh Gadre updated SOLR-12514:

Attachment: SOLR-12514.patch

> Rule-base Authorization plugin skips authorization if querying node does not 
> have collection replica
> 
>
> Key: SOLR-12514
> URL: https://issues.apache.org/jira/browse/SOLR-12514
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: security
>Affects Versions: 7.3.1
>Reporter: Mahesh Kumar Vasanthu Somashekar
>Priority: Major
> Attachments: SOLR-12514.patch, Screen Shot 2018-06-24 at 9.36.45 
> PM.png, security.json
>
>
> Solr serves client requests going throught 3 steps - init(), authorize() and 
> handle-request ([link 
> git-link|https://github.com/apache/lucene-solr/blob/releases/lucene-solr/7.3.1/solr/core/src/java/org/apache/solr/servlet/HttpSolrCall.java#L471]).
>  init() initializes all required information to be used by authorize(). 
> init() skips initializing if request is to be served remotely, which leads to 
> skipping authorization step ([link 
> git-link|https://github.com/apache/lucene-solr/blob/releases/lucene-solr/7.3.1/solr/core/src/java/org/apache/solr/servlet/HttpSolrCall.java#L291]).
>  init() relies on 'cores' object which only has information of local node 
> (which is perfect as per design). It should actually be getting security 
> information (security.json) from zookeeper, which has global view of the 
> cluster.
>  
> Example:
>  SolrCloud setup consists of 2 nodes (solr-7.3.1):
>  live_nodes: [
>  "localhost:8983_solr",
>  "localhost:8984_solr",
>  ]
> Two collections are created - 'collection-rf-1' with RF=1 and 
> 'collection-rf-2' with RF=2.
> Two users are created - 'collection-rf-1-user' and 'collection-rf-2-user'.
> Security configuration is as below (security.json attached):
>  "authorization":{
>  "class":"solr.RuleBasedAuthorizationPlugin",
>  "permissions":[
> { "name":"read", "collection":"collection-rf-2", "role":"collection-rf-2", 
> "index":1}
> ,
> { "name":"read", "collection":"collection-rf-1", "role":"collection-rf-1", 
> "index":2}
> ,
> { "name":"read", "role":"*", "index":3}
> ,
>  ...
>  "user-role":
> { "collection-rf-1-user":[ "collection-rf-1"], "collection-rf-2-user":[ 
> "collection-rf-2"]}
> ,
>  ...
>  
> Basically, its setup to that 'collection-rf-1-user' user can only access 
> 'collection-rf-1' collection and 'collection-rf-2-user' user can only access 
> 'collection-rf-2' collection.
> Also note that 'collection-rf-1' collection replica is only on 
> 'localhost:8983_solr' node, whereas ''collection-rf-2' collection replica is 
> on both live nodes.
>  
> Authorization does not work as expected for 'collection-rf-1' collection:
> $ curl -u collection-rf-2-user:password 
> 'http://*localhost:8983*/solr/collection-rf-1/select?q=*:*'
>  
>  
>  
>  Error 403 Unauthorized request, Response code: 403
>  
>  HTTP ERROR 403
>  Problem accessing /solr/collection-rf-1/select. Reason:
>   Unauthorized request, Response code: 403
>  
>  
> $ curl -u collection-rf-2-user:password 
> 'http://*localhost:8984*/solr/collection-rf-1/select?q=*:*'
>  {
>  "responseHeader":{
>  "zkConnected":true,
>  "status":0,
>  "QTime":0,
>  "params":{
>  "q":"*:*"}},
>  "response":{"numFound":0,"start":0,"docs":[]
>  }}
>  
> Whereas authorization works perfectly for 'collection-rf-2' collection (as 
> both nodes have replica):
> $ curl -u collection-rf-1-user:password 
> 'http://*localhost:8984*/solr/collection-rf-2/select?q=*:*'
>  
>  
>  
>  Error 403 Unauthorized request, Response code: 403
>  
>  HTTP ERROR 403
>  Problem accessing /solr/collection-rf-2/select. Reason:
>   Unauthorized request, Response code: 403
>  
>  
> $ curl -u collection-rf-1-user:password 
> 'http://*localhost:8983*/solr/collection-rf-2/select?q=*:*'
>  
>  
>  
>  Error 403 Unauthorized request, Response code: 403
>  
>  HTTP ERROR 403
>  Problem accessing /solr/collection-rf-2/select. Reason:
>   Unauthorized request, Response code: 403
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-BadApples-Tests-7.x - Build # 88 - Still Unstable

2018-06-25 Thread Apache Jenkins Server

Build: https://builds.apache.org/job/Lucene-Solr-BadApples-Tests-7.x/88/

2 tests failed.
FAILED:  org.apache.solr.cloud.cdcr.CdcrBidirectionalTest.testBiDir

Error Message:
Captured an uncaught exception in thread: Thread[id=31432, 
name=cdcr-replicator-10196-thread-1, state=RUNNABLE, 
group=TGRP-CdcrBidirectionalTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught 
exception in thread: Thread[id=31432, name=cdcr-replicator-10196-thread-1, 
state=RUNNABLE, group=TGRP-CdcrBidirectionalTest]
Caused by: java.lang.AssertionError
at __randomizedtesting.SeedInfo.seed([8FBDC81BB7DE2F32]:0)
at 
org.apache.solr.update.CdcrUpdateLog$CdcrLogReader.forwardSeek(CdcrUpdateLog.java:611)
at org.apache.solr.handler.CdcrReplicator.run(CdcrReplicator.java:125)
at 
org.apache.solr.handler.CdcrReplicatorScheduler.lambda$null$0(CdcrReplicatorScheduler.java:81)
at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)


FAILED:  
org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testSplitIntegration

Error Message:
last state: 
DocCollection(testSplitIntegration_collection//clusterstate.json/107)={   
"replicationFactor":"1",   "pullReplicas":"0",   
"router":{"name":"compositeId"},   "maxShardsPerNode":"2",   
"autoAddReplicas":"false",   "nrtReplicas":"2",   "tlogReplicas":"0",   
"autoCreated":"true",   "shards":{ "shard2":{   "replicas":{ 
"core_node3":{   
"core":"testSplitIntegration_collection_shard2_replica_n3",   
"leader":"true",   "SEARCHER.searcher.maxDoc":11,   
"SEARCHER.searcher.deletedDocs":0,   "INDEX.sizeInBytes":15740, 
  "node_name":"127.0.0.1:1_solr",   "state":"active",   
"type":"NRT",   "INDEX.sizeInGB":1.4659017324447632E-5,   
"SEARCHER.searcher.numDocs":11}, "core_node4":{   
"core":"testSplitIntegration_collection_shard2_replica_n4",   
"SEARCHER.searcher.maxDoc":11,   "SEARCHER.searcher.deletedDocs":0, 
  "INDEX.sizeInBytes":15740,   "node_name":"127.0.0.1:10001_solr",  
 "state":"active",   "type":"NRT",   
"INDEX.sizeInGB":1.4659017324447632E-5,   
"SEARCHER.searcher.numDocs":11}},   "range":"0-7fff",   
"state":"active"}, "shard1":{   "replicas":{ "core_node1":{ 
  "core":"testSplitIntegration_collection_shard1_replica_n1",   
"leader":"true",   "SEARCHER.searcher.maxDoc":14,   
"SEARCHER.searcher.deletedDocs":0,   "INDEX.sizeInBytes":17240, 
  "node_name":"127.0.0.1:1_solr",   "state":"active",   
"type":"NRT",   "INDEX.sizeInGB":1.605600118637085E-5,   
"SEARCHER.searcher.numDocs":14}, "core_node2":{   
"core":"testSplitIntegration_collection_shard1_replica_n2",   
"SEARCHER.searcher.maxDoc":14,   "SEARCHER.searcher.deletedDocs":0, 
  "INDEX.sizeInBytes":17240,   "node_name":"127.0.0.1:10001_solr",  
 "state":"active",   "type":"NRT",   
"INDEX.sizeInGB":1.605600118637085E-5,   
"SEARCHER.searcher.numDocs":14}},   "range":"8000-",   
"state":"active"}}}

Stack Trace:
java.util.concurrent.TimeoutException: last state: 
DocCollection(testSplitIntegration_collection//clusterstate.json/107)={
  "replicationFactor":"1",
  "pullReplicas":"0",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"2",
  "autoAddReplicas":"false",
  "nrtReplicas":"2",
  "tlogReplicas":"0",
  "autoCreated":"true",
  "shards":{
"shard2":{
  "replicas":{
"core_node3":{
  "core":"testSplitIntegration_collection_shard2_replica_n3",
  "leader":"true",
  "SEARCHER.searcher.maxDoc":11,
  "SEARCHER.searcher.deletedDocs":0,
  "INDEX.sizeInBytes":15740,
  "node_name":"127.0.0.1:1_solr",
  "state":"active",
  "type":"NRT",
  "INDEX.sizeInGB":1.4659017324447632E-5,
  "SEARCHER.searcher.numDocs":11},
"core_node4":{
  "core":"testSplitIntegration_collection_shard2_replica_n4",
  "SEARCHER.searcher.maxDoc":11,
  "SEARCHER.searcher.deletedDocs":0,
  "INDEX.sizeInBytes":15740,
  "node_name":"127.0.0.1:10001_solr",
  "state":"active",
  "type":"NRT",
  "INDEX.sizeInGB":1.4659017324447632E-5,
  "SEARCHER.searcher.numDocs":11}},
  "range":"0-7fff",
  "state":"active"},
"shard1":{
  "replicas":{
"core_node1":{

[jira] [Created] (LUCENE-8369) Remove the spatial module as it is obsolete

2018-06-25 Thread David Smiley (JIRA)

David Smiley created LUCENE-8369:


 Summary: Remove the spatial module as it is obsolete
 Key: LUCENE-8369
 URL: https://issues.apache.org/jira/browse/LUCENE-8369
 Project: Lucene - Core
  Issue Type: Task
  Components: modules/spatial
Reporter: David Smiley
Assignee: David Smiley


The "spatial" module is at this juncture nearly empty with only a couple 
utilities that aren't used by anything in the entire codebase -- 
GeoRelationUtils, and MortonEncoder.  Perhaps it should have been removed 
earlier in LUCENE-7664 which was the removal of GeoPointField which was 
essentially why the module existed.  Better late than never.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12514) Rule-base Authorization plugin skips authorization if querying node does not have collection replica

2018-06-25 Thread Hrishikesh Gadre (JIRA)



[ 
https://issues.apache.org/jira/browse/SOLR-12514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522660#comment-16522660
 ] 

Hrishikesh Gadre commented on SOLR-12514:
-

Yes I also have observed this behavior with [Solr Sentry authorization 
plugin|https://github.com/apache/sentry/blob/74842080936b93a8ef9b874774fd841764adc42c/sentry-binding/sentry-binding-solr/src/main/java/org/apache/sentry/binding/solr/authz/SentrySolrPluginImpl.java].
 The root cause was the fact that the authorization framework relies on 
PermissionNameProvider interface to figure out the permissions to be checked 
against the authorization database. The individual request handlers in Solr are 
expected to implement this interface so as to define which permissions to check 
for which functionality. Now in case a core is not available on a given Solr 
server, the authorization context provides null value for the request handler. 
The Sentry plugin performs an "instanceof" check against the request handler 
reference. If this check fails (either because the request handler is null or 
does not implement PermissionNameProvider interface), it approves the request 
anyway. This is required since there are many request handlers in Solr which 
don't support authorization (Ref: SOLR-11623) and the authorization plugin 
needs to play nice with these request handlers. Now once the request is 
approved by the authorization plugin, the forwarded request is executed as a 
solr admin user (which has all the permissions). As a result, an user without 
authorization can see the query results.

The fix was to use the "proxy users" functionality provided by the Hadoop 
authentication plugin. In this case, Solr automatically applies a "doAs" 
parameter to the query being forwarded to the remote solr server. This forces 
the remote Solr instance to treat this as a request from original user instead 
of from solr admin. Hence when it performs the authorization check, the request 
is rejected appropriately. This fix works well in our environment because we 
only support Hadoop authentication plugin which has "proxy users" support. We 
will need to think about generic solution for it to be applicable for all 
users. Let me upload the patch for reference. 

> Rule-base Authorization plugin skips authorization if querying node does not 
> have collection replica
> 
>
> Key: SOLR-12514
> URL: https://issues.apache.org/jira/browse/SOLR-12514
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: security
>Affects Versions: 7.3.1
>Reporter: Mahesh Kumar Vasanthu Somashekar
>Priority: Major
> Attachments: Screen Shot 2018-06-24 at 9.36.45 PM.png, security.json
>
>
> Solr serves client requests going throught 3 steps - init(), authorize() and 
> handle-request ([link 
> git-link|https://github.com/apache/lucene-solr/blob/releases/lucene-solr/7.3.1/solr/core/src/java/org/apache/solr/servlet/HttpSolrCall.java#L471]).
>  init() initializes all required information to be used by authorize(). 
> init() skips initializing if request is to be served remotely, which leads to 
> skipping authorization step ([link 
> git-link|https://github.com/apache/lucene-solr/blob/releases/lucene-solr/7.3.1/solr/core/src/java/org/apache/solr/servlet/HttpSolrCall.java#L291]).
>  init() relies on 'cores' object which only has information of local node 
> (which is perfect as per design). It should actually be getting security 
> information (security.json) from zookeeper, which has global view of the 
> cluster.
>  
> Example:
>  SolrCloud setup consists of 2 nodes (solr-7.3.1):
>  live_nodes: [
>  "localhost:8983_solr",
>  "localhost:8984_solr",
>  ]
> Two collections are created - 'collection-rf-1' with RF=1 and 
> 'collection-rf-2' with RF=2.
> Two users are created - 'collection-rf-1-user' and 'collection-rf-2-user'.
> Security configuration is as below (security.json attached):
>  "authorization":{
>  "class":"solr.RuleBasedAuthorizationPlugin",
>  "permissions":[
> { "name":"read", "collection":"collection-rf-2", "role":"collection-rf-2", 
> "index":1}
> ,
> { "name":"read", "collection":"collection-rf-1", "role":"collection-rf-1", 
> "index":2}
> ,
> { "name":"read", "role":"*", "index":3}
> ,
>  ...
>  "user-role":
> { "collection-rf-1-user":[ "collection-rf-1"], "collection-rf-2-user":[ 
> "collection-rf-2"]}
> ,
>  ...
>  
> Basically, its setup to that 'collection-rf-1-user' user can only access 
> 'collection-rf-1' collection and 'collection-rf-2-user' user can only access 
> 'collection-rf-2' collection.
> Also note that 'collection-rf-1' collection replica is only on 
> 'localhost:8983_solr' node, whereas ''collection-rf-2' collection replica

[jira] [Commented] (SOLR-12441) Add deeply nested documents URP

2018-06-25 Thread David Smiley (JIRA)



[ 
https://issues.apache.org/jira/browse/SOLR-12441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522648#comment-16522648
 ] 

David Smiley commented on SOLR-12441:
-

A more concrete straw-man proposal is as follows:  In 
{{SolrCore.loadUpdateProcessorChains()}} conditionally add a new 
NestedUpdateProcessorFactory immediately prior to RunUpdateProcessorFactory.  
In NestedUpdateProcessorFactory.getInstance(), conditionally wrap the 
RunUpdateProcessorFactory dependent on the 
{{indexSchema.isUsableForChildDocs()}}.  This proposal is for 8.0.  
NestedUpdateProcessorFactory would add "\_root\_" in this plan.  If we all 
generally agree with this, such a proposal would be its own issue and not this 
one, I think.

If someone doesn't want nested docs, then they shouldn't define "_root_" in the 
schema.  There are now other fields to deal with as well though -- like 
\_nestPath\_ and \_nestParent\_ -- where should these be defined?  We could 
simply add these in the schema by default.  NestedUpdateProcessorFactory could 
simply look at the schema for their existence to see if it should populate 
these (or not) instead of bothering with any configuration of the URP -- thus 
it's easier and more automatic.  It is kinda a shame that a from-scratch schema 
would require adding a bunch new fields but ah well.

> Add deeply nested documents URP
> ---
>
> Key: SOLR-12441
> URL: https://issues.apache.org/jira/browse/SOLR-12441
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: mosh
>Assignee: David Smiley
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> As discussed in 
> [SOLR-12298|https://issues.apache.org/jira/browse/SOLR-12298], there ought to 
> be an URP to add metadata fields to childDocuments in order to allow a 
> transformer to rebuild the original document hierarchy.
> {quote}I propose we add the following fields:
>  # __nestParent__
>  # _nestLevel_
>  # __nestPath__
> __nestParent__: This field wild will store the document's parent docId, to be 
> used for building the whole hierarchy, using a new document transformer, as 
> suggested by Jan on the mailing list.
> _nestLevel_: This field will store the level of the specified field in the 
> document, using an int value. This field can be used for the parentFilter, 
> eliminating the need to provide a parentFilter, which will be set by default 
> as "_level_:queriedFieldLevel".
> _nestLevel_: This field will contain the full path, separated by a specific 
> reserved char e.g., '.'
>  for example: "first.second.third".
>  This will enable users to search for a specific path, or provide a regular 
> expression to search for fields sharing the same name in different levels of 
> the document, filtering using the level key if needed.
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-12458) ADLS support for SOLR

2018-06-25 Thread Mike Wingert (JIRA)



 [ 
https://issues.apache.org/jira/browse/SOLR-12458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mike Wingert updated SOLR-12458:

Attachment: SOLR-12458.patch

> ADLS support for SOLR
> -
>
> Key: SOLR-12458
> URL: https://issues.apache.org/jira/browse/SOLR-12458
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Hadoop Integration
>Affects Versions: master (8.0)
>Reporter: Mike Wingert
>Priority: Minor
>  Labels: features
> Fix For: master (8.0)
>
> Attachments: SOLR-12458.patch, SOLR-12458.patch, SOLR-12458.patch, 
> SOLR-12458.patch, SOLR-12458.patch, SOLR-12458.patch, SOLR-12458.patch, 
> SOLR-12458.patch
>
>
> This is to track ADLS support for SOLR.
> ADLS is a HDFS like API available in Microsoft Azure.   
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12516) JSON "range" facets don't refine sub-facets under special buckets (before,after,between)

2018-06-25 Thread Hoss Man (JIRA)



[ 
https://issues.apache.org/jira/browse/SOLR-12516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522632#comment-16522632
 ] 

Hoss Man commented on SOLR-12516:
-

Attached a trivial test patch that modifies the existing 
TestJsonFacetRefinement check of a range facet w/a single bucket to also 
specify {{other:all}} so that the {{between}} bucket should have the same count 
& sub-facet results as the existing single bucket – but it does not.

> JSON "range" facets don't refine sub-facets under special buckets 
> (before,after,between)
> 
>
> Key: SOLR-12516
> URL: https://issues.apache.org/jira/browse/SOLR-12516
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Facet Module
>Reporter: Hoss Man
>Priority: Major
> Attachments: SOLR-12516.patch
>
>
> {{FacetRangeMerger extends FacetRequestSortedMerger}} ... however 
> {{FacetRangeMerger}} does not override {{getRefinement(...)}} which means 
> only {{FacetRequestSortedMerger.buckets}} is evaluated and considered for 
> refinement. The additional, special purpose, {{FacetBucket}} instances 
> tracked in {{FacetRangeMerger}} are never considered for refinement.
> In a simple range facet this doesn't cause any problems because these buckets 
> are returned by every shard on the phase#1 request -- but *if a sub-facet 
> (such as a field facet) is nested under a range facet then the buckets 
> returned by the sub-facets for the before/between/after buckets will never be 
> refined* ... the phase#1 sub-facet buckets will be merged as.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-12516) JSON "range" facets don't refine sub-facets under special buckets (before,after,between)

2018-06-25 Thread Hoss Man (JIRA)



 [ 
https://issues.apache.org/jira/browse/SOLR-12516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hoss Man updated SOLR-12516:

Attachment: SOLR-12516.patch

> JSON "range" facets don't refine sub-facets under special buckets 
> (before,after,between)
> 
>
> Key: SOLR-12516
> URL: https://issues.apache.org/jira/browse/SOLR-12516
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Facet Module
>Reporter: Hoss Man
>Priority: Major
> Attachments: SOLR-12516.patch
>
>
> {{FacetRangeMerger extends FacetRequestSortedMerger}} ... however 
> {{FacetRangeMerger}} does not override {{getRefinement(...)}} which means 
> only {{FacetRequestSortedMerger.buckets}} is evaluated and considered for 
> refinement. The additional, special purpose, {{FacetBucket}} instances 
> tracked in {{FacetRangeMerger}} are never considered for refinement.
> In a simple range facet this doesn't cause any problems because these buckets 
> are returned by every shard on the phase#1 request -- but *if a sub-facet 
> (such as a field facet) is nested under a range facet then the buckets 
> returned by the sub-facets for the before/between/after buckets will never be 
> refined* ... the phase#1 sub-facet buckets will be merged as.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-5211) updating parent as childless makes old children orphans

2018-06-25 Thread David Smiley (JIRA)



[ 
https://issues.apache.org/jira/browse/SOLR-5211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522624#comment-16522624
 ] 

David Smiley commented on SOLR-5211:


Feel free to propose a trial patch with such a change.  I wonder if your 
proposal would be better/worse -- I'm not sure.  Note that there are some 
external callers that want a Lucene Document from the command as well.

A cursory look at your patch looked fine... and I can see that there is some 
awkwardness in conditionally adding the block ID prompting your suggestion to 
decouple Document conversion about of the command.  Separately in SOLR-12441 I 
proposed maybe \_root\_ manipulation should be done in an URP, the one being 
added in that issue.

> updating parent as childless makes old children orphans
> ---
>
> Key: SOLR-5211
> URL: https://issues.apache.org/jira/browse/SOLR-5211
> Project: Solr
>  Issue Type: Sub-task
>  Components: update
>Affects Versions: 4.5, 6.0
>Reporter: Mikhail Khludnev
>Assignee: Mikhail Khludnev
>Priority: Major
> Attachments: SOLR-5211.patch, SOLR-5211.patch, SOLR-5211.patch
>
>
> if I have parent with children in the index, I can send update omitting 
> children. as a result old children become orphaned. 
> I suppose separate \_root_ fields makes much trouble. I propose to extend 
> notion of uniqueKey, and let it spans across blocks that makes updates 
> unambiguous.  
> WDYT? Do you like to see a test proves this issue?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12516) JSON "range" facets don't refine sub-facets under special buckets (before,after,between)

2018-06-25 Thread Hoss Man (JIRA)



[ 
https://issues.apache.org/jira/browse/SOLR-12516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522598#comment-16522598
 ] 

Hoss Man commented on SOLR-12516:
-

I only discovered this because of a randomized failure from 
{{CurrencyRangeFacetCloudTest}} while testing out SOLR-12343 -- that patch 
(currently) enforces that every shard which has at least one bucket for a field 
facet must "contribute" to every bucket (either in phase#1 or via refinement) 
to be returned -- and it can cause the existing 
{{testJsonRangeFacetWithSubFacet}} to fail for some (randomized) shard count 
because of this bug.

I'll try to work up a more isolated, generalized, test case that shows 
incorrect counts (on a simple/non-currency) for a range facet w/o SOLR-12343 in 
place

> JSON "range" facets don't refine sub-facets under special buckets 
> (before,after,between)
> 
>
> Key: SOLR-12516
> URL: https://issues.apache.org/jira/browse/SOLR-12516
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Facet Module
>Reporter: Hoss Man
>Priority: Major
>
> {{FacetRangeMerger extends FacetRequestSortedMerger}} ... however 
> {{FacetRangeMerger}} does not override {{getRefinement(...)}} which means 
> only {{FacetRequestSortedMerger.buckets}} is evaluated and considered for 
> refinement. The additional, special purpose, {{FacetBucket}} instances 
> tracked in {{FacetRangeMerger}} are never considered for refinement.
> In a simple range facet this doesn't cause any problems because these buckets 
> are returned by every shard on the phase#1 request -- but *if a sub-facet 
> (such as a field facet) is nested under a range facet then the buckets 
> returned by the sub-facets for the before/between/after buckets will never be 
> refined* ... the phase#1 sub-facet buckets will be merged as.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12441) Add deeply nested documents URP

2018-06-25 Thread David Smiley (JIRA)



[ 
https://issues.apache.org/jira/browse/SOLR-12441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522593#comment-16522593
 ] 

David Smiley commented on SOLR-12441:
-

Thanks for the PR [~moshebla]!

With 8.0 on the horizon this fall, we can think of the best solution that isn't 
necessarily fully backwards compatible. 
Perhaps this URP should handle \_root\_ as well to thus unify where fields 
involved in nested docs are added?  Although that raises other questions like 
how might this URP be registered automatically?  Wether it should be an URP vs 
something baked in?  I think it should be an URP but it'd be nice if an URP 
like this could be registered automatically... but under what circumstances and 
how to control that -- I don't know (though I have ideas). 

CC [~osavrasov] as this is related to SOLR-5211 in a sense

> Add deeply nested documents URP
> ---
>
> Key: SOLR-12441
> URL: https://issues.apache.org/jira/browse/SOLR-12441
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: mosh
>Assignee: David Smiley
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> As discussed in 
> [SOLR-12298|https://issues.apache.org/jira/browse/SOLR-12298], there ought to 
> be an URP to add metadata fields to childDocuments in order to allow a 
> transformer to rebuild the original document hierarchy.
> {quote}I propose we add the following fields:
>  # __nestParent__
>  # _nestLevel_
>  # __nestPath__
> __nestParent__: This field wild will store the document's parent docId, to be 
> used for building the whole hierarchy, using a new document transformer, as 
> suggested by Jan on the mailing list.
> _nestLevel_: This field will store the level of the specified field in the 
> document, using an int value. This field can be used for the parentFilter, 
> eliminating the need to provide a parentFilter, which will be set by default 
> as "_level_:queriedFieldLevel".
> _nestLevel_: This field will contain the full path, separated by a specific 
> reserved char e.g., '.'
>  for example: "first.second.third".
>  This will enable users to search for a specific path, or provide a regular 
> expression to search for fields sharing the same name in different levels of 
> the document, filtering using the level key if needed.
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-12516) JSON "range" facets don't refine sub-facets under special buckets (before,after,between)

2018-06-25 Thread Hoss Man (JIRA)

Hoss Man created SOLR-12516:
---

 Summary: JSON "range" facets don't refine sub-facets under special 
buckets (before,after,between)
 Key: SOLR-12516
 URL: https://issues.apache.org/jira/browse/SOLR-12516
 Project: Solr
  Issue Type: Bug
  Security Level: Public (Default Security Level. Issues are Public)
  Components: Facet Module
Reporter: Hoss Man


{{FacetRangeMerger extends FacetRequestSortedMerger}} ... however 
{{FacetRangeMerger}} does not override {{getRefinement(...)}} which means only 
{{FacetRequestSortedMerger.buckets}} is evaluated and considered for 
refinement. The additional, special purpose, {{FacetBucket}} instances tracked 
in {{FacetRangeMerger}} are never considered for refinement.

In a simple range facet this doesn't cause any problems because these buckets 
are returned by every shard on the phase#1 request -- but *if a sub-facet (such 
as a field facet) is nested under a range facet then the buckets returned by 
the sub-facets for the before/between/after buckets will never be refined* ... 
the phase#1 sub-facet buckets will be merged as.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12514) Rule-base Authorization plugin skips authorization if querying node does not have collection replica

2018-06-25 Thread Mahesh Kumar Vasanthu Somashekar (JIRA)



[ 
https://issues.apache.org/jira/browse/SOLR-12514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522548#comment-16522548
 ] 

Mahesh Kumar Vasanthu Somashekar commented on SOLR-12514:
-

Hi [~varunthacker]

Yes, I'm using Solr-7.3.1

 

Create another user without 'search-user' role, meaning user doesn't have 
access to 'test_collection'.

Run query using this user hitting all Solr nodes.

You would see 200 response from nodes which doesn't have replica (which is not 
expected) and 403 response from nodes which has replica (expected).

 

Thanks.

> Rule-base Authorization plugin skips authorization if querying node does not 
> have collection replica
> 
>
> Key: SOLR-12514
> URL: https://issues.apache.org/jira/browse/SOLR-12514
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: security
>Affects Versions: 7.3.1
>Reporter: Mahesh Kumar Vasanthu Somashekar
>Priority: Major
> Attachments: Screen Shot 2018-06-24 at 9.36.45 PM.png, security.json
>
>
> Solr serves client requests going throught 3 steps - init(), authorize() and 
> handle-request ([link 
> git-link|https://github.com/apache/lucene-solr/blob/releases/lucene-solr/7.3.1/solr/core/src/java/org/apache/solr/servlet/HttpSolrCall.java#L471]).
>  init() initializes all required information to be used by authorize(). 
> init() skips initializing if request is to be served remotely, which leads to 
> skipping authorization step ([link 
> git-link|https://github.com/apache/lucene-solr/blob/releases/lucene-solr/7.3.1/solr/core/src/java/org/apache/solr/servlet/HttpSolrCall.java#L291]).
>  init() relies on 'cores' object which only has information of local node 
> (which is perfect as per design). It should actually be getting security 
> information (security.json) from zookeeper, which has global view of the 
> cluster.
>  
> Example:
>  SolrCloud setup consists of 2 nodes (solr-7.3.1):
>  live_nodes: [
>  "localhost:8983_solr",
>  "localhost:8984_solr",
>  ]
> Two collections are created - 'collection-rf-1' with RF=1 and 
> 'collection-rf-2' with RF=2.
> Two users are created - 'collection-rf-1-user' and 'collection-rf-2-user'.
> Security configuration is as below (security.json attached):
>  "authorization":{
>  "class":"solr.RuleBasedAuthorizationPlugin",
>  "permissions":[
> { "name":"read", "collection":"collection-rf-2", "role":"collection-rf-2", 
> "index":1}
> ,
> { "name":"read", "collection":"collection-rf-1", "role":"collection-rf-1", 
> "index":2}
> ,
> { "name":"read", "role":"*", "index":3}
> ,
>  ...
>  "user-role":
> { "collection-rf-1-user":[ "collection-rf-1"], "collection-rf-2-user":[ 
> "collection-rf-2"]}
> ,
>  ...
>  
> Basically, its setup to that 'collection-rf-1-user' user can only access 
> 'collection-rf-1' collection and 'collection-rf-2-user' user can only access 
> 'collection-rf-2' collection.
> Also note that 'collection-rf-1' collection replica is only on 
> 'localhost:8983_solr' node, whereas ''collection-rf-2' collection replica is 
> on both live nodes.
>  
> Authorization does not work as expected for 'collection-rf-1' collection:
> $ curl -u collection-rf-2-user:password 
> 'http://*localhost:8983*/solr/collection-rf-1/select?q=*:*'
>  
>  
>  
>  Error 403 Unauthorized request, Response code: 403
>  
>  HTTP ERROR 403
>  Problem accessing /solr/collection-rf-1/select. Reason:
>   Unauthorized request, Response code: 403
>  
>  
> $ curl -u collection-rf-2-user:password 
> 'http://*localhost:8984*/solr/collection-rf-1/select?q=*:*'
>  {
>  "responseHeader":{
>  "zkConnected":true,
>  "status":0,
>  "QTime":0,
>  "params":{
>  "q":"*:*"}},
>  "response":{"numFound":0,"start":0,"docs":[]
>  }}
>  
> Whereas authorization works perfectly for 'collection-rf-2' collection (as 
> both nodes have replica):
> $ curl -u collection-rf-1-user:password 
> 'http://*localhost:8984*/solr/collection-rf-2/select?q=*:*'
>  
>  
>  
>  Error 403 Unauthorized request, Response code: 403
>  
>  HTTP ERROR 403
>  Problem accessing /solr/collection-rf-2/select. Reason:
>   Unauthorized request, Response code: 403
>  
>  
> $ curl -u collection-rf-1-user:password 
> 'http://*localhost:8983*/solr/collection-rf-2/select?q=*:*'
>  
>  
>  
>  Error 403 Unauthorized request, Response code: 403
>  
>  HTTP ERROR 403
>  Problem accessing /solr/collection-rf-2/select. Reason:
>   Unauthorized request, Response code: 403
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] lucene-solr pull request #410: SOLR-12441: add deeply nested URP for nested ...

2018-06-25 Thread dsmiley

Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/410#discussion_r197869098
  
--- Diff: solr/core/src/java/org/apache/solr/schema/IndexSchema.java ---
@@ -104,9 +104,12 @@
   public static final String FIELD_TYPE = "fieldType";
   public static final String FIELD_TYPES = FIELD_TYPE + "s";
   public static final String INTERNAL_POLY_FIELD_PREFIX = "*" + 
FieldType.POLY_FIELD_SEPARATOR;
+  public static final String LEVEL_FIELD_NAME = "_nestLevel_";
--- End diff --

I think these should nest ones should start with "NEST_"


---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] lucene-solr pull request #410: SOLR-12441: add deeply nested URP for nested ...

2018-06-25 Thread dsmiley

Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/410#discussion_r197857625
  
--- Diff: 
solr/core/src/java/org/apache/solr/update/processor/DeeplyNestedUpdateProcessor.java
 ---
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.update.processor;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.EnumSet;
+import java.util.Locale;
+import java.util.Objects;
+
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.SolrInputField;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.update.AddUpdateCommand;
+import static 
org.apache.solr.update.processor.DeeplyNestedUpdateProcessorFactory.NestedFlag;
+
+public class DeeplyNestedUpdateProcessor extends UpdateRequestProcessor {
+  private EnumSet fields;
+  SolrQueryRequest req;
+
+
+  protected DeeplyNestedUpdateProcessor(SolrQueryRequest req, 
SolrQueryResponse rsp, EnumSet fields, UpdateRequestProcessor next) 
{
+super(next);
+this.req = req;
+this.fields = fields;
+  }
+
+  @Override
+  public void processAdd(AddUpdateCommand cmd) throws IOException {
+SolrInputDocument doc = cmd.getSolrInputDocument();
+processDocChildren(doc, null);
+super.processAdd(cmd);
+  }
+
+  private void processDocChildren(SolrInputDocument doc, String fullPath) {
+for(SolrInputField field: doc.values()) {
+  if(field.getFirstValue() instanceof SolrInputDocument) {
+Object val = field.getValue();
+fullPath = Objects.isNull(fullPath) ? field.getName(): 
String.format(Locale.ROOT,"%s.%s", fullPath, field.getName());
+if (val instanceof Collection) {
--- End diff --

SolrInputField is directly Iterable over its values, so you needn't call 
getValue and check its type.


---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] lucene-solr pull request #410: SOLR-12441: add deeply nested URP for nested ...

2018-06-25 Thread dsmiley

Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/410#discussion_r197865187
  
--- Diff: 
solr/core/src/java/org/apache/solr/update/processor/DeeplyNestedUpdateProcessor.java
 ---
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.update.processor;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.EnumSet;
+import java.util.Locale;
+import java.util.Objects;
+
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.SolrInputField;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.update.AddUpdateCommand;
+import static 
org.apache.solr.update.processor.DeeplyNestedUpdateProcessorFactory.NestedFlag;
+
+public class DeeplyNestedUpdateProcessor extends UpdateRequestProcessor {
+  private EnumSet fields;
+  SolrQueryRequest req;
+
+
+  protected DeeplyNestedUpdateProcessor(SolrQueryRequest req, 
SolrQueryResponse rsp, EnumSet fields, UpdateRequestProcessor next) 
{
+super(next);
+this.req = req;
+this.fields = fields;
+  }
+
+  @Override
+  public void processAdd(AddUpdateCommand cmd) throws IOException {
+SolrInputDocument doc = cmd.getSolrInputDocument();
+processDocChildren(doc, null);
+super.processAdd(cmd);
+  }
+
+  private void processDocChildren(SolrInputDocument doc, String fullPath) {
+for(SolrInputField field: doc.values()) {
+  if(field.getFirstValue() instanceof SolrInputDocument) {
+Object val = field.getValue();
+fullPath = Objects.isNull(fullPath) ? field.getName(): 
String.format(Locale.ROOT,"%s.%s", fullPath, field.getName());
+if (val instanceof Collection) {
+  for(Object childDoc: (Collection) val) {
+if(childDoc instanceof SolrInputDocument) {
+  processChildDoc((SolrInputDocument) childDoc, doc, fullPath);
+}
+  }
+} else {
+  processChildDoc((SolrInputDocument) val, doc, fullPath);
+}
+  }
+}
+  }
+
+  private void processChildDoc(SolrInputDocument sdoc, SolrInputDocument 
parent, String fullPath) {
+if(fields == NestedFlag.ALL) {
+  setPathField(sdoc, fullPath);
+  setParentKey(sdoc, parent);
+  setLevelKey(sdoc, fullPath);
+} else {
+  if(fields.contains(NestedFlag.PATH)) {
+setPathField(sdoc, fullPath);
+  }
+  if (fields.contains(NestedFlag.PARENT)) {
+setParentKey(sdoc, parent);
+  }
+  if(fields.contains(NestedFlag.LEVEL)) {
+setLevelKey(sdoc, fullPath);
+  }
+}
+processDocChildren(sdoc, fullPath);
+  }
+
+  private void setLevelKey(SolrInputDocument sdoc, String fullPath) {
+sdoc.addField(IndexSchema.LEVEL_FIELD_NAME, 
fullPath.split("\\.").length);
+  }
+
+  private void setParentKey(SolrInputDocument sdoc, SolrInputDocument 
parent) {
+sdoc.addField(IndexSchema.PARENT_FIELD_NAME, 
parent.getFieldValue(req.getSchema().getUniqueKeyField().getName()));
--- End diff --

For these "set" methods, I think we want to be calling *setField* and not 
*addField*; no?  Granted it'd be weird if there was an existing value... and if 
there was, maybe we shouldn't overwrite it?  Probably shouldn't.


---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] lucene-solr pull request #410: SOLR-12441: add deeply nested URP for nested ...

2018-06-25 Thread dsmiley

Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/410#discussion_r197866818
  
--- Diff: 
solr/core/src/java/org/apache/solr/update/processor/DeeplyNestedUpdateProcessorFactory.java
 ---
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.update.processor;
+
+import java.util.EnumSet;
+import java.util.List;
+import java.util.Locale;
+import java.util.stream.Collectors;
+
+import org.apache.solr.common.SolrException;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.response.SolrQueryResponse;
+
+import static org.apache.solr.common.SolrException.ErrorCode.SERVER_ERROR;
+
+public class DeeplyNestedUpdateProcessorFactory extends 
UpdateRequestProcessorFactory {
+
+  private EnumSet fields;
+  private static final List allowedConfFields = 
NestedFlag.ALL.stream().map(e -> 
e.toString().toLowerCase(Locale.ROOT)).collect(Collectors.toList());
+
+  public UpdateRequestProcessor getInstance(SolrQueryRequest req, 
SolrQueryResponse rsp, UpdateRequestProcessor next ) {
+return new DeeplyNestedUpdateProcessor(req, rsp, fields, next);
+  }
+
+  @Override
+  public void init( NamedList args )
+  {
+Object tmp = args.remove("fields");
+if (null == tmp) {
+  throw new SolrException(SERVER_ERROR,
+  "'versionField' must be configured");
+}
+if (! (tmp instanceof String) ) {
+  throw new SolrException(SERVER_ERROR,
+  "'versionField' must be configured as a ");
+}
+List fields = StrUtils.splitSmart((String)tmp, ',');
+if(!allowedConfFields.containsAll(fields)) {
+  throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, 
"Deeply Nested URP may only contain: " + StringUtils.join(allowedConfFields, ", 
") +
+  " got: " + StringUtils.join(fields, ", ") + " instead");
+}
+this.fields = fields.size() == NestedFlag.values().length ? 
NestedFlag.ALL: generateNestedFlags(fields);
+  }
+
+  private static EnumSet generateNestedFlags(List 
fields) {
+return EnumSet.copyOf(fields.stream().map(e -> 
NestedFlag.valueOf(e.toUpperCase(Locale.ROOT))).collect(Collectors.toList()));
+  }
+
+  public enum NestedFlag {
+PATH,
+PARENT,
+LEVEL;
+
+public static final EnumSet ALL = 
EnumSet.allOf(NestedFlag.class);
--- End diff --

Nice


---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] lucene-solr pull request #410: SOLR-12441: add deeply nested URP for nested ...

2018-06-25 Thread dsmiley

Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/410#discussion_r197859682
  
--- Diff: 
solr/core/src/java/org/apache/solr/update/processor/DeeplyNestedUpdateProcessor.java
 ---
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.update.processor;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.EnumSet;
+import java.util.Locale;
+import java.util.Objects;
+
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.SolrInputField;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.update.AddUpdateCommand;
+import static 
org.apache.solr.update.processor.DeeplyNestedUpdateProcessorFactory.NestedFlag;
+
+public class DeeplyNestedUpdateProcessor extends UpdateRequestProcessor {
+  private EnumSet fields;
+  SolrQueryRequest req;
+
+
+  protected DeeplyNestedUpdateProcessor(SolrQueryRequest req, 
SolrQueryResponse rsp, EnumSet fields, UpdateRequestProcessor next) 
{
+super(next);
+this.req = req;
+this.fields = fields;
+  }
+
+  @Override
+  public void processAdd(AddUpdateCommand cmd) throws IOException {
+SolrInputDocument doc = cmd.getSolrInputDocument();
+processDocChildren(doc, null);
+super.processAdd(cmd);
+  }
+
+  private void processDocChildren(SolrInputDocument doc, String fullPath) {
+for(SolrInputField field: doc.values()) {
+  if(field.getFirstValue() instanceof SolrInputDocument) {
+Object val = field.getValue();
+fullPath = Objects.isNull(fullPath) ? field.getName(): 
String.format(Locale.ROOT,"%s.%s", fullPath, field.getName());
--- End diff --

I can't say I'm a fan of String.format -- it usually is just a more verbose 
& complex way to do trivial string concatenation.  Granted String.format 
_sometimes_ has value (e.g. formatting floating point); it isn't exercised here.

Shouldn't we throw a BAD_REQUEST exception if field.getName() contains the 
character we're using to join ('.')?

There's an O(N^2) string concatenation here where N is the depth of the 
tree but hopefully the docs won't be *that* nested to worry about this.  We 
could add a comment to at least acknowledge this.


---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] lucene-solr pull request #410: SOLR-12441: add deeply nested URP for nested ...

2018-06-25 Thread dsmiley

Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/410#discussion_r197862734
  
--- Diff: 
solr/core/src/java/org/apache/solr/update/processor/DeeplyNestedUpdateProcessor.java
 ---
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.update.processor;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.EnumSet;
+import java.util.Locale;
+import java.util.Objects;
+
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.SolrInputField;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.update.AddUpdateCommand;
+import static 
org.apache.solr.update.processor.DeeplyNestedUpdateProcessorFactory.NestedFlag;
+
+public class DeeplyNestedUpdateProcessor extends UpdateRequestProcessor {
+  private EnumSet fields;
+  SolrQueryRequest req;
+
+
+  protected DeeplyNestedUpdateProcessor(SolrQueryRequest req, 
SolrQueryResponse rsp, EnumSet fields, UpdateRequestProcessor next) 
{
+super(next);
+this.req = req;
+this.fields = fields;
+  }
+
+  @Override
+  public void processAdd(AddUpdateCommand cmd) throws IOException {
+SolrInputDocument doc = cmd.getSolrInputDocument();
+processDocChildren(doc, null);
+super.processAdd(cmd);
+  }
+
+  private void processDocChildren(SolrInputDocument doc, String fullPath) {
+for(SolrInputField field: doc.values()) {
+  if(field.getFirstValue() instanceof SolrInputDocument) {
+Object val = field.getValue();
+fullPath = Objects.isNull(fullPath) ? field.getName(): 
String.format(Locale.ROOT,"%s.%s", fullPath, field.getName());
+if (val instanceof Collection) {
+  for(Object childDoc: (Collection) val) {
+if(childDoc instanceof SolrInputDocument) {
+  processChildDoc((SolrInputDocument) childDoc, doc, fullPath);
+}
+  }
+} else {
+  processChildDoc((SolrInputDocument) val, doc, fullPath);
+}
+  }
+}
+  }
+
+  private void processChildDoc(SolrInputDocument sdoc, SolrInputDocument 
parent, String fullPath) {
+if(fields == NestedFlag.ALL) {
+  setPathField(sdoc, fullPath);
+  setParentKey(sdoc, parent);
+  setLevelKey(sdoc, fullPath);
+} else {
+  if(fields.contains(NestedFlag.PATH)) {
+setPathField(sdoc, fullPath);
+  }
+  if (fields.contains(NestedFlag.PARENT)) {
+setParentKey(sdoc, parent);
+  }
+  if(fields.contains(NestedFlag.LEVEL)) {
+setLevelKey(sdoc, fullPath);
+  }
+}
+processDocChildren(sdoc, fullPath);
+  }
+
+  private void setLevelKey(SolrInputDocument sdoc, String fullPath) {
+sdoc.addField(IndexSchema.LEVEL_FIELD_NAME, 
fullPath.split("\\.").length);
--- End diff --

yuck; sorry.  Perhaps instead of pre-concatenating the full path, we 
instead pass full path around as a List to the various arguments and 
then we only materialize it in setPathField?  This may also ameliorate the 
O(N^2) problem although we would be re-concatenating the earlier part of the 
string but that doesn't seem like a big deal.


---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] lucene-solr pull request #410: SOLR-12441: add deeply nested URP for nested ...

2018-06-25 Thread dsmiley

Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/410#discussion_r197861078
  
--- Diff: 
solr/core/src/java/org/apache/solr/update/processor/DeeplyNestedUpdateProcessor.java
 ---
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.update.processor;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.EnumSet;
+import java.util.Locale;
+import java.util.Objects;
+
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.SolrInputField;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.update.AddUpdateCommand;
+import static 
org.apache.solr.update.processor.DeeplyNestedUpdateProcessorFactory.NestedFlag;
+
+public class DeeplyNestedUpdateProcessor extends UpdateRequestProcessor {
+  private EnumSet fields;
+  SolrQueryRequest req;
+
+
+  protected DeeplyNestedUpdateProcessor(SolrQueryRequest req, 
SolrQueryResponse rsp, EnumSet fields, UpdateRequestProcessor next) 
{
+super(next);
+this.req = req;
+this.fields = fields;
+  }
+
+  @Override
+  public void processAdd(AddUpdateCommand cmd) throws IOException {
+SolrInputDocument doc = cmd.getSolrInputDocument();
+processDocChildren(doc, null);
+super.processAdd(cmd);
+  }
+
+  private void processDocChildren(SolrInputDocument doc, String fullPath) {
+for(SolrInputField field: doc.values()) {
+  if(field.getFirstValue() instanceof SolrInputDocument) {
+Object val = field.getValue();
+fullPath = Objects.isNull(fullPath) ? field.getName(): 
String.format(Locale.ROOT,"%s.%s", fullPath, field.getName());
+if (val instanceof Collection) {
+  for(Object childDoc: (Collection) val) {
+if(childDoc instanceof SolrInputDocument) {
+  processChildDoc((SolrInputDocument) childDoc, doc, fullPath);
+}
+  }
+} else {
+  processChildDoc((SolrInputDocument) val, doc, fullPath);
+}
+  }
+}
+  }
+
+  private void processChildDoc(SolrInputDocument sdoc, SolrInputDocument 
parent, String fullPath) {
+if(fields == NestedFlag.ALL) {
--- End diff --

I don't think we need special-case ALL


---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] lucene-solr pull request #410: SOLR-12441: add deeply nested URP for nested ...

2018-06-25 Thread dsmiley

Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/410#discussion_r197868389
  
--- Diff: 
solr/core/src/java/org/apache/solr/update/processor/DeeplyNestedUpdateProcessor.java
 ---
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.update.processor;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.EnumSet;
+import java.util.Locale;
+import java.util.Objects;
+
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.SolrInputField;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.update.AddUpdateCommand;
+import static 
org.apache.solr.update.processor.DeeplyNestedUpdateProcessorFactory.NestedFlag;
+
+public class DeeplyNestedUpdateProcessor extends UpdateRequestProcessor {
--- End diff --

I wonder if we want to call this as-such.  Can this be used if the nesting 
is "shallow"?  Perhaps remove the "Deeply" part.  Perhaps this URP should 
handle "_root_" as well to thus unify where nested child doc fields are added?  
Although that raises other questions.  With 8.0 on the horizon this fall, we 
can think of the best solution that isn't necessarily fully backwards 
compatible.  This conversation should probably be taken to the JIRA as it's a 
bigger question and not a code detail.


---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12441) Add deeply nested documents URP

2018-06-25 Thread David Smiley (JIRA)



[ 
https://issues.apache.org/jira/browse/SOLR-12441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522513#comment-16522513
 ] 

David Smiley commented on SOLR-12441:
-

CC [~mkhludnev] and [~anshum] as you may be interested in this issue (plus 
related ones on the parent issue) based on your more extensive experience with 
(deeply) nested documents.

> Add deeply nested documents URP
> ---
>
> Key: SOLR-12441
> URL: https://issues.apache.org/jira/browse/SOLR-12441
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: mosh
>Assignee: David Smiley
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> As discussed in 
> [SOLR-12298|https://issues.apache.org/jira/browse/SOLR-12298], there ought to 
> be an URP to add metadata fields to childDocuments in order to allow a 
> transformer to rebuild the original document hierarchy.
> {quote}I propose we add the following fields:
>  # __nestParent__
>  # _nestLevel_
>  # __nestPath__
> __nestParent__: This field wild will store the document's parent docId, to be 
> used for building the whole hierarchy, using a new document transformer, as 
> suggested by Jan on the mailing list.
> _nestLevel_: This field will store the level of the specified field in the 
> document, using an int value. This field can be used for the parentFilter, 
> eliminating the need to provide a parentFilter, which will be set by default 
> as "_level_:queriedFieldLevel".
> _nestLevel_: This field will contain the full path, separated by a specific 
> reserved char e.g., '.'
>  for example: "first.second.third".
>  This will enable users to search for a specific path, or provide a regular 
> expression to search for fields sharing the same name in different levels of 
> the document, filtering using the level key if needed.
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [DISCUSS] Geo/spatial organization in Lucene

2018-06-25 Thread Nicholas Knize

+1

On Mon, Jun 25, 2018, 11:32 AM Ignacio Vera Sequeiros  wrote:

> +1
> --
> *From:* Alan Woodward 
> *Sent:* Monday, June 25, 2018 5:56:16 PM
> *To:* dev@lucene.apache.org
>
> *Subject:* Re: [DISCUSS] Geo/spatial organization in Lucene
> +1 to move LatLonPoint and friends to core, and nuke the spatial module
>
> On 25 Jun 2018, at 16:32, David Smiley  wrote:
>
> Okay fine, I'm not going to block spatial stuff going into core.
>  (celebration).  I foresee the spatial stuff there growing beyond the one
> default impl though.
>
> Perhaps most of us are still not happy with seeing spatial code across so
> many modules?  Nick and I have voiced this concern so far.  Given the
> pittance of utility of what's in the spatial module today, can we agree to
> simply remove it?
>
> I pity users trying to figure out what is where to make sense of it.  I
> wonder how new users discover/browse to look around -- I'm too used to the
> codebase to have any idea what newbies do.  That seems to be this:
> http://lucene.apache.org/core/7_3_1/index.html  Each module only gets one
> terse sentence fragment.  It'd be nice to have potentially a paragraph of
> information?  Even without more verbage, the spatial ones could have better
> descriptions.  I propose these changes:
>
> * spatial:  remove it :-)   -- see above
> * spatial3d: Computational geometry on the surface of a sphere or
> ellipsoid, including Lucene index & search solutions
> * spatial-extras: Spatial code that has external dependencies like
> Spatial4j and JTS, including Lucene index & search solutions
>
> perhaps "spatial-sphere" might be a more meaningful name than spatial3d?
> Yes, it's ellipsoidal but sphere is close enough ;-)
>
> ~ David
>
> On Mon, Jun 25, 2018 at 10:42 AM Michael McCandless <
> luc...@mikemccandless.com> wrote:
>
>> I also favor B: move the common case, good performing spatial
>> implementations to core, but still bake new things in sandbox.  LatLonPoint
>> has baked way too long already!  The addition of first class (codec
>> support) KD trees in Lucene (dimensional points) was/is really a game
>> changer for Lucene supporting common geo spatial applications.
>>
>> It would be nice to find a better name than geo3d / spatial3d: it
>> confuses 100% of the people I explain it to, on first impression :)  But we
>> should tackle that separately/later.
>>
>> Merging the 2D/3D abstractions sounds a little too ambitious at this
>> point, so I think it's fine to leave them separate for now.
>>
>> Mike McCandless
>>
>> http://blog.mikemccandless.com
>>
>> On Wed, Jun 20, 2018 at 1:00 PM, Nicholas Knize  wrote:
>>
>>> If I were to pick between the two, I also have a preference for B.  I've
>>> also tried to keep this whole spatial organization rather simple:
>>>
>>> core - simple spatial capabilities needed by the 99% spatial use case
>>> (e.g., web mapping). Includes LatLonPoint, polygon & distance search
>>> (everything currently in sandbox). Lightweight, and no dependencies or
>>> complexities. If one wants simple and fast point search, all you need is
>>> the core module.
>>>
>>> spatial - dependency free. Expands on core spatial to include simple
>>> shape searching. Uses internal relations. Everything confined to core and
>>> spatial modules.
>>>
>>> spatial-extras - expanded spatial capabilities. Welcomes third-party
>>> dependencies (e.g., S3, SIS, Proj4J). Targets more advanced/expert GIS
>>> use-cases.
>>>
>>> geo3d - trades speed for accuracy. I've always struggled with the name,
>>> since it implies 3D shapes/point cloud support. But history has shown
>>> considering a name change to be a bike-shedding endeavor.
>>>
>>> At the end of the day I'm up for whatever makes most sense for everyone
>>> here. Lord knows we could use more people helping out on geo.
>>>
>>> - Nick
>>>
>>>
>>>
>>> On Wed, Jun 20, 2018 at 11:40 AM Adrien Grand  wrote:
>>>
 I have a slight preference for B similarly to how StandardAnalyzer is
 in core and other analyzers are in analysis, but no strong feelings. In any
 case I agree that both A and B would be much better than the current
 situation.


 Le mer. 20 juin 2018 à 18:09, David Smiley 
 a écrit :

> I think everyone agrees the current state of spatial code organization
> in Lucene is not desirable.  We have a spatial module that has almost
> nothing in it, we have mature spatial code in the sandbox that needs to
> "graduate" somewhere, and we've got a handful of geo utilities in Lucene
> core (mostly because I didn't notice).  No agreement has been reached on
> what the desired state should be.
>
> I'd like to hear opinions on this from members of the community.  I am
> especially interested in listening to people that normally don't seem to
> speak up about spatial matters. Perhaps Uwe Schindlerand Alan Woodward – I
> respect both of you guys a ton for your tenure with Lucene and aren't too
>

[jira] [Commented] (SOLR-12441) Add deeply nested documents URP

2018-06-25 Thread David Smiley (JIRA)



[ 
https://issues.apache.org/jira/browse/SOLR-12441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522510#comment-16522510
 ] 

David Smiley commented on SOLR-12441:
-

Can you please provide further justification for \_nestLevel\_ like a more 
fleshed out scenario?  It seems to only be a matter of query convenience based 
on your description (i.e. there are other ways to accomplish the same query), 
and we're paying index weight for it (albeit small) so I don't quite see if 
it's worth it.

> Add deeply nested documents URP
> ---
>
> Key: SOLR-12441
> URL: https://issues.apache.org/jira/browse/SOLR-12441
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: mosh
>Assignee: David Smiley
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> As discussed in 
> [SOLR-12298|https://issues.apache.org/jira/browse/SOLR-12298], there ought to 
> be an URP to add metadata fields to childDocuments in order to allow a 
> transformer to rebuild the original document hierarchy.
> {quote}I propose we add the following fields:
>  # __nestParent__
>  # _nestLevel_
>  # __nestPath__
> __nestParent__: This field wild will store the document's parent docId, to be 
> used for building the whole hierarchy, using a new document transformer, as 
> suggested by Jan on the mailing list.
> _nestLevel_: This field will store the level of the specified field in the 
> document, using an int value. This field can be used for the parentFilter, 
> eliminating the need to provide a parentFilter, which will be set by default 
> as "_level_:queriedFieldLevel".
> _nestLevel_: This field will contain the full path, separated by a specific 
> reserved char e.g., '.'
>  for example: "first.second.third".
>  This will enable users to search for a specific path, or provide a regular 
> expression to search for fields sharing the same name in different levels of 
> the document, filtering using the level key if needed.
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11933) DIH gui shouldn't have "clean" be checked by default

2018-06-25 Thread Eric Pugh (JIRA)



[ 
https://issues.apache.org/jira/browse/SOLR-11933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522507#comment-16522507
 ] 

Eric Pugh commented on SOLR-11933:
--

Thanks [~kshitij91] and [~dsmiley] for the +1's.   I created SOLR-12515 to 
incorporate the suggestion by [~elyograg].Love to get some feedback!

> DIH gui shouldn't have "clean" be checked by default
> 
>
> Key: SOLR-11933
> URL: https://issues.apache.org/jira/browse/SOLR-11933
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: contrib - DataImportHandler
>Affects Versions: 7.2
>Reporter: Eric Pugh
>Assignee: Tomás Fernández Löbbe
>Priority: Minor
> Fix For: 7.3, master (8.0)
>
> Attachments: fef1d06a2eb15a0fd36eb91124af413a19d95528.diff
>
>
> The DIH webapp by default has the "clean" checkbox enabled.   Clean is very 
> dangerous because you delete all the data first, and then load the data.   
> Making this the default choice is bad UX.  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-12515) DIH gui should be intelligent about the "clean" being checked option

2018-06-25 Thread Eric Pugh (JIRA)



 [ 
https://issues.apache.org/jira/browse/SOLR-12515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Pugh updated SOLR-12515:
-
Attachment: SOLR-12515.patch

> DIH gui should be intelligent about the "clean" being checked option
> 
>
> Key: SOLR-12515
> URL: https://issues.apache.org/jira/browse/SOLR-12515
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: contrib - DataImportHandler
>Affects Versions: 7.3.1
>Reporter: Eric Pugh
>Priority: Minor
> Attachments: SOLR-12515.patch
>
>
> The DIH webapp by default has the "clean" checkbox enabled.   Clean is very 
> dangerous because you delete all the data first, and then load the data.   
> A previous ticket, SOLR-11933 had some suggestions on how best to solve this 
> problem, specifically:
> bq. I think the default of the 'clean' checkbox in the UI should change 
> depending on the type of import selected. This is what happens when using the 
> API directly and the clean parameter is not present.
> bq. 
> bq. If full-import is selected, it should be on, if delta-import is selected, 
> it should be off. (with the option for the user to change the setting, of 
> course)
> This patch applies this logic change.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-12515) DIH gui should be intelligent about the "clean" being checked option

2018-06-25 Thread Eric Pugh (JIRA)

Eric Pugh created SOLR-12515:


 Summary: DIH gui should be intelligent about the "clean" being 
checked option
 Key: SOLR-12515
 URL: https://issues.apache.org/jira/browse/SOLR-12515
 Project: Solr
  Issue Type: Improvement
  Security Level: Public (Default Security Level. Issues are Public)
  Components: contrib - DataImportHandler
Affects Versions: 7.3.1
Reporter: Eric Pugh


The DIH webapp by default has the "clean" checkbox enabled.   Clean is very 
dangerous because you delete all the data first, and then load the data.   

A previous ticket, SOLR-11933 had some suggestions on how best to solve this 
problem, specifically:

bq. I think the default of the 'clean' checkbox in the UI should change 
depending on the type of import selected. This is what happens when using the 
API directly and the clean parameter is not present.
bq. 
bq. If full-import is selected, it should be on, if delta-import is selected, 
it should be off. (with the option for the user to change the setting, of 
course)

This patch applies this logic change.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [DISCUSS] Geo/spatial organization in Lucene

2018-06-25 Thread Ignacio Vera Sequeiros

+1

From: Alan Woodward 
Sent: Monday, June 25, 2018 5:56:16 PM
To: dev@lucene.apache.org
Subject: Re: [DISCUSS] Geo/spatial organization in Lucene

+1 to move LatLonPoint and friends to core, and nuke the spatial module

On 25 Jun 2018, at 16:32, David Smiley 
mailto:david.w.smi...@gmail.com>> wrote:

Okay fine, I'm not going to block spatial stuff going into core.  
(celebration).  I foresee the spatial stuff there growing beyond the one 
default impl though.

Perhaps most of us are still not happy with seeing spatial code across so many 
modules?  Nick and I have voiced this concern so far.  Given the pittance of 
utility of what's in the spatial module today, can we agree to simply remove it?

I pity users trying to figure out what is where to make sense of it.  I wonder 
how new users discover/browse to look around -- I'm too used to the codebase to 
have any idea what newbies do.  That seems to be this: 
http://lucene.apache.org/core/7_3_1/index.html  Each module only gets one terse 
sentence fragment.  It'd be nice to have potentially a paragraph of 
information?  Even without more verbage, the spatial ones could have better 
descriptions.  I propose these changes:

* spatial:  remove it :-)   -- see above
* spatial3d: Computational geometry on the surface of a sphere or ellipsoid, 
including Lucene index & search solutions
* spatial-extras: Spatial code that has external dependencies like Spatial4j 
and JTS, including Lucene index & search solutions

perhaps "spatial-sphere" might be a more meaningful name than spatial3d?  Yes, 
it's ellipsoidal but sphere is close enough ;-)

~ David

On Mon, Jun 25, 2018 at 10:42 AM Michael McCandless 
mailto:luc...@mikemccandless.com>> wrote:
I also favor B: move the common case, good performing spatial implementations 
to core, but still bake new things in sandbox.  LatLonPoint has baked way too 
long already!  The addition of first class (codec support) KD trees in Lucene 
(dimensional points) was/is really a game changer for Lucene supporting common 
geo spatial applications.

It would be nice to find a better name than geo3d / spatial3d: it confuses 100% 
of the people I explain it to, on first impression :)  But we should tackle 
that separately/later.

Merging the 2D/3D abstractions sounds a little too ambitious at this point, so 
I think it's fine to leave them separate for now.

Mike McCandless

http://blog.mikemccandless.com

On Wed, Jun 20, 2018 at 1:00 PM, Nicholas Knize 
mailto:nkn...@gmail.com>> wrote:
If I were to pick between the two, I also have a preference for B.  I've also 
tried to keep this whole spatial organization rather simple:

core - simple spatial capabilities needed by the 99% spatial use case (e.g., 
web mapping). Includes LatLonPoint, polygon & distance search (everything 
currently in sandbox). Lightweight, and no dependencies or complexities. If one 
wants simple and fast point search, all you need is the core module.

spatial - dependency free. Expands on core spatial to include simple shape 
searching. Uses internal relations. Everything confined to core and spatial 
modules.

spatial-extras - expanded spatial capabilities. Welcomes third-party 
dependencies (e.g., S3, SIS, Proj4J). Targets more advanced/expert GIS 
use-cases.

geo3d - trades speed for accuracy. I've always struggled with the name, since 
it implies 3D shapes/point cloud support. But history has shown considering a 
name change to be a bike-shedding endeavor.

At the end of the day I'm up for whatever makes most sense for everyone here. 
Lord knows we could use more people helping out on geo.

- Nick

On Wed, Jun 20, 2018 at 11:40 AM Adrien Grand 
mailto:jpou...@gmail.com>> wrote:
I have a slight preference for B similarly to how StandardAnalyzer is in core 
and other analyzers are in analysis, but no strong feelings. In any case I 
agree that both A and B would be much better than the current situation.

Le mer. 20 juin 2018 à 18:09, David Smiley 
mailto:david.w.smi...@gmail.com>> a écrit :
I think everyone agrees the current state of spatial code organization in 
Lucene is not desirable.  We have a spatial module that has almost nothing in 
it, we have mature spatial code in the sandbox that needs to "graduate" 
somewhere, and we've got a handful of geo utilities in Lucene core (mostly 
because I didn't notice).  No agreement has been reached on what the desired 
state should be.

I'd like to hear opinions on this from members of the community.  I am 
especially interested in listening to people that normally don't seem to speak 
up about spatial matters. Perhaps Uwe Schindlerand Alan Woodward – I respect 
both of you guys a ton for your tenure with Lucene and aren't too pushy with 
your opinions. I can be convinced to change my mind, especially if coming from 
you two.  Of course anyone can respond -- this is an open discussion!

As I understand it, there are two proposals loosely defined

[jira] [Commented] (SOLR-12513) Reproducing TestCodecSupport.testMixedCompressionMode failure

2018-06-25 Thread Erick Erickson (JIRA)



[ 
https://issues.apache.org/jira/browse/SOLR-12513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522489#comment-16522489
 ] 

Erick Erickson commented on SOLR-12513:
---

Thanks [~steve_rowe]

I regenerated the hints file, so the actual code changes are very small 
compared to the total patch size.

I have yet to run precommit and full tests on this one, but after I do I'll 
check this patch in.

> Reproducing TestCodecSupport.testMixedCompressionMode failure
> -
>
> Key: SOLR-12513
> URL: https://issues.apache.org/jira/browse/SOLR-12513
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Steve Rowe
>Assignee: Erick Erickson
>Priority: Major
> Attachments: SOLR-12513.patch
>
>
> From [https://jenkins.thetaphi.de/job/Lucene-Solr-master-Windows/7375/] 
> (reproduces for me on Linux):
> {noformat}
> Checking out Revision 3b9d3a760a432b97aad2c08b2f778fa2344eb14a 
> (refs/remotes/origin/master)
> [...]
>[junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestCodecSupport 
> -Dtests.method=testMixedCompressionMode -Dtests.seed=F584376A20EC9B2D 
> -Dtests.slow=true -Dtests.locale=ha-GH -Dtests.timezone=Africa/Nairobi 
> -Dtests.asserts=true -Dtests.file.encoding=UTF-8
>[junit4] FAILURE 0.59s J1 | TestCodecSupport.testMixedCompressionMode <<<
>[junit4]> Throwable #1: org.junit.ComparisonFailure: Expecting 
> compression mode string to be BEST_SPEED but got: BEST_COMPRESSION
>[junit4]>  SegmentInfo: _5(8.0.0):c1
>[junit4]>  SegmentInfos: segments_e: _7(8.0.0):c2 _5(8.0.0):c1
>[junit4]>  Codec: Lucene70 expected: but 
> was:
>[junit4]>  at 
> __randomizedtesting.SeedInfo.seed([F584376A20EC9B2D:2BF1400E658E6E9C]:0)
>[junit4]>  at 
> org.apache.solr.core.TestCodecSupport.lambda$assertCompressionMode$0(TestCodecSupport.java:115)
>[junit4]>  at 
> org.apache.solr.core.SolrCore.withSearcher(SolrCore.java:1874)
>[junit4]>  at 
> org.apache.solr.core.TestCodecSupport.assertCompressionMode(TestCodecSupport.java:112)
>[junit4]>  at 
> org.apache.solr.core.TestCodecSupport.testMixedCompressionMode(TestCodecSupport.java:157)
>[junit4]>  at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>[junit4]>  at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>[junit4]>  at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>[junit4]>  at 
> java.base/java.lang.reflect.Method.invoke(Method.java:564)
>[junit4]>  at java.base/java.lang.Thread.run(Thread.java:844)
> [...]
>[junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, 
> docValues:{}, maxPointsInLeafNode=1658, maxMBSortInHeap=7.737749859284375, 
> sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@2da9a1c5),
>  locale=ha-GH, timezone=Africa/Nairobi
>[junit4]   2> NOTE: Windows 10 10.0 amd64/Oracle Corporation 10.0.1 
> (64-bit)/cpus=3,threads=1,free=39067800,total=97320960
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-12513) Reproducing TestCodecSupport.testMixedCompressionMode failure

2018-06-25 Thread Erick Erickson (JIRA)



 [ 
https://issues.apache.org/jira/browse/SOLR-12513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Erick Erickson updated SOLR-12513:
--
Attachment: SOLR-12513.patch

> Reproducing TestCodecSupport.testMixedCompressionMode failure
> -
>
> Key: SOLR-12513
> URL: https://issues.apache.org/jira/browse/SOLR-12513
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Steve Rowe
>Assignee: Erick Erickson
>Priority: Major
> Attachments: SOLR-12513.patch
>
>
> From [https://jenkins.thetaphi.de/job/Lucene-Solr-master-Windows/7375/] 
> (reproduces for me on Linux):
> {noformat}
> Checking out Revision 3b9d3a760a432b97aad2c08b2f778fa2344eb14a 
> (refs/remotes/origin/master)
> [...]
>[junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestCodecSupport 
> -Dtests.method=testMixedCompressionMode -Dtests.seed=F584376A20EC9B2D 
> -Dtests.slow=true -Dtests.locale=ha-GH -Dtests.timezone=Africa/Nairobi 
> -Dtests.asserts=true -Dtests.file.encoding=UTF-8
>[junit4] FAILURE 0.59s J1 | TestCodecSupport.testMixedCompressionMode <<<
>[junit4]> Throwable #1: org.junit.ComparisonFailure: Expecting 
> compression mode string to be BEST_SPEED but got: BEST_COMPRESSION
>[junit4]>  SegmentInfo: _5(8.0.0):c1
>[junit4]>  SegmentInfos: segments_e: _7(8.0.0):c2 _5(8.0.0):c1
>[junit4]>  Codec: Lucene70 expected: but 
> was:
>[junit4]>  at 
> __randomizedtesting.SeedInfo.seed([F584376A20EC9B2D:2BF1400E658E6E9C]:0)
>[junit4]>  at 
> org.apache.solr.core.TestCodecSupport.lambda$assertCompressionMode$0(TestCodecSupport.java:115)
>[junit4]>  at 
> org.apache.solr.core.SolrCore.withSearcher(SolrCore.java:1874)
>[junit4]>  at 
> org.apache.solr.core.TestCodecSupport.assertCompressionMode(TestCodecSupport.java:112)
>[junit4]>  at 
> org.apache.solr.core.TestCodecSupport.testMixedCompressionMode(TestCodecSupport.java:157)
>[junit4]>  at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>[junit4]>  at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>[junit4]>  at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>[junit4]>  at 
> java.base/java.lang.reflect.Method.invoke(Method.java:564)
>[junit4]>  at java.base/java.lang.Thread.run(Thread.java:844)
> [...]
>[junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, 
> docValues:{}, maxPointsInLeafNode=1658, maxMBSortInHeap=7.737749859284375, 
> sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@2da9a1c5),
>  locale=ha-GH, timezone=Africa/Nairobi
>[junit4]   2> NOTE: Windows 10 10.0 amd64/Oracle Corporation 10.0.1 
> (64-bit)/cpus=3,threads=1,free=39067800,total=97320960
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-11216) Race condition in peerSync

2018-06-25 Thread Steve Rowe (JIRA)



[ 
https://issues.apache.org/jira/browse/SOLR-11216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522479#comment-16522479
 ] 

Steve Rowe edited comment on SOLR-11216 at 6/25/18 4:19 PM:


Policeman Jenkins found a reproducing seed for a 
{{TestStressCloudBlindAtomicUpdates -Dtests.method.test_dv_idx()}} failure that 
{{git bisect}} blames on commit {{412116f}} on this issue 
[https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/2190/]:

{noformat}
Checking out Revision 82b793df56c8c9fb50c29f46f39465453a87f2b2 
(refs/remotes/origin/branch_7x)
[...]
   [junit4]   2> NOTE: reproduce with: ant test  
-Dtestcase=TestStressCloudBlindAtomicUpdates -Dtests.method=test_dv_idx 
-Dtests.seed=9E9025C836D166FA -Dtests.multiplier=3 -Dtests.slow=true 
-Dtests.locale=ga -Dtests.timezone=Europe/Monaco -Dtests.asserts=true 
-Dtests.file.encoding=UTF-8
   [junit4] FAILURE 11.4s J1 | TestStressCloudBlindAtomicUpdates.test_dv_idx <<<
   [junit4]> Throwable #1: java.lang.AssertionError: Some docs had errors 
-- check logs expected:<0> but was:<7>
   [junit4]>at 
__randomizedtesting.SeedInfo.seed([9E9025C836D166FA:BEC5000EA0DCB1B]:0)
   [junit4]>at 
org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.checkField(TestStressCloudBlindAtomicUpdates.java:342)
   [junit4]>at 
org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.test_dv_idx(TestStressCloudBlindAtomicUpdates.java:231)
   [junit4]>at java.lang.Thread.run(Thread.java:748)
[...]
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70), 
sim=RandomSimilarity(queryNorm=true): {}, locale=ga, timezone=Europe/Monaco
   [junit4]   2> NOTE: Linux 4.13.0-41-generic i386/Oracle Corporation 
1.8.0_172 (32-bit)/cpus=8,threads=1,free=238679536,total=536870912
{noformat}


was (Author: steve_rowe):
Policeman Jenkins found a reproducing seed for a {}} failure that {{git 
bisect}} blames on commit {{412116f}} on this issue 
[https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/2190/]:

{noformat}
Checking out Revision 82b793df56c8c9fb50c29f46f39465453a87f2b2 
(refs/remotes/origin/branch_7x)
[...]
   [junit4]   2> NOTE: reproduce with: ant test  
-Dtestcase=TestStressCloudBlindAtomicUpdates -Dtests.method=test_dv_idx 
-Dtests.seed=9E9025C836D166FA -Dtests.multiplier=3 -Dtests.slow=true 
-Dtests.locale=ga -Dtests.timezone=Europe/Monaco -Dtests.asserts=true 
-Dtests.file.encoding=UTF-8
   [junit4] FAILURE 11.4s J1 | TestStressCloudBlindAtomicUpdates.test_dv_idx <<<
   [junit4]> Throwable #1: java.lang.AssertionError: Some docs had errors 
-- check logs expected:<0> but was:<7>
   [junit4]>at 
__randomizedtesting.SeedInfo.seed([9E9025C836D166FA:BEC5000EA0DCB1B]:0)
   [junit4]>at 
org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.checkField(TestStressCloudBlindAtomicUpdates.java:342)
   [junit4]>at 
org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.test_dv_idx(TestStressCloudBlindAtomicUpdates.java:231)
   [junit4]>at java.lang.Thread.run(Thread.java:748)
[...]
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70), 
sim=RandomSimilarity(queryNorm=true): {}, locale=ga, timezone=Europe/Monaco
   [junit4]   2> NOTE: Linux 4.13.0-41-generic i386/Oracle Corporation 
1.8.0_172 (32-bit)/cpus=8,threads=1,free=238679536,total=536870912
{noformat}

> Race condition in peerSync
> --
>
> Key: SOLR-11216
> URL: https://issues.apache.org/jira/browse/SOLR-11216
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 7.4
>Reporter: Cao Manh Dat
>Assignee: Cao Manh Dat
>Priority: Major
> Fix For: master (8.0), 7.5
>
> Attachments: SOLR-11216.patch, SOLR-11216.patch, SOLR-11216.patch, 
> SOLR-11216.patch, SOLR-11216.patch, SOLR-11216.patch, SOLR-11216.patch
>
>
> When digging into SOLR-10126. I found a case that can make peerSync fail.
> * leader and replica receive update from 1 to 4
> * replica stop
> * replica miss updates 5, 6
> * replica start recovery
> ## replica buffer updates 7, 8
> ## replica request versions from leader, 
> ## in the same time leader receive update 9, so it will return updates from 1 
> to 9 (for request versions) when replica get recent versions ( so it will be 
> 1,2,3,4,5,6,7,8,9 )
> ## replica do peersync and request updates 5, 6, 9 from leader 
> ## replica apply updates 5, 6, 9. Its index does not have update 7, 8 and 
> maxVersionSpecified for fingerprint is 9, therefore compare fingerprint will 
> fail



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail:

[jira] [Comment Edited] (SOLR-11216) Race condition in peerSync

2018-06-25 Thread Steve Rowe (JIRA)



[ 
https://issues.apache.org/jira/browse/SOLR-11216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522479#comment-16522479
 ] 

Steve Rowe edited comment on SOLR-11216 at 6/25/18 4:19 PM:


Policeman Jenkins found a reproducing seed for a 
{{TestStressCloudBlindAtomicUpdates.test_dv_idx()}} failure that {{git bisect}} 
blames on commit {{412116f}} on this issue 
[https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/2190/]:

{noformat}
Checking out Revision 82b793df56c8c9fb50c29f46f39465453a87f2b2 
(refs/remotes/origin/branch_7x)
[...]
   [junit4]   2> NOTE: reproduce with: ant test  
-Dtestcase=TestStressCloudBlindAtomicUpdates -Dtests.method=test_dv_idx 
-Dtests.seed=9E9025C836D166FA -Dtests.multiplier=3 -Dtests.slow=true 
-Dtests.locale=ga -Dtests.timezone=Europe/Monaco -Dtests.asserts=true 
-Dtests.file.encoding=UTF-8
   [junit4] FAILURE 11.4s J1 | TestStressCloudBlindAtomicUpdates.test_dv_idx <<<
   [junit4]> Throwable #1: java.lang.AssertionError: Some docs had errors 
-- check logs expected:<0> but was:<7>
   [junit4]>at 
__randomizedtesting.SeedInfo.seed([9E9025C836D166FA:BEC5000EA0DCB1B]:0)
   [junit4]>at 
org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.checkField(TestStressCloudBlindAtomicUpdates.java:342)
   [junit4]>at 
org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.test_dv_idx(TestStressCloudBlindAtomicUpdates.java:231)
   [junit4]>at java.lang.Thread.run(Thread.java:748)
[...]
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70), 
sim=RandomSimilarity(queryNorm=true): {}, locale=ga, timezone=Europe/Monaco
   [junit4]   2> NOTE: Linux 4.13.0-41-generic i386/Oracle Corporation 
1.8.0_172 (32-bit)/cpus=8,threads=1,free=238679536,total=536870912
{noformat}


was (Author: steve_rowe):
Policeman Jenkins found a reproducing seed for a 
{{TestStressCloudBlindAtomicUpdates -Dtests.method.test_dv_idx()}} failure that 
{{git bisect}} blames on commit {{412116f}} on this issue 
[https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/2190/]:

{noformat}
Checking out Revision 82b793df56c8c9fb50c29f46f39465453a87f2b2 
(refs/remotes/origin/branch_7x)
[...]
   [junit4]   2> NOTE: reproduce with: ant test  
-Dtestcase=TestStressCloudBlindAtomicUpdates -Dtests.method=test_dv_idx 
-Dtests.seed=9E9025C836D166FA -Dtests.multiplier=3 -Dtests.slow=true 
-Dtests.locale=ga -Dtests.timezone=Europe/Monaco -Dtests.asserts=true 
-Dtests.file.encoding=UTF-8
   [junit4] FAILURE 11.4s J1 | TestStressCloudBlindAtomicUpdates.test_dv_idx <<<
   [junit4]> Throwable #1: java.lang.AssertionError: Some docs had errors 
-- check logs expected:<0> but was:<7>
   [junit4]>at 
__randomizedtesting.SeedInfo.seed([9E9025C836D166FA:BEC5000EA0DCB1B]:0)
   [junit4]>at 
org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.checkField(TestStressCloudBlindAtomicUpdates.java:342)
   [junit4]>at 
org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.test_dv_idx(TestStressCloudBlindAtomicUpdates.java:231)
   [junit4]>at java.lang.Thread.run(Thread.java:748)
[...]
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70), 
sim=RandomSimilarity(queryNorm=true): {}, locale=ga, timezone=Europe/Monaco
   [junit4]   2> NOTE: Linux 4.13.0-41-generic i386/Oracle Corporation 
1.8.0_172 (32-bit)/cpus=8,threads=1,free=238679536,total=536870912
{noformat}

> Race condition in peerSync
> --
>
> Key: SOLR-11216
> URL: https://issues.apache.org/jira/browse/SOLR-11216
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 7.4
>Reporter: Cao Manh Dat
>Assignee: Cao Manh Dat
>Priority: Major
> Fix For: master (8.0), 7.5
>
> Attachments: SOLR-11216.patch, SOLR-11216.patch, SOLR-11216.patch, 
> SOLR-11216.patch, SOLR-11216.patch, SOLR-11216.patch, SOLR-11216.patch
>
>
> When digging into SOLR-10126. I found a case that can make peerSync fail.
> * leader and replica receive update from 1 to 4
> * replica stop
> * replica miss updates 5, 6
> * replica start recovery
> ## replica buffer updates 7, 8
> ## replica request versions from leader, 
> ## in the same time leader receive update 9, so it will return updates from 1 
> to 9 (for request versions) when replica get recent versions ( so it will be 
> 1,2,3,4,5,6,7,8,9 )
> ## replica do peersync and request updates 5, 6, 9 from leader 
> ## replica apply updates 5, 6, 9. Its index does not have update 7, 8 and 
> maxVersionSpecified for fingerprint is 9, therefore compare fingerprint will 
> fail



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail:

[jira] [Commented] (SOLR-11216) Race condition in peerSync

2018-06-25 Thread Steve Rowe (JIRA)



[ 
https://issues.apache.org/jira/browse/SOLR-11216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522479#comment-16522479
 ] 

Steve Rowe commented on SOLR-11216:
---

Policeman Jenkins found a reproducing seed for a {}} failure that {{git 
bisect}} blames on commit {{412116f}} on this issue 
[https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/2190/]:

{noformat}
Checking out Revision 82b793df56c8c9fb50c29f46f39465453a87f2b2 
(refs/remotes/origin/branch_7x)
[...]
   [junit4]   2> NOTE: reproduce with: ant test  
-Dtestcase=TestStressCloudBlindAtomicUpdates -Dtests.method=test_dv_idx 
-Dtests.seed=9E9025C836D166FA -Dtests.multiplier=3 -Dtests.slow=true 
-Dtests.locale=ga -Dtests.timezone=Europe/Monaco -Dtests.asserts=true 
-Dtests.file.encoding=UTF-8
   [junit4] FAILURE 11.4s J1 | TestStressCloudBlindAtomicUpdates.test_dv_idx <<<
   [junit4]> Throwable #1: java.lang.AssertionError: Some docs had errors 
-- check logs expected:<0> but was:<7>
   [junit4]>at 
__randomizedtesting.SeedInfo.seed([9E9025C836D166FA:BEC5000EA0DCB1B]:0)
   [junit4]>at 
org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.checkField(TestStressCloudBlindAtomicUpdates.java:342)
   [junit4]>at 
org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.test_dv_idx(TestStressCloudBlindAtomicUpdates.java:231)
   [junit4]>at java.lang.Thread.run(Thread.java:748)
[...]
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70), 
sim=RandomSimilarity(queryNorm=true): {}, locale=ga, timezone=Europe/Monaco
   [junit4]   2> NOTE: Linux 4.13.0-41-generic i386/Oracle Corporation 
1.8.0_172 (32-bit)/cpus=8,threads=1,free=238679536,total=536870912
{noformat}

> Race condition in peerSync
> --
>
> Key: SOLR-11216
> URL: https://issues.apache.org/jira/browse/SOLR-11216
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 7.4
>Reporter: Cao Manh Dat
>Assignee: Cao Manh Dat
>Priority: Major
> Fix For: master (8.0), 7.5
>
> Attachments: SOLR-11216.patch, SOLR-11216.patch, SOLR-11216.patch, 
> SOLR-11216.patch, SOLR-11216.patch, SOLR-11216.patch, SOLR-11216.patch
>
>
> When digging into SOLR-10126. I found a case that can make peerSync fail.
> * leader and replica receive update from 1 to 4
> * replica stop
> * replica miss updates 5, 6
> * replica start recovery
> ## replica buffer updates 7, 8
> ## replica request versions from leader, 
> ## in the same time leader receive update 9, so it will return updates from 1 
> to 9 (for request versions) when replica get recent versions ( so it will be 
> 1,2,3,4,5,6,7,8,9 )
> ## replica do peersync and request updates 5, 6, 9 from leader 
> ## replica apply updates 5, 6, 9. Its index does not have update 7, 8 and 
> maxVersionSpecified for fingerprint is 9, therefore compare fingerprint will 
> fail



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [DISCUSS] Geo/spatial organization in Lucene

2018-06-25 Thread Alan Woodward

+1 to move LatLonPoint and friends to core, and nuke the spatial module

> On 25 Jun 2018, at 16:32, David Smiley  wrote:
> 
> Okay fine, I'm not going to block spatial stuff going into core.  
> (celebration).  I foresee the spatial stuff there growing beyond the one 
> default impl though.
> 
> Perhaps most of us are still not happy with seeing spatial code across so 
> many modules?  Nick and I have voiced this concern so far.  Given the 
> pittance of utility of what's in the spatial module today, can we agree to 
> simply remove it?
> 
> I pity users trying to figure out what is where to make sense of it.  I 
> wonder how new users discover/browse to look around -- I'm too used to the 
> codebase to have any idea what newbies do.  That seems to be this: 
> http://lucene.apache.org/core/7_3_1/index.html 
>   Each module only gets one 
> terse sentence fragment.  It'd be nice to have potentially a paragraph of 
> information?  Even without more verbage, the spatial ones could have better 
> descriptions.  I propose these changes:
> 
> * spatial:  remove it :-)   -- see above
> * spatial3d: Computational geometry on the surface of a sphere or ellipsoid, 
> including Lucene index & search solutions
> * spatial-extras: Spatial code that has external dependencies like Spatial4j 
> and JTS, including Lucene index & search solutions
> 
> perhaps "spatial-sphere" might be a more meaningful name than spatial3d?  
> Yes, it's ellipsoidal but sphere is close enough ;-)
> 
> ~ David
> 
> On Mon, Jun 25, 2018 at 10:42 AM Michael McCandless 
> mailto:luc...@mikemccandless.com>> wrote:
> I also favor B: move the common case, good performing spatial implementations 
> to core, but still bake new things in sandbox.  LatLonPoint has baked way too 
> long already!  The addition of first class (codec support) KD trees in Lucene 
> (dimensional points) was/is really a game changer for Lucene supporting 
> common geo spatial applications.
> 
> It would be nice to find a better name than geo3d / spatial3d: it confuses 
> 100% of the people I explain it to, on first impression :)  But we should 
> tackle that separately/later.
> 
> Merging the 2D/3D abstractions sounds a little too ambitious at this point, 
> so I think it's fine to leave them separate for now.
> 
> Mike McCandless
> 
> http://blog.mikemccandless.com 
> 
> On Wed, Jun 20, 2018 at 1:00 PM, Nicholas Knize  > wrote:
> If I were to pick between the two, I also have a preference for B.  I've also 
> tried to keep this whole spatial organization rather simple:
> 
> core - simple spatial capabilities needed by the 99% spatial use case (e.g., 
> web mapping). Includes LatLonPoint, polygon & distance search (everything 
> currently in sandbox). Lightweight, and no dependencies or complexities. If 
> one wants simple and fast point search, all you need is the core module.
> 
> spatial - dependency free. Expands on core spatial to include simple shape 
> searching. Uses internal relations. Everything confined to core and spatial 
> modules.
> 
> spatial-extras - expanded spatial capabilities. Welcomes third-party 
> dependencies (e.g., S3, SIS, Proj4J). Targets more advanced/expert GIS 
> use-cases.
> 
> geo3d - trades speed for accuracy. I've always struggled with the name, since 
> it implies 3D shapes/point cloud support. But history has shown considering a 
> name change to be a bike-shedding endeavor. 
> 
> At the end of the day I'm up for whatever makes most sense for everyone here. 
> Lord knows we could use more people helping out on geo.
> 
> - Nick
> 
> 
> 
> On Wed, Jun 20, 2018 at 11:40 AM Adrien Grand  > wrote:
> I have a slight preference for B similarly to how StandardAnalyzer is in core 
> and other analyzers are in analysis, but no strong feelings. In any case I 
> agree that both A and B would be much better than the current situation.
> 
> 
> Le mer. 20 juin 2018 à 18:09, David Smiley  > a écrit :
> I think everyone agrees the current state of spatial code organization in 
> Lucene is not desirable.  We have a spatial module that has almost nothing in 
> it, we have mature spatial code in the sandbox that needs to "graduate" 
> somewhere, and we've got a handful of geo utilities in Lucene core (mostly 
> because I didn't notice).  No agreement has been reached on what the desired 
> state should be.
> 
> I'd like to hear opinions on this from members of the community.  I am 
> especially interested in listening to people that normally don't seem to 
> speak up about spatial matters. Perhaps Uwe Schindlerand Alan Woodward – I 
> respect both of you guys a ton for your tenure with Lucene and aren't too 
> pushy with your opinions. I can be convinced to change my mind, especially if 
> coming from you two.  Of course anyone can respond -- this is an open 
> discussion!
> 
> As I understand

[jira] [Commented] (SOLR-11616) Backup failing on a constantly changing index with NoSuchFileException

2018-06-25 Thread Varun Thacker (JIRA)



[ 
https://issues.apache.org/jira/browse/SOLR-11616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522450#comment-16522450
 ] 

Varun Thacker commented on SOLR-11616:
--

Hi Mikhail,

I don't believe so. This would lead to a leaking SolrIndexSearcher under a 
condition which I am not sure when can it happen.

> Backup failing on a constantly changing index with NoSuchFileException
> --
>
> Key: SOLR-11616
> URL: https://issues.apache.org/jira/browse/SOLR-11616
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Varun Thacker
>Assignee: Varun Thacker
>Priority: Major
> Fix For: 7.2, master (8.0)
>
> Attachments: SOLR-11616.patch, SOLR-11616.patch, solr-6.3.log, 
> solr-7.1.log
>
>
> As reported by several users on SOLR-9120 , Solr backups fail with 
> NoSuchFileException on a constantly changing index. 
> Users linked SOLR-9120 to the root cause as the stack trace is the same , but 
> the fix proposed there won't fix backups to stop failing.
> We need to implement a similar fix in {{SnapShooter#createSnapshot}} to fix 
> the problem



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11807) maxShardsPerNode=-1 needs special handling while restoring collections

2018-06-25 Thread Varun Thacker (JIRA)



[ 
https://issues.apache.org/jira/browse/SOLR-11807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522449#comment-16522449
 ] 

Varun Thacker commented on SOLR-11807:
--

Thanks Steve! I'll look into it tomorrow. I beasted this test quite a bit 
before committing but looks like Jenkins uncovered something 

> maxShardsPerNode=-1 needs special handling while restoring collections
> --
>
> Key: SOLR-11807
> URL: https://issues.apache.org/jira/browse/SOLR-11807
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Backup/Restore
>Reporter: Varun Thacker
>Assignee: Varun Thacker
>Priority: Minor
> Fix For: master (8.0), 7.5
>
> Attachments: SOLR-11807.patch, SOLR-11807.patch, SOLR-11807.patch
>
>
> When you start Solr 6.6. and run the cloud example here's the log excerpt :
> {code:java}
> Connecting to ZooKeeper at localhost:9983 ...
> INFO  - 2018-06-20 13:44:47.491; 
> org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider; Cluster at 
> localhost:9983 ready
> ...
> Creating new collection 'gettingstarted' using command:
> http://localhost:8983/solr/admin/collections?action=CREATE=gettingstarted=2=2=2=gettingstarted{code}
> maxShardsPerNode get's set to 2 . 
>  
> Compare this to Solr 7.3 
> {code:java}
> INFO  - 2018-06-20 13:55:33.823; 
> org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider; Cluster at 
> localhost:9983 ready
> Created collection 'gettingstarted' with 2 shard(s), 2 replica(s) with 
> config-set 'gettingstarted'{code}
> So something changed and now we no longer set maxShardsPerNode and it 
> defaults to -1 . 
>  
> -1 has special handing while creating a collection ( it means max int ) . 
> This special handling is not there while restoring a collection and hence 
> this fails
> We should not set maxShardsPerNode to -1 in the first place
> Steps to reproduce:
> 1. ./bin/solr start -e cloud -noprompt : This creates a 2 node cluster and a 
> gettingstarted collection which 2X2
>  2. Add 4 docs (id=1,2,3,4) with commit=true and openSearcher=true (default)
>  3. Call backup: 
> [http://localhost:8983/solr/admin/collections?action=BACKUP=gettingstarted_backup=gettingstarted=/Users/varunthacker/solr-7.1.0]
>  4. Call restore:
>  
> [http://localhost:8983/solr/admin/collections?action=restore=gettingstarted_backup=restore_gettingstarted=/Users/varunthacker/solr-7.1.0]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11807) maxShardsPerNode=-1 needs special handling while restoring collections

2018-06-25 Thread Steve Rowe (JIRA)



[ 
https://issues.apache.org/jira/browse/SOLR-11807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522446#comment-16522446
 ] 

Steve Rowe commented on SOLR-11807:
---

Policeman Jenkins found a reproducing seed for a 
{{TestLocalFSCloudBackupRestore}} failure that {{git bisect}} blames on commit 
{{3a2ec9b}} on this issue 
[https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/2183/]:

{noformat}
Checking out Revision 82b793df56c8c9fb50c29f46f39465453a87f2b2 
(refs/remotes/origin/branch_7x)
[...]
   [junit4]   2> NOTE: reproduce with: ant test  
-Dtestcase=TestLocalFSCloudBackupRestore -Dtests.method=test 
-Dtests.seed=7CEE52DAF9E4606 -Dtests.multiplier=3 -Dtests.slow=true 
-Dtests.locale=bs-Cyrl -Dtests.timezone=America/Boise -Dtests.asserts=true 
-Dtests.file.encoding=UTF-8
   [junit4] FAILURE 15.8s J2 | TestLocalFSCloudBackupRestore.test <<<
   [junit4]> Throwable #1: java.lang.AssertionError: Node 
127.0.0.1:40147_solr has 6 replicas. Expected num replicas : 3 state file 
   [junit4]> 
DocCollection(backuprestore_restored//collections/backuprestore_restored/state.json/13)={
   [junit4]>   "pullReplicas":0,
   [junit4]>   "replicationFactor":2,
   [junit4]>   "shards":{
   [junit4]> "shard2":{
   [junit4]>   "range":"0-7fff",
   [junit4]>   "state":"active",
   [junit4]>   "replicas":{
   [junit4]> "core_node122":{
   [junit4]>   "core":"backuprestore_restored_shard2_replica_n121",
   [junit4]>   "base_url":"https://127.0.0.1:40147/solr;,
   [junit4]>   "node_name":"127.0.0.1:40147_solr",
   [junit4]>   "state":"active",
   [junit4]>   "type":"NRT",
   [junit4]>   "force_set_state":"false",
   [junit4]>   "leader":"true"},
   [junit4]> "core_node128":{
   [junit4]>   "core":"backuprestore_restored_shard2_replica_n127",
   [junit4]>   "base_url":"https://127.0.0.1:40147/solr;,
   [junit4]>   "node_name":"127.0.0.1:40147_solr",
   [junit4]>   "state":"active",
   [junit4]>   "type":"NRT",
   [junit4]>   "force_set_state":"false"}},
   [junit4]>   "stateTimestamp":"1529839664323341145"},
   [junit4]> "shard1_1":{
   [junit4]>   "range":"c000-",
   [junit4]>   "state":"active",
   [junit4]>   "replicas":{
   [junit4]> "core_node124":{
   [junit4]>   
"core":"backuprestore_restored_shard1_1_replica_n123",
   [junit4]>   "base_url":"https://127.0.0.1:40147/solr;,
   [junit4]>   "node_name":"127.0.0.1:40147_solr",
   [junit4]>   "state":"active",
   [junit4]>   "type":"NRT",
   [junit4]>   "force_set_state":"false",
   [junit4]>   "leader":"true"},
   [junit4]> "core_node130":{
   [junit4]>   
"core":"backuprestore_restored_shard1_1_replica_n129",
   [junit4]>   "base_url":"https://127.0.0.1:40147/solr;,
   [junit4]>   "node_name":"127.0.0.1:40147_solr",
   [junit4]>   "state":"active",
   [junit4]>   "type":"NRT",
   [junit4]>   "force_set_state":"false"}},
   [junit4]>   "stateTimestamp":"1529839664323360463"},
   [junit4]> "shard1_0":{
   [junit4]>   "range":"8000-bfff",
   [junit4]>   "state":"active",
   [junit4]>   "replicas":{
   [junit4]> "core_node126":{
   [junit4]>   
"core":"backuprestore_restored_shard1_0_replica_n125",
   [junit4]>   "base_url":"https://127.0.0.1:40147/solr;,
   [junit4]>   "node_name":"127.0.0.1:40147_solr",
   [junit4]>   "state":"active",
   [junit4]>   "type":"NRT",
   [junit4]>   "force_set_state":"false",
   [junit4]>   "leader":"true"},
   [junit4]> "core_node132":{
   [junit4]>   
"core":"backuprestore_restored_shard1_0_replica_n131",
   [junit4]>   "base_url":"https://127.0.0.1:40147/solr;,
   [junit4]>   "node_name":"127.0.0.1:40147_solr",
   [junit4]>   "state":"active",
   [junit4]>   "type":"NRT",
   [junit4]>   "force_set_state":"false"}},
   [junit4]>   "stateTimestamp":"1529839664323379971"}},
   [junit4]>   "router":{"name":"compositeId"},
   [junit4]>   "maxShardsPerNode":"6",
   [junit4]>   "autoAddReplicas":"true",
   [junit4]>   "nrtReplicas":2,
   [junit4]>   "tlogReplicas":0}
   [junit4]>at 
__randomizedtesting.SeedInfo.seed([7CEE52DAF9E4606:8F9ADAF701622BFE]:0)
   [junit4]>at 
org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.lambda$testBackupAndRestore$1(AbstractCloudBackupRestoreTestCase.java:346)
   [junit4]>at

[jira] [Commented] (SOLR-11616) Backup failing on a constantly changing index with NoSuchFileException

2018-06-25 Thread Mikhail Khludnev (JIRA)



[ 
https://issues.apache.org/jira/browse/SOLR-11616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522445#comment-16522445
 ] 

Mikhail Khludnev commented on SOLR-11616:
-

Hello, [~varunthacker], can SOLR-11616 cause "incomplete" backups as well? 

> Backup failing on a constantly changing index with NoSuchFileException
> --
>
> Key: SOLR-11616
> URL: https://issues.apache.org/jira/browse/SOLR-11616
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Varun Thacker
>Assignee: Varun Thacker
>Priority: Major
> Fix For: 7.2, master (8.0)
>
> Attachments: SOLR-11616.patch, SOLR-11616.patch, solr-6.3.log, 
> solr-7.1.log
>
>
> As reported by several users on SOLR-9120 , Solr backups fail with 
> NoSuchFileException on a constantly changing index. 
> Users linked SOLR-9120 to the root cause as the stack trace is the same , but 
> the fix proposed there won't fix backups to stop failing.
> We need to implement a similar fix in {{SnapShooter#createSnapshot}} to fix 
> the problem



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

BadApple report

2018-06-25 Thread Erick Erickson

I've attached the latest. I was chatting offline and "it was noted"
that these reports are pretty big, which encourages people to not
read, well, all 977 lines. Here's what I'll try to do:

The "interesting" part of the report is the tests  I'll change, i.e.
annotate or un-annotate. The rest if background in case people are
interested.

I can put the _actions_ I'll take on Thursday in the body of the email
and attach the full file for background.



  **Annotations will be removed from the following tests because they
haven't failed in the last 4 weeks.

   CdcrBootstrapTest.testBootstrapWithContinousIndexingOnSourceCluster
   ChaosMonkeySafeLeaderTest.test
   CollectionsAPIDistributedZkTest.testCollectionsAPI
   ComputePlanActionTest.testNodeLost
   ConcurrentCreateRoutedAliasTest
   DeleteReplicaTest.deleteReplicaOnIndexing
   DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica
   IndexSizeTriggerTest
   LeaderFailoverAfterPartitionTest.test
   LeaderVoteWaitTimeoutTest.basicTest
   NodeLostTriggerTest.testListenerAcceptance
   ReplaceNodeNoTargetTest
   ReplaceNodeNoTargetTest.test
   TestCollapseQParserPlugin.testStringCollapse
   TestHdfsUpdateLog.testFSThreadSafety
   TestInPlaceUpdatesDistrib.test
   TestMinMaxOnMultiValuedField.testDoubleFieldCache
   TestMinMaxOnMultiValuedField.testFloatFieldCache
   TestMinMaxOnMultiValuedField.testIntFieldCache
   TestMinMaxOnMultiValuedField.testLongFieldCache
   TestMoreLikeThis.testMultiFieldShouldReturnPerFieldBooleanQuery
   TestSQLHandler.doTest
   UIMABaseAnalyzerTest.testRandomStrings
   UIMABaseAnalyzerTest.testRandomStringsWithConfigurationParameters
   UIMATypeAwareAnalyzerTest.testRandomStrings
   UnloadDistributedZkTest.test

 Will BadApple these tests **
   Report   Pct runsfails   test
 0123   5.2 2022101  CdcrBidirectionalTest.testBiDir
 0123   0.9 1877  9
DeleteReplicaTest.deleteReplicaFromClusterState
 0123  13.4 2088435  IndexSizeTriggerTest.testMergeIntegration
 0123  25.0 2330288  IndexSizeTriggerTest.testMixedBounds
 0123  36.8 2088744  IndexSizeTriggerTest.testSplitIntegration
 0123  17.4 2239399  IndexSizeTriggerTest.testTrigger
 0123   0.2 1860  5  RecoveryZkTest(suite)
 0123   0.2 1861  6  RecoveryZkTest.test
 0123   0.6 1877 34  SolrRrdBackendFactoryTest.testBasic
 0123   0.4 1866  8  TestCloudPivotFacet.test
 0123   1.5 2020103  TestComputePlanAction.testNodeAdded
 0123   2.2 1932 52  TestExecutePlanAction.testExecute
 0123   2.8  138  7
TestGenericDistributedQueue.testDistributedQueue
 0123  26.0  567129  TestLargeCluster.testAddNode
 0123   0.2 1910 15  TestRandomChains.testRandomChains
 0123   1.0 1908 33
TestRandomChains.testRandomChainsWithLargeStrings
 0123   5.8 1600 79  TestRecovery.testExistOldBufferLog


Also note that I expect a little thrashing because I'm un-annotating
tests. That's why I'm commenting them out rather than removing them,
if failures start to thrash we'll know.

Erick
DO NOT ENABLE LIST:
'IndexSizeTriggerTest.testMergeIntegration'
'IndexSizeTriggerTest.testMixedBounds'
'IndexSizeTriggerTest.testSplitIntegration'
'IndexSizeTriggerTest.testTrigger'
'TestControlledRealTimeReopenThread.testCRTReopen'
'TestICUNormalizer2CharFilter.testRandomStrings'
'TestICUTokenizerCJK'
'TestImpersonationWithHadoopAuth.testForwarding'
'TestLTRReRankingPipeline.testDifferentTopN'
'TestRandomChains'

Processing file (History bit 3): HOSS-06-25.csv
Processing file (History bit 2): HOSS-06-18.csv
Processing file (History bit 1): HOSS-06-11.csv
Processing file (History bit 0): HOSS-06-05.csv


**Annotated tests/suites that didn't fail in the last 4 weeks.

  **Tests and suites removed from the next two lists because they were 
specified in 'doNotEnable' in the properties file
 TestControlledRealTimeReopenThread.testCRTReopen
 TestICUNormalizer2CharFilter.testRandomStrings
 TestICUTokenizerCJK
 TestImpersonationWithHadoopAuth.testForwarding
 TestLTRReRankingPipeline.testDifferentTopN

  **Annotations will be removed from the following tests because they haven't 
failed in the last month.

  **Methods: 26
   CdcrBootstrapTest.testBootstrapWithContinousIndexingOnSourceCluster
   ChaosMonkeySafeLeaderTest.test
   CollectionsAPIDistributedZkTest.testCollectionsAPI
   ComputePlanActionTest.testNodeLost
   ConcurrentCreateRoutedAliasTest
   DeleteReplicaTest.deleteReplicaOnIndexing
   DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica
   IndexSizeTriggerTest
   LeaderFailoverAfterPartitionTest.test
   LeaderVoteWaitTimeoutTest.basicTest
   NodeLostTriggerTest.testListenerAcceptance
   ReplaceNodeNoTargetTest
   ReplaceNodeNoTargetTest.test

Re: [DISCUSS] Geo/spatial organization in Lucene

2018-06-25 Thread David Smiley

Okay fine, I'm not going to block spatial stuff going into core.
 (celebration).  I foresee the spatial stuff there growing beyond the one
default impl though.

Perhaps most of us are still not happy with seeing spatial code across so
many modules?  Nick and I have voiced this concern so far.  Given the
pittance of utility of what's in the spatial module today, can we agree to
simply remove it?

I pity users trying to figure out what is where to make sense of it.  I
wonder how new users discover/browse to look around -- I'm too used to the
codebase to have any idea what newbies do.  That seems to be this:
http://lucene.apache.org/core/7_3_1/index.html  Each module only gets one
terse sentence fragment.  It'd be nice to have potentially a paragraph of
information?  Even without more verbage, the spatial ones could have better
descriptions.  I propose these changes:

* spatial:  remove it :-)   -- see above
* spatial3d: Computational geometry on the surface of a sphere or
ellipsoid, including Lucene index & search solutions
* spatial-extras: Spatial code that has external dependencies like
Spatial4j and JTS, including Lucene index & search solutions

perhaps "spatial-sphere" might be a more meaningful name than spatial3d?
Yes, it's ellipsoidal but sphere is close enough ;-)

~ David

On Mon, Jun 25, 2018 at 10:42 AM Michael McCandless <
luc...@mikemccandless.com> wrote:

> I also favor B: move the common case, good performing spatial
> implementations to core, but still bake new things in sandbox.  LatLonPoint
> has baked way too long already!  The addition of first class (codec
> support) KD trees in Lucene (dimensional points) was/is really a game
> changer for Lucene supporting common geo spatial applications.
>
> It would be nice to find a better name than geo3d / spatial3d: it confuses
> 100% of the people I explain it to, on first impression :)  But we should
> tackle that separately/later.
>
> Merging the 2D/3D abstractions sounds a little too ambitious at this
> point, so I think it's fine to leave them separate for now.
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> On Wed, Jun 20, 2018 at 1:00 PM, Nicholas Knize  wrote:
>
>> If I were to pick between the two, I also have a preference for B.  I've
>> also tried to keep this whole spatial organization rather simple:
>>
>> core - simple spatial capabilities needed by the 99% spatial use case
>> (e.g., web mapping). Includes LatLonPoint, polygon & distance search
>> (everything currently in sandbox). Lightweight, and no dependencies or
>> complexities. If one wants simple and fast point search, all you need is
>> the core module.
>>
>> spatial - dependency free. Expands on core spatial to include simple
>> shape searching. Uses internal relations. Everything confined to core and
>> spatial modules.
>>
>> spatial-extras - expanded spatial capabilities. Welcomes third-party
>> dependencies (e.g., S3, SIS, Proj4J). Targets more advanced/expert GIS
>> use-cases.
>>
>> geo3d - trades speed for accuracy. I've always struggled with the name,
>> since it implies 3D shapes/point cloud support. But history has shown
>> considering a name change to be a bike-shedding endeavor.
>>
>> At the end of the day I'm up for whatever makes most sense for everyone
>> here. Lord knows we could use more people helping out on geo.
>>
>> - Nick
>>
>>
>>
>> On Wed, Jun 20, 2018 at 11:40 AM Adrien Grand  wrote:
>>
>>> I have a slight preference for B similarly to how StandardAnalyzer is in
>>> core and other analyzers are in analysis, but no strong feelings. In any
>>> case I agree that both A and B would be much better than the current
>>> situation.
>>>
>>>
>>> Le mer. 20 juin 2018 à 18:09, David Smiley  a
>>> écrit :
>>>
 I think everyone agrees the current state of spatial code organization
 in Lucene is not desirable.  We have a spatial module that has almost
 nothing in it, we have mature spatial code in the sandbox that needs to
 "graduate" somewhere, and we've got a handful of geo utilities in Lucene
 core (mostly because I didn't notice).  No agreement has been reached on
 what the desired state should be.

 I'd like to hear opinions on this from members of the community.  I am
 especially interested in listening to people that normally don't seem to
 speak up about spatial matters. Perhaps Uwe Schindlerand Alan Woodward – I
 respect both of you guys a ton for your tenure with Lucene and aren't too
 pushy with your opinions. I can be convinced to change my mind, especially
 if coming from you two.  Of course anyone can respond -- this is an open
 discussion!

 As I understand it, there are two proposals loosely defined as follows:

 (A) Common spatial needs will be met in the "spatial" module.  The
 Lucene "spatial" module, currently in a weird gutted state, should have
 basically all spatial code currently in sandbox plus all geo stuff in
 Lucene core. Thus there will be no

[jira] [Comment Edited] (SOLR-12513) Reproducing TestCodecSupport.testMixedCompressionMode failure

2018-06-25 Thread Steve Rowe (JIRA)



[ 
https://issues.apache.org/jira/browse/SOLR-12513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522417#comment-16522417
 ] 

Steve Rowe edited comment on SOLR-12513 at 6/25/18 3:14 PM:


{quote}There are two mentions in: 
 /lucene/tools/junit4/cached-timehints.txt
 I'm not familiar with this file, should the references just be removed? Steve 
Rowe do you know?
{quote}
That file is generated locally based on test timings. I think it's okay to 
ignore them, but if you're worried about it you can regenerate the file using 
{{ant test-updatecache}} (if you're curious you can see what it does in 
{{lucene/common-build.xml}}.


was (Author: steve_rowe):
{quote}
There are two mentions in: 
/lucene/tools/junit4/cached-timehints.txt
I'm not familiar with this file, should the references just be removed? Steve 
Rowe do you know?
{quote}

That file is generated locally based on test timings.  I think it's okay to 
ignore them, but if you're worried about it you can regenerate the file using 
{{test-updatecache}} (if you're curious you can see what it does in 
{{lucene/common-build.xml}}.

> Reproducing TestCodecSupport.testMixedCompressionMode failure
> -
>
> Key: SOLR-12513
> URL: https://issues.apache.org/jira/browse/SOLR-12513
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Steve Rowe
>Assignee: Erick Erickson
>Priority: Major
>
> From [https://jenkins.thetaphi.de/job/Lucene-Solr-master-Windows/7375/] 
> (reproduces for me on Linux):
> {noformat}
> Checking out Revision 3b9d3a760a432b97aad2c08b2f778fa2344eb14a 
> (refs/remotes/origin/master)
> [...]
>[junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestCodecSupport 
> -Dtests.method=testMixedCompressionMode -Dtests.seed=F584376A20EC9B2D 
> -Dtests.slow=true -Dtests.locale=ha-GH -Dtests.timezone=Africa/Nairobi 
> -Dtests.asserts=true -Dtests.file.encoding=UTF-8
>[junit4] FAILURE 0.59s J1 | TestCodecSupport.testMixedCompressionMode <<<
>[junit4]> Throwable #1: org.junit.ComparisonFailure: Expecting 
> compression mode string to be BEST_SPEED but got: BEST_COMPRESSION
>[junit4]>  SegmentInfo: _5(8.0.0):c1
>[junit4]>  SegmentInfos: segments_e: _7(8.0.0):c2 _5(8.0.0):c1
>[junit4]>  Codec: Lucene70 expected: but 
> was:
>[junit4]>  at 
> __randomizedtesting.SeedInfo.seed([F584376A20EC9B2D:2BF1400E658E6E9C]:0)
>[junit4]>  at 
> org.apache.solr.core.TestCodecSupport.lambda$assertCompressionMode$0(TestCodecSupport.java:115)
>[junit4]>  at 
> org.apache.solr.core.SolrCore.withSearcher(SolrCore.java:1874)
>[junit4]>  at 
> org.apache.solr.core.TestCodecSupport.assertCompressionMode(TestCodecSupport.java:112)
>[junit4]>  at 
> org.apache.solr.core.TestCodecSupport.testMixedCompressionMode(TestCodecSupport.java:157)
>[junit4]>  at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>[junit4]>  at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>[junit4]>  at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>[junit4]>  at 
> java.base/java.lang.reflect.Method.invoke(Method.java:564)
>[junit4]>  at java.base/java.lang.Thread.run(Thread.java:844)
> [...]
>[junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, 
> docValues:{}, maxPointsInLeafNode=1658, maxMBSortInHeap=7.737749859284375, 
> sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@2da9a1c5),
>  locale=ha-GH, timezone=Africa/Nairobi
>[junit4]   2> NOTE: Windows 10 10.0 amd64/Oracle Corporation 10.0.1 
> (64-bit)/cpus=3,threads=1,free=39067800,total=97320960
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12513) Reproducing TestCodecSupport.testMixedCompressionMode failure

2018-06-25 Thread Steve Rowe (JIRA)



[ 
https://issues.apache.org/jira/browse/SOLR-12513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522417#comment-16522417
 ] 

Steve Rowe commented on SOLR-12513:
---

{quote}
There are two mentions in: 
/lucene/tools/junit4/cached-timehints.txt
I'm not familiar with this file, should the references just be removed? Steve 
Rowe do you know?
{quote}

That file is generated locally based on test timings.  I think it's okay to 
ignore them, but if you're worried about it you can regenerate the file using 
{{test-updatecache}} (if you're curious you can see what it does in 
{{lucene/common-build.xml}}.

> Reproducing TestCodecSupport.testMixedCompressionMode failure
> -
>
> Key: SOLR-12513
> URL: https://issues.apache.org/jira/browse/SOLR-12513
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Steve Rowe
>Assignee: Erick Erickson
>Priority: Major
>
> From [https://jenkins.thetaphi.de/job/Lucene-Solr-master-Windows/7375/] 
> (reproduces for me on Linux):
> {noformat}
> Checking out Revision 3b9d3a760a432b97aad2c08b2f778fa2344eb14a 
> (refs/remotes/origin/master)
> [...]
>[junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestCodecSupport 
> -Dtests.method=testMixedCompressionMode -Dtests.seed=F584376A20EC9B2D 
> -Dtests.slow=true -Dtests.locale=ha-GH -Dtests.timezone=Africa/Nairobi 
> -Dtests.asserts=true -Dtests.file.encoding=UTF-8
>[junit4] FAILURE 0.59s J1 | TestCodecSupport.testMixedCompressionMode <<<
>[junit4]> Throwable #1: org.junit.ComparisonFailure: Expecting 
> compression mode string to be BEST_SPEED but got: BEST_COMPRESSION
>[junit4]>  SegmentInfo: _5(8.0.0):c1
>[junit4]>  SegmentInfos: segments_e: _7(8.0.0):c2 _5(8.0.0):c1
>[junit4]>  Codec: Lucene70 expected: but 
> was:
>[junit4]>  at 
> __randomizedtesting.SeedInfo.seed([F584376A20EC9B2D:2BF1400E658E6E9C]:0)
>[junit4]>  at 
> org.apache.solr.core.TestCodecSupport.lambda$assertCompressionMode$0(TestCodecSupport.java:115)
>[junit4]>  at 
> org.apache.solr.core.SolrCore.withSearcher(SolrCore.java:1874)
>[junit4]>  at 
> org.apache.solr.core.TestCodecSupport.assertCompressionMode(TestCodecSupport.java:112)
>[junit4]>  at 
> org.apache.solr.core.TestCodecSupport.testMixedCompressionMode(TestCodecSupport.java:157)
>[junit4]>  at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>[junit4]>  at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>[junit4]>  at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>[junit4]>  at 
> java.base/java.lang.reflect.Method.invoke(Method.java:564)
>[junit4]>  at java.base/java.lang.Thread.run(Thread.java:844)
> [...]
>[junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {}, 
> docValues:{}, maxPointsInLeafNode=1658, maxMBSortInHeap=7.737749859284375, 
> sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@2da9a1c5),
>  locale=ha-GH, timezone=Africa/Nairobi
>[junit4]   2> NOTE: Windows 10 10.0 amd64/Oracle Corporation 10.0.1 
> (64-bit)/cpus=3,threads=1,free=39067800,total=97320960
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-repro - Build # 883 - Still Unstable

2018-06-25 Thread Apache Jenkins Server

Build: https://builds.apache.org/job/Lucene-Solr-repro/883/

[...truncated 29 lines...]
[repro] Jenkins log URL: 
https://builds.apache.org/job/Lucene-Solr-BadApples-Tests-master/83/consoleText

[repro] Revision: 3b9d3a760a432b97aad2c08b2f778fa2344eb14a

[repro] Repro line:  ant test  -Dtestcase=TestLargeCluster 
-Dtests.method=testAddNode -Dtests.seed=C03129E07D75B94 -Dtests.multiplier=2 
-Dtests.slow=true -Dtests.badapples=true -Dtests.locale=sr 
-Dtests.timezone=America/Noronha -Dtests.asserts=true 
-Dtests.file.encoding=US-ASCII

[repro] Repro line:  ant test  -Dtestcase=AddReplicaTest -Dtests.method=test 
-Dtests.seed=C03129E07D75B94 -Dtests.multiplier=2 -Dtests.slow=true 
-Dtests.badapples=true -Dtests.locale=en-MT -Dtests.timezone=Europe/Berlin 
-Dtests.asserts=true -Dtests.file.encoding=US-ASCII

[repro] Repro line:  ant test  -Dtestcase=IndexSizeTriggerTest 
-Dtests.method=testMergeIntegration -Dtests.seed=C03129E07D75B94 
-Dtests.multiplier=2 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=ga 
-Dtests.timezone=Etc/GMT-11 -Dtests.asserts=true -Dtests.file.encoding=US-ASCII

[repro] Repro line:  ant test  -Dtestcase=IndexSizeTriggerTest 
-Dtests.method=testSplitIntegration -Dtests.seed=C03129E07D75B94 
-Dtests.multiplier=2 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=ga 
-Dtests.timezone=Etc/GMT-11 -Dtests.asserts=true -Dtests.file.encoding=US-ASCII

[repro] git rev-parse --abbrev-ref HEAD
[repro] git rev-parse HEAD
[repro] Initial local git branch/revision: 
8c714348aeea51df19e7603905f85995bcf0371c
[repro] git fetch
[repro] git checkout 3b9d3a760a432b97aad2c08b2f778fa2344eb14a

[...truncated 2 lines...]
[repro] git merge --ff-only

[...truncated 1 lines...]
[repro] ant clean

[...truncated 6 lines...]
[repro] Test suites by module:
[repro]solr/core
[repro]   TestLargeCluster
[repro]   AddReplicaTest
[repro]   IndexSizeTriggerTest
[repro] ant compile-test

[...truncated 3300 lines...]
[repro] ant test-nocompile -Dtests.dups=5 -Dtests.maxfailures=15 
-Dtests.class="*.TestLargeCluster|*.AddReplicaTest|*.IndexSizeTriggerTest" 
-Dtests.showOutput=onerror  -Dtests.seed=C03129E07D75B94 -Dtests.multiplier=2 
-Dtests.slow=true -Dtests.badapples=true -Dtests.locale=sr 
-Dtests.timezone=America/Noronha -Dtests.asserts=true 
-Dtests.file.encoding=US-ASCII

[...truncated 79663 lines...]
[repro] Setting last failure code to 256

[repro] Failures:
[repro]   0/5 failed: org.apache.solr.cloud.AddReplicaTest
[repro]   4/5 failed: org.apache.solr.cloud.autoscaling.sim.TestLargeCluster
[repro]   5/5 failed: org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest

[repro] Re-testing 100% failures at the tip of master
[repro] git fetch
[repro] git checkout master

[...truncated 3 lines...]
[repro] git merge --ff-only

[...truncated 1 lines...]
[repro] ant clean

[...truncated 8 lines...]
[repro] Test suites by module:
[repro]solr/core
[repro]   IndexSizeTriggerTest
[repro] ant compile-test

[...truncated 3300 lines...]
[repro] ant test-nocompile -Dtests.dups=5 -Dtests.maxfailures=5 
-Dtests.class="*.IndexSizeTriggerTest" -Dtests.showOutput=onerror  
-Dtests.seed=C03129E07D75B94 -Dtests.multiplier=2 -Dtests.slow=true 
-Dtests.badapples=true -Dtests.locale=ga -Dtests.timezone=Etc/GMT-11 
-Dtests.asserts=true -Dtests.file.encoding=US-ASCII

[...truncated 27784 lines...]
[repro] Setting last failure code to 256

[repro] Failures at the tip of master:
[repro]   5/5 failed: org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest

[repro] Re-testing 100% failures at the tip of master without a seed
[repro] ant clean

[...truncated 8 lines...]
[repro] Test suites by module:
[repro]solr/core
[repro]   IndexSizeTriggerTest
[repro] ant compile-test

[...truncated 3300 lines...]
[repro] ant test-nocompile -Dtests.dups=5 -Dtests.maxfailures=5 
-Dtests.class="*.IndexSizeTriggerTest" -Dtests.showOutput=onerror  
-Dtests.multiplier=2 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=ga 
-Dtests.timezone=Etc/GMT-11 -Dtests.asserts=true -Dtests.file.encoding=US-ASCII

[...truncated 36899 lines...]
[repro] Setting last failure code to 256

[repro] Failures at the tip of master without a seed:
[repro]   4/5 failed: org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest
[repro] git checkout 8c714348aeea51df19e7603905f85995bcf0371c

[...truncated 8 lines...]
[repro] Exiting with code 256

[...truncated 5 lines...]

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9598) Solr RESTORE api doesn't wait for the restored collection to be fully ready for usage

2018-06-25 Thread Varun Thacker (JIRA)



[ 
https://issues.apache.org/jira/browse/SOLR-9598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522402#comment-16522402
 ] 

Varun Thacker commented on SOLR-9598:
-

I don't think so. from what I can tell this sounds like the problem the new 
"waitFor" flag was added to the collection APIs ? 

> Solr RESTORE api doesn't wait for the restored collection to be fully ready 
> for usage
> -
>
> Key: SOLR-9598
> URL: https://issues.apache.org/jira/browse/SOLR-9598
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Backup/Restore
>Affects Versions: 6.2
>Reporter: Hrishikesh Gadre
>Priority: Major
>
> As part of the RESTORE operation, Solr creates a new collection and adds 
> necessary number of replicas to each shard. The problem is that this 
> operation doesn't wait for this new collection to be fully ready for usage 
> (e.g. querying and indexing). This requires extra checks on the client side 
> to make sure that the recovery is complete and reflected in cluster status 
> stored in Zookeeper. e.g. refer to the backup/restore unit test for this 
> check,
> https://github.com/apache/lucene-solr/blob/722e82712435ecf46c9868137d885484152f749b/solr/core/src/test/org/apache/solr/cloud/AbstractCloudBackupRestoreTestCase.java#L234
> Ideally this check should be implemented in the RESTORE operation itself. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-9598) Solr RESTORE api doesn't wait for the restored collection to be fully ready for usage

2018-06-25 Thread Varun Thacker (JIRA)



 [ 
https://issues.apache.org/jira/browse/SOLR-9598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Thacker updated SOLR-9598:

Component/s: Backup/Restore

> Solr RESTORE api doesn't wait for the restored collection to be fully ready 
> for usage
> -
>
> Key: SOLR-9598
> URL: https://issues.apache.org/jira/browse/SOLR-9598
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Backup/Restore
>Affects Versions: 6.2
>Reporter: Hrishikesh Gadre
>Priority: Major
>
> As part of the RESTORE operation, Solr creates a new collection and adds 
> necessary number of replicas to each shard. The problem is that this 
> operation doesn't wait for this new collection to be fully ready for usage 
> (e.g. querying and indexing). This requires extra checks on the client side 
> to make sure that the recovery is complete and reflected in cluster status 
> stored in Zookeeper. e.g. refer to the backup/restore unit test for this 
> check,
> https://github.com/apache/lucene-solr/blob/722e82712435ecf46c9868137d885484152f749b/solr/core/src/test/org/apache/solr/cloud/AbstractCloudBackupRestoreTestCase.java#L234
> Ideally this check should be implemented in the RESTORE operation itself. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: SuRf FST implementation

2018-06-25 Thread Michael McCandless

Thanks for sharing, John; this looks interesting!  Would be nice to build a
codec using that reference impl (hmm, except it is C++)

FST in this context stands for "Fast Succinct Trie" not "Finite State
Transducer" and it seems to be a data structure similar to (but claimed
better than) bloom filters, in that it can efficiently evaluate set
membership with a tunable chance for being wrong (false positive), but it'd
be nice to see some numbers on a real Lucene index / use case.

Mike McCandless

http://blog.mikemccandless.com

On Sat, Jun 23, 2018 at 6:04 PM, John Wang  wrote:

> The SigMod paper describes a more compact FST implementation looks really
> interesting:
>
>  http://www.cs.cmu.edu/~huanche1/publications/surf_paper.pdf
>
>  (reference implementation: https://github.com/efficient/SuRF)
>
> Was wondering if Lucene's FST implementation used by term dictionary can
> take advantage of this.
>
> Thanks
>
> -John
>

[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build # 4697 - Unstable!

2018-06-25 Thread Policeman Jenkins Server

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/4697/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseSerialGC

2 tests failed.
FAILED:  org.apache.solr.handler.component.DistributedFacetPivotSmallTest.test

Error Message:
Error from server at http://127.0.0.1:55979//collection1: ERROR: [doc=19] Error 
adding field 'company_t'='microsoft polecat' msg=Multiple values encountered 
for non multiValued copy field text: microsoft polecat

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
from server at http://127.0.0.1:55979//collection1: ERROR: [doc=19] Error 
adding field 'company_t'='microsoft polecat' msg=Multiple values encountered 
for non multiValued copy field text: microsoft polecat
at 
__randomizedtesting.SeedInfo.seed([818134CA5B3EE43C:9D50B10F5C289C4]:0)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:173)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:138)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:152)
at 
org.apache.solr.BaseDistributedSearchTestCase.indexDoc(BaseDistributedSearchTestCase.java:483)
at 
org.apache.solr.BaseDistributedSearchTestCase.index(BaseDistributedSearchTestCase.java:476)
at 
org.apache.solr.handler.component.DistributedFacetPivotSmallTest.test(DistributedFacetPivotSmallTest.java:54)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at

Re: [DISCUSS] Geo/spatial organization in Lucene

2018-06-25 Thread Michael McCandless

I also favor B: move the common case, good performing spatial
implementations to core, but still bake new things in sandbox.  LatLonPoint
has baked way too long already!  The addition of first class (codec
support) KD trees in Lucene (dimensional points) was/is really a game
changer for Lucene supporting common geo spatial applications.

It would be nice to find a better name than geo3d / spatial3d: it confuses
100% of the people I explain it to, on first impression :)  But we should
tackle that separately/later.

Merging the 2D/3D abstractions sounds a little too ambitious at this point,
so I think it's fine to leave them separate for now.

Mike McCandless

http://blog.mikemccandless.com

On Wed, Jun 20, 2018 at 1:00 PM, Nicholas Knize  wrote:

> If I were to pick between the two, I also have a preference for B.  I've
> also tried to keep this whole spatial organization rather simple:
>
> core - simple spatial capabilities needed by the 99% spatial use case
> (e.g., web mapping). Includes LatLonPoint, polygon & distance search
> (everything currently in sandbox). Lightweight, and no dependencies or
> complexities. If one wants simple and fast point search, all you need is
> the core module.
>
> spatial - dependency free. Expands on core spatial to include simple shape
> searching. Uses internal relations. Everything confined to core and spatial
> modules.
>
> spatial-extras - expanded spatial capabilities. Welcomes third-party
> dependencies (e.g., S3, SIS, Proj4J). Targets more advanced/expert GIS
> use-cases.
>
> geo3d - trades speed for accuracy. I've always struggled with the name,
> since it implies 3D shapes/point cloud support. But history has shown
> considering a name change to be a bike-shedding endeavor.
>
> At the end of the day I'm up for whatever makes most sense for everyone
> here. Lord knows we could use more people helping out on geo.
>
> - Nick
>
>
>
> On Wed, Jun 20, 2018 at 11:40 AM Adrien Grand  wrote:
>
>> I have a slight preference for B similarly to how StandardAnalyzer is in
>> core and other analyzers are in analysis, but no strong feelings. In any
>> case I agree that both A and B would be much better than the current
>> situation.
>>
>>
>> Le mer. 20 juin 2018 à 18:09, David Smiley  a
>> écrit :
>>
>>> I think everyone agrees the current state of spatial code organization
>>> in Lucene is not desirable.  We have a spatial module that has almost
>>> nothing in it, we have mature spatial code in the sandbox that needs to
>>> "graduate" somewhere, and we've got a handful of geo utilities in Lucene
>>> core (mostly because I didn't notice).  No agreement has been reached on
>>> what the desired state should be.
>>>
>>> I'd like to hear opinions on this from members of the community.  I am
>>> especially interested in listening to people that normally don't seem to
>>> speak up about spatial matters. Perhaps Uwe Schindlerand Alan Woodward – I
>>> respect both of you guys a ton for your tenure with Lucene and aren't too
>>> pushy with your opinions. I can be convinced to change my mind, especially
>>> if coming from you two.  Of course anyone can respond -- this is an open
>>> discussion!
>>>
>>> As I understand it, there are two proposals loosely defined as follows:
>>>
>>> (A) Common spatial needs will be met in the "spatial" module.  The
>>> Lucene "spatial" module, currently in a weird gutted state, should have
>>> basically all spatial code currently in sandbox plus all geo stuff in
>>> Lucene core. Thus there will be no geo stuff in Lucene core.
>>>
>>> (B) Common spatial needs will be met by Lucene core.  Lucene core should
>>> expand it's current "geo" utilities to include the spatial stuff currently
>>> in the sandbox module.  It'd also take on what little remains in the Lucene
>>> spatial module and thus we can remove the spatial module.
>>>
>>> With either plan if a user has certain advanced/specialized needs they
>>> may need to go to spatial3d or spatial-extras modules.  These would be
>>> untouched in both proposals.
>>>
>>> I'm in favor of (A) on the grounds that we have modules for special
>>> feature areas, and spatial should be no different.  My gut estimation is
>>> that 75-90% of apps do not have spatial requirements and need not depend on
>>> any spatial module.  Other modules are probably used more (e.g. queries,
>>> suggest, etc.)
>>>
>>> Respectfully,
>>>   ~ David
>>>
>>> p.s. if I mischaracterized any proposal or overlooked another then I'm
>>> sorry, please correct me.
>>> --
>>> Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
>>> LinkedIn: http://linkedin.com/in/davidwsmiley | Book: http://www.
>>> solrenterprisesearchserver.com
>>>
>> --
> Nicholas Knize  |  Geospatial Software Guy  |  Elasticsearch & Apache
> Lucene  |  nkn...@apache.org
>

[JENKINS] Lucene-Solr-repro - Build # 882 - Still Unstable

2018-06-25 Thread Apache Jenkins Server

Build: https://builds.apache.org/job/Lucene-Solr-repro/882/

[...truncated 29 lines...]
[repro] Jenkins log URL: 
https://builds.apache.org/job/Lucene-Solr-BadApples-Tests-7.x/87/consoleText

[repro] Revision: 82b793df56c8c9fb50c29f46f39465453a87f2b2

[repro] Repro line:  ant test  -Dtestcase=TestTriggerIntegration 
-Dtests.method=testEventFromRestoredState -Dtests.seed=649A597248CB264A 
-Dtests.multiplier=2 -Dtests.slow=true -Dtests.badapples=true 
-Dtests.locale=es-EC -Dtests.timezone=America/Montreal -Dtests.asserts=true 
-Dtests.file.encoding=US-ASCII

[repro] Repro line:  ant test  -Dtestcase=TestTriggerIntegration 
-Dtests.method=testNodeAddedTriggerRestoreState -Dtests.seed=649A597248CB264A 
-Dtests.multiplier=2 -Dtests.slow=true -Dtests.badapples=true 
-Dtests.locale=es-EC -Dtests.timezone=America/Montreal -Dtests.asserts=true 
-Dtests.file.encoding=US-ASCII

[repro] Repro line:  ant test  -Dtestcase=TestTriggerIntegration 
-Dtests.method=testTriggerThrottling -Dtests.seed=649A597248CB264A 
-Dtests.multiplier=2 -Dtests.slow=true -Dtests.badapples=true 
-Dtests.locale=es-EC -Dtests.timezone=America/Montreal -Dtests.asserts=true 
-Dtests.file.encoding=US-ASCII

[repro] Repro line:  ant test  -Dtestcase=TestTriggerIntegration 
-Dtests.method=testNodeLostTriggerRestoreState -Dtests.seed=649A597248CB264A 
-Dtests.multiplier=2 -Dtests.slow=true -Dtests.badapples=true 
-Dtests.locale=es-EC -Dtests.timezone=America/Montreal -Dtests.asserts=true 
-Dtests.file.encoding=US-ASCII

[repro] Repro line:  ant test  -Dtestcase=TestTriggerIntegration 
-Dtests.method=testListeners -Dtests.seed=649A597248CB264A -Dtests.multiplier=2 
-Dtests.slow=true -Dtests.badapples=true -Dtests.locale=es-EC 
-Dtests.timezone=America/Montreal -Dtests.asserts=true 
-Dtests.file.encoding=US-ASCII

[repro] Repro line:  ant test  -Dtestcase=TestTriggerIntegration 
-Dtests.method=testSearchRate -Dtests.seed=649A597248CB264A 
-Dtests.multiplier=2 -Dtests.slow=true -Dtests.badapples=true 
-Dtests.locale=es-EC -Dtests.timezone=America/Montreal -Dtests.asserts=true 
-Dtests.file.encoding=US-ASCII

[repro] Repro line:  ant test  -Dtestcase=TestTriggerIntegration 
-Dtests.method=testNodeMarkersRegistration -Dtests.seed=649A597248CB264A 
-Dtests.multiplier=2 -Dtests.slow=true -Dtests.badapples=true 
-Dtests.locale=es-EC -Dtests.timezone=America/Montreal -Dtests.asserts=true 
-Dtests.file.encoding=US-ASCII

[repro] Repro line:  ant test  -Dtestcase=TestTriggerIntegration 
-Dtests.method=testEventQueue -Dtests.seed=649A597248CB264A 
-Dtests.multiplier=2 -Dtests.slow=true -Dtests.badapples=true 
-Dtests.locale=es-EC -Dtests.timezone=America/Montreal -Dtests.asserts=true 
-Dtests.file.encoding=US-ASCII

[repro] Repro line:  ant test  -Dtestcase=TestLargeCluster 
-Dtests.method=testAddNode -Dtests.seed=649A597248CB264A -Dtests.multiplier=2 
-Dtests.slow=true -Dtests.badapples=true -Dtests.locale=mt 
-Dtests.timezone=America/Goose_Bay -Dtests.asserts=true 
-Dtests.file.encoding=US-ASCII

[repro] Repro line:  ant test  -Dtestcase=HttpPartitionTest -Dtests.method=test 
-Dtests.seed=649A597248CB264A -Dtests.multiplier=2 -Dtests.slow=true 
-Dtests.badapples=true -Dtests.locale=tr -Dtests.timezone=America/Hermosillo 
-Dtests.asserts=true -Dtests.file.encoding=US-ASCII

[repro] Repro line:  ant test  -Dtestcase=HttpPartitionTest 
-Dtests.seed=649A597248CB264A -Dtests.multiplier=2 -Dtests.slow=true 
-Dtests.badapples=true -Dtests.locale=tr -Dtests.timezone=America/Hermosillo 
-Dtests.asserts=true -Dtests.file.encoding=US-ASCII

[repro] git rev-parse --abbrev-ref HEAD
[repro] git rev-parse HEAD
[repro] Initial local git branch/revision: 
8c714348aeea51df19e7603905f85995bcf0371c
[repro] git fetch
[repro] git checkout 82b793df56c8c9fb50c29f46f39465453a87f2b2

[...truncated 2 lines...]
[repro] git merge --ff-only

[...truncated 1 lines...]
[repro] ant clean

[...truncated 6 lines...]
[repro] Test suites by module:
[repro]solr/core
[repro]   TestLargeCluster
[repro]   TestTriggerIntegration
[repro]   HttpPartitionTest
[repro] ant compile-test

[...truncated 3318 lines...]
[repro] ant test-nocompile -Dtests.dups=5 -Dtests.maxfailures=15 
-Dtests.class="*.TestLargeCluster|*.TestTriggerIntegration|*.HttpPartitionTest" 
-Dtests.showOutput=onerror  -Dtests.seed=649A597248CB264A -Dtests.multiplier=2 
-Dtests.slow=true -Dtests.badapples=true -Dtests.locale=mt 
-Dtests.timezone=America/Goose_Bay -Dtests.asserts=true 
-Dtests.file.encoding=US-ASCII

[...truncated 12487 lines...]
[repro] Setting last failure code to 256

[repro] Failures:
[repro]   0/5 failed: org.apache.solr.cloud.HttpPartitionTest
[repro]   1/5 failed: org.apache.solr.cloud.autoscaling.sim.TestLargeCluster
[repro]   4/5 failed: 
org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration
[repro] git checkout 8c714348aeea51df19e7603905f85995bcf0371c

[...truncated 2 lines...]
[repro] Exiting with code 256

[...truncated 5 lines...]

Re: Mailing List Subscription

2018-06-25 Thread Erick Erickson

Afshin:

We can't particularly do much unless you tell us _what_ you've tried
and _what_ the results were.

Please follow the instructions here:
http://lucene.apache.org/solr/community.html#mailing-lists-irc. You
must use the _exact_ same e-mail as you used to subscribe.

If the initial try doesn't work and following the suggestions at the
"problems" link doesn't work for you, let us know. But note you need
to show us the _entire_ return header to allow anyone to diagnose the
problem.

Best,
Erick


On Mon, Jun 25, 2018 at 3:17 AM, Afshin Khosravi  wrote:
> Can you also help me unsubscribe?  Nothing seem to work!  Thank you
>
> Afshin S. Khosravi
> CEO
> Trilogy Integrated Resources, LLC.
> www.trilogyir.com
> www.networkofcare.org
> 415.225.4044
>
> On Jun 25, 2018, at 2:28 AM, Jan Høydahl 
> mailto:jan@cominvent.com>> wrote:
>
> Here are instructions on how to subscribe to the lists: 
> http://lucene.apache.org/solr/community.html#mailing-lists-irc
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
>
> 25. jun. 2018 kl. 06:07 skrev Rainman Sián 
> mailto:rainman.s...@gmail.com>>:
>
> Hello
>
> I'm Rainman, I have worked with Solr in a couple of projects in the past and 
> about to start a new one.
>
> I want to be part of this list and collaborate to the project,
>
> Best regards,
>
> --
> Rainman Sián
>
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9598) Solr RESTORE api doesn't wait for the restored collection to be fully ready for usage

2018-06-25 Thread Mikhail Khludnev (JIRA)



[ 
https://issues.apache.org/jira/browse/SOLR-9598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522350#comment-16522350
 ] 

Mikhail Khludnev commented on SOLR-9598:


is it related to SOLR-12291 ? 

> Solr RESTORE api doesn't wait for the restored collection to be fully ready 
> for usage
> -
>
> Key: SOLR-9598
> URL: https://issues.apache.org/jira/browse/SOLR-9598
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 6.2
>Reporter: Hrishikesh Gadre
>Priority: Major
>
> As part of the RESTORE operation, Solr creates a new collection and adds 
> necessary number of replicas to each shard. The problem is that this 
> operation doesn't wait for this new collection to be fully ready for usage 
> (e.g. querying and indexing). This requires extra checks on the client side 
> to make sure that the recovery is complete and reflected in cluster status 
> stored in Zookeeper. e.g. refer to the backup/restore unit test for this 
> check,
> https://github.com/apache/lucene-solr/blob/722e82712435ecf46c9868137d885484152f749b/solr/core/src/test/org/apache/solr/cloud/AbstractCloudBackupRestoreTestCase.java#L234
> Ideally this check should be implemented in the RESTORE operation itself. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-repro - Build # 881 - Unstable

2018-06-25 Thread Apache Jenkins Server

Build: https://builds.apache.org/job/Lucene-Solr-repro/881/

[...truncated 28 lines...]
[repro] Jenkins log URL: 
https://builds.apache.org/job/Lucene-Solr-BadApples-NightlyTests-master/16/consoleText

[repro] Revision: 3b9d3a760a432b97aad2c08b2f778fa2344eb14a

[repro] Ant options: -Dtests.multiplier=2 
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/test-data/enwiki.random.lines.txt
[repro] Repro line:  ant test  -Dtestcase=HdfsRestartWhileUpdatingTest 
-Dtests.method=test -Dtests.seed=ED1B0E4F1ACD66D7 -Dtests.multiplier=2 
-Dtests.nightly=true -Dtests.slow=true -Dtests.badapples=true 
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/test-data/enwiki.random.lines.txt
 -Dtests.locale=ar-YE -Dtests.timezone=Pacific/Marquesas -Dtests.asserts=true 
-Dtests.file.encoding=UTF-8

[repro] Repro line:  ant test  -Dtestcase=HdfsRestartWhileUpdatingTest 
-Dtests.seed=ED1B0E4F1ACD66D7 -Dtests.multiplier=2 -Dtests.nightly=true 
-Dtests.slow=true -Dtests.badapples=true 
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/test-data/enwiki.random.lines.txt
 -Dtests.locale=ar-YE -Dtests.timezone=Pacific/Marquesas -Dtests.asserts=true 
-Dtests.file.encoding=UTF-8

[repro] Repro line:  ant test  -Dtestcase=TestPullReplica 
-Dtests.seed=ED1B0E4F1ACD66D7 -Dtests.multiplier=2 -Dtests.nightly=true 
-Dtests.slow=true -Dtests.badapples=true 
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/test-data/enwiki.random.lines.txt
 -Dtests.locale=ar-SD -Dtests.timezone=Africa/Bujumbura -Dtests.asserts=true 
-Dtests.file.encoding=UTF-8

[repro] git rev-parse --abbrev-ref HEAD
[repro] git rev-parse HEAD
[repro] Initial local git branch/revision: 
8c714348aeea51df19e7603905f85995bcf0371c
[repro] git fetch
[repro] git checkout 3b9d3a760a432b97aad2c08b2f778fa2344eb14a

[...truncated 2 lines...]
[repro] git merge --ff-only

[...truncated 1 lines...]
[repro] ant clean

[...truncated 6 lines...]
[repro] Test suites by module:
[repro]solr/core
[repro]   TestPullReplica
[repro]   HdfsRestartWhileUpdatingTest
[repro] ant compile-test

[...truncated 3300 lines...]
[repro] ant test-nocompile -Dtests.dups=5 -Dtests.maxfailures=10 
-Dtests.class="*.TestPullReplica|*.HdfsRestartWhileUpdatingTest" 
-Dtests.showOutput=onerror -Dtests.multiplier=2 
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/test-data/enwiki.random.lines.txt
 -Dtests.seed=ED1B0E4F1ACD66D7 -Dtests.multiplier=2 -Dtests.nightly=true 
-Dtests.slow=true -Dtests.badapples=true 
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-NightlyTests-master/test-data/enwiki.random.lines.txt
 -Dtests.locale=ar-SD -Dtests.timezone=Africa/Bujumbura -Dtests.asserts=true 
-Dtests.file.encoding=UTF-8

[...truncated 26133 lines...]
[repro] Setting last failure code to 256

[repro] Failures:
[repro]   0/5 failed: org.apache.solr.cloud.TestPullReplica
[repro]   1/5 failed: org.apache.solr.cloud.hdfs.HdfsRestartWhileUpdatingTest
[repro] git checkout 8c714348aeea51df19e7603905f85995bcf0371c

[...truncated 2 lines...]
[repro] Exiting with code 256

[...truncated 6 lines...]

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (SOLR-12506) Add SolrJ support for the modify collection API

2018-06-25 Thread Shalin Shekhar Mangar (JIRA)



 [ 
https://issues.apache.org/jira/browse/SOLR-12506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shalin Shekhar Mangar resolved SOLR-12506.
--
Resolution: Fixed

Committed.

master: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/8c714348
branch_7x: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/0db26c9f

> Add SolrJ support for the modify collection API
> ---
>
> Key: SOLR-12506
> URL: https://issues.apache.org/jira/browse/SOLR-12506
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrJ
>Reporter: Shalin Shekhar Mangar
>Assignee: Shalin Shekhar Mangar
>Priority: Minor
> Fix For: master (8.0), 7.5
>
> Attachments: SOLR-12506.patch, SOLR-12506.patch
>
>
> We should add support for the modify collection API in SolrJ



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (SOLR-12468) Update Jetty to 9.4.11.v20180605

2018-06-25 Thread Shalin Shekhar Mangar (JIRA)



 [ 
https://issues.apache.org/jira/browse/SOLR-12468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shalin Shekhar Mangar resolved SOLR-12468.
--
   Resolution: Fixed
Fix Version/s: 7.5
   master (8.0)

Committed.

Master: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/ffd99443
branch_7x: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/6c27059d

> Update Jetty to 9.4.11.v20180605
> 
>
> Key: SOLR-12468
> URL: https://issues.apache.org/jira/browse/SOLR-12468
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Michael Braun
>Assignee: Shalin Shekhar Mangar
>Priority: Major
> Fix For: master (8.0), 7.5
>
> Attachments: SOLR-12468.patch
>
>
> Summary of changes from 
> https://github.com/eclipse/jetty.project/blob/jetty-9.4.x/VERSION.txt
> {code}
> jetty-9.4.11.v20180605 - 05 June 2018
>  + 1785 Support for vhost@connectorname syntax of virtual hosts
>  + 2346 Revert stack trace logging for HTTPChannel.onException
>  + 2439 Remove HTTP/2 data copy
>  + 2472 central.maven.org doesn't work with https
>  + 2484 Repeated null check in MimeTypes.getDefaultMimeByExtension
>  + 2496 Jetty Maven Plugin should skip execution on projects it cannot support
>  + 2516 NPE at SslClientConnectionFactory.newConnection()
>  + 2518 HttpClient cannot handle bad servers that report multiple 100-continue
>responses in the same conversation
>  + 2525 Deprecate BlockingTimeout mechanism for removal in future release
>  + 2529 HttpParser cleanup
>  + 2532 Improve parser handing of tokens
>  + 2545 Slow HTTP2 per-stream download performance
>  + 2546 Incorrect parsing of PROXY protocol v2
>  + 2548 Possible deadlock failing HTTP/2 stream creation
>  + 2549 ConsumeAll and requestRecycle
>  + 2550 Coalesce overlapping HTTP requested byte ranges
>  + 2556 "file:" prefix in jetty.base variable
>  + 2559 Use Configurator declared in ServerEndpointConfig over one declared in
>the @ServerEndpoint annotation
>  + 2560 PathResource exception handling
>  + 2568 QueuedThreadPool.getBusyThreads() should take into account
>ReservedThreadExecutor.getAvailable()
>  + 2571 Jetty Client 9.4.x incorrectly handles too large fields from nginx 
> 1.14
>server
>  + 2574 Clarify max request queued exception message
>  + 2575 Work around broken OSGi implementations Bundle.getEntry() behavior 
> returning
>with unescaped URLs
>  + 2580 Stop creating unnecessary exceptions with MultiException
>  + 2586 Update to asm 6.2
>  + 2603 WebSocket ByteAccumulator initialized with wrong maximum
>  + 2604 WebSocket ByteAccumulator should report sizes in
>MessageTooLargeException
>  + 2616 Trailers preventing client from processing all the data
>  + 2619 QueuedThreadPool race can shrink newly created idle threads before use
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

1 2 >

1 - 100 of 140 matches

Mail list logo