[jira] [Commented] (LUCENE-8585) Create jump-tables for DocValues at index-time
[
https://issues.apache.org/jira/browse/LUCENE-8585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745859#comment-16745859
]
Toke Eskildsen commented on LUCENE-8585:
Ah yes, {{TestLucene80DocValuesFormat}}, thanks.
I'll await the ping from [~romseygeek] and if positive, merge {{master}} in, do
a last {{ant test}} and merge back to {{master}}.
Considering your involvement in this, I suggest I set the credits in the
{{CHANGES.txt}} to (Toke Eskildsen, Adrien Grand)?
> Create jump-tables for DocValues at index-time
> --
>
> Key: LUCENE-8585
> URL: https://issues.apache.org/jira/browse/LUCENE-8585
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: 8.0
>Reporter: Toke Eskildsen
>Priority: Minor
> Labels: performance
> Attachments: LUCENE-8585.patch, LUCENE-8585.patch,
> make_patch_lucene8585.sh
>
> Time Spent: 10h 10m
> Remaining Estimate: 0h
>
> As noted in LUCENE-7589, lookup of DocValues should use jump-tables to avoid
> long iterative walks. This is implemented in LUCENE-8374 at search-time
> (first request for DocValues from a field in a segment), with the benefit of
> working without changes to existing Lucene 7 indexes and the downside of
> introducing a startup time penalty and a memory overhead.
> As discussed in LUCENE-8374, the codec should be updated to create these
> jump-tables at index time. This eliminates the segment-open time & memory
> penalties, with the potential downside of increasing index-time for DocValues.
> The three elements of LUCENE-8374 should be transferable to index-time
> without much alteration of the core structures:
> * {{IndexedDISI}} block offset and index skips: A {{long}} (64 bits) for
> every 65536 documents, containing the offset of the block in 33 bits and the
> index (number of set bits) up to the block in 31 bits.
> It can be build sequentially and should be stored as a simple sequence of
> consecutive longs for caching of lookups.
> As it is fairly small, relative to document count, it might be better to
> simply memory cache it?
> * {{IndexedDISI}} DENSE (> 4095, < 65536 set bits) blocks: A {{short}} (16
> bits) for every 8 {{longs}} (512 bits) for a total of 256 bytes/DENSE_block.
> Each {{short}} represents the number of set bits up to right before the
> corresponding sub-block of 512 docIDs.
> The \{{shorts}} can be computed sequentially or when the DENSE block is
> flushed (probably the easiest). They should be stored as a simple sequence of
> consecutive shorts for caching of lookups, one logically independent sequence
> for each DENSE block. The logical position would be one sequence at the start
> of every DENSE block.
> Whether it is best to read all the 16 {{shorts}} up front when a DENSE block
> is accessed or whether it is best to only read any individual {{short}} when
> needed is not clear at this point.
> * Variable Bits Per Value: A {{long}} (64 bits) for every 16384 numeric
> values. Each {{long}} holds the offset to the corresponding block of values.
> The offsets can be computed sequentially and should be stored as a simple
> sequence of consecutive {{longs}} for caching of lookups.
> The vBPV-offsets has the largest space overhead og the 3 jump-tables and a
> lot of the 64 bits in each long are not used for most indexes. They could be
> represented as a simple {{PackedInts}} sequence or {{MonotonicLongValues}},
> with the downsides of a potential lookup-time overhead and the need for doing
> the compression after all offsets has been determined.
> I have no experience with the codec-parts responsible for creating
> index-structures. I'm quite willing to take a stab at this, although I
> probably won't do much about it before January 2019. Should anyone else wish
> to adopt this JIRA-issue or co-work on it, I'll be happy to share.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] tokee commented on a change in pull request #525: LUCENE-8585: Index-time jump-tables for DocValues
tokee commented on a change in pull request #525: LUCENE-8585: Index-time jump-tables for DocValues URL: https://github.com/apache/lucene-solr/pull/525#discussion_r248948819 ## File path: lucene/test-framework/src/java/org/apache/lucene/index/BaseDocValuesFormatTestCase.java ## @@ -1216,7 +1219,7 @@ private void doTestNumericsVsStoredFields(double density, LongSupplier longs) th doc.add(dvField); // index some docs -int numDocs = atLeast(300); +int numDocs = atLeast((int) (minDocs*1.172)); Review comment: Yeah, that should have been explained. It is because I made the minDocs variable. Originally the test was fixed with 256 as minDocs. `256*1.172=300`, so with this trick the `numDocs/minDocs`-factor stayed constant for the old tests. I have probably been overthinking this. Should I set it to something less puzzling like 1.5 or 2? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[JENKINS] Lucene-Solr-7.x-MacOSX (64bit/jdk-9) - Build # 1029 - Unstable!
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-MacOSX/1029/
Java: 64bit/jdk-9 -XX:+UseCompressedOops -XX:+UseSerialGC
1 tests failed.
FAILED: org.apache.solr.core.TestDynamicLoading.testDynamicLoading
Error Message:
Could not get expected value
'org.apache.solr.core.BlobStoreTestRequestHandler' for path
'overlay/requestHandler/\/test1/class' full output: { "responseHeader":{
"status":0, "QTime":0}, "overlay":{ "znodeVersion":0,
"runtimeLib":{"colltest":{ "name":"colltest", "version":1,
from server: null
Stack Trace:
java.lang.AssertionError: Could not get expected value
'org.apache.solr.core.BlobStoreTestRequestHandler' for path
'overlay/requestHandler/\/test1/class' full output: {
"responseHeader":{
"status":0,
"QTime":0},
"overlay":{
"znodeVersion":0,
"runtimeLib":{"colltest":{
"name":"colltest",
"version":1, from server: null
at
__randomizedtesting.SeedInfo.seed([8A06D6D5754450F5:524BFB828299F555]:0)
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.solr.core.TestSolrConfigHandler.testForResponseElement(TestSolrConfigHandler.java:590)
at
org.apache.solr.core.TestDynamicLoading.testDynamicLoading(TestDynamicLoading.java:80)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1075)
at
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1047)
at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
org.ap
[JENKINS] Lucene-Solr-BadApples-7.x-Linux (64bit/jdk-9.0.4) - Build # 149 - Failure!
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-BadApples-7.x-Linux/149/ Java: 64bit/jdk-9.0.4 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC All tests passed Build Log: [...truncated 1943 lines...] [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/lucene/build/core/test/temp/junit4-J0-20190118_054249_8218449212046283322016.syserr [junit4] >>> JVM J0 emitted unexpected output (verbatim) [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. [junit4] <<< JVM J0: EOF [...truncated 5 lines...] [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/lucene/build/core/test/temp/junit4-J2-20190118_054249_8209106140052814722655.syserr [junit4] >>> JVM J2 emitted unexpected output (verbatim) [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. [junit4] <<< JVM J2: EOF [...truncated 5 lines...] [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/lucene/build/core/test/temp/junit4-J1-20190118_054249_821726649132168379225.syserr [junit4] >>> JVM J1 emitted unexpected output (verbatim) [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. [junit4] <<< JVM J1: EOF [...truncated 292 lines...] [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/lucene/build/test-framework/test/temp/junit4-J1-20190118_055113_800765498382137989224.syserr [junit4] >>> JVM J1 emitted unexpected output (verbatim) [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. [junit4] <<< JVM J1: EOF [...truncated 12 lines...] [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/lucene/build/test-framework/test/temp/junit4-J0-20190118_055113_8009754459724734856457.syserr [junit4] >>> JVM J0 emitted unexpected output (verbatim) [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. [junit4] <<< JVM J0: EOF [...truncated 3 lines...] [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/lucene/build/test-framework/test/temp/junit4-J2-20190118_055113_80010881172302610510731.syserr [junit4] >>> JVM J2 emitted unexpected output (verbatim) [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. [junit4] <<< JVM J2: EOF [...truncated 1080 lines...] [junit4] JVM J0: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/lucene/build/analysis/common/test/temp/junit4-J0-20190118_055244_18710630833757237963361.syserr [junit4] >>> JVM J0 emitted unexpected output (verbatim) [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. [junit4] <<< JVM J0: EOF [...truncated 3 lines...] [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/lucene/build/analysis/common/test/temp/junit4-J1-20190118_055244_25512691703270409103193.syserr [junit4] >>> JVM J1 emitted unexpected output (verbatim) [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. [junit4] <<< JVM J1: EOF [...truncated 3 lines...] [junit4] JVM J2: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/lucene/build/analysis/common/test/temp/junit4-J2-20190118_055244_2466247560683876065323.syserr [junit4] >>> JVM J2 emitted unexpected output (verbatim) [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. [junit4] <<< JVM J2: EOF [...truncated 255 lines...] [junit4] JVM J1: stderr was not empty, see: /home/jenkins/workspace/Lucene-Solr-BadApples-7.x-Linux/lucene/build/analysis/icu/test/temp/junit4-J1-20190118_055503_36216093639320001391190.syserr [junit4] >>> JVM J1 emitted unexpected output (verbatim) [junit4] Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. [junit4] <<< JVM J1: EOF
[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build # 5024 - Unstable!
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/5024/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseG1GC
3 tests failed.
FAILED: org.apache.solr.cloud.MetricsHistoryIntegrationTest.testGet
Error Message:
Stack Trace:
java.lang.AssertionError
at
__randomizedtesting.SeedInfo.seed([4871674043380643:4577F4C3A0516BE5]:0)
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertNotNull(Assert.java:712)
at org.junit.Assert.assertNotNull(Assert.java:722)
at
org.apache.solr.cloud.MetricsHistoryIntegrationTest.testGet(MetricsHistoryIntegrationTest.java:138)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.cloud.MetricsHistoryIntegrationTest.testList
Error Message:
{} expected:<3> but was:<0>
Stack Trace:
java.lang.AssertionError: {} expected:<3> but was:<0>
at
__randomizedtesting.SeedInfo.seed([4871
[JENKINS-EA] Lucene-Solr-8.x-Windows (64bit/jdk-12-ea+23) - Build # 14 - Unstable!
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-8.x-Windows/14/ Java: 64bit/jdk-12-ea+23 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC 2 tests failed. FAILED: org.apache.solr.cloud.OverseerRolesTest.testOverseerRole Error Message: Timed out waiting for overseer state change Stack Trace: java.lang.AssertionError: Timed out waiting for overseer state change at __randomizedtesting.SeedInfo.seed([A6786B1D1F0FB08B:47B3968924BC865A]:0) at org.junit.Assert.fail(Assert.java:88) at org.apache.solr.cloud.OverseerRolesTest.waitForNewOverseer(OverseerRolesTest.java:63) at org.apache.solr.cloud.OverseerRolesTest.testOverseerRole(OverseerRolesTest.java:145) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:567) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938) at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974) at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) at java.base/java.lang.Thread.run(Thread.java:835) FAILED: org.apache.solr.core.ExitableDirectoryReaderTest.testQueryResults Error Message: Should NOT have inserted partial results! expected:<4> but was:<5> Stack Trace: java.lang.AssertionError: Should NOT have inserted partial results! expected:<4>
[JENKINS] Lucene-Solr-Tests-8.x - Build # 8 - Still Unstable
Build: https://builds.apache.org/job/Lucene-Solr-Tests-8.x/8/ 4 tests failed. FAILED: junit.framework.TestSuite.org.apache.solr.cloud.autoscaling.AutoScalingHandlerTest Error Message: ObjectTracker found 2 object(s) that were not released!!! [SolrZkClient, ZkStateReader] org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.common.cloud.SolrZkClient at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42) at org.apache.solr.common.cloud.SolrZkClient.(SolrZkClient.java:203) at org.apache.solr.common.cloud.SolrZkClient.(SolrZkClient.java:126) at org.apache.solr.common.cloud.SolrZkClient.(SolrZkClient.java:116) at org.apache.solr.common.cloud.ZkStateReader.(ZkStateReader.java:306) at org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider.connect(ZkClientClusterStateProvider.java:160) at org.apache.solr.client.solrj.impl.CloudSolrClient.connect(CloudSolrClient.java:399) at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:827) at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817) at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:207) at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:224) at org.apache.solr.client.solrj.impl.SolrClientCloudManager.request(SolrClientCloudManager.java:115) at org.apache.solr.cloud.autoscaling.ExecutePlanAction.waitForTaskToFinish(ExecutePlanAction.java:132) at org.apache.solr.cloud.autoscaling.ExecutePlanAction.process(ExecutePlanAction.java:85) at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:324) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.common.cloud.ZkStateReader at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42) at org.apache.solr.common.cloud.ZkStateReader.(ZkStateReader.java:328) at org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider.connect(ZkClientClusterStateProvider.java:160) at org.apache.solr.client.solrj.impl.CloudSolrClient.connect(CloudSolrClient.java:399) at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:827) at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817) at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:207) at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:224) at org.apache.solr.client.solrj.impl.SolrClientCloudManager.request(SolrClientCloudManager.java:115) at org.apache.solr.cloud.autoscaling.ExecutePlanAction.waitForTaskToFinish(ExecutePlanAction.java:132) at org.apache.solr.cloud.autoscaling.ExecutePlanAction.process(ExecutePlanAction.java:85) at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:324) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) expected null, but was:(SolrZkClient.java:203) at org.apache.solr.common.cloud.SolrZkClient.(SolrZkClient.java:126) at org.apache.solr.common.cloud.SolrZkClient.(SolrZkClient.java:116) at org.apache.solr.common.cloud.ZkStateReader.(ZkStateReader.java:306) at org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider.connect(ZkClientClusterStateProvider.java:160) at org.apache.solr.client.solrj.impl.CloudSolrClient.connect(CloudSolrClient.java:399) at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:827) at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817) at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:207) at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:224) at org.apache.solr.client.solrj.impl.SolrClientCloudManager.request(SolrClientCloudManager.java:115) at org.apache.solr.cloud.autoscaling.ExecutePlanAction.waitForTaskToFinish(ExecutePlanAction.java:132) at org.apache.solr.cloud.autoscaling.ExecutePlanAction.pr
[JENKINS] Lucene-Solr-repro - Build # 2693 - Unstable
Build: https://builds.apache.org/job/Lucene-Solr-repro/2693/ [...truncated 65 lines...] [repro] Jenkins log URL: https://builds.apache.org/job/Lucene-Solr-Tests-8.x/7/consoleText [repro] Revision: 59ba68697060b48d147b9e118ed946545932d491 [repro] Repro line: ant test -Dtestcase=TestSimTriggerIntegration -Dtests.method=testSearchRate -Dtests.seed=701375370C0E791F -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=ar-JO -Dtests.timezone=Africa/Kampala -Dtests.asserts=true -Dtests.file.encoding=UTF-8 [repro] Repro line: ant test -Dtestcase=DeleteReplicaTest -Dtests.method=deleteReplicaFromClusterStateLegacy -Dtests.seed=701375370C0E791F -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=en -Dtests.timezone=America/Swift_Current -Dtests.asserts=true -Dtests.file.encoding=UTF-8 [repro] git rev-parse --abbrev-ref HEAD [repro] git rev-parse HEAD [repro] Initial local git branch/revision: 75a7827bf44aa42b54702f3c07bc01e0fa58a8b7 [repro] git fetch [repro] git checkout 59ba68697060b48d147b9e118ed946545932d491 [...truncated 2 lines...] [repro] git merge --ff-only [...truncated 1 lines...] [repro] ant clean [...truncated 6 lines...] [repro] Test suites by module: [repro]solr/core [repro] TestSimTriggerIntegration [repro] DeleteReplicaTest [repro] ant compile-test [...truncated 3570 lines...] [repro] ant test-nocompile -Dtests.dups=5 -Dtests.maxfailures=10 -Dtests.class="*.TestSimTriggerIntegration|*.DeleteReplicaTest" -Dtests.showOutput=onerror -Dtests.seed=701375370C0E791F -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=ar-JO -Dtests.timezone=Africa/Kampala -Dtests.asserts=true -Dtests.file.encoding=UTF-8 [...truncated 2229 lines...] [repro] Setting last failure code to 256 [repro] Failures: [repro] 0/5 failed: org.apache.solr.cloud.DeleteReplicaTest [repro] 1/5 failed: org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration [repro] git checkout 75a7827bf44aa42b54702f3c07bc01e0fa58a8b7 [...truncated 2 lines...] [repro] Exiting with code 256 [...truncated 6 lines...] - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1753 - Still unstable
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1753/ 3 tests failed. FAILED: junit.framework.TestSuite.org.apache.solr.cloud.TlogReplayBufferedWhileIndexingTest Error Message: ObjectTracker found 2 object(s) that were not released!!! [SolrZkClient, ZkStateReader] org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.common.cloud.SolrZkClient at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42) at org.apache.solr.common.cloud.SolrZkClient.(SolrZkClient.java:203) at org.apache.solr.common.cloud.SolrZkClient.(SolrZkClient.java:126) at org.apache.solr.common.cloud.SolrZkClient.(SolrZkClient.java:116) at org.apache.solr.common.cloud.ZkStateReader.(ZkStateReader.java:306) at org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider.connect(ZkClientClusterStateProvider.java:160) at org.apache.solr.client.solrj.impl.CloudSolrClient.connect(CloudSolrClient.java:399) at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:827) at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:950) at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817) at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:207) at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:224) at org.apache.solr.cloud.StoppableIndexingThread.indexDocs(StoppableIndexingThread.java:181) at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:115) org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.common.cloud.ZkStateReader at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42) at org.apache.solr.common.cloud.ZkStateReader.(ZkStateReader.java:328) at org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider.connect(ZkClientClusterStateProvider.java:160) at org.apache.solr.client.solrj.impl.CloudSolrClient.connect(CloudSolrClient.java:399) at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:827) at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:950) at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817) at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:207) at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:224) at org.apache.solr.cloud.StoppableIndexingThread.indexDocs(StoppableIndexingThread.java:181) at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:115) expected null, but was:(SolrZkClient.java:203) at org.apache.solr.common.cloud.SolrZkClient.(SolrZkClient.java:126) at org.apache.solr.common.cloud.SolrZkClient.(SolrZkClient.java:116) at org.apache.solr.common.cloud.ZkStateReader.(ZkStateReader.java:306) at org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider.connect(ZkClientClusterStateProvider.java:160) at org.apache.solr.client.solrj.impl.CloudSolrClient.connect(CloudSolrClient.java:399) at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:827) at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:950) at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817) at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:207) at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:224) at org.apache.solr.cloud.StoppableIndexingThread.indexDocs(StoppableIndexingThread.java:181) at org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:115) org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.common.cloud.ZkStateReader at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42) at org.apache.solr.common.cloud.ZkStateReader.(ZkStateReader.java:328) at org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider.connect(ZkClientClusterStateProvider.java:160) at org.apache.solr.client.solrj.impl.CloudSolrClient.connect(CloudSolrClient.java:399) at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:827) at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:950) at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817) at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:207) at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:224) at org.apache.solr.cloud.StoppableIndexingThread.indexDocs(StoppableIndexingThread.java:181) at org.apache.solr.cloud.StoppableIndexingThread.run(St
[jira] [Commented] (SOLR-12281) Document the new index back-compat guarantees
[
https://issues.apache.org/jira/browse/SOLR-12281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745674#comment-16745674
]
Erick Erickson commented on SOLR-12281:
---
Or it's a sub-task...
> Document the new index back-compat guarantees
> -
>
> Key: SOLR-12281
> URL: https://issues.apache.org/jira/browse/SOLR-12281
> Project: Solr
> Issue Type: Task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: documentation
>Affects Versions: 8.0
>Reporter: Jan Høydahl
>Priority: Blocker
> Fix For: 8.0
>
>
> Refer to discussions in LUCENE-8264. Since 7.x a lucene segment will carry
> information about *first created* version, and in 8.0 Lucene will refuse to
> read an index created pre-7.0, even if the index is "upgraded" and rewritten
> in 7.x format.
> This new policy breaks with our historic N-1 guarantee, in that you now will
> be *forced* to reindex from source every 2 major versions, starting from 8.0.
> There is a chance that the requirement will be relaxed in 9.0 so that 9.x
> will be able to read 7.x indices but that I guess depends on what breaking
> index changes are made before 9.0.
> This all must be documented clearly both in CHANGES and in RefGuide. Now we
> have wordings that you can upgrade 6->7 and then 7->8.
> Related to this we should also touch parts of our documentation that deals
> with stored/docvalues, and add a recommendation to keep things stored because
> you *will* need to reindex.
> Probably this also affects backup/restore in that if you plan to upgrade to
> 8.x you will need to make sure that none of your segments are created
> pre-7.x. And perhaps it affects CDCR as well if one cluster is upgraded
> before the other etc.
> Then finally -- and this is beyond just documentation -- we should look into
> better product support for various re-index scenarios. We already have the
> streaming {{update()}} feature, backup/restore, manual /export to json etc.
> But what about:
> * A nice Admin UI on top of the streaming capabilities, where you enter the
> URL of the remote (old) system and then you get a list of collections that
> you can select from, and click "GO" and go for lunch, then when you return
> everything is reindexed into the new cluster.
> * Have the install script warn you when doing an in-place upgrade from 7.x
> to 8.0?
> * If Solr 8.x is started and detects a pre-7.0 index, then log big ERROR,
> and show a big red "UPGRADE" button in the Admin UI. Clicking this button
> would trigger some magic code that re-builds all collections from source (if
> stored/DV fields can be found in the old indices).
> I've put this as a blocker issue for now, in that we at least need some
> documentation updates before 8.0, and ideally some tools as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
Re: maxShardsPerNode always over-riden by autoscaling?
Sorry I meant still shows in /collections/myCollection/state.json (bleh
don't know why I always seem to say cluster state when I mean collection
sate)
I also can't account for why the ticket I linked says "Thirdly, the
maxShardsPerNode is not stored in collection state and therefore it is only
applicable during the time of collection creation."
Except perhaps that it shows up at the bottom of the state.json file
-Gus
On Thu, Jan 17, 2019 at 7:39 PM Gus Heck wrote:
> I had a 8.0.0 cluster strangely put 2 shards on the same node and then 3
> on the same node the next collection I created. Both had maxShardsPerNode
> set to 1 (and it still shows in cluster_state.json).
>
> After some research I found https://jira.apache.org/jira/browse/SOLR-11239
>
> The 8x documentation says nothing about this in the vicinity of the
> maxShardsPerNode attribute, and when I checked I found that by default the
> cluster comes with 2 listeners and two triggers (.auto_add_replicas and
> .scheduled_maintenance). (they seem to be hard coded for automatic addition
> in OverSeerTriggerThread:
>
> @Override
> public void run() {
> int lastZnodeVersion = znodeVersion;
>
> // we automatically add a trigger for auto add replicas if it does not
> exists already
> // we also automatically add a scheduled maintenance trigger
> while (!isClosed) {
> try {
> if (Thread.currentThread().isInterrupted()) {
> log.warn("Interrupted");
> break;
> }
> AutoScalingConfig autoScalingConfig =
> cloudManager.getDistribStateManager().getAutoScalingConfig();
> AutoScalingConfig updatedConfig =
> withAutoAddReplicasTrigger(autoScalingConfig);
> updatedConfig = withScheduledMaintenanceTrigger(updatedConfig);
>
> So I think this means that it's not possible for maxShardsPerNode to ever
> have an effect?
>
> If so, we need to formally remove it and provide a clear example of how to
> achieve the same effect with the autoscaling framework referenced from the
> place where this attribute was documented.
>
> -Gus
>
>
--
http://www.the111shift.com
maxShardsPerNode always over-riden by autoscaling?
I had a 8.0.0 cluster strangely put 2 shards on the same node and then 3 on
the same node the next collection I created. Both had maxShardsPerNode set
to 1 (and it still shows in cluster_state.json).
After some research I found https://jira.apache.org/jira/browse/SOLR-11239
The 8x documentation says nothing about this in the vicinity of the
maxShardsPerNode attribute, and when I checked I found that by default the
cluster comes with 2 listeners and two triggers (.auto_add_replicas and
.scheduled_maintenance). (they seem to be hard coded for automatic addition
in OverSeerTriggerThread:
@Override
public void run() {
int lastZnodeVersion = znodeVersion;
// we automatically add a trigger for auto add replicas if it does not
exists already
// we also automatically add a scheduled maintenance trigger
while (!isClosed) {
try {
if (Thread.currentThread().isInterrupted()) {
log.warn("Interrupted");
break;
}
AutoScalingConfig autoScalingConfig =
cloudManager.getDistribStateManager().getAutoScalingConfig();
AutoScalingConfig updatedConfig =
withAutoAddReplicasTrigger(autoScalingConfig);
updatedConfig = withScheduledMaintenanceTrigger(updatedConfig);
So I think this means that it's not possible for maxShardsPerNode to ever
have an effect?
If so, we need to formally remove it and provide a clear example of how to
achieve the same effect with the autoscaling framework referenced from the
place where this attribute was documented.
-Gus
[jira] [Commented] (SOLR-13145) Provide better error when ref guide deps are not installed
[
https://issues.apache.org/jira/browse/SOLR-13145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745612#comment-16745612
]
Jan Høydahl commented on SOLR-13145:
+1
The error could also refer to README.adoc in that folder
([https://github.com/apache/lucene-solr/blob/master/solr/solr-ref-guide/README.adoc])
which lists the exact requirements for the ref guide build environment. The
build depends on many specific older versions of tools and won't work if you
e.g. use latest/default version of jekyll, so a check for tool existence is not
enough.
What if the ant build inside {{solr-ref-guide}} did a check for all gems and
versions as the very first task?
> Provide better error when ref guide deps are not installed
> --
>
> Key: SOLR-13145
> URL: https://issues.apache.org/jira/browse/SOLR-13145
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Build
>Affects Versions: 8.0
>Reporter: Gus Heck
>Assignee: Gus Heck
>Priority: Trivial
> Attachments: SOLR-13145.patch
>
>
> Recently I tried to build the ref guide only to find I didn't have the deps
> installed on that particular machine (mostly I do solr dev on my desktop
> cause it's faster). The error I got was unenlightening:
> {code:java}
> build-site:
> [java] Exception in thread "main" java.lang.NullPointerException
> [java] at CheckLinksAndAnchors.main(CheckLinksAndAnchors.java:156)
> BUILD FAILED
> {code}
> This happens because CheckLinksAndAnchors.main() is trying to list the files
> in the directory into which Jeckyll didn't put the html and getting null.
> I think the problem here is that we let the build continue when Jeckyll
> failed. Attaching patch to fail out when Jeckyll is not found with this
> message:
> {code:java}
> -build-site:
> [echo] Running Jekyll...
> [exec] rbenv: jekyll: command not found
> BUILD FAILED
> /Users/gus/projects/solr/lucene-solr/solr/solr-ref-guide/build.xml:299: exec
> returned: 127
> {code}
> That at least will point the dev in the right direction, since clearly
> something required is not found.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13145) Provide better error when ref guide deps are not installed
[
https://issues.apache.org/jira/browse/SOLR-13145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745603#comment-16745603
]
Gus Heck commented on SOLR-13145:
-
Patch is extremely simple, will commit tomorrow unless someone has an alternate
suggestion.
> Provide better error when ref guide deps are not installed
> --
>
> Key: SOLR-13145
> URL: https://issues.apache.org/jira/browse/SOLR-13145
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Build
>Affects Versions: 8.0
>Reporter: Gus Heck
>Assignee: Gus Heck
>Priority: Trivial
> Attachments: SOLR-13145.patch
>
>
> Recently I tried to build the ref guide only to find I didn't have the deps
> installed on that particular machine (mostly I do solr dev on my desktop
> cause it's faster). The error I got was unenlightening:
> {code:java}
> build-site:
> [java] Exception in thread "main" java.lang.NullPointerException
> [java] at CheckLinksAndAnchors.main(CheckLinksAndAnchors.java:156)
> BUILD FAILED
> {code}
> This happens because CheckLinksAndAnchors.main() is trying to list the files
> in the directory into which Jeckyll didn't put the html and getting null.
> I think the problem here is that we let the build continue when Jeckyll
> failed. Attaching patch to fail out when Jeckyll is not found with this
> message:
> {code:java}
> -build-site:
> [echo] Running Jekyll...
> [exec] rbenv: jekyll: command not found
> BUILD FAILED
> /Users/gus/projects/solr/lucene-solr/solr/solr-ref-guide/build.xml:299: exec
> returned: 127
> {code}
> That at least will point the dev in the right direction, since clearly
> something required is not found.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-13145) Provide better error when ref guide deps are not installed
[
https://issues.apache.org/jira/browse/SOLR-13145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gus Heck updated SOLR-13145:
Attachment: SOLR-13145.patch
> Provide better error when ref guide deps are not installed
> --
>
> Key: SOLR-13145
> URL: https://issues.apache.org/jira/browse/SOLR-13145
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Build
>Affects Versions: 8.0
>Reporter: Gus Heck
>Assignee: Gus Heck
>Priority: Trivial
> Attachments: SOLR-13145.patch
>
>
> Recently I tried to build the ref guide only to find I didn't have the deps
> installed on that particular machine (mostly I do solr dev on my desktop
> cause it's faster). The error I got was unenlightening:
> {code:java}
> build-site:
> [java] Exception in thread "main" java.lang.NullPointerException
> [java] at CheckLinksAndAnchors.main(CheckLinksAndAnchors.java:156)
> BUILD FAILED
> {code}
> This happens because CheckLinksAndAnchors.main() is trying to list the files
> in the directory into which Jeckyll didn't put the html and getting null.
> I think the problem here is that we let the build continue when Jeckyll
> failed. Attaching patch to fail out when Jeckyll is not found with this
> message:
> {code:java}
> -build-site:
> [echo] Running Jekyll...
> [exec] rbenv: jekyll: command not found
> BUILD FAILED
> /Users/gus/projects/solr/lucene-solr/solr/solr-ref-guide/build.xml:299: exec
> returned: 127
> {code}
> That at least will point the dev in the right direction, since clearly
> something required is not found.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[JENKINS] Lucene-Solr-master-Windows (64bit/jdk-9.0.4) - Build # 7699 - Unstable!
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Windows/7699/ Java: 64bit/jdk-9.0.4 -XX:+UseCompressedOops -XX:+UseSerialGC 1 tests failed. FAILED: junit.framework.TestSuite.org.apache.solr.client.solrj.request.TestCoreAdmin Error Message: Could not remove the following files (in the order of attempts): C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-solrj\test\J1\temp\solr.client.solrj.request.TestCoreAdmin_415EC4309285C204-001\tempDir-009: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-solrj\test\J1\temp\solr.client.solrj.request.TestCoreAdmin_415EC4309285C204-001\tempDir-009 Stack Trace: java.io.IOException: Could not remove the following files (in the order of attempts): C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-solrj\test\J1\temp\solr.client.solrj.request.TestCoreAdmin_415EC4309285C204-001\tempDir-009: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-solrj\test\J1\temp\solr.client.solrj.request.TestCoreAdmin_415EC4309285C204-001\tempDir-009 at __randomizedtesting.SeedInfo.seed([415EC4309285C204]:0) at org.apache.lucene.util.IOUtils.rm(IOUtils.java:319) at org.apache.lucene.util.TestRuleTemporaryFilesCleanup.afterAlways(TestRuleTemporaryFilesCleanup.java:216) at com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterAlways(TestRuleAdapter.java:31) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) at java.base/java.lang.Thread.run(Thread.java:844) Build Log: [...truncated 16601 lines...] [junit4] Suite: org.apache.solr.client.solrj.request.TestCoreAdmin [junit4] 2> Creating dataDir: C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-solrj\test\J1\temp\solr.client.solrj.request.TestCoreAdmin_415EC4309285C204-001\init-core-data-001 [junit4] 2> 338536 INFO (SUITE-TestCoreAdmin-seed#[415EC4309285C204]-worker) [] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true [junit4] 2> 338538 INFO (SUITE-TestCoreAdmin-seed#[415EC4309285C204]-worker) [] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason="", value=0.0/0.0, ssl=0.0/0.0, clientAuth=0.0/0.0) [junit4] 2> 338538 INFO (SUITE-TestCoreAdmin-seed#[415EC4309285C204]-worker) [] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom [junit4] 2> 338555 INFO (TEST-TestCoreAdmin.testCustomUlogDir-seed#[415EC4309285C204]) [] o.a.s.SolrTestCaseJ4 ###Starting testCustomUlogDir [junit4] 1> Solr home: C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-solrj\test\J1\temp\solr.client.solrj.request.TestCoreAdmin_415EC4309285C204-001\solrHome-001 [junit4] 2> 338556 INFO (TEST-TestCoreAdmin.testCustomUlogDir-seed#[415EC4309285C204]) [] o.a.s.c.SolrXmlConfig Loading container configuration from C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-solrj\test\J1\temp\solr.client.solrj.request.TestCoreAdmin_415EC4309285C204-001\solrHome-001\solr.xml [junit4] 2> 338564 INFO (TEST-TestCoreAdmin.testCustomUlogDir-seed#[415EC4309285C204]) [] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@4b431465, but no JMX reporters were configured - adding default JMX reporter. [junit4] 2> 338645 INFO (TEST-TestCoreAdmin.testCustomUlogDir-seed#[415EC4309285C204]) [] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=true] [junit4] 2> 338647 WARN (TEST-TestCoreAdmin.testCustomUlogDir-seed#[415EC4309285C204]) [] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@6f6e8f04[provider=null,keyStore=null,trustStore=null] [junit4] 2> 338651 WARN (TEST-TestCoreAdmin.testCustomUlogDir-seed#[415EC4309285C204]) [] o.e.j.u.s.S.config No Client End
[jira] [Commented] (SOLR-13144) Reference guide section on IndexUpgraderTool needs to be updated
[ https://issues.apache.org/jira/browse/SOLR-13144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745592#comment-16745592 ] Jan Høydahl commented on SOLR-13144: Let's convert this to a sub-task of SOLR-12281 and also add other sub tasks, as I believe we need far more documentation changes than just this one chapter due to the new restrictions. > Reference guide section on IndexUpgraderTool needs to be updated > > > Key: SOLR-13144 > URL: https://issues.apache.org/jira/browse/SOLR-13144 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: documentation >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Minor > Attachments: SOLR-13144.patch > > > The section on IndexUpgrader Tool (and I'll remove the space too) leads to > the conclusion that you can successfully upgrade an index X->X+1->X+2 which > isn't true. I'll attach a proposed doc change in a minute. > I'm thinking of only putting this in master and 8x on the theory that AFAIK, > the enforcement of this restriction is only 8x and later. > This is doc-only, if no objections I'll commit this weekend at the latest. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-13145) Provide better error when ref guide deps are not installed
Gus Heck created SOLR-13145:
---
Summary: Provide better error when ref guide deps are not installed
Key: SOLR-13145
URL: https://issues.apache.org/jira/browse/SOLR-13145
Project: Solr
Issue Type: Improvement
Security Level: Public (Default Security Level. Issues are Public)
Components: Build
Affects Versions: 8.0
Reporter: Gus Heck
Assignee: Gus Heck
Recently I tried to build the ref guide only to find I didn't have the deps
installed on that particular machine (mostly I do solr dev on my desktop cause
it's faster). The error I got was unenlightening:
{code:java}
build-site:
[java] Exception in thread "main" java.lang.NullPointerException
[java] at CheckLinksAndAnchors.main(CheckLinksAndAnchors.java:156)
BUILD FAILED
{code}
This happens because CheckLinksAndAnchors.main() is trying to list the files in
the directory into which Jeckyll didn't put the html and getting null. I
think the problem here is that we let the build continue when Jeckyll failed.
Attaching patch to fail out when Jeckyll is not found with this message:
{code:java}
-build-site:
[echo] Running Jekyll...
[exec] rbenv: jekyll: command not found
BUILD FAILED
/Users/gus/projects/solr/lucene-solr/solr/solr-ref-guide/build.xml:299: exec
returned: 127
{code}
That at least will point the dev in the right direction, since clearly
something required is not found.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-8291) NPE calling export handler when useFilterForSortedQuery=true
[
https://issues.apache.org/jira/browse/SOLR-8291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16744483#comment-16744483
]
Rahul Goswami edited comment on SOLR-8291 at 1/17/19 9:44 PM:
--
Facing the same issue on 7.2.1 while using /export handler.
In my case useFilterForSortedQuery=false and I get the NPE in
ExportWriter.java when the /stream handler is invoked with a search() streaming
expression with qt="/export" containing fq="\{!collapse field=id_field
sort="time desc"} (among other fq's. I tried eliminating one fq at a time to
find the problematic one. The one with collapse parser is what makes it fail).
Below is the stacktrace (Same as Ron's above):
org.apache.solr.servlet.HttpSolrCall null:java.lang.NullPointerException
at org.apache.lucene.util.BitSetIterator.(BitSetIterator.java:61)
at org.apache.solr.handler.ExportWriter.writeDocs(ExportWriter.java:243)
at org.apache.solr.handler.ExportWriter.lambda$null$1(ExportWriter.java:222)
at
org.apache.solr.response.JSONWriter.writeIterator(JSONResponseWriter.java:523)
at
org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:180)
at org.apache.solr.response.JSONWriter$2.put(JSONResponseWriter.java:559)
at org.apache.solr.handler.ExportWriter.lambda$null$2(ExportWriter.java:222)
at org.apache.solr.response.JSONWriter.writeMap(JSONResponseWriter.java:547)
at
org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:198)
at org.apache.solr.response.JSONWriter$2.put(JSONResponseWriter.java:559)
at org.apache.solr.handler.ExportWriter.lambda$write$3(ExportWriter.java:220)
at org.apache.solr.response.JSONWriter.writeMap(JSONResponseWriter.java:547)
at org.apache.solr.handler.ExportWriter.write(ExportWriter.java:218)
at org.apache.solr.core.SolrCore$3.write(SolrCore.java:2627)
at
org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:49)
at org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:788)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:525)
Is Ron's patch the way to go?
was (Author: rahul196...@gmail.com):
Facing the same issue on 7.2.1 while using /export handler.
In my case useFilterForSortedQuery=false and I get the NPE in
ExportWriter.java when the /stream handler is invoked with a search() streaming
expression with qt="/export" containing fq="\{!collapse field=id_field
sort="time desc"} (among other fq's. I tried eliminating one fq at a time to
find the problematic one. The one with collapse parser is what makes it fail)
Is Ron's patch the way to go?
> NPE calling export handler when useFilterForSortedQuery=true
>
>
> Key: SOLR-8291
> URL: https://issues.apache.org/jira/browse/SOLR-8291
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
>Affects Versions: 5.2.1
>Reporter: Ray
>Priority: Major
> Attachments: SOLR-8291.patch, solr.log
>
>
> *Updated*: The stacktrace below was created when the solrconfig.xml has the
> following element:
> {code}
> true
> {code}
> It was determined that useFilterForSortedQuery is incompatible with the
> /export handler.
> See the comments near the end of the ticket for a potential work around if
> this flag needs to be set.
> Get NPE during calling export handler, here is the stack trace:
> at org.apache.lucene.util.BitSetIterator.(BitSetIterator.java:58)
> at
> org.apache.solr.response.SortingResponseWriter.write(SortingResponseWriter.java:138)
> at
> org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:53)
> at
> org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:727)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:459)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:274)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:242)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:274)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:242)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161)
> at
> org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:181)
> at
> org.jboss.modcluster.catali
[jira] [Comment Edited] (SOLR-8291) NPE calling export handler when useFilterForSortedQuery=true
[
https://issues.apache.org/jira/browse/SOLR-8291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16744483#comment-16744483
]
Rahul Goswami edited comment on SOLR-8291 at 1/17/19 9:42 PM:
--
Facing the same issue on 7.2.1 while using /export handler.
In my case useFilterForSortedQuery=false and I get the NPE in
ExportWriter.java when the /stream handler is invoked with a search() streaming
expression with qt="/export" containing fq="\{!collapse field=id_field
sort="time desc"} (among other fq's. I tried eliminating one fq at a time to
find the problematic one. The one with collapse parser is what makes it fail)
Is Ron's patch the way to go?
was (Author: rahul196...@gmail.com):
Facing the same issue on 7.2.1 while using /export handler.
In my case useFilterForSortedQuery=false and I get the NPE in ExportWriter.java
when the stream request with /export handler contains an fq="\{!collapse
field=id_field sort="time desc"}
Is Ron's patch the way to go?
> NPE calling export handler when useFilterForSortedQuery=true
>
>
> Key: SOLR-8291
> URL: https://issues.apache.org/jira/browse/SOLR-8291
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
>Affects Versions: 5.2.1
>Reporter: Ray
>Priority: Major
> Attachments: SOLR-8291.patch, solr.log
>
>
> *Updated*: The stacktrace below was created when the solrconfig.xml has the
> following element:
> {code}
> true
> {code}
> It was determined that useFilterForSortedQuery is incompatible with the
> /export handler.
> See the comments near the end of the ticket for a potential work around if
> this flag needs to be set.
> Get NPE during calling export handler, here is the stack trace:
> at org.apache.lucene.util.BitSetIterator.(BitSetIterator.java:58)
> at
> org.apache.solr.response.SortingResponseWriter.write(SortingResponseWriter.java:138)
> at
> org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:53)
> at
> org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:727)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:459)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:274)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:242)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:274)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:242)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161)
> at
> org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:181)
> at
> org.jboss.modcluster.catalina.CatalinaContext$RequestListenerValve.event(CatalinaContext.java:285)
> at
> org.jboss.modcluster.catalina.CatalinaContext$RequestListenerValve.invoke(CatalinaContext.java:261)
> at
> org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:88)
> at
> org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:100)
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:159)
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
> at
> org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)
> at
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:567)
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
> at
> org.jboss.web.tomcat.service.request.ActiveRequestResponseCacheValve.invoke(ActiveRequestResponseCacheValve.java:53)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:362)
> at
> org.apache.coyote.ajp.AjpAprProcessor.process(AjpAprProcessor.java:489)
> at
> org.apache.coyote.ajp.AjpAprProtocol$AjpConnectionHandler.process(AjpAprProtocol.java:452)
> at
> org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:2019)
> at java.lang.Thread.run(Thread.java:745)
> It seems there are some FixedBitSet was set to null
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e
[jira] [Comment Edited] (SOLR-8291) NPE calling export handler when useFilterForSortedQuery=true
[
https://issues.apache.org/jira/browse/SOLR-8291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16744483#comment-16744483
]
Rahul Goswami edited comment on SOLR-8291 at 1/17/19 9:36 PM:
--
Facing the same issue on 7.2.1 while using /export handler.
In my case useFilterForSortedQuery=false and I get the NPE in ExportWriter.java
when the stream request with /export handler contains an fq="\{!collapse
field=id_field sort="time desc"}
Is Ron's patch the way to go?
was (Author: rahul196...@gmail.com):
Facing the same issue on 7.2.1 while using /export handler... Is Ron's patch
the way to go?
> NPE calling export handler when useFilterForSortedQuery=true
>
>
> Key: SOLR-8291
> URL: https://issues.apache.org/jira/browse/SOLR-8291
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
>Affects Versions: 5.2.1
>Reporter: Ray
>Priority: Major
> Attachments: SOLR-8291.patch, solr.log
>
>
> *Updated*: The stacktrace below was created when the solrconfig.xml has the
> following element:
> {code}
> true
> {code}
> It was determined that useFilterForSortedQuery is incompatible with the
> /export handler.
> See the comments near the end of the ticket for a potential work around if
> this flag needs to be set.
> Get NPE during calling export handler, here is the stack trace:
> at org.apache.lucene.util.BitSetIterator.(BitSetIterator.java:58)
> at
> org.apache.solr.response.SortingResponseWriter.write(SortingResponseWriter.java:138)
> at
> org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:53)
> at
> org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:727)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:459)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:274)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:242)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:274)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:242)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161)
> at
> org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:181)
> at
> org.jboss.modcluster.catalina.CatalinaContext$RequestListenerValve.event(CatalinaContext.java:285)
> at
> org.jboss.modcluster.catalina.CatalinaContext$RequestListenerValve.invoke(CatalinaContext.java:261)
> at
> org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:88)
> at
> org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:100)
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:159)
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
> at
> org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)
> at
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:567)
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
> at
> org.jboss.web.tomcat.service.request.ActiveRequestResponseCacheValve.invoke(ActiveRequestResponseCacheValve.java:53)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:362)
> at
> org.apache.coyote.ajp.AjpAprProcessor.process(AjpAprProcessor.java:489)
> at
> org.apache.coyote.ajp.AjpAprProtocol$AjpConnectionHandler.process(AjpAprProtocol.java:452)
> at
> org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:2019)
> at java.lang.Thread.run(Thread.java:745)
> It seems there are some FixedBitSet was set to null
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[JENKINS] Lucene-Solr-8.x-Solaris (64bit/jdk1.8.0) - Build # 12 - Unstable!
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-8.x-Solaris/12/ Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseSerialGC 2 tests failed. FAILED: org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration.testEventQueue Error Message: action wasn't interrupted Stack Trace: java.lang.AssertionError: action wasn't interrupted at __randomizedtesting.SeedInfo.seed([6A2867FFC357BF29:A39D2551CA3079DC]:0) at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.assertTrue(Assert.java:41) at org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration.testEventQueue(TestSimTriggerIntegration.java:718) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938) at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974) at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) at java.lang.Thread.run(Thread.java:748) FAILED: org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration.testNodeMarkersRegistration Error Message: Path /autoscaling/nodeAdded/127.0.0.1:10029_solr wasn't created Stack Trace: java.lang.AssertionError: Path /autoscaling/nodeAdded/127.0.0.1:1002
[jira] [Created] (LUCENE-8647) Create a generic method in the Explanation class to get explanation attributes
Sambhav Kothari created LUCENE-8647: --- Summary: Create a generic method in the Explanation class to get explanation attributes Key: LUCENE-8647 URL: https://issues.apache.org/jira/browse/LUCENE-8647 Project: Lucene - Core Issue Type: Improvement Components: core/search Affects Versions: 7.5 Reporter: Sambhav Kothari Initial proposal can be found here - [http://mail-archives.apache.org/mod_mbox/lucene-java-user/201901.mbox/%3CCADitDkmnv4Vup5VG2ZwLUOv5VM=itu-y8eugxxmtoppzpcy...@mail.gmail.com%3E] Currently the Explanation class has a toHtml and toString method, but it doesn't seem to have any method to output the explanation as a NamedList. Part of the reason is that NamedList is a Solr only concept and it cannot exist in lucene. But this leads to utility functions like (https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302214165a4d/solr/core/src/java/org/apache/solr/response/transform/ExplainAugmenterFactory.java#L115 and https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302214165a4d/solr/core/src/java/org/apache/solr/util/SolrPluginUtils.java#L400-L466 which assume a particular structure about explanations. Explanation should instead have a toMap method which returns a Map of key-value pairs which can then be converted to a NamedList in Solr. This can be exceptionally useful if we have a class inheriting from Explanation that adds more attributes. For example for Learning to Rank model explanations we can have a ModelExplanation class which adds doc-specific feature scores and other machine learning model explanations to the output. It also allows the ExplainAugmenter and other classes to have a more generic way of getting an Explanation's data in a structured format. The way it currently does so is very ugly - https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302214165a4d/solr/core/src/java/org/apache/solr/response/transform/ExplainAugmenterFactory.java#L115 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-8647) Create a generic method in the Explanation class to get explanation attributes
[ https://issues.apache.org/jira/browse/LUCENE-8647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sambhav Kothari updated LUCENE-8647: Description: Initial proposal and discussion can be found here - [http://mail-archives.apache.org/mod_mbox/lucene-java-user/201901.mbox/%3CCADitDkmnv4Vup5VG2ZwLUOv5VM=itu-y8eugxxmtoppzpcy...@mail.gmail.com%3E] Currently the Explanation class has a toString method, but it doesn't seem to have any method to output the explanation as a NamedList. Part of the reason is that NamedList is a Solr only concept and it cannot exist in lucene. But this leads to utility functions like ([https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302214165a4d/solr/core/src/java/org/apache/solr/response/transform/ExplainAugmenterFactory.java#L115] and [https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302214165a4d/solr/core/src/java/org/apache/solr/util/SolrPluginUtils.java#L400-L466] which assume a particular structure about explanations. Explanation should instead have a toMap method which returns a Map of key-value pairs which can then be converted to a NamedList in Solr. This can be exceptionally useful if we have a class inheriting from Explanation that adds more attributes. For example for Learning to Rank model explanations we can have a ModelExplanation class which adds doc-specific feature scores and other machine learning model explanations to the output. It also allows the ExplainAugmenter and other classes to have a more generic way of getting an Explanation's data in a structured format. The way it currently does so is very ugly - [https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302214165a4d/solr/core/src/java/org/apache/solr/response/transform/ExplainAugmenterFactory.java#L115] This also implies that we shouldn't keep Explanation as a final class anymore. Storing and parsing things like feature values in the explanation description doesn't seem like the right way. was: Initial proposal and discussion can be found here - [http://mail-archives.apache.org/mod_mbox/lucene-java-user/201901.mbox/%3CCADitDkmnv4Vup5VG2ZwLUOv5VM=itu-y8eugxxmtoppzpcy...@mail.gmail.com%3E] Currently the Explanation class has a toString method, but it doesn't seem to have any method to output the explanation as a NamedList. Part of the reason is that NamedList is a Solr only concept and it cannot exist in lucene. But this leads to utility functions like ([https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302214165a4d/solr/core/src/java/org/apache/solr/response/transform/ExplainAugmenterFactory.java#L115] and [https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302214165a4d/solr/core/src/java/org/apache/solr/util/SolrPluginUtils.java#L400-L466] which assume a particular structure about explanations. Explanation should instead have a toMap method which returns a Map of key-value pairs which can then be converted to a NamedList in Solr. This can be exceptionally useful if we have a class inheriting from Explanation that adds more attributes. For example for Learning to Rank model explanations we can have a ModelExplanation class which adds doc-specific feature scores and other machine learning model explanations to the output. It also allows the ExplainAugmenter and other classes to have a more generic way of getting an Explanation's data in a structured format. The way it currently does so is very ugly - [https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302214165a4d/solr/core/src/java/org/apache/solr/response/transform/ExplainAugmenterFactory.java#L115] This also implies that we shouldn't keep Explanation as a final class anymore. Storing and parsing things in explanation description doesn't seem like the right way. > Create a generic method in the Explanation class to get explanation attributes > -- > > Key: LUCENE-8647 > URL: https://issues.apache.org/jira/browse/LUCENE-8647 > Project: Lucene - Core > Issue Type: Improvement > Components: core/search >Affects Versions: 7.5 >Reporter: Sambhav Kothari >Priority: Minor > > Initial proposal and discussion can be found here - > > [http://mail-archives.apache.org/mod_mbox/lucene-java-user/201901.mbox/%3CCADitDkmnv4Vup5VG2ZwLUOv5VM=itu-y8eugxxmtoppzpcy...@mail.gmail.com%3E] > > Currently the Explanation class has a toString method, but it doesn't seem to > have any method to output the explanation as a NamedList. Part of the reason > is that NamedList is a Solr only concept and it cannot exist in lucene. But > this leads to utility functions like > ([https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea13
[jira] [Comment Edited] (LUCENE-8647) Create a generic method in the Explanation class to get explanation attributes
[
https://issues.apache.org/jira/browse/LUCENE-8647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745452#comment-16745452
]
Sambhav Kothari edited comment on LUCENE-8647 at 1/17/19 8:04 PM:
--
Example implementation of a possibly useful subclass in contrib/ltr -
{code:java}
/**
* A special explanation class that stores feature scores for a specific doc
that a model is scoring.
*/
public class ModelExplanation extends Explanation {
private Map featureImportance = new HashMap<>();
private Map featureValues = new HashMap<>();
public Map getFeatureImportance() {
return featureImportance;
}
public void setFeatureImportance(Map featureImportance) {
this.featureImportance = featureImportance;
}
public Map getFeatureValues() {
return featureValues;
}
public void setFeatureValues(Map featureValues) {
this.featureValues = featureValues;
}
@Override
public LinkedHashMap toMap() {
LinkedHashMap data = super.toMap();
data.put("featureImportance", getFeatureImportance());
data.put("featureValues", getFeatureValues());
return data;
}
}
{code}
was (Author: samj1912):
Example implementation of a subclass -
{code:java}
/**
* A special explanation class that stores feature scores for a specific doc
that a model is scoring.
*/
public class ModelExplanation extends Explanation {
private Map featureImportance = new HashMap<>();
private Map featureValues = new HashMap<>();
public Map getFeatureImportance() {
return featureImportance;
}
public void setFeatureImportance(Map featureImportance) {
this.featureImportance = featureImportance;
}
public Map getFeatureValues() {
return featureValues;
}
public void setFeatureValues(Map featureValues) {
this.featureValues = featureValues;
}
@Override
public LinkedHashMap toMap() {
LinkedHashMap data = super.toMap();
data.put("featureImportance", getFeatureImportance());
data.put("featureValues", getFeatureValues());
return data;
}
}
{code}
> Create a generic method in the Explanation class to get explanation attributes
> --
>
> Key: LUCENE-8647
> URL: https://issues.apache.org/jira/browse/LUCENE-8647
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/search
>Affects Versions: 7.5
>Reporter: Sambhav Kothari
>Priority: Minor
>
> Initial proposal and discussion can be found here -
>
> [http://mail-archives.apache.org/mod_mbox/lucene-java-user/201901.mbox/%3CCADitDkmnv4Vup5VG2ZwLUOv5VM=itu-y8eugxxmtoppzpcy...@mail.gmail.com%3E]
>
> Currently the Explanation class has a toString method, but it doesn't seem to
> have any method to output the explanation as a NamedList. Part of the reason
> is that NamedList is a Solr only concept and it cannot exist in lucene. But
> this leads to utility functions like
> ([https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302214165a4d/solr/core/src/java/org/apache/solr/response/transform/ExplainAugmenterFactory.java#L115]
> and
> [https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302214165a4d/solr/core/src/java/org/apache/solr/util/SolrPluginUtils.java#L400-L466]
> which assume a particular structure about explanations.
>
> Explanation should instead have a toMap method which returns a Map of
> key-value pairs which can then be converted to a NamedList in Solr. This can
> be exceptionally useful if we have a class inheriting from Explanation that
> adds more attributes.
>
> For example for Learning to Rank model explanations we can have a
> ModelExplanation class which adds doc-specific feature scores and other
> machine learning model explanations to the output. It also allows the
> ExplainAugmenter and other classes to have a more generic way of getting an
> Explanation's data in a structured format.
>
> The way it currently does so is very ugly -
> [https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302214165a4d/solr/core/src/java/org/apache/solr/response/transform/ExplainAugmenterFactory.java#L115]
>
> This also implies that we shouldn't keep Explanation as a final class
> anymore. Storing and parsing things like feature values in the explanation
> description doesn't seem like the right way.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8647) Create a generic method in the Explanation class to get explanation attributes
[
https://issues.apache.org/jira/browse/LUCENE-8647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745452#comment-16745452
]
Sambhav Kothari commented on LUCENE-8647:
-
Example implementation of a subclass -
{code:java}
/**
* A special explanation class that stores feature scores for a specific doc
that a model is scoring.
*/
public class ModelExplanation extends Explanation {
private Map featureImportance = new HashMap<>();
private Map featureValues = new HashMap<>();
public Map getFeatureImportance() {
return featureImportance;
}
public void setFeatureImportance(Map featureImportance) {
this.featureImportance = featureImportance;
}
public Map getFeatureValues() {
return featureValues;
}
public void setFeatureValues(Map featureValues) {
this.featureValues = featureValues;
}
@Override
public LinkedHashMap toMap() {
LinkedHashMap data = super.toMap();
data.put("featureImportance", getFeatureImportance());
data.put("featureValues", getFeatureValues());
return data;
}
}
{code}
> Create a generic method in the Explanation class to get explanation attributes
> --
>
> Key: LUCENE-8647
> URL: https://issues.apache.org/jira/browse/LUCENE-8647
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/search
>Affects Versions: 7.5
>Reporter: Sambhav Kothari
>Priority: Minor
>
> Initial proposal and discussion can be found here -
>
> [http://mail-archives.apache.org/mod_mbox/lucene-java-user/201901.mbox/%3CCADitDkmnv4Vup5VG2ZwLUOv5VM=itu-y8eugxxmtoppzpcy...@mail.gmail.com%3E]
>
> Currently the Explanation class has a toString method, but it doesn't seem to
> have any method to output the explanation as a NamedList. Part of the reason
> is that NamedList is a Solr only concept and it cannot exist in lucene. But
> this leads to utility functions like
> ([https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302214165a4d/solr/core/src/java/org/apache/solr/response/transform/ExplainAugmenterFactory.java#L115]
> and
> [https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302214165a4d/solr/core/src/java/org/apache/solr/util/SolrPluginUtils.java#L400-L466]
> which assume a particular structure about explanations.
>
> Explanation should instead have a toMap method which returns a Map of
> key-value pairs which can then be converted to a NamedList in Solr. This can
> be exceptionally useful if we have a class inheriting from Explanation that
> adds more attributes.
>
> For example for Learning to Rank model explanations we can have a
> ModelExplanation class which adds doc-specific feature scores and other
> machine learning model explanations to the output. It also allows the
> ExplainAugmenter and other classes to have a more generic way of getting an
> Explanation's data in a structured format.
>
> The way it currently does so is very ugly -
> [https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302214165a4d/solr/core/src/java/org/apache/solr/response/transform/ExplainAugmenterFactory.java#L115]
>
> This also implies that we shouldn't keep Explanation as a final class
> anymore. Storing and parsing things like feature values in the explanation
> description doesn't seem like the right way.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-8647) Create a generic method in the Explanation class to get explanation attributes
[ https://issues.apache.org/jira/browse/LUCENE-8647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sambhav Kothari updated LUCENE-8647: Description: Initial proposal and discussion can be found here - [http://mail-archives.apache.org/mod_mbox/lucene-java-user/201901.mbox/%3CCADitDkmnv4Vup5VG2ZwLUOv5VM=itu-y8eugxxmtoppzpcy...@mail.gmail.com%3E] Currently the Explanation class has a toString method, but it doesn't seem to have any method to output the explanation as a NamedList. Part of the reason is that NamedList is a Solr only concept and it cannot exist in lucene. But this leads to utility functions like ([https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302214165a4d/solr/core/src/java/org/apache/solr/response/transform/ExplainAugmenterFactory.java#L115] and [https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302214165a4d/solr/core/src/java/org/apache/solr/util/SolrPluginUtils.java#L400-L466] which assume a particular structure about explanations. Explanation should instead have a toMap method which returns a Map of key-value pairs which can then be converted to a NamedList in Solr. This can be exceptionally useful if we have a class inheriting from Explanation that adds more attributes. For example for Learning to Rank model explanations we can have a ModelExplanation class which adds doc-specific feature scores and other machine learning model explanations to the output. It also allows the ExplainAugmenter and other classes to have a more generic way of getting an Explanation's data in a structured format. The way it currently does so is very ugly - [https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302214165a4d/solr/core/src/java/org/apache/solr/response/transform/ExplainAugmenterFactory.java#L115] This also implies that we shouldn't keep Explanation as a final class anymore. Storing and parsing things in explanation description doesn't seem like the right way. was: Initial proposal can be found here - [http://mail-archives.apache.org/mod_mbox/lucene-java-user/201901.mbox/%3CCADitDkmnv4Vup5VG2ZwLUOv5VM=itu-y8eugxxmtoppzpcy...@mail.gmail.com%3E] Currently the Explanation class has a toString method, but it doesn't seem to have any method to output the explanation as a NamedList. Part of the reason is that NamedList is a Solr only concept and it cannot exist in lucene. But this leads to utility functions like ([https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302214165a4d/solr/core/src/java/org/apache/solr/response/transform/ExplainAugmenterFactory.java#L115] and [https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302214165a4d/solr/core/src/java/org/apache/solr/util/SolrPluginUtils.java#L400-L466] which assume a particular structure about explanations. Explanation should instead have a toMap method which returns a Map of key-value pairs which can then be converted to a NamedList in Solr. This can be exceptionally useful if we have a class inheriting from Explanation that adds more attributes. For example for Learning to Rank model explanations we can have a ModelExplanation class which adds doc-specific feature scores and other machine learning model explanations to the output. It also allows the ExplainAugmenter and other classes to have a more generic way of getting an Explanation's data in a structured format. The way it currently does so is very ugly - [https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302214165a4d/solr/core/src/java/org/apache/solr/response/transform/ExplainAugmenterFactory.java#L115] This also implies that we shouldn't keep Explanation as a final class anymore. Storing and parsing things in explanation description doesn't seem like the right way. > Create a generic method in the Explanation class to get explanation attributes > -- > > Key: LUCENE-8647 > URL: https://issues.apache.org/jira/browse/LUCENE-8647 > Project: Lucene - Core > Issue Type: Improvement > Components: core/search >Affects Versions: 7.5 >Reporter: Sambhav Kothari >Priority: Minor > > Initial proposal and discussion can be found here - > > [http://mail-archives.apache.org/mod_mbox/lucene-java-user/201901.mbox/%3CCADitDkmnv4Vup5VG2ZwLUOv5VM=itu-y8eugxxmtoppzpcy...@mail.gmail.com%3E] > > Currently the Explanation class has a toString method, but it doesn't seem to > have any method to output the explanation as a NamedList. Part of the reason > is that NamedList is a Solr only concept and it cannot exist in lucene. But > this leads to utility functions like > ([https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302214165a4d/solr/core/src/java
[jira] [Updated] (LUCENE-8647) Create a generic method in the Explanation class to get explanation attributes
[ https://issues.apache.org/jira/browse/LUCENE-8647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sambhav Kothari updated LUCENE-8647: Description: Initial proposal can be found here - [http://mail-archives.apache.org/mod_mbox/lucene-java-user/201901.mbox/%3CCADitDkmnv4Vup5VG2ZwLUOv5VM=itu-y8eugxxmtoppzpcy...@mail.gmail.com%3E] Currently the Explanation class has a toString method, but it doesn't seem to have any method to output the explanation as a NamedList. Part of the reason is that NamedList is a Solr only concept and it cannot exist in lucene. But this leads to utility functions like ([https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302214165a4d/solr/core/src/java/org/apache/solr/response/transform/ExplainAugmenterFactory.java#L115] and [https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302214165a4d/solr/core/src/java/org/apache/solr/util/SolrPluginUtils.java#L400-L466] which assume a particular structure about explanations. Explanation should instead have a toMap method which returns a Map of key-value pairs which can then be converted to a NamedList in Solr. This can be exceptionally useful if we have a class inheriting from Explanation that adds more attributes. For example for Learning to Rank model explanations we can have a ModelExplanation class which adds doc-specific feature scores and other machine learning model explanations to the output. It also allows the ExplainAugmenter and other classes to have a more generic way of getting an Explanation's data in a structured format. The way it currently does so is very ugly - [https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302214165a4d/solr/core/src/java/org/apache/solr/response/transform/ExplainAugmenterFactory.java#L115] This also implies that we shouldn't keep Explanation as a final class anymore. Storing and parsing things in explanation description doesn't seem like the right way. was: Initial proposal can be found here - [http://mail-archives.apache.org/mod_mbox/lucene-java-user/201901.mbox/%3CCADitDkmnv4Vup5VG2ZwLUOv5VM=itu-y8eugxxmtoppzpcy...@mail.gmail.com%3E] Currently the Explanation class has a toString method, but it doesn't seem to have any method to output the explanation as a NamedList. Part of the reason is that NamedList is a Solr only concept and it cannot exist in lucene. But this leads to utility functions like ([https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302214165a4d/solr/core/src/java/org/apache/solr/response/transform/ExplainAugmenterFactory.java#L115] and [https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302214165a4d/solr/core/src/java/org/apache/solr/util/SolrPluginUtils.java#L400-L466] which assume a particular structure about explanations. Explanation should instead have a toMap method which returns a Map of key-value pairs which can then be converted to a NamedList in Solr. This can be exceptionally useful if we have a class inheriting from Explanation that adds more attributes. For example for Learning to Rank model explanations we can have a ModelExplanation class which adds doc-specific feature scores and other machine learning model explanations to the output. It also allows the ExplainAugmenter and other classes to have a more generic way of getting an Explanation's data in a structured format. The way it currently does so is very ugly - [https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302214165a4d/solr/core/src/java/org/apache/solr/response/transform/ExplainAugmenterFactory.java#L115] > Create a generic method in the Explanation class to get explanation attributes > -- > > Key: LUCENE-8647 > URL: https://issues.apache.org/jira/browse/LUCENE-8647 > Project: Lucene - Core > Issue Type: Improvement > Components: core/search >Affects Versions: 7.5 >Reporter: Sambhav Kothari >Priority: Minor > > Initial proposal can be found here - > > [http://mail-archives.apache.org/mod_mbox/lucene-java-user/201901.mbox/%3CCADitDkmnv4Vup5VG2ZwLUOv5VM=itu-y8eugxxmtoppzpcy...@mail.gmail.com%3E] > > Currently the Explanation class has a toString method, but it doesn't seem to > have any method to output the explanation as a NamedList. Part of the reason > is that NamedList is a Solr only concept and it cannot exist in lucene. But > this leads to utility functions like > ([https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302214165a4d/solr/core/src/java/org/apache/solr/response/transform/ExplainAugmenterFactory.java#L115] > and > [https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302214165a4d/solr/core/src/java/org/apache/solr/u
[jira] [Updated] (LUCENE-8647) Create a generic method in the Explanation class to get explanation attributes
[ https://issues.apache.org/jira/browse/LUCENE-8647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sambhav Kothari updated LUCENE-8647: Description: Initial proposal can be found here - [http://mail-archives.apache.org/mod_mbox/lucene-java-user/201901.mbox/%3CCADitDkmnv4Vup5VG2ZwLUOv5VM=itu-y8eugxxmtoppzpcy...@mail.gmail.com%3E] Currently the Explanation class has a toString method, but it doesn't seem to have any method to output the explanation as a NamedList. Part of the reason is that NamedList is a Solr only concept and it cannot exist in lucene. But this leads to utility functions like ([https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302214165a4d/solr/core/src/java/org/apache/solr/response/transform/ExplainAugmenterFactory.java#L115] and [https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302214165a4d/solr/core/src/java/org/apache/solr/util/SolrPluginUtils.java#L400-L466] which assume a particular structure about explanations. Explanation should instead have a toMap method which returns a Map of key-value pairs which can then be converted to a NamedList in Solr. This can be exceptionally useful if we have a class inheriting from Explanation that adds more attributes. For example for Learning to Rank model explanations we can have a ModelExplanation class which adds doc-specific feature scores and other machine learning model explanations to the output. It also allows the ExplainAugmenter and other classes to have a more generic way of getting an Explanation's data in a structured format. The way it currently does so is very ugly - [https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302214165a4d/solr/core/src/java/org/apache/solr/response/transform/ExplainAugmenterFactory.java#L115] was: Initial proposal can be found here - [http://mail-archives.apache.org/mod_mbox/lucene-java-user/201901.mbox/%3CCADitDkmnv4Vup5VG2ZwLUOv5VM=itu-y8eugxxmtoppzpcy...@mail.gmail.com%3E] Currently the Explanation class has a toHtml and toString method, but it doesn't seem to have any method to output the explanation as a NamedList. Part of the reason is that NamedList is a Solr only concept and it cannot exist in lucene. But this leads to utility functions like (https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302214165a4d/solr/core/src/java/org/apache/solr/response/transform/ExplainAugmenterFactory.java#L115 and https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302214165a4d/solr/core/src/java/org/apache/solr/util/SolrPluginUtils.java#L400-L466 which assume a particular structure about explanations. Explanation should instead have a toMap method which returns a Map of key-value pairs which can then be converted to a NamedList in Solr. This can be exceptionally useful if we have a class inheriting from Explanation that adds more attributes. For example for Learning to Rank model explanations we can have a ModelExplanation class which adds doc-specific feature scores and other machine learning model explanations to the output. It also allows the ExplainAugmenter and other classes to have a more generic way of getting an Explanation's data in a structured format. The way it currently does so is very ugly - https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302214165a4d/solr/core/src/java/org/apache/solr/response/transform/ExplainAugmenterFactory.java#L115 > Create a generic method in the Explanation class to get explanation attributes > -- > > Key: LUCENE-8647 > URL: https://issues.apache.org/jira/browse/LUCENE-8647 > Project: Lucene - Core > Issue Type: Improvement > Components: core/search >Affects Versions: 7.5 >Reporter: Sambhav Kothari >Priority: Minor > > Initial proposal can be found here - > > [http://mail-archives.apache.org/mod_mbox/lucene-java-user/201901.mbox/%3CCADitDkmnv4Vup5VG2ZwLUOv5VM=itu-y8eugxxmtoppzpcy...@mail.gmail.com%3E] > > Currently the Explanation class has a toString method, but it doesn't seem to > have any method to output the explanation as a NamedList. Part of the reason > is that NamedList is a Solr only concept and it cannot exist in lucene. But > this leads to utility functions like > ([https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302214165a4d/solr/core/src/java/org/apache/solr/response/transform/ExplainAugmenterFactory.java#L115] > and > [https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302214165a4d/solr/core/src/java/org/apache/solr/util/SolrPluginUtils.java#L400-L466] > which assume a particular structure about explanations. > > Explanation should instead have a toMap method which returns a Map
[jira] [Commented] (SOLR-13143) Output from ExplainAugmenterFactory and DebugQuery for rerank queries is not same
[
https://issues.apache.org/jira/browse/SOLR-13143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745425#comment-16745425
]
Sambhav Kothari commented on SOLR-13143:
They are in the same ballpark, but different issues IMHO. The scores are
correct for re-ranking. It was only the explain doc transformer that was giving
the wrong output.
> Output from ExplainAugmenterFactory and DebugQuery for rerank queries is not
> same
> -
>
> Key: SOLR-13143
> URL: https://issues.apache.org/jira/browse/SOLR-13143
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: search
>Affects Versions: 7.5
>Reporter: Sambhav Kothari
>Priority: Minor
> Attachments: bug.patch
>
>
> Currently, if we use the ExplainAugmenterFactory with LtR, instead of using
> the
> model/re-rankers explain method, it uses the default query explain (tf-idf
> explanation). This happens because the BasicResultContext doesn't wrap the
>
> query([https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302]
>
> 214165a4d/solr/core/src/java/org/apache/solr/response/BasicResultContext.java#L6
> 7) with the RankQuery when its set to context's query, which is then used by
> the ExplainAugmenterFactory.
>
> ([https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c45230221416]
>
> 5a4d/solr/core/src/java/org/apache/solr/response/transform/ExplainAugmenterFacto
> ry.java#L111).
> As a result there are discrepancies between queries like -
> [http://localhost:8983/solr/collection1/select?q=*:*&collection=collectionName&wt]
> =json&fl=[explain style=nl],score&rq=\{!ltr model=linear-model}
> [http://localhost:8983/solr/collection1/select?q=*:*&collection=collectionName&wt]
> =json&fl=score&rq=\{!ltr model=linear-model}&debugQuery=true
> the former outputs the explain from the SimilarityScorer's explain while the
> latter uses the correct LtR ModelScorer's explain.
> There are a few other problems with the explain augmenter - for eg. it
> doesn't
> work with grouping (although the other doc transformers like LtR's
> LTRFeatureLoggerTransformerFactory work with grouping)
> Mailing List -
>
> http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201901.mbox/%3C5C3879D702C30144003902B0_0_18802%40msllnjpmsgsv06%3E
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[JENKINS] Lucene-Solr-NightlyTests-8.x - Build # 2 - Still Unstable
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-8.x/2/
5 tests failed.
FAILED: org.apache.solr.cloud.UnloadDistributedZkTest.test
Error Message:
Captured an uncaught exception in thread: Thread[id=194737,
name=testExecutor-15307-thread-18, state=RUNNABLE,
group=TGRP-UnloadDistributedZkTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught
exception in thread: Thread[id=194737, name=testExecutor-15307-thread-18,
state=RUNNABLE, group=TGRP-UnloadDistributedZkTest]
Caused by: java.lang.RuntimeException:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting
response from server at: http://127.0.0.1:39746/_
at __randomizedtesting.SeedInfo.seed([19B0826919B86C5E]:0)
at
org.apache.solr.cloud.BasicDistributedZkTest.lambda$createCollectionInOneInstance$1(BasicDistributedZkTest.java:659)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.client.solrj.SolrServerException: Timeout occured
while waiting response from server at: http://127.0.0.1:39746/_
at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:661)
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:256)
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:245)
at
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:207)
at
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:224)
at
org.apache.solr.cloud.BasicDistributedZkTest.lambda$createCollectionInOneInstance$1(BasicDistributedZkTest.java:657)
... 4 more
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at
org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
at
org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
at
org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
at
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
at
org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
at
org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
at
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
at
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
at
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
at
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:549)
... 9 more
FAILED:
org.apache.solr.cloud.api.collections.CollectionsAPIDistributedZkTest.testCollectionsAPI
Error Message:
Failed while waiting for active collection Timeout waiting to see state for
collection=awhollynewcollection_0
:DocCollection(awhollynewcollection_0//collections/awhollynewcollection_0/state.json/8)={
"pullReplicas":"0", "replicationFactor":"4", "shards":{ "shard1":{
"range":"8000-bfff", "state":"active", "replicas":{
"core_node4":{ "core":"awhollynewcollection_0_shard1_replica_n1",
"base_url":"http://127.0.0.1:39109/solr";,
"node_name":"127.0.0.1:39109_solr", "state":"down",
"type":"NRT", "force_set_state":"false"}, "core_node8":{
"core":"awhollynewcollection_0_shard1_replica_n2",
"base_url":"http://1
[jira] [Commented] (SOLR-13143) Output from ExplainAugmenterFactory and DebugQuery for rerank queries is not same
[
https://issues.apache.org/jira/browse/SOLR-13143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745387#comment-16745387
]
Erick Erickson commented on SOLR-13143:
---
Is this the same as SOLR-13126?
> Output from ExplainAugmenterFactory and DebugQuery for rerank queries is not
> same
> -
>
> Key: SOLR-13143
> URL: https://issues.apache.org/jira/browse/SOLR-13143
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: search
>Affects Versions: 7.5
>Reporter: Sambhav Kothari
>Priority: Minor
> Attachments: bug.patch
>
>
> Currently, if we use the ExplainAugmenterFactory with LtR, instead of using
> the
> model/re-rankers explain method, it uses the default query explain (tf-idf
> explanation). This happens because the BasicResultContext doesn't wrap the
>
> query([https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302]
>
> 214165a4d/solr/core/src/java/org/apache/solr/response/BasicResultContext.java#L6
> 7) with the RankQuery when its set to context's query, which is then used by
> the ExplainAugmenterFactory.
>
> ([https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c45230221416]
>
> 5a4d/solr/core/src/java/org/apache/solr/response/transform/ExplainAugmenterFacto
> ry.java#L111).
> As a result there are discrepancies between queries like -
> [http://localhost:8983/solr/collection1/select?q=*:*&collection=collectionName&wt]
> =json&fl=[explain style=nl],score&rq=\{!ltr model=linear-model}
> [http://localhost:8983/solr/collection1/select?q=*:*&collection=collectionName&wt]
> =json&fl=score&rq=\{!ltr model=linear-model}&debugQuery=true
> the former outputs the explain from the SimilarityScorer's explain while the
> latter uses the correct LtR ModelScorer's explain.
> There are a few other problems with the explain augmenter - for eg. it
> doesn't
> work with grouping (although the other doc transformers like LtR's
> LTRFeatureLoggerTransformerFactory work with grouping)
> Mailing List -
>
> http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201901.mbox/%3C5C3879D702C30144003902B0_0_18802%40msllnjpmsgsv06%3E
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13143) Output from ExplainAugmenterFactory and DebugQuery for rerank queries is not same
[
https://issues.apache.org/jira/browse/SOLR-13143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745388#comment-16745388
]
Erick Erickson commented on SOLR-13143:
---
These are at least in the same ballpark.
> Output from ExplainAugmenterFactory and DebugQuery for rerank queries is not
> same
> -
>
> Key: SOLR-13143
> URL: https://issues.apache.org/jira/browse/SOLR-13143
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: search
>Affects Versions: 7.5
>Reporter: Sambhav Kothari
>Priority: Minor
> Attachments: bug.patch
>
>
> Currently, if we use the ExplainAugmenterFactory with LtR, instead of using
> the
> model/re-rankers explain method, it uses the default query explain (tf-idf
> explanation). This happens because the BasicResultContext doesn't wrap the
>
> query([https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302]
>
> 214165a4d/solr/core/src/java/org/apache/solr/response/BasicResultContext.java#L6
> 7) with the RankQuery when its set to context's query, which is then used by
> the ExplainAugmenterFactory.
>
> ([https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c45230221416]
>
> 5a4d/solr/core/src/java/org/apache/solr/response/transform/ExplainAugmenterFacto
> ry.java#L111).
> As a result there are discrepancies between queries like -
> [http://localhost:8983/solr/collection1/select?q=*:*&collection=collectionName&wt]
> =json&fl=[explain style=nl],score&rq=\{!ltr model=linear-model}
> [http://localhost:8983/solr/collection1/select?q=*:*&collection=collectionName&wt]
> =json&fl=score&rq=\{!ltr model=linear-model}&debugQuery=true
> the former outputs the explain from the SimilarityScorer's explain while the
> latter uses the correct LtR ModelScorer's explain.
> There are a few other problems with the explain augmenter - for eg. it
> doesn't
> work with grouping (although the other doc transformers like LtR's
> LTRFeatureLoggerTransformerFactory work with grouping)
> Mailing List -
>
> http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201901.mbox/%3C5C3879D702C30144003902B0_0_18802%40msllnjpmsgsv06%3E
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[JENKINS] Lucene-Solr-7.x-Linux (64bit/jdk-10.0.1) - Build # 3415 - Unstable!
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/3415/ Java: 64bit/jdk-10.0.1 -XX:+UseCompressedOops -XX:+UseG1GC 1 tests failed. FAILED: junit.framework.TestSuite.org.apache.solr.handler.TestReplicationHandler Error Message: ObjectTracker found 4 object(s) that were not released!!! [InternalHttpClient, MockDirectoryWrapper, SolrCore, MockDirectoryWrapper] org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.http.impl.client.InternalHttpClient at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42) at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:321) at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:330) at org.apache.solr.handler.IndexFetcher.createHttpClient(IndexFetcher.java:225) at org.apache.solr.handler.IndexFetcher.(IndexFetcher.java:267) at org.apache.solr.handler.ReplicationHandler.inform(ReplicationHandler.java:1202) at org.apache.solr.core.SolrResourceLoader.inform(SolrResourceLoader.java:696) at org.apache.solr.core.SolrCore.(SolrCore.java:1000) at org.apache.solr.core.SolrCore.(SolrCore.java:874) at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1178) at org.apache.solr.core.CoreContainer.lambda$load$13(CoreContainer.java:690) at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:844) org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MockDirectoryWrapper at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42) at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348) at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:95) at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:770) at org.apache.solr.core.SolrCore.(SolrCore.java:967) at org.apache.solr.core.SolrCore.(SolrCore.java:874) at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1178) at org.apache.solr.core.CoreContainer.lambda$load$13(CoreContainer.java:690) at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:844) org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.core.SolrCore at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42) at org.apache.solr.core.SolrCore.(SolrCore.java:1054) at org.apache.solr.core.SolrCore.(SolrCore.java:874) at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1178) at org.apache.solr.core.CoreContainer.lambda$load$13(CoreContainer.java:690) at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:844) org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.lucene.store.MockDirectoryWrapper at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42) at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348) at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:359) at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:738) at org.apache.solr.core.SolrCore.(SolrCore.java:967) at org.apache.solr.core.SolrCore.(SolrCore.java:874) at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1178) at org.apache.solr.core.CoreContainer.lambda$load$13(CoreContainer.java:690) at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(Instrumente
[jira] [Commented] (SOLR-13144) Reference guide section on IndexUpgraderTool needs to be updated
[ https://issues.apache.org/jira/browse/SOLR-13144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745359#comment-16745359 ] Cassandra Targett commented on SOLR-13144: -- Please also see SOLR-12281. Updating a single page is only be a part of the overall story that needs to be described in detail in at least a couple places of the Guide (page for upgrading to 8.0, etc.) > Reference guide section on IndexUpgraderTool needs to be updated > > > Key: SOLR-13144 > URL: https://issues.apache.org/jira/browse/SOLR-13144 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: documentation >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Minor > Attachments: SOLR-13144.patch > > > The section on IndexUpgrader Tool (and I'll remove the space too) leads to > the conclusion that you can successfully upgrade an index X->X+1->X+2 which > isn't true. I'll attach a proposed doc change in a minute. > I'm thinking of only putting this in master and 8x on the theory that AFAIK, > the enforcement of this restriction is only 8x and later. > This is doc-only, if no objections I'll commit this weekend at the latest. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8585) Create jump-tables for DocValues at index-time
[
https://issues.apache.org/jira/browse/LUCENE-8585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745344#comment-16745344
]
Adrien Grand commented on LUCENE-8585:
--
bq. BaseDocValuesFormatTestCase focuses on Variable Bits Per Value block jumps
Did you mean TestLucene80DocValuesFormat?
+1 to merge this PR
bq. I need a bit of guidance on how to fit this into the 8.0-release.
I haven't seen a call to cut a RC yet so I assume it's ok to get this change in
[~romseygeek]?
> Create jump-tables for DocValues at index-time
> --
>
> Key: LUCENE-8585
> URL: https://issues.apache.org/jira/browse/LUCENE-8585
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: 8.0
>Reporter: Toke Eskildsen
>Priority: Minor
> Labels: performance
> Attachments: LUCENE-8585.patch, LUCENE-8585.patch,
> make_patch_lucene8585.sh
>
> Time Spent: 10h
> Remaining Estimate: 0h
>
> As noted in LUCENE-7589, lookup of DocValues should use jump-tables to avoid
> long iterative walks. This is implemented in LUCENE-8374 at search-time
> (first request for DocValues from a field in a segment), with the benefit of
> working without changes to existing Lucene 7 indexes and the downside of
> introducing a startup time penalty and a memory overhead.
> As discussed in LUCENE-8374, the codec should be updated to create these
> jump-tables at index time. This eliminates the segment-open time & memory
> penalties, with the potential downside of increasing index-time for DocValues.
> The three elements of LUCENE-8374 should be transferable to index-time
> without much alteration of the core structures:
> * {{IndexedDISI}} block offset and index skips: A {{long}} (64 bits) for
> every 65536 documents, containing the offset of the block in 33 bits and the
> index (number of set bits) up to the block in 31 bits.
> It can be build sequentially and should be stored as a simple sequence of
> consecutive longs for caching of lookups.
> As it is fairly small, relative to document count, it might be better to
> simply memory cache it?
> * {{IndexedDISI}} DENSE (> 4095, < 65536 set bits) blocks: A {{short}} (16
> bits) for every 8 {{longs}} (512 bits) for a total of 256 bytes/DENSE_block.
> Each {{short}} represents the number of set bits up to right before the
> corresponding sub-block of 512 docIDs.
> The \{{shorts}} can be computed sequentially or when the DENSE block is
> flushed (probably the easiest). They should be stored as a simple sequence of
> consecutive shorts for caching of lookups, one logically independent sequence
> for each DENSE block. The logical position would be one sequence at the start
> of every DENSE block.
> Whether it is best to read all the 16 {{shorts}} up front when a DENSE block
> is accessed or whether it is best to only read any individual {{short}} when
> needed is not clear at this point.
> * Variable Bits Per Value: A {{long}} (64 bits) for every 16384 numeric
> values. Each {{long}} holds the offset to the corresponding block of values.
> The offsets can be computed sequentially and should be stored as a simple
> sequence of consecutive {{longs}} for caching of lookups.
> The vBPV-offsets has the largest space overhead og the 3 jump-tables and a
> lot of the 64 bits in each long are not used for most indexes. They could be
> represented as a simple {{PackedInts}} sequence or {{MonotonicLongValues}},
> with the downsides of a potential lookup-time overhead and the need for doing
> the compression after all offsets has been determined.
> I have no experience with the codec-parts responsible for creating
> index-structures. I'm quite willing to take a stab at this, although I
> probably won't do much about it before January 2019. Should anyone else wish
> to adopt this JIRA-issue or co-work on it, I'll be happy to share.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] jpountz commented on a change in pull request #525: LUCENE-8585: Index-time jump-tables for DocValues
jpountz commented on a change in pull request #525: LUCENE-8585: Index-time
jump-tables for DocValues
URL: https://github.com/apache/lucene-solr/pull/525#discussion_r248773989
##
File path:
lucene/core/src/java/org/apache/lucene/codecs/lucene80/Lucene80DocValuesConsumer.java
##
@@ -0,0 +1,663 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.codecs.lucene80;
+
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+import org.apache.lucene.codecs.CodecUtil;
+import org.apache.lucene.codecs.DocValuesConsumer;
+import org.apache.lucene.codecs.DocValuesProducer;
+import org.apache.lucene.index.BinaryDocValues;
+import org.apache.lucene.index.DocValues;
+import org.apache.lucene.index.EmptyDocValuesProducer;
+import org.apache.lucene.index.FieldInfo;
+import org.apache.lucene.index.IndexFileNames;
+import org.apache.lucene.index.SegmentWriteState;
+import org.apache.lucene.index.SortedDocValues;
+import org.apache.lucene.index.SortedNumericDocValues;
+import org.apache.lucene.index.SortedSetDocValues;
+import org.apache.lucene.index.TermsEnum;
+import org.apache.lucene.search.DocIdSetIterator;
+import org.apache.lucene.search.SortedSetSelector;
+import org.apache.lucene.store.GrowableByteArrayDataOutput;
+import org.apache.lucene.store.IndexOutput;
+import org.apache.lucene.store.RAMOutputStream;
+import org.apache.lucene.util.ArrayUtil;
+import org.apache.lucene.util.BytesRef;
+import org.apache.lucene.util.BytesRefBuilder;
+import org.apache.lucene.util.IOUtils;
+import org.apache.lucene.util.MathUtil;
+import org.apache.lucene.util.StringHelper;
+import org.apache.lucene.util.packed.DirectMonotonicWriter;
+import org.apache.lucene.util.packed.DirectWriter;
+
+import static
org.apache.lucene.codecs.lucene80.Lucene80DocValuesFormat.DIRECT_MONOTONIC_BLOCK_SHIFT;
+import static
org.apache.lucene.codecs.lucene80.Lucene80DocValuesFormat.NUMERIC_BLOCK_SHIFT;
+import static
org.apache.lucene.codecs.lucene80.Lucene80DocValuesFormat.NUMERIC_BLOCK_SIZE;
+
+/** writer for {@link Lucene80DocValuesFormat} */
+final class Lucene80DocValuesConsumer extends DocValuesConsumer implements
Closeable {
+
+ IndexOutput data, meta;
+ final int maxDoc;
+
+ /** expert: Creates a new writer */
+ public Lucene80DocValuesConsumer(SegmentWriteState state, String dataCodec,
String dataExtension, String metaCodec, String metaExtension) throws
IOException {
+boolean success = false;
+try {
+ String dataName = IndexFileNames.segmentFileName(state.segmentInfo.name,
state.segmentSuffix, dataExtension);
+ data = state.directory.createOutput(dataName, state.context);
+ CodecUtil.writeIndexHeader(data, dataCodec,
Lucene80DocValuesFormat.VERSION_CURRENT, state.segmentInfo.getId(),
state.segmentSuffix);
+ String metaName = IndexFileNames.segmentFileName(state.segmentInfo.name,
state.segmentSuffix, metaExtension);
+ meta = state.directory.createOutput(metaName, state.context);
+ CodecUtil.writeIndexHeader(meta, metaCodec,
Lucene80DocValuesFormat.VERSION_CURRENT, state.segmentInfo.getId(),
state.segmentSuffix);
+ maxDoc = state.segmentInfo.maxDoc();
+ success = true;
+} finally {
+ if (!success) {
+IOUtils.closeWhileHandlingException(this);
+ }
+}
+ }
+
+ @Override
+ public void close() throws IOException {
+boolean success = false;
+try {
+ if (meta != null) {
+meta.writeInt(-1); // write EOF marker
+CodecUtil.writeFooter(meta); // write checksum
+ }
+ if (data != null) {
+CodecUtil.writeFooter(data); // write checksum
+ }
+ success = true;
+} finally {
+ if (success) {
+IOUtils.close(data, meta);
+ } else {
+IOUtils.closeWhileHandlingException(data, meta);
+ }
+ meta = data = null;
+}
+ }
+
+ @Override
+ public void addNumericField(FieldInfo field, DocValuesProducer
valuesProducer) throws IOException {
+meta.writeInt(field.number);
+meta.writeByte(Lucene80DocValuesFormat.NUMERIC);
+
+writeValues(field, n
[GitHub] jpountz commented on a change in pull request #525: LUCENE-8585: Index-time jump-tables for DocValues
jpountz commented on a change in pull request #525: LUCENE-8585: Index-time jump-tables for DocValues URL: https://github.com/apache/lucene-solr/pull/525#discussion_r248777308 ## File path: lucene/test-framework/src/java/org/apache/lucene/index/BaseDocValuesFormatTestCase.java ## @@ -1216,7 +1219,7 @@ private void doTestNumericsVsStoredFields(double density, LongSupplier longs) th doc.add(dvField); // index some docs -int numDocs = atLeast(300); +int numDocs = atLeast((int) (minDocs*1.172)); Review comment: why this 1.172 factor? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[JENKINS] Lucene-Solr-NightlyTests-7.x - Build # 434 - Still Unstable
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.x/434/ 1 tests failed. FAILED: org.apache.solr.cloud.ForceLeaderTest.testReplicasInLIRNoLeader Error Message: Timeout occured while waiting response from server at: http://127.0.0.1:34433/lo_/xt/forceleader_test_collection Stack Trace: org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:34433/lo_/xt/forceleader_test_collection at __randomizedtesting.SeedInfo.seed([1C56C4CDC8AEA95D:FAC1F00DF12C503C]:0) at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654) at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255) at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244) at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:484) at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:414) at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1110) at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884) at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817) at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194) at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:504) at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:479) at org.apache.solr.cloud.ForceLeaderTest.testReplicasInLIRNoLeader(ForceLeaderTest.java:294) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938) at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974) at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988) at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1075) at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1047) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRul
[jira] [Updated] (SOLR-13144) Reference guide section on IndexUpgraderTool needs to be updated
[ https://issues.apache.org/jira/browse/SOLR-13144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erick Erickson updated SOLR-13144: -- Attachment: SOLR-13144.patch > Reference guide section on IndexUpgraderTool needs to be updated > > > Key: SOLR-13144 > URL: https://issues.apache.org/jira/browse/SOLR-13144 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: documentation >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Minor > Attachments: SOLR-13144.patch > > > The section on IndexUpgrader Tool (and I'll remove the space too) leads to > the conclusion that you can successfully upgrade an index X->X+1->X+2 which > isn't true. I'll attach a proposed doc change in a minute. > I'm thinking of only putting this in master and 8x on the theory that AFAIK, > the enforcement of this restriction is only 8x and later. > This is doc-only, if no objections I'll commit this weekend at the latest. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-13144) Reference guide section on IndexUpgraderTool needs to be updated
Erick Erickson created SOLR-13144: - Summary: Reference guide section on IndexUpgraderTool needs to be updated Key: SOLR-13144 URL: https://issues.apache.org/jira/browse/SOLR-13144 Project: Solr Issue Type: Improvement Security Level: Public (Default Security Level. Issues are Public) Components: documentation Reporter: Erick Erickson Assignee: Erick Erickson The section on IndexUpgrader Tool (and I'll remove the space too) leads to the conclusion that you can successfully upgrade an index X->X+1->X+2 which isn't true. I'll attach a proposed doc change in a minute. I'm thinking of only putting this in master and 8x on the theory that AFAIK, the enforcement of this restriction is only 8x and later. This is doc-only, if no objections I'll commit this weekend at the latest. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: HTTP/2 support completely missing in CHANGES.txt
Hi Uwe, All announement in solr/CHANGES.txt were removed in a merge, I added them back. Please kindly review them. Thanks! On Thu, Jan 17, 2019 at 1:51 PM Uwe Schindler wrote: > Hi, > > I just reviewed the Lucene/Solr 8 changes text (in preparation to my > FOSDEM talk) and figured out that there is no announcement about the > recently added support for HTTP/2 in the Solr inter-node communication and > inside the SolrJ client. We should really add some announement in the "new > features". We should also give a statement that HTTPS support in > combination with HTTP/2 only works with Java 9 or later. > > Who wants to add this? > > Uwe > > - > Uwe Schindler > Achterdiek 19, D-28357 Bremen > http://www.thetaphi.de > eMail: u...@thetaphi.de > > > > - > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > > -- *Best regards,* *Cao Mạnh Đạt* *D.O.B : 31-07-1991Cell: (+84) 946.328.329E-mail: caomanhdat...@gmail.com *
[jira] [Commented] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap
[
https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745253#comment-16745253
]
Michael McCandless commented on LUCENE-8635:
OK thanks [~sokolov]. I'll try to also run bench on wikibig and report back.
I think doing a single method call instead of the two (seek + read) via
{{RandomAccessInput}} must be helping.
{quote}The thing that makes me want to be careful here is that access to the
terms index is very random, so things might degrade badly if the OS cache
doesn't hold the whole terms index in memory.
{quote}
I think net/net we are already relying on OS to do the right thing here. As
things stand today, the OS could also swap out the heap pages that hold the
FST's {{byte[]}} depending on its swappiness (on Linux).
{quote}I'm not super familiar with the FST internals, I wonder whether there
are changes that we could make to it so that it would be more disk-friendly,
eg. by seeking backward as little as possible when looking up a key?
{quote}
{{We used to have a }}{{pack}} method in FST that would 1) try to further
compress the {{byte[]}} size by moving nodes "closer" to the nodes that
transitioned to them, and 2) reversing the bytes. But we removed that method
because it added complexity and nobody was really using it and sometimes it
even made the FST bigger!
Maybe, we could bring the method back, but only part 2) of it, and always call
it at the end of building an FST? That should be simpler code (without part
1), and should achieve sequential reads of at least the bytes to decode a
single transition; maybe it gives a performance jump independent of this
change? But I think we really should explore that independently of this issue
... I think as long as additional performance tests show only these smallish
impacts to real queries we should just make the change across the board for
terms dictionary index?
> Lazy loading Lucene FST offheap using mmap
> --
>
> Key: LUCENE-8635
> URL: https://issues.apache.org/jira/browse/LUCENE-8635
> Project: Lucene - Core
> Issue Type: New Feature
> Components: core/FSTs
> Environment: I used below setup for es_rally tests:
> single node i3.xlarge running ES 6.5
> es_rally was running on another i3.xlarge instance
>Reporter: Ankit Jain
>Priority: Major
> Attachments: offheap.patch, ra.patch, rally_benchmark.xlsx
>
>
> Currently, FST loads all the terms into heap memory during index open. This
> causes frequent JVM OOM issues if the term size gets big. A better way of
> doing this will be to lazily load FST using mmap. That ensures only the
> required terms get loaded into memory.
>
> Lucene can expose API for providing list of fields to load terms offheap. I'm
> planning to take following approach for this:
> # Add a boolean property fstOffHeap in FieldInfo
> # Pass list of offheap fields to lucene during index open (ALL can be
> special keyword for loading ALL fields offheap)
> # Initialize the fstOffHeap property during lucene index open
> # FieldReader invokes default FST constructor or OffHeap constructor based
> on fstOffHeap field
>
> I created a patch (that loads all fields offheap), did some benchmarks using
> es_rally and results look good.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] janhoy opened a new pull request #341: SOLR-12131: ExternalRoleRuleBasedAuthorizationPlugin
janhoy opened a new pull request #341: SOLR-12131: ExternalRoleRuleBasedAuthorizationPlugin URL: https://github.com/apache/lucene-solr/pull/341 ...which gets user's roles from request This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] janhoy commented on issue #341: SOLR-12131: ExternalRoleRuleBasedAuthorizationPlugin
janhoy commented on issue #341: SOLR-12131: ExternalRoleRuleBasedAuthorizationPlugin URL: https://github.com/apache/lucene-solr/pull/341#issuecomment-455227661 Sorry, reopening as only part of this will be part of SOLR-12121, the new Authz plugin will still be committed in this issue This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Closed] (SOLR-12131) Authorization plugin support for getting user's roles from the outside
[
https://issues.apache.org/jira/browse/SOLR-12131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jan Høydahl closed SOLR-12131.
--
> Authorization plugin support for getting user's roles from the outside
> --
>
> Key: SOLR-12131
> URL: https://issues.apache.org/jira/browse/SOLR-12131
> Project: Solr
> Issue Type: New Feature
> Security Level: Public(Default Security Level. Issues are Public)
> Components: security
>Reporter: Jan Høydahl
>Assignee: Jan Høydahl
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Currently the {{RuleBasedAuthorizationPlugin}} relies on explicitly mapping
> users to roles. However, when users are authenticated by an external Identity
> service (e.g. JWT as implemented in SOLR-12121), that external service keeps
> track of the user's roles, and will pass that as a "claim" in the token (JWT).
> In order for Solr to be able to Authorise requests based on those roles, the
> Authorization plugin should be able to accept (verified) roles from the
> request instead of explicit mapping.
> Suggested approach is to create a new interface {{VerifiedUserRoles}} and a
> {{PrincipalWithUserRoles}} which implements the interface. The Authorization
> plugin can then pull the roles from request. By piggy-backing on the
> Principal, we have a seamless way to transfer extra external information, and
> there is also a natural relationship:
> {code:java}
> User Authentication -> Role validation -> Creating a Principal{code}
> I plan to add the interface, the custom Principal class and restructure
> {{RuleBasedAuthorizationPlugin}} in an abstract base class and two
> implementations: {{RuleBasedAuthorizationPlugin}} (as today) and a new
> {{ExternalRoleRuleBasedAuthorizationPlugin.}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] janhoy closed pull request #341: SOLR-12131: ExternalRoleRuleBasedAuthorizationPlugin
janhoy closed pull request #341: SOLR-12131: ExternalRoleRuleBasedAuthorizationPlugin URL: https://github.com/apache/lucene-solr/pull/341 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] janhoy commented on issue #341: SOLR-12131: ExternalRoleRuleBasedAuthorizationPlugin
janhoy commented on issue #341: SOLR-12131: ExternalRoleRuleBasedAuthorizationPlugin URL: https://github.com/apache/lucene-solr/pull/341#issuecomment-455226804 Closing, will be implemented in SOLR-12121 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-12131) Authorization plugin support for getting user's roles from the outside
[
https://issues.apache.org/jira/browse/SOLR-12131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jan Høydahl resolved SOLR-12131.
Resolution: Later
Fix Version/s: (was: 8.0)
This feature will be committed as part of SOLR-12121. Resolving this
> Authorization plugin support for getting user's roles from the outside
> --
>
> Key: SOLR-12131
> URL: https://issues.apache.org/jira/browse/SOLR-12131
> Project: Solr
> Issue Type: New Feature
> Security Level: Public(Default Security Level. Issues are Public)
> Components: security
>Reporter: Jan Høydahl
>Assignee: Jan Høydahl
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently the {{RuleBasedAuthorizationPlugin}} relies on explicitly mapping
> users to roles. However, when users are authenticated by an external Identity
> service (e.g. JWT as implemented in SOLR-12121), that external service keeps
> track of the user's roles, and will pass that as a "claim" in the token (JWT).
> In order for Solr to be able to Authorise requests based on those roles, the
> Authorization plugin should be able to accept (verified) roles from the
> request instead of explicit mapping.
> Suggested approach is to create a new interface {{VerifiedUserRoles}} and a
> {{PrincipalWithUserRoles}} which implements the interface. The Authorization
> plugin can then pull the roles from request. By piggy-backing on the
> Principal, we have a seamless way to transfer extra external information, and
> there is also a natural relationship:
> {code:java}
> User Authentication -> Role validation -> Creating a Principal{code}
> I plan to add the interface, the custom Principal class and restructure
> {{RuleBasedAuthorizationPlugin}} in an abstract base class and two
> implementations: {{RuleBasedAuthorizationPlugin}} (as today) and a new
> {{ExternalRoleRuleBasedAuthorizationPlugin.}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12121) JWT Authentication plugin
[
https://issues.apache.org/jira/browse/SOLR-12121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745226#comment-16745226
]
Jan Høydahl commented on SOLR-12121:
Up to date with master, fixed some documentation, going to commit to master
soon.
> JWT Authentication plugin
> -
>
> Key: SOLR-12121
> URL: https://issues.apache.org/jira/browse/SOLR-12121
> Project: Solr
> Issue Type: New Feature
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Authentication
>Reporter: Jan Høydahl
>Assignee: Jan Høydahl
>Priority: Major
> Fix For: master (9.0)
>
> Attachments: image-2018-08-27-13-04-04-183.png
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> A new Authentication plugin that will accept a [Json Web
> Token|https://en.wikipedia.org/wiki/JSON_Web_Token] (JWT) in the
> Authorization header and validate it by checking the cryptographic signature.
> The plugin will not perform the authentication itself but assert that the
> user was authenticated by the service that issued the JWT token.
> JWT defined a number of standard claims, and user principal can be fetched
> from the {{sub}} (subject) claim and passed on to Solr. The plugin will
> always check the {{exp}} (expiry) claim and optionally enforce checks on the
> {{iss}} (issuer) and {{aud}} (audience) claims.
> The first version of the plugin will only support RSA signing keys and will
> support fetching the public key of the issuer through a [Json Web
> Key|https://tools.ietf.org/html/rfc7517] (JWK) file, either from a https URL
> or from local file.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-12121) JWT Authentication plugin
[
https://issues.apache.org/jira/browse/SOLR-12121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jan Høydahl updated SOLR-12121:
---
Fix Version/s: (was: 8.0)
> JWT Authentication plugin
> -
>
> Key: SOLR-12121
> URL: https://issues.apache.org/jira/browse/SOLR-12121
> Project: Solr
> Issue Type: New Feature
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Authentication
>Reporter: Jan Høydahl
>Assignee: Jan Høydahl
>Priority: Major
> Fix For: master (9.0)
>
> Attachments: image-2018-08-27-13-04-04-183.png
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> A new Authentication plugin that will accept a [Json Web
> Token|https://en.wikipedia.org/wiki/JSON_Web_Token] (JWT) in the
> Authorization header and validate it by checking the cryptographic signature.
> The plugin will not perform the authentication itself but assert that the
> user was authenticated by the service that issued the JWT token.
> JWT defined a number of standard claims, and user principal can be fetched
> from the {{sub}} (subject) claim and passed on to Solr. The plugin will
> always check the {{exp}} (expiry) claim and optionally enforce checks on the
> {{iss}} (issuer) and {{aud}} (audience) claims.
> The first version of the plugin will only support RSA signing keys and will
> support fetching the public key of the issuer through a [Json Web
> Key|https://tools.ietf.org/html/rfc7517] (JWK) file, either from a https URL
> or from local file.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8642) RamUsageTester.sizeOf ignores arrays and collections if --illegal-access=deny
[ https://issues.apache.org/jira/browse/LUCENE-8642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745190#comment-16745190 ] Dawid Weiss commented on LUCENE-8642: - Yes, I'll work on it. Sorry for the noise. > RamUsageTester.sizeOf ignores arrays and collections if --illegal-access=deny > - > > Key: LUCENE-8642 > URL: https://issues.apache.org/jira/browse/LUCENE-8642 > Project: Lucene - Core > Issue Type: Bug >Reporter: Dawid Weiss >Assignee: Dawid Weiss >Priority: Minor > Fix For: master (9.0) > > > We have a silent fall-through on catch clause there if a field is > inaccessible. This causes all collections to not be recursively descended > into. > We could hack it so that collection objects are counted "manually". Or we > could propagate illegal access exception from ram tester, detect which tests/ > objects do try to measure collection objects and either remove them (if it's > not possible to measure accurately) or change them (if it is possible to > estimate the size in a different way). > Looking for feedback on this one. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13143) Output from ExplainAugmenterFactory and DebugQuery for rerank queries is not same
[
https://issues.apache.org/jira/browse/SOLR-13143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745178#comment-16745178
]
Sambhav Kothari commented on SOLR-13143:
https://github.com/apache/lucene-solr/pull/539
> Output from ExplainAugmenterFactory and DebugQuery for rerank queries is not
> same
> -
>
> Key: SOLR-13143
> URL: https://issues.apache.org/jira/browse/SOLR-13143
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: search
>Affects Versions: 7.5
>Reporter: Sambhav Kothari
>Priority: Minor
> Attachments: bug.patch
>
>
> Currently, if we use the ExplainAugmenterFactory with LtR, instead of using
> the
> model/re-rankers explain method, it uses the default query explain (tf-idf
> explanation). This happens because the BasicResultContext doesn't wrap the
>
> query([https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302]
>
> 214165a4d/solr/core/src/java/org/apache/solr/response/BasicResultContext.java#L6
> 7) with the RankQuery when its set to context's query, which is then used by
> the ExplainAugmenterFactory.
>
> ([https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c45230221416]
>
> 5a4d/solr/core/src/java/org/apache/solr/response/transform/ExplainAugmenterFacto
> ry.java#L111).
> As a result there are discrepancies between queries like -
> [http://localhost:8983/solr/collection1/select?q=*:*&collection=collectionName&wt]
> =json&fl=[explain style=nl],score&rq=\{!ltr model=linear-model}
> [http://localhost:8983/solr/collection1/select?q=*:*&collection=collectionName&wt]
> =json&fl=score&rq=\{!ltr model=linear-model}&debugQuery=true
> the former outputs the explain from the SimilarityScorer's explain while the
> latter uses the correct LtR ModelScorer's explain.
> There are a few other problems with the explain augmenter - for eg. it
> doesn't
> work with grouping (although the other doc transformers like LtR's
> LTRFeatureLoggerTransformerFactory work with grouping)
> Mailing List -
>
> http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201901.mbox/%3C5C3879D702C30144003902B0_0_18802%40msllnjpmsgsv06%3E
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12638) Support atomic updates of nested/child documents for nested-enabled schema
[ https://issues.apache.org/jira/browse/SOLR-12638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745179#comment-16745179 ] Yosef Brown commented on SOLR-12638: If we require the child document's id to begin with the parent's id, and the parent document id itself is a composite (https://lucene.apache.org/solr/guide/7_5/shards-and-indexing-data-in-solrcloud.html#document-routing advocates using id prefixes for shard routing), the child document will end up with 2 exclamation separators (and schemas with more levels of nesting will have even more exclamations). What is the impact of having multiple exclamations in a (child) document id? > Support atomic updates of nested/child documents for nested-enabled schema > -- > > Key: SOLR-12638 > URL: https://issues.apache.org/jira/browse/SOLR-12638 > Project: Solr > Issue Type: Sub-task > Security Level: Public(Default Security Level. Issues are Public) >Reporter: mosh >Priority: Major > Attachments: SOLR-12638-delete-old-block-no-commit.patch, > SOLR-12638-nocommit.patch > > Time Spent: 6h 40m > Remaining Estimate: 0h > > I have been toying with the thought of using this transformer in conjunction > with NestedUpdateProcessor and AtomicUpdate to allow SOLR to completely > re-index the entire nested structure. This is just a thought, I am still > thinking about implementation details. Hopefully I will be able to post a > more concrete proposal soon. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-13143) Output from ExplainAugmenterFactory and DebugQuery for rerank queries is not same
[
https://issues.apache.org/jira/browse/SOLR-13143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745004#comment-16745004
]
Sambhav Kothari edited comment on SOLR-13143 at 1/17/19 2:56 PM:
-
Attaching the failing test as a patch. You can run it via
{code:java}
cd solr/core
ant test -Dtestcase=TestReRankQParserPlugin{code}
was (Author: samj1912):
Attaching the failing test as a patch. You can run it vie
{code:java}
ant test -Dtestcase=TestReRankQParserPlugin{code}
> Output from ExplainAugmenterFactory and DebugQuery for rerank queries is not
> same
> -
>
> Key: SOLR-13143
> URL: https://issues.apache.org/jira/browse/SOLR-13143
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: search
>Affects Versions: 7.5
>Reporter: Sambhav Kothari
>Priority: Minor
> Attachments: bug.patch
>
>
> Currently, if we use the ExplainAugmenterFactory with LtR, instead of using
> the
> model/re-rankers explain method, it uses the default query explain (tf-idf
> explanation). This happens because the BasicResultContext doesn't wrap the
>
> query([https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302]
>
> 214165a4d/solr/core/src/java/org/apache/solr/response/BasicResultContext.java#L6
> 7) with the RankQuery when its set to context's query, which is then used by
> the ExplainAugmenterFactory.
>
> ([https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c45230221416]
>
> 5a4d/solr/core/src/java/org/apache/solr/response/transform/ExplainAugmenterFacto
> ry.java#L111).
> As a result there are discrepancies between queries like -
> [http://localhost:8983/solr/collection1/select?q=*:*&collection=collectionName&wt]
> =json&fl=[explain style=nl],score&rq=\{!ltr model=linear-model}
> [http://localhost:8983/solr/collection1/select?q=*:*&collection=collectionName&wt]
> =json&fl=score&rq=\{!ltr model=linear-model}&debugQuery=true
> the former outputs the explain from the SimilarityScorer's explain while the
> latter uses the correct LtR ModelScorer's explain.
> There are a few other problems with the explain augmenter - for eg. it
> doesn't
> work with grouping (although the other doc transformers like LtR's
> LTRFeatureLoggerTransformerFactory work with grouping)
> Mailing List -
>
> http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201901.mbox/%3C5C3879D702C30144003902B0_0_18802%40msllnjpmsgsv06%3E
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] samj1912 opened a new pull request #539: Add unit test for differences between ExplainAugmenterFactory and debug explanations
samj1912 opened a new pull request #539: Add unit test for differences between ExplainAugmenterFactory and debug explanations URL: https://github.com/apache/lucene-solr/pull/539 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13129) Document nested child docs in the ref guide
[ https://issues.apache.org/jira/browse/SOLR-13129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745090#comment-16745090 ] David Smiley commented on SOLR-13129: - Oh yes; I should have stated that more explicitly. > Document nested child docs in the ref guide > --- > > Key: SOLR-13129 > URL: https://issues.apache.org/jira/browse/SOLR-13129 > Project: Solr > Issue Type: Sub-task > Security Level: Public(Default Security Level. Issues are Public) > Components: documentation >Affects Versions: 8.0 >Reporter: David Smiley >Priority: Major > Fix For: 8.0 > > > Solr 8.0 will have nicer support for nested child documents than its > predecessors. This should be documented in one place in the ref guide (to > the extent that makes sense). Users need to the schema ramifications (incl. > special fields and that some aspects are optional and when), what a nested > document "looks like" (XML, JSON, SolrJ), how to use the child doc > transformer, how to use block join queries, and get some overview of how this > all works. Maybe mention some plausible future enhancements / direction this > is going in (e.g. path query language?). Some of this is already done but > it's in various places and could be moved. Unlike other features which > conveniently fit into one spot in the documentation (like a query parser), > this is a more complex issue that has multiple aspects – more > "cross-cutting", and so IMO doesn't belong in the current doc pigeon holes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8646) Add multi-term intervals
[ https://issues.apache.org/jira/browse/LUCENE-8646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745079#comment-16745079 ] Alan Woodward commented on LUCENE-8646: --- Here's a patch with one possible implementation. It adds support for prefix and wildcard expansions, and limits both to 128 terms. Expansion is done per-leaf reader (no rewriting) which I think is fine as we're not aggregating term stats or anything, so there's no need for a global view. I haven't added support for Regexp-type intervals as the API starts getting very clunky; we might want to revisit the Regexp object API itself, or maybe add the ability to build intervals directly from an Automaton. > Add multi-term intervals > > > Key: LUCENE-8646 > URL: https://issues.apache.org/jira/browse/LUCENE-8646 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Alan Woodward >Priority: Major > Attachments: LUCENE-8646.patch > > > We currently have no support for wildcard-type intervals. I'd like to > explore adding some very basic support for prefix and wildcard interval > sources, but we need to ensure that we don't end up with the same performance > issues that dog SpanMultiTermQueryWrapper. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8581) Change LatLonShape encoding to use 4 BYTES Per Dimension
[
https://issues.apache.org/jira/browse/LUCENE-8581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745085#comment-16745085
]
ASF subversion and git services commented on LUCENE-8581:
-
Commit 618812899cb1a6fc8ce7fa917ec4bc569686b0ff in lucene-solr's branch
refs/heads/branch_7x from David Wayne Smiley
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=6188128 ]
LUCENE-8581: Unreference LatLonPoint.BYTES from LatLonShape & Rectangle2D.
(cherry picked from commit 70dd3ee06a49261da1c9d2c220f59ac9e73fc4c8)
> Change LatLonShape encoding to use 4 BYTES Per Dimension
>
>
> Key: LUCENE-8581
> URL: https://issues.apache.org/jira/browse/LUCENE-8581
> Project: Lucene - Core
> Issue Type: New Feature
>Reporter: Nicholas Knize
>Assignee: Ignacio Vera
>Priority: Major
> Fix For: 8.0, 7.7
>
> Attachments: LUCENE-8581.patch, LUCENE-8581.patch, LUCENE-8581.patch,
> LUCENE-8581.patch, LUCENE-8581.patch, LUCENE-8581.patch, LUCENE-8581.patch,
> LUCENE-8581.patch, LUCENE-8581_bytes.patch
>
>
> {{LatLonShape}} tessellated triangles currently use a relatively naive
> encoding with the first four dimensions as the bounding box of the triangle
> and the last three dimensions as the vertices of the triangle. To encode the
> {{x,y}} vertices in the last three dimensions requires {{bytesPerDim}} to be
> set to 8, with 4 bytes for the x & y axis, respectively. We can reduce
> {{bytesPerDim}} to 4 by encoding the index(es) of the vertices shared by the
> bounding box along with the orientation of the triangle. This also opens the
> door for supporting {{CONTAINS}} queries.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-13143) Output from ExplainAugmenterFactory and DebugQuery for rerank queries is not same
[
https://issues.apache.org/jira/browse/SOLR-13143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sambhav Kothari updated SOLR-13143:
---
Attachment: (was: bug.patch)
> Output from ExplainAugmenterFactory and DebugQuery for rerank queries is not
> same
> -
>
> Key: SOLR-13143
> URL: https://issues.apache.org/jira/browse/SOLR-13143
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: search
>Affects Versions: 7.5
>Reporter: Sambhav Kothari
>Priority: Minor
> Attachments: bug.patch
>
>
> Currently, if we use the ExplainAugmenterFactory with LtR, instead of using
> the
> model/re-rankers explain method, it uses the default query explain (tf-idf
> explanation). This happens because the BasicResultContext doesn't wrap the
>
> query([https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302]
>
> 214165a4d/solr/core/src/java/org/apache/solr/response/BasicResultContext.java#L6
> 7) with the RankQuery when its set to context's query, which is then used by
> the ExplainAugmenterFactory.
>
> ([https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c45230221416]
>
> 5a4d/solr/core/src/java/org/apache/solr/response/transform/ExplainAugmenterFacto
> ry.java#L111).
> As a result there are discrepancies between queries like -
> [http://localhost:8983/solr/collection1/select?q=*:*&collection=collectionName&wt]
> =json&fl=[explain style=nl],score&rq=\{!ltr model=linear-model}
> [http://localhost:8983/solr/collection1/select?q=*:*&collection=collectionName&wt]
> =json&fl=score&rq=\{!ltr model=linear-model}&debugQuery=true
> the former outputs the explain from the SimilarityScorer's explain while the
> latter uses the correct LtR ModelScorer's explain.
> There are a few other problems with the explain augmenter - for eg. it
> doesn't
> work with grouping (although the other doc transformers like LtR's
> LTRFeatureLoggerTransformerFactory work with grouping)
> Mailing List -
>
> http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201901.mbox/%3C5C3879D702C30144003902B0_0_18802%40msllnjpmsgsv06%3E
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-13143) Output from ExplainAugmenterFactory and DebugQuery for rerank queries is not same
[
https://issues.apache.org/jira/browse/SOLR-13143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sambhav Kothari updated SOLR-13143:
---
Attachment: (was: bug.patch)
> Output from ExplainAugmenterFactory and DebugQuery for rerank queries is not
> same
> -
>
> Key: SOLR-13143
> URL: https://issues.apache.org/jira/browse/SOLR-13143
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: search
>Affects Versions: 7.5
>Reporter: Sambhav Kothari
>Priority: Minor
> Attachments: bug.patch
>
>
> Currently, if we use the ExplainAugmenterFactory with LtR, instead of using
> the
> model/re-rankers explain method, it uses the default query explain (tf-idf
> explanation). This happens because the BasicResultContext doesn't wrap the
>
> query([https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302]
>
> 214165a4d/solr/core/src/java/org/apache/solr/response/BasicResultContext.java#L6
> 7) with the RankQuery when its set to context's query, which is then used by
> the ExplainAugmenterFactory.
>
> ([https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c45230221416]
>
> 5a4d/solr/core/src/java/org/apache/solr/response/transform/ExplainAugmenterFacto
> ry.java#L111).
> As a result there are discrepancies between queries like -
> [http://localhost:8983/solr/collection1/select?q=*:*&collection=collectionName&wt]
> =json&fl=[explain style=nl],score&rq=\{!ltr model=linear-model}
> [http://localhost:8983/solr/collection1/select?q=*:*&collection=collectionName&wt]
> =json&fl=score&rq=\{!ltr model=linear-model}&debugQuery=true
> the former outputs the explain from the SimilarityScorer's explain while the
> latter uses the correct LtR ModelScorer's explain.
> There are a few other problems with the explain augmenter - for eg. it
> doesn't
> work with grouping (although the other doc transformers like LtR's
> LTRFeatureLoggerTransformerFactory work with grouping)
> Mailing List -
>
> http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201901.mbox/%3C5C3879D702C30144003902B0_0_18802%40msllnjpmsgsv06%3E
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-13143) Output from ExplainAugmenterFactory and DebugQuery for rerank queries is not same
[
https://issues.apache.org/jira/browse/SOLR-13143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sambhav Kothari updated SOLR-13143:
---
Attachment: bug.patch
> Output from ExplainAugmenterFactory and DebugQuery for rerank queries is not
> same
> -
>
> Key: SOLR-13143
> URL: https://issues.apache.org/jira/browse/SOLR-13143
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: search
>Affects Versions: 7.5
>Reporter: Sambhav Kothari
>Priority: Minor
> Attachments: bug.patch
>
>
> Currently, if we use the ExplainAugmenterFactory with LtR, instead of using
> the
> model/re-rankers explain method, it uses the default query explain (tf-idf
> explanation). This happens because the BasicResultContext doesn't wrap the
>
> query([https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302]
>
> 214165a4d/solr/core/src/java/org/apache/solr/response/BasicResultContext.java#L6
> 7) with the RankQuery when its set to context's query, which is then used by
> the ExplainAugmenterFactory.
>
> ([https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c45230221416]
>
> 5a4d/solr/core/src/java/org/apache/solr/response/transform/ExplainAugmenterFacto
> ry.java#L111).
> As a result there are discrepancies between queries like -
> [http://localhost:8983/solr/collection1/select?q=*:*&collection=collectionName&wt]
> =json&fl=[explain style=nl],score&rq=\{!ltr model=linear-model}
> [http://localhost:8983/solr/collection1/select?q=*:*&collection=collectionName&wt]
> =json&fl=score&rq=\{!ltr model=linear-model}&debugQuery=true
> the former outputs the explain from the SimilarityScorer's explain while the
> latter uses the correct LtR ModelScorer's explain.
> There are a few other problems with the explain augmenter - for eg. it
> doesn't
> work with grouping (although the other doc transformers like LtR's
> LTRFeatureLoggerTransformerFactory work with grouping)
> Mailing List -
>
> http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201901.mbox/%3C5C3879D702C30144003902B0_0_18802%40msllnjpmsgsv06%3E
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-13143) Output from ExplainAugmenterFactory and DebugQuery for rerank queries is not same
[
https://issues.apache.org/jira/browse/SOLR-13143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sambhav Kothari updated SOLR-13143:
---
Attachment: bug.patch
> Output from ExplainAugmenterFactory and DebugQuery for rerank queries is not
> same
> -
>
> Key: SOLR-13143
> URL: https://issues.apache.org/jira/browse/SOLR-13143
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: search
>Affects Versions: 7.5
>Reporter: Sambhav Kothari
>Priority: Minor
> Attachments: bug.patch, bug.patch
>
>
> Currently, if we use the ExplainAugmenterFactory with LtR, instead of using
> the
> model/re-rankers explain method, it uses the default query explain (tf-idf
> explanation). This happens because the BasicResultContext doesn't wrap the
>
> query([https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302]
>
> 214165a4d/solr/core/src/java/org/apache/solr/response/BasicResultContext.java#L6
> 7) with the RankQuery when its set to context's query, which is then used by
> the ExplainAugmenterFactory.
>
> ([https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c45230221416]
>
> 5a4d/solr/core/src/java/org/apache/solr/response/transform/ExplainAugmenterFacto
> ry.java#L111).
> As a result there are discrepancies between queries like -
> [http://localhost:8983/solr/collection1/select?q=*:*&collection=collectionName&wt]
> =json&fl=[explain style=nl],score&rq=\{!ltr model=linear-model}
> [http://localhost:8983/solr/collection1/select?q=*:*&collection=collectionName&wt]
> =json&fl=score&rq=\{!ltr model=linear-model}&debugQuery=true
> the former outputs the explain from the SimilarityScorer's explain while the
> latter uses the correct LtR ModelScorer's explain.
> There are a few other problems with the explain augmenter - for eg. it
> doesn't
> work with grouping (although the other doc transformers like LtR's
> LTRFeatureLoggerTransformerFactory work with grouping)
> Mailing List -
>
> http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201901.mbox/%3C5C3879D702C30144003902B0_0_18802%40msllnjpmsgsv06%3E
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8646) Add multi-term intervals
[ https://issues.apache.org/jira/browse/LUCENE-8646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745081#comment-16745081 ] Adrien Grand commented on LUCENE-8646: -- +1 Thanks for enforcing a maximum number of terms. > Add multi-term intervals > > > Key: LUCENE-8646 > URL: https://issues.apache.org/jira/browse/LUCENE-8646 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Alan Woodward >Priority: Major > Attachments: LUCENE-8646.patch > > > We currently have no support for wildcard-type intervals. I'd like to > explore adding some very basic support for prefix and wildcard interval > sources, but we need to ensure that we don't end up with the same performance > issues that dog SpanMultiTermQueryWrapper. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8581) Change LatLonShape encoding to use 4 BYTES Per Dimension
[
https://issues.apache.org/jira/browse/LUCENE-8581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745080#comment-16745080
]
ASF subversion and git services commented on LUCENE-8581:
-
Commit 28f2f2d5340db84da8ce4e265726685ca7740095 in lucene-solr's branch
refs/heads/branch_8x from David Wayne Smiley
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=28f2f2d ]
LUCENE-8581: Unreference LatLonPoint.BYTES from LatLonShape & Rectangle2D.
(cherry picked from commit 70dd3ee06a49261da1c9d2c220f59ac9e73fc4c8)
> Change LatLonShape encoding to use 4 BYTES Per Dimension
>
>
> Key: LUCENE-8581
> URL: https://issues.apache.org/jira/browse/LUCENE-8581
> Project: Lucene - Core
> Issue Type: New Feature
>Reporter: Nicholas Knize
>Assignee: Ignacio Vera
>Priority: Major
> Fix For: 8.0, 7.7
>
> Attachments: LUCENE-8581.patch, LUCENE-8581.patch, LUCENE-8581.patch,
> LUCENE-8581.patch, LUCENE-8581.patch, LUCENE-8581.patch, LUCENE-8581.patch,
> LUCENE-8581.patch, LUCENE-8581_bytes.patch
>
>
> {{LatLonShape}} tessellated triangles currently use a relatively naive
> encoding with the first four dimensions as the bounding box of the triangle
> and the last three dimensions as the vertices of the triangle. To encode the
> {{x,y}} vertices in the last three dimensions requires {{bytesPerDim}} to be
> set to 8, with 4 bytes for the x & y axis, respectively. We can reduce
> {{bytesPerDim}} to 4 by encoding the index(es) of the vertices shared by the
> bounding box along with the orientation of the triangle. This also opens the
> door for supporting {{CONTAINS}} queries.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-8646) Add multi-term intervals
[ https://issues.apache.org/jira/browse/LUCENE-8646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Woodward updated LUCENE-8646: -- Attachment: LUCENE-8646.patch > Add multi-term intervals > > > Key: LUCENE-8646 > URL: https://issues.apache.org/jira/browse/LUCENE-8646 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Alan Woodward >Priority: Major > Attachments: LUCENE-8646.patch > > > We currently have no support for wildcard-type intervals. I'd like to > explore adding some very basic support for prefix and wildcard interval > sources, but we need to ensure that we don't end up with the same performance > issues that dog SpanMultiTermQueryWrapper. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8581) Change LatLonShape encoding to use 4 BYTES Per Dimension
[
https://issues.apache.org/jira/browse/LUCENE-8581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745072#comment-16745072
]
ASF subversion and git services commented on LUCENE-8581:
-
Commit 70dd3ee06a49261da1c9d2c220f59ac9e73fc4c8 in lucene-solr's branch
refs/heads/master from David Wayne Smiley
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=70dd3ee ]
LUCENE-8581: Unreference LatLonPoint.BYTES from LatLonShape & Rectangle2D.
> Change LatLonShape encoding to use 4 BYTES Per Dimension
>
>
> Key: LUCENE-8581
> URL: https://issues.apache.org/jira/browse/LUCENE-8581
> Project: Lucene - Core
> Issue Type: New Feature
>Reporter: Nicholas Knize
>Assignee: Ignacio Vera
>Priority: Major
> Fix For: 8.0, 7.7
>
> Attachments: LUCENE-8581.patch, LUCENE-8581.patch, LUCENE-8581.patch,
> LUCENE-8581.patch, LUCENE-8581.patch, LUCENE-8581.patch, LUCENE-8581.patch,
> LUCENE-8581.patch, LUCENE-8581_bytes.patch
>
>
> {{LatLonShape}} tessellated triangles currently use a relatively naive
> encoding with the first four dimensions as the bounding box of the triangle
> and the last three dimensions as the vertices of the triangle. To encode the
> {{x,y}} vertices in the last three dimensions requires {{bytesPerDim}} to be
> set to 8, with 4 bytes for the x & y axis, respectively. We can reduce
> {{bytesPerDim}} to 4 by encoding the index(es) of the vertices shared by the
> bounding box along with the orientation of the triangle. This also opens the
> door for supporting {{CONTAINS}} queries.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8645) Add fixed field intervals
[ https://issues.apache.org/jira/browse/LUCENE-8645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745064#comment-16745064 ] Alan Woodward commented on LUCENE-8645: --- Here is a patch - this turns out be remarkably simple to implement. > Add fixed field intervals > - > > Key: LUCENE-8645 > URL: https://issues.apache.org/jira/browse/LUCENE-8645 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Alan Woodward >Priority: Major > Attachments: LUCENE-8645.patch > > > It can be useful to report intervals from one fields as if they came from > another. For example, fast prefix searches can be implemented by indexing > text into two fields, one with the full terms and one with edge-ngrams > enabled; to do proximity searches against terms and prefixes, you could wrap > a term query against the ngrammed field so that its intervals appear to come > from the normal field, and use it an an ordered or unordered interval. > This is analogous to the FieldMaskingSpanQuery, but has the advantage that we > don't use term statistics for scoring interval queries, so there is no issue > with mixing up field weights from different fields. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (LUCENE-8646) Add multi-term intervals
Alan Woodward created LUCENE-8646: - Summary: Add multi-term intervals Key: LUCENE-8646 URL: https://issues.apache.org/jira/browse/LUCENE-8646 Project: Lucene - Core Issue Type: Improvement Reporter: Alan Woodward Assignee: Alan Woodward We currently have no support for wildcard-type intervals. I'd like to explore adding some very basic support for prefix and wildcard interval sources, but we need to ensure that we don't end up with the same performance issues that dog SpanMultiTermQueryWrapper. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-13143) Output from ExplainAugmenterFactory and DebugQuery for rerank queries is not same
[
https://issues.apache.org/jira/browse/SOLR-13143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745004#comment-16745004
]
Sambhav Kothari edited comment on SOLR-13143 at 1/17/19 1:53 PM:
-
Attaching the failing test as a patch. You can run it vie
```
ant test -Dtestcase=TestReRankQParserPlugin
```
was (Author: samj1912):
Attaching the failing test as a patch.
> Output from ExplainAugmenterFactory and DebugQuery for rerank queries is not
> same
> -
>
> Key: SOLR-13143
> URL: https://issues.apache.org/jira/browse/SOLR-13143
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: search
>Affects Versions: 7.5
>Reporter: Sambhav Kothari
>Priority: Minor
> Attachments: bug.patch
>
>
> Currently, if we use the ExplainAugmenterFactory with LtR, instead of using
> the
> model/re-rankers explain method, it uses the default query explain (tf-idf
> explanation). This happens because the BasicResultContext doesn't wrap the
>
> query([https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302]
>
> 214165a4d/solr/core/src/java/org/apache/solr/response/BasicResultContext.java#L6
> 7) with the RankQuery when its set to context's query, which is then used by
> the ExplainAugmenterFactory.
>
> ([https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c45230221416]
>
> 5a4d/solr/core/src/java/org/apache/solr/response/transform/ExplainAugmenterFacto
> ry.java#L111).
> As a result there are discrepancies between queries like -
> [http://localhost:8983/solr/collection1/select?q=*:*&collection=collectionName&wt]
> =json&fl=[explain style=nl],score&rq=\{!ltr model=linear-model}
> [http://localhost:8983/solr/collection1/select?q=*:*&collection=collectionName&wt]
> =json&fl=score&rq=\{!ltr model=linear-model}&debugQuery=true
> the former outputs the explain from the SimilarityScorer's explain while the
> latter uses the correct LtR ModelScorer's explain.
> There are a few other problems with the explain augmenter - for eg. it
> doesn't
> work with grouping (although the other doc transformers like LtR's
> LTRFeatureLoggerTransformerFactory work with grouping)
> Mailing List -
>
> http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201901.mbox/%3C5C3879D702C30144003902B0_0_18802%40msllnjpmsgsv06%3E
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-13143) Output from ExplainAugmenterFactory and DebugQuery for rerank queries is not same
[
https://issues.apache.org/jira/browse/SOLR-13143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745004#comment-16745004
]
Sambhav Kothari edited comment on SOLR-13143 at 1/17/19 1:53 PM:
-
Attaching the failing test as a patch. You can run it vie
{code:java}
ant test -Dtestcase=TestReRankQParserPlugin{code}
was (Author: samj1912):
Attaching the failing test as a patch. You can run it vie
```
ant test -Dtestcase=TestReRankQParserPlugin
```
> Output from ExplainAugmenterFactory and DebugQuery for rerank queries is not
> same
> -
>
> Key: SOLR-13143
> URL: https://issues.apache.org/jira/browse/SOLR-13143
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: search
>Affects Versions: 7.5
>Reporter: Sambhav Kothari
>Priority: Minor
> Attachments: bug.patch
>
>
> Currently, if we use the ExplainAugmenterFactory with LtR, instead of using
> the
> model/re-rankers explain method, it uses the default query explain (tf-idf
> explanation). This happens because the BasicResultContext doesn't wrap the
>
> query([https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302]
>
> 214165a4d/solr/core/src/java/org/apache/solr/response/BasicResultContext.java#L6
> 7) with the RankQuery when its set to context's query, which is then used by
> the ExplainAugmenterFactory.
>
> ([https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c45230221416]
>
> 5a4d/solr/core/src/java/org/apache/solr/response/transform/ExplainAugmenterFacto
> ry.java#L111).
> As a result there are discrepancies between queries like -
> [http://localhost:8983/solr/collection1/select?q=*:*&collection=collectionName&wt]
> =json&fl=[explain style=nl],score&rq=\{!ltr model=linear-model}
> [http://localhost:8983/solr/collection1/select?q=*:*&collection=collectionName&wt]
> =json&fl=score&rq=\{!ltr model=linear-model}&debugQuery=true
> the former outputs the explain from the SimilarityScorer's explain while the
> latter uses the correct LtR ModelScorer's explain.
> There are a few other problems with the explain augmenter - for eg. it
> doesn't
> work with grouping (although the other doc transformers like LtR's
> LTRFeatureLoggerTransformerFactory work with grouping)
> Mailing List -
>
> http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201901.mbox/%3C5C3879D702C30144003902B0_0_18802%40msllnjpmsgsv06%3E
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
HTTP/2 support completely missing in CHANGES.txt
Hi, I just reviewed the Lucene/Solr 8 changes text (in preparation to my FOSDEM talk) and figured out that there is no announcement about the recently added support for HTTP/2 in the Solr inter-node communication and inside the SolrJ client. We should really add some announement in the "new features". We should also give a statement that HTTPS support in combination with HTTP/2 only works with Java 9 or later. Who wants to add this? Uwe - Uwe Schindler Achterdiek 19, D-28357 Bremen http://www.thetaphi.de eMail: u...@thetaphi.de - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-8645) Add fixed field intervals
[ https://issues.apache.org/jira/browse/LUCENE-8645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Woodward updated LUCENE-8645: -- Attachment: LUCENE-8645.patch > Add fixed field intervals > - > > Key: LUCENE-8645 > URL: https://issues.apache.org/jira/browse/LUCENE-8645 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Alan Woodward >Priority: Major > Attachments: LUCENE-8645.patch > > > It can be useful to report intervals from one fields as if they came from > another. For example, fast prefix searches can be implemented by indexing > text into two fields, one with the full terms and one with edge-ngrams > enabled; to do proximity searches against terms and prefixes, you could wrap > a term query against the ngrammed field so that its intervals appear to come > from the normal field, and use it an an ordered or unordered interval. > This is analogous to the FieldMaskingSpanQuery, but has the advantage that we > don't use term statistics for scoring interval queries, so there is no issue > with mixing up field weights from different fields. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (LUCENE-8645) Add fixed field intervals
Alan Woodward created LUCENE-8645: - Summary: Add fixed field intervals Key: LUCENE-8645 URL: https://issues.apache.org/jira/browse/LUCENE-8645 Project: Lucene - Core Issue Type: Improvement Reporter: Alan Woodward Assignee: Alan Woodward It can be useful to report intervals from one fields as if they came from another. For example, fast prefix searches can be implemented by indexing text into two fields, one with the full terms and one with edge-ngrams enabled; to do proximity searches against terms and prefixes, you could wrap a term query against the ngrammed field so that its intervals appear to come from the normal field, and use it an an ordered or unordered interval. This is analogous to the FieldMaskingSpanQuery, but has the advantage that we don't use term statistics for scoring interval queries, so there is no issue with mixing up field weights from different fields. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13116) Add Admin UI login support for Kerberos
[ https://issues.apache.org/jira/browse/SOLR-13116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745044#comment-16745044 ] Jan Høydahl commented on SOLR-13116: Committed. I do not see any need for RefGuide updates, since the Kerberos page already says that users with a valid ticket can use the Admin UI. Wdyt? > Add Admin UI login support for Kerberos > --- > > Key: SOLR-13116 > URL: https://issues.apache.org/jira/browse/SOLR-13116 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: Admin UI >Affects Versions: 8.0, 7.7 >Reporter: Jan Høydahl >Assignee: Jan Høydahl >Priority: Major > Fix For: 8.0, 7.7 > > Attachments: SOLR-13116.patch, SOLR-13116.patch, eventual_auth.png, > improved_login_page.png > > > Spinoff from SOLR-7896. Kerberos auth plugin should get Admin UI Login > support. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13116) Add Admin UI login support for Kerberos
[ https://issues.apache.org/jira/browse/SOLR-13116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745040#comment-16745040 ] ASF subversion and git services commented on SOLR-13116: Commit 2be452d4730cc4d16773b98aa46ff50b3df7c10a in lucene-solr's branch refs/heads/branch_8x from Jan Høydahl [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=2be452d ] SOLR-13116: Add Admin UI login support for Kerberos (cherry picked from commit e515a9140677d76d4d74a32f6c8ca03a2b2f72c5) > Add Admin UI login support for Kerberos > --- > > Key: SOLR-13116 > URL: https://issues.apache.org/jira/browse/SOLR-13116 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: Admin UI >Affects Versions: 8.0, 7.7 >Reporter: Jan Høydahl >Assignee: Jan Høydahl >Priority: Major > Fix For: 8.0, 7.7 > > Attachments: SOLR-13116.patch, SOLR-13116.patch, eventual_auth.png, > improved_login_page.png > > > Spinoff from SOLR-7896. Kerberos auth plugin should get Admin UI Login > support. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8585) Create jump-tables for DocValues at index-time
[
https://issues.apache.org/jira/browse/LUCENE-8585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745060#comment-16745060
]
Toke Eskildsen commented on LUCENE-8585:
I am the one thanking for top-notch feedback. The current pull-request is way
better than the first take.
I have cleaned up the jump-tables relevant testing by pulling that code out of
the general {{BaseDocValuesFormatTestCase}} to {{TestLucene80DocValuesFormat}}
and expanded it a bit. {{TestIndexedDISI}} tests jump-tabls at
{{IndexedDIDI}}-level, so {{BaseDocValuesFormatTestCase}} focuses on Variable
Bits Per Value block jumps. This reduces the test time overhead significantly
and should address the rest of the issues you have raised on the pull request.
Our schedules seem to fit poorly, so there have been long delays in our
communication. I'll prioritize to react quickly during the next week, in the
hope that we can get this pushed to master.
I need a bit of guidance on how to fit this into the 8.0-release. Shall I just
merge the {{master}}-branch into the {{LUCENE-8585}}-branch to keep the
pull-request current?
> Create jump-tables for DocValues at index-time
> --
>
> Key: LUCENE-8585
> URL: https://issues.apache.org/jira/browse/LUCENE-8585
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Affects Versions: 8.0
>Reporter: Toke Eskildsen
>Priority: Minor
> Labels: performance
> Attachments: LUCENE-8585.patch, LUCENE-8585.patch,
> make_patch_lucene8585.sh
>
> Time Spent: 9h 40m
> Remaining Estimate: 0h
>
> As noted in LUCENE-7589, lookup of DocValues should use jump-tables to avoid
> long iterative walks. This is implemented in LUCENE-8374 at search-time
> (first request for DocValues from a field in a segment), with the benefit of
> working without changes to existing Lucene 7 indexes and the downside of
> introducing a startup time penalty and a memory overhead.
> As discussed in LUCENE-8374, the codec should be updated to create these
> jump-tables at index time. This eliminates the segment-open time & memory
> penalties, with the potential downside of increasing index-time for DocValues.
> The three elements of LUCENE-8374 should be transferable to index-time
> without much alteration of the core structures:
> * {{IndexedDISI}} block offset and index skips: A {{long}} (64 bits) for
> every 65536 documents, containing the offset of the block in 33 bits and the
> index (number of set bits) up to the block in 31 bits.
> It can be build sequentially and should be stored as a simple sequence of
> consecutive longs for caching of lookups.
> As it is fairly small, relative to document count, it might be better to
> simply memory cache it?
> * {{IndexedDISI}} DENSE (> 4095, < 65536 set bits) blocks: A {{short}} (16
> bits) for every 8 {{longs}} (512 bits) for a total of 256 bytes/DENSE_block.
> Each {{short}} represents the number of set bits up to right before the
> corresponding sub-block of 512 docIDs.
> The \{{shorts}} can be computed sequentially or when the DENSE block is
> flushed (probably the easiest). They should be stored as a simple sequence of
> consecutive shorts for caching of lookups, one logically independent sequence
> for each DENSE block. The logical position would be one sequence at the start
> of every DENSE block.
> Whether it is best to read all the 16 {{shorts}} up front when a DENSE block
> is accessed or whether it is best to only read any individual {{short}} when
> needed is not clear at this point.
> * Variable Bits Per Value: A {{long}} (64 bits) for every 16384 numeric
> values. Each {{long}} holds the offset to the corresponding block of values.
> The offsets can be computed sequentially and should be stored as a simple
> sequence of consecutive {{longs}} for caching of lookups.
> The vBPV-offsets has the largest space overhead og the 3 jump-tables and a
> lot of the 64 bits in each long are not used for most indexes. They could be
> represented as a simple {{PackedInts}} sequence or {{MonotonicLongValues}},
> with the downsides of a potential lookup-time overhead and the need for doing
> the compression after all offsets has been determined.
> I have no experience with the codec-parts responsible for creating
> index-structures. I'm quite willing to take a stab at this, although I
> probably won't do much about it before January 2019. Should anyone else wish
> to adopt this JIRA-issue or co-work on it, I'll be happy to share.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-12121) JWT Authentication plugin
[
https://issues.apache.org/jira/browse/SOLR-12121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jan Høydahl updated SOLR-12121:
---
Fix Version/s: master (9.0)
> JWT Authentication plugin
> -
>
> Key: SOLR-12121
> URL: https://issues.apache.org/jira/browse/SOLR-12121
> Project: Solr
> Issue Type: New Feature
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Authentication
>Reporter: Jan Høydahl
>Assignee: Jan Høydahl
>Priority: Major
> Fix For: 8.0, master (9.0)
>
> Attachments: image-2018-08-27-13-04-04-183.png
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> A new Authentication plugin that will accept a [Json Web
> Token|https://en.wikipedia.org/wiki/JSON_Web_Token] (JWT) in the
> Authorization header and validate it by checking the cryptographic signature.
> The plugin will not perform the authentication itself but assert that the
> user was authenticated by the service that issued the JWT token.
> JWT defined a number of standard claims, and user principal can be fetched
> from the {{sub}} (subject) claim and passed on to Solr. The plugin will
> always check the {{exp}} (expiry) claim and optionally enforce checks on the
> {{iss}} (issuer) and {{aud}} (audience) claims.
> The first version of the plugin will only support RSA signing keys and will
> support fetching the public key of the issuer through a [Json Web
> Key|https://tools.ietf.org/html/rfc7517] (JWK) file, either from a https URL
> or from local file.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-13116) Add Admin UI login support for Kerberos
[ https://issues.apache.org/jira/browse/SOLR-13116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jan Høydahl resolved SOLR-13116. Resolution: Fixed > Add Admin UI login support for Kerberos > --- > > Key: SOLR-13116 > URL: https://issues.apache.org/jira/browse/SOLR-13116 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: Admin UI >Affects Versions: 8.0, 7.7 >Reporter: Jan Høydahl >Assignee: Jan Høydahl >Priority: Major > Fix For: 8.0, 7.7 > > Attachments: SOLR-13116.patch, SOLR-13116.patch, eventual_auth.png, > improved_login_page.png > > > Spinoff from SOLR-7896. Kerberos auth plugin should get Admin UI Login > support. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13116) Add Admin UI login support for Kerberos
[ https://issues.apache.org/jira/browse/SOLR-13116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745043#comment-16745043 ] ASF subversion and git services commented on SOLR-13116: Commit c028bd7bae65df736877e3eeaf1b373add83309a in lucene-solr's branch refs/heads/branch_7x from Jan Høydahl [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=c028bd7 ] SOLR-13116: Add Admin UI login support for Kerberos (cherry picked from commit e515a9140677d76d4d74a32f6c8ca03a2b2f72c5) > Add Admin UI login support for Kerberos > --- > > Key: SOLR-13116 > URL: https://issues.apache.org/jira/browse/SOLR-13116 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: Admin UI >Affects Versions: 8.0, 7.7 >Reporter: Jan Høydahl >Assignee: Jan Høydahl >Priority: Major > Fix For: 8.0, 7.7 > > Attachments: SOLR-13116.patch, SOLR-13116.patch, eventual_auth.png, > improved_login_page.png > > > Spinoff from SOLR-7896. Kerberos auth plugin should get Admin UI Login > support. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11126) Node-level health check handler
[ https://issues.apache.org/jira/browse/SOLR-11126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745041#comment-16745041 ] Jan Høydahl commented on SOLR-11126: This issue can be resolved? > Node-level health check handler > --- > > Key: SOLR-11126 > URL: https://issues.apache.org/jira/browse/SOLR-11126 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Anshum Gupta >Assignee: Shalin Shekhar Mangar >Priority: Major > Fix For: 8.0 > > Attachments: SOLR-11126-v2.patch, SOLR-11126.patch, SOLR-11126.patch, > SOLR-11126.patch, SOLR-11126.patch, SOLR-11126.patch > > > Solr used to have the PING handler at core level, but with SolrCloud, we are > missing a node level health check handler. It would be good to have. The API > would look like: > * solr/admin/health (v1 API) > * solr/node/health (v2 API) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13116) Add Admin UI login support for Kerberos
[ https://issues.apache.org/jira/browse/SOLR-13116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745039#comment-16745039 ] ASF subversion and git services commented on SOLR-13116: Commit e515a9140677d76d4d74a32f6c8ca03a2b2f72c5 in lucene-solr's branch refs/heads/master from Jan Høydahl [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=e515a91 ] SOLR-13116: Add Admin UI login support for Kerberos > Add Admin UI login support for Kerberos > --- > > Key: SOLR-13116 > URL: https://issues.apache.org/jira/browse/SOLR-13116 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: Admin UI >Affects Versions: 8.0, 7.7 >Reporter: Jan Høydahl >Assignee: Jan Høydahl >Priority: Major > Fix For: 8.0, 7.7 > > Attachments: SOLR-13116.patch, SOLR-13116.patch, eventual_auth.png, > improved_login_page.png > > > Spinoff from SOLR-7896. Kerberos auth plugin should get Admin UI Login > support. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-13143) Output from ExplainAugmenterFactory and DebugQuery for rerank queries is not same
[
https://issues.apache.org/jira/browse/SOLR-13143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sambhav Kothari updated SOLR-13143:
---
Description:
Currently, if we use the ExplainAugmenterFactory with LtR, instead of using the
model/re-rankers explain method, it uses the default query explain (tf-idf
explanation). This happens because the BasicResultContext doesn't wrap the
query([https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302]
214165a4d/solr/core/src/java/org/apache/solr/response/BasicResultContext.java#L6
7) with the RankQuery when its set to context's query, which is then used by
the ExplainAugmenterFactory.
([https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c45230221416]
5a4d/solr/core/src/java/org/apache/solr/response/transform/ExplainAugmenterFacto
ry.java#L111).
As a result there are discrepancies between queries like -
[http://localhost:8983/solr/collection1/select?q=*:*&collection=collectionName&wt]
=json&fl=[explain style=nl],score&rq=\{!ltr model=linear-model}
[http://localhost:8983/solr/collection1/select?q=*:*&collection=collectionName&wt]
=json&fl=score&rq=\{!ltr model=linear-model}&debugQuery=true
the former outputs the explain from the SimilarityScorer's explain while the
latter uses the correct LtR ModelScorer's explain.
There are a few other problems with the explain augmenter - for eg. it doesn't
work with grouping (although the other doc transformers like LtR's
LTRFeatureLoggerTransformerFactory work with grouping)
Mailing List -
http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201901.mbox/%3C5C3879D702C30144003902B0_0_18802%40msllnjpmsgsv06%3E
was:
Currently, if we use the ExplainAugmenterFactory with LtR, instead of using the
model/re-rankers explain method, it uses the default query explain (tf-idf
explanation). This happens because the BasicResultContext doesn't wrap the
query(https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302
214165a4d/solr/core/src/java/org/apache/solr/response/BasicResultContext.java#L6
7) with the RankQuery when its set to context's query, which is then used by
the ExplainAugmenterFactory.
(https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c45230221416
5a4d/solr/core/src/java/org/apache/solr/response/transform/ExplainAugmenterFacto
ry.java#L111).
As a result there are discrepancies between queries like -
http://localhost:8983/solr/collection1/select?q=*:*&collection=collectionName&wt
=json&fl=[explain style=nl],score&rq=\{!ltr model=linear-model}
http://localhost:8983/solr/collection1/select?q=*:*&collection=collectionName&wt
=json&fl=score&rq=\{!ltr model=linear-model}&debugQuery=true
the former outputs the explain from the SimilarityScorer's explain while the
latter uses the correct LtR ModelScorer's explain.
There are a few other problems with the explain augmenter - for eg. it doesn't
work with grouping (although the other doc transformers like LtR's
LTRFeatureLoggerTransformerFactory work with grouping)
> Output from ExplainAugmenterFactory and DebugQuery for rerank queries is not
> same
> -
>
> Key: SOLR-13143
> URL: https://issues.apache.org/jira/browse/SOLR-13143
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: search
>Affects Versions: 7.5
>Reporter: Sambhav Kothari
>Priority: Minor
> Attachments: bug.patch
>
>
> Currently, if we use the ExplainAugmenterFactory with LtR, instead of using
> the
> model/re-rankers explain method, it uses the default query explain (tf-idf
> explanation). This happens because the BasicResultContext doesn't wrap the
>
> query([https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302]
>
> 214165a4d/solr/core/src/java/org/apache/solr/response/BasicResultContext.java#L6
> 7) with the RankQuery when its set to context's query, which is then used by
> the ExplainAugmenterFactory.
>
> ([https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c45230221416]
>
> 5a4d/solr/core/src/java/org/apache/solr/response/transform/ExplainAugmenterFacto
> ry.java#L111).
> As a result there are discrepancies between queries like -
> [http://localhost:8983/solr/collection1/select?q=*:*&collection=collectionName&wt]
> =json&fl=[explain style=nl],score&rq=\{!ltr model=linear-model}
> [http://localhost:8983/solr/collection1/select?q=*:*&collection=collectionName&wt]
> =json&fl=score&rq=\{!ltr model=linear-model}&debugQuery=true
> the former outputs the explain from the SimilarityScorer's explain while the
> latter uses the correct LtR ModelScorer's explain.
> There are a few other problems with the explain augmenter - for eg. it
>
[jira] [Commented] (SOLR-12955) Refactor DistributedUpdateProcessor
[
https://issues.apache.org/jira/browse/SOLR-12955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745036#comment-16745036
]
Lucene/Solr QA commented on SOLR-12955:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
51s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Release audit (RAT) {color} |
{color:green} 1m 44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Check forbidden APIs {color} |
{color:green} 1m 37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Validate source patterns {color} |
{color:green} 1m 37s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 41m
21s{color} | {color:green} core in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
29s{color} | {color:green} test-framework in the patch passed. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 48m 16s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | SOLR-12955 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12955119/SOLR-12955.patch |
| Optional Tests | compile javac unit ratsources checkforbiddenapis
validatesourcepatterns |
| uname | Linux lucene1-us-west 4.4.0-137-generic #163~14.04.1-Ubuntu SMP Mon
Sep 24 17:14:57 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | ant |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-SOLR-Build/sourcedir/dev-tools/test-patch/lucene-solr-yetus-personality.sh
|
| git revision | master / f2352e9 |
| ant | version: Apache Ant(TM) version 1.9.3 compiled on July 24 2018 |
| Default Java | 1.8.0_191 |
| Test Results |
https://builds.apache.org/job/PreCommit-SOLR-Build/263/testReport/ |
| modules | C: solr/core solr/test-framework U: solr |
| Console output |
https://builds.apache.org/job/PreCommit-SOLR-Build/263/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> Refactor DistributedUpdateProcessor
> ---
>
> Key: SOLR-12955
> URL: https://issues.apache.org/jira/browse/SOLR-12955
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Bar Rotstein
>Assignee: David Smiley
>Priority: Major
> Attachments: SOLR-12955-no-commit.patch, SOLR-12955.patch,
> SOLR-12955.patch, SOLR-12955.patch
>
> Time Spent: 8h 20m
> Remaining Estimate: 0h
>
> Lately As I was skimming through Solr's code base I noticed that
> DistributedUpdateProcessor has a lot of nested if else statements, which
> hampers code readability.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-13136) Queries fail during shard creation [testcase included]
[ https://issues.apache.org/jira/browse/SOLR-13136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bram Van Dam updated SOLR-13136: Issue Type: Bug (was: Improvement) > Queries fail during shard creation [testcase included] > -- > > Key: SOLR-13136 > URL: https://issues.apache.org/jira/browse/SOLR-13136 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Affects Versions: 7.6 >Reporter: Bram Van Dam >Priority: Major > Attachments: TestSearchDuringShardCreation.java > > > Shard creation takes a certain amount of time. Queries launched between the > start of shard creation and the full "upness" of the shard will fail: "Error > from server at foo: no servers hosting shard: bar". > Ideally, shards should not be used in queries before their creation is fully > completed. A new shard is empty anyway, so its upness couldn't even influence > the query results in any way. > I'm attaching a testcase which reproduces the problem. Tested on two > machines, but your mileage may vary assuming this is a concurrency issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8642) RamUsageTester.sizeOf ignores arrays and collections if --illegal-access=deny
[ https://issues.apache.org/jira/browse/LUCENE-8642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745015#comment-16745015 ] Uwe Schindler commented on LUCENE-8642: --- I think the approach is fine, we just need more test runs to catch all problems with the new approach. > RamUsageTester.sizeOf ignores arrays and collections if --illegal-access=deny > - > > Key: LUCENE-8642 > URL: https://issues.apache.org/jira/browse/LUCENE-8642 > Project: Lucene - Core > Issue Type: Bug >Reporter: Dawid Weiss >Assignee: Dawid Weiss >Priority: Minor > Fix For: master (9.0) > > > We have a silent fall-through on catch clause there if a field is > inaccessible. This causes all collections to not be recursively descended > into. > We could hack it so that collection objects are counted "manually". Or we > could propagate illegal access exception from ram tester, detect which tests/ > objects do try to measure collection objects and either remove them (if it's > not possible to measure accurately) or change them (if it is possible to > estimate the size in a different way). > Looking for feedback on this one. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13143) Output from ExplainAugmenterFactory and DebugQuery for rerank queries is not same
[
https://issues.apache.org/jira/browse/SOLR-13143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745004#comment-16745004
]
Sambhav Kothari commented on SOLR-13143:
Attaching the failing test as a patch.
> Output from ExplainAugmenterFactory and DebugQuery for rerank queries is not
> same
> -
>
> Key: SOLR-13143
> URL: https://issues.apache.org/jira/browse/SOLR-13143
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: search
>Affects Versions: 7.5
>Reporter: Sambhav Kothari
>Priority: Minor
> Labels: patch
> Attachments: bug.patch
>
>
> Currently, if we use the ExplainAugmenterFactory with LtR, instead of using
> the
> model/re-rankers explain method, it uses the default query explain (tf-idf
> explanation). This happens because the BasicResultContext doesn't wrap the
> query(https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302
> 214165a4d/solr/core/src/java/org/apache/solr/response/BasicResultContext.java#L6
> 7) with the RankQuery when its set to context's query, which is then used by
> the ExplainAugmenterFactory.
> (https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c45230221416
> 5a4d/solr/core/src/java/org/apache/solr/response/transform/ExplainAugmenterFacto
> ry.java#L111).
> As a result there are discrepancies between queries like -
> http://localhost:8983/solr/collection1/select?q=*:*&collection=collectionName&wt
> =json&fl=[explain style=nl],score&rq=\{!ltr model=linear-model}
> http://localhost:8983/solr/collection1/select?q=*:*&collection=collectionName&wt
> =json&fl=score&rq=\{!ltr model=linear-model}&debugQuery=true
> the former outputs the explain from the SimilarityScorer's explain while the
> latter uses the correct LtR ModelScorer's explain.
> There are a few other problems with the explain augmenter - for eg. it
> doesn't
> work with grouping (although the other doc transformers like LtR's
> LTRFeatureLoggerTransformerFactory work with grouping)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-13143) Output from ExplainAugmenterFactory and DebugQuery for rerank queries is not same
[
https://issues.apache.org/jira/browse/SOLR-13143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sambhav Kothari updated SOLR-13143:
---
Labels: (was: patch)
> Output from ExplainAugmenterFactory and DebugQuery for rerank queries is not
> same
> -
>
> Key: SOLR-13143
> URL: https://issues.apache.org/jira/browse/SOLR-13143
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: search
>Affects Versions: 7.5
>Reporter: Sambhav Kothari
>Priority: Minor
> Attachments: bug.patch
>
>
> Currently, if we use the ExplainAugmenterFactory with LtR, instead of using
> the
> model/re-rankers explain method, it uses the default query explain (tf-idf
> explanation). This happens because the BasicResultContext doesn't wrap the
> query(https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302
> 214165a4d/solr/core/src/java/org/apache/solr/response/BasicResultContext.java#L6
> 7) with the RankQuery when its set to context's query, which is then used by
> the ExplainAugmenterFactory.
> (https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c45230221416
> 5a4d/solr/core/src/java/org/apache/solr/response/transform/ExplainAugmenterFacto
> ry.java#L111).
> As a result there are discrepancies between queries like -
> http://localhost:8983/solr/collection1/select?q=*:*&collection=collectionName&wt
> =json&fl=[explain style=nl],score&rq=\{!ltr model=linear-model}
> http://localhost:8983/solr/collection1/select?q=*:*&collection=collectionName&wt
> =json&fl=score&rq=\{!ltr model=linear-model}&debugQuery=true
> the former outputs the explain from the SimilarityScorer's explain while the
> latter uses the correct LtR ModelScorer's explain.
> There are a few other problems with the explain augmenter - for eg. it
> doesn't
> work with grouping (although the other doc transformers like LtR's
> LTRFeatureLoggerTransformerFactory work with grouping)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-13143) Output from ExplainAugmenterFactory and DebugQuery for rerank queries is not same
Sambhav Kothari created SOLR-13143:
--
Summary: Output from ExplainAugmenterFactory and DebugQuery for
rerank queries is not same
Key: SOLR-13143
URL: https://issues.apache.org/jira/browse/SOLR-13143
Project: Solr
Issue Type: Bug
Security Level: Public (Default Security Level. Issues are Public)
Components: search
Affects Versions: 7.5
Reporter: Sambhav Kothari
Attachments: bug.patch
Currently, if we use the ExplainAugmenterFactory with LtR, instead of using the
model/re-rankers explain method, it uses the default query explain (tf-idf
explanation). This happens because the BasicResultContext doesn't wrap the
query(https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302
214165a4d/solr/core/src/java/org/apache/solr/response/BasicResultContext.java#L6
7) with the RankQuery when its set to context's query, which is then used by
the ExplainAugmenterFactory.
(https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c45230221416
5a4d/solr/core/src/java/org/apache/solr/response/transform/ExplainAugmenterFacto
ry.java#L111).
As a result there are discrepancies between queries like -
http://localhost:8983/solr/collection1/select?q=*:*&collection=collectionName&wt
=json&fl=[explain style=nl],score&rq=\{!ltr model=linear-model}
http://localhost:8983/solr/collection1/select?q=*:*&collection=collectionName&wt
=json&fl=score&rq=\{!ltr model=linear-model}&debugQuery=true
the former outputs the explain from the SimilarityScorer's explain while the
latter uses the correct LtR ModelScorer's explain.
There are a few other problems with the explain augmenter - for eg. it doesn't
work with grouping (although the other doc transformers like LtR's
LTRFeatureLoggerTransformerFactory work with grouping)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8642) RamUsageTester.sizeOf ignores arrays and collections if --illegal-access=deny
[ https://issues.apache.org/jira/browse/LUCENE-8642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16745001#comment-16745001 ] Dawid Weiss commented on LUCENE-8642: - Had to revert from master because it does show some errors there in my local runs (accounting differences). > RamUsageTester.sizeOf ignores arrays and collections if --illegal-access=deny > - > > Key: LUCENE-8642 > URL: https://issues.apache.org/jira/browse/LUCENE-8642 > Project: Lucene - Core > Issue Type: Bug >Reporter: Dawid Weiss >Assignee: Dawid Weiss >Priority: Minor > Fix For: master (9.0) > > > We have a silent fall-through on catch clause there if a field is > inaccessible. This causes all collections to not be recursively descended > into. > We could hack it so that collection objects are counted "manually". Or we > could propagate illegal access exception from ram tester, detect which tests/ > objects do try to measure collection objects and either remove them (if it's > not possible to measure accurately) or change them (if it is possible to > estimate the size in a different way). > Looking for feedback on this one. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[JENKINS] Lucene-Solr-repro - Build # 2691 - Unstable
Build: https://builds.apache.org/job/Lucene-Solr-repro/2691/ [...truncated 29 lines...] [repro] Jenkins log URL: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.x/433/consoleText [repro] Revision: 6d63958821232699f0a8423d9b21d4915bfba64e [repro] Ant options: -Dtests.multiplier=2 -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/test-data/enwiki.random.lines.txt [repro] Repro line: ant test -Dtestcase=UnloadDistributedZkTest -Dtests.method=test -Dtests.seed=376100A047775059 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/test-data/enwiki.random.lines.txt -Dtests.locale=ar-SA -Dtests.timezone=SystemV/PST8 -Dtests.asserts=true -Dtests.file.encoding=US-ASCII [repro] Repro line: ant test -Dtestcase=LIROnShardRestartTest -Dtests.method=testAllReplicasInLIR -Dtests.seed=376100A047775059 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/test-data/enwiki.random.lines.txt -Dtests.locale=ja-JP -Dtests.timezone=Pacific/Pitcairn -Dtests.asserts=true -Dtests.file.encoding=US-ASCII [repro] Repro line: ant test -Dtestcase=HdfsUnloadDistributedZkTest -Dtests.method=test -Dtests.seed=376100A047775059 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/test-data/enwiki.random.lines.txt -Dtests.locale=hu-HU -Dtests.timezone=Indian/Chagos -Dtests.asserts=true -Dtests.file.encoding=US-ASCII [repro] git rev-parse --abbrev-ref HEAD [repro] git rev-parse HEAD [repro] Initial local git branch/revision: f2352e9456e9520c75a792533c3de3bf7e58f777 [repro] git fetch [repro] git checkout 6d63958821232699f0a8423d9b21d4915bfba64e [...truncated 2 lines...] [repro] git merge --ff-only [...truncated 1 lines...] [repro] ant clean [...truncated 6 lines...] [repro] Test suites by module: [repro]solr/core [repro] LIROnShardRestartTest [repro] HdfsUnloadDistributedZkTest [repro] UnloadDistributedZkTest [repro] ant compile-test [...truncated 3605 lines...] [repro] ant test-nocompile -Dtests.dups=5 -Dtests.maxfailures=15 -Dtests.class="*.LIROnShardRestartTest|*.HdfsUnloadDistributedZkTest|*.UnloadDistributedZkTest" -Dtests.showOutput=onerror -Dtests.multiplier=2 -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/test-data/enwiki.random.lines.txt -Dtests.seed=376100A047775059 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/test-data/enwiki.random.lines.txt -Dtests.locale=ja-JP -Dtests.timezone=Pacific/Pitcairn -Dtests.asserts=true -Dtests.file.encoding=US-ASCII [...truncated 6208 lines...] [repro] Setting last failure code to 256 [repro] Failures: [repro] 0/5 failed: org.apache.solr.cloud.UnloadDistributedZkTest [repro] 0/5 failed: org.apache.solr.cloud.hdfs.HdfsUnloadDistributedZkTest [repro] 2/5 failed: org.apache.solr.cloud.LIROnShardRestartTest [repro] git checkout f2352e9456e9520c75a792533c3de3bf7e58f777 [...truncated 2 lines...] [repro] Exiting with code 256 [...truncated 6 lines...] - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13125) Optimize Queries when sorting by router.field
[
https://issues.apache.org/jira/browse/SOLR-13125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16744974#comment-16744974
]
Lucene/Solr QA commented on SOLR-13125:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
44s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Release audit (RAT) {color} |
{color:green} 1m 32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Check forbidden APIs {color} |
{color:green} 1m 32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Validate source patterns {color} |
{color:green} 1m 32s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 39m
19s{color} | {color:green} core in the patch passed. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 45m 14s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | SOLR-13125 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12955049/SOLR-13125.patch |
| Optional Tests | compile javac unit ratsources checkforbiddenapis
validatesourcepatterns |
| uname | Linux lucene1-us-west 4.4.0-137-generic #163~14.04.1-Ubuntu SMP Mon
Sep 24 17:14:57 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | ant |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-SOLR-Build/sourcedir/dev-tools/test-patch/lucene-solr-yetus-personality.sh
|
| git revision | master / a16f083 |
| ant | version: Apache Ant(TM) version 1.9.3 compiled on July 24 2018 |
| Default Java | 1.8.0_191 |
| Test Results |
https://builds.apache.org/job/PreCommit-SOLR-Build/262/testReport/ |
| modules | C: solr/core U: solr/core |
| Console output |
https://builds.apache.org/job/PreCommit-SOLR-Build/262/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> Optimize Queries when sorting by router.field
> -
>
> Key: SOLR-13125
> URL: https://issues.apache.org/jira/browse/SOLR-13125
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: mosh
>Priority: Minor
> Attachments: SOLR-13125-no-commit.patch, SOLR-13125.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We are currently testing TRA using Solr 7.7, having >300 shards in the alias,
> with much growth in the coming months.
> The "hot" data(in our case, more recent) will be stored on stronger
> nodes(SSD, more RAM, etc).
> A proposal of optimizing queries sorted by router.field(the field which TRA
> uses to route the data to the correct collection) has emerged.
> Perhaps, in queries which are sorted by router.field, Solr could be smart
> enough to wait for the more recent collections, and in case the limit was
> reached cancel other queries(or just not block and wait for the results)?
> For example:
> When querying a TRA which with a filter on a different field than
> router.field, but sorting by router.field desc, limit=100.
> Since this is a TRA, solr will issue queries for all the collections in the
> alias.
> But to optimize this particular type of query, Solr could wait for the most
> recent collection in the TRA, see whether the result set matches or exceeds
> the limit. If so, the query could be returned to the user without waiting for
> the rest of the shards. If not, the issuing node will block until the second
> query returns, and so forth, until the limit of the request is reached.
> This might also be useful for deep paging, querying each collection and only
> skipping to the next once there are no more results in the specified
> collection.
> Thoughts or inputs are always welcome.
> This is just my two cents, and I'm always happy to brainstorm.
> Thanks in advance.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
--
[jira] [Commented] (LUCENE-8641) TermInSetQueryTest.testRamBytesUsed fails with an assertion on Java > 1.8
[
https://issues.apache.org/jira/browse/LUCENE-8641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16744965#comment-16744965
]
ASF subversion and git services commented on LUCENE-8641:
-
Commit f2352e9456e9520c75a792533c3de3bf7e58f777 in lucene-solr's branch
refs/heads/master from Dawid Weiss
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=f2352e9 ]
Revert "LUCENE-8642, LUCENE-8641: correct RamUsageTester.sizeOf's handling of
ByteBuffers. Throw exceptions on denied reflection to catch problems early.
This affects tests only."
This reverts commit a16f0833edb744e7bd0dfadce7e7d41e1d3aee2c.
> TermInSetQueryTest.testRamBytesUsed fails with an assertion on Java > 1.8
> -
>
> Key: LUCENE-8641
> URL: https://issues.apache.org/jira/browse/LUCENE-8641
> Project: Lucene - Core
> Issue Type: Bug
>Reporter: Dawid Weiss
>Assignee: Dawid Weiss
>Priority: Minor
> Fix For: master (9.0)
>
>
> So, this is strange. I get a fully reproducible numbers going in for the same
> seed in this test, but the failures observed on Jenkins are non-reproducible
> in Idea (but they can be reproduced from ant).
> An example:
> {code}
> ant test -Dtestcase=TermInSetQueryTest -Dtests.method=testRamBytesUsed
> -Dtests.seed=D8EACD24EB26B7FD -Dtests.multiplier=3 -Dtests.slow=true
> -Dtests.badapples=true -Dtests.locale=kw
> -Dtests.timezone=America/North_Dakota/New_Salem -Dtests.asserts=true
> -Dtests.file.encoding=UTF-8
> {code}
> Something is odd.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8642) RamUsageTester.sizeOf ignores arrays and collections if --illegal-access=deny
[ https://issues.apache.org/jira/browse/LUCENE-8642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16744964#comment-16744964 ] ASF subversion and git services commented on LUCENE-8642: - Commit f2352e9456e9520c75a792533c3de3bf7e58f777 in lucene-solr's branch refs/heads/master from Dawid Weiss [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=f2352e9 ] Revert "LUCENE-8642, LUCENE-8641: correct RamUsageTester.sizeOf's handling of ByteBuffers. Throw exceptions on denied reflection to catch problems early. This affects tests only." This reverts commit a16f0833edb744e7bd0dfadce7e7d41e1d3aee2c. > RamUsageTester.sizeOf ignores arrays and collections if --illegal-access=deny > - > > Key: LUCENE-8642 > URL: https://issues.apache.org/jira/browse/LUCENE-8642 > Project: Lucene - Core > Issue Type: Bug >Reporter: Dawid Weiss >Assignee: Dawid Weiss >Priority: Minor > Fix For: master (9.0) > > > We have a silent fall-through on catch clause there if a field is > inaccessible. This causes all collections to not be recursively descended > into. > We could hack it so that collection objects are counted "manually". Or we > could propagate illegal access exception from ram tester, detect which tests/ > objects do try to measure collection objects and either remove them (if it's > not possible to measure accurately) or change them (if it is possible to > estimate the size in a different way). > Looking for feedback on this one. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Reopened] (LUCENE-8642) RamUsageTester.sizeOf ignores arrays and collections if --illegal-access=deny
[ https://issues.apache.org/jira/browse/LUCENE-8642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dawid Weiss reopened LUCENE-8642: - Reopening for backports. > RamUsageTester.sizeOf ignores arrays and collections if --illegal-access=deny > - > > Key: LUCENE-8642 > URL: https://issues.apache.org/jira/browse/LUCENE-8642 > Project: Lucene - Core > Issue Type: Bug >Reporter: Dawid Weiss >Assignee: Dawid Weiss >Priority: Minor > Fix For: master (9.0) > > > We have a silent fall-through on catch clause there if a field is > inaccessible. This causes all collections to not be recursively descended > into. > We could hack it so that collection objects are counted "manually". Or we > could propagate illegal access exception from ram tester, detect which tests/ > objects do try to measure collection objects and either remove them (if it's > not possible to measure accurately) or change them (if it is possible to > estimate the size in a different way). > Looking for feedback on this one. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-13129) Document nested child docs in the ref guide
[
https://issues.apache.org/jira/browse/SOLR-13129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16744960#comment-16744960
]
mosh edited comment on SOLR-13129 at 1/17/19 11:58 AM:
---
this is a more complex issue that has multiple aspects – more "cross-cutting",
and so IMO doesn't belong in the current doc pigeon holes.\{quote}
I have been wondering whether this topic deserves its own page in the ref
guide.
WDYT?
was (Author: moshebla):
{quote}this is a more complex issue that has multiple aspects – more
"cross-cutting", and so IMO doesn't belong in the current doc pigeon
holes.\{quote}
I am wondering whether this topic deserves its own page in the ref guide.
WDYT?
> Document nested child docs in the ref guide
> ---
>
> Key: SOLR-13129
> URL: https://issues.apache.org/jira/browse/SOLR-13129
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: documentation
>Affects Versions: 8.0
>Reporter: David Smiley
>Priority: Major
> Fix For: 8.0
>
>
> Solr 8.0 will have nicer support for nested child documents than its
> predecessors. This should be documented in one place in the ref guide (to
> the extent that makes sense). Users need to the schema ramifications (incl.
> special fields and that some aspects are optional and when), what a nested
> document "looks like" (XML, JSON, SolrJ), how to use the child doc
> transformer, how to use block join queries, and get some overview of how this
> all works. Maybe mention some plausible future enhancements / direction this
> is going in (e.g. path query language?). Some of this is already done but
> it's in various places and could be moved. Unlike other features which
> conveniently fit into one spot in the documentation (like a query parser),
> this is a more complex issue that has multiple aspects – more
> "cross-cutting", and so IMO doesn't belong in the current doc pigeon holes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13129) Document nested child docs in the ref guide
[
https://issues.apache.org/jira/browse/SOLR-13129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16744960#comment-16744960
]
mosh commented on SOLR-13129:
-
{quote}this is a more complex issue that has multiple aspects – more
"cross-cutting", and so IMO doesn't belong in the current doc pigeon
holes.\{quote}
I am wondering whether this topic deserves its own page in the ref guide.
WDYT?
> Document nested child docs in the ref guide
> ---
>
> Key: SOLR-13129
> URL: https://issues.apache.org/jira/browse/SOLR-13129
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: documentation
>Affects Versions: 8.0
>Reporter: David Smiley
>Priority: Major
> Fix For: 8.0
>
>
> Solr 8.0 will have nicer support for nested child documents than its
> predecessors. This should be documented in one place in the ref guide (to
> the extent that makes sense). Users need to the schema ramifications (incl.
> special fields and that some aspects are optional and when), what a nested
> document "looks like" (XML, JSON, SolrJ), how to use the child doc
> transformer, how to use block join queries, and get some overview of how this
> all works. Maybe mention some plausible future enhancements / direction this
> is going in (e.g. path query language?). Some of this is already done but
> it's in various places and could be moved. Unlike other features which
> conveniently fit into one spot in the documentation (like a query parser),
> this is a more complex issue that has multiple aspects – more
> "cross-cutting", and so IMO doesn't belong in the current doc pigeon holes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8642) RamUsageTester.sizeOf ignores arrays and collections if --illegal-access=deny
[ https://issues.apache.org/jira/browse/LUCENE-8642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16744956#comment-16744956 ] Alan Woodward commented on LUCENE-8642: --- Thanks [~dweiss]! > RamUsageTester.sizeOf ignores arrays and collections if --illegal-access=deny > - > > Key: LUCENE-8642 > URL: https://issues.apache.org/jira/browse/LUCENE-8642 > Project: Lucene - Core > Issue Type: Bug >Reporter: Dawid Weiss >Assignee: Dawid Weiss >Priority: Minor > Fix For: master (9.0) > > > We have a silent fall-through on catch clause there if a field is > inaccessible. This causes all collections to not be recursively descended > into. > We could hack it so that collection objects are counted "manually". Or we > could propagate illegal access exception from ram tester, detect which tests/ > objects do try to measure collection objects and either remove them (if it's > not possible to measure accurately) or change them (if it is possible to > estimate the size in a different way). > Looking for feedback on this one. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8641) TermInSetQueryTest.testRamBytesUsed fails with an assertion on Java > 1.8
[
https://issues.apache.org/jira/browse/LUCENE-8641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16744950#comment-16744950
]
ASF subversion and git services commented on LUCENE-8641:
-
Commit 7bcad94325c1c4168d35a23fa35850f245dafb80 in lucene-solr's branch
refs/heads/branch_8x from Dawid Weiss
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=7bcad94 ]
Revert "LUCENE-8642, LUCENE-8641: correct RamUsageTester.sizeOf's handling of
ByteBuffers. Throw exceptions on denied reflection to catch problems early.
This affects tests only."
This reverts commit b4524d902df8536887b0caa108634fee8599b65c.
> TermInSetQueryTest.testRamBytesUsed fails with an assertion on Java > 1.8
> -
>
> Key: LUCENE-8641
> URL: https://issues.apache.org/jira/browse/LUCENE-8641
> Project: Lucene - Core
> Issue Type: Bug
>Reporter: Dawid Weiss
>Assignee: Dawid Weiss
>Priority: Minor
> Fix For: master (9.0)
>
>
> So, this is strange. I get a fully reproducible numbers going in for the same
> seed in this test, but the failures observed on Jenkins are non-reproducible
> in Idea (but they can be reproduced from ant).
> An example:
> {code}
> ant test -Dtestcase=TermInSetQueryTest -Dtests.method=testRamBytesUsed
> -Dtests.seed=D8EACD24EB26B7FD -Dtests.multiplier=3 -Dtests.slow=true
> -Dtests.badapples=true -Dtests.locale=kw
> -Dtests.timezone=America/North_Dakota/New_Salem -Dtests.asserts=true
> -Dtests.file.encoding=UTF-8
> {code}
> Something is odd.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8642) RamUsageTester.sizeOf ignores arrays and collections if --illegal-access=deny
[ https://issues.apache.org/jira/browse/LUCENE-8642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16744947#comment-16744947 ] ASF subversion and git services commented on LUCENE-8642: - Commit f592041f0cdc0b60b8c76486ff78cded2821cc78 in lucene-solr's branch refs/heads/branch_7x from Dawid Weiss [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=f592041 ] Revert "LUCENE-8642, LUCENE-8641: correct RamUsageTester.sizeOf's handling of ByteBuffers. Throw exceptions on denied reflection to catch problems early. This affects tests only." This reverts commit ffd8e88774e0eefbcb41720d7a4713f3cf6ad90c. > RamUsageTester.sizeOf ignores arrays and collections if --illegal-access=deny > - > > Key: LUCENE-8642 > URL: https://issues.apache.org/jira/browse/LUCENE-8642 > Project: Lucene - Core > Issue Type: Bug >Reporter: Dawid Weiss >Assignee: Dawid Weiss >Priority: Minor > Fix For: master (9.0) > > > We have a silent fall-through on catch clause there if a field is > inaccessible. This causes all collections to not be recursively descended > into. > We could hack it so that collection objects are counted "manually". Or we > could propagate illegal access exception from ram tester, detect which tests/ > objects do try to measure collection objects and either remove them (if it's > not possible to measure accurately) or change them (if it is possible to > estimate the size in a different way). > Looking for feedback on this one. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
