[jira] [Resolved] (LUCENE-10141) Update releaseWizard for 8x to correctly create back-compat indices and update Version in main after repo split
[ https://issues.apache.org/jira/browse/LUCENE-10141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timothy Potter resolved LUCENE-10141. - Lucene Fields: (was: New) Resolution: Fixed > Update releaseWizard for 8x to correctly create back-compat indices and > update Version in main after repo split > --- > > Key: LUCENE-10141 > URL: https://issues.apache.org/jira/browse/LUCENE-10141 > Project: Lucene - Core > Issue Type: Task > Components: release wizard >Reporter: Timothy Potter >Assignee: Timothy Potter >Priority: Major > Fix For: 8.11 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Need to update the release wizard in 8x to create the back-compat indices and > update the Version info so that issues like: > https://issues.apache.org/jira/browse/LUCENE-10131 don't impact future 8x > release managers. Hopefully an 8.11 is NOT needed but release managers have > enough on their plate to get right that we should fix this if possible. If > not, we at least need to document the process of doing it manually. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-10141) Update releaseWizard for 8x to correctly create back-compat indices and update Version in main after repo split
[ https://issues.apache.org/jira/browse/LUCENE-10141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timothy Potter updated LUCENE-10141: Fix Version/s: 8.10.1 8.11 > Update releaseWizard for 8x to correctly create back-compat indices and > update Version in main after repo split > --- > > Key: LUCENE-10141 > URL: https://issues.apache.org/jira/browse/LUCENE-10141 > Project: Lucene - Core > Issue Type: Task > Components: release wizard >Reporter: Timothy Potter >Assignee: Timothy Potter >Priority: Major > Fix For: 8.11, 8.10.1 > > > Need to update the release wizard in 8x to create the back-compat indices and > update the Version info so that issues like: > https://issues.apache.org/jira/browse/LUCENE-10131 don't impact future 8x > release managers. Hopefully an 8.11 is NOT needed but release managers have > enough on their plate to get right that we should fix this if possible. If > not, we at least need to document the process of doing it manually. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-10131) Release wizard for 8x fails to generate Backcompat Indexes for unstable branch (main)
[
https://issues.apache.org/jira/browse/LUCENE-10131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timothy Potter resolved LUCENE-10131.
-
Fix Version/s: main (9.0)
Assignee: Timothy Potter
Resolution: Fixed
> Release wizard for 8x fails to generate Backcompat Indexes for unstable
> branch (main)
> -
>
> Key: LUCENE-10131
> URL: https://issues.apache.org/jira/browse/LUCENE-10131
> Project: Lucene - Core
> Issue Type: Test
>Reporter: Timothy Potter
>Assignee: Timothy Potter
>Priority: Major
> Fix For: main (9.0)
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> {code}
> org.apache.lucene.backward_index.TestBackwardsCompatibility > test suite's
> output saved to
> /Users/tjp/.lucene-releases/8.10.0/lucene-solr/lucene/backward-codecs/build/test-results/test/outputs/OUTPUT-org.apache.lucene.backward_index.TestBackwardsCompatibility.txt,
> copied below:
>> java.lang.AssertionError: Extra backcompat test files:
>> 8.10.0-cfs
>> at
> __randomizedtesting.SeedInfo.seed([14048777FC78168A:4DB683748EF00A6]:0)
>> at org.junit.Assert.fail(Assert.java:89)
>> at
> org.apache.lucene.backward_index.TestBackwardsCompatibility.testAllVersionsTested(TestBackwardsCompatibility.java:787)
>> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>> at
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1754)
>> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:942)
>> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:978)
>> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:992)
>> at
> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:44)
>> at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
>> at
> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
>> at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
>> at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
>> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
>> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:819)
>> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:470)
>> at
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:951)
>> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:836)
>> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:887)
>> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:898)
>> at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
>> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
>> at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>> at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
>> at
>
[jira] [Created] (LUCENE-10141) Update releaseWizard for 8x to correctly create back-compat indices and update Version in main after repo split
Timothy Potter created LUCENE-10141: --- Summary: Update releaseWizard for 8x to correctly create back-compat indices and update Version in main after repo split Key: LUCENE-10141 URL: https://issues.apache.org/jira/browse/LUCENE-10141 Project: Lucene - Core Issue Type: Task Components: release wizard Reporter: Timothy Potter Assignee: Timothy Potter Need to update the release wizard in 8x to create the back-compat indices and update the Version info so that issues like: https://issues.apache.org/jira/browse/LUCENE-10131 don't impact future 8x release managers. Hopefully an 8.11 is NOT needed but release managers have enough on their plate to get right that we should fix this if possible. If not, we at least need to document the process of doing it manually. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-10131) Release wizard for 8x fails to generate Backcompat Indexes for unstable branch (main)
[
https://issues.apache.org/jira/browse/LUCENE-10131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17421785#comment-17421785
]
Timothy Potter commented on LUCENE-10131:
-
Here's the command the releaseWizard ran against main:
{code}
python3 -u dev-tools/scripts/addBackcompatIndexes.py --temp-dir
/Users/tjp/.lucene-releases/8.10.0/backcompat 8.10.0 && git add
lucene/backward-codecs/src/test/org/apache/lucene/backward_index/' in folder
'/Users/tjp/.lucene-releases/8.10.0/lucene-solr
{code}
> Release wizard for 8x fails to generate Backcompat Indexes for unstable
> branch (main)
> -
>
> Key: LUCENE-10131
> URL: https://issues.apache.org/jira/browse/LUCENE-10131
> Project: Lucene - Core
> Issue Type: Test
>Reporter: Timothy Potter
>Priority: Major
>
> {code}
> org.apache.lucene.backward_index.TestBackwardsCompatibility > test suite's
> output saved to
> /Users/tjp/.lucene-releases/8.10.0/lucene-solr/lucene/backward-codecs/build/test-results/test/outputs/OUTPUT-org.apache.lucene.backward_index.TestBackwardsCompatibility.txt,
> copied below:
>> java.lang.AssertionError: Extra backcompat test files:
>> 8.10.0-cfs
>> at
> __randomizedtesting.SeedInfo.seed([14048777FC78168A:4DB683748EF00A6]:0)
>> at org.junit.Assert.fail(Assert.java:89)
>> at
> org.apache.lucene.backward_index.TestBackwardsCompatibility.testAllVersionsTested(TestBackwardsCompatibility.java:787)
>> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>> at
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1754)
>> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:942)
>> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:978)
>> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:992)
>> at
> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:44)
>> at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
>> at
> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
>> at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
>> at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
>> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
>> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:819)
>> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:470)
>> at
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:951)
>> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:836)
>> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:887)
>> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:898)
>> at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
>> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
>> at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>> at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
[jira] [Created] (LUCENE-10131) Release wizard for 8x fails to generate Backcompat Indexes for unstable branch (main)
Timothy Potter created LUCENE-10131:
---
Summary: Release wizard for 8x fails to generate Backcompat
Indexes for unstable branch (main)
Key: LUCENE-10131
URL: https://issues.apache.org/jira/browse/LUCENE-10131
Project: Lucene - Core
Issue Type: Test
Reporter: Timothy Potter
{code}
org.apache.lucene.backward_index.TestBackwardsCompatibility > test suite's
output saved to
/Users/tjp/.lucene-releases/8.10.0/lucene-solr/lucene/backward-codecs/build/test-results/test/outputs/OUTPUT-org.apache.lucene.backward_index.TestBackwardsCompatibility.txt,
copied below:
> java.lang.AssertionError: Extra backcompat test files:
> 8.10.0-cfs
> at
__randomizedtesting.SeedInfo.seed([14048777FC78168A:4DB683748EF00A6]:0)
> at org.junit.Assert.fail(Assert.java:89)
> at
org.apache.lucene.backward_index.TestBackwardsCompatibility.testAllVersionsTested(TestBackwardsCompatibility.java:787)
> at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1754)
> at
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:942)
> at
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:978)
> at
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:992)
> at
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:44)
> at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> at
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
> at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
> at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
> at
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:819)
> at
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:470)
> at
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:951)
> at
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:836)
> at
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:887)
> at
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:898)
> at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
> at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
> at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
> at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
> at
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
[jira] [Created] (SOLR-15199) bin/solr should expose all actions provided by SolrCLI, such as "api"
Timothy Potter created SOLR-15199:
-
Summary: bin/solr should expose all actions provided by SolrCLI,
such as "api"
Key: SOLR-15199
URL: https://issues.apache.org/jira/browse/SOLR-15199
Project: Solr
Issue Type: Improvement
Security Level: Public (Default Security Level. Issues are Public)
Components: scripts and tools
Reporter: Timothy Potter
Assignee: Timothy Potter
I have need to invoke the "api" action in SolrCLI via bin/solr, but it's not
exposed, so I have to resort to doing this for K8s probes that require auth:
{code}
java -Dbasicauth="$(cat /etc/secrets/dev-solrcloud-basic-auth/username):$(cat
/etc/secrets/dev-solrcloud-basic-auth/password)"
-Djavax.net.ssl.keyStore="$(echo $SOLR_SSL_KEY_STORE)"
-Djavax.net.ssl.keyStorePassword="$(echo $SOLR_SSL_KEY_STORE_PASSWORD)"
-Djavax.net.ssl.trustStore="$(echo $SOLR_SSL_TRUST_STORE)"
-Djavax.net.ssl.trustStorePassword="$(echo $SOLR_SSL_TRUST_STORE_PASSWORD)"
-Dsolr.ssl.checkPeerName=false
-Dsolr.httpclient.builder.factory=org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory
-Dsolr.install.dir="/opt/solr"
-Dlog4j.configurationFile="/opt/solr/server/resources/log4j2-console.xml"
-classpath
"/opt/solr/server/solr-webapp/webapp/WEB-INF/lib/*:/opt/solr/server/lib/ext/*:/opt/solr/server/lib/*"
org.apache.solr.util.SolrCLI api -get
https://localhost:8983/solr/admin/info/system
{code}
In general, we should just have bin/solr fall-thru commands it doesn't
recognize to SolrCLI and let that fail if the command isn't supported. That
way, bin/solr won't need to be updated everytime SolrCLI implements a new
action.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-15181) TestCodecSupport.testDynamicFieldsDocValuesFormats & TestCodecSupport.testDocValuesFormats are failing
[
https://issues.apache.org/jira/browse/SOLR-15181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timothy Potter updated SOLR-15181:
--
Fix Version/s: master (9.0)
Resolution: Fixed
Status: Resolved (was: Patch Available)
> TestCodecSupport.testDynamicFieldsDocValuesFormats &
> TestCodecSupport.testDocValuesFormats are failing
> --
>
> Key: SOLR-15181
> URL: https://issues.apache.org/jira/browse/SOLR-15181
> Project: Solr
> Issue Type: Test
>Reporter: Zach Chen
>Assignee: Timothy Potter
>Priority: Major
> Fix For: master (9.0)
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> I noticed two tests were consistently failing with latest head of master
> branch
> (https://github.com/apache/lucene-solr/tree/4d7b2aebfe01e2a8d0407c179238bb5e092b8d2a):
> Reproduce with
> {code:java}
> ./gradlew clean; ./gradlew :solr:core:test --tests
> "org.apache.solr.core.TestCodecSupport.testDynamicFieldsDocValuesFormats"
> -Ptests.jvms=6 -Ptests.jvmargs=-XX:TieredStopAtLevel=1
> -Ptests.seed=70C20159183C205E -Ptests.file.encoding=US-ASCII
> {code}
> Error stacktrace
> {code:java}
> 2> 3655 INFO
> (TEST-TestCodecSupport.testDynamicFieldsDocValuesFormats-seed#[70C20159183C205E])
> [ ] o.a.s.SolrTestCaseJ4 ###Ending testDynamicFieldsDocValuesFormats
> > org.junit.ComparisonFailure: expected: but
> was:
> > at
> __randomizedtesting.SeedInfo.seed([70C20159183C205E:90115F9F0EA2960A]:0)
> > at org.junit.Assert.assertEquals(Assert.java:117)
> > at org.junit.Assert.assertEquals(Assert.java:146)
> > at
> org.apache.solr.core.TestCodecSupport.testDynamicFieldsDocValuesFormats(TestCodecSupport.java:89)
> {code}
>
> Reproduce with
> {code:java}
> ./gradlew clean; ./gradlew :solr:core:test --tests
> "org.apache.solr.core.TestCodecSupport.testDocValuesFormats" -Ptests.jvms=6
> -Ptests.jvmargs=-XX:TieredStopAtLevel=1 -Ptests.seed=70C20159183C205E
> -Ptests.file.encoding=US-ASCII{code}
> Error stacktrace
> {code:java}
> 2> 4261 INFO
> (TEST-TestCodecSupport.testDocValuesFormats-seed#[70C20159183C205E]) [ ]
> o.a.s.SolrTestCaseJ4 ###Ending testDocValuesFormats
> > org.junit.ComparisonFailure: expected: but
> was:
> > at
> __randomizedtesting.SeedInfo.seed([70C20159183C205E:7C99C9EBF03D80A1]:0)
> > at org.junit.Assert.assertEquals(Assert.java:117)
> > at org.junit.Assert.assertEquals(Assert.java:146)
> > at
> org.apache.solr.core.TestCodecSupport.testDocValuesFormats(TestCodecSupport.java:65)
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-15181) TestCodecSupport.testDynamicFieldsDocValuesFormats & TestCodecSupport.testDocValuesFormats are failing
[
https://issues.apache.org/jira/browse/SOLR-15181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289403#comment-17289403
]
Timothy Potter commented on SOLR-15181:
---
ok PR up for that https://github.com/apache/lucene-solr/pull/2424
> TestCodecSupport.testDynamicFieldsDocValuesFormats &
> TestCodecSupport.testDocValuesFormats are failing
> --
>
> Key: SOLR-15181
> URL: https://issues.apache.org/jira/browse/SOLR-15181
> Project: Solr
> Issue Type: Test
>Reporter: Zach Chen
>Assignee: Timothy Potter
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> I noticed two tests were consistently failing with latest head of master
> branch
> (https://github.com/apache/lucene-solr/tree/4d7b2aebfe01e2a8d0407c179238bb5e092b8d2a):
> Reproduce with
> {code:java}
> ./gradlew clean; ./gradlew :solr:core:test --tests
> "org.apache.solr.core.TestCodecSupport.testDynamicFieldsDocValuesFormats"
> -Ptests.jvms=6 -Ptests.jvmargs=-XX:TieredStopAtLevel=1
> -Ptests.seed=70C20159183C205E -Ptests.file.encoding=US-ASCII
> {code}
> Error stacktrace
> {code:java}
> 2> 3655 INFO
> (TEST-TestCodecSupport.testDynamicFieldsDocValuesFormats-seed#[70C20159183C205E])
> [ ] o.a.s.SolrTestCaseJ4 ###Ending testDynamicFieldsDocValuesFormats
> > org.junit.ComparisonFailure: expected: but
> was:
> > at
> __randomizedtesting.SeedInfo.seed([70C20159183C205E:90115F9F0EA2960A]:0)
> > at org.junit.Assert.assertEquals(Assert.java:117)
> > at org.junit.Assert.assertEquals(Assert.java:146)
> > at
> org.apache.solr.core.TestCodecSupport.testDynamicFieldsDocValuesFormats(TestCodecSupport.java:89)
> {code}
>
> Reproduce with
> {code:java}
> ./gradlew clean; ./gradlew :solr:core:test --tests
> "org.apache.solr.core.TestCodecSupport.testDocValuesFormats" -Ptests.jvms=6
> -Ptests.jvmargs=-XX:TieredStopAtLevel=1 -Ptests.seed=70C20159183C205E
> -Ptests.file.encoding=US-ASCII{code}
> Error stacktrace
> {code:java}
> 2> 4261 INFO
> (TEST-TestCodecSupport.testDocValuesFormats-seed#[70C20159183C205E]) [ ]
> o.a.s.SolrTestCaseJ4 ###Ending testDocValuesFormats
> > org.junit.ComparisonFailure: expected: but
> was:
> > at
> __randomizedtesting.SeedInfo.seed([70C20159183C205E:7C99C9EBF03D80A1]:0)
> > at org.junit.Assert.assertEquals(Assert.java:117)
> > at org.junit.Assert.assertEquals(Assert.java:146)
> > at
> org.apache.solr.core.TestCodecSupport.testDocValuesFormats(TestCodecSupport.java:65)
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Assigned] (SOLR-15181) TestCodecSupport.testDynamicFieldsDocValuesFormats & TestCodecSupport.testDocValuesFormats are failing
[
https://issues.apache.org/jira/browse/SOLR-15181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timothy Potter reassigned SOLR-15181:
-
Assignee: Timothy Potter
> TestCodecSupport.testDynamicFieldsDocValuesFormats &
> TestCodecSupport.testDocValuesFormats are failing
> --
>
> Key: SOLR-15181
> URL: https://issues.apache.org/jira/browse/SOLR-15181
> Project: Solr
> Issue Type: Test
>Reporter: Zach Chen
>Assignee: Timothy Potter
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> I noticed two tests were consistently failing with latest head of master
> branch
> (https://github.com/apache/lucene-solr/tree/4d7b2aebfe01e2a8d0407c179238bb5e092b8d2a):
> Reproduce with
> {code:java}
> ./gradlew clean; ./gradlew :solr:core:test --tests
> "org.apache.solr.core.TestCodecSupport.testDynamicFieldsDocValuesFormats"
> -Ptests.jvms=6 -Ptests.jvmargs=-XX:TieredStopAtLevel=1
> -Ptests.seed=70C20159183C205E -Ptests.file.encoding=US-ASCII
> {code}
> Error stacktrace
> {code:java}
> 2> 3655 INFO
> (TEST-TestCodecSupport.testDynamicFieldsDocValuesFormats-seed#[70C20159183C205E])
> [ ] o.a.s.SolrTestCaseJ4 ###Ending testDynamicFieldsDocValuesFormats
> > org.junit.ComparisonFailure: expected: but
> was:
> > at
> __randomizedtesting.SeedInfo.seed([70C20159183C205E:90115F9F0EA2960A]:0)
> > at org.junit.Assert.assertEquals(Assert.java:117)
> > at org.junit.Assert.assertEquals(Assert.java:146)
> > at
> org.apache.solr.core.TestCodecSupport.testDynamicFieldsDocValuesFormats(TestCodecSupport.java:89)
> {code}
>
> Reproduce with
> {code:java}
> ./gradlew clean; ./gradlew :solr:core:test --tests
> "org.apache.solr.core.TestCodecSupport.testDocValuesFormats" -Ptests.jvms=6
> -Ptests.jvmargs=-XX:TieredStopAtLevel=1 -Ptests.seed=70C20159183C205E
> -Ptests.file.encoding=US-ASCII{code}
> Error stacktrace
> {code:java}
> 2> 4261 INFO
> (TEST-TestCodecSupport.testDocValuesFormats-seed#[70C20159183C205E]) [ ]
> o.a.s.SolrTestCaseJ4 ###Ending testDocValuesFormats
> > org.junit.ComparisonFailure: expected: but
> was:
> > at
> __randomizedtesting.SeedInfo.seed([70C20159183C205E:7C99C9EBF03D80A1]:0)
> > at org.junit.Assert.assertEquals(Assert.java:117)
> > at org.junit.Assert.assertEquals(Assert.java:146)
> > at
> org.apache.solr.core.TestCodecSupport.testDocValuesFormats(TestCodecSupport.java:65)
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-15181) TestCodecSupport.testDynamicFieldsDocValuesFormats & TestCodecSupport.testDocValuesFormats are failing
[
https://issues.apache.org/jira/browse/SOLR-15181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timothy Potter updated SOLR-15181:
--
Status: Patch Available (was: Open)
> TestCodecSupport.testDynamicFieldsDocValuesFormats &
> TestCodecSupport.testDocValuesFormats are failing
> --
>
> Key: SOLR-15181
> URL: https://issues.apache.org/jira/browse/SOLR-15181
> Project: Solr
> Issue Type: Test
>Reporter: Zach Chen
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> I noticed two tests were consistently failing with latest head of master
> branch
> (https://github.com/apache/lucene-solr/tree/4d7b2aebfe01e2a8d0407c179238bb5e092b8d2a):
> Reproduce with
> {code:java}
> ./gradlew clean; ./gradlew :solr:core:test --tests
> "org.apache.solr.core.TestCodecSupport.testDynamicFieldsDocValuesFormats"
> -Ptests.jvms=6 -Ptests.jvmargs=-XX:TieredStopAtLevel=1
> -Ptests.seed=70C20159183C205E -Ptests.file.encoding=US-ASCII
> {code}
> Error stacktrace
> {code:java}
> 2> 3655 INFO
> (TEST-TestCodecSupport.testDynamicFieldsDocValuesFormats-seed#[70C20159183C205E])
> [ ] o.a.s.SolrTestCaseJ4 ###Ending testDynamicFieldsDocValuesFormats
> > org.junit.ComparisonFailure: expected: but
> was:
> > at
> __randomizedtesting.SeedInfo.seed([70C20159183C205E:90115F9F0EA2960A]:0)
> > at org.junit.Assert.assertEquals(Assert.java:117)
> > at org.junit.Assert.assertEquals(Assert.java:146)
> > at
> org.apache.solr.core.TestCodecSupport.testDynamicFieldsDocValuesFormats(TestCodecSupport.java:89)
> {code}
>
> Reproduce with
> {code:java}
> ./gradlew clean; ./gradlew :solr:core:test --tests
> "org.apache.solr.core.TestCodecSupport.testDocValuesFormats" -Ptests.jvms=6
> -Ptests.jvmargs=-XX:TieredStopAtLevel=1 -Ptests.seed=70C20159183C205E
> -Ptests.file.encoding=US-ASCII{code}
> Error stacktrace
> {code:java}
> 2> 4261 INFO
> (TEST-TestCodecSupport.testDocValuesFormats-seed#[70C20159183C205E]) [ ]
> o.a.s.SolrTestCaseJ4 ###Ending testDocValuesFormats
> > org.junit.ComparisonFailure: expected: but
> was:
> > at
> __randomizedtesting.SeedInfo.seed([70C20159183C205E:7C99C9EBF03D80A1]:0)
> > at org.junit.Assert.assertEquals(Assert.java:117)
> > at org.junit.Assert.assertEquals(Assert.java:146)
> > at
> org.apache.solr.core.TestCodecSupport.testDocValuesFormats(TestCodecSupport.java:65)
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-15181) TestCodecSupport.testDynamicFieldsDocValuesFormats & TestCodecSupport.testDocValuesFormats are failing
[
https://issues.apache.org/jira/browse/SOLR-15181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289328#comment-17289328
]
Timothy Potter commented on SOLR-15181:
---
fwiw ~ they pass with this change, but I'm not sure if that's the "right"
solution here:
{code}
diff --git a/solr/core/src/test-files/solr/collection1/conf/schema_codec.xml
b/solr/core/src/test-files/solr/collection1/conf/schema_codec.xml
index b8a1a0b5e56..e73b621fe63 100644
--- a/solr/core/src/test-files/solr/collection1/conf/schema_codec.xml
+++ b/solr/core/src/test-files/solr/collection1/conf/schema_codec.xml
@@ -22,7 +22,7 @@
-
+
{code}
> TestCodecSupport.testDynamicFieldsDocValuesFormats &
> TestCodecSupport.testDocValuesFormats are failing
> --
>
> Key: SOLR-15181
> URL: https://issues.apache.org/jira/browse/SOLR-15181
> Project: Solr
> Issue Type: Test
>Reporter: Zach Chen
>Priority: Major
>
> I noticed two tests were consistently failing with latest head of master
> branch
> (https://github.com/apache/lucene-solr/tree/4d7b2aebfe01e2a8d0407c179238bb5e092b8d2a):
> Reproduce with
> {code:java}
> ./gradlew clean; ./gradlew :solr:core:test --tests
> "org.apache.solr.core.TestCodecSupport.testDynamicFieldsDocValuesFormats"
> -Ptests.jvms=6 -Ptests.jvmargs=-XX:TieredStopAtLevel=1
> -Ptests.seed=70C20159183C205E -Ptests.file.encoding=US-ASCII
> {code}
> Error stacktrace
> {code:java}
> 2> 3655 INFO
> (TEST-TestCodecSupport.testDynamicFieldsDocValuesFormats-seed#[70C20159183C205E])
> [ ] o.a.s.SolrTestCaseJ4 ###Ending testDynamicFieldsDocValuesFormats
> > org.junit.ComparisonFailure: expected: but
> was:
> > at
> __randomizedtesting.SeedInfo.seed([70C20159183C205E:90115F9F0EA2960A]:0)
> > at org.junit.Assert.assertEquals(Assert.java:117)
> > at org.junit.Assert.assertEquals(Assert.java:146)
> > at
> org.apache.solr.core.TestCodecSupport.testDynamicFieldsDocValuesFormats(TestCodecSupport.java:89)
> {code}
>
> Reproduce with
> {code:java}
> ./gradlew clean; ./gradlew :solr:core:test --tests
> "org.apache.solr.core.TestCodecSupport.testDocValuesFormats" -Ptests.jvms=6
> -Ptests.jvmargs=-XX:TieredStopAtLevel=1 -Ptests.seed=70C20159183C205E
> -Ptests.file.encoding=US-ASCII{code}
> Error stacktrace
> {code:java}
> 2> 4261 INFO
> (TEST-TestCodecSupport.testDocValuesFormats-seed#[70C20159183C205E]) [ ]
> o.a.s.SolrTestCaseJ4 ###Ending testDocValuesFormats
> > org.junit.ComparisonFailure: expected: but
> was:
> > at
> __randomizedtesting.SeedInfo.seed([70C20159183C205E:7C99C9EBF03D80A1]:0)
> > at org.junit.Assert.assertEquals(Assert.java:117)
> > at org.junit.Assert.assertEquals(Assert.java:146)
> > at
> org.apache.solr.core.TestCodecSupport.testDocValuesFormats(TestCodecSupport.java:65)
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-15179) Release wizard should have the option to source Apache credentials and signing key passphrase from local file vs. prompting the user
Timothy Potter created SOLR-15179: - Summary: Release wizard should have the option to source Apache credentials and signing key passphrase from local file vs. prompting the user Key: SOLR-15179 URL: https://issues.apache.org/jira/browse/SOLR-15179 Project: Solr Issue Type: Improvement Security Level: Public (Default Security Level. Issues are Public) Components: release-scripts Reporter: Timothy Potter It would be a nice improvement for the release wizard to source Apache credentials and the signing key passphrase from a local file, say `~/.lucene-release-wizard-input` or similar. This should be optional but use it instead of prompting the release manager for these credentials multiple times. The wizard should add a reminder step at the beginning to create this file and at end of the process to delete this file. cc [~janhoy] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (SOLR-15135) Admin UI does not accurately reflect node PerReplicaStates status
[ https://issues.apache.org/jira/browse/SOLR-15135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timothy Potter resolved SOLR-15135. --- Resolution: Fixed > Admin UI does not accurately reflect node PerReplicaStates status > - > > Key: SOLR-15135 > URL: https://issues.apache.org/jira/browse/SOLR-15135 > Project: Solr > Issue Type: Bug > Components: Admin UI >Affects Versions: 8.8 >Reporter: Mike Drob >Assignee: Timothy Potter >Priority: Blocker > Fix For: master (9.0), 8.9, 8.8.1 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > When creating collections using the new PRS, the admin UI will show them as > down because it only looks at state.json and not the correct PRS node. > cc: [~noble] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-15135) Admin UI does not accurately reflect node PerReplicaStates status
[ https://issues.apache.org/jira/browse/SOLR-15135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timothy Potter updated SOLR-15135: -- Fix Version/s: 8.8.1 8.9 master (9.0) > Admin UI does not accurately reflect node PerReplicaStates status > - > > Key: SOLR-15135 > URL: https://issues.apache.org/jira/browse/SOLR-15135 > Project: Solr > Issue Type: Bug > Components: Admin UI >Affects Versions: 8.8 >Reporter: Mike Drob >Assignee: Timothy Potter >Priority: Blocker > Fix For: master (9.0), 8.9, 8.8.1 > > > When creating collections using the new PRS, the admin UI will show them as > down because it only looks at state.json and not the correct PRS node. > cc: [~noble] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Assigned] (SOLR-15135) Admin UI does not accurately reflect node PerReplicaStates status
[ https://issues.apache.org/jira/browse/SOLR-15135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timothy Potter reassigned SOLR-15135: - Assignee: Timothy Potter > Admin UI does not accurately reflect node PerReplicaStates status > - > > Key: SOLR-15135 > URL: https://issues.apache.org/jira/browse/SOLR-15135 > Project: Solr > Issue Type: Bug > Components: Admin UI >Affects Versions: 8.8 >Reporter: Mike Drob >Assignee: Timothy Potter >Priority: Blocker > > When creating collections using the new PRS, the admin UI will show them as > down because it only looks at state.json and not the correct PRS node. > cc: [~noble] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-15135) Admin UI does not accurately reflect node PerReplicaStates status
[
https://issues.apache.org/jira/browse/SOLR-15135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17285396#comment-17285396
]
Timothy Potter commented on SOLR-15135:
---
This is would be a trivial change in the {{ZookeeperInfoHandler}} if only there
was some utility method that creates the equivalent structure of {{state.json}}
from {{PerReplicaStates}} ... does such a util exist [~ichattopadhyaya]?
> Admin UI does not accurately reflect node PerReplicaStates status
> -
>
> Key: SOLR-15135
> URL: https://issues.apache.org/jira/browse/SOLR-15135
> Project: Solr
> Issue Type: Bug
> Components: Admin UI
>Affects Versions: 8.8
>Reporter: Mike Drob
>Priority: Blocker
>
> When creating collections using the new PRS, the admin UI will show them as
> down because it only looks at state.json and not the correct PRS node.
> cc: [~noble]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-15138) PerReplicaStates does not scale to large collections as well as state.json
[ https://issues.apache.org/jira/browse/SOLR-15138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17284929#comment-17284929 ] Timothy Potter commented on SOLR-15138: --- changing the fix version to 8.9 as it was reverted from branch_8_8 with this commit: https://github.com/apache/lucene-solr/commit/5f065acfbdb10e35633859520bb59122f4809f0b (hence not in 8.8.1) > PerReplicaStates does not scale to large collections as well as state.json > -- > > Key: SOLR-15138 > URL: https://issues.apache.org/jira/browse/SOLR-15138 > Project: Solr > Issue Type: Bug > Components: SolrCloud >Affects Versions: 8.8 >Reporter: Mike Drob >Assignee: Noble Paul >Priority: Major > Fix For: 8.9 > > Time Spent: 3h 20m > Remaining Estimate: 0h > > I was testing PRS collection creation with larger collections today > (previously I had tested with many small collections) and it seemed to be > having trouble keeping up. > > I was running a 4 node instance, each JVM with 4G Heap in k8s, and a single > zookeeper. > > With this cluster configuration, I am able to create several (at least 10) > collections with 11 shards and 11 replicas using the "old way" of keeping > state. These collections are created serially, waiting for all replicas to be > active before proceeding. > However, when attempting to do the same with PRS, the creation stalls on > collection 2 or 3, with several replicas stuck in a "down" state. Further, > when attempting to delete these collections using the regular API it > sometimes takes several attempts after getting stuck a few times as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-15138) PerReplicaStates does not scale to large collections as well as state.json
[ https://issues.apache.org/jira/browse/SOLR-15138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timothy Potter updated SOLR-15138: -- Fix Version/s: (was: 8.8.1) 8.9 > PerReplicaStates does not scale to large collections as well as state.json > -- > > Key: SOLR-15138 > URL: https://issues.apache.org/jira/browse/SOLR-15138 > Project: Solr > Issue Type: Bug > Components: SolrCloud >Affects Versions: 8.8 >Reporter: Mike Drob >Assignee: Noble Paul >Priority: Major > Fix For: 8.9 > > Time Spent: 3h 20m > Remaining Estimate: 0h > > I was testing PRS collection creation with larger collections today > (previously I had tested with many small collections) and it seemed to be > having trouble keeping up. > > I was running a 4 node instance, each JVM with 4G Heap in k8s, and a single > zookeeper. > > With this cluster configuration, I am able to create several (at least 10) > collections with 11 shards and 11 replicas using the "old way" of keeping > state. These collections are created serially, waiting for all replicas to be > active before proceeding. > However, when attempting to do the same with PRS, the creation stalls on > collection 2 or 3, with several replicas stuck in a "down" state. Further, > when attempting to delete these collections using the regular API it > sometimes takes several attempts after getting stuck a few times as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-15145) Older versions of SolrJ (pre-8.8.0) hit an NPE when computing the base_url for core node props
[
https://issues.apache.org/jira/browse/SOLR-15145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timothy Potter updated SOLR-15145:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Older versions of SolrJ (pre-8.8.0) hit an NPE when computing the base_url
> for core node props
> --
>
> Key: SOLR-15145
> URL: https://issues.apache.org/jira/browse/SOLR-15145
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrJ
>Affects Versions: 8.8
>Reporter: Timothy Potter
>Assignee: Timothy Potter
>Priority: Critical
> Fix For: 8.8.1
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> From the mailing list:
> {code}
> Caused by: java.lang.NullPointerException
> at
> deployment.uleaf.ear//org.apache.solr.common.cloud.ZkCoreNodeProps.getCoreUrl(ZkCoreNodeProps.java:53)
> at
> deployment.uleaf.ear//org.apache.solr.client.solrj.impl.BaseCloudSolrClient.lambda$sendRequest$2(BaseCloudSolrClient.java:1161)
> at java.base/java.util.ArrayList.forEach(ArrayList.java:1540)
> at
> deployment.uleaf.ear//org.apache.solr.client.solrj.impl.BaseCloudSolrClient.sendRequest(BaseCloudSolrClient.java:1159)
> at
> deployment.uleaf.ear//org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:934)
> ... 166 more
> {code}
> see:
> https://lists.apache.org/thread.html/r3d131030f0a7026235451f71fabdae6d6b7c2f955822c75dcad4d41f%40%3Csolr-user.lucene.apache.org%3E
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-15148) Include a backcompat utility class with a main in solrj that we can use to test older SolrJ against a release candidate
Timothy Potter created SOLR-15148: - Summary: Include a backcompat utility class with a main in solrj that we can use to test older SolrJ against a release candidate Key: SOLR-15148 URL: https://issues.apache.org/jira/browse/SOLR-15148 Project: Solr Issue Type: New Feature Security Level: Public (Default Security Level. Issues are Public) Components: SolrJ Reporter: Timothy Potter Changes to SolrJ in 8.8.0 (SOLR-12182) broke backcompat (fix is SOLR-15145) should have been caught during RC smoke testing. A simple utility class that we can run during RC smoke testing to catch back-compat breaks like this would be useful. To keep things simple, the smoke tester can download the previous version of SolrJ from Maven central and invoke this Backcompat app (embedded in the SolrJ JAR) against the new Solr server in the RC. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-15145) Older versions of SolrJ (pre-8.8.0) hit an NPE when computing the base_url for core node props
[
https://issues.apache.org/jira/browse/SOLR-15145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timothy Potter updated SOLR-15145:
--
Status: Patch Available (was: Open)
> Older versions of SolrJ (pre-8.8.0) hit an NPE when computing the base_url
> for core node props
> --
>
> Key: SOLR-15145
> URL: https://issues.apache.org/jira/browse/SOLR-15145
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrJ
>Affects Versions: 8.8
>Reporter: Timothy Potter
>Assignee: Timothy Potter
>Priority: Critical
> Fix For: 8.8.1
>
>
> From the mailing list:
> {code}
> Caused by: java.lang.NullPointerException
> at
> deployment.uleaf.ear//org.apache.solr.common.cloud.ZkCoreNodeProps.getCoreUrl(ZkCoreNodeProps.java:53)
> at
> deployment.uleaf.ear//org.apache.solr.client.solrj.impl.BaseCloudSolrClient.lambda$sendRequest$2(BaseCloudSolrClient.java:1161)
> at java.base/java.util.ArrayList.forEach(ArrayList.java:1540)
> at
> deployment.uleaf.ear//org.apache.solr.client.solrj.impl.BaseCloudSolrClient.sendRequest(BaseCloudSolrClient.java:1159)
> at
> deployment.uleaf.ear//org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:934)
> ... 166 more
> {code}
> see:
> https://lists.apache.org/thread.html/r3d131030f0a7026235451f71fabdae6d6b7c2f955822c75dcad4d41f%40%3Csolr-user.lucene.apache.org%3E
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-15145) Older versions of SolrJ (pre-8.8.0) hit an NPE when computing the base_url for core node props
[
https://issues.apache.org/jira/browse/SOLR-15145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17281362#comment-17281362
]
Timothy Potter commented on SOLR-15145:
---
After some digging around in the code again, there's really no way we can
remove `base_url` from the stored state in ZK for 8.8 w/o breaking back-compat
for SolrJ if clients are using {{ZkClientClusterStateProvider}} to load cluster
state directly from ZK. I should have caught this when implementing SOLR-12182
:( Obviously we can't make any changes to existing pre 8.8.0 SolrJ clients ...
Perhaps we could introduce a System property in 8.8.1 that re-enables storing
the `base_url` in ZK for these back-compat use cases? That will at least give
people an out when they can't upgrade their SolrJ version. Given the rolling
restart problem, I'm inclined to make this System property default to storing
the `base_url` instead of having to explicitly turn this behavior on, but want
to think about this a bit...
> Older versions of SolrJ (pre-8.8.0) hit an NPE when computing the base_url
> for core node props
> --
>
> Key: SOLR-15145
> URL: https://issues.apache.org/jira/browse/SOLR-15145
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrJ
>Affects Versions: 8.8
>Reporter: Timothy Potter
>Assignee: Timothy Potter
>Priority: Critical
> Fix For: 8.8.1
>
>
> From the mailing list:
> {code}
> Caused by: java.lang.NullPointerException
> at
> deployment.uleaf.ear//org.apache.solr.common.cloud.ZkCoreNodeProps.getCoreUrl(ZkCoreNodeProps.java:53)
> at
> deployment.uleaf.ear//org.apache.solr.client.solrj.impl.BaseCloudSolrClient.lambda$sendRequest$2(BaseCloudSolrClient.java:1161)
> at java.base/java.util.ArrayList.forEach(ArrayList.java:1540)
> at
> deployment.uleaf.ear//org.apache.solr.client.solrj.impl.BaseCloudSolrClient.sendRequest(BaseCloudSolrClient.java:1159)
> at
> deployment.uleaf.ear//org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:934)
> ... 166 more
> {code}
> see:
> https://lists.apache.org/thread.html/r3d131030f0a7026235451f71fabdae6d6b7c2f955822c75dcad4d41f%40%3Csolr-user.lucene.apache.org%3E
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-15145) Older versions of SolrJ (pre-8.8.0) hit an NPE when computing the base_url for core node props
[
https://issues.apache.org/jira/browse/SOLR-15145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timothy Potter updated SOLR-15145:
--
Priority: Critical (was: Major)
> Older versions of SolrJ (pre-8.8.0) hit an NPE when computing the base_url
> for core node props
> --
>
> Key: SOLR-15145
> URL: https://issues.apache.org/jira/browse/SOLR-15145
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrJ
>Affects Versions: 8.8
>Reporter: Timothy Potter
>Assignee: Timothy Potter
>Priority: Critical
> Fix For: 8.8.1
>
>
> From the mailing list:
> {code}
> Caused by: java.lang.NullPointerException
> at
> deployment.uleaf.ear//org.apache.solr.common.cloud.ZkCoreNodeProps.getCoreUrl(ZkCoreNodeProps.java:53)
> at
> deployment.uleaf.ear//org.apache.solr.client.solrj.impl.BaseCloudSolrClient.lambda$sendRequest$2(BaseCloudSolrClient.java:1161)
> at java.base/java.util.ArrayList.forEach(ArrayList.java:1540)
> at
> deployment.uleaf.ear//org.apache.solr.client.solrj.impl.BaseCloudSolrClient.sendRequest(BaseCloudSolrClient.java:1159)
> at
> deployment.uleaf.ear//org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:934)
> ... 166 more
> {code}
> see:
> https://lists.apache.org/thread.html/r3d131030f0a7026235451f71fabdae6d6b7c2f955822c75dcad4d41f%40%3Csolr-user.lucene.apache.org%3E
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-15145) Older versions of SolrJ (pre-8.8.0) hit an NPE when computing the base_url for core node props
[
https://issues.apache.org/jira/browse/SOLR-15145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17281317#comment-17281317
]
Timothy Potter edited comment on SOLR-15145 at 2/8/21, 7:48 PM:
Actually now that I think about this some more, this may also impact rolling
upgrades from pre 8.8 to an 8.8 cluster as upgrade nodes re-joining the cluster
will not persist the {{base_url}} in stored cluster state, which could cause
issues with leader election / recoveries as there will be a mix of node
versions in the cluster temporarily.
Tried this out locally and older nodes see errors like this while new nodes are
coming back up:
{code}
2021-02-08 19:42:20.107 ERROR
(OverseerStateUpdate-72081179721793543-dev-solrcloud-0.dev:80_solr-n_00)
[ ] o.a.s.c.Overseer Overseer could not process the current clusterstate
state update message, skipping the message: {
"operation":"leader",
"shard":"shard1",
"collection":"sop1",
"node_name":"dev-solrcloud-2.dev:80_solr",
"core":"sop1_shard1_replica_n1",
"state":"active"} => java.lang.NullPointerException
at
org.apache.solr.common.cloud.ZkCoreNodeProps.getCoreUrl(ZkCoreNodeProps.java:53)
java.lang.NullPointerException: null
at
org.apache.solr.common.cloud.ZkCoreNodeProps.getCoreUrl(ZkCoreNodeProps.java:53)
~[?:?]
at
org.apache.solr.cloud.overseer.SliceMutator.setShardLeader(SliceMutator.java:140)
~[?:?]
at
org.apache.solr.cloud.Overseer$ClusterStateUpdater.processMessage(Overseer.java:420)
~[?:?]
at
org.apache.solr.cloud.Overseer$ClusterStateUpdater.processQueueItem(Overseer.java:309)
~[?:?]
at
org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:253) ~[?:?]
at java.lang.Thread.run(Unknown Source) [?:?]
{code}
Unfortunately, my test collection did not come back cleanly either, now it
seems to have some core name mismatch?
{code}
2021-02-08 19:46:46.651 WARN
(coreZkRegister-3-thread-2-processing-n:dev-solrcloud-1.dev:80_solr) [c:sop1
s:shard1 r:core_node5 x:sop1_shard1_replica_n2] o.a.s.c.ZkController Still
seeing conflicting information about the leader of shard shard1 for collection
sop1 after 30 seconds; our state says
http://dev-solrcloud-1.dev:80/solr/sop1_shard1_replica_n2/, but ZooKeeper says
http://dev-solrcloud-2.dev:80/solr/sop1_shard1_replica_n1/
{code}
was (Author: thelabdude):
Actually now that I think about this some more, this may also impact rolling
upgrades from pre 8.8 to an 8.8 cluster as upgrade nodes re-joining the cluster
will not persist the {{base_url}} in stored cluster state, which could cause
issues with leader election / recoveries as there will be a mix of node
versions in the cluster temporarily.
Tried this out locally and older nodes see errors like this while new nodes are
coming back up:
{code}
2021-02-08 19:42:20.107 ERROR
(OverseerStateUpdate-72081179721793543-dev-solrcloud-0.dev:80_solr-n_00)
[ ] o.a.s.c.Overseer Overseer could not process the current clusterstate
state update message, skipping the message: {
"operation":"leader",
"shard":"shard1",
"collection":"sop1",
"node_name":"dev-solrcloud-2.dev:80_solr",
"core":"sop1_shard1_replica_n1",
"state":"active"} => java.lang.NullPointerException
at
org.apache.solr.common.cloud.ZkCoreNodeProps.getCoreUrl(ZkCoreNodeProps.java:53)
java.lang.NullPointerException: null
at
org.apache.solr.common.cloud.ZkCoreNodeProps.getCoreUrl(ZkCoreNodeProps.java:53)
~[?:?]
at
org.apache.solr.cloud.overseer.SliceMutator.setShardLeader(SliceMutator.java:140)
~[?:?]
at
org.apache.solr.cloud.Overseer$ClusterStateUpdater.processMessage(Overseer.java:420)
~[?:?]
at
org.apache.solr.cloud.Overseer$ClusterStateUpdater.processQueueItem(Overseer.java:309)
~[?:?]
at
org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:253) ~[?:?]
at java.lang.Thread.run(Unknown Source) [?:?]
{code}
> Older versions of SolrJ (pre-8.8.0) hit an NPE when computing the base_url
> for core node props
> --
>
> Key: SOLR-15145
> URL: https://issues.apache.org/jira/browse/SOLR-15145
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrJ
>Affects Versions: 8.8
>Reporter: Timothy Potter
>Assignee: Timothy Potter
>Priority: Major
> Fix For: 8.8.1
>
>
> From the mailing list:
> {code}
> Caused by: java.lang.NullPointerException
> at
> deployment.uleaf.ear//org.apache.solr.common.cloud.ZkCoreNodeProps.getCoreUrl(ZkCoreNodeProps.java:53)
> at
>
[jira] [Comment Edited] (SOLR-15145) Older versions of SolrJ (pre-8.8.0) hit an NPE when computing the base_url for core node props
[
https://issues.apache.org/jira/browse/SOLR-15145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17281317#comment-17281317
]
Timothy Potter edited comment on SOLR-15145 at 2/8/21, 7:43 PM:
Actually now that I think about this some more, this may also impact rolling
upgrades from pre 8.8 to an 8.8 cluster as upgrade nodes re-joining the cluster
will not persist the {{base_url}} in stored cluster state, which could cause
issues with leader election / recoveries as there will be a mix of node
versions in the cluster temporarily.
Tried this out locally and older nodes see errors like this while new nodes are
coming back up:
{code}
2021-02-08 19:42:20.107 ERROR
(OverseerStateUpdate-72081179721793543-dev-solrcloud-0.dev:80_solr-n_00)
[ ] o.a.s.c.Overseer Overseer could not process the current clusterstate
state update message, skipping the message: {
"operation":"leader",
"shard":"shard1",
"collection":"sop1",
"node_name":"dev-solrcloud-2.dev:80_solr",
"core":"sop1_shard1_replica_n1",
"state":"active"} => java.lang.NullPointerException
at
org.apache.solr.common.cloud.ZkCoreNodeProps.getCoreUrl(ZkCoreNodeProps.java:53)
java.lang.NullPointerException: null
at
org.apache.solr.common.cloud.ZkCoreNodeProps.getCoreUrl(ZkCoreNodeProps.java:53)
~[?:?]
at
org.apache.solr.cloud.overseer.SliceMutator.setShardLeader(SliceMutator.java:140)
~[?:?]
at
org.apache.solr.cloud.Overseer$ClusterStateUpdater.processMessage(Overseer.java:420)
~[?:?]
at
org.apache.solr.cloud.Overseer$ClusterStateUpdater.processQueueItem(Overseer.java:309)
~[?:?]
at
org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:253) ~[?:?]
at java.lang.Thread.run(Unknown Source) [?:?]
{code}
was (Author: thelabdude):
Actually now that I think about this some more, this may also impact rolling
upgrades from pre 8.8 to an 8.8 cluster as upgrade nodes re-joining the cluster
will not persist the {{base_url}} in stored cluster state, which could cause
issues with leader election / recoveries as there will be a mix of node
versions in the cluster temporarily.
> Older versions of SolrJ (pre-8.8.0) hit an NPE when computing the base_url
> for core node props
> --
>
> Key: SOLR-15145
> URL: https://issues.apache.org/jira/browse/SOLR-15145
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrJ
>Affects Versions: 8.8
>Reporter: Timothy Potter
>Assignee: Timothy Potter
>Priority: Major
> Fix For: 8.8.1
>
>
> From the mailing list:
> {code}
> Caused by: java.lang.NullPointerException
> at
> deployment.uleaf.ear//org.apache.solr.common.cloud.ZkCoreNodeProps.getCoreUrl(ZkCoreNodeProps.java:53)
> at
> deployment.uleaf.ear//org.apache.solr.client.solrj.impl.BaseCloudSolrClient.lambda$sendRequest$2(BaseCloudSolrClient.java:1161)
> at java.base/java.util.ArrayList.forEach(ArrayList.java:1540)
> at
> deployment.uleaf.ear//org.apache.solr.client.solrj.impl.BaseCloudSolrClient.sendRequest(BaseCloudSolrClient.java:1159)
> at
> deployment.uleaf.ear//org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:934)
> ... 166 more
> {code}
> see:
> https://lists.apache.org/thread.html/r3d131030f0a7026235451f71fabdae6d6b7c2f955822c75dcad4d41f%40%3Csolr-user.lucene.apache.org%3E
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-15145) Older versions of SolrJ (pre-8.8.0) hit an NPE when computing the base_url for core node props
[
https://issues.apache.org/jira/browse/SOLR-15145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17281317#comment-17281317
]
Timothy Potter commented on SOLR-15145:
---
Actually now that I think about this some more, this may also impact rolling
upgrades from pre 8.8 to an 8.8 cluster as upgrade nodes re-joining the cluster
will not persist the {{base_url}} in stored cluster state, which could cause
issues with leader election / recoveries as there will be a mix of node
versions in the cluster temporarily.
> Older versions of SolrJ (pre-8.8.0) hit an NPE when computing the base_url
> for core node props
> --
>
> Key: SOLR-15145
> URL: https://issues.apache.org/jira/browse/SOLR-15145
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrJ
>Affects Versions: 8.8
>Reporter: Timothy Potter
>Assignee: Timothy Potter
>Priority: Major
> Fix For: 8.8.1
>
>
> From the mailing list:
> {code}
> Caused by: java.lang.NullPointerException
> at
> deployment.uleaf.ear//org.apache.solr.common.cloud.ZkCoreNodeProps.getCoreUrl(ZkCoreNodeProps.java:53)
> at
> deployment.uleaf.ear//org.apache.solr.client.solrj.impl.BaseCloudSolrClient.lambda$sendRequest$2(BaseCloudSolrClient.java:1161)
> at java.base/java.util.ArrayList.forEach(ArrayList.java:1540)
> at
> deployment.uleaf.ear//org.apache.solr.client.solrj.impl.BaseCloudSolrClient.sendRequest(BaseCloudSolrClient.java:1159)
> at
> deployment.uleaf.ear//org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:934)
> ... 166 more
> {code}
> see:
> https://lists.apache.org/thread.html/r3d131030f0a7026235451f71fabdae6d6b7c2f955822c75dcad4d41f%40%3Csolr-user.lucene.apache.org%3E
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-15145) Older versions of SolrJ (pre-8.8.0) hit an NPE when computing the base_url for core node props
[
https://issues.apache.org/jira/browse/SOLR-15145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17281145#comment-17281145
]
Timothy Potter commented on SOLR-15145:
---
The work-around would be to upgrade client applications to use SolrJ 8.8.0 when
hitting a Solr 8.8.0 server, but that's not always practical / achievable in
production deployments. Thus we'll need to get a fix into 8.8.1. In the
meantime, users that cannot upgrade their client applications to SolrJ 8.8,
should hold off on upgrading the server.
> Older versions of SolrJ (pre-8.8.0) hit an NPE when computing the base_url
> for core node props
> --
>
> Key: SOLR-15145
> URL: https://issues.apache.org/jira/browse/SOLR-15145
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrJ
>Affects Versions: 8.8
>Reporter: Timothy Potter
>Assignee: Timothy Potter
>Priority: Major
>
> From the mailing list:
> {code}
> Caused by: java.lang.NullPointerException
> at
> deployment.uleaf.ear//org.apache.solr.common.cloud.ZkCoreNodeProps.getCoreUrl(ZkCoreNodeProps.java:53)
> at
> deployment.uleaf.ear//org.apache.solr.client.solrj.impl.BaseCloudSolrClient.lambda$sendRequest$2(BaseCloudSolrClient.java:1161)
> at java.base/java.util.ArrayList.forEach(ArrayList.java:1540)
> at
> deployment.uleaf.ear//org.apache.solr.client.solrj.impl.BaseCloudSolrClient.sendRequest(BaseCloudSolrClient.java:1159)
> at
> deployment.uleaf.ear//org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:934)
> ... 166 more
> {code}
> see:
> https://lists.apache.org/thread.html/r3d131030f0a7026235451f71fabdae6d6b7c2f955822c75dcad4d41f%40%3Csolr-user.lucene.apache.org%3E
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-15145) Older versions of SolrJ (pre-8.8.0) hit an NPE when computing the base_url for core node props
[
https://issues.apache.org/jira/browse/SOLR-15145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timothy Potter updated SOLR-15145:
--
Fix Version/s: 8.8.1
> Older versions of SolrJ (pre-8.8.0) hit an NPE when computing the base_url
> for core node props
> --
>
> Key: SOLR-15145
> URL: https://issues.apache.org/jira/browse/SOLR-15145
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrJ
>Affects Versions: 8.8
>Reporter: Timothy Potter
>Assignee: Timothy Potter
>Priority: Major
> Fix For: 8.8.1
>
>
> From the mailing list:
> {code}
> Caused by: java.lang.NullPointerException
> at
> deployment.uleaf.ear//org.apache.solr.common.cloud.ZkCoreNodeProps.getCoreUrl(ZkCoreNodeProps.java:53)
> at
> deployment.uleaf.ear//org.apache.solr.client.solrj.impl.BaseCloudSolrClient.lambda$sendRequest$2(BaseCloudSolrClient.java:1161)
> at java.base/java.util.ArrayList.forEach(ArrayList.java:1540)
> at
> deployment.uleaf.ear//org.apache.solr.client.solrj.impl.BaseCloudSolrClient.sendRequest(BaseCloudSolrClient.java:1159)
> at
> deployment.uleaf.ear//org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:934)
> ... 166 more
> {code}
> see:
> https://lists.apache.org/thread.html/r3d131030f0a7026235451f71fabdae6d6b7c2f955822c75dcad4d41f%40%3Csolr-user.lucene.apache.org%3E
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-15145) Older versions of SolrJ (pre-8.8.0) hit an NPE when computing the base_url for core node props
Timothy Potter created SOLR-15145:
-
Summary: Older versions of SolrJ (pre-8.8.0) hit an NPE when
computing the base_url for core node props
Key: SOLR-15145
URL: https://issues.apache.org/jira/browse/SOLR-15145
Project: Solr
Issue Type: Bug
Security Level: Public (Default Security Level. Issues are Public)
Components: SolrJ
Affects Versions: 8.8
Reporter: Timothy Potter
Assignee: Timothy Potter
>From the mailing list:
{code}
Caused by: java.lang.NullPointerException
at
deployment.uleaf.ear//org.apache.solr.common.cloud.ZkCoreNodeProps.getCoreUrl(ZkCoreNodeProps.java:53)
at
deployment.uleaf.ear//org.apache.solr.client.solrj.impl.BaseCloudSolrClient.lambda$sendRequest$2(BaseCloudSolrClient.java:1161)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1540)
at
deployment.uleaf.ear//org.apache.solr.client.solrj.impl.BaseCloudSolrClient.sendRequest(BaseCloudSolrClient.java:1159)
at
deployment.uleaf.ear//org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:934)
... 166 more
{code}
see:
https://lists.apache.org/thread.html/r3d131030f0a7026235451f71fabdae6d6b7c2f955822c75dcad4d41f%40%3Csolr-user.lucene.apache.org%3E
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (SOLR-15128) nodeName does not contain expected ':' separator: localhost
[
https://issues.apache.org/jira/browse/SOLR-15128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timothy Potter resolved SOLR-15128.
---
Resolution: Won't Fix
Ignore this ;-) Was using the wrong method to get the nodeName in some code
that only exists on master. {{zkcontroller.getNodeName}} is the correct way to
get the nodeName for converting to a URL.
> nodeName does not contain expected ':' separator: localhost
> ---
>
> Key: SOLR-15128
> URL: https://issues.apache.org/jira/browse/SOLR-15128
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Environment: Only seems to affect master, 8.8 is not affected.
>Reporter: Timothy Potter
>Assignee: Timothy Potter
>Priority: Major
> Fix For: master (9.0)
>
>
> {code}
> "error":{"msg":"nodeName does not contain expected ':' separator:
> localhost","trace":"java.lang.IllegalArgumentException: nodeName does not
> contain expected ':' separator: localhost\n\tat
> org.apache.solr.common.util.Utils.getBaseUrlForNodeName(Utils.java:764)\n\tat
> org.apache.solr.common.util.Utils.getBaseUrlForNodeName(Utils.java:759)\n\tat
> org.apache.solr.common.cloud.UrlScheme.getBaseUrlForNodeName(UrlScheme.java:54)\n\ta{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-15128) nodeName does not contain expected ':' separator: localhost
[
https://issues.apache.org/jira/browse/SOLR-15128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17277376#comment-17277376
]
Timothy Potter commented on SOLR-15128:
---
Not sure why master is seeing {{localhost}} as the nodeName w/o port on it, but
this is breaking the parsing in {{Utils.getBaseUrlFromNodeName}}
> nodeName does not contain expected ':' separator: localhost
> ---
>
> Key: SOLR-15128
> URL: https://issues.apache.org/jira/browse/SOLR-15128
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Environment: Only seems to affect master, 8.8 is not affected.
>Reporter: Timothy Potter
>Assignee: Timothy Potter
>Priority: Major
> Fix For: master (9.0)
>
>
> {code}
> "error":{"msg":"nodeName does not contain expected ':' separator:
> localhost","trace":"java.lang.IllegalArgumentException: nodeName does not
> contain expected ':' separator: localhost\n\tat
> org.apache.solr.common.util.Utils.getBaseUrlForNodeName(Utils.java:764)\n\tat
> org.apache.solr.common.util.Utils.getBaseUrlForNodeName(Utils.java:759)\n\tat
> org.apache.solr.common.cloud.UrlScheme.getBaseUrlForNodeName(UrlScheme.java:54)\n\ta{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-15128) nodeName does not contain expected ':' separator: localhost
Timothy Potter created SOLR-15128:
-
Summary: nodeName does not contain expected ':' separator:
localhost
Key: SOLR-15128
URL: https://issues.apache.org/jira/browse/SOLR-15128
Project: Solr
Issue Type: Bug
Security Level: Public (Default Security Level. Issues are Public)
Environment: Only seems to affect master, 8.8 is not affected.
Reporter: Timothy Potter
Assignee: Timothy Potter
{code}
"error":{"msg":"nodeName does not contain expected ':' separator:
localhost","trace":"java.lang.IllegalArgumentException: nodeName does not
contain expected ':' separator: localhost\n\tat
org.apache.solr.common.util.Utils.getBaseUrlForNodeName(Utils.java:764)\n\tat
org.apache.solr.common.util.Utils.getBaseUrlForNodeName(Utils.java:759)\n\tat
org.apache.solr.common.cloud.UrlScheme.getBaseUrlForNodeName(UrlScheme.java:54)\n\ta{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-15128) nodeName does not contain expected ':' separator: localhost
[
https://issues.apache.org/jira/browse/SOLR-15128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timothy Potter updated SOLR-15128:
--
Fix Version/s: master (9.0)
> nodeName does not contain expected ':' separator: localhost
> ---
>
> Key: SOLR-15128
> URL: https://issues.apache.org/jira/browse/SOLR-15128
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Environment: Only seems to affect master, 8.8 is not affected.
>Reporter: Timothy Potter
>Assignee: Timothy Potter
>Priority: Major
> Fix For: master (9.0)
>
>
> {code}
> "error":{"msg":"nodeName does not contain expected ':' separator:
> localhost","trace":"java.lang.IllegalArgumentException: nodeName does not
> contain expected ':' separator: localhost\n\tat
> org.apache.solr.common.util.Utils.getBaseUrlForNodeName(Utils.java:764)\n\tat
> org.apache.solr.common.util.Utils.getBaseUrlForNodeName(Utils.java:759)\n\tat
> org.apache.solr.common.cloud.UrlScheme.getBaseUrlForNodeName(UrlScheme.java:54)\n\ta{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-15098) Refactor cloud panel to not need base_url
[
https://issues.apache.org/jira/browse/SOLR-15098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timothy Potter updated SOLR-15098:
--
Description:
Work done in SOLR-12182 removed {{base_url}} but the JS for the Admin UI cloud
panel still depends on that property. We should refactor the JS code to not
require this property. This introduced some unfortunate regression in 8.8-rc1:
https://issues.apache.org/jira/browse/SOLR-15097
Also, there is no unit test for the {{ZookeeperInfoHandler}} used by the Cloud
panel, requiring manual UI testing. The fix for this issue should include a
unit test for this critical component in Solr!
was:
Work done in SOLR-12182 removed base_url but the JS for the Admin UI cloud
panel still depends on that property. We should refactor the JS code to not
require this property. This introduced some unfortunate regression in 8.8-rc1:
https://issues.apache.org/jira/browse/SOLR-15097
Also, there is no unit test for the {{ZookeeperInfoHandler}} used by the Cloud
panel, requiring manual UI testing. The fix for this issue should include a
unit test for this critical component in Solr!
> Refactor cloud panel to not need base_url
> -
>
> Key: SOLR-15098
> URL: https://issues.apache.org/jira/browse/SOLR-15098
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: master (9.0)
>Reporter: Timothy Potter
>Assignee: Timothy Potter
>Priority: Major
>
> Work done in SOLR-12182 removed {{base_url}} but the JS for the Admin UI
> cloud panel still depends on that property. We should refactor the JS code to
> not require this property. This introduced some unfortunate regression in
> 8.8-rc1: https://issues.apache.org/jira/browse/SOLR-15097
> Also, there is no unit test for the {{ZookeeperInfoHandler}} used by the
> Cloud panel, requiring manual UI testing. The fix for this issue should
> include a unit test for this critical component in Solr!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-15098) Refactor cloud panel to not need base_url
Timothy Potter created SOLR-15098:
-
Summary: Refactor cloud panel to not need base_url
Key: SOLR-15098
URL: https://issues.apache.org/jira/browse/SOLR-15098
Project: Solr
Issue Type: Improvement
Security Level: Public (Default Security Level. Issues are Public)
Affects Versions: master (9.0)
Reporter: Timothy Potter
Assignee: Timothy Potter
Work done in SOLR-12182 removed base_url but the JS for the Admin UI cloud
panel still depends on that property. We should refactor the JS code to not
require this property. This introduced some unfortunate regression in 8.8-rc1:
https://issues.apache.org/jira/browse/SOLR-15097
Also, there is no unit test for the {{ZookeeperInfoHandler}} used by the Cloud
panel, requiring manual UI testing. The fix for this issue should include a
unit test for this critical component in Solr!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-15097) Cluster graph in admin UI is broken
[ https://issues.apache.org/jira/browse/SOLR-15097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17269696#comment-17269696 ] Timothy Potter commented on SOLR-15097: --- Doesn't affect master, only 8x, must have been a mistake during the backport to 8x > Cluster graph in admin UI is broken > --- > > Key: SOLR-15097 > URL: https://issues.apache.org/jira/browse/SOLR-15097 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Anshum Gupta >Assignee: Timothy Potter >Priority: Blocker > Fix For: 8.8 > > Time Spent: 20m > Remaining Estimate: 0h > > Cluster graph is broken in the Admin UI and doesn't reflect any collections. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-15097) Cluster graph in admin UI is broken
[ https://issues.apache.org/jira/browse/SOLR-15097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timothy Potter updated SOLR-15097: -- Fix Version/s: 8.8 > Cluster graph in admin UI is broken > --- > > Key: SOLR-15097 > URL: https://issues.apache.org/jira/browse/SOLR-15097 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Affects Versions: 8.8 >Reporter: Anshum Gupta >Assignee: Timothy Potter >Priority: Critical > Fix For: 8.8 > > > Cluster graph is broken in the Admin UI and doesn't reflect any collections. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-15097) Cluster graph in admin UI is broken
[ https://issues.apache.org/jira/browse/SOLR-15097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timothy Potter updated SOLR-15097: -- Priority: Blocker (was: Critical) > Cluster graph in admin UI is broken > --- > > Key: SOLR-15097 > URL: https://issues.apache.org/jira/browse/SOLR-15097 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Affects Versions: 8.8 >Reporter: Anshum Gupta >Assignee: Timothy Potter >Priority: Blocker > Fix For: 8.8 > > > Cluster graph is broken in the Admin UI and doesn't reflect any collections. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-15097) Cluster graph in admin UI is broken
[ https://issues.apache.org/jira/browse/SOLR-15097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timothy Potter updated SOLR-15097: -- Affects Version/s: 8.8 > Cluster graph in admin UI is broken > --- > > Key: SOLR-15097 > URL: https://issues.apache.org/jira/browse/SOLR-15097 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Affects Versions: 8.8 >Reporter: Anshum Gupta >Assignee: Timothy Potter >Priority: Critical > > Cluster graph is broken in the Admin UI and doesn't reflect any collections. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-15097) Cluster graph in admin UI is broken
[ https://issues.apache.org/jira/browse/SOLR-15097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17269680#comment-17269680 ] Timothy Potter commented on SOLR-15097: --- Regression caused by fix for SOLR-12182 > Cluster graph in admin UI is broken > --- > > Key: SOLR-15097 > URL: https://issues.apache.org/jira/browse/SOLR-15097 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Anshum Gupta >Assignee: Timothy Potter >Priority: Critical > > Cluster graph is broken in the Admin UI and doesn't reflect any collections. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Assigned] (SOLR-15097) Cluster graph in admin UI is broken
[ https://issues.apache.org/jira/browse/SOLR-15097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timothy Potter reassigned SOLR-15097: - Assignee: Timothy Potter > Cluster graph in admin UI is broken > --- > > Key: SOLR-15097 > URL: https://issues.apache.org/jira/browse/SOLR-15097 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Anshum Gupta >Assignee: Timothy Potter >Priority: Critical > > Cluster graph is broken in the Admin UI and doesn't reflect any collections. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-15082) payload function (FloatPayloadValueSource) should work with the payloads created by TokenOffsetPayloadTokenFilterFactory
Timothy Potter created SOLR-15082:
-
Summary: payload function (FloatPayloadValueSource) should work
with the payloads created by TokenOffsetPayloadTokenFilterFactory
Key: SOLR-15082
URL: https://issues.apache.org/jira/browse/SOLR-15082
Project: Solr
Issue Type: Improvement
Security Level: Public (Default Security Level. Issues are Public)
Reporter: Timothy Potter
Assignee: Timothy Potter
Index term offsets as payloads using {{TokenOffsetPayloadTokenFilterFactory}}
but wasn't able to leverage them in the {{payload}} function due to:
{code}
No payload decoder found for field: test_offsets_as_payload
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-15036) Use plist automatically for executing a facet expression against a collection alias backed by multiple collections
[
https://issues.apache.org/jira/browse/SOLR-15036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timothy Potter updated SOLR-15036:
--
Fix Version/s: master (9.0)
8.8
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Use plist automatically for executing a facet expression against a collection
> alias backed by multiple collections
> --
>
> Key: SOLR-15036
> URL: https://issues.apache.org/jira/browse/SOLR-15036
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: streaming expressions
>Reporter: Timothy Potter
>Assignee: Timothy Potter
>Priority: Major
> Fix For: 8.8, master (9.0)
>
> Attachments: relay-approach.patch
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> For analytics use cases, streaming expressions make it possible to compute
> basic aggregations (count, min, max, sum, and avg) over massive data sets.
> Moreover, with massive data sets, it is common to use collection aliases over
> many underlying collections, for instance time-partitioned aliases backed by
> a set of collections, each covering a specific time range. In some cases, we
> can end up with many collections (think 50-60) each with 100's of shards.
> Aliases help insulate client applications from complex collection topologies
> on the server side.
> Let's take a basic facet expression that computes some useful aggregation
> metrics:
> {code:java}
> facet(
> some_alias,
> q="*:*",
> fl="a_i",
> sort="a_i asc",
> buckets="a_i",
> bucketSorts="count(*) asc",
> bucketSizeLimit=1,
> sum(a_d), avg(a_d), min(a_d), max(a_d), count(*)
> )
> {code}
> Behind the scenes, the {{FacetStream}} sends a JSON facet request to Solr
> which then expands the alias to a list of collections. For each collection,
> the top-level distributed query controller gathers a candidate set of
> replicas to query and then scatters {{distrib=false}} queries to each replica
> in the list. For instance, if we have 60 collections with 200 shards each,
> then this results in 12,000 shard requests from the query controller node to
> the other nodes in the cluster. The requests are sent in an async manner (see
> {{SearchHandler}} and {{HttpShardHandler}}) In my testing, we’ve seen cases
> where we hit 18,000 replicas and these queries don’t always come back in a
> timely manner. Put simply, this also puts a lot of load on the top-level
> query controller node in terms of open connections and new object creation.
> Instead, we can use {{plist}} to send the JSON facet query to each collection
> in the alias in parallel, which reduces the overhead of each top-level
> distributed query from 12,000 to 200 in my example above. With this approach,
> you’ll then need to sort the tuples back from each collection and do a
> rollup, something like:
> {code:java}
> select(
> rollup(
> sort(
> plist(
> select(facet(coll1,q="*:*", fl="a_i", sort="a_i asc", buckets="a_i",
> bucketSorts="count(*) asc", bucketSizeLimit=1, sum(a_d), avg(a_d),
> min(a_d), max(a_d), count(*)),a_i,sum(a_d) as the_sum, avg(a_d) as the_avg,
> min(a_d) as the_min, max(a_d) as the_max, count(*) as cnt),
> select(facet(coll2,q="*:*", fl="a_i", sort="a_i asc", buckets="a_i",
> bucketSorts="count(*) asc", bucketSizeLimit=1, sum(a_d), avg(a_d),
> min(a_d), max(a_d), count(*)),a_i,sum(a_d) as the_sum, avg(a_d) as the_avg,
> min(a_d) as the_min, max(a_d) as the_max, count(*) as cnt)
> ),
> by="a_i asc"
> ),
> over="a_i",
> sum(the_sum), avg(the_avg), min(the_min), max(the_max), sum(cnt)
> ),
> a_i, sum(the_sum) as the_sum, avg(the_avg) as the_avg, min(the_min) as
> the_min, max(the_max) as the_max, sum(cnt) as cnt
> )
> {code}
> One thing to point out is that you can’t just avg. the averages back from
> each collection in the rollup. It needs to be a *weighted avg.* when rolling
> up the avg. from each facet expression in the plist. However, we have the
> count per collection, so this is doable but will require some changes to the
> rollup expression to support weighted average.
> While this plist approach is doable, it’s a pain for users to have to create
> the rollup / sort over plist expression for collection aliases. After all,
> aliases are supposed to hide these types of complexities from client
> applications!
> The point of this ticket is to investigate the feasibility of auto-wrapping
> the facet expression with a rollup / sort / plist when the collection
> argument is an alias with multiple collections; other stream sources will be
> considered after
[jira] [Commented] (SOLR-15036) Use plist automatically for executing a facet expression against a collection alias backed by multiple collections
[
https://issues.apache.org/jira/browse/SOLR-15036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17262786#comment-17262786
]
Timothy Potter commented on SOLR-15036:
---
Thanks. Updated the impl. to use {{HashRollupStream}} (doesn't seem to be
documented btw) ... definitely much cleaner ;-)
> Use plist automatically for executing a facet expression against a collection
> alias backed by multiple collections
> --
>
> Key: SOLR-15036
> URL: https://issues.apache.org/jira/browse/SOLR-15036
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: streaming expressions
>Reporter: Timothy Potter
>Assignee: Timothy Potter
>Priority: Major
> Attachments: relay-approach.patch
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> For analytics use cases, streaming expressions make it possible to compute
> basic aggregations (count, min, max, sum, and avg) over massive data sets.
> Moreover, with massive data sets, it is common to use collection aliases over
> many underlying collections, for instance time-partitioned aliases backed by
> a set of collections, each covering a specific time range. In some cases, we
> can end up with many collections (think 50-60) each with 100's of shards.
> Aliases help insulate client applications from complex collection topologies
> on the server side.
> Let's take a basic facet expression that computes some useful aggregation
> metrics:
> {code:java}
> facet(
> some_alias,
> q="*:*",
> fl="a_i",
> sort="a_i asc",
> buckets="a_i",
> bucketSorts="count(*) asc",
> bucketSizeLimit=1,
> sum(a_d), avg(a_d), min(a_d), max(a_d), count(*)
> )
> {code}
> Behind the scenes, the {{FacetStream}} sends a JSON facet request to Solr
> which then expands the alias to a list of collections. For each collection,
> the top-level distributed query controller gathers a candidate set of
> replicas to query and then scatters {{distrib=false}} queries to each replica
> in the list. For instance, if we have 60 collections with 200 shards each,
> then this results in 12,000 shard requests from the query controller node to
> the other nodes in the cluster. The requests are sent in an async manner (see
> {{SearchHandler}} and {{HttpShardHandler}}) In my testing, we’ve seen cases
> where we hit 18,000 replicas and these queries don’t always come back in a
> timely manner. Put simply, this also puts a lot of load on the top-level
> query controller node in terms of open connections and new object creation.
> Instead, we can use {{plist}} to send the JSON facet query to each collection
> in the alias in parallel, which reduces the overhead of each top-level
> distributed query from 12,000 to 200 in my example above. With this approach,
> you’ll then need to sort the tuples back from each collection and do a
> rollup, something like:
> {code:java}
> select(
> rollup(
> sort(
> plist(
> select(facet(coll1,q="*:*", fl="a_i", sort="a_i asc", buckets="a_i",
> bucketSorts="count(*) asc", bucketSizeLimit=1, sum(a_d), avg(a_d),
> min(a_d), max(a_d), count(*)),a_i,sum(a_d) as the_sum, avg(a_d) as the_avg,
> min(a_d) as the_min, max(a_d) as the_max, count(*) as cnt),
> select(facet(coll2,q="*:*", fl="a_i", sort="a_i asc", buckets="a_i",
> bucketSorts="count(*) asc", bucketSizeLimit=1, sum(a_d), avg(a_d),
> min(a_d), max(a_d), count(*)),a_i,sum(a_d) as the_sum, avg(a_d) as the_avg,
> min(a_d) as the_min, max(a_d) as the_max, count(*) as cnt)
> ),
> by="a_i asc"
> ),
> over="a_i",
> sum(the_sum), avg(the_avg), min(the_min), max(the_max), sum(cnt)
> ),
> a_i, sum(the_sum) as the_sum, avg(the_avg) as the_avg, min(the_min) as
> the_min, max(the_max) as the_max, sum(cnt) as cnt
> )
> {code}
> One thing to point out is that you can’t just avg. the averages back from
> each collection in the rollup. It needs to be a *weighted avg.* when rolling
> up the avg. from each facet expression in the plist. However, we have the
> count per collection, so this is doable but will require some changes to the
> rollup expression to support weighted average.
> While this plist approach is doable, it’s a pain for users to have to create
> the rollup / sort over plist expression for collection aliases. After all,
> aliases are supposed to hide these types of complexities from client
> applications!
> The point of this ticket is to investigate the feasibility of auto-wrapping
> the facet expression with a rollup / sort / plist when the collection
> argument is an alias with multiple collections; other stream sources will be
> considered after facet is
[jira] [Commented] (SOLR-15036) Use plist automatically for executing a facet expression against a collection alias backed by multiple collections
[
https://issues.apache.org/jira/browse/SOLR-15036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17261575#comment-17261575
]
Timothy Potter commented on SOLR-15036:
---
Ok thank you for the guidance. Tiered makes sense now (y). So I'll clean that
up in the PR and add the explicit opt-in in the backport for 8.x.
Btw ~ while in the {{FacetStream}} code, I noticed it doesn't support basic
auth like {{SolrStream}} does, i.e. there's no way to pass basic auth
credentials to the {{QueryRequest}} constructed in the open method that I can
see ... is this by design or just oversight?
> Use plist automatically for executing a facet expression against a collection
> alias backed by multiple collections
> --
>
> Key: SOLR-15036
> URL: https://issues.apache.org/jira/browse/SOLR-15036
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: streaming expressions
>Reporter: Timothy Potter
>Assignee: Timothy Potter
>Priority: Major
> Attachments: relay-approach.patch
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> For analytics use cases, streaming expressions make it possible to compute
> basic aggregations (count, min, max, sum, and avg) over massive data sets.
> Moreover, with massive data sets, it is common to use collection aliases over
> many underlying collections, for instance time-partitioned aliases backed by
> a set of collections, each covering a specific time range. In some cases, we
> can end up with many collections (think 50-60) each with 100's of shards.
> Aliases help insulate client applications from complex collection topologies
> on the server side.
> Let's take a basic facet expression that computes some useful aggregation
> metrics:
> {code:java}
> facet(
> some_alias,
> q="*:*",
> fl="a_i",
> sort="a_i asc",
> buckets="a_i",
> bucketSorts="count(*) asc",
> bucketSizeLimit=1,
> sum(a_d), avg(a_d), min(a_d), max(a_d), count(*)
> )
> {code}
> Behind the scenes, the {{FacetStream}} sends a JSON facet request to Solr
> which then expands the alias to a list of collections. For each collection,
> the top-level distributed query controller gathers a candidate set of
> replicas to query and then scatters {{distrib=false}} queries to each replica
> in the list. For instance, if we have 60 collections with 200 shards each,
> then this results in 12,000 shard requests from the query controller node to
> the other nodes in the cluster. The requests are sent in an async manner (see
> {{SearchHandler}} and {{HttpShardHandler}}) In my testing, we’ve seen cases
> where we hit 18,000 replicas and these queries don’t always come back in a
> timely manner. Put simply, this also puts a lot of load on the top-level
> query controller node in terms of open connections and new object creation.
> Instead, we can use {{plist}} to send the JSON facet query to each collection
> in the alias in parallel, which reduces the overhead of each top-level
> distributed query from 12,000 to 200 in my example above. With this approach,
> you’ll then need to sort the tuples back from each collection and do a
> rollup, something like:
> {code:java}
> select(
> rollup(
> sort(
> plist(
> select(facet(coll1,q="*:*", fl="a_i", sort="a_i asc", buckets="a_i",
> bucketSorts="count(*) asc", bucketSizeLimit=1, sum(a_d), avg(a_d),
> min(a_d), max(a_d), count(*)),a_i,sum(a_d) as the_sum, avg(a_d) as the_avg,
> min(a_d) as the_min, max(a_d) as the_max, count(*) as cnt),
> select(facet(coll2,q="*:*", fl="a_i", sort="a_i asc", buckets="a_i",
> bucketSorts="count(*) asc", bucketSizeLimit=1, sum(a_d), avg(a_d),
> min(a_d), max(a_d), count(*)),a_i,sum(a_d) as the_sum, avg(a_d) as the_avg,
> min(a_d) as the_min, max(a_d) as the_max, count(*) as cnt)
> ),
> by="a_i asc"
> ),
> over="a_i",
> sum(the_sum), avg(the_avg), min(the_min), max(the_max), sum(cnt)
> ),
> a_i, sum(the_sum) as the_sum, avg(the_avg) as the_avg, min(the_min) as
> the_min, max(the_max) as the_max, sum(cnt) as cnt
> )
> {code}
> One thing to point out is that you can’t just avg. the averages back from
> each collection in the rollup. It needs to be a *weighted avg.* when rolling
> up the avg. from each facet expression in the plist. However, we have the
> count per collection, so this is doable but will require some changes to the
> rollup expression to support weighted average.
> While this plist approach is doable, it’s a pain for users to have to create
> the rollup / sort over plist expression for collection aliases. After all,
> aliases are supposed to hide these types of
[jira] [Commented] (SOLR-15036) Use plist automatically for executing a facet expression against a collection alias backed by multiple collections
[
https://issues.apache.org/jira/browse/SOLR-15036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17261555#comment-17261555
]
Timothy Potter commented on SOLR-15036:
---
Also, regarding composability ... I think that approach has served us well
historically as the streaming expression module matured. However, I believe
going forward that we can start to build more intelligence into the expressions
themselves vs. putting the onus on users to figure out the optimal way to
compose an expression, especially for common analytics use cases. Either that
or we should start to re-invest in the Solr SQL to streaming expression
translator to apply more intelligence / optimizations automatically.
> Use plist automatically for executing a facet expression against a collection
> alias backed by multiple collections
> --
>
> Key: SOLR-15036
> URL: https://issues.apache.org/jira/browse/SOLR-15036
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: streaming expressions
>Reporter: Timothy Potter
>Assignee: Timothy Potter
>Priority: Major
> Attachments: relay-approach.patch
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> For analytics use cases, streaming expressions make it possible to compute
> basic aggregations (count, min, max, sum, and avg) over massive data sets.
> Moreover, with massive data sets, it is common to use collection aliases over
> many underlying collections, for instance time-partitioned aliases backed by
> a set of collections, each covering a specific time range. In some cases, we
> can end up with many collections (think 50-60) each with 100's of shards.
> Aliases help insulate client applications from complex collection topologies
> on the server side.
> Let's take a basic facet expression that computes some useful aggregation
> metrics:
> {code:java}
> facet(
> some_alias,
> q="*:*",
> fl="a_i",
> sort="a_i asc",
> buckets="a_i",
> bucketSorts="count(*) asc",
> bucketSizeLimit=1,
> sum(a_d), avg(a_d), min(a_d), max(a_d), count(*)
> )
> {code}
> Behind the scenes, the {{FacetStream}} sends a JSON facet request to Solr
> which then expands the alias to a list of collections. For each collection,
> the top-level distributed query controller gathers a candidate set of
> replicas to query and then scatters {{distrib=false}} queries to each replica
> in the list. For instance, if we have 60 collections with 200 shards each,
> then this results in 12,000 shard requests from the query controller node to
> the other nodes in the cluster. The requests are sent in an async manner (see
> {{SearchHandler}} and {{HttpShardHandler}}) In my testing, we’ve seen cases
> where we hit 18,000 replicas and these queries don’t always come back in a
> timely manner. Put simply, this also puts a lot of load on the top-level
> query controller node in terms of open connections and new object creation.
> Instead, we can use {{plist}} to send the JSON facet query to each collection
> in the alias in parallel, which reduces the overhead of each top-level
> distributed query from 12,000 to 200 in my example above. With this approach,
> you’ll then need to sort the tuples back from each collection and do a
> rollup, something like:
> {code:java}
> select(
> rollup(
> sort(
> plist(
> select(facet(coll1,q="*:*", fl="a_i", sort="a_i asc", buckets="a_i",
> bucketSorts="count(*) asc", bucketSizeLimit=1, sum(a_d), avg(a_d),
> min(a_d), max(a_d), count(*)),a_i,sum(a_d) as the_sum, avg(a_d) as the_avg,
> min(a_d) as the_min, max(a_d) as the_max, count(*) as cnt),
> select(facet(coll2,q="*:*", fl="a_i", sort="a_i asc", buckets="a_i",
> bucketSorts="count(*) asc", bucketSizeLimit=1, sum(a_d), avg(a_d),
> min(a_d), max(a_d), count(*)),a_i,sum(a_d) as the_sum, avg(a_d) as the_avg,
> min(a_d) as the_min, max(a_d) as the_max, count(*) as cnt)
> ),
> by="a_i asc"
> ),
> over="a_i",
> sum(the_sum), avg(the_avg), min(the_min), max(the_max), sum(cnt)
> ),
> a_i, sum(the_sum) as the_sum, avg(the_avg) as the_avg, min(the_min) as
> the_min, max(the_max) as the_max, sum(cnt) as cnt
> )
> {code}
> One thing to point out is that you can’t just avg. the averages back from
> each collection in the rollup. It needs to be a *weighted avg.* when rolling
> up the avg. from each facet expression in the plist. However, we have the
> count per collection, so this is doable but will require some changes to the
> rollup expression to support weighted average.
> While this plist approach is doable, it’s a pain for users to have to create
> the rollup / sort
[jira] [Comment Edited] (SOLR-15036) Use plist automatically for executing a facet expression against a collection alias backed by multiple collections
[
https://issues.apache.org/jira/browse/SOLR-15036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17261550#comment-17261550
]
Timothy Potter edited comment on SOLR-15036 at 1/8/21, 7:36 PM:
Thanks for looking [~jbernste]
I don't see any real risk to existing applications with this functionality (I
know ~ famous last words right ;-). The auto-wrapping only happens if the facet
expression only requests metrics that are safe to rollup, namely count, sum,
min, max, or avg. If while attempting to auto-wrap, we encounter a metric that
isn't safe to rollup, the impl. falls back to non-parallel.
In terms of being simpler, I'd counter that the current impl. is not simple for
users with collection alias and that the additions here are not very
complicated. The diff of {{FacetStream}} looks big because of code reformatting
to match our standard as there were many improperly formatted sections in this
file. I'm happy to undo that and just add my changes if that makes a review
easier, I think you'll see the changes are quite simple.
I'm ok with this being an explicit opt-in (vs. opt-out as I have it) your
{{tiered="true"}} idea. However, I prefer the parameter be named {{plist}} or
{{parallel}} instead of {{tiered}} if you're ok with that? The reason it is
opt-out now is because I didn't want users to have to explicitly specify this
when using a collection alias, i.e. opt-in requires changes to existing apps to
make use of this. However, I appreciate your concern about existing apps and
feel it's OK to require a change to enable an optimization, esp. if we keep the
System property to allow enabling this globally vs. in each expression
specifically.
was (Author: thelabdude):
Thanks for looking [~jbernste]
I don't see any real risk to existing applications with this functionality (I
know ~ famous last words right ;-). The auto-wrapping only happens if the facet
expression only requests metrics that are safe to rollup, namely count, sum,
min, max, or avg. If while attempting to auto-wrap, we encounter a metric that
isn't safe to rollup, the impl. falls back to non-parallel.
In terms of being simpler, I'd counter that the current impl. is not simple for
users with collection alias and that the additions here are not very
complicated. The diff of {{FacetStream}} looks big because of code reformatting
to match our standard as there were many improperly formatted sections in this
file. I'm happy to undo that and just add my changes if that makes a review
easier, I think you'll see the changes are quite simple.
I'm ok with this being an explicit opt-in (vs. opt-out as I have it) your
{{tiered="true"}} idea. However, I prefer the parameter be named {{plist}} or
{{parallel} instead of {{tiered}} if you're ok with that? The reason it is
opt-out now is because I didn't want users to have to explicitly specify this
when using a collection alias, i.e. opt-in requires changes to existing apps to
make use of this. However, I appreciate your concern about existing apps and
feel it's OK to require a change to enable an optimization, esp. if we keep the
System property to allow enabling this globally vs. in each expression
specifically.
> Use plist automatically for executing a facet expression against a collection
> alias backed by multiple collections
> --
>
> Key: SOLR-15036
> URL: https://issues.apache.org/jira/browse/SOLR-15036
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: streaming expressions
>Reporter: Timothy Potter
>Assignee: Timothy Potter
>Priority: Major
> Attachments: relay-approach.patch
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> For analytics use cases, streaming expressions make it possible to compute
> basic aggregations (count, min, max, sum, and avg) over massive data sets.
> Moreover, with massive data sets, it is common to use collection aliases over
> many underlying collections, for instance time-partitioned aliases backed by
> a set of collections, each covering a specific time range. In some cases, we
> can end up with many collections (think 50-60) each with 100's of shards.
> Aliases help insulate client applications from complex collection topologies
> on the server side.
> Let's take a basic facet expression that computes some useful aggregation
> metrics:
> {code:java}
> facet(
> some_alias,
> q="*:*",
> fl="a_i",
> sort="a_i asc",
> buckets="a_i",
> bucketSorts="count(*) asc",
> bucketSizeLimit=1,
> sum(a_d), avg(a_d), min(a_d), max(a_d), count(*)
> )
> {code}
> Behind the scenes, the {{FacetStream}} sends a JSON
[jira] [Commented] (SOLR-15036) Use plist automatically for executing a facet expression against a collection alias backed by multiple collections
[
https://issues.apache.org/jira/browse/SOLR-15036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17261550#comment-17261550
]
Timothy Potter commented on SOLR-15036:
---
Thanks for looking [~jbernste]
I don't see any real risk to existing applications with this functionality (I
know ~ famous last words right ;-). The auto-wrapping only happens if the facet
expression only requests metrics that are safe to rollup, namely count, sum,
min, max, or avg. If while attempting to auto-wrap, we encounter a metric that
isn't safe to rollup, the impl. falls back to non-parallel.
In terms of being simpler, I'd counter that the current impl. is not simple for
users with collection alias and that the additions here are not very
complicated. The diff of {{FacetStream}} looks big because of code reformatting
to match our standard as there were many improperly formatted sections in this
file. I'm happy to undo that and just add my changes if that makes a review
easier, I think you'll see the changes are quite simple.
I'm ok with this being an explicit opt-in (vs. opt-out as I have it) your
{{tiered="true"}} idea. However, I prefer the parameter be named {{plist}} or
{{parallel} instead of {{tiered}} if you're ok with that? The reason it is
opt-out now is because I didn't want users to have to explicitly specify this
when using a collection alias, i.e. opt-in requires changes to existing apps to
make use of this. However, I appreciate your concern about existing apps and
feel it's OK to require a change to enable an optimization, esp. if we keep the
System property to allow enabling this globally vs. in each expression
specifically.
> Use plist automatically for executing a facet expression against a collection
> alias backed by multiple collections
> --
>
> Key: SOLR-15036
> URL: https://issues.apache.org/jira/browse/SOLR-15036
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: streaming expressions
>Reporter: Timothy Potter
>Assignee: Timothy Potter
>Priority: Major
> Attachments: relay-approach.patch
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> For analytics use cases, streaming expressions make it possible to compute
> basic aggregations (count, min, max, sum, and avg) over massive data sets.
> Moreover, with massive data sets, it is common to use collection aliases over
> many underlying collections, for instance time-partitioned aliases backed by
> a set of collections, each covering a specific time range. In some cases, we
> can end up with many collections (think 50-60) each with 100's of shards.
> Aliases help insulate client applications from complex collection topologies
> on the server side.
> Let's take a basic facet expression that computes some useful aggregation
> metrics:
> {code:java}
> facet(
> some_alias,
> q="*:*",
> fl="a_i",
> sort="a_i asc",
> buckets="a_i",
> bucketSorts="count(*) asc",
> bucketSizeLimit=1,
> sum(a_d), avg(a_d), min(a_d), max(a_d), count(*)
> )
> {code}
> Behind the scenes, the {{FacetStream}} sends a JSON facet request to Solr
> which then expands the alias to a list of collections. For each collection,
> the top-level distributed query controller gathers a candidate set of
> replicas to query and then scatters {{distrib=false}} queries to each replica
> in the list. For instance, if we have 60 collections with 200 shards each,
> then this results in 12,000 shard requests from the query controller node to
> the other nodes in the cluster. The requests are sent in an async manner (see
> {{SearchHandler}} and {{HttpShardHandler}}) In my testing, we’ve seen cases
> where we hit 18,000 replicas and these queries don’t always come back in a
> timely manner. Put simply, this also puts a lot of load on the top-level
> query controller node in terms of open connections and new object creation.
> Instead, we can use {{plist}} to send the JSON facet query to each collection
> in the alias in parallel, which reduces the overhead of each top-level
> distributed query from 12,000 to 200 in my example above. With this approach,
> you’ll then need to sort the tuples back from each collection and do a
> rollup, something like:
> {code:java}
> select(
> rollup(
> sort(
> plist(
> select(facet(coll1,q="*:*", fl="a_i", sort="a_i asc", buckets="a_i",
> bucketSorts="count(*) asc", bucketSizeLimit=1, sum(a_d), avg(a_d),
> min(a_d), max(a_d), count(*)),a_i,sum(a_d) as the_sum, avg(a_d) as the_avg,
> min(a_d) as the_min, max(a_d) as the_max, count(*) as cnt),
> select(facet(coll2,q="*:*", fl="a_i", sort="a_i asc",
[jira] [Updated] (SOLR-15036) Use plist automatically for executing a facet expression against a collection alias backed by multiple collections
[
https://issues.apache.org/jira/browse/SOLR-15036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timothy Potter updated SOLR-15036:
--
Status: Patch Available (was: Open)
> Use plist automatically for executing a facet expression against a collection
> alias backed by multiple collections
> --
>
> Key: SOLR-15036
> URL: https://issues.apache.org/jira/browse/SOLR-15036
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: streaming expressions
>Reporter: Timothy Potter
>Assignee: Timothy Potter
>Priority: Major
> Attachments: relay-approach.patch
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> For analytics use cases, streaming expressions make it possible to compute
> basic aggregations (count, min, max, sum, and avg) over massive data sets.
> Moreover, with massive data sets, it is common to use collection aliases over
> many underlying collections, for instance time-partitioned aliases backed by
> a set of collections, each covering a specific time range. In some cases, we
> can end up with many collections (think 50-60) each with 100's of shards.
> Aliases help insulate client applications from complex collection topologies
> on the server side.
> Let's take a basic facet expression that computes some useful aggregation
> metrics:
> {code:java}
> facet(
> some_alias,
> q="*:*",
> fl="a_i",
> sort="a_i asc",
> buckets="a_i",
> bucketSorts="count(*) asc",
> bucketSizeLimit=1,
> sum(a_d), avg(a_d), min(a_d), max(a_d), count(*)
> )
> {code}
> Behind the scenes, the {{FacetStream}} sends a JSON facet request to Solr
> which then expands the alias to a list of collections. For each collection,
> the top-level distributed query controller gathers a candidate set of
> replicas to query and then scatters {{distrib=false}} queries to each replica
> in the list. For instance, if we have 60 collections with 200 shards each,
> then this results in 12,000 shard requests from the query controller node to
> the other nodes in the cluster. The requests are sent in an async manner (see
> {{SearchHandler}} and {{HttpShardHandler}}) In my testing, we’ve seen cases
> where we hit 18,000 replicas and these queries don’t always come back in a
> timely manner. Put simply, this also puts a lot of load on the top-level
> query controller node in terms of open connections and new object creation.
> Instead, we can use {{plist}} to send the JSON facet query to each collection
> in the alias in parallel, which reduces the overhead of each top-level
> distributed query from 12,000 to 200 in my example above. With this approach,
> you’ll then need to sort the tuples back from each collection and do a
> rollup, something like:
> {code:java}
> select(
> rollup(
> sort(
> plist(
> select(facet(coll1,q="*:*", fl="a_i", sort="a_i asc", buckets="a_i",
> bucketSorts="count(*) asc", bucketSizeLimit=1, sum(a_d), avg(a_d),
> min(a_d), max(a_d), count(*)),a_i,sum(a_d) as the_sum, avg(a_d) as the_avg,
> min(a_d) as the_min, max(a_d) as the_max, count(*) as cnt),
> select(facet(coll2,q="*:*", fl="a_i", sort="a_i asc", buckets="a_i",
> bucketSorts="count(*) asc", bucketSizeLimit=1, sum(a_d), avg(a_d),
> min(a_d), max(a_d), count(*)),a_i,sum(a_d) as the_sum, avg(a_d) as the_avg,
> min(a_d) as the_min, max(a_d) as the_max, count(*) as cnt)
> ),
> by="a_i asc"
> ),
> over="a_i",
> sum(the_sum), avg(the_avg), min(the_min), max(the_max), sum(cnt)
> ),
> a_i, sum(the_sum) as the_sum, avg(the_avg) as the_avg, min(the_min) as
> the_min, max(the_max) as the_max, sum(cnt) as cnt
> )
> {code}
> One thing to point out is that you can’t just avg. the averages back from
> each collection in the rollup. It needs to be a *weighted avg.* when rolling
> up the avg. from each facet expression in the plist. However, we have the
> count per collection, so this is doable but will require some changes to the
> rollup expression to support weighted average.
> While this plist approach is doable, it’s a pain for users to have to create
> the rollup / sort over plist expression for collection aliases. After all,
> aliases are supposed to hide these types of complexities from client
> applications!
> The point of this ticket is to investigate the feasibility of auto-wrapping
> the facet expression with a rollup / sort / plist when the collection
> argument is an alias with multiple collections; other stream sources will be
> considered after facet is proven out.
> Lastly, I also considered an alternative approach of doing a parallel relay
> on the server side. The idea is
[jira] [Commented] (SOLR-15036) Use plist automatically for executing a facet expression against a collection alias backed by multiple collections
[
https://issues.apache.org/jira/browse/SOLR-15036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17261434#comment-17261434
]
Timothy Potter commented on SOLR-15036:
---
[~jbernste] I think this is ready to go and would appreciate your eyes on it:
https://github.com/apache/lucene-solr/pull/2132
> Use plist automatically for executing a facet expression against a collection
> alias backed by multiple collections
> --
>
> Key: SOLR-15036
> URL: https://issues.apache.org/jira/browse/SOLR-15036
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: streaming expressions
>Reporter: Timothy Potter
>Assignee: Timothy Potter
>Priority: Major
> Attachments: relay-approach.patch
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> For analytics use cases, streaming expressions make it possible to compute
> basic aggregations (count, min, max, sum, and avg) over massive data sets.
> Moreover, with massive data sets, it is common to use collection aliases over
> many underlying collections, for instance time-partitioned aliases backed by
> a set of collections, each covering a specific time range. In some cases, we
> can end up with many collections (think 50-60) each with 100's of shards.
> Aliases help insulate client applications from complex collection topologies
> on the server side.
> Let's take a basic facet expression that computes some useful aggregation
> metrics:
> {code:java}
> facet(
> some_alias,
> q="*:*",
> fl="a_i",
> sort="a_i asc",
> buckets="a_i",
> bucketSorts="count(*) asc",
> bucketSizeLimit=1,
> sum(a_d), avg(a_d), min(a_d), max(a_d), count(*)
> )
> {code}
> Behind the scenes, the {{FacetStream}} sends a JSON facet request to Solr
> which then expands the alias to a list of collections. For each collection,
> the top-level distributed query controller gathers a candidate set of
> replicas to query and then scatters {{distrib=false}} queries to each replica
> in the list. For instance, if we have 60 collections with 200 shards each,
> then this results in 12,000 shard requests from the query controller node to
> the other nodes in the cluster. The requests are sent in an async manner (see
> {{SearchHandler}} and {{HttpShardHandler}}) In my testing, we’ve seen cases
> where we hit 18,000 replicas and these queries don’t always come back in a
> timely manner. Put simply, this also puts a lot of load on the top-level
> query controller node in terms of open connections and new object creation.
> Instead, we can use {{plist}} to send the JSON facet query to each collection
> in the alias in parallel, which reduces the overhead of each top-level
> distributed query from 12,000 to 200 in my example above. With this approach,
> you’ll then need to sort the tuples back from each collection and do a
> rollup, something like:
> {code:java}
> select(
> rollup(
> sort(
> plist(
> select(facet(coll1,q="*:*", fl="a_i", sort="a_i asc", buckets="a_i",
> bucketSorts="count(*) asc", bucketSizeLimit=1, sum(a_d), avg(a_d),
> min(a_d), max(a_d), count(*)),a_i,sum(a_d) as the_sum, avg(a_d) as the_avg,
> min(a_d) as the_min, max(a_d) as the_max, count(*) as cnt),
> select(facet(coll2,q="*:*", fl="a_i", sort="a_i asc", buckets="a_i",
> bucketSorts="count(*) asc", bucketSizeLimit=1, sum(a_d), avg(a_d),
> min(a_d), max(a_d), count(*)),a_i,sum(a_d) as the_sum, avg(a_d) as the_avg,
> min(a_d) as the_min, max(a_d) as the_max, count(*) as cnt)
> ),
> by="a_i asc"
> ),
> over="a_i",
> sum(the_sum), avg(the_avg), min(the_min), max(the_max), sum(cnt)
> ),
> a_i, sum(the_sum) as the_sum, avg(the_avg) as the_avg, min(the_min) as
> the_min, max(the_max) as the_max, sum(cnt) as cnt
> )
> {code}
> One thing to point out is that you can’t just avg. the averages back from
> each collection in the rollup. It needs to be a *weighted avg.* when rolling
> up the avg. from each facet expression in the plist. However, we have the
> count per collection, so this is doable but will require some changes to the
> rollup expression to support weighted average.
> While this plist approach is doable, it’s a pain for users to have to create
> the rollup / sort over plist expression for collection aliases. After all,
> aliases are supposed to hide these types of complexities from client
> applications!
> The point of this ticket is to investigate the feasibility of auto-wrapping
> the facet expression with a rollup / sort / plist when the collection
> argument is an alias with multiple collections; other stream sources will be
> considered after facet
[jira] [Updated] (SOLR-15059) Default Grafana dashboard needs to expose graphs for monitoring query performance
[
https://issues.apache.org/jira/browse/SOLR-15059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timothy Potter updated SOLR-15059:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Default Grafana dashboard needs to expose graphs for monitoring query
> performance
> -
>
> Key: SOLR-15059
> URL: https://issues.apache.org/jira/browse/SOLR-15059
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Grafana Dashboard, metrics
>Reporter: Timothy Potter
>Assignee: Timothy Potter
>Priority: Major
> Fix For: 8.8, master (9.0)
>
> Attachments: Screen Shot 2020-12-23 at 10.22.43 AM.png
>
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> The default Grafana dashboard doesn't expose graphs for monitoring query
> performance. For instance, if I want to see QPS for a collection, that's not
> shown in the default dashboard. Same for quantiles like p95 query latency.
> After some digging, these metrics are available in the output from
> {{/admin/metrics}} but are not exported by the exporter.
> This PR proposes to enhance the default dashboard with a new Query Metrics
> section with the following metrics:
> * Distributed QPS per Collection (aggregated across all cores)
> * Distributed QPS per Solr Node (aggregated across all base_url)
> * QPS 1-min rate per core
> * QPS 5-min rate per core
> * Top-level Query latency p99, p95, p75
> * Local (non-distrib) query count per core (this is important for determining
> if there is unbalanced load)
> * Local (non-distrib) query rate per core (1-min)
> * Local (non-distrib) p95 per core
> Also, the {{solr-exporter-config.xml}} uses {{jq}} queries to pull metrics
> from the output from {{/admin/metrics}}. This file is huge and contains a
> bunch of {{jq}} boilerplate. Moreover, I'm introducing another 15-20 metrics
> in this PR, it only makes the file more verbose.
> Thus, I'm also introducing support for jq templates so as to reduce
> boilerplate, reduce syntax errors, and improve readability. For instance the
> query metrics I'm adding to the config look like this:
> {code}
>
> $jq:core-query(1minRate, endswith(".distrib.requestTimes"))
>
>
> $jq:core-query(5minRate, endswith(".distrib.requestTimes"))
>
> {code}
> Instead of duplicating the complicated {{jq}} query for each metric. The
> templates are optional and only should be used if a given jq structure is
> repeated 3 or more times. Otherwise, inlining the jq query is still
> supported. Here's how the templates work:
> {code}
> A regex with named groups is used to match template references to template
> + vars using the basic pattern:
> $jq:( , , , )
> For instance,
> $jq:core(requests_total, endswith(".requestTimes"), count, COUNTER)
> TEMPLATE = core
> UNIQUE = requests_total (unique suffix for this metric, results in a metric
> named "solr_metrics_core_requests_total")
> KEYSELECTOR = endswith(".requestTimes") (filter to select the specific key
> for this metric)
> METRIC = count
> TYPE = COUNTER
> Some templates may have a default type, so you can omit that from your
> template reference, such as:
> $jq:core(requests_total, endswith(".requestTimes"), count)
> Uses the defaultType=COUNTER as many uses of the core template are counts.
> If a template reference omits the metric, then the unique suffix is used,
> for instance:
> $jq:core-query(1minRate, endswith(".distrib.requestTimes"))
> Creates a GAUGE metric (default type) named
> "solr_metrics_core_query_1minRate" using the 1minRate value from the selected
> JSON object.
> {code}
> Just so people don't have to go digging in the large diff on the config XML,
> here are the query metrics I'm adding to the exporter config with use of the
> templates idea:
> {code}
>
> $jq:core-query(errors_1minRate, select(.key |
> endswith(".errors")), 1minRate)
>
>
> $jq:core-query(client_errors_1minRate, select(.key |
> endswith(".clientErrors")), 1minRate)
>
>
> $jq:core-query(1minRate, select(.key |
> endswith(".distrib.requestTimes")), 1minRate)
>
>
> $jq:core-query(5minRate, select(.key |
> endswith(".distrib.requestTimes")), 5minRate)
>
>
> $jq:core-query(median_ms, select(.key |
> endswith(".distrib.requestTimes")), median_ms)
>
>
> $jq:core-query(p75_ms, select(.key |
> endswith(".distrib.requestTimes")), p75_ms)
>
>
> $jq:core-query(p95_ms,
[jira] [Updated] (SOLR-15059) Default Grafana dashboard needs to expose graphs for monitoring query performance
[
https://issues.apache.org/jira/browse/SOLR-15059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timothy Potter updated SOLR-15059:
--
Description:
The default Grafana dashboard doesn't expose graphs for monitoring query
performance. For instance, if I want to see QPS for a collection, that's not
shown in the default dashboard. Same for quantiles like p95 query latency.
After some digging, these metrics are available in the output from
{{/admin/metrics}} but are not exported by the exporter.
This PR proposes to enhance the default dashboard with a new Query Metrics
section with the following metrics:
* Distributed QPS per Collection (aggregated across all cores)
* Distributed QPS per Solr Node (aggregated across all base_url)
* QPS 1-min rate per core
* QPS 5-min rate per core
* Top-level Query latency p99, p95, p75
* Local (non-distrib) query count per core (this is important for determining
if there is unbalanced load)
* Local (non-distrib) query rate per core (1-min)
* Local (non-distrib) p95 per core
Also, the {{solr-exporter-config.xml}} uses {{jq}} queries to pull metrics from
the output from {{/admin/metrics}}. This file is huge and contains a bunch of
{{jq}} boilerplate. Moreover, I'm introducing another 15-20 metrics in this PR,
it only makes the file more verbose.
Thus, I'm also introducing support for jq templates so as to reduce
boilerplate, reduce syntax errors, and improve readability. For instance the
query metrics I'm adding to the config look like this:
{code}
$jq:core-query(1minRate, endswith(".distrib.requestTimes"))
$jq:core-query(5minRate, endswith(".distrib.requestTimes"))
{code}
Instead of duplicating the complicated {{jq}} query for each metric. The
templates are optional and only should be used if a given jq structure is
repeated 3 or more times. Otherwise, inlining the jq query is still supported.
Here's how the templates work:
{code}
A regex with named groups is used to match template references to template +
vars using the basic pattern:
$jq:( , , , )
For instance,
$jq:core(requests_total, endswith(".requestTimes"), count, COUNTER)
TEMPLATE = core
UNIQUE = requests_total (unique suffix for this metric, results in a metric
named "solr_metrics_core_requests_total")
KEYSELECTOR = endswith(".requestTimes") (filter to select the specific key
for this metric)
METRIC = count
TYPE = COUNTER
Some templates may have a default type, so you can omit that from your
template reference, such as:
$jq:core(requests_total, endswith(".requestTimes"), count)
Uses the defaultType=COUNTER as many uses of the core template are counts.
If a template reference omits the metric, then the unique suffix is used, for
instance:
$jq:core-query(1minRate, endswith(".distrib.requestTimes"))
Creates a GAUGE metric (default type) named
"solr_metrics_core_query_1minRate" using the 1minRate value from the selected
JSON object.
{code}
Just so people don't have to go digging in the large diff on the config XML,
here are the query metrics I'm adding to the exporter config with use of the
templates idea:
{code}
$jq:core-query(errors_1minRate, select(.key | endswith(".errors")),
1minRate)
$jq:core-query(client_errors_1minRate, select(.key |
endswith(".clientErrors")), 1minRate)
$jq:core-query(1minRate, select(.key |
endswith(".distrib.requestTimes")), 1minRate)
$jq:core-query(5minRate, select(.key |
endswith(".distrib.requestTimes")), 5minRate)
$jq:core-query(median_ms, select(.key |
endswith(".distrib.requestTimes")), median_ms)
$jq:core-query(p75_ms, select(.key |
endswith(".distrib.requestTimes")), p75_ms)
$jq:core-query(p95_ms, select(.key |
endswith(".distrib.requestTimes")), p95_ms)
$jq:core-query(p99_ms, select(.key |
endswith(".distrib.requestTimes")), p99_ms)
$jq:core-query(mean_rate, select(.key |
endswith(".distrib.requestTimes")), meanRate)
$jq:core-query(local_1minRate, select(.key |
endswith(".local.requestTimes")), 1minRate)
$jq:core-query(local_5minRate, select(.key |
endswith(".local.requestTimes")), 5minRate)
$jq:core-query(local_median_ms, select(.key |
endswith(".local.requestTimes")), median_ms)
$jq:core-query(local_p75_ms, select(.key |
endswith(".local.requestTimes")), p75_ms)
$jq:core-query(local_p95_ms, select(.key |
endswith(".local.requestTimes")), p95_ms)
[jira] [Resolved] (SOLR-15058) Solr can not get correct hostname when hostname contain '_'
[ https://issues.apache.org/jira/browse/SOLR-15058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timothy Potter resolved SOLR-15058. --- Fix Version/s: master (9.0) 8.8 Resolution: Fixed > Solr can not get correct hostname when hostname contain '_' > --- > > Key: SOLR-15058 > URL: https://issues.apache.org/jira/browse/SOLR-15058 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Su Sasa >Assignee: Timothy Potter >Priority: Minor > Fix For: 8.8, master (9.0) > > Attachments: > 0001-SOLR-15058-get-the-correct-hostname-when-hostname-co.patch > > Time Spent: 2h > Remaining Estimate: 0h > > Utils.getBaseUrlForNodeName split the nodename by the first '_' > In ZK, Solr add '_solr' to the end of the hostname > so there should split by the last '_' -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Assigned] (SOLR-15058) Solr can not get correct hostname when hostname contain '_'
[ https://issues.apache.org/jira/browse/SOLR-15058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timothy Potter reassigned SOLR-15058: - Assignee: Timothy Potter > Solr can not get correct hostname when hostname contain '_' > --- > > Key: SOLR-15058 > URL: https://issues.apache.org/jira/browse/SOLR-15058 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Su Sasa >Assignee: Timothy Potter >Priority: Minor > Attachments: > 0001-SOLR-15058-get-the-correct-hostname-when-hostname-co.patch > > Time Spent: 40m > Remaining Estimate: 0h > > Utils.getBaseUrlForNodeName split the nodename by the first '_' > In ZK, Solr add '_solr' to the end of the hostname > so there should split by the last '_' -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-15058) Solr can not get correct hostname when hostname contain '_'
[ https://issues.apache.org/jira/browse/SOLR-15058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17256071#comment-17256071 ] Timothy Potter commented on SOLR-15058: --- Let's get this fixed for 8.8 ... I'll work to get it committed as I've been in this code recently as well. > Solr can not get correct hostname when hostname contain '_' > --- > > Key: SOLR-15058 > URL: https://issues.apache.org/jira/browse/SOLR-15058 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Su Sasa >Assignee: Timothy Potter >Priority: Minor > Attachments: > 0001-SOLR-15058-get-the-correct-hostname-when-hostname-co.patch > > Time Spent: 40m > Remaining Estimate: 0h > > Utils.getBaseUrlForNodeName split the nodename by the first '_' > In ZK, Solr add '_solr' to the end of the hostname > so there should split by the last '_' -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-15059) Default Grafana dashboard needs to expose graphs for monitoring query performance
[
https://issues.apache.org/jira/browse/SOLR-15059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timothy Potter updated SOLR-15059:
--
Description:
The default Grafana dashboard doesn't expose graphs for monitoring query
performance. For instance, if I want to see QPS for a collection, that's not
shown in the default dashboard. Same for quantiles like p95 query latency.
After some digging, these metrics are available in the output from
{{/admin/metrics}} but are not exported by the exporter.
This PR proposes to enhance the default dashboard with a new Query Metrics
section with the following metrics:
* Distributed QPS per Collection (aggregated across all cores)
* Distributed QPS per Solr Node (aggregated across all base_url)
* QPS 1-min rate per core
* QPS 5-min rate per core
* Top-level Query latency p99, p95, p75
* Local (non-distrib) query count per core (this is important for determining
if there is unbalanced load)
* Local (non-distrib) query rate per core (1-min)
* Local (non-distrib) p95 per core
Also, the {{solr-exporter-config.xml}} uses {{jq}} queries to pull metrics from
the output from {{/admin/metrics}}. This file is huge and contains a bunch of
{{jq}} boilerplate. Moreover, I'm introducing another 15-20 metrics in this PR,
it only makes the file more verbose.
Thus, I'm also introducing support for jq templates so as to reduce
boilerplate, reduce syntax errors, and improve readability. For instance the
query metrics I'm adding to the config look like this:
{code}
$jq:core-query(1minRate, endswith(".distrib.requestTimes"))
$jq:core-query(5minRate, endswith(".distrib.requestTimes"))
{code}
Instead of duplicating the complicated {{jq}} query for each metric. The
templates are optional and only should be used if a given jq structure is
repeated 3 or more times. Otherwise, inlining the jq query is still supported.
Here's how the templates work:
{code}
A regex with named groups is used to match template references to template +
vars using the basic pattern:
$jq:( , , , )
For instance,
$jq:core(requests_total, endswith(".requestTimes"), count, COUNTER)
TEMPLATE = core
UNIQUE = requests_total (unique suffix for this metric, results in a metric
named "solr_metrics_core_requests_total")
KEYSELECTOR = endswith(".requestTimes") (filter to select the specific key
for this metric)
METRIC = count
TYPE = COUNTER
Some templates may have a default type, so you can omit that from your
template reference, such as:
$jq:core(requests_total, endswith(".requestTimes"), count)
Uses the defaultType=COUNTER as many uses of the core template are counts.
If a template reference omits the metric, then the unique suffix is used, for
instance:
$jq:core-query(1minRate, endswith(".distrib.requestTimes"))
Creates a GAUGE metric (default type) named
"solr_metrics_core_query_1minRate" using the 1minRate value from the selected
JSON object.
{code}
Just so people don't have to go digging in the large diff on the config XML,
here are the query metrics I'm adding to the exporter config with use of the
templates idea:
{code}
$jq:core-query(errors_1minRate, endswith(".errors"), 1minRate)
$jq:core-query(client_errors_1minRate, endswith(".clientErrors"),
1minRate)
$jq:core-query(1minRate, endswith(".distrib.requestTimes"))
$jq:core-query(5minRate, endswith(".distrib.requestTimes"))
$jq:core-query(median_ms, endswith(".distrib.requestTimes"))
$jq:core-query(p75_ms, endswith(".distrib.requestTimes"))
$jq:core-query(p95_ms, endswith(".distrib.requestTimes"))
$jq:core-query(p99_ms, endswith(".distrib.requestTimes"))
$jq:core-query(mean_rate, endswith(".distrib.requestTimes"),
meanRate)
$jq:core-query(local_1minRate, endswith(".local.requestTimes"),
1minRate)
$jq:core-query(local_5minRate, endswith(".local.requestTimes"),
5minRate)
$jq:core-query(local_median_ms, endswith(".local.requestTimes"),
median_ms)
$jq:core-query(local_p75_ms, endswith(".local.requestTimes"),
p75_ms)
$jq:core-query(local_p95_ms, endswith(".local.requestTimes"),
p95_ms)
$jq:core-query(local_p99_ms, endswith(".local.requestTimes"),
p99_ms)
$jq:core-query(local_mean_rate, endswith(".local.requestTimes"),
meanRate)
$jq:core-query(local_count, endswith(".local.requestTimes"), count,
[jira] [Updated] (SOLR-15059) Default Grafana dashboard needs to expose graphs for monitoring query performance
[
https://issues.apache.org/jira/browse/SOLR-15059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timothy Potter updated SOLR-15059:
--
Attachment: Screen Shot 2020-12-23 at 10.22.43 AM.png
Status: Open (was: Open)
> Default Grafana dashboard needs to expose graphs for monitoring query
> performance
> -
>
> Key: SOLR-15059
> URL: https://issues.apache.org/jira/browse/SOLR-15059
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Grafana Dashboard, metrics
>Reporter: Timothy Potter
>Assignee: Timothy Potter
>Priority: Major
> Fix For: 8.8, master (9.0)
>
> Attachments: Screen Shot 2020-12-23 at 10.22.43 AM.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The default Grafana dashboard doesn't expose graphs for monitoring query
> performance. For instance, if I want to see QPS for a collection, that's not
> shown in the default dashboard. Same for quantiles like p95 query latency.
> After some digging, these metrics are available in the output from
> {{/admin/metrics}} but are not exported by the exporter.
> This PR proposes to enhance the default dashboard with a new Query Metrics
> section with the following metrics:
> * Distributed QPS per Collection (aggregated across all cores)
> * Distributed QPS per Solr Node (aggregated across all base_url)
> * QPS 1-min rate per core
> * QPS 5-min rate per core
> * Top-level Query latency p99, p95, p75
> * Local (non-distrib) query count per core (this is important for determining
> if there is unbalanced load)
> * Local (non-distrib) query rate per core (1-min)
> * Local (non-distrib) p95 per core
> Also, the {{solr-exporter-config.xml}} uses {{jq}} queries to pull metrics
> from the output from {{/admin/metrics}}. This file is huge and contains a
> bunch of {{jq}} boilerplate. Moreover, I'm introducing another 15-20 metrics
> in this PR, it only makes the file more verbose.
> Thus, I'm also introducing support for jq templates so as to reduce
> boilerplate, reduce syntax errors, and improve readability. For instance the
> query metrics I'm adding to the config look like this:
> {code}
>
> $jq:core-query(1minRate, endswith(".distrib.requestTimes"))
>
>
> $jq:core-query(5minRate, endswith(".distrib.requestTimes"))
>
> {code}
> Instead of duplicating the complicated {{jq}} query for each metric. The
> templates are optional and only should be used if a given jq structure is
> repeated 3 or more times. Otherwise, inlining the jq query is still
> supported. Here's how the templates work:
> {code}
> A regex with named groups is used to match template references to template
> + vars using the basic pattern:
> $jq:( , , , )
> For instance,
> $jq:core(requests_total, endswith(".requestTimes"), count, COUNTER)
> TEMPLATE = core
> UNIQUE = requests_total (unique suffix for this metric, results in a metric
> named "solr_metrics_core_requests_total")
> KEYSELECTOR = endswith(".requestTimes") (filter to select the specific key
> for this metric)
> METRIC = count
> TYPE = COUNTER
> Some templates may have a default type, so you can omit that from your
> template reference, such as:
> $jq:core(requests_total, endswith(".requestTimes"), count)
> Uses the defaultType=COUNTER as many uses of the core template are counts.
> If a template reference omits the metric, then the unique suffix is used,
> for instance:
> $jq:core-query(1minRate, endswith(".distrib.requestTimes"))
> Creates a GAUGE metric (default type) named
> "solr_metrics_core_query_1minRate" using the 1minRate value from the selected
> JSON object.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-15059) Default Grafana dashboard needs to expose graphs for monitoring query performance
[
https://issues.apache.org/jira/browse/SOLR-15059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timothy Potter updated SOLR-15059:
--
Status: Patch Available (was: Open)
> Default Grafana dashboard needs to expose graphs for monitoring query
> performance
> -
>
> Key: SOLR-15059
> URL: https://issues.apache.org/jira/browse/SOLR-15059
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Grafana Dashboard, metrics
>Reporter: Timothy Potter
>Assignee: Timothy Potter
>Priority: Major
> Fix For: 8.8, master (9.0)
>
> Attachments: Screen Shot 2020-12-23 at 10.22.43 AM.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The default Grafana dashboard doesn't expose graphs for monitoring query
> performance. For instance, if I want to see QPS for a collection, that's not
> shown in the default dashboard. Same for quantiles like p95 query latency.
> After some digging, these metrics are available in the output from
> {{/admin/metrics}} but are not exported by the exporter.
> This PR proposes to enhance the default dashboard with a new Query Metrics
> section with the following metrics:
> * Distributed QPS per Collection (aggregated across all cores)
> * Distributed QPS per Solr Node (aggregated across all base_url)
> * QPS 1-min rate per core
> * QPS 5-min rate per core
> * Top-level Query latency p99, p95, p75
> * Local (non-distrib) query count per core (this is important for determining
> if there is unbalanced load)
> * Local (non-distrib) query rate per core (1-min)
> * Local (non-distrib) p95 per core
> Also, the {{solr-exporter-config.xml}} uses {{jq}} queries to pull metrics
> from the output from {{/admin/metrics}}. This file is huge and contains a
> bunch of {{jq}} boilerplate. Moreover, I'm introducing another 15-20 metrics
> in this PR, it only makes the file more verbose.
> Thus, I'm also introducing support for jq templates so as to reduce
> boilerplate, reduce syntax errors, and improve readability. For instance the
> query metrics I'm adding to the config look like this:
> {code}
>
> $jq:core-query(1minRate, endswith(".distrib.requestTimes"))
>
>
> $jq:core-query(5minRate, endswith(".distrib.requestTimes"))
>
> {code}
> Instead of duplicating the complicated {{jq}} query for each metric. The
> templates are optional and only should be used if a given jq structure is
> repeated 3 or more times. Otherwise, inlining the jq query is still
> supported. Here's how the templates work:
> {code}
> A regex with named groups is used to match template references to template
> + vars using the basic pattern:
> $jq:( , , , )
> For instance,
> $jq:core(requests_total, endswith(".requestTimes"), count, COUNTER)
> TEMPLATE = core
> UNIQUE = requests_total (unique suffix for this metric, results in a metric
> named "solr_metrics_core_requests_total")
> KEYSELECTOR = endswith(".requestTimes") (filter to select the specific key
> for this metric)
> METRIC = count
> TYPE = COUNTER
> Some templates may have a default type, so you can omit that from your
> template reference, such as:
> $jq:core(requests_total, endswith(".requestTimes"), count)
> Uses the defaultType=COUNTER as many uses of the core template are counts.
> If a template reference omits the metric, then the unique suffix is used,
> for instance:
> $jq:core-query(1minRate, endswith(".distrib.requestTimes"))
> Creates a GAUGE metric (default type) named
> "solr_metrics_core_query_1minRate" using the 1minRate value from the selected
> JSON object.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-15059) Default Grafana dashboard needs to expose graphs for monitoring query performance
[
https://issues.apache.org/jira/browse/SOLR-15059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timothy Potter updated SOLR-15059:
--
Description:
The default Grafana dashboard doesn't expose graphs for monitoring query
performance. For instance, if I want to see QPS for a collection, that's not
shown in the default dashboard. Same for quantiles like p95 query latency.
After some digging, these metrics are available in the output from
{{/admin/metrics}} but are not exported by the exporter.
This PR proposes to enhance the default dashboard with a new Query Metrics
section with the following metrics:
* Distributed QPS per Collection (aggregated across all cores)
* Distributed QPS per Solr Node (aggregated across all base_url)
* QPS 1-min rate per core
* QPS 5-min rate per core
* Top-level Query latency p99, p95, p75
* Local (non-distrib) query count per core (this is important for determining
if there is unbalanced load)
* Local (non-distrib) query rate per core (1-min)
* Local (non-distrib) p95 per core
Also, the {{solr-exporter-config.xml}} uses {{jq}} queries to pull metrics from
the output from {{/admin/metrics}}. This file is huge and contains a bunch of
{{jq}} boilerplate. Moreover, I'm introducing another 15-20 metrics in this PR,
it only makes the file more verbose.
Thus, I'm also introducing support for jq templates so as to reduce
boilerplate, reduce syntax errors, and improve readability. For instance the
query metrics I'm adding to the config look like this:
{code}
$jq:core-query(1minRate, endswith(".distrib.requestTimes"))
$jq:core-query(5minRate, endswith(".distrib.requestTimes"))
{code}
Instead of duplicating the complicated {{jq}} query for each metric. The
templates are optional and only should be used if a given jq structure is
repeated 3 or more times. Otherwise, inlining the jq query is still supported.
Here's how the templates work:
{code}
A regex with named groups is used to match template references to template +
vars using the basic pattern:
$jq:( , , , )
For instance,
$jq:core(requests_total, endswith(".requestTimes"), count, COUNTER)
TEMPLATE = core
UNIQUE = requests_total (unique suffix for this metric, results in a metric
named "solr_metrics_core_requests_total")
KEYSELECTOR = endswith(".requestTimes") (filter to select the specific key
for this metric)
METRIC = count
TYPE = COUNTER
Some templates may have a default type, so you can omit that from your
template reference, such as:
$jq:core(requests_total, endswith(".requestTimes"), count)
Uses the defaultType=COUNTER as many uses of the core template are counts.
If a template reference omits the metric, then the unique suffix is used, for
instance:
$jq:core-query(1minRate, endswith(".distrib.requestTimes"))
Creates a GAUGE metric (default type) named
"solr_metrics_core_query_1minRate" using the 1minRate value from the selected
JSON object.
{code}
was:
The default Grafana dashboard doesn't expose graphs for monitoring query
performance. For instance, if I want to see QPS for a collection, that's not
shown in the default dashboard. Same for quantiles like p95 query latency.
After some digging, these metrics are available in the output from
{{/admin/metrics}} but are not exported by the exporter.
This PR proposes to enhance the default dashboard with a new Query Metrics
section with the following metrics:
* Distributed QPS per Collection (aggregated across all cores)
* Distributed QPS per Solr Node (aggregated across all base_url)
* QPS 1-min rate per core
* QPS 5-min rate per core
* Top-level Query latency p99, p95, p75
* Local (non-distrib) query count per core (this is important for determining
if there is unbalanced load)
* Local (non-distrib) query rate per core (1-min)
* Local (non-distrib) p95 per core
Also, the {{solr-exporter-config.xml}} uses {{jq}} queries to pull metrics from
the output from {{/admin/metrics}}. This file is huge and contains a bunch of
{{jq}} boilerplate. Moreover, I'm introducing another 15-20 metrics in this PR,
it only makes the file more verbose.
Thus, I'm also introducing support for jq templates so as to reduce
boilerplate, reduce syntax errors, and improve readability. For instance the
query metrics I'm adding to the config look like this:
{code}
$jq:core-query(1minRate, endswith(".distrib.requestTimes"))
$jq:core-query(5minRate, endswith(".distrib.requestTimes"))
{code}
Instead of duplicating the complicated {{jq}} query for each metric. The
templates are optional and only should be used if a given jq structure is
repeated 3 or more times. Otherwise, inlining the jq query is still supported.
> Default Grafana dashboard needs to expose graphs for monitoring query
> performance
>
[jira] [Updated] (SOLR-15059) Default Grafana dashboard needs to expose graphs for monitoring query performance
[
https://issues.apache.org/jira/browse/SOLR-15059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timothy Potter updated SOLR-15059:
--
Fix Version/s: master (9.0)
8.8
> Default Grafana dashboard needs to expose graphs for monitoring query
> performance
> -
>
> Key: SOLR-15059
> URL: https://issues.apache.org/jira/browse/SOLR-15059
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Grafana Dashboard, metrics
>Reporter: Timothy Potter
>Assignee: Timothy Potter
>Priority: Major
> Fix For: 8.8, master (9.0)
>
>
> The default Grafana dashboard doesn't expose graphs for monitoring query
> performance. For instance, if I want to see QPS for a collection, that's not
> shown in the default dashboard. Same for quantiles like p95 query latency.
> After some digging, these metrics are available in the output from
> {{/admin/metrics}} but are not exported by the exporter.
> This PR proposes to enhance the default dashboard with a new Query Metrics
> section with the following metrics:
> * Distributed QPS per Collection (aggregated across all cores)
> * Distributed QPS per Solr Node (aggregated across all base_url)
> * QPS 1-min rate per core
> * QPS 5-min rate per core
> * Top-level Query latency p99, p95, p75
> * Local (non-distrib) query count per core (this is important for determining
> if there is unbalanced load)
> * Local (non-distrib) query rate per core (1-min)
> * Local (non-distrib) p95 per core
> Also, the {{solr-exporter-config.xml}} uses {{jq}} queries to pull metrics
> from the output from {{/admin/metrics}}. This file is huge and contains a
> bunch of {{jq}} boilerplate. Moreover, I'm introducing another 15-20 metrics
> in this PR, it only makes the file more verbose.
> Thus, I'm also introducing support for jq templates so as to reduce
> boilerplate, reduce syntax errors, and improve readability. For instance the
> query metrics I'm adding to the config look like this:
> {code}
>
> $jq:core-query(1minRate, endswith(".distrib.requestTimes"))
>
>
> $jq:core-query(5minRate, endswith(".distrib.requestTimes"))
>
> {code}
> Instead of duplicating the complicated {{jq}} query for each metric. The
> templates are optional and only should be used if a given jq structure is
> repeated 3 or more times. Otherwise, inlining the jq query is still supported.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-15059) Default Grafana dashboard needs to expose graphs for monitoring query performance
Timothy Potter created SOLR-15059:
-
Summary: Default Grafana dashboard needs to expose graphs for
monitoring query performance
Key: SOLR-15059
URL: https://issues.apache.org/jira/browse/SOLR-15059
Project: Solr
Issue Type: Improvement
Security Level: Public (Default Security Level. Issues are Public)
Components: Grafana Dashboard, metrics
Reporter: Timothy Potter
Assignee: Timothy Potter
The default Grafana dashboard doesn't expose graphs for monitoring query
performance. For instance, if I want to see QPS for a collection, that's not
shown in the default dashboard. Same for quantiles like p95 query latency.
After some digging, these metrics are available in the output from
{{/admin/metrics}} but are not exported by the exporter.
This PR proposes to enhance the default dashboard with a new Query Metrics
section with the following metrics:
* Distributed QPS per Collection (aggregated across all cores)
* Distributed QPS per Solr Node (aggregated across all base_url)
* QPS 1-min rate per core
* QPS 5-min rate per core
* Top-level Query latency p99, p95, p75
* Local (non-distrib) query count per core (this is important for determining
if there is unbalanced load)
* Local (non-distrib) query rate per core (1-min)
* Local (non-distrib) p95 per core
Also, the {{solr-exporter-config.xml}} uses {{jq}} queries to pull metrics from
the output from {{/admin/metrics}}. This file is huge and contains a bunch of
{{jq}} boilerplate. Moreover, I'm introducing another 15-20 metrics in this PR,
it only makes the file more verbose.
Thus, I'm also introducing support for jq templates so as to reduce
boilerplate, reduce syntax errors, and improve readability. For instance the
query metrics I'm adding to the config look like this:
{code}
$jq:core-query(1minRate, endswith(".distrib.requestTimes"))
$jq:core-query(5minRate, endswith(".distrib.requestTimes"))
{code}
Instead of duplicating the complicated {{jq}} query for each metric. The
templates are optional and only should be used if a given jq structure is
repeated 3 or more times. Otherwise, inlining the jq query is still supported.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-15054) Reproducible test failure ConfigureRecoveryStrategyTest.testAlmostAllMethodsAreFinal
[ https://issues.apache.org/jira/browse/SOLR-15054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timothy Potter updated SOLR-15054: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Reproducible test failure > ConfigureRecoveryStrategyTest.testAlmostAllMethodsAreFinal > > > Key: SOLR-15054 > URL: https://issues.apache.org/jira/browse/SOLR-15054 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: Tests >Affects Versions: master (9.0) >Reporter: Erick Erickson >Assignee: Timothy Potter >Priority: Major > Fix For: 8.8, master (9.0) > > Attachments: Screen Shot 2020-12-16 at 2.59.57 PM.png > > Time Spent: 50m > Remaining Estimate: 0h > > Reproduce with: > ./gradlew test --tests > ConfigureRecoveryStrategyTest.testAlmostAllMethodsAreFinal > -Dtests.seed=6E36F4B900CC2D0B -Dtests.multiplier=3 -Dtests.slow=true > -Dtests.locale=ccp -Dtests.timezone=ART -Dtests.asserts=true > -Dtests.file.encoding=UTF-8 > Git bisect identifies SOLR-12182 as the commit that introduced this, pinging > [~thelabdude] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-15054) Reproducible test failure ConfigureRecoveryStrategyTest.testAlmostAllMethodsAreFinal
[ https://issues.apache.org/jira/browse/SOLR-15054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timothy Potter updated SOLR-15054: -- Status: Patch Available (was: Open) > Reproducible test failure > ConfigureRecoveryStrategyTest.testAlmostAllMethodsAreFinal > > > Key: SOLR-15054 > URL: https://issues.apache.org/jira/browse/SOLR-15054 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: Tests >Affects Versions: master (9.0) >Reporter: Erick Erickson >Assignee: Timothy Potter >Priority: Major > Attachments: Screen Shot 2020-12-16 at 2.59.57 PM.png > > Time Spent: 0.5h > Remaining Estimate: 0h > > Reproduce with: > ./gradlew test --tests > ConfigureRecoveryStrategyTest.testAlmostAllMethodsAreFinal > -Dtests.seed=6E36F4B900CC2D0B -Dtests.multiplier=3 -Dtests.slow=true > -Dtests.locale=ccp -Dtests.timezone=ART -Dtests.asserts=true > -Dtests.file.encoding=UTF-8 > Git bisect identifies SOLR-12182 as the commit that introduced this, pinging > [~thelabdude] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-15054) Reproducible test failure ConfigureRecoveryStrategyTest.testAlmostAllMethodsAreFinal
[ https://issues.apache.org/jira/browse/SOLR-15054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17250680#comment-17250680 ] Timothy Potter commented on SOLR-15054: --- Fails on 8x too ... so 2nd PR up for that ... > Reproducible test failure > ConfigureRecoveryStrategyTest.testAlmostAllMethodsAreFinal > > > Key: SOLR-15054 > URL: https://issues.apache.org/jira/browse/SOLR-15054 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: Tests >Affects Versions: master (9.0) >Reporter: Erick Erickson >Priority: Major > Attachments: Screen Shot 2020-12-16 at 2.59.57 PM.png > > Time Spent: 0.5h > Remaining Estimate: 0h > > Reproduce with: > ./gradlew test --tests > ConfigureRecoveryStrategyTest.testAlmostAllMethodsAreFinal > -Dtests.seed=6E36F4B900CC2D0B -Dtests.multiplier=3 -Dtests.slow=true > -Dtests.locale=ccp -Dtests.timezone=ART -Dtests.asserts=true > -Dtests.file.encoding=UTF-8 > Git bisect identifies SOLR-12182 as the commit that introduced this, pinging > [~thelabdude] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Assigned] (SOLR-15054) Reproducible test failure ConfigureRecoveryStrategyTest.testAlmostAllMethodsAreFinal
[ https://issues.apache.org/jira/browse/SOLR-15054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timothy Potter reassigned SOLR-15054: - Assignee: Timothy Potter > Reproducible test failure > ConfigureRecoveryStrategyTest.testAlmostAllMethodsAreFinal > > > Key: SOLR-15054 > URL: https://issues.apache.org/jira/browse/SOLR-15054 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: Tests >Affects Versions: master (9.0) >Reporter: Erick Erickson >Assignee: Timothy Potter >Priority: Major > Attachments: Screen Shot 2020-12-16 at 2.59.57 PM.png > > Time Spent: 0.5h > Remaining Estimate: 0h > > Reproduce with: > ./gradlew test --tests > ConfigureRecoveryStrategyTest.testAlmostAllMethodsAreFinal > -Dtests.seed=6E36F4B900CC2D0B -Dtests.multiplier=3 -Dtests.slow=true > -Dtests.locale=ccp -Dtests.timezone=ART -Dtests.asserts=true > -Dtests.file.encoding=UTF-8 > Git bisect identifies SOLR-12182 as the commit that introduced this, pinging > [~thelabdude] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-15054) Reproducible test failure ConfigureRecoveryStrategyTest.testAlmostAllMethodsAreFinal
[ https://issues.apache.org/jira/browse/SOLR-15054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timothy Potter updated SOLR-15054: -- Fix Version/s: master (9.0) 8.8 > Reproducible test failure > ConfigureRecoveryStrategyTest.testAlmostAllMethodsAreFinal > > > Key: SOLR-15054 > URL: https://issues.apache.org/jira/browse/SOLR-15054 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: Tests >Affects Versions: master (9.0) >Reporter: Erick Erickson >Assignee: Timothy Potter >Priority: Major > Fix For: 8.8, master (9.0) > > Attachments: Screen Shot 2020-12-16 at 2.59.57 PM.png > > Time Spent: 0.5h > Remaining Estimate: 0h > > Reproduce with: > ./gradlew test --tests > ConfigureRecoveryStrategyTest.testAlmostAllMethodsAreFinal > -Dtests.seed=6E36F4B900CC2D0B -Dtests.multiplier=3 -Dtests.slow=true > -Dtests.locale=ccp -Dtests.timezone=ART -Dtests.asserts=true > -Dtests.file.encoding=UTF-8 > Git bisect identifies SOLR-12182 as the commit that introduced this, pinging > [~thelabdude] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-15054) Reproducible test failure ConfigureRecoveryStrategyTest.testAlmostAllMethodsAreFinal
[ https://issues.apache.org/jira/browse/SOLR-15054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timothy Potter updated SOLR-15054: -- Attachment: Screen Shot 2020-12-16 at 2.59.57 PM.png > Reproducible test failure > ConfigureRecoveryStrategyTest.testAlmostAllMethodsAreFinal > > > Key: SOLR-15054 > URL: https://issues.apache.org/jira/browse/SOLR-15054 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: Tests >Affects Versions: master (9.0) >Reporter: Erick Erickson >Priority: Major > Attachments: Screen Shot 2020-12-16 at 2.59.57 PM.png > > > Reproduce with: > ./gradlew test --tests > ConfigureRecoveryStrategyTest.testAlmostAllMethodsAreFinal > -Dtests.seed=6E36F4B900CC2D0B -Dtests.multiplier=3 -Dtests.slow=true > -Dtests.locale=ccp -Dtests.timezone=ART -Dtests.asserts=true > -Dtests.file.encoding=UTF-8 > Git bisect identifies SOLR-12182 as the commit that introduced this, pinging > [~thelabdude] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-15054) Reproducible test failure ConfigureRecoveryStrategyTest.testAlmostAllMethodsAreFinal
[
https://issues.apache.org/jira/browse/SOLR-15054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17250662#comment-17250662
]
Timothy Potter edited comment on SOLR-15054 at 12/16/20, 10:00 PM:
---
I think it was Mike's changes in this merge actually ->
https://github.com/apache/lucene-solr/pull/2120/files#diff-838f63a8d7b764661b66c758c19cfe6f3d6454b328a1b5f52f75480530b0cfe7R428
... Unless I'm missing something there [~erickerickson], where did you see it
was my changes for SOLR-12182?
Regardless, I can fix it up no problem ... I don't think the test logic should
require {{private}} methods to be {{final}}. In fact, my IDE actually marks
that as unnecessary ;-) So I think the right fix here is to change the test as
follows:
{code}
diff --git
a/solr/core/src/test/org/apache/solr/core/ConfigureRecoveryStrategyTest.java
b/solr/core/src/test/org/apache/solr/core/ConfigureRecoveryStrategyTest.java
index 0a988f6e747..246ed981137 100644
--- a/solr/core/src/test/org/apache/solr/core/ConfigureRecoveryStrategyTest.java
+++ b/solr/core/src/test/org/apache/solr/core/ConfigureRecoveryStrategyTest.java
@@ -66,7 +66,7 @@ public class ConfigureRecoveryStrategyTest extends
SolrTestCaseJ4 {
public void testAlmostAllMethodsAreFinal() throws Exception {
for (Method m : RecoveryStrategy.class.getDeclaredMethods()) {
- if (Modifier.isStatic(m.getModifiers())) continue;
+ if (Modifier.isStatic(m.getModifiers()) ||
Modifier.isPrivate(m.getModifiers())) continue;
final String methodName = m.getName();
if ("getReplicateLeaderUrl".equals(methodName)) {
assertFalse(m.toString(), Modifier.isFinal(m.getModifiers()));
{code}
was (Author: thelabdude):
I think it was Mike's changes in this merge actually -> ... Unless I'm missing
something there [~erickerickson]. Regardless, I don't think the test logic
should require {{private}} methods to be {{final}}. In fact, my IDE actually
marks that as unnecessary ;-) So I think the right fix here is to change the
test as follows:
{code}
diff --git
a/solr/core/src/test/org/apache/solr/core/ConfigureRecoveryStrategyTest.java
b/solr/core/src/test/org/apache/solr/core/ConfigureRecoveryStrategyTest.java
index 0a988f6e747..246ed981137 100644
--- a/solr/core/src/test/org/apache/solr/core/ConfigureRecoveryStrategyTest.java
+++ b/solr/core/src/test/org/apache/solr/core/ConfigureRecoveryStrategyTest.java
@@ -66,7 +66,7 @@ public class ConfigureRecoveryStrategyTest extends
SolrTestCaseJ4 {
public void testAlmostAllMethodsAreFinal() throws Exception {
for (Method m : RecoveryStrategy.class.getDeclaredMethods()) {
- if (Modifier.isStatic(m.getModifiers())) continue;
+ if (Modifier.isStatic(m.getModifiers()) ||
Modifier.isPrivate(m.getModifiers())) continue;
final String methodName = m.getName();
if ("getReplicateLeaderUrl".equals(methodName)) {
assertFalse(m.toString(), Modifier.isFinal(m.getModifiers()));
{code}
> Reproducible test failure
> ConfigureRecoveryStrategyTest.testAlmostAllMethodsAreFinal
>
>
> Key: SOLR-15054
> URL: https://issues.apache.org/jira/browse/SOLR-15054
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Tests
>Affects Versions: master (9.0)
>Reporter: Erick Erickson
>Priority: Major
> Attachments: Screen Shot 2020-12-16 at 2.59.57 PM.png
>
>
> Reproduce with:
> ./gradlew test --tests
> ConfigureRecoveryStrategyTest.testAlmostAllMethodsAreFinal
> -Dtests.seed=6E36F4B900CC2D0B -Dtests.multiplier=3 -Dtests.slow=true
> -Dtests.locale=ccp -Dtests.timezone=ART -Dtests.asserts=true
> -Dtests.file.encoding=UTF-8
> Git bisect identifies SOLR-12182 as the commit that introduced this, pinging
> [~thelabdude]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-15054) Reproducible test failure ConfigureRecoveryStrategyTest.testAlmostAllMethodsAreFinal
[
https://issues.apache.org/jira/browse/SOLR-15054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17250662#comment-17250662
]
Timothy Potter commented on SOLR-15054:
---
I think it was Mike's changes in this merge actually -> ... Unless I'm missing
something there [~erickerickson]. Regardless, I don't think the test logic
should require {{private}} methods to be {{final}}. In fact, my IDE actually
marks that as unnecessary ;-) So I think the right fix here is to change the
test as follows:
{code}
diff --git
a/solr/core/src/test/org/apache/solr/core/ConfigureRecoveryStrategyTest.java
b/solr/core/src/test/org/apache/solr/core/ConfigureRecoveryStrategyTest.java
index 0a988f6e747..246ed981137 100644
--- a/solr/core/src/test/org/apache/solr/core/ConfigureRecoveryStrategyTest.java
+++ b/solr/core/src/test/org/apache/solr/core/ConfigureRecoveryStrategyTest.java
@@ -66,7 +66,7 @@ public class ConfigureRecoveryStrategyTest extends
SolrTestCaseJ4 {
public void testAlmostAllMethodsAreFinal() throws Exception {
for (Method m : RecoveryStrategy.class.getDeclaredMethods()) {
- if (Modifier.isStatic(m.getModifiers())) continue;
+ if (Modifier.isStatic(m.getModifiers()) ||
Modifier.isPrivate(m.getModifiers())) continue;
final String methodName = m.getName();
if ("getReplicateLeaderUrl".equals(methodName)) {
assertFalse(m.toString(), Modifier.isFinal(m.getModifiers()));
{code}
> Reproducible test failure
> ConfigureRecoveryStrategyTest.testAlmostAllMethodsAreFinal
>
>
> Key: SOLR-15054
> URL: https://issues.apache.org/jira/browse/SOLR-15054
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Tests
>Affects Versions: master (9.0)
>Reporter: Erick Erickson
>Priority: Major
>
> Reproduce with:
> ./gradlew test --tests
> ConfigureRecoveryStrategyTest.testAlmostAllMethodsAreFinal
> -Dtests.seed=6E36F4B900CC2D0B -Dtests.multiplier=3 -Dtests.slow=true
> -Dtests.locale=ccp -Dtests.timezone=ART -Dtests.asserts=true
> -Dtests.file.encoding=UTF-8
> Git bisect identifies SOLR-12182 as the commit that introduced this, pinging
> [~thelabdude]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-15054) Reproducible test failure ConfigureRecoveryStrategyTest.testAlmostAllMethodsAreFinal
[ https://issues.apache.org/jira/browse/SOLR-15054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17250644#comment-17250644 ] Timothy Potter commented on SOLR-15054: --- taking a look, thanks [~erickerickson] > Reproducible test failure > ConfigureRecoveryStrategyTest.testAlmostAllMethodsAreFinal > > > Key: SOLR-15054 > URL: https://issues.apache.org/jira/browse/SOLR-15054 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: Tests >Affects Versions: master (9.0) >Reporter: Erick Erickson >Priority: Major > > Reproduce with: > ./gradlew test --tests > ConfigureRecoveryStrategyTest.testAlmostAllMethodsAreFinal > -Dtests.seed=6E36F4B900CC2D0B -Dtests.multiplier=3 -Dtests.slow=true > -Dtests.locale=ccp -Dtests.timezone=ART -Dtests.asserts=true > -Dtests.file.encoding=UTF-8 > Git bisect identifies SOLR-12182 as the commit that introduced this, pinging > [~thelabdude] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-15036) Use plist automatically for executing a facet expression against a collection alias backed by multiple collections
[
https://issues.apache.org/jira/browse/SOLR-15036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17249765#comment-17249765
]
Timothy Potter commented on SOLR-15036:
---
Confirmed, it works nicely now! Thanks for your help Joel
{code}
{count(*)=6, a_i=0, max(max(a_d))=2.2515625018914305,
min(min(a_d))=-0.5859583807765252, sum(sum(a_d))=5.894460990302006,
wsum(avg(a_d), count(*))=0.9824101650503342}
{count(*)=4, a_i=1, max(max(a_d))=3.338305310115201,
min(min(a_d))=0.03050220236482515, sum(sum(a_d))=12.517492417715335,
wsum(avg(a_d), count(*))=2.086248736285889}
{count(*)=4, a_i=2, max(max(a_d))=4.832815828279073,
min(min(a_d))=3.16905458918893, sum(sum(a_d))=24.076139429000165,
wsum(avg(a_d), count(*))=4.012689904833361}
{count(*)=4, a_i=3, max(max(a_d))=5.66831997419713,
min(min(a_d))=2.902262184046103, sum(sum(a_d))=22.58303980377591,
wsum(avg(a_d), count(*))=3.763839967295984}
{count(*)=4, a_i=4, max(max(a_d))=6.531585917691583,
min(min(a_d))=2.6395698661907963, sum(sum(a_d))=28.243748570490624,
wsum(avg(a_d), count(*))=4.707291428415103}
{count(*)=5, a_i=5, max(max(a_d))=7.555382540979672,
min(min(a_d))=4.808772939476107, sum(sum(a_d))=37.88196903407075,
wsum(avg(a_d), count(*))=6.313661505678459}
{count(*)=5, a_i=6, max(max(a_d))=8.416136012729918,
min(min(a_d))=5.422492404700898, sum(sum(a_d))=39.25679972070782,
wsum(avg(a_d), count(*))=6.542799953451303}
{count(*)=5, a_i=7, max(max(a_d))=8.667999236934058,
min(min(a_d))=6.934577412906803, sum(sum(a_d))=46.7622185952807, wsum(avg(a_d),
count(*))=7.793703099213451}
{count(*)=5, a_i=8, max(max(a_d))=9.566181963643201,
min(min(a_d))=7.4397380388592556, sum(sum(a_d))=53.296172957938325,
wsum(avg(a_d), count(*))=8.88269549298972}
{count(*)=4, a_i=9, max(max(a_d))=12.251349466753346,
min(min(a_d))=9.232427215193514, sum(sum(a_d))=63.46244550204135,
wsum(avg(a_d), count(*))=10.577074250340223}
{code}
> Use plist automatically for executing a facet expression against a collection
> alias backed by multiple collections
> --
>
> Key: SOLR-15036
> URL: https://issues.apache.org/jira/browse/SOLR-15036
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: streaming expressions
>Reporter: Timothy Potter
>Assignee: Timothy Potter
>Priority: Major
> Attachments: relay-approach.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> For analytics use cases, streaming expressions make it possible to compute
> basic aggregations (count, min, max, sum, and avg) over massive data sets.
> Moreover, with massive data sets, it is common to use collection aliases over
> many underlying collections, for instance time-partitioned aliases backed by
> a set of collections, each covering a specific time range. In some cases, we
> can end up with many collections (think 50-60) each with 100's of shards.
> Aliases help insulate client applications from complex collection topologies
> on the server side.
> Let's take a basic facet expression that computes some useful aggregation
> metrics:
> {code:java}
> facet(
> some_alias,
> q="*:*",
> fl="a_i",
> sort="a_i asc",
> buckets="a_i",
> bucketSorts="count(*) asc",
> bucketSizeLimit=1,
> sum(a_d), avg(a_d), min(a_d), max(a_d), count(*)
> )
> {code}
> Behind the scenes, the {{FacetStream}} sends a JSON facet request to Solr
> which then expands the alias to a list of collections. For each collection,
> the top-level distributed query controller gathers a candidate set of
> replicas to query and then scatters {{distrib=false}} queries to each replica
> in the list. For instance, if we have 60 collections with 200 shards each,
> then this results in 12,000 shard requests from the query controller node to
> the other nodes in the cluster. The requests are sent in an async manner (see
> {{SearchHandler}} and {{HttpShardHandler}}) In my testing, we’ve seen cases
> where we hit 18,000 replicas and these queries don’t always come back in a
> timely manner. Put simply, this also puts a lot of load on the top-level
> query controller node in terms of open connections and new object creation.
> Instead, we can use {{plist}} to send the JSON facet query to each collection
> in the alias in parallel, which reduces the overhead of each top-level
> distributed query from 12,000 to 200 in my example above. With this approach,
> you’ll then need to sort the tuples back from each collection and do a
> rollup, something like:
> {code:java}
> select(
> rollup(
> sort(
> plist(
> select(facet(coll1,q="*:*", fl="a_i", sort="a_i asc", buckets="a_i",
> bucketSorts="count(*) asc",
[jira] [Commented] (SOLR-15036) Use plist automatically for executing a facet expression against a collection alias backed by multiple collections
[
https://issues.apache.org/jira/browse/SOLR-15036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17249747#comment-17249747
]
Timothy Potter commented on SOLR-15036:
---
Oh darn, I should have spotted that! Sorry for the noise there [~jbernste] ...
seems like we could improve the error handling in the drill code to barf if the
user is trying to compute metrics for fields not in the fl? That could help
with silly mistakes like this ...
> Use plist automatically for executing a facet expression against a collection
> alias backed by multiple collections
> --
>
> Key: SOLR-15036
> URL: https://issues.apache.org/jira/browse/SOLR-15036
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: streaming expressions
>Reporter: Timothy Potter
>Assignee: Timothy Potter
>Priority: Major
> Attachments: relay-approach.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> For analytics use cases, streaming expressions make it possible to compute
> basic aggregations (count, min, max, sum, and avg) over massive data sets.
> Moreover, with massive data sets, it is common to use collection aliases over
> many underlying collections, for instance time-partitioned aliases backed by
> a set of collections, each covering a specific time range. In some cases, we
> can end up with many collections (think 50-60) each with 100's of shards.
> Aliases help insulate client applications from complex collection topologies
> on the server side.
> Let's take a basic facet expression that computes some useful aggregation
> metrics:
> {code:java}
> facet(
> some_alias,
> q="*:*",
> fl="a_i",
> sort="a_i asc",
> buckets="a_i",
> bucketSorts="count(*) asc",
> bucketSizeLimit=1,
> sum(a_d), avg(a_d), min(a_d), max(a_d), count(*)
> )
> {code}
> Behind the scenes, the {{FacetStream}} sends a JSON facet request to Solr
> which then expands the alias to a list of collections. For each collection,
> the top-level distributed query controller gathers a candidate set of
> replicas to query and then scatters {{distrib=false}} queries to each replica
> in the list. For instance, if we have 60 collections with 200 shards each,
> then this results in 12,000 shard requests from the query controller node to
> the other nodes in the cluster. The requests are sent in an async manner (see
> {{SearchHandler}} and {{HttpShardHandler}}) In my testing, we’ve seen cases
> where we hit 18,000 replicas and these queries don’t always come back in a
> timely manner. Put simply, this also puts a lot of load on the top-level
> query controller node in terms of open connections and new object creation.
> Instead, we can use {{plist}} to send the JSON facet query to each collection
> in the alias in parallel, which reduces the overhead of each top-level
> distributed query from 12,000 to 200 in my example above. With this approach,
> you’ll then need to sort the tuples back from each collection and do a
> rollup, something like:
> {code:java}
> select(
> rollup(
> sort(
> plist(
> select(facet(coll1,q="*:*", fl="a_i", sort="a_i asc", buckets="a_i",
> bucketSorts="count(*) asc", bucketSizeLimit=1, sum(a_d), avg(a_d),
> min(a_d), max(a_d), count(*)),a_i,sum(a_d) as the_sum, avg(a_d) as the_avg,
> min(a_d) as the_min, max(a_d) as the_max, count(*) as cnt),
> select(facet(coll2,q="*:*", fl="a_i", sort="a_i asc", buckets="a_i",
> bucketSorts="count(*) asc", bucketSizeLimit=1, sum(a_d), avg(a_d),
> min(a_d), max(a_d), count(*)),a_i,sum(a_d) as the_sum, avg(a_d) as the_avg,
> min(a_d) as the_min, max(a_d) as the_max, count(*) as cnt)
> ),
> by="a_i asc"
> ),
> over="a_i",
> sum(the_sum), avg(the_avg), min(the_min), max(the_max), sum(cnt)
> ),
> a_i, sum(the_sum) as the_sum, avg(the_avg) as the_avg, min(the_min) as
> the_min, max(the_max) as the_max, sum(cnt) as cnt
> )
> {code}
> One thing to point out is that you can’t just avg. the averages back from
> each collection in the rollup. It needs to be a *weighted avg.* when rolling
> up the avg. from each facet expression in the plist. However, we have the
> count per collection, so this is doable but will require some changes to the
> rollup expression to support weighted average.
> While this plist approach is doable, it’s a pain for users to have to create
> the rollup / sort over plist expression for collection aliases. After all,
> aliases are supposed to hide these types of complexities from client
> applications!
> The point of this ticket is to investigate the feasibility of auto-wrapping
> the facet expression with a rollup
[jira] [Commented] (SOLR-15036) Use plist automatically for executing a facet expression against a collection alias backed by multiple collections
[
https://issues.apache.org/jira/browse/SOLR-15036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17249387#comment-17249387
]
Timothy Potter commented on SOLR-15036:
---
[~jbernste] Does drill only support counting per dimension?
For my test case, I'm expecting these tuples:
{code}
{sum(a_d)=5.894460990302005, a_i=0, count(*)=6, max(a_d)=2.2515625018914305,
avg(a_d)=0.9824101650503341, min(a_d)=-0.5859583807765252}
{sum(a_d)=12.517492417715335, a_i=1, count(*)=6, max(a_d)=3.338305310115201,
avg(a_d)=2.086248736285889, min(a_d)=0.03050220236482515}
{sum(a_d)=24.076139429000165, a_i=2, count(*)=6, max(a_d)=4.832815828279073,
avg(a_d)=4.01268990483336, min(a_d)=3.16905458918893}
{sum(a_d)=22.583039803775907, a_i=3, count(*)=6, max(a_d)=5.66831997419713,
avg(a_d)=3.7638399672959846, min(a_d)=2.902262184046103}
{sum(a_d)=28.243748570490624, a_i=4, count(*)=6, max(a_d)=6.531585917691583,
avg(a_d)=4.707291428415104, min(a_d)=2.6395698661907963}
{sum(a_d)=37.88196903407075, a_i=5, count(*)=6, max(a_d)=7.555382540979672,
avg(a_d)=6.313661505678458, min(a_d)=4.808772939476107}
{sum(a_d)=39.25679972070782, a_i=6, count(*)=6, max(a_d)=8.416136012729918,
avg(a_d)=6.542799953451302, min(a_d)=5.422492404700898}
{sum(a_d)=46.7622185952807, a_i=7, count(*)=6, max(a_d)=8.667999236934058,
avg(a_d)=7.793703099213451, min(a_d)=6.934577412906803}
{sum(a_d)=53.296172957938325, a_i=8, count(*)=6, max(a_d)=9.566181963643201,
avg(a_d)=8.88269549298972, min(a_d)=7.4397380388592556}
{sum(a_d)=63.46244550204135, a_i=9, count(*)=6, max(a_d)=12.251349466753346,
avg(a_d)=10.577074250340225, min(a_d)=9.232427215193514}
{code}
Which is what is produced by the facet expression in my description for this
JIRA.
But {{drill}} doesn't seem like it's aggregating by the {{a_i}} dimension as
expected. The following drill expression (w/o rollup for now) produces the
following tuples for the same setup ^
{code}
drill(
SOME_ALIAS_WITH_MANY_COLLS,
q="*:*",
fl="a_i",
sort="a_i asc",
rollup(
input(),
over="a_i",
sum(a_d), avg(a_d), min(a_d), max(a_d), count(*)
)
)
{code}
Produces Tuple stream:
{code}
{sum(a_d)=0.0, count(*)=1, a_i=0, avg(a_d)=0.0,
max(a_d)=-1.7976931348623157E308, min(a_d)=1.7976931348623157E308}
{sum(a_d)=0.0, count(*)=3, a_i=0, avg(a_d)=0.0,
max(a_d)=-1.7976931348623157E308, min(a_d)=1.7976931348623157E308}
{sum(a_d)=0.0, count(*)=1, a_i=0, avg(a_d)=0.0,
max(a_d)=-1.7976931348623157E308, min(a_d)=1.7976931348623157E308}
{sum(a_d)=0.0, count(*)=1, a_i=0, avg(a_d)=0.0,
max(a_d)=-1.7976931348623157E308, min(a_d)=1.7976931348623157E308}
{sum(a_d)=0.0, count(*)=2, a_i=1, avg(a_d)=0.0,
max(a_d)=-1.7976931348623157E308, min(a_d)=1.7976931348623157E308}
{sum(a_d)=0.0, count(*)=1, a_i=1, avg(a_d)=0.0,
max(a_d)=-1.7976931348623157E308, min(a_d)=1.7976931348623157E308}
{sum(a_d)=0.0, count(*)=2, a_i=1, avg(a_d)=0.0,
max(a_d)=-1.7976931348623157E308, min(a_d)=1.7976931348623157E308}
{sum(a_d)=0.0, count(*)=1, a_i=1, avg(a_d)=0.0,
max(a_d)=-1.7976931348623157E308, min(a_d)=1.7976931348623157E308}
{sum(a_d)=0.0, count(*)=1, a_i=2, avg(a_d)=0.0,
max(a_d)=-1.7976931348623157E308, min(a_d)=1.7976931348623157E308}
{sum(a_d)=0.0, count(*)=2, a_i=2, avg(a_d)=0.0,
max(a_d)=-1.7976931348623157E308, min(a_d)=1.7976931348623157E308}
{sum(a_d)=0.0, count(*)=2, a_i=2, avg(a_d)=0.0,
max(a_d)=-1.7976931348623157E308, min(a_d)=1.7976931348623157E308}
{sum(a_d)=0.0, count(*)=1, a_i=2, avg(a_d)=0.0,
max(a_d)=-1.7976931348623157E308, min(a_d)=1.7976931348623157E308}
{sum(a_d)=0.0, count(*)=1, a_i=3, avg(a_d)=0.0,
max(a_d)=-1.7976931348623157E308, min(a_d)=1.7976931348623157E308}
{sum(a_d)=0.0, count(*)=1, a_i=3, avg(a_d)=0.0,
max(a_d)=-1.7976931348623157E308, min(a_d)=1.7976931348623157E308}
{sum(a_d)=0.0, count(*)=1, a_i=3, avg(a_d)=0.0,
max(a_d)=-1.7976931348623157E308, min(a_d)=1.7976931348623157E308}
{sum(a_d)=0.0, count(*)=2, a_i=3, avg(a_d)=0.0,
max(a_d)=-1.7976931348623157E308, min(a_d)=1.7976931348623157E308}
{sum(a_d)=0.0, count(*)=1, a_i=3, avg(a_d)=0.0,
max(a_d)=-1.7976931348623157E308, min(a_d)=1.7976931348623157E308}
{sum(a_d)=0.0, count(*)=1, a_i=4, avg(a_d)=0.0,
max(a_d)=-1.7976931348623157E308, min(a_d)=1.7976931348623157E308}
{sum(a_d)=0.0, count(*)=1, a_i=4, avg(a_d)=0.0,
max(a_d)=-1.7976931348623157E308, min(a_d)=1.7976931348623157E308}
{sum(a_d)=0.0, count(*)=1, a_i=4, avg(a_d)=0.0,
max(a_d)=-1.7976931348623157E308, min(a_d)=1.7976931348623157E308}
{sum(a_d)=0.0, count(*)=1, a_i=4, avg(a_d)=0.0,
max(a_d)=-1.7976931348623157E308, min(a_d)=1.7976931348623157E308}
{sum(a_d)=0.0, count(*)=1, a_i=4, avg(a_d)=0.0,
max(a_d)=-1.7976931348623157E308, min(a_d)=1.7976931348623157E308}
{sum(a_d)=0.0, count(*)=1, a_i=4, avg(a_d)=0.0,
max(a_d)=-1.7976931348623157E308, min(a_d)=1.7976931348623157E308}
{sum(a_d)=0.0, count(*)=2, a_i=5, avg(a_d)=0.0,
max(a_d)=-1.7976931348623157E308, min(a_d)=1.7976931348623157E308}
[jira] [Resolved] (SOLR-15046) bin/solr check to see if solr.jetty.https.port should be set always returns true
[
https://issues.apache.org/jira/browse/SOLR-15046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timothy Potter resolved SOLR-15046.
---
Fix Version/s: 8.8
Resolution: Fixed
> bin/solr check to see if solr.jetty.https.port should be set always returns
> true
>
>
> Key: SOLR-15046
> URL: https://issues.apache.org/jira/browse/SOLR-15046
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: scripts and tools
>Affects Versions: 8.8
>Reporter: Timothy Potter
>Assignee: Timothy Potter
>Priority: Minor
> Fix For: 8.8
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> This always returns {{true}} on Mac OS X (not sure about Linux):
> {code}
> if [ "$SOLR_SSL_ENABLED" ]; then
> # If using SSL and solr.jetty.https.port not set explicitly, use the
> jetty.port
> SSL_PORT_PROP="-Dsolr.jetty.https.port=$SOLR_PORT"
> SOLR_OPTS+=($SOLR_SSL_OPTS "$SSL_PORT_PROP")
> fi
> {code}
> On master, we do {{if [ "$SOLR_SSL_ENABLED" == "true" ]; then}}
> The problem is that ZkContainer (when zkRun enabled) looks to see if
> {{solr.jetty.https.port}} is set, and if so, then it assumes TLS is enabled.
> So this little BASH bug causes Solr's HttpShardHandler to connect via https.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-15036) Use plist automatically for executing a facet expression against a collection alias backed by multiple collections
[
https://issues.apache.org/jira/browse/SOLR-15036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17249142#comment-17249142
]
Timothy Potter commented on SOLR-15036:
---
Thank you! I read that in the docs but it didn't quite sink in I guess.
> Use plist automatically for executing a facet expression against a collection
> alias backed by multiple collections
> --
>
> Key: SOLR-15036
> URL: https://issues.apache.org/jira/browse/SOLR-15036
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: streaming expressions
>Reporter: Timothy Potter
>Assignee: Timothy Potter
>Priority: Major
> Attachments: relay-approach.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> For analytics use cases, streaming expressions make it possible to compute
> basic aggregations (count, min, max, sum, and avg) over massive data sets.
> Moreover, with massive data sets, it is common to use collection aliases over
> many underlying collections, for instance time-partitioned aliases backed by
> a set of collections, each covering a specific time range. In some cases, we
> can end up with many collections (think 50-60) each with 100's of shards.
> Aliases help insulate client applications from complex collection topologies
> on the server side.
> Let's take a basic facet expression that computes some useful aggregation
> metrics:
> {code:java}
> facet(
> some_alias,
> q="*:*",
> fl="a_i",
> sort="a_i asc",
> buckets="a_i",
> bucketSorts="count(*) asc",
> bucketSizeLimit=1,
> sum(a_d), avg(a_d), min(a_d), max(a_d), count(*)
> )
> {code}
> Behind the scenes, the {{FacetStream}} sends a JSON facet request to Solr
> which then expands the alias to a list of collections. For each collection,
> the top-level distributed query controller gathers a candidate set of
> replicas to query and then scatters {{distrib=false}} queries to each replica
> in the list. For instance, if we have 60 collections with 200 shards each,
> then this results in 12,000 shard requests from the query controller node to
> the other nodes in the cluster. The requests are sent in an async manner (see
> {{SearchHandler}} and {{HttpShardHandler}}) In my testing, we’ve seen cases
> where we hit 18,000 replicas and these queries don’t always come back in a
> timely manner. Put simply, this also puts a lot of load on the top-level
> query controller node in terms of open connections and new object creation.
> Instead, we can use {{plist}} to send the JSON facet query to each collection
> in the alias in parallel, which reduces the overhead of each top-level
> distributed query from 12,000 to 200 in my example above. With this approach,
> you’ll then need to sort the tuples back from each collection and do a
> rollup, something like:
> {code:java}
> select(
> rollup(
> sort(
> plist(
> select(facet(coll1,q="*:*", fl="a_i", sort="a_i asc", buckets="a_i",
> bucketSorts="count(*) asc", bucketSizeLimit=1, sum(a_d), avg(a_d),
> min(a_d), max(a_d), count(*)),a_i,sum(a_d) as the_sum, avg(a_d) as the_avg,
> min(a_d) as the_min, max(a_d) as the_max, count(*) as cnt),
> select(facet(coll2,q="*:*", fl="a_i", sort="a_i asc", buckets="a_i",
> bucketSorts="count(*) asc", bucketSizeLimit=1, sum(a_d), avg(a_d),
> min(a_d), max(a_d), count(*)),a_i,sum(a_d) as the_sum, avg(a_d) as the_avg,
> min(a_d) as the_min, max(a_d) as the_max, count(*) as cnt)
> ),
> by="a_i asc"
> ),
> over="a_i",
> sum(the_sum), avg(the_avg), min(the_min), max(the_max), sum(cnt)
> ),
> a_i, sum(the_sum) as the_sum, avg(the_avg) as the_avg, min(the_min) as
> the_min, max(the_max) as the_max, sum(cnt) as cnt
> )
> {code}
> One thing to point out is that you can’t just avg. the averages back from
> each collection in the rollup. It needs to be a *weighted avg.* when rolling
> up the avg. from each facet expression in the plist. However, we have the
> count per collection, so this is doable but will require some changes to the
> rollup expression to support weighted average.
> While this plist approach is doable, it’s a pain for users to have to create
> the rollup / sort over plist expression for collection aliases. After all,
> aliases are supposed to hide these types of complexities from client
> applications!
> The point of this ticket is to investigate the feasibility of auto-wrapping
> the facet expression with a rollup / sort / plist when the collection
> argument is an alias with multiple collections; other stream sources will be
> considered after facet is proven out.
> Lastly, I also considered an alternative
[jira] [Commented] (SOLR-15026) MiniSolrCloudCluster can inconsistently get confused about when it's using SSL
[
https://issues.apache.org/jira/browse/SOLR-15026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17249140#comment-17249140
]
Timothy Potter commented on SOLR-15026:
---
See: SOLR-15046
> MiniSolrCloudCluster can inconsistently get confused about when it's using SSL
> --
>
> Key: SOLR-15026
> URL: https://issues.apache.org/jira/browse/SOLR-15026
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Chris M. Hostetter
>Priority: Major
>
> A new test added in SOLR-14934 caused the following reproducible failure to
> pop up on jenkins...
> {noformat}
> hossman@slate:~/lucene/dev [j11] [master] $ ./gradlew -p solr/test-framework/
> test --tests MiniSolrCloudClusterTest.testSolrHomeAndResourceLoaders
> -Dtests.seed=806A85748BD81F48 -Dtests.multiplier=2 -Dtests.slow=true
> -Dtests.locale=ln-CG -Dtests.timezone=Asia/Thimbu -Dtests.asserts=true
> -Dtests.file.encoding=UTF-8
> Starting a Gradle Daemon (subsequent builds will be faster)
> > Task :randomizationInfo
> Running tests with randomization seed: tests.seed=806A85748BD81F48
> > Task :solr:test-framework:test
> org.apache.solr.cloud.MiniSolrCloudClusterTest >
> testSolrHomeAndResourceLoaders FAILED
> org.apache.solr.client.solrj.SolrServerException: IOException occurred
> when talking to server at: https://127.0.0.1:38681/solr
> at
> __randomizedtesting.SeedInfo.seed([806A85748BD81F48:37548FA7602CB5FD]:0)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:712)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:269)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:251)
> at
> org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:390)
> at
> org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:360)
> at
> org.apache.solr.client.solrj.impl.BaseCloudSolrClient.sendRequest(BaseCloudSolrClient.java:1168)
> at
> org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:931)
> at
> org.apache.solr.client.solrj.impl.BaseCloudSolrClient.request(BaseCloudSolrClient.java:865)
> at
> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:229)
> at
> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:246)
> at
> org.apache.solr.cloud.MiniSolrCloudClusterTest.testSolrHomeAndResourceLoaders(MiniSolrCloudClusterTest.java:125)
> ...
> Caused by:
> javax.net.ssl.SSLException: Unsupported or unrecognized SSL message
> at
> java.base/sun.security.ssl.SSLSocketInputRecord.handleUnknownRecord(SSLSocketInputRecord.java:439)
> {noformat}
> The problem sems to be that even though the MiniSolrCloudCluster being
> instantiated isn't _intentionally_ using any SSL randomization (it just uses
> {{JettyConfig.builder().build()}} the CloudSolrClient returned by
> {{cluster.getSolrClient()}} is evidnetly picking up the ranodmized SSL and
> trying to use it to talk to the cluster.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-15046) bin/solr check to see if solr.jetty.https.port should be set always returns true
[
https://issues.apache.org/jira/browse/SOLR-15046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17249139#comment-17249139
]
Timothy Potter commented on SOLR-15046:
---
This problem was surfaced by the changes introduced in SOLR-12182, but the bug
is actually in the {{bin/solr}} script.
> bin/solr check to see if solr.jetty.https.port should be set always returns
> true
>
>
> Key: SOLR-15046
> URL: https://issues.apache.org/jira/browse/SOLR-15046
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: scripts and tools
>Affects Versions: 8.8
>Reporter: Timothy Potter
>Assignee: Timothy Potter
>Priority: Minor
>
> This always returns {{true}} on Mac OS X (not sure about Linux):
> {code}
> if [ "$SOLR_SSL_ENABLED" ]; then
> # If using SSL and solr.jetty.https.port not set explicitly, use the
> jetty.port
> SSL_PORT_PROP="-Dsolr.jetty.https.port=$SOLR_PORT"
> SOLR_OPTS+=($SOLR_SSL_OPTS "$SSL_PORT_PROP")
> fi
> {code}
> On master, we do {{ if [ "$SOLR_SSL_ENABLED" == "true" ]; then}}
> The problem is that ZkContainer (when zkRun enabled) looks to see if
> {{solr.jetty.https.port}} is set, and if so, then it assumes TLS is enabled.
> So this little BASH bug causes Solr's HttpShardHandler to connect via https.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-15046) bin/solr check to see if solr.jetty.https.port should be set always returns true
[
https://issues.apache.org/jira/browse/SOLR-15046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timothy Potter updated SOLR-15046:
--
Description:
This always returns {{true}} on Mac OS X (not sure about Linux):
{code}
if [ "$SOLR_SSL_ENABLED" ]; then
# If using SSL and solr.jetty.https.port not set explicitly, use the
jetty.port
SSL_PORT_PROP="-Dsolr.jetty.https.port=$SOLR_PORT"
SOLR_OPTS+=($SOLR_SSL_OPTS "$SSL_PORT_PROP")
fi
{code}
On master, we do {{if [ "$SOLR_SSL_ENABLED" == "true" ]; then}}
The problem is that ZkContainer (when zkRun enabled) looks to see if
{{solr.jetty.https.port}} is set, and if so, then it assumes TLS is enabled. So
this little BASH bug causes Solr's HttpShardHandler to connect via https.
was:
This always returns {{true}} on Mac OS X (not sure about Linux):
{code}
if [ "$SOLR_SSL_ENABLED" ]; then
# If using SSL and solr.jetty.https.port not set explicitly, use the
jetty.port
SSL_PORT_PROP="-Dsolr.jetty.https.port=$SOLR_PORT"
SOLR_OPTS+=($SOLR_SSL_OPTS "$SSL_PORT_PROP")
fi
{code}
On master, we do {{ if [ "$SOLR_SSL_ENABLED" == "true" ]; then}}
The problem is that ZkContainer (when zkRun enabled) looks to see if
{{solr.jetty.https.port}} is set, and if so, then it assumes TLS is enabled. So
this little BASH bug causes Solr's HttpShardHandler to connect via https.
> bin/solr check to see if solr.jetty.https.port should be set always returns
> true
>
>
> Key: SOLR-15046
> URL: https://issues.apache.org/jira/browse/SOLR-15046
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: scripts and tools
>Affects Versions: 8.8
>Reporter: Timothy Potter
>Assignee: Timothy Potter
>Priority: Minor
>
> This always returns {{true}} on Mac OS X (not sure about Linux):
> {code}
> if [ "$SOLR_SSL_ENABLED" ]; then
> # If using SSL and solr.jetty.https.port not set explicitly, use the
> jetty.port
> SSL_PORT_PROP="-Dsolr.jetty.https.port=$SOLR_PORT"
> SOLR_OPTS+=($SOLR_SSL_OPTS "$SSL_PORT_PROP")
> fi
> {code}
> On master, we do {{if [ "$SOLR_SSL_ENABLED" == "true" ]; then}}
> The problem is that ZkContainer (when zkRun enabled) looks to see if
> {{solr.jetty.https.port}} is set, and if so, then it assumes TLS is enabled.
> So this little BASH bug causes Solr's HttpShardHandler to connect via https.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-15046) bin/solr check to see if solr.jetty.https.port should be set always returns true
Timothy Potter created SOLR-15046:
-
Summary: bin/solr check to see if solr.jetty.https.port should be
set always returns true
Key: SOLR-15046
URL: https://issues.apache.org/jira/browse/SOLR-15046
Project: Solr
Issue Type: Bug
Security Level: Public (Default Security Level. Issues are Public)
Components: scripts and tools
Affects Versions: 8.8
Reporter: Timothy Potter
Assignee: Timothy Potter
This always returns {{true}} on Mac OS X (not sure about Linux):
{code}
if [ "$SOLR_SSL_ENABLED" ]; then
# If using SSL and solr.jetty.https.port not set explicitly, use the
jetty.port
SSL_PORT_PROP="-Dsolr.jetty.https.port=$SOLR_PORT"
SOLR_OPTS+=($SOLR_SSL_OPTS "$SSL_PORT_PROP")
fi
{code}
On master, we do {{ if [ "$SOLR_SSL_ENABLED" == "true" ]; then}}
The problem is that ZkContainer (when zkRun enabled) looks to see if
{{solr.jetty.https.port}} is set, and if so, then it assumes TLS is enabled. So
this little BASH bug causes Solr's HttpShardHandler to connect via https.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-12182) Can not switch urlScheme in 7x if there are any cores in the cluster
[
https://issues.apache.org/jira/browse/SOLR-12182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17249133#comment-17249133
]
Timothy Potter commented on SOLR-12182:
---
It looks like it is bad BASH logic :P This always evaluates to {{true}} on Mac
OS X -->
https://github.com/apache/lucene-solr/blob/branch_8x/solr/bin/solr#L2154 ... it
was fixed on master so this is only an issue with 8x
> Can not switch urlScheme in 7x if there are any cores in the cluster
>
>
> Key: SOLR-12182
> URL: https://issues.apache.org/jira/browse/SOLR-12182
> Project: Solr
> Issue Type: Bug
>Affects Versions: 7.0, 7.1, 7.2
>Reporter: Anshum Gupta
>Assignee: Timothy Potter
>Priority: Major
> Fix For: 8.8, master (9.0)
>
> Attachments: SOLR-12182.patch, SOLR-12182_20200423.patch
>
> Time Spent: 5h 50m
> Remaining Estimate: 0h
>
> I was trying to enable TLS on a cluster that was already in use i.e. had
> existing collections and ended up with down cores, that wouldn't come up and
> the following core init errors in the logs:
> *org.apache.solr.common.SolrException:org.apache.solr.common.SolrException:
> replica with coreNodeName core_node4 exists but with a different name or
> base_url.*
> What is happening here is that the core/replica is defined in the
> clusterstate with the urlScheme as part of it's base URL e.g.
> *"base_url":"http:hostname:port/solr"*.
> Switching the urlScheme in Solr breaks this convention as the host now uses
> HTTPS instead.
> Actually, I ran into this with an older version because I was running with
> *legacyCloud=false* and then realized that we switched that to the default
> behavior only in 7x i.e while most users did not hit this issue with older
> versions, unless they overrode the legacyCloud value explicitly, users
> running 7x are bound to run into this more often.
> Switching the value of legacyCloud to true, bouncing the cluster so that the
> clusterstate gets flushed, and then setting it back to false is a workaround
> but a bit risky one if you don't know if you have any old cores lying around.
> Ideally, I think we shouldn't prepend the urlScheme to the base_url value and
> use the urlScheme on the fly to construct it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-12182) Can not switch urlScheme in 7x if there are any cores in the cluster
[
https://issues.apache.org/jira/browse/SOLR-12182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17249118#comment-17249118
]
Timothy Potter commented on SOLR-12182:
---
It's because {{solr.jetty.https.port}} is set; ZkContainer sees this property
and then assumes TLS is enabled. I'm not clear why {{solr.jetty.https.port}} is
set for the {{bin/solr start -c}} though? It seems like the bug is elsewhere
b/c {{solr.jetty.https.port}} should not be set if TLS is not enabled. I'll
open a new JIRA and track it down. Thanks for finding [~jbernste]
> Can not switch urlScheme in 7x if there are any cores in the cluster
>
>
> Key: SOLR-12182
> URL: https://issues.apache.org/jira/browse/SOLR-12182
> Project: Solr
> Issue Type: Bug
>Affects Versions: 7.0, 7.1, 7.2
>Reporter: Anshum Gupta
>Assignee: Timothy Potter
>Priority: Major
> Fix For: 8.8, master (9.0)
>
> Attachments: SOLR-12182.patch, SOLR-12182_20200423.patch
>
> Time Spent: 5h 50m
> Remaining Estimate: 0h
>
> I was trying to enable TLS on a cluster that was already in use i.e. had
> existing collections and ended up with down cores, that wouldn't come up and
> the following core init errors in the logs:
> *org.apache.solr.common.SolrException:org.apache.solr.common.SolrException:
> replica with coreNodeName core_node4 exists but with a different name or
> base_url.*
> What is happening here is that the core/replica is defined in the
> clusterstate with the urlScheme as part of it's base URL e.g.
> *"base_url":"http:hostname:port/solr"*.
> Switching the urlScheme in Solr breaks this convention as the host now uses
> HTTPS instead.
> Actually, I ran into this with an older version because I was running with
> *legacyCloud=false* and then realized that we switched that to the default
> behavior only in 7x i.e while most users did not hit this issue with older
> versions, unless they overrode the legacyCloud value explicitly, users
> running 7x are bound to run into this more often.
> Switching the value of legacyCloud to true, bouncing the cluster so that the
> clusterstate gets flushed, and then setting it back to false is a workaround
> but a bit risky one if you don't know if you have any old cores lying around.
> Ideally, I think we shouldn't prepend the urlScheme to the base_url value and
> use the urlScheme on the fly to construct it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-15036) Use plist automatically for executing a facet expression against a collection alias backed by multiple collections
[
https://issues.apache.org/jira/browse/SOLR-15036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17249101#comment-17249101
]
Timothy Potter commented on SOLR-15036:
---
Ok, I have a fix for that issue but the bigger issue is whether the expression
is correct and if so, why is it not rolling up over the shards on the client?
i.e. i'm getting multiple results per bucket instead of one final result per
bucket?
> Use plist automatically for executing a facet expression against a collection
> alias backed by multiple collections
> --
>
> Key: SOLR-15036
> URL: https://issues.apache.org/jira/browse/SOLR-15036
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: streaming expressions
>Reporter: Timothy Potter
>Assignee: Timothy Potter
>Priority: Major
> Attachments: relay-approach.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> For analytics use cases, streaming expressions make it possible to compute
> basic aggregations (count, min, max, sum, and avg) over massive data sets.
> Moreover, with massive data sets, it is common to use collection aliases over
> many underlying collections, for instance time-partitioned aliases backed by
> a set of collections, each covering a specific time range. In some cases, we
> can end up with many collections (think 50-60) each with 100's of shards.
> Aliases help insulate client applications from complex collection topologies
> on the server side.
> Let's take a basic facet expression that computes some useful aggregation
> metrics:
> {code:java}
> facet(
> some_alias,
> q="*:*",
> fl="a_i",
> sort="a_i asc",
> buckets="a_i",
> bucketSorts="count(*) asc",
> bucketSizeLimit=1,
> sum(a_d), avg(a_d), min(a_d), max(a_d), count(*)
> )
> {code}
> Behind the scenes, the {{FacetStream}} sends a JSON facet request to Solr
> which then expands the alias to a list of collections. For each collection,
> the top-level distributed query controller gathers a candidate set of
> replicas to query and then scatters {{distrib=false}} queries to each replica
> in the list. For instance, if we have 60 collections with 200 shards each,
> then this results in 12,000 shard requests from the query controller node to
> the other nodes in the cluster. The requests are sent in an async manner (see
> {{SearchHandler}} and {{HttpShardHandler}}) In my testing, we’ve seen cases
> where we hit 18,000 replicas and these queries don’t always come back in a
> timely manner. Put simply, this also puts a lot of load on the top-level
> query controller node in terms of open connections and new object creation.
> Instead, we can use {{plist}} to send the JSON facet query to each collection
> in the alias in parallel, which reduces the overhead of each top-level
> distributed query from 12,000 to 200 in my example above. With this approach,
> you’ll then need to sort the tuples back from each collection and do a
> rollup, something like:
> {code:java}
> select(
> rollup(
> sort(
> plist(
> select(facet(coll1,q="*:*", fl="a_i", sort="a_i asc", buckets="a_i",
> bucketSorts="count(*) asc", bucketSizeLimit=1, sum(a_d), avg(a_d),
> min(a_d), max(a_d), count(*)),a_i,sum(a_d) as the_sum, avg(a_d) as the_avg,
> min(a_d) as the_min, max(a_d) as the_max, count(*) as cnt),
> select(facet(coll2,q="*:*", fl="a_i", sort="a_i asc", buckets="a_i",
> bucketSorts="count(*) asc", bucketSizeLimit=1, sum(a_d), avg(a_d),
> min(a_d), max(a_d), count(*)),a_i,sum(a_d) as the_sum, avg(a_d) as the_avg,
> min(a_d) as the_min, max(a_d) as the_max, count(*) as cnt)
> ),
> by="a_i asc"
> ),
> over="a_i",
> sum(the_sum), avg(the_avg), min(the_min), max(the_max), sum(cnt)
> ),
> a_i, sum(the_sum) as the_sum, avg(the_avg) as the_avg, min(the_min) as
> the_min, max(the_max) as the_max, sum(cnt) as cnt
> )
> {code}
> One thing to point out is that you can’t just avg. the averages back from
> each collection in the rollup. It needs to be a *weighted avg.* when rolling
> up the avg. from each facet expression in the plist. However, we have the
> count per collection, so this is doable but will require some changes to the
> rollup expression to support weighted average.
> While this plist approach is doable, it’s a pain for users to have to create
> the rollup / sort over plist expression for collection aliases. After all,
> aliases are supposed to hide these types of complexities from client
> applications!
> The point of this ticket is to investigate the feasibility of auto-wrapping
> the facet expression with a rollup / sort / plist when the
[jira] [Commented] (SOLR-15036) Use plist automatically for executing a facet expression against a collection alias backed by multiple collections
[
https://issues.apache.org/jira/browse/SOLR-15036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17249078#comment-17249078
]
Timothy Potter commented on SOLR-15036:
---
[~jbernste] does this {{drill}} expr look correct? I'm not seeing the results
from each shard get rolled up into final results? Do I have to do that part
myself? If so, that's kind of weird
{code}
drill(
SOME_ALIAS_WITH_MANY_COLLS,
q="*:*",
fl="a_i",
sort="a_i asc",
rollup(
input(),
over="a_i",
sum(a_d), avg(a_d), min(a_d), max(a_d), count(*)
)
)
{code}
Also, I was getting a number of NullPointerExceptions in the metrics
computation in the ExportWriter, such as:
{code}
java.util.concurrent.ExecutionException: java.io.IOException: -->
http://127.0.0.1:62300/solr/coll1_shard2_replica_n2/:
java.lang.NullPointerException
at
org.apache.solr.client.solrj.io.stream.metrics.SumMetric.update(SumMetric.java:75)
at
org.apache.solr.client.solrj.io.stream.RollupStream.read(RollupStream.java:252)
at
org.apache.solr.handler.export.ExportWriter.lambda$writeDocs$7(ExportWriter.java:378)
at
org.apache.solr.handler.export.ExportBuffers.run(ExportBuffers.java:217)
at
org.apache.solr.handler.export.ExportWriter.writeDocs(ExportWriter.java:371)
at
org.apache.solr.handler.export.ExportWriter.lambda$write$1(ExportWriter.java:288)
{code}
> Use plist automatically for executing a facet expression against a collection
> alias backed by multiple collections
> --
>
> Key: SOLR-15036
> URL: https://issues.apache.org/jira/browse/SOLR-15036
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: streaming expressions
>Reporter: Timothy Potter
>Assignee: Timothy Potter
>Priority: Major
> Attachments: relay-approach.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> For analytics use cases, streaming expressions make it possible to compute
> basic aggregations (count, min, max, sum, and avg) over massive data sets.
> Moreover, with massive data sets, it is common to use collection aliases over
> many underlying collections, for instance time-partitioned aliases backed by
> a set of collections, each covering a specific time range. In some cases, we
> can end up with many collections (think 50-60) each with 100's of shards.
> Aliases help insulate client applications from complex collection topologies
> on the server side.
> Let's take a basic facet expression that computes some useful aggregation
> metrics:
> {code:java}
> facet(
> some_alias,
> q="*:*",
> fl="a_i",
> sort="a_i asc",
> buckets="a_i",
> bucketSorts="count(*) asc",
> bucketSizeLimit=1,
> sum(a_d), avg(a_d), min(a_d), max(a_d), count(*)
> )
> {code}
> Behind the scenes, the {{FacetStream}} sends a JSON facet request to Solr
> which then expands the alias to a list of collections. For each collection,
> the top-level distributed query controller gathers a candidate set of
> replicas to query and then scatters {{distrib=false}} queries to each replica
> in the list. For instance, if we have 60 collections with 200 shards each,
> then this results in 12,000 shard requests from the query controller node to
> the other nodes in the cluster. The requests are sent in an async manner (see
> {{SearchHandler}} and {{HttpShardHandler}}) In my testing, we’ve seen cases
> where we hit 18,000 replicas and these queries don’t always come back in a
> timely manner. Put simply, this also puts a lot of load on the top-level
> query controller node in terms of open connections and new object creation.
> Instead, we can use {{plist}} to send the JSON facet query to each collection
> in the alias in parallel, which reduces the overhead of each top-level
> distributed query from 12,000 to 200 in my example above. With this approach,
> you’ll then need to sort the tuples back from each collection and do a
> rollup, something like:
> {code:java}
> select(
> rollup(
> sort(
> plist(
> select(facet(coll1,q="*:*", fl="a_i", sort="a_i asc", buckets="a_i",
> bucketSorts="count(*) asc", bucketSizeLimit=1, sum(a_d), avg(a_d),
> min(a_d), max(a_d), count(*)),a_i,sum(a_d) as the_sum, avg(a_d) as the_avg,
> min(a_d) as the_min, max(a_d) as the_max, count(*) as cnt),
> select(facet(coll2,q="*:*", fl="a_i", sort="a_i asc", buckets="a_i",
> bucketSorts="count(*) asc", bucketSizeLimit=1, sum(a_d), avg(a_d),
> min(a_d), max(a_d), count(*)),a_i,sum(a_d) as the_sum, avg(a_d) as the_avg,
> min(a_d) as the_min, max(a_d) as the_max, count(*) as cnt)
> ),
> by="a_i asc"
> ),
>
[jira] [Comment Edited] (SOLR-15026) MiniSolrCloudCluster can inconsistently get confused about when it's using SSL
[
https://issues.apache.org/jira/browse/SOLR-15026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17249062#comment-17249062
]
Timothy Potter edited comment on SOLR-15026 at 12/14/20, 3:52 PM:
--
Not related to this issue ^ for sure ... so maybe open a new JIRA and I'll take
a look.
How did you start the server? {{ant server}} doesn't start a Solr, just builds
so you can start one right? When I run your 2 commands, I get:
{code}
ant server
bin/solr create -c test -s 1 -d _default
Failed to determine the port of a local Solr instance, cannot create test!
{code}
If I start, then create works:
{code}
bin/solr start
Waiting up to 180 seconds to see Solr running on port 8983 [-]
Started Solr server on port 8983 (pid=18118). Happy searching!
$ bin/solr create -c test -s 1 -d _default
WARNING: Using _default configset with data driven schema functionality. NOT
RECOMMENDED for production use.
To turn off: bin/solr config -c test -p 8983 -action set-user-property
-property update.autoCreateFields -value false
Created new core 'test'
{code}
was (Author: thelabdude):
Not related to this issue ^ for sure ... so maybe open a new JIRA and I'll take
a look.
How did you start the server? {{ant server}} doesn't start a Solr, just builds
so you can start one right? When I run your 2 commands, I get:
{code}
ant server
bin/solr create -c test -s 1 -d _default
Failed to determine the port of a local Solr instance, cannot create test!
{code}
If I start, then create works:
{code}
bin/solr start
*** [WARN] *** Your open file limit is currently 2560.
It should be set to 65000 to avoid operational disruption.
If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in
your profile or solr.in.sh
*** [WARN] *** Your Max Processes Limit is currently 11136.
It should be set to 65000 to avoid operational disruption.
If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in
your profile or solr.in.sh
Waiting up to 180 seconds to see Solr running on port 8983 [-]
Started Solr server on port 8983 (pid=18118). Happy searching!
(⎈ |docker-desktop:tjp)~/dev/oss/lucene-solr/8x/solr$ bin/solr create -c test
-s 1 -d _default
WARNING: Using _default configset with data driven schema functionality. NOT
RECOMMENDED for production use.
To turn off: bin/solr config -c test -p 8983 -action set-user-property
-property update.autoCreateFields -value false
Created new core 'test'
{code}
> MiniSolrCloudCluster can inconsistently get confused about when it's using SSL
> --
>
> Key: SOLR-15026
> URL: https://issues.apache.org/jira/browse/SOLR-15026
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Chris M. Hostetter
>Priority: Major
>
> A new test added in SOLR-14934 caused the following reproducible failure to
> pop up on jenkins...
> {noformat}
> hossman@slate:~/lucene/dev [j11] [master] $ ./gradlew -p solr/test-framework/
> test --tests MiniSolrCloudClusterTest.testSolrHomeAndResourceLoaders
> -Dtests.seed=806A85748BD81F48 -Dtests.multiplier=2 -Dtests.slow=true
> -Dtests.locale=ln-CG -Dtests.timezone=Asia/Thimbu -Dtests.asserts=true
> -Dtests.file.encoding=UTF-8
> Starting a Gradle Daemon (subsequent builds will be faster)
> > Task :randomizationInfo
> Running tests with randomization seed: tests.seed=806A85748BD81F48
> > Task :solr:test-framework:test
> org.apache.solr.cloud.MiniSolrCloudClusterTest >
> testSolrHomeAndResourceLoaders FAILED
> org.apache.solr.client.solrj.SolrServerException: IOException occurred
> when talking to server at: https://127.0.0.1:38681/solr
> at
> __randomizedtesting.SeedInfo.seed([806A85748BD81F48:37548FA7602CB5FD]:0)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:712)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:269)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:251)
> at
> org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:390)
> at
> org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:360)
> at
> org.apache.solr.client.solrj.impl.BaseCloudSolrClient.sendRequest(BaseCloudSolrClient.java:1168)
> at
> org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:931)
> at
> org.apache.solr.client.solrj.impl.BaseCloudSolrClient.request(BaseCloudSolrClient.java:865)
> at
> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:229)
> at
>
[jira] [Comment Edited] (SOLR-15026) MiniSolrCloudCluster can inconsistently get confused about when it's using SSL
[
https://issues.apache.org/jira/browse/SOLR-15026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17249062#comment-17249062
]
Timothy Potter edited comment on SOLR-15026 at 12/14/20, 3:51 PM:
--
Not related to this issue ^ for sure ... so maybe open a new JIRA and I'll take
a look.
How did you start the server? {{ant server}} doesn't start a Solr, just builds
so you can start one right? When I run your 2 commands, I get:
{code}
ant server
bin/solr create -c test -s 1 -d _default
Failed to determine the port of a local Solr instance, cannot create test!
{code}
If I start, then create works:
{code}
bin/solr start
*** [WARN] *** Your open file limit is currently 2560.
It should be set to 65000 to avoid operational disruption.
If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in
your profile or solr.in.sh
*** [WARN] *** Your Max Processes Limit is currently 11136.
It should be set to 65000 to avoid operational disruption.
If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in
your profile or solr.in.sh
Waiting up to 180 seconds to see Solr running on port 8983 [-]
Started Solr server on port 8983 (pid=18118). Happy searching!
(⎈ |docker-desktop:tjp)~/dev/oss/lucene-solr/8x/solr$ bin/solr create -c test
-s 1 -d _default
WARNING: Using _default configset with data driven schema functionality. NOT
RECOMMENDED for production use.
To turn off: bin/solr config -c test -p 8983 -action set-user-property
-property update.autoCreateFields -value false
Created new core 'test'
{code}
was (Author: thelabdude):
Not related to this issue ^ for sure ... so maybe open a new JIRA and I'll take
a look.
How did you start the server? {{ant server}} doesn't start a Solr, just builds
so you can start one right? When I run your 2 commands, I get:
{code}
ant server
bin/solr create -c test -s 1 -d _default
Failed to determine the port of a local Solr instance, cannot create test!
{code}
> MiniSolrCloudCluster can inconsistently get confused about when it's using SSL
> --
>
> Key: SOLR-15026
> URL: https://issues.apache.org/jira/browse/SOLR-15026
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Chris M. Hostetter
>Priority: Major
>
> A new test added in SOLR-14934 caused the following reproducible failure to
> pop up on jenkins...
> {noformat}
> hossman@slate:~/lucene/dev [j11] [master] $ ./gradlew -p solr/test-framework/
> test --tests MiniSolrCloudClusterTest.testSolrHomeAndResourceLoaders
> -Dtests.seed=806A85748BD81F48 -Dtests.multiplier=2 -Dtests.slow=true
> -Dtests.locale=ln-CG -Dtests.timezone=Asia/Thimbu -Dtests.asserts=true
> -Dtests.file.encoding=UTF-8
> Starting a Gradle Daemon (subsequent builds will be faster)
> > Task :randomizationInfo
> Running tests with randomization seed: tests.seed=806A85748BD81F48
> > Task :solr:test-framework:test
> org.apache.solr.cloud.MiniSolrCloudClusterTest >
> testSolrHomeAndResourceLoaders FAILED
> org.apache.solr.client.solrj.SolrServerException: IOException occurred
> when talking to server at: https://127.0.0.1:38681/solr
> at
> __randomizedtesting.SeedInfo.seed([806A85748BD81F48:37548FA7602CB5FD]:0)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:712)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:269)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:251)
> at
> org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:390)
> at
> org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:360)
> at
> org.apache.solr.client.solrj.impl.BaseCloudSolrClient.sendRequest(BaseCloudSolrClient.java:1168)
> at
> org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:931)
> at
> org.apache.solr.client.solrj.impl.BaseCloudSolrClient.request(BaseCloudSolrClient.java:865)
> at
> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:229)
> at
> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:246)
> at
> org.apache.solr.cloud.MiniSolrCloudClusterTest.testSolrHomeAndResourceLoaders(MiniSolrCloudClusterTest.java:125)
> ...
> Caused by:
> javax.net.ssl.SSLException: Unsupported or unrecognized SSL message
> at
> java.base/sun.security.ssl.SSLSocketInputRecord.handleUnknownRecord(SSLSocketInputRecord.java:439)
> {noformat}
> The problem sems to be that even though the MiniSolrCloudCluster being
> instantiated isn't _intentionally_ using
[jira] [Commented] (SOLR-15026) MiniSolrCloudCluster can inconsistently get confused about when it's using SSL
[
https://issues.apache.org/jira/browse/SOLR-15026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17249062#comment-17249062
]
Timothy Potter commented on SOLR-15026:
---
Not related to this issue ^ for sure ... so maybe open a new JIRA and I'll take
a look.
How did you start the server? {{ant server}} doesn't start a Solr, just builds
so you can start one right? When I run your 2 commands, I get:
{code}
ant server
bin/solr create -c test -s 1 -d _default
Failed to determine the port of a local Solr instance, cannot create test!
{code}
> MiniSolrCloudCluster can inconsistently get confused about when it's using SSL
> --
>
> Key: SOLR-15026
> URL: https://issues.apache.org/jira/browse/SOLR-15026
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Chris M. Hostetter
>Priority: Major
>
> A new test added in SOLR-14934 caused the following reproducible failure to
> pop up on jenkins...
> {noformat}
> hossman@slate:~/lucene/dev [j11] [master] $ ./gradlew -p solr/test-framework/
> test --tests MiniSolrCloudClusterTest.testSolrHomeAndResourceLoaders
> -Dtests.seed=806A85748BD81F48 -Dtests.multiplier=2 -Dtests.slow=true
> -Dtests.locale=ln-CG -Dtests.timezone=Asia/Thimbu -Dtests.asserts=true
> -Dtests.file.encoding=UTF-8
> Starting a Gradle Daemon (subsequent builds will be faster)
> > Task :randomizationInfo
> Running tests with randomization seed: tests.seed=806A85748BD81F48
> > Task :solr:test-framework:test
> org.apache.solr.cloud.MiniSolrCloudClusterTest >
> testSolrHomeAndResourceLoaders FAILED
> org.apache.solr.client.solrj.SolrServerException: IOException occurred
> when talking to server at: https://127.0.0.1:38681/solr
> at
> __randomizedtesting.SeedInfo.seed([806A85748BD81F48:37548FA7602CB5FD]:0)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:712)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:269)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:251)
> at
> org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:390)
> at
> org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:360)
> at
> org.apache.solr.client.solrj.impl.BaseCloudSolrClient.sendRequest(BaseCloudSolrClient.java:1168)
> at
> org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:931)
> at
> org.apache.solr.client.solrj.impl.BaseCloudSolrClient.request(BaseCloudSolrClient.java:865)
> at
> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:229)
> at
> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:246)
> at
> org.apache.solr.cloud.MiniSolrCloudClusterTest.testSolrHomeAndResourceLoaders(MiniSolrCloudClusterTest.java:125)
> ...
> Caused by:
> javax.net.ssl.SSLException: Unsupported or unrecognized SSL message
> at
> java.base/sun.security.ssl.SSLSocketInputRecord.handleUnknownRecord(SSLSocketInputRecord.java:439)
> {noformat}
> The problem sems to be that even though the MiniSolrCloudCluster being
> instantiated isn't _intentionally_ using any SSL randomization (it just uses
> {{JettyConfig.builder().build()}} the CloudSolrClient returned by
> {{cluster.getSolrClient()}} is evidnetly picking up the ranodmized SSL and
> trying to use it to talk to the cluster.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Assigned] (SOLR-6443) TestManagedResourceStorage fails on Jenkins with SolrCore.getOpenCount()==2
[ https://issues.apache.org/jira/browse/SOLR-6443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timothy Potter reassigned SOLR-6443: Assignee: (was: Timothy Potter) > TestManagedResourceStorage fails on Jenkins with SolrCore.getOpenCount()==2 > --- > > Key: SOLR-6443 > URL: https://issues.apache.org/jira/browse/SOLR-6443 > Project: Solr > Issue Type: Bug > Components: Schema and Analysis >Reporter: Timothy Potter >Priority: Major > > FAILED: > junit.framework.TestSuite.org.apache.solr.rest.TestManagedResourceStorage > Error Message: > SolrCore.getOpenCount()==2 > Stack Trace: > java.lang.RuntimeException: SolrCore.getOpenCount()==2 > at __randomizedtesting.SeedInfo.seed([A491D1FD4CEF5EF8]:0) > at org.apache.solr.util.TestHarness.close(TestHarness.java:332) > at org.apache.solr.SolrTestCaseJ4.deleteCore(SolrTestCaseJ4.java:620) > at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:183) > at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:484) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Assigned] (SOLR-9008) Investigate feasibilty and impact of using SparseFixedBitSet where Solr is currently using FixedBitSet
[
https://issues.apache.org/jira/browse/SOLR-9008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timothy Potter reassigned SOLR-9008:
Assignee: (was: Timothy Potter)
> Investigate feasibilty and impact of using SparseFixedBitSet where Solr is
> currently using FixedBitSet
> --
>
> Key: SOLR-9008
> URL: https://issues.apache.org/jira/browse/SOLR-9008
> Project: Solr
> Issue Type: Improvement
>Reporter: Timothy Potter
>Priority: Major
>
> Found this gem in one of Mike's blog posts:
> {quote}
> But with 5.0.0, Lucene now supports random-writable and advance-able sparse
> bitsets (RoaringDocIdSet and SparseFixedBitSet), so the heap required is in
> proportion to how many bits are set, not how many total documents exist in
> the index.
> {quote}
> http://blog.mikemccandless.com/2014/11/apache-lucene-500-is-coming.html
> I don't see any uses of either of these classes in Solr code but from a quick
> look, sounds compelling for saving memory, such as when caching fq's
> This ticket is for exploring where Solr can leverage these structures and
> whether there's an improvement in performance and/or memory usage.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Assigned] (SOLR-14766) Deprecate ManagedResources from Solr
[ https://issues.apache.org/jira/browse/SOLR-14766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timothy Potter reassigned SOLR-14766: - Assignee: (was: Timothy Potter) > Deprecate ManagedResources from Solr > > > Key: SOLR-14766 > URL: https://issues.apache.org/jira/browse/SOLR-14766 > Project: Solr > Issue Type: Task >Reporter: Noble Paul >Priority: Major > Labels: deprecation > Attachments: SOLR-14766.patch > > Time Spent: 1h > Remaining Estimate: 0h > > This feature has the following problems. > * It's insecure because it is using restlet > * Nobody knows that code enough to even remove the restlet dependency > * Restlest dependency on Solr exists just because of this > We should deprecate this from 8.7 and remove it from master -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Assigned] (SOLR-15026) MiniSolrCloudCluster can inconsistently get confused about when it's using SSL
[
https://issues.apache.org/jira/browse/SOLR-15026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timothy Potter reassigned SOLR-15026:
-
Assignee: (was: Timothy Potter)
> MiniSolrCloudCluster can inconsistently get confused about when it's using SSL
> --
>
> Key: SOLR-15026
> URL: https://issues.apache.org/jira/browse/SOLR-15026
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Chris M. Hostetter
>Priority: Major
>
> A new test added in SOLR-14934 caused the following reproducible failure to
> pop up on jenkins...
> {noformat}
> hossman@slate:~/lucene/dev [j11] [master] $ ./gradlew -p solr/test-framework/
> test --tests MiniSolrCloudClusterTest.testSolrHomeAndResourceLoaders
> -Dtests.seed=806A85748BD81F48 -Dtests.multiplier=2 -Dtests.slow=true
> -Dtests.locale=ln-CG -Dtests.timezone=Asia/Thimbu -Dtests.asserts=true
> -Dtests.file.encoding=UTF-8
> Starting a Gradle Daemon (subsequent builds will be faster)
> > Task :randomizationInfo
> Running tests with randomization seed: tests.seed=806A85748BD81F48
> > Task :solr:test-framework:test
> org.apache.solr.cloud.MiniSolrCloudClusterTest >
> testSolrHomeAndResourceLoaders FAILED
> org.apache.solr.client.solrj.SolrServerException: IOException occurred
> when talking to server at: https://127.0.0.1:38681/solr
> at
> __randomizedtesting.SeedInfo.seed([806A85748BD81F48:37548FA7602CB5FD]:0)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:712)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:269)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:251)
> at
> org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:390)
> at
> org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:360)
> at
> org.apache.solr.client.solrj.impl.BaseCloudSolrClient.sendRequest(BaseCloudSolrClient.java:1168)
> at
> org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:931)
> at
> org.apache.solr.client.solrj.impl.BaseCloudSolrClient.request(BaseCloudSolrClient.java:865)
> at
> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:229)
> at
> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:246)
> at
> org.apache.solr.cloud.MiniSolrCloudClusterTest.testSolrHomeAndResourceLoaders(MiniSolrCloudClusterTest.java:125)
> ...
> Caused by:
> javax.net.ssl.SSLException: Unsupported or unrecognized SSL message
> at
> java.base/sun.security.ssl.SSLSocketInputRecord.handleUnknownRecord(SSLSocketInputRecord.java:439)
> {noformat}
> The problem sems to be that even though the MiniSolrCloudCluster being
> instantiated isn't _intentionally_ using any SSL randomization (it just uses
> {{JettyConfig.builder().build()}} the CloudSolrClient returned by
> {{cluster.getSolrClient()}} is evidnetly picking up the ranodmized SSL and
> trying to use it to talk to the cluster.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-15036) Use plist automatically for executing a facet expression against a collection alias backed by multiple collections
[
https://issues.apache.org/jira/browse/SOLR-15036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17246624#comment-17246624
]
Timothy Potter commented on SOLR-15036:
---
I plan to spend some time today learning the inner-workings of the {{drill}}
source. Also, I wouldn't get too hung up on my example only having a single
dimension, as it's just an example. When I test this in a large cluster, I'll
test with several dimensions as well, some with high cardinality.
> Use plist automatically for executing a facet expression against a collection
> alias backed by multiple collections
> --
>
> Key: SOLR-15036
> URL: https://issues.apache.org/jira/browse/SOLR-15036
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: streaming expressions
>Reporter: Timothy Potter
>Assignee: Timothy Potter
>Priority: Major
> Attachments: relay-approach.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> For analytics use cases, streaming expressions make it possible to compute
> basic aggregations (count, min, max, sum, and avg) over massive data sets.
> Moreover, with massive data sets, it is common to use collection aliases over
> many underlying collections, for instance time-partitioned aliases backed by
> a set of collections, each covering a specific time range. In some cases, we
> can end up with many collections (think 50-60) each with 100's of shards.
> Aliases help insulate client applications from complex collection topologies
> on the server side.
> Let's take a basic facet expression that computes some useful aggregation
> metrics:
> {code:java}
> facet(
> some_alias,
> q="*:*",
> fl="a_i",
> sort="a_i asc",
> buckets="a_i",
> bucketSorts="count(*) asc",
> bucketSizeLimit=1,
> sum(a_d), avg(a_d), min(a_d), max(a_d), count(*)
> )
> {code}
> Behind the scenes, the {{FacetStream}} sends a JSON facet request to Solr
> which then expands the alias to a list of collections. For each collection,
> the top-level distributed query controller gathers a candidate set of
> replicas to query and then scatters {{distrib=false}} queries to each replica
> in the list. For instance, if we have 60 collections with 200 shards each,
> then this results in 12,000 shard requests from the query controller node to
> the other nodes in the cluster. The requests are sent in an async manner (see
> {{SearchHandler}} and {{HttpShardHandler}}) In my testing, we’ve seen cases
> where we hit 18,000 replicas and these queries don’t always come back in a
> timely manner. Put simply, this also puts a lot of load on the top-level
> query controller node in terms of open connections and new object creation.
> Instead, we can use {{plist}} to send the JSON facet query to each collection
> in the alias in parallel, which reduces the overhead of each top-level
> distributed query from 12,000 to 200 in my example above. With this approach,
> you’ll then need to sort the tuples back from each collection and do a
> rollup, something like:
> {code:java}
> select(
> rollup(
> sort(
> plist(
> select(facet(coll1,q="*:*", fl="a_i", sort="a_i asc", buckets="a_i",
> bucketSorts="count(*) asc", bucketSizeLimit=1, sum(a_d), avg(a_d),
> min(a_d), max(a_d), count(*)),a_i,sum(a_d) as the_sum, avg(a_d) as the_avg,
> min(a_d) as the_min, max(a_d) as the_max, count(*) as cnt),
> select(facet(coll2,q="*:*", fl="a_i", sort="a_i asc", buckets="a_i",
> bucketSorts="count(*) asc", bucketSizeLimit=1, sum(a_d), avg(a_d),
> min(a_d), max(a_d), count(*)),a_i,sum(a_d) as the_sum, avg(a_d) as the_avg,
> min(a_d) as the_min, max(a_d) as the_max, count(*) as cnt)
> ),
> by="a_i asc"
> ),
> over="a_i",
> sum(the_sum), avg(the_avg), min(the_min), max(the_max), sum(cnt)
> ),
> a_i, sum(the_sum) as the_sum, avg(the_avg) as the_avg, min(the_min) as
> the_min, max(the_max) as the_max, sum(cnt) as cnt
> )
> {code}
> One thing to point out is that you can’t just avg. the averages back from
> each collection in the rollup. It needs to be a *weighted avg.* when rolling
> up the avg. from each facet expression in the plist. However, we have the
> count per collection, so this is doable but will require some changes to the
> rollup expression to support weighted average.
> While this plist approach is doable, it’s a pain for users to have to create
> the rollup / sort over plist expression for collection aliases. After all,
> aliases are supposed to hide these types of complexities from client
> applications!
> The point of this ticket is to investigate the feasibility of auto-wrapping
> the
[jira] [Commented] (SOLR-15036) Use plist automatically for executing a facet expression against a collection alias backed by multiple collections
[
https://issues.apache.org/jira/browse/SOLR-15036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17246031#comment-17246031
]
Timothy Potter commented on SOLR-15036:
---
[~atri] the patch isn't the solution I'm going for, as I explained in the
description ...
Regarding drill, I'll have to investigate its performance compared to {{plist}}
and {{facet}}. However, since it's based on {{/export}} seems like it would be
a lot of I/O out each Solr node instead of just relying on the efficient JSON
facet implementation? I certainly don't want to {{/export}} 1B rows to count
them when I can just facet instead. What would a {{drill}} expression look like
that does the same as my example in the description?
> Use plist automatically for executing a facet expression against a collection
> alias backed by multiple collections
> --
>
> Key: SOLR-15036
> URL: https://issues.apache.org/jira/browse/SOLR-15036
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: streaming expressions
>Reporter: Timothy Potter
>Assignee: Timothy Potter
>Priority: Major
> Attachments: relay-approach.patch
>
>
> For analytics use cases, streaming expressions make it possible to compute
> basic aggregations (count, min, max, sum, and avg) over massive data sets.
> Moreover, with massive data sets, it is common to use collection aliases over
> many underlying collections, for instance time-partitioned aliases backed by
> a set of collections, each covering a specific time range. In some cases, we
> can end up with many collections (think 50-60) each with 100's of shards.
> Aliases help insulate client applications from complex collection topologies
> on the server side.
> Let's take a basic facet expression that computes some useful aggregation
> metrics:
> {code:java}
> facet(
> some_alias,
> q="*:*",
> fl="a_i",
> sort="a_i asc",
> buckets="a_i",
> bucketSorts="count(*) asc",
> bucketSizeLimit=1,
> sum(a_d), avg(a_d), min(a_d), max(a_d), count(*)
> )
> {code}
> Behind the scenes, the {{FacetStream}} sends a JSON facet request to Solr
> which then expands the alias to a list of collections. For each collection,
> the top-level distributed query controller gathers a candidate set of
> replicas to query and then scatters {{distrib=false}} queries to each replica
> in the list. For instance, if we have 60 collections with 200 shards each,
> then this results in 12,000 shard requests from the query controller node to
> the other nodes in the cluster. The requests are sent in an async manner (see
> {{SearchHandler}} and {{HttpShardHandler}}) In my testing, we’ve seen cases
> where we hit 18,000 replicas and these queries don’t always come back in a
> timely manner. Put simply, this also puts a lot of load on the top-level
> query controller node in terms of open connections and new object creation.
> Instead, we can use {{plist}} to send the JSON facet query to each collection
> in the alias in parallel, which reduces the overhead of each top-level
> distributed query from 12,000 to 200 in my example above. With this approach,
> you’ll then need to sort the tuples back from each collection and do a
> rollup, something like:
> {code:java}
> select(
> rollup(
> sort(
> plist(
> select(facet(coll1,q="*:*", fl="a_i", sort="a_i asc", buckets="a_i",
> bucketSorts="count(*) asc", bucketSizeLimit=1, sum(a_d), avg(a_d),
> min(a_d), max(a_d), count(*)),a_i,sum(a_d) as the_sum, avg(a_d) as the_avg,
> min(a_d) as the_min, max(a_d) as the_max, count(*) as cnt),
> select(facet(coll2,q="*:*", fl="a_i", sort="a_i asc", buckets="a_i",
> bucketSorts="count(*) asc", bucketSizeLimit=1, sum(a_d), avg(a_d),
> min(a_d), max(a_d), count(*)),a_i,sum(a_d) as the_sum, avg(a_d) as the_avg,
> min(a_d) as the_min, max(a_d) as the_max, count(*) as cnt)
> ),
> by="a_i asc"
> ),
> over="a_i",
> sum(the_sum), avg(the_avg), min(the_min), max(the_max), sum(cnt)
> ),
> a_i, sum(the_sum) as the_sum, avg(the_avg) as the_avg, min(the_min) as
> the_min, max(the_max) as the_max, sum(cnt) as cnt
> )
> {code}
> One thing to point out is that you can’t just avg. the averages back from
> each collection in the rollup. It needs to be a *weighted avg.* when rolling
> up the avg. from each facet expression in the plist. However, we have the
> count per collection, so this is doable but will require some changes to the
> rollup expression to support weighted average.
> While this plist approach is doable, it’s a pain for users to have to create
> the rollup / sort over plist expression for collection
[jira] [Created] (SOLR-15036) Use plist automatically for executing a facet expression against a collection alias backed by multiple collections
Timothy Potter created SOLR-15036:
-
Summary: Use plist automatically for executing a facet expression
against a collection alias backed by multiple collections
Key: SOLR-15036
URL: https://issues.apache.org/jira/browse/SOLR-15036
Project: Solr
Issue Type: Improvement
Security Level: Public (Default Security Level. Issues are Public)
Components: streaming expressions
Reporter: Timothy Potter
Assignee: Timothy Potter
Attachments: relay-approach.patch
For analytics use cases, streaming expressions make it possible to compute
basic aggregations (count, min, max, sum, and avg) over massive data sets.
Moreover, with massive data sets, it is common to use collection aliases over
many underlying collections, for instance time-partitioned aliases backed by a
set of collections, each covering a specific time range. In some cases, we can
end up with many collections (think 50-60) each with 100's of shards. Aliases
help insulate client applications from complex collection topologies on the
server side.
Let's take a basic facet expression that computes some useful aggregation
metrics:
{code:java}
facet(
some_alias,
q="*:*",
fl="a_i",
sort="a_i asc",
buckets="a_i",
bucketSorts="count(*) asc",
bucketSizeLimit=1,
sum(a_d), avg(a_d), min(a_d), max(a_d), count(*)
)
{code}
Behind the scenes, the {{FacetStream}} sends a JSON facet request to Solr which
then expands the alias to a list of collections. For each collection, the
top-level distributed query controller gathers a candidate set of replicas to
query and then scatters {{distrib=false}} queries to each replica in the list.
For instance, if we have 60 collections with 200 shards each, then this results
in 12,000 shard requests from the query controller node to the other nodes in
the cluster. The requests are sent in an async manner (see {{SearchHandler}}
and {{HttpShardHandler}}) In my testing, we’ve seen cases where we hit 18,000
replicas and these queries don’t always come back in a timely manner. Put
simply, this also puts a lot of load on the top-level query controller node in
terms of open connections and new object creation.
Instead, we can use {{plist}} to send the JSON facet query to each collection
in the alias in parallel, which reduces the overhead of each top-level
distributed query from 12,000 to 200 in my example above. With this approach,
you’ll then need to sort the tuples back from each collection and do a rollup,
something like:
{code:java}
select(
rollup(
sort(
plist(
select(facet(coll1,q="*:*", fl="a_i", sort="a_i asc", buckets="a_i",
bucketSorts="count(*) asc", bucketSizeLimit=1, sum(a_d), avg(a_d),
min(a_d), max(a_d), count(*)),a_i,sum(a_d) as the_sum, avg(a_d) as the_avg,
min(a_d) as the_min, max(a_d) as the_max, count(*) as cnt),
select(facet(coll2,q="*:*", fl="a_i", sort="a_i asc", buckets="a_i",
bucketSorts="count(*) asc", bucketSizeLimit=1, sum(a_d), avg(a_d),
min(a_d), max(a_d), count(*)),a_i,sum(a_d) as the_sum, avg(a_d) as the_avg,
min(a_d) as the_min, max(a_d) as the_max, count(*) as cnt)
),
by="a_i asc"
),
over="a_i",
sum(the_sum), avg(the_avg), min(the_min), max(the_max), sum(cnt)
),
a_i, sum(the_sum) as the_sum, avg(the_avg) as the_avg, min(the_min) as
the_min, max(the_max) as the_max, sum(cnt) as cnt
)
{code}
One thing to point out is that you can’t just avg. the averages back from each
collection in the rollup. It needs to be a *weighted avg.* when rolling up the
avg. from each facet expression in the plist. However, we have the count per
collection, so this is doable but will require some changes to the rollup
expression to support weighted average.
While this plist approach is doable, it’s a pain for users to have to create
the rollup / sort over plist expression for collection aliases. After all,
aliases are supposed to hide these types of complexities from client
applications!
The point of this ticket is to investigate the feasibility of auto-wrapping the
facet expression with a rollup / sort / plist when the collection argument is
an alias with multiple collections; other stream sources will be considered
after facet is proven out.
Lastly, I also considered an alternative approach of doing a parallel relay on
the server side. The idea is similar to {{plist}} but instead of this being
driven on the client side, the {{FacetModule}} can create intermediate queries
(I called them {{relay}} queries in my impl.) that help distribute the load. In
my example above, there would be 60 such relay queries, each sent to a replica
for each collection in the alias, which then sends the {{distrib=false}}
queries to each replica. The relay query response handler collects the facet
responses from each replica before sending back to the top-level query
[jira] [Commented] (SOLR-15026) MiniSolrCloudCluster can inconsistently get confused about when it's using SSL
[
https://issues.apache.org/jira/browse/SOLR-15026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17245463#comment-17245463
]
Timothy Potter commented on SOLR-15026:
---
Hi [~hossman] I actually think the right fix for this test class is to just add
the {{SuppressSSL}} annotation as you've already done. Alternatively, we could
pass the SSLConfig from the {{SolrTestCaseJ4}} to the JettyConfig builder such
as:
{{JettyConfig.builder().withSSLConfig(sslConfig.buildServerSSLConfig()).build()}}.
But, that's already being done in {{TestMiniSolrCloudClusterSSL}} for SSL
specific tests, seems unnecessary to add SSL randomization to
{{MiniSolrCloudClusterTest}}. If you agree, we can just mark this as won't fix
and leave the {{@SolrTestCaseJ4.SuppressSSL}} annotation in place or I can
update the test to pass the SSLConfig and remove the annotation.
> MiniSolrCloudCluster can inconsistently get confused about when it's using SSL
> --
>
> Key: SOLR-15026
> URL: https://issues.apache.org/jira/browse/SOLR-15026
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Chris M. Hostetter
>Assignee: Timothy Potter
>Priority: Major
>
> A new test added in SOLR-14934 caused the following reproducible failure to
> pop up on jenkins...
> {noformat}
> hossman@slate:~/lucene/dev [j11] [master] $ ./gradlew -p solr/test-framework/
> test --tests MiniSolrCloudClusterTest.testSolrHomeAndResourceLoaders
> -Dtests.seed=806A85748BD81F48 -Dtests.multiplier=2 -Dtests.slow=true
> -Dtests.locale=ln-CG -Dtests.timezone=Asia/Thimbu -Dtests.asserts=true
> -Dtests.file.encoding=UTF-8
> Starting a Gradle Daemon (subsequent builds will be faster)
> > Task :randomizationInfo
> Running tests with randomization seed: tests.seed=806A85748BD81F48
> > Task :solr:test-framework:test
> org.apache.solr.cloud.MiniSolrCloudClusterTest >
> testSolrHomeAndResourceLoaders FAILED
> org.apache.solr.client.solrj.SolrServerException: IOException occurred
> when talking to server at: https://127.0.0.1:38681/solr
> at
> __randomizedtesting.SeedInfo.seed([806A85748BD81F48:37548FA7602CB5FD]:0)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:712)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:269)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:251)
> at
> org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:390)
> at
> org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:360)
> at
> org.apache.solr.client.solrj.impl.BaseCloudSolrClient.sendRequest(BaseCloudSolrClient.java:1168)
> at
> org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:931)
> at
> org.apache.solr.client.solrj.impl.BaseCloudSolrClient.request(BaseCloudSolrClient.java:865)
> at
> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:229)
> at
> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:246)
> at
> org.apache.solr.cloud.MiniSolrCloudClusterTest.testSolrHomeAndResourceLoaders(MiniSolrCloudClusterTest.java:125)
> ...
> Caused by:
> javax.net.ssl.SSLException: Unsupported or unrecognized SSL message
> at
> java.base/sun.security.ssl.SSLSocketInputRecord.handleUnknownRecord(SSLSocketInputRecord.java:439)
> {noformat}
> The problem sems to be that even though the MiniSolrCloudCluster being
> instantiated isn't _intentionally_ using any SSL randomization (it just uses
> {{JettyConfig.builder().build()}} the CloudSolrClient returned by
> {{cluster.getSolrClient()}} is evidnetly picking up the ranodmized SSL and
> trying to use it to talk to the cluster.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14987) SolrStream ends up creating a new HttpSolrClient for every replica being queried instead of reusing for the same node
[
https://issues.apache.org/jira/browse/SOLR-14987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timothy Potter updated SOLR-14987:
--
Fix Version/s: master (9.0)
8.8
Resolution: Fixed
Status: Resolved (was: Patch Available)
> SolrStream ends up creating a new HttpSolrClient for every replica being
> queried instead of reusing for the same node
> -
>
> Key: SOLR-14987
> URL: https://issues.apache.org/jira/browse/SOLR-14987
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: streaming expressions
>Reporter: Timothy Potter
>Assignee: Timothy Potter
>Priority: Major
> Fix For: 8.8, master (9.0)
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> Looking into some streaming expression performance issues when there are many
> collections with many shards being queried and I found that SolrStream's open
> method creates a new \{{HttpSolrClient}} for every replica being queried. For
> instance, in my test case, I have 10 collections with 100 shards each (rf=1)
> and I get 1000 HttpSolrClient instances in my SolrClientCache. If I reuse
> HttpSolrClient's per node hosting a replica (so 10 in my case), the query
> time for my expression drops by half (not too mention the reduced allocation
> load on the JVM).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-8673) o.a.s.search.facet classes not public/extendable
[
https://issues.apache.org/jira/browse/SOLR-8673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17245398#comment-17245398
]
Timothy Potter commented on SOLR-8673:
--
[~mkhl] it looks like this commit to 8x is causing precommit to fail, can you
take a look please? Probably just a matter of removing the `@Override`
annotation:
{code}
common.compile-test:
[javac] Compiling 1 source file to 8x/solr/build/solr-core/classes/test
[javac]
8x/solr/core/src/test/org/apache/solr/search/function/AggValueSourceTest.java:47:
error: method does not override or implement a method from a supertype
[javac] @Override
[javac] ^
[javac] 1 error
{code}
Also, looks like the status of this Jira needs to be updated, e.g. fixVersion,
etc.
> o.a.s.search.facet classes not public/extendable
>
>
> Key: SOLR-8673
> URL: https://issues.apache.org/jira/browse/SOLR-8673
> Project: Solr
> Issue Type: Improvement
> Components: Facet Module
>Affects Versions: 5.4.1
>Reporter: Markus Jelsma
>Priority: Major
> Fix For: 6.2, 7.0
>
> Attachments: SOLR-8673.patch, SOLR-8673.patch
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> It is not easy to create a custom JSON facet function. A simple function
> based on AvgAgg quickly results in the following compilation failures:
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.3:compile (default-compile)
> on project openindex-solr: Compilation failure: Compilation failure:
> [ERROR]
> /home/markus/projects/openindex/solr/trunk/src/main/java/i.o.s.search/facet/CustomAvgAgg.java:[22,36]
> org.apache.solr.search.facet.FacetContext is not public in
> org.apache.solr.search.facet; cannot be accessed from outside package
> [ERROR]
> /home/markus/projects/openindex/solr/trunk/src/main/java/i.o.s.search/facet/CustomAvgAgg.java:[23,36]
> org.apache.solr.search.facet.FacetDoubleMerger is not public in
> org.apache.solr.search.facet; cannot be accessed from outside package
> [ERROR]
> /home/markus/projects/openindex/solr/trunk/src/main/java/i.o.s.search/facet/CustomAvgAgg.java:[40,32]
> cannot find symbol
> [ERROR] symbol: class FacetContext
> [ERROR] location: class i.o.s.search.facet.CustomAvgAgg
> [ERROR]
> /home/markus/projects/openindex/solr/trunk/src/main/java/i.o.s.search/facet/CustomAvgAgg.java:[49,39]
> cannot find symbol
> [ERROR] symbol: class FacetDoubleMerger
> [ERROR] location: class i.o.s.search.facet.CustomAvgAgg
> [ERROR]
> /home/markus/projects/openindex/solr/trunk/src/main/java/i.o.s.search/facet/CustomAvgAgg.java:[54,43]
> cannot find symbol
> [ERROR] symbol: class Context
> [ERROR] location: class i.o.s.search.facet.CustomAvgAgg.Merger
> [ERROR]
> /home/markus/projects/openindex/solr/trunk/src/main/java/i.o.s.search/facet/CustomAvgAgg.java:[41,16]
> cannot find symbol
> [ERROR] symbol: class AvgSlotAcc
> [ERROR] location: class i.o.s.search.facet.CustomAvgAgg
> [ERROR]
> /home/markus/projects/openindex/solr/trunk/src/main/java/i.o.s.search/facet/CustomAvgAgg.java:[46,12]
> incompatible types: i.o.s.search.facet.CustomAvgAgg.Merger cannot be
> converted to org.apache.solr.search.facet.FacetMerger
> [ERROR]
> /home/markus/projects/openindex/solr/trunk/src/main/java/i.o.s.search/facet/CustomAvgAgg.java:[53,5]
> method does not override or implement a method from a supertype
> [ERROR]
> /home/markus/projects/openindex/solr/trunk/src/main/java/i.o.s.search/facet/CustomAvgAgg.java:[60,5]
> method does not override or implement a method from a supertype
> {code}
> It seems lots of classes are tucked away in FacetModule, which we can't reach
> from outside.
> Originates from this thread:
> http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201602.mbox/%3ccab_8yd9ldbg_0zxm_h1igkfm6bqeypd5ilyy7tty8cztscv...@mail.gmail.com%3E
> ( also available at
> https://lists.apache.org/thread.html/9fddcad3136ec908ce1c57881f8d3069e5d153f08b71f80f3e18d995%401455019826%40%3Csolr-user.lucene.apache.org%3E
> )
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Assigned] (SOLR-15026) MiniSolrCloudCluster can inconsistently get confused about when it's using SSL
[
https://issues.apache.org/jira/browse/SOLR-15026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timothy Potter reassigned SOLR-15026:
-
Assignee: Timothy Potter
> MiniSolrCloudCluster can inconsistently get confused about when it's using SSL
> --
>
> Key: SOLR-15026
> URL: https://issues.apache.org/jira/browse/SOLR-15026
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Chris M. Hostetter
>Assignee: Timothy Potter
>Priority: Major
>
> A new test added in SOLR-14934 caused the following reproducible failure to
> pop up on jenkins...
> {noformat}
> hossman@slate:~/lucene/dev [j11] [master] $ ./gradlew -p solr/test-framework/
> test --tests MiniSolrCloudClusterTest.testSolrHomeAndResourceLoaders
> -Dtests.seed=806A85748BD81F48 -Dtests.multiplier=2 -Dtests.slow=true
> -Dtests.locale=ln-CG -Dtests.timezone=Asia/Thimbu -Dtests.asserts=true
> -Dtests.file.encoding=UTF-8
> Starting a Gradle Daemon (subsequent builds will be faster)
> > Task :randomizationInfo
> Running tests with randomization seed: tests.seed=806A85748BD81F48
> > Task :solr:test-framework:test
> org.apache.solr.cloud.MiniSolrCloudClusterTest >
> testSolrHomeAndResourceLoaders FAILED
> org.apache.solr.client.solrj.SolrServerException: IOException occurred
> when talking to server at: https://127.0.0.1:38681/solr
> at
> __randomizedtesting.SeedInfo.seed([806A85748BD81F48:37548FA7602CB5FD]:0)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:712)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:269)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:251)
> at
> org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:390)
> at
> org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:360)
> at
> org.apache.solr.client.solrj.impl.BaseCloudSolrClient.sendRequest(BaseCloudSolrClient.java:1168)
> at
> org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:931)
> at
> org.apache.solr.client.solrj.impl.BaseCloudSolrClient.request(BaseCloudSolrClient.java:865)
> at
> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:229)
> at
> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:246)
> at
> org.apache.solr.cloud.MiniSolrCloudClusterTest.testSolrHomeAndResourceLoaders(MiniSolrCloudClusterTest.java:125)
> ...
> Caused by:
> javax.net.ssl.SSLException: Unsupported or unrecognized SSL message
> at
> java.base/sun.security.ssl.SSLSocketInputRecord.handleUnknownRecord(SSLSocketInputRecord.java:439)
> {noformat}
> The problem sems to be that even though the MiniSolrCloudCluster being
> instantiated isn't _intentionally_ using any SSL randomization (it just uses
> {{JettyConfig.builder().build()}} the CloudSolrClient returned by
> {{cluster.getSolrClient()}} is evidnetly picking up the ranodmized SSL and
> trying to use it to talk to the cluster.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-15026) MiniSolrCloudCluster can inconsistently get confused about when it's using SSL
[
https://issues.apache.org/jira/browse/SOLR-15026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17243396#comment-17243396
]
Timothy Potter commented on SOLR-15026:
---
I've been all over this code recently, so will take this up unless you've
already started on it [~hossman]
> MiniSolrCloudCluster can inconsistently get confused about when it's using SSL
> --
>
> Key: SOLR-15026
> URL: https://issues.apache.org/jira/browse/SOLR-15026
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Chris M. Hostetter
>Assignee: Timothy Potter
>Priority: Major
>
> A new test added in SOLR-14934 caused the following reproducible failure to
> pop up on jenkins...
> {noformat}
> hossman@slate:~/lucene/dev [j11] [master] $ ./gradlew -p solr/test-framework/
> test --tests MiniSolrCloudClusterTest.testSolrHomeAndResourceLoaders
> -Dtests.seed=806A85748BD81F48 -Dtests.multiplier=2 -Dtests.slow=true
> -Dtests.locale=ln-CG -Dtests.timezone=Asia/Thimbu -Dtests.asserts=true
> -Dtests.file.encoding=UTF-8
> Starting a Gradle Daemon (subsequent builds will be faster)
> > Task :randomizationInfo
> Running tests with randomization seed: tests.seed=806A85748BD81F48
> > Task :solr:test-framework:test
> org.apache.solr.cloud.MiniSolrCloudClusterTest >
> testSolrHomeAndResourceLoaders FAILED
> org.apache.solr.client.solrj.SolrServerException: IOException occurred
> when talking to server at: https://127.0.0.1:38681/solr
> at
> __randomizedtesting.SeedInfo.seed([806A85748BD81F48:37548FA7602CB5FD]:0)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:712)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:269)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:251)
> at
> org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:390)
> at
> org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:360)
> at
> org.apache.solr.client.solrj.impl.BaseCloudSolrClient.sendRequest(BaseCloudSolrClient.java:1168)
> at
> org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:931)
> at
> org.apache.solr.client.solrj.impl.BaseCloudSolrClient.request(BaseCloudSolrClient.java:865)
> at
> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:229)
> at
> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:246)
> at
> org.apache.solr.cloud.MiniSolrCloudClusterTest.testSolrHomeAndResourceLoaders(MiniSolrCloudClusterTest.java:125)
> ...
> Caused by:
> javax.net.ssl.SSLException: Unsupported or unrecognized SSL message
> at
> java.base/sun.security.ssl.SSLSocketInputRecord.handleUnknownRecord(SSLSocketInputRecord.java:439)
> {noformat}
> The problem sems to be that even though the MiniSolrCloudCluster being
> instantiated isn't _intentionally_ using any SSL randomization (it just uses
> {{JettyConfig.builder().build()}} the CloudSolrClient returned by
> {{cluster.getSolrClient()}} is evidnetly picking up the ranodmized SSL and
> trying to use it to talk to the cluster.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-12182) Can not switch urlScheme in 7x if there are any cores in the cluster
[ https://issues.apache.org/jira/browse/SOLR-12182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17242826#comment-17242826 ] Timothy Potter commented on SOLR-12182: --- Yes, I'm aware of that [~hossman] ... wasn't going to backport this given the scope but changed my mind. I'll fix CHANGES.txt in master > Can not switch urlScheme in 7x if there are any cores in the cluster > > > Key: SOLR-12182 > URL: https://issues.apache.org/jira/browse/SOLR-12182 > Project: Solr > Issue Type: Bug >Affects Versions: 7.0, 7.1, 7.2 >Reporter: Anshum Gupta >Assignee: Timothy Potter >Priority: Major > Fix For: 8.8, master (9.0) > > Attachments: SOLR-12182.patch, SOLR-12182_20200423.patch > > Time Spent: 5.5h > Remaining Estimate: 0h > > I was trying to enable TLS on a cluster that was already in use i.e. had > existing collections and ended up with down cores, that wouldn't come up and > the following core init errors in the logs: > *org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: > replica with coreNodeName core_node4 exists but with a different name or > base_url.* > What is happening here is that the core/replica is defined in the > clusterstate with the urlScheme as part of it's base URL e.g. > *"base_url":"http:hostname:port/solr"*. > Switching the urlScheme in Solr breaks this convention as the host now uses > HTTPS instead. > Actually, I ran into this with an older version because I was running with > *legacyCloud=false* and then realized that we switched that to the default > behavior only in 7x i.e while most users did not hit this issue with older > versions, unless they overrode the legacyCloud value explicitly, users > running 7x are bound to run into this more often. > Switching the value of legacyCloud to true, bouncing the cluster so that the > clusterstate gets flushed, and then setting it back to false is a workaround > but a bit risky one if you don't know if you have any old cores lying around. > Ideally, I think we shouldn't prepend the urlScheme to the base_url value and > use the urlScheme on the fly to construct it. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
