[jira] [Commented] (SOLR-14610) ReflectMapWriter to use VarHandle instead of old legacy reflection
[ https://issues.apache.org/jira/browse/SOLR-14610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17154210#comment-17154210 ] Noble Paul commented on SOLR-14610: --- Now, it is all cached in a static Map. Which means it is a lot faster that doing reflection all over again. So, in the cached object, the VarHandle is resolved and stored > ReflectMapWriter to use VarHandle instead of old legacy reflection > -- > > Key: SOLR-14610 > URL: https://issues.apache.org/jira/browse/SOLR-14610 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Noble Paul >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > > The same reason why we changed to MethodHandles in SOLR-14404 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13989) Move all hadoop related code to a contrib module
[
https://issues.apache.org/jira/browse/SOLR-13989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17154207#comment-17154207
]
David Smiley commented on SOLR-13989:
-
The package manager might not currently be able to load certain things
mentioned above but my approach in
[https://github.com/apache/lucene-solr/pull/1109] made it near universal what
it could load. While that particular PR got into a tug of war of how to do
things, I hope to resurrect another incarnation of it after some further
SolrResourceLoader refactorings I'll undergo. That may then unblock the issue
here (HDFS).
> Move all hadoop related code to a contrib module
>
>
> Key: SOLR-13989
> URL: https://issues.apache.org/jira/browse/SOLR-13989
> Project: Solr
> Issue Type: Task
> Components: Hadoop Integration
>Reporter: Shalin Shekhar Mangar
>Priority: Major
> Fix For: master (9.0)
>
>
> Spin off from SOLR-13986:
> {quote}
> It seems really important to move or remove this hadoop shit out of the solr
> core: It is really unreasonable that solr core depends on hadoop. that's
> gonna simply block any progress improving its security, because solr code
> will get dragged down by hadoop's code.
> {quote}
> We should move all hadoop related dependencies to a separate contrib module
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14610) ReflectMapWriter to use VarHandle instead of old legacy reflection
[ https://issues.apache.org/jira/browse/SOLR-14610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17154205#comment-17154205 ] David Smiley commented on SOLR-14610: - Since ReflectMapWriter does all logic on demand, I wonder if there is any perf benefit from VarHandle stuff. In SOLR-14404 the situation was different because the VarHandle was looked up only once and re-used, saved in the AnnotatedApi.Cmd.method. CC [~uschindler] > ReflectMapWriter to use VarHandle instead of old legacy reflection > -- > > Key: SOLR-14610 > URL: https://issues.apache.org/jira/browse/SOLR-14610 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Noble Paul >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > > The same reason why we changed to MethodHandles in SOLR-14404 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13132) Improve JSON "terms" facet performance when sorted by relatedness
[
https://issues.apache.org/jira/browse/SOLR-13132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17154192#comment-17154192
]
Chris M. Hostetter commented on SOLR-13132:
---
Hey Michael, just to sanity check one thing...
{quote}The results below are mostly for without any filterCache,...
{quote}
Are the {{MASTER}} results you reported here from testing with
"filterCacheSize=0" ... which would make them an apples to apples comparison
with your {{"SOLR-13132 sweep_collection=false, filterCacheSize=0}} " results?
(i'm guessing so based on your comment above, but i don't see any sort of log
in your tgz showing exactly what commands were used to run each test, and i see
a default size of 4096 in your solrconfig.xml, so i wanted to be certain what
you ment)
Assuming i'm understanding correctly, then what i see here is what we expected:
for the "common" case of sorting on {{relatedness()}} (high cardinality fields
– larger then filterCache size – w/ non-trivially FG sets) sweeping is big
improvement, and in the _uncommon_ case of low cardinality fields and/or small
FG sets there is only a small (negative) impact of using the new code – and
that seems to go away (numbers close enough to be noise) by specyifing
"{{sweep_collection:false}} ".
sound right?
So i think we're good to go – we just need the corrections/updates to the ref
guide. If you can please remove the incorrect stuff i mentioned in my last
comment, and replace it with a note explaining when/why people _may_ want to
consider use {{sweep_collection: false}} (ie: atypical SKG situations where
cardinality or FG set size is very low) I'll try to merge & backport ASAP.
> Improve JSON "terms" facet performance when sorted by relatedness
> --
>
> Key: SOLR-13132
> URL: https://issues.apache.org/jira/browse/SOLR-13132
> Project: Solr
> Issue Type: Improvement
> Components: Facet Module
>Affects Versions: 7.4, master (9.0)
>Reporter: Michael Gibney
>Priority: Major
> Attachments: SOLR-13132-benchmarks.tgz,
> SOLR-13132-with-cache-01.patch, SOLR-13132-with-cache.patch,
> SOLR-13132.patch, SOLR-13132_testSweep.patch
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> When sorting buckets by {{relatedness}}, JSON "terms" facet must calculate
> {{relatedness}} for every term.
> The current implementation uses a standard uninverted approach (either
> {{docValues}} or {{UnInvertedField}}) to get facet counts over the domain
> base docSet, and then uses that initial pass as a pre-filter for a
> second-pass, inverted approach of fetching docSets for each relevant term
> (i.e., {{count > minCount}}?) and calculating intersection size of those sets
> with the domain base docSet.
> Over high-cardinality fields, the overhead of per-term docSet creation and
> set intersection operations increases request latency to the point where
> relatedness sort may not be usable in practice (for my use case, even after
> applying the patch for SOLR-13108, for a field with ~220k unique terms per
> core, QTime for high-cardinality domain docSets were, e.g.: cardinality
> 1816684=9000ms, cardinality 5032902=18000ms).
> The attached patch brings the above example QTimes down to a manageable
> ~300ms and ~250ms respectively. The approach calculates uninverted facet
> counts over domain base, foreground, and background docSets in parallel in a
> single pass. This allows us to take advantage of the efficiencies built into
> the standard uninverted {{FacetFieldProcessorByArray[DV|UIF]}}), and avoids
> the per-term docSet creation and set intersection overhead.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14638) Edismax boost function zero score
[
https://issues.apache.org/jira/browse/SOLR-14638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17154187#comment-17154187
]
David Smiley commented on SOLR-14638:
-
I browsed through the commits between 7.7.1 and 7.7.2 and the one that seems
like it could be related is this: SOLR-13126
You might have to use \{{if(exists(field),field,0)}} or similar.
> Edismax boost function zero score
> -
>
> Key: SOLR-14638
> URL: https://issues.apache.org/jira/browse/SOLR-14638
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 7.7.1, 7.7.2
> Environment: I am using the [https://hub.docker.com/_/solr] image for
> 7.7.1 and 7.7.2 versions without any major changes.
>Reporter: Victor Zharikov
>Priority: Critical
>
> I'm using edismax to boost a score.
> Using the 'field(field_name)' function's boost, I multiply the points by the
> value of this field.
> Its behavior without any patch notes became noticeably different between
> 7.7.1 and 7.7.2.
> On version 7.7.1, the record for which the 'field_name' field is not filled
> has regular score. And the record score with field_name = 2.0, for example,
> is multiplied by two.
> On version 7.7.2, a record with the field 'field_name' not filled in has ZERO
> score. And for the record with field_name = 2.0, the points are still
> multiplied by two from the normal result.
> It completely breaks the score ranking.
> Example .
> Version 7.7.1
> no field_name "score": 32.586094
> field_name = 2.0 "score": 65.17219
> no field_name "score": 0.0
> field_name = 2.0 "score": 65.17219
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14636) Provide a reference implementation for SolrCloud that is stable and fast.
[ https://issues.apache.org/jira/browse/SOLR-14636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17154182#comment-17154182 ] Mark Robert Miller commented on SOLR-14636: --- Okay, I've gotten around that I think. I'll push current state in the morning. The Solr Core tests are essentially _damn fast_ (they can be still be faster) HOwever, there is some work to do around hardening, both while running via single JVM as well as lots of various parallel JVM runs. So there are some bugs to find, no doubt, but that's really the whole point of this stage in the code. Startup and shutdown *rip*. And like a high speed train, things go bad unless the rail is solid. > Provide a reference implementation for SolrCloud that is stable and fast. > - > > Key: SOLR-14636 > URL: https://issues.apache.org/jira/browse/SOLR-14636 > Project: Solr > Issue Type: Task > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Mark Robert Miller >Assignee: Mark Robert Miller >Priority: Major > > SolrCloud powers critical infrastructure and needs the ability to run quickly > with stability. This reference implementation will allow for this. > > *status*: alpha > *tests***: > * *core*: passing with ignores (not solid*) > * *solrj*: tbd > * *test-framework*: tbd > * *contrib/analysis-extras*: tbd > * *contrib/analytics*: tbd > * *contrib/clustering*: tbd > * *contrib/dataimporthandler*: tbd > * *contrib/dataimporthandler-extras*: tbd > * *contrib/extraction*: tbd > * *contrib/jaegertracer-configurator*: tbd > * *contrib/langid*: tbd > * *contrib/prometheus-exporter*: tbd > * *contrib/velocity*: tbd > _* Running tests quickly and efficiently with strict policing will more > frequently find bugs and requires a period of hardening._ > _** Non Nightly currently, Nightly comes last._ -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14599) Introduce cluster level plugins through package manager
[
https://issues.apache.org/jira/browse/SOLR-14599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17154180#comment-17154180
]
David Smiley commented on SOLR-14599:
-
Ishan: BTW the "manual" instructions need not imply hand-edit of configs; using
the config API to accomplish the same applies as here as well. You might then
ask why doesn't the package include those self-configure instructions. The
answer is when those instructions aren't suitable to the user. For example
anything involving schema analysis chains usually is something customized /
integrated with existing choices carefully.
{quote}The {{package-manager.adoc}} is specifically mean to contain the CLI
commands
{quote}
Oh; I didn't know that. That's kind of a limiting scope but I can see that the
majority of the content relates to the CLI; Security is an exception. Still;
"internals" feels wrong, and yet another page seems a bit much for just this.
What if this information was added to the very bottom and was clearly
delineated? Basically like how the Security part is added.
> Introduce cluster level plugins through package manager
> ---
>
> Key: SOLR-14599
> URL: https://issues.apache.org/jira/browse/SOLR-14599
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Package Manager
>Reporter: Ishan Chattopadhyaya
>Assignee: Ishan Chattopadhyaya
>Priority: Blocker
> Fix For: 8.6
>
> Attachments: SOLR-14599.patch
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> SOLR-14404 made it possible to write request handlers that are registered at
> core container level. This makes it possible for packages to have two types
> of plugins:
> # Collection level
> # Cluster level
> This issue intends to introduce the latter via package manager. The manifest
> for a package will now specify the type of the plugin. Such plugins can be
> deployed directly without specifying a collection.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14636) Provide a reference implementation for SolrCloud that is stable and fast.
[ https://issues.apache.org/jira/browse/SOLR-14636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17154138#comment-17154138 ] Mark Robert Miller commented on SOLR-14636: --- Ugh, startup got a little too fast and exposed a couple new holes. Small delay. > Provide a reference implementation for SolrCloud that is stable and fast. > - > > Key: SOLR-14636 > URL: https://issues.apache.org/jira/browse/SOLR-14636 > Project: Solr > Issue Type: Task > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Mark Robert Miller >Assignee: Mark Robert Miller >Priority: Major > > SolrCloud powers critical infrastructure and needs the ability to run quickly > with stability. This reference implementation will allow for this. > > *status*: alpha > *tests***: > * *core*: passing with ignores (not solid*) > * *solrj*: tbd > * *test-framework*: tbd > * *contrib/analysis-extras*: tbd > * *contrib/analytics*: tbd > * *contrib/clustering*: tbd > * *contrib/dataimporthandler*: tbd > * *contrib/dataimporthandler-extras*: tbd > * *contrib/extraction*: tbd > * *contrib/jaegertracer-configurator*: tbd > * *contrib/langid*: tbd > * *contrib/prometheus-exporter*: tbd > * *contrib/velocity*: tbd > _* Running tests quickly and efficiently with strict policing will more > frequently find bugs and requires a period of hardening._ > _** Non Nightly currently, Nightly comes last._ -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-14469) Removed deprecated code in solr/core (master only)
[
https://issues.apache.org/jira/browse/SOLR-14469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17154040#comment-17154040
]
Uwe Schindler edited comment on SOLR-14469 at 7/8/20, 9:59 PM:
---
Adding {{Xlint:deprecation}} makes sense for a job like the one here: remove
deprecations before a relaese. But it should not be a general flag enabled for
javac, as it makes adding new deprecations also close to impossible.
Maybe add something like a gradle flag "-Ddisallow.deprecations=true" that can
be used for the person who wants to cleanup deprecations before a major version.
was (Author: thetaphi):
Adding {{Xlint:deprecation}} makes sense for a job like the one here: remove
deprecations before a relaese. But it should not be a general flag enabled for
javac, as it makes adding new deprecations also close to impossible.
> Removed deprecated code in solr/core (master only)
> --
>
> Key: SOLR-14469
> URL: https://issues.apache.org/jira/browse/SOLR-14469
> Project: Solr
> Issue Type: Sub-task
>Reporter: Erick Erickson
>Assignee: Erick Erickson
>Priority: Major
>
> I'm currently working on getting all the warnings out of the code, so this is
> something of a placeholder for a week or two.
> There will be sub-tasks, please create them when you start working on a
> project.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14469) Removed deprecated code in solr/core (master only)
[
https://issues.apache.org/jira/browse/SOLR-14469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17154040#comment-17154040
]
Uwe Schindler commented on SOLR-14469:
--
Adding {{Xlint:deprecation}} makes sense for a job like the one here: remove
deprecations before a relaese. But it should not be a general flag enabled for
javac, as it makes adding new deprecations also close to impossible.
> Removed deprecated code in solr/core (master only)
> --
>
> Key: SOLR-14469
> URL: https://issues.apache.org/jira/browse/SOLR-14469
> Project: Solr
> Issue Type: Sub-task
>Reporter: Erick Erickson
>Assignee: Erick Erickson
>Priority: Major
>
> I'm currently working on getting all the warnings out of the code, so this is
> something of a placeholder for a week or two.
> There will be sub-tasks, please create them when you start working on a
> project.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14469) Removed deprecated code in solr/core (master only)
[
https://issues.apache.org/jira/browse/SOLR-14469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17154037#comment-17154037
]
Uwe Schindler commented on SOLR-14469:
--
The negatives are currently no-ops, because we only explicitely enable warnings.
I don't think (as David Smiley also mentioned), that {{Xlint:deprecation}}
(enable!!!) should be added, because this makes usage of Solr's own
deprecations impossible. This is by the way one reason why we have
forbiddenapis and its "jdk-deprecated" signatures file. It differs from this
warning in a way that only calls to Java are forbidden, but you still can call
all other code marked as deprecated, including your own.
So in short: Use forbiddenapis to explicitely disable deprecated stuff (like
done with jdk's deprecations). On the Solr/Lucene API side, we should have
clear rules how the message looks like. But this more something for the
source-patterns checker.
> Removed deprecated code in solr/core (master only)
> --
>
> Key: SOLR-14469
> URL: https://issues.apache.org/jira/browse/SOLR-14469
> Project: Solr
> Issue Type: Sub-task
>Reporter: Erick Erickson
>Assignee: Erick Erickson
>Priority: Major
>
> I'm currently working on getting all the warnings out of the code, so this is
> something of a placeholder for a week or two.
> There will be sub-tasks, please create them when you start working on a
> project.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14469) Removed deprecated code in solr/core (master only)
[ https://issues.apache.org/jira/browse/SOLR-14469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17154030#comment-17154030 ] Erick Erickson commented on SOLR-14469: --- Remember to take out negatives from defaults-java.gradle > Removed deprecated code in solr/core (master only) > -- > > Key: SOLR-14469 > URL: https://issues.apache.org/jira/browse/SOLR-14469 > Project: Solr > Issue Type: Sub-task >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Major > > I'm currently working on getting all the warnings out of the code, so this is > something of a placeholder for a week or two. > There will be sub-tasks, please create them when you start working on a > project. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9411) Fail complation on warnings, 9x gradle-only
[ https://issues.apache.org/jira/browse/LUCENE-9411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17154029#comment-17154029 ] Erick Erickson commented on LUCENE-9411: True, I can remove the negatives, I'll deal with that in SOLR-14469 > Fail complation on warnings, 9x gradle-only > --- > > Key: LUCENE-9411 > URL: https://issues.apache.org/jira/browse/LUCENE-9411 > Project: Lucene - Core > Issue Type: Improvement > Components: general/build >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Major > Labels: build > Fix For: master (9.0) > > Attachments: LUCENE-9411-explicit-warnings.patch, LUCENE-9411.patch, > LUCENE-9411.patch, LUCENE-9411.patch, LUCENE-9411.patch, > annotations-warnings.patch > > > Moving this over here from SOLR-11973 since it's part of the build system and > affects Lucene as well as Solr. You might want to see the discussion there. > We have a clean compile for both Solr and Lucene, no rawtypes, unchecked, > try, etc. warnings. There are some peculiar warnings (things like > SuppressFBWarnings, i.e. FindBugs) that I'm not sure about at all, but let's > assume those are not a problem. Now I'd like to start failing the compilation > if people write new code that generates warnings. > From what I can tell, just adding the flag is easy in both the Gradle and Ant > builds. I still have to prove out that adding -Werrors does what I expect, > i.e. succeeds now and fails when I introduce warnings. > But let's assume that works. Are there objections to this idea generally? I > hope to have some data by next Monday. > FWIW, the Lucene code base had far fewer issues than Solr, but > common-build.xml is in Lucene. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13132) Improve JSON "terms" facet performance when sorted by relatedness
[
https://issues.apache.org/jira/browse/SOLR-13132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17153994#comment-17153994
]
Michael Gibney commented on SOLR-13132:
---
I just attached [^SOLR-13132-benchmarks.tgz], some naive/sanity-check
benchmarks. The results below are mostly for without any filterCache, but I
included hooks in the included scripts to easily change the filterCache size
for evaluation. For purpose of this evaluation, fgSet == base search result
domain. All results discussed here are for single-valued string fields, but
multivalued string fields are also included in the benchmark attachment
(results for multi-valued didn't differ substantially from those for
single-valued).
There's a row for each docset domain recall percentage (percentage of \*:*
domain returned by main query/fg), and a column for each field cardinality;
cell values indicate latency (QTime) in ms against a single core with 3 million
docs, no deletes; each value is the average of 10 repeated invocations of the
the relevant request (standard deviation isn't captured here, but was quite
low, fwiw).
{code:java}
MASTER:
cdnlty: 10 100 1k 10k 100k1m
.1% 23 32 75 124 118 109
1% 28 43 98 412 1057582
10% 78 90 135 430 32915530
20% 139 151 192 484 33537610
30% 198 207 250 538 34629537
40% 253 265 307 595 347411206
50% 321 341 374 655 349713155
{code}
{code:java}
SOLR-13132 sweep_collection=false, filterCacheSize=0
cdnlty: 10 100 1k 10k 100k1m
.1% 24 37 74 119 108 74
1% 29 44 97 403 1021563
10% 77 92 136 417 31615356
20% 144 156 197 486 32337257
30% 199 209 254 534 33229224
40% 254 276 314 599 339310937
50% 323 352 368 643 340312718
{code}
{code:java}
SOLR-13132 sweep_collection=true
cdnlty: 10 100 1k 10k 100k1m
.1% 99 99 106 111 114 145
1% 102 106 112 110 122 157
10% 168 173 177 175 193 241
20% 241 245 249 249 266 341
30% 307 312 318 313 337 409
40% 382 386 390 390 414 494
50% 449 455 459 460 487 569
{code}
{code:java}
SOLR-13132 sweep_collection=false, filterCacheSize=12000 (very large!)
.1% 21 17 33 45 37 57
1% 7 17 41 270 885 525
10% 35 44 57 375 32395866
20% 71 78 89 204 33247800
30% 97 108 120 220 33359684
40% 130 141 152 248 325811582
50% 159 171 183 270 329913754
{code}
The last of these is with a _very_ large filterCache configured. It pretty
clearly benefits cardinality through 10k, but has a slight negative impact on
100k and above (filterCache overhead with no benefit). This is as expected;
it's worth noting that in real-world situations, filterCache is likely to be
smaller and also used by other features, so this test probably underestimates
the negative system-wide impact of an undersized filterCache, and also
underestimates the sufficient filterCache size threshold wrt field cardinality.
Because there were low-level changes to the code to collect counts, I also did
a sanity check comparing simple facet term count performance (no skg) of master
wrt SOLR-13132. I won't post the results in line here, but there appeared to be
no change whatsoever. The results are included in the attached tar file (under
the "results" directory).
Side note: the negative performance impact of sweep collection for
low-cardinality docsets is mainly due to the fact that the full count
accumulation domain (for sweep collection) becomes the union of the result
DocSet, fg DocSet, and bg DocSet. Where bg DocSet is often large (e.g., \*:*),
we're essentially just seeing the effect of collecting term facet counts over a
high cardinality DocSet domain. As a contrived example, setting an artificially
restricted bgSet (e.g., {{\{!prefix f=id v=99}}}) is considerably faster:
{code:java}
SOLR-13132 sweep_collection=true, tiny bgSet
cdnlty: 10 100 1k 10k 100k1m
.1% 2 1 1 2 2 8
1% 4 6 6 6 9 17
10% 50 48 46 47 55 78
20% 91 92 93 93 104 146
30% 135 137 138 141 152 198
40% 180 182 185 186 198 253
50% 223 226 228 229 245 316
{code}
> Improve JSON "terms" facet
[jira] [Commented] (LUCENE-9411) Fail complation on warnings, 9x gradle-only
[ https://issues.apache.org/jira/browse/LUCENE-9411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17153983#comment-17153983 ] Uwe Schindler commented on LUCENE-9411: --- I think, whenever we change release version, we have to add new warning types and analyze how it fails. I am sure the new warnings are less serious. It will be mostly stuff like missing break in try. I would like to figure out how Werror or is affected by doclint stuff. > Fail complation on warnings, 9x gradle-only > --- > > Key: LUCENE-9411 > URL: https://issues.apache.org/jira/browse/LUCENE-9411 > Project: Lucene - Core > Issue Type: Improvement > Components: general/build >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Major > Labels: build > Fix For: master (9.0) > > Attachments: LUCENE-9411-explicit-warnings.patch, LUCENE-9411.patch, > LUCENE-9411.patch, LUCENE-9411.patch, LUCENE-9411.patch, > annotations-warnings.patch > > > Moving this over here from SOLR-11973 since it's part of the build system and > affects Lucene as well as Solr. You might want to see the discussion there. > We have a clean compile for both Solr and Lucene, no rawtypes, unchecked, > try, etc. warnings. There are some peculiar warnings (things like > SuppressFBWarnings, i.e. FindBugs) that I'm not sure about at all, but let's > assume those are not a problem. Now I'd like to start failing the compilation > if people write new code that generates warnings. > From what I can tell, just adding the flag is easy in both the Gradle and Ant > builds. I still have to prove out that adding -Werrors does what I expect, > i.e. succeeds now and fails when I introduce warnings. > But let's assume that works. Are there objections to this idea generally? I > hope to have some data by next Monday. > FWIW, the Lucene code base had far fewer issues than Solr, but > common-build.xml is in Lucene. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-9411) Fail complation on warnings, 9x gradle-only
[ https://issues.apache.org/jira/browse/LUCENE-9411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uwe Schindler resolved LUCENE-9411. --- Resolution: Fixed > Fail complation on warnings, 9x gradle-only > --- > > Key: LUCENE-9411 > URL: https://issues.apache.org/jira/browse/LUCENE-9411 > Project: Lucene - Core > Issue Type: Improvement > Components: general/build >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Major > Labels: build > Fix For: master (9.0) > > Attachments: LUCENE-9411-explicit-warnings.patch, LUCENE-9411.patch, > LUCENE-9411.patch, LUCENE-9411.patch, LUCENE-9411.patch, > annotations-warnings.patch > > > Moving this over here from SOLR-11973 since it's part of the build system and > affects Lucene as well as Solr. You might want to see the discussion there. > We have a clean compile for both Solr and Lucene, no rawtypes, unchecked, > try, etc. warnings. There are some peculiar warnings (things like > SuppressFBWarnings, i.e. FindBugs) that I'm not sure about at all, but let's > assume those are not a problem. Now I'd like to start failing the compilation > if people write new code that generates warnings. > From what I can tell, just adding the flag is easy in both the Gradle and Ant > builds. I still have to prove out that adding -Werrors does what I expect, > i.e. succeeds now and fails when I introduce warnings. > But let's assume that works. Are there objections to this idea generally? I > hope to have some data by next Monday. > FWIW, the Lucene code base had far fewer issues than Solr, but > common-build.xml is in Lucene. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9411) Fail complation on warnings, 9x gradle-only
[ https://issues.apache.org/jira/browse/LUCENE-9411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17153970#comment-17153970 ] Uwe Schindler commented on LUCENE-9411: --- Looks OK. The negatives can now be removed (deprecation and serial). > Fail complation on warnings, 9x gradle-only > --- > > Key: LUCENE-9411 > URL: https://issues.apache.org/jira/browse/LUCENE-9411 > Project: Lucene - Core > Issue Type: Improvement > Components: general/build >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Major > Labels: build > Fix For: master (9.0) > > Attachments: LUCENE-9411-explicit-warnings.patch, LUCENE-9411.patch, > LUCENE-9411.patch, LUCENE-9411.patch, LUCENE-9411.patch, > annotations-warnings.patch > > > Moving this over here from SOLR-11973 since it's part of the build system and > affects Lucene as well as Solr. You might want to see the discussion there. > We have a clean compile for both Solr and Lucene, no rawtypes, unchecked, > try, etc. warnings. There are some peculiar warnings (things like > SuppressFBWarnings, i.e. FindBugs) that I'm not sure about at all, but let's > assume those are not a problem. Now I'd like to start failing the compilation > if people write new code that generates warnings. > From what I can tell, just adding the flag is easy in both the Gradle and Ant > builds. I still have to prove out that adding -Werrors does what I expect, > i.e. succeeds now and fails when I introduce warnings. > But let's assume that works. Are there objections to this idea generally? I > hope to have some data by next Monday. > FWIW, the Lucene code base had far fewer issues than Solr, but > common-build.xml is in Lucene. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9411) Fail complation on warnings, 9x gradle-only
[ https://issues.apache.org/jira/browse/LUCENE-9411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17153969#comment-17153969 ] Erick Erickson commented on LUCENE-9411: [~uschindler] Ok, I pushed the change. If that's what you had in mind, please go ahead and close this again. And thanks for catching this, I never thought of it... > Fail complation on warnings, 9x gradle-only > --- > > Key: LUCENE-9411 > URL: https://issues.apache.org/jira/browse/LUCENE-9411 > Project: Lucene - Core > Issue Type: Improvement > Components: general/build >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Major > Labels: build > Fix For: master (9.0) > > Attachments: LUCENE-9411-explicit-warnings.patch, LUCENE-9411.patch, > LUCENE-9411.patch, LUCENE-9411.patch, LUCENE-9411.patch, > annotations-warnings.patch > > > Moving this over here from SOLR-11973 since it's part of the build system and > affects Lucene as well as Solr. You might want to see the discussion there. > We have a clean compile for both Solr and Lucene, no rawtypes, unchecked, > try, etc. warnings. There are some peculiar warnings (things like > SuppressFBWarnings, i.e. FindBugs) that I'm not sure about at all, but let's > assume those are not a problem. Now I'd like to start failing the compilation > if people write new code that generates warnings. > From what I can tell, just adding the flag is easy in both the Gradle and Ant > builds. I still have to prove out that adding -Werrors does what I expect, > i.e. succeeds now and fails when I introduce warnings. > But let's assume that works. Are there objections to this idea generally? I > hope to have some data by next Monday. > FWIW, the Lucene code base had far fewer issues than Solr, but > common-build.xml is in Lucene. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9411) Fail complation on warnings, 9x gradle-only
[ https://issues.apache.org/jira/browse/LUCENE-9411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17153965#comment-17153965 ] ASF subversion and git services commented on LUCENE-9411: - Commit 294caa81540f169d537a98c952a39d124db102b2 in lucene-solr's branch refs/heads/master from Erick Erickson [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=294caa8 ] LUCENE-9411: Fail complation on warnings, 9x gradle-only. Explicitly list warnings to check for > Fail complation on warnings, 9x gradle-only > --- > > Key: LUCENE-9411 > URL: https://issues.apache.org/jira/browse/LUCENE-9411 > Project: Lucene - Core > Issue Type: Improvement > Components: general/build >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Major > Labels: build > Fix For: master (9.0) > > Attachments: LUCENE-9411-explicit-warnings.patch, LUCENE-9411.patch, > LUCENE-9411.patch, LUCENE-9411.patch, LUCENE-9411.patch, > annotations-warnings.patch > > > Moving this over here from SOLR-11973 since it's part of the build system and > affects Lucene as well as Solr. You might want to see the discussion there. > We have a clean compile for both Solr and Lucene, no rawtypes, unchecked, > try, etc. warnings. There are some peculiar warnings (things like > SuppressFBWarnings, i.e. FindBugs) that I'm not sure about at all, but let's > assume those are not a problem. Now I'd like to start failing the compilation > if people write new code that generates warnings. > From what I can tell, just adding the flag is easy in both the Gradle and Ant > builds. I still have to prove out that adding -Werrors does what I expect, > i.e. succeeds now and fails when I introduce warnings. > But let's assume that works. Are there objections to this idea generally? I > hope to have some data by next Monday. > FWIW, the Lucene code base had far fewer issues than Solr, but > common-build.xml is in Lucene. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-14638) Edismax boost function zero score
Victor Zharikov created SOLR-14638: -- Summary: Edismax boost function zero score Key: SOLR-14638 URL: https://issues.apache.org/jira/browse/SOLR-14638 Project: Solr Issue Type: Bug Security Level: Public (Default Security Level. Issues are Public) Affects Versions: 7.7.2, 7.7.1 Environment: I am using the [https://hub.docker.com/_/solr] image for 7.7.1 and 7.7.2 versions without any major changes. Reporter: Victor Zharikov I'm using edismax to boost a score. Using the 'field(field_name)' function's boost, I multiply the points by the value of this field. Its behavior without any patch notes became noticeably different between 7.7.1 and 7.7.2. On version 7.7.1, the record for which the 'field_name' field is not filled has regular score. And the record score with field_name = 2.0, for example, is multiplied by two. On version 7.7.2, a record with the field 'field_name' not filled in has ZERO score. And for the record with field_name = 2.0, the points are still multiplied by two from the normal result. It completely breaks the score ranking. Example . Version 7.7.1 no field_name "score": 32.586094 field_name = 2.0 "score": 65.17219 no field_name "score": 0.0 field_name = 2.0 "score": 65.17219 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14637) Update CloudSolrClient examples
[
https://issues.apache.org/jira/browse/SOLR-14637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17153874#comment-17153874
]
Lucene/Solr QA commented on SOLR-14637:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
|| || || || {color:brown} master Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} Release audit (RAT) {color} |
{color:green} 0m 2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Validate source patterns {color} |
{color:green} 0m 2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Validate ref guide {color} |
{color:green} 0m 2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:black}{color} | {color:black} {color} | {color:black} 1m 38s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | SOLR-14637 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13007306/SOLR-14637.patch |
| Optional Tests | ratsources validatesourcepatterns validaterefguide |
| uname | Linux lucene1-us-west 4.15.0-108-generic #109-Ubuntu SMP Fri Jun 19
11:33:10 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | ant |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-SOLR-Build/sourcedir/dev-tools/test-patch/lucene-solr-yetus-personality.sh
|
| git revision | master / 3b8ae56b39c |
| ant | version: Apache Ant(TM) version 1.10.5 compiled on March 28 2019 |
| modules | C: solr/solr-ref-guide U: solr/solr-ref-guide |
| Console output |
https://builds.apache.org/job/PreCommit-SOLR-Build/777/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> Update CloudSolrClient examples
> ---
>
> Key: SOLR-14637
> URL: https://issues.apache.org/jira/browse/SOLR-14637
> Project: Solr
> Issue Type: Task
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Andras Salamon
>Priority: Minor
> Attachments: SOLR-14637.patch
>
>
> CloudSolrClient.Builder() is deprecated ( SOLR-11629 ) but in the
> documentation we still use this constructor. I think it would be better to
> use a non-deprecated constructor in the examples.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] markharwood opened a new pull request #1659: Add case insensitive support to regexp query.
markharwood opened a new pull request #1659: URL: https://github.com/apache/lucene-solr/pull/1659 Backport of 887fe4c83d4114c6238265ca7f05aa491525af9d This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] ctargett commented on a change in pull request #1626: SOLR-14588: Implement Circuit Breakers
ctargett commented on a change in pull request #1626: URL: https://github.com/apache/lucene-solr/pull/1626#discussion_r451754651 ## File path: solr/solr-ref-guide/src/index.adoc ## @@ -10,6 +10,7 @@ streaming-expressions, \ solrcloud, \ legacy-scaling-and-distribution, \ +circuit-breakers, \ Review comment: I missed this before it was committed - it shouldn't be a top-level page IMO. I'm not sure where to put it, and I'm also working on a major re-org of all pages, so I'll leave it for now and figure out where it should go as part of the re-org process. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (SOLR-14566) Record "request-id" on coordinator log messages
[
https://issues.apache.org/jira/browse/SOLR-14566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Gerlowski resolved SOLR-14566.
Fix Version/s: 8.7
master (9.0)
Resolution: Fixed
> Record "request-id" on coordinator log messages
> ---
>
> Key: SOLR-14566
> URL: https://issues.apache.org/jira/browse/SOLR-14566
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Jason Gerlowski
>Assignee: Jason Gerlowski
>Priority: Minor
> Fix For: master (9.0), 8.7
>
> Time Spent: 6h 10m
> Remaining Estimate: 0h
>
> Currently, in SolrCore.java we log each search request that comes through
> each core as it is finishing. This includes the path, query-params, QTime,
> and status. In the case of a distributed search both the "coordinator" node
> and each of the per-shard requests produce a log message.
> When Solr is fielding many identical queries, such as those created by a
> healthcheck or dashboard, it can be hard when examining logs to link the
> per-shard requests with the "cooordinator" request that came in upstream.
> One thing that would make this easier is if the {{NOW}} param added to
> per-shard requests is also included in the log message from the
> "coordinator". While {{NOW}} isn't unique strictly speaking, it often is in
> practice, and along with the query-params would allow debuggers to associate
> shard requests with coordinator requests a large majority of the time.
> An alternative approach would be to create a {{qid}} or {{query-uuid}} when
> the coordinator starts its work that can be logged everywhere. This provides
> a stronger expectation around uniqueness, but would require UUID generation
> on the coordinator, which may be non-negligible work at high QPS (maybe? I
> have no idea). It also loses the neatness of reusing data already present on
> the shard requests.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14566) Record "request-id" on coordinator log messages
[
https://issues.apache.org/jira/browse/SOLR-14566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17153830#comment-17153830
]
Jason Gerlowski commented on SOLR-14566:
There was a good bit of feedback on the PR in the links section above from
David, Shalin, Jan, and others. We settled on using a "rid" field with a
default value of {{-}}. An
non-default value can be used by explicitly including the "rid" parameter in
your request. The "rid" behavior can be disabled by sending requests with a
{{disableRequestId=true}} query param.
David brought up that really we should be taking this request-id and recording
it using MDC, so that it's recorded on _all_ log messages for a given request.
I agreed with him, but deferred on making that change here on grounds of
"scope". The MDC approach is more powerful, but also a much larger, more
complex change. That's why SOLR-8274 has been stalled out as long as it has,
honestly. Rather than let the perfect get in the way of the good, I committed
the simpler "rid" approach as-is so that it's available to users while the much
cooler MDC approach comes together.
> Record "request-id" on coordinator log messages
> ---
>
> Key: SOLR-14566
> URL: https://issues.apache.org/jira/browse/SOLR-14566
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Jason Gerlowski
>Assignee: Jason Gerlowski
>Priority: Minor
> Time Spent: 6h 10m
> Remaining Estimate: 0h
>
> Currently, in SolrCore.java we log each search request that comes through
> each core as it is finishing. This includes the path, query-params, QTime,
> and status. In the case of a distributed search both the "coordinator" node
> and each of the per-shard requests produce a log message.
> When Solr is fielding many identical queries, such as those created by a
> healthcheck or dashboard, it can be hard when examining logs to link the
> per-shard requests with the "cooordinator" request that came in upstream.
> One thing that would make this easier is if the {{NOW}} param added to
> per-shard requests is also included in the log message from the
> "coordinator". While {{NOW}} isn't unique strictly speaking, it often is in
> practice, and along with the query-params would allow debuggers to associate
> shard requests with coordinator requests a large majority of the time.
> An alternative approach would be to create a {{qid}} or {{query-uuid}} when
> the coordinator starts its work that can be logged everywhere. This provides
> a stronger expectation around uniqueness, but would require UUID generation
> on the coordinator, which may be non-negligible work at high QPS (maybe? I
> have no idea). It also loses the neatness of reusing data already present on
> the shard requests.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14636) Provide a reference implementation for SolrCloud that is stable and fast.
[ https://issues.apache.org/jira/browse/SOLR-14636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17153823#comment-17153823 ] Mark Robert Miller commented on SOLR-14636: --- First milestone to shoot for, *milestone one*, make the current thing work. I am not a software developer. I don't plan, I don't design, I don't care about solutions other than that they work. So in this stage, I just fix what's there. I don't care if it's silly, heavy, annoying, obscure. Fix what's there, let the software decide how it should be done. I don't pull out HDFS, I don't remove auto scaling, I don't punt DiH, I don't remove legacyCloud. With one exception. I only support collectionStateFormat=2. *Milestone two* allows for for further development, larger changes, new approaches. > Provide a reference implementation for SolrCloud that is stable and fast. > - > > Key: SOLR-14636 > URL: https://issues.apache.org/jira/browse/SOLR-14636 > Project: Solr > Issue Type: Task > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Mark Robert Miller >Assignee: Mark Robert Miller >Priority: Major > > SolrCloud powers critical infrastructure and needs the ability to run quickly > with stability. This reference implementation will allow for this. > > *status*: alpha > *tests***: > * *core*: passing with ignores (not solid*) > * *solrj*: tbd > * *test-framework*: tbd > * *contrib/analysis-extras*: tbd > * *contrib/analytics*: tbd > * *contrib/clustering*: tbd > * *contrib/dataimporthandler*: tbd > * *contrib/dataimporthandler-extras*: tbd > * *contrib/extraction*: tbd > * *contrib/jaegertracer-configurator*: tbd > * *contrib/langid*: tbd > * *contrib/prometheus-exporter*: tbd > * *contrib/velocity*: tbd > _* Running tests quickly and efficiently with strict policing will more > frequently find bugs and requires a period of hardening._ > _** Non Nightly currently, Nightly comes last._ -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14566) Record "request-id" on coordinator log messages
[
https://issues.apache.org/jira/browse/SOLR-14566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Gerlowski updated SOLR-14566:
---
Summary: Record "request-id" on coordinator log messages (was: Record
"NOW" on "coordinator" log messages)
> Record "request-id" on coordinator log messages
> ---
>
> Key: SOLR-14566
> URL: https://issues.apache.org/jira/browse/SOLR-14566
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Jason Gerlowski
>Assignee: Jason Gerlowski
>Priority: Minor
> Time Spent: 6h 10m
> Remaining Estimate: 0h
>
> Currently, in SolrCore.java we log each search request that comes through
> each core as it is finishing. This includes the path, query-params, QTime,
> and status. In the case of a distributed search both the "coordinator" node
> and each of the per-shard requests produce a log message.
> When Solr is fielding many identical queries, such as those created by a
> healthcheck or dashboard, it can be hard when examining logs to link the
> per-shard requests with the "cooordinator" request that came in upstream.
> One thing that would make this easier is if the {{NOW}} param added to
> per-shard requests is also included in the log message from the
> "coordinator". While {{NOW}} isn't unique strictly speaking, it often is in
> practice, and along with the query-params would allow debuggers to associate
> shard requests with coordinator requests a large majority of the time.
> An alternative approach would be to create a {{qid}} or {{query-uuid}} when
> the coordinator starts its work that can be logged everywhere. This provides
> a stronger expectation around uniqueness, but would require UUID generation
> on the coordinator, which may be non-negligible work at high QPS (maybe? I
> have no idea). It also loses the neatness of reusing data already present on
> the shard requests.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-13132) Improve JSON "terms" facet performance when sorted by relatedness
[
https://issues.apache.org/jira/browse/SOLR-13132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Gibney updated SOLR-13132:
--
Attachment: SOLR-13132-benchmarks.tgz
> Improve JSON "terms" facet performance when sorted by relatedness
> --
>
> Key: SOLR-13132
> URL: https://issues.apache.org/jira/browse/SOLR-13132
> Project: Solr
> Issue Type: Improvement
> Components: Facet Module
>Affects Versions: 7.4, master (9.0)
>Reporter: Michael Gibney
>Priority: Major
> Attachments: SOLR-13132-benchmarks.tgz,
> SOLR-13132-with-cache-01.patch, SOLR-13132-with-cache.patch,
> SOLR-13132.patch, SOLR-13132_testSweep.patch
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> When sorting buckets by {{relatedness}}, JSON "terms" facet must calculate
> {{relatedness}} for every term.
> The current implementation uses a standard uninverted approach (either
> {{docValues}} or {{UnInvertedField}}) to get facet counts over the domain
> base docSet, and then uses that initial pass as a pre-filter for a
> second-pass, inverted approach of fetching docSets for each relevant term
> (i.e., {{count > minCount}}?) and calculating intersection size of those sets
> with the domain base docSet.
> Over high-cardinality fields, the overhead of per-term docSet creation and
> set intersection operations increases request latency to the point where
> relatedness sort may not be usable in practice (for my use case, even after
> applying the patch for SOLR-13108, for a field with ~220k unique terms per
> core, QTime for high-cardinality domain docSets were, e.g.: cardinality
> 1816684=9000ms, cardinality 5032902=18000ms).
> The attached patch brings the above example QTimes down to a manageable
> ~300ms and ~250ms respectively. The approach calculates uninverted facet
> counts over domain base, foreground, and background docSets in parallel in a
> single pass. This allows us to take advantage of the efficiencies built into
> the standard uninverted {{FacetFieldProcessorByArray[DV|UIF]}}), and avoids
> the per-term docSet creation and set intersection overhead.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (SOLR-12309) CloudSolrClient.Builder constructors are not well documented
[
https://issues.apache.org/jira/browse/SOLR-12309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Eric Pugh resolved SOLR-12309.
Resolution: Fixed
Not providing a version since it was way back in 7x branch this was fixed...
> CloudSolrClient.Builder constructors are not well documented
>
>
> Key: SOLR-12309
> URL: https://issues.apache.org/jira/browse/SOLR-12309
> Project: Solr
> Issue Type: Bug
> Components: clients - java
>Affects Versions: 7.3
>Reporter: Shawn Heisey
>Priority: Minor
> Attachments: SOLR-12309.patch, fluent-builder-fail-compile-time.patch
>
>
> I was having a lot of trouble figuring out how to create a CloudSolrClient
> object without using deprecated code.
> The no-arg constructor on the Builder object is deprecated, and the two
> remaining methods have similar signatures to each other. It is not at all
> obvious how to successfully call the one that uses ZooKeeper to connect. The
> javadoc is silent on the issue. I did finally figure it out with a lot of
> googling, and I would like to save others the hassle.
> I believe that this is what the javadoc for the third ctor should say:
>
> Provide a series of ZooKeeper hosts which will be used when configuring
> CloudSolrClient instances. Optionally, include a chroot to be used when
> accessing the ZooKeeper database.
> Here are a couple of examples. The first one has no chroot, the second one
> does:
> new CloudSolrClient.Builder(zkHosts, Optional.empty())
> new CloudSolrClient.Builder(zkHosts, Optional.of("/solr"))
>
> The javadoc for the URL-based method should probably say something to
> indicate that it is easy to confuse with the ZK-based method.
> I have not yet looked at the current reference guide to see if that has any
> clarification.
> Is it a good idea to completely eliminate the ability to create a cloud
> client using a single string that matches the zkHost value used when starting
> Solr in cloud mode?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-12309) CloudSolrClient.Builder constructors are not well documented
[
https://issues.apache.org/jira/browse/SOLR-12309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17153771#comment-17153771
]
David Eric Pugh commented on SOLR-12309:
Makes sense to me! [~elyograg] please reopen this if you disagree.
> CloudSolrClient.Builder constructors are not well documented
>
>
> Key: SOLR-12309
> URL: https://issues.apache.org/jira/browse/SOLR-12309
> Project: Solr
> Issue Type: Bug
> Components: clients - java
>Affects Versions: 7.3
>Reporter: Shawn Heisey
>Priority: Minor
> Attachments: SOLR-12309.patch, fluent-builder-fail-compile-time.patch
>
>
> I was having a lot of trouble figuring out how to create a CloudSolrClient
> object without using deprecated code.
> The no-arg constructor on the Builder object is deprecated, and the two
> remaining methods have similar signatures to each other. It is not at all
> obvious how to successfully call the one that uses ZooKeeper to connect. The
> javadoc is silent on the issue. I did finally figure it out with a lot of
> googling, and I would like to save others the hassle.
> I believe that this is what the javadoc for the third ctor should say:
>
> Provide a series of ZooKeeper hosts which will be used when configuring
> CloudSolrClient instances. Optionally, include a chroot to be used when
> accessing the ZooKeeper database.
> Here are a couple of examples. The first one has no chroot, the second one
> does:
> new CloudSolrClient.Builder(zkHosts, Optional.empty())
> new CloudSolrClient.Builder(zkHosts, Optional.of("/solr"))
>
> The javadoc for the URL-based method should probably say something to
> indicate that it is easy to confuse with the ZK-based method.
> I have not yet looked at the current reference guide to see if that has any
> clarification.
> Is it a good idea to completely eliminate the ability to create a cloud
> client using a single string that matches the zkHost value used when starting
> Solr in cloud mode?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14636) Provide a reference implementation for SolrCloud that is stable and fast.
[ https://issues.apache.org/jira/browse/SOLR-14636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Robert Miller updated SOLR-14636: -- Description: SolrCloud powers critical infrastructure and needs the ability to run quickly with stability. This reference implementation will allow for this. *status*: alpha *tests***: * *core*: passing with ignores (not solid*) * *solrj*: tbd * *test-framework*: tbd * *contrib/analysis-extras*: tbd * *contrib/analytics*: tbd * *contrib/clustering*: tbd * *contrib/dataimporthandler*: tbd * *contrib/dataimporthandler-extras*: tbd * *contrib/extraction*: tbd * *contrib/jaegertracer-configurator*: tbd * *contrib/langid*: tbd * *contrib/prometheus-exporter*: tbd * *contrib/velocity*: tbd _* Running tests quickly and efficiently with strict policing will more frequently find bugs and requires a period of hardening._ _** Non Nightly currently, Nightly comes last._ was: SolrCloud powers critical infrastructure and needs the ability to run quickly with stability. This reference implementation will allow for this. *status*: alpha *tests*: * *core*: passing with ignores (not solid*) * *solrj*: tbd * *test-framework*: tbd * *contrib/analysis-extras*: tbd * *contrib/analytics*: tbd * *contrib/clustering*: tbd * *contrib/dataimporthandler*: tbd * *contrib/dataimporthandler-extras*: tbd * *contrib/extraction*: tbd * *contrib/jaegertracer-configurator*: tbd * *contrib/langid*: tbd * *contrib/prometheus-exporter*: tbd * *contrib/velocity*: tbd _* Running tests quickly and efficiently with strict policing will more frequently find bugs and requires a period of hardening._ > Provide a reference implementation for SolrCloud that is stable and fast. > - > > Key: SOLR-14636 > URL: https://issues.apache.org/jira/browse/SOLR-14636 > Project: Solr > Issue Type: Task > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Mark Robert Miller >Assignee: Mark Robert Miller >Priority: Major > > SolrCloud powers critical infrastructure and needs the ability to run quickly > with stability. This reference implementation will allow for this. > > *status*: alpha > *tests***: > * *core*: passing with ignores (not solid*) > * *solrj*: tbd > * *test-framework*: tbd > * *contrib/analysis-extras*: tbd > * *contrib/analytics*: tbd > * *contrib/clustering*: tbd > * *contrib/dataimporthandler*: tbd > * *contrib/dataimporthandler-extras*: tbd > * *contrib/extraction*: tbd > * *contrib/jaegertracer-configurator*: tbd > * *contrib/langid*: tbd > * *contrib/prometheus-exporter*: tbd > * *contrib/velocity*: tbd > _* Running tests quickly and efficiently with strict policing will more > frequently find bugs and requires a period of hardening._ > _** Non Nightly currently, Nightly comes last._ -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-9411) Fail complation on warnings, 9x gradle-only
[ https://issues.apache.org/jira/browse/LUCENE-9411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erick Erickson updated LUCENE-9411: --- Attachment: LUCENE-9411-explicit-warnings.patch Status: Reopened (was: Reopened) Uwe: Well, maybe I did understand your suggestion but didn't explain myself clearly ;). Your solution works, and is easy to implement. However, it leaves open the possibility that the new compiler warnings in version X will generate hundreds to thousands of new compiler warnings. How likely that is IDK. I just hope there's not a new warning in version X that's as bad as "rawtypes". Unfortunately, I don't see any way around that, we'll have to go with listing them explicitly since it's certainly not acceptable to break Jenkins builds for Java 11+ for this. We get a lot of benefit, much of that benefit because of your work to stop using the latest Java for this issue... 8,000 warnings in Solr later I have some evidence if we don't start breaking locally, people won't fix these by checking Jenkins. Lucene was a lot cleaner, still about 100 warnings. Mostly I'm whining because it was a boatload of work to clean up the crap we'd accumulated and I doubt I'll be willing to do anything of a similar magnitude when we upgrade to the next version. But that's not the immediate problem. I think the attached patch is what you're looking for? I'll run check and push it in a bit. > Fail complation on warnings, 9x gradle-only > --- > > Key: LUCENE-9411 > URL: https://issues.apache.org/jira/browse/LUCENE-9411 > Project: Lucene - Core > Issue Type: Improvement > Components: general/build >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Major > Labels: build > Fix For: master (9.0) > > Attachments: LUCENE-9411-explicit-warnings.patch, LUCENE-9411.patch, > LUCENE-9411.patch, LUCENE-9411.patch, LUCENE-9411.patch, > annotations-warnings.patch > > > Moving this over here from SOLR-11973 since it's part of the build system and > affects Lucene as well as Solr. You might want to see the discussion there. > We have a clean compile for both Solr and Lucene, no rawtypes, unchecked, > try, etc. warnings. There are some peculiar warnings (things like > SuppressFBWarnings, i.e. FindBugs) that I'm not sure about at all, but let's > assume those are not a problem. Now I'd like to start failing the compilation > if people write new code that generates warnings. > From what I can tell, just adding the flag is easy in both the Gradle and Ant > builds. I still have to prove out that adding -Werrors does what I expect, > i.e. succeeds now and fails when I introduce warnings. > But let's assume that works. Are there objections to this idea generally? I > hope to have some data by next Monday. > FWIW, the Lucene code base had far fewer issues than Solr, but > common-build.xml is in Lucene. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-12309) CloudSolrClient.Builder constructors are not well documented
[
https://issues.apache.org/jira/browse/SOLR-12309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17153716#comment-17153716
]
Andras Salamon commented on SOLR-12309:
---
It seems to me javadoc has been updated two years ago. I think this jira can be
resolved.
> CloudSolrClient.Builder constructors are not well documented
>
>
> Key: SOLR-12309
> URL: https://issues.apache.org/jira/browse/SOLR-12309
> Project: Solr
> Issue Type: Bug
> Components: clients - java
>Affects Versions: 7.3
>Reporter: Shawn Heisey
>Priority: Minor
> Attachments: SOLR-12309.patch, fluent-builder-fail-compile-time.patch
>
>
> I was having a lot of trouble figuring out how to create a CloudSolrClient
> object without using deprecated code.
> The no-arg constructor on the Builder object is deprecated, and the two
> remaining methods have similar signatures to each other. It is not at all
> obvious how to successfully call the one that uses ZooKeeper to connect. The
> javadoc is silent on the issue. I did finally figure it out with a lot of
> googling, and I would like to save others the hassle.
> I believe that this is what the javadoc for the third ctor should say:
>
> Provide a series of ZooKeeper hosts which will be used when configuring
> CloudSolrClient instances. Optionally, include a chroot to be used when
> accessing the ZooKeeper database.
> Here are a couple of examples. The first one has no chroot, the second one
> does:
> new CloudSolrClient.Builder(zkHosts, Optional.empty())
> new CloudSolrClient.Builder(zkHosts, Optional.of("/solr"))
>
> The javadoc for the URL-based method should probably say something to
> indicate that it is easy to confuse with the ZK-based method.
> I have not yet looked at the current reference guide to see if that has any
> clarification.
> Is it a good idea to completely eliminate the ability to create a cloud
> client using a single string that matches the zkHost value used when starting
> Solr in cloud mode?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-12847) Cut over implementation of maxShardsPerNode to a collection policy
[ https://issues.apache.org/jira/browse/SOLR-12847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17153715#comment-17153715 ] ASF subversion and git services commented on SOLR-12847: Commit cf742f45963f4747e7041e8131248bc3a2b44864 in lucene-solr's branch refs/heads/master from Andrzej Bialecki [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=cf742f4 ] SOLR-12847: Remove support for maxShardsPerNode. > Cut over implementation of maxShardsPerNode to a collection policy > -- > > Key: SOLR-12847 > URL: https://issues.apache.org/jira/browse/SOLR-12847 > Project: Solr > Issue Type: Bug > Components: AutoScaling, SolrCloud >Reporter: Shalin Shekhar Mangar >Assignee: Andrzej Bialecki >Priority: Major > Fix For: master (9.0) > > Time Spent: 40m > Remaining Estimate: 0h > > We've back and forth over handling maxShardsPerNode with autoscaling policies > (see SOLR-11005 for history). Now that we've reimplemented support for > creating collections with maxShardsPerNode when autoscaling policy is > enabled, we should re-look at how it is implemented. > I propose that we fold maxShardsPerNode (if specified) to a collection level > policy that overrides the corresponding default in cluster policy (see > SOLR-12845). We'll need to ensure that if maxShardsPerNode is specified then > the user sees neither violations nor corresponding suggestions because of the > default cluster policy. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (SOLR-12847) Cut over implementation of maxShardsPerNode to a collection policy
[ https://issues.apache.org/jira/browse/SOLR-12847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrzej Bialecki resolved SOLR-12847. - Resolution: Fixed > Cut over implementation of maxShardsPerNode to a collection policy > -- > > Key: SOLR-12847 > URL: https://issues.apache.org/jira/browse/SOLR-12847 > Project: Solr > Issue Type: Bug > Components: AutoScaling, SolrCloud >Reporter: Shalin Shekhar Mangar >Assignee: Andrzej Bialecki >Priority: Major > Fix For: master (9.0) > > Time Spent: 40m > Remaining Estimate: 0h > > We've back and forth over handling maxShardsPerNode with autoscaling policies > (see SOLR-11005 for history). Now that we've reimplemented support for > creating collections with maxShardsPerNode when autoscaling policy is > enabled, we should re-look at how it is implemented. > I propose that we fold maxShardsPerNode (if specified) to a collection level > policy that overrides the corresponding default in cluster policy (see > SOLR-12845). We'll need to ensure that if maxShardsPerNode is specified then > the user sees neither violations nor corresponding suggestions because of the > default cluster policy. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (SOLR-10814) Solr RuleBasedAuthorization config doesn't work seamlessly with kerberos authentication
[ https://issues.apache.org/jira/browse/SOLR-10814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Drob resolved SOLR-10814. -- Fix Version/s: master (9.0) Assignee: Mike Drob Resolution: Fixed [~noble.paul] - responding to your question on the PR - the public touch points are covered in the ref guide. If the example there is insufficient, please let me know. > Solr RuleBasedAuthorization config doesn't work seamlessly with kerberos > authentication > --- > > Key: SOLR-10814 > URL: https://issues.apache.org/jira/browse/SOLR-10814 > Project: Solr > Issue Type: Bug >Affects Versions: 6.2 >Reporter: Hrishikesh Gadre >Assignee: Mike Drob >Priority: Major > Fix For: master (9.0) > > Attachments: SOLR-10814.patch > > Time Spent: 40m > Remaining Estimate: 0h > > Solr allows configuring roles to control user access to the system. This is > accomplished through rule-based permission definitions which are assigned to > users. > The authorization framework in Solr passes the information about the request > (to be authorized) using an instance of AuthorizationContext class. Currently > the only way to extract authenticated user is via getUserPrincipal() method > which returns an instance of java.security.Principal class. The > RuleBasedAuthorizationPlugin implementation invokes getName() method on the > Principal instance to fetch the list of associated roles. > https://github.com/apache/lucene-solr/blob/2271e73e763b17f971731f6f69d6ffe46c40b944/solr/core/src/java/org/apache/solr/security/RuleBasedAuthorizationPlugin.java#L156 > In case of basic authentication mechanism, the principal is the userName. > Hence it works fine. But in case of kerberos authentication, the user > principal also contains the RELM information e.g. instead of foo, it would > return f...@example.com. This means if the user changes the authentication > mechanism, he would also need to change the user-role mapping in > authorization section to use f...@example.com instead of foo. This is not > good from usability perspective. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] madrob commented on pull request #1619: SOLR-10814 Add short-name feature to RuleBasedAuthz plugin
madrob commented on pull request #1619: URL: https://github.com/apache/lucene-solr/pull/1619#issuecomment-655589483 Committed in d3f4b21deb0. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] madrob closed pull request #1619: SOLR-10814 Add short-name feature to RuleBasedAuthz plugin
madrob closed pull request #1619: URL: https://github.com/apache/lucene-solr/pull/1619 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-10814) Solr RuleBasedAuthorization config doesn't work seamlessly with kerberos authentication
[ https://issues.apache.org/jira/browse/SOLR-10814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17153689#comment-17153689 ] ASF subversion and git services commented on SOLR-10814: Commit fc5887181b75d7e622ca31ebf0531f8d3b7599d8 in lucene-solr's branch refs/heads/master from Mike Drob [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=fc58871 ] SOLR-10814 changes entry > Solr RuleBasedAuthorization config doesn't work seamlessly with kerberos > authentication > --- > > Key: SOLR-10814 > URL: https://issues.apache.org/jira/browse/SOLR-10814 > Project: Solr > Issue Type: Bug >Affects Versions: 6.2 >Reporter: Hrishikesh Gadre >Priority: Major > Attachments: SOLR-10814.patch > > Time Spent: 20m > Remaining Estimate: 0h > > Solr allows configuring roles to control user access to the system. This is > accomplished through rule-based permission definitions which are assigned to > users. > The authorization framework in Solr passes the information about the request > (to be authorized) using an instance of AuthorizationContext class. Currently > the only way to extract authenticated user is via getUserPrincipal() method > which returns an instance of java.security.Principal class. The > RuleBasedAuthorizationPlugin implementation invokes getName() method on the > Principal instance to fetch the list of associated roles. > https://github.com/apache/lucene-solr/blob/2271e73e763b17f971731f6f69d6ffe46c40b944/solr/core/src/java/org/apache/solr/security/RuleBasedAuthorizationPlugin.java#L156 > In case of basic authentication mechanism, the principal is the userName. > Hence it works fine. But in case of kerberos authentication, the user > principal also contains the RELM information e.g. instead of foo, it would > return f...@example.com. This means if the user changes the authentication > mechanism, he would also need to change the user-role mapping in > authorization section to use f...@example.com instead of foo. This is not > good from usability perspective. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14636) Provide a reference implementation for SolrCloud that is stable and fast.
[ https://issues.apache.org/jira/browse/SOLR-14636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Robert Miller updated SOLR-14636: -- Description: SolrCloud powers critical infrastructure and needs the ability to run quickly with stability. This reference implementation will allow for this. *status*: alpha *tests*: * *core*: passing with ignores (not solid*) * *solrj*: tbd * *test-framework*: tbd * *contrib/analysis-extras*: tbd * *contrib/analytics*: tbd * *contrib/clustering*: tbd * *contrib/dataimporthandler*: tbd * *contrib/dataimporthandler-extras*: tbd * *contrib/extraction*: tbd * *contrib/jaegertracer-configurator*: tbd * *contrib/langid*: tbd * *contrib/prometheus-exporter*: tbd * *contrib/velocity*: tbd _* Running tests quickly and efficiently with strict policing will more frequently find bugs and requires a period of hardening._ was: SolrCloud powers critical infrastructure and needs the ability to run quickly with stability. This reference implementation will allow for this. *status*: alpha *tests*: * core: passing with ignores (not solid*) * solrj: tbd * contrib: tbd _* Running tests quickly and efficiently with strict policing will more frequently finds bugs and requires a period of hardening._ > Provide a reference implementation for SolrCloud that is stable and fast. > - > > Key: SOLR-14636 > URL: https://issues.apache.org/jira/browse/SOLR-14636 > Project: Solr > Issue Type: Task > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Mark Robert Miller >Assignee: Mark Robert Miller >Priority: Major > > SolrCloud powers critical infrastructure and needs the ability to run quickly > with stability. This reference implementation will allow for this. > > *status*: alpha > *tests*: > * *core*: passing with ignores (not solid*) > * *solrj*: tbd > * *test-framework*: tbd > * *contrib/analysis-extras*: tbd > * *contrib/analytics*: tbd > * *contrib/clustering*: tbd > * *contrib/dataimporthandler*: tbd > * *contrib/dataimporthandler-extras*: tbd > * *contrib/extraction*: tbd > * *contrib/jaegertracer-configurator*: tbd > * *contrib/langid*: tbd > * *contrib/prometheus-exporter*: tbd > * *contrib/velocity*: tbd > _* Running tests quickly and efficiently with strict policing will more > frequently find bugs and requires a period of hardening._ -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-9386) RegExpQuery - add case insensitive matching option
[ https://issues.apache.org/jira/browse/LUCENE-9386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Harwood resolved LUCENE-9386. -- Resolution: Fixed Addressed in https://github.com/apache/lucene-solr/commit/887fe4c83d4114c6238265ca7f05aa491525af9d > RegExpQuery - add case insensitive matching option > -- > > Key: LUCENE-9386 > URL: https://issues.apache.org/jira/browse/LUCENE-9386 > Project: Lucene - Core > Issue Type: Improvement > Components: core/search >Reporter: Mark Harwood >Priority: Minor > > In searches sometimes case sensitivity is important and sometimes not. > However, users don't want to have to index two versions of their data > (lowercased and original) in order to service both case sensitive and case > insensitive queries. To get around this users have been commonly seen to take > a user query e.g. `powershell.exe` and search for it with the regex > `[Pp][Oo][Ww][Ee][Rr][Ss][Hh][Ee][Ll][Ll]\.[Ee][Xx][Ee]`. > The proposal is that we add an extra "case insensitive" option to the RegExp > query flags to automatically do this sort of expansion when we create > Automatons. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9386) RegExpQuery - add case insensitive matching option
[ https://issues.apache.org/jira/browse/LUCENE-9386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17153677#comment-17153677 ] ASF subversion and git services commented on LUCENE-9386: - Commit 887fe4c83d4114c6238265ca7f05aa491525af9d in lucene-solr's branch refs/heads/master from markharwood [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=887fe4c ] LUCENE-9386 add case insensitive RegExp matching option (#1541) Added case insensitive search option (currently only works with ASCII characters) > RegExpQuery - add case insensitive matching option > -- > > Key: LUCENE-9386 > URL: https://issues.apache.org/jira/browse/LUCENE-9386 > Project: Lucene - Core > Issue Type: Improvement > Components: core/search >Reporter: Mark Harwood >Priority: Minor > > In searches sometimes case sensitivity is important and sometimes not. > However, users don't want to have to index two versions of their data > (lowercased and original) in order to service both case sensitive and case > insensitive queries. To get around this users have been commonly seen to take > a user query e.g. `powershell.exe` and search for it with the regex > `[Pp][Oo][Ww][Ee][Rr][Ss][Hh][Ee][Ll][Ll]\.[Ee][Xx][Ee]`. > The proposal is that we add an extra "case insensitive" option to the RegExp > query flags to automatically do this sort of expansion when we create > Automatons. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] markharwood merged pull request #1541: RegExp - add case insensitive matching option
markharwood merged pull request #1541: URL: https://github.com/apache/lucene-solr/pull/1541 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14636) Provide a reference implementation for SolrCloud that is stable and fast.
[ https://issues.apache.org/jira/browse/SOLR-14636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Robert Miller updated SOLR-14636: -- Description: SolrCloud powers critical infrastructure and needs the ability to run quickly with stability. This reference implementation will allow for this. *status*: alpha *tests*: * core: passing with ignores (not solid*) * solrj: tbd * contrib: tbd _* Running tests quickly and efficiently with strict policing will more frequently finds bugs and requires a period of hardening._ was: SolrCloud powers critical infrastructure and needs the ability to run quickly with stability. This reference implementation will allow for this. *status*: alpha *tests*: * core: passing with ignores (not solid*) * solrj: tbd * contrib: tbd * Running tests quickly and efficiently with strict policing will more frequently finds bugs and requires a period of hardening. > Provide a reference implementation for SolrCloud that is stable and fast. > - > > Key: SOLR-14636 > URL: https://issues.apache.org/jira/browse/SOLR-14636 > Project: Solr > Issue Type: Task > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Mark Robert Miller >Assignee: Mark Robert Miller >Priority: Major > > SolrCloud powers critical infrastructure and needs the ability to run quickly > with stability. This reference implementation will allow for this. > > *status*: alpha > *tests*: > * core: passing with ignores (not solid*) > * solrj: tbd > * contrib: tbd > _* Running tests quickly and efficiently with strict policing will more > frequently finds bugs and requires a period of hardening._ -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14636) Provide a reference implementation for SolrCloud that is stable and fast.
[ https://issues.apache.org/jira/browse/SOLR-14636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Robert Miller updated SOLR-14636: -- Description: SolrCloud powers critical infrastructure and needs the ability to run quickly with stability. This reference implementation will allow for this. *status*: alpha *tests*: * core: passing with ignores (not solid*) * solrj: tbd * contrib: tbd * Running tests quickly and efficiently with strict policing will more frequently finds bugs and requires a period of hardening. was: SolrCloud powers critical infrastructure and needs the ability to run quickly with stability. This reference implementation will allow for this. *status*: alpha *tests*: * core: passing with ignores * solrj: tbd * contrib: tbd > Provide a reference implementation for SolrCloud that is stable and fast. > - > > Key: SOLR-14636 > URL: https://issues.apache.org/jira/browse/SOLR-14636 > Project: Solr > Issue Type: Task > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Mark Robert Miller >Assignee: Mark Robert Miller >Priority: Major > > SolrCloud powers critical infrastructure and needs the ability to run quickly > with stability. This reference implementation will allow for this. > > *status*: alpha > *tests*: > * core: passing with ignores (not solid*) > * solrj: tbd > * contrib: tbd > * Running tests quickly and efficiently with strict policing will more > frequently finds bugs and requires a period of hardening. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14636) Provide a reference implementation for SolrCloud that is stable and fast.
[ https://issues.apache.org/jira/browse/SOLR-14636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17153661#comment-17153661 ] Mark Robert Miller commented on SOLR-14636: --- I've only been working on this in earnest for a couple weeks, but it shouldn't take too long to finish, I've done this before a few times. I'll keep the current status of the branch in the description and share the branch later today. > Provide a reference implementation for SolrCloud that is stable and fast. > - > > Key: SOLR-14636 > URL: https://issues.apache.org/jira/browse/SOLR-14636 > Project: Solr > Issue Type: Task > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Mark Robert Miller >Assignee: Mark Robert Miller >Priority: Major > > SolrCloud powers critical infrastructure and needs the ability to run quickly > with stability. This reference implementation will allow for this. > > *status*: alpha > *tests*: > * core: passing with ignores > * solrj: tbd > * contrib: tbd -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14637) Update CloudSolrClient examples
[ https://issues.apache.org/jira/browse/SOLR-14637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17153658#comment-17153658 ] David Eric Pugh commented on SOLR-14637: What do you think about fleshing out https://lucene.apache.org/solr/guide/8_5/using-solrj.html a bit with the CloudSolrClient? The https://lucene.apache.org/solr/guide/8_5/using-solrj.html#base-urls, and the next paragraphs below it don't talk about CloudSolrClient, and maybe having a paragraph about how to use CloudSolrClient would be good? > Update CloudSolrClient examples > --- > > Key: SOLR-14637 > URL: https://issues.apache.org/jira/browse/SOLR-14637 > Project: Solr > Issue Type: Task > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Andras Salamon >Priority: Minor > Attachments: SOLR-14637.patch > > > CloudSolrClient.Builder() is deprecated ( SOLR-11629 ) but in the > documentation we still use this constructor. I think it would be better to > use a non-deprecated constructor in the examples. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14637) Update CloudSolrClient examples
[ https://issues.apache.org/jira/browse/SOLR-14637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17153653#comment-17153653 ] David Eric Pugh commented on SOLR-14637: [~asalamon74]is that the simplest example? Wow.. It's a lot more complex then what was previous > Update CloudSolrClient examples > --- > > Key: SOLR-14637 > URL: https://issues.apache.org/jira/browse/SOLR-14637 > Project: Solr > Issue Type: Task > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Andras Salamon >Priority: Minor > Attachments: SOLR-14637.patch > > > CloudSolrClient.Builder() is deprecated ( SOLR-11629 ) but in the > documentation we still use this constructor. I think it would be better to > use a non-deprecated constructor in the examples. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14636) Provide a reference implementation for SolrCloud that is stable and fast.
[ https://issues.apache.org/jira/browse/SOLR-14636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Robert Miller updated SOLR-14636: -- Description: SolrCloud powers critical infrastructure and needs the ability to run quickly with stability. This reference implementation will allow for this. *status*: alpha *tests*: * core: passing with ignores * solrj: tbd * contrib: tbd was:SolrCloud powers critical infrastructure and needs the ability to run quickly with stability. This reference implementation will allow for this. > Provide a reference implementation for SolrCloud that is stable and fast. > - > > Key: SOLR-14636 > URL: https://issues.apache.org/jira/browse/SOLR-14636 > Project: Solr > Issue Type: Task > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Mark Robert Miller >Assignee: Mark Robert Miller >Priority: Major > > SolrCloud powers critical infrastructure and needs the ability to run quickly > with stability. This reference implementation will allow for this. > > *status*: alpha > *tests*: > * core: passing with ignores > * solrj: tbd > * contrib: tbd -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14637) Update CloudSolrClient examples
[ https://issues.apache.org/jira/browse/SOLR-14637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Salamon updated SOLR-14637: -- Status: Patch Available (was: Open) > Update CloudSolrClient examples > --- > > Key: SOLR-14637 > URL: https://issues.apache.org/jira/browse/SOLR-14637 > Project: Solr > Issue Type: Task > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Andras Salamon >Priority: Minor > Attachments: SOLR-14637.patch > > > CloudSolrClient.Builder() is deprecated ( SOLR-11629 ) but in the > documentation we still use this constructor. I think it would be better to > use a non-deprecated constructor in the examples. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14637) Update CloudSolrClient examples
[ https://issues.apache.org/jira/browse/SOLR-14637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Salamon updated SOLR-14637: -- Attachment: SOLR-14637.patch > Update CloudSolrClient examples > --- > > Key: SOLR-14637 > URL: https://issues.apache.org/jira/browse/SOLR-14637 > Project: Solr > Issue Type: Task > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Andras Salamon >Priority: Minor > Attachments: SOLR-14637.patch > > > CloudSolrClient.Builder() is deprecated ( SOLR-11629 ) but in the > documentation we still use this constructor. I think it would be better to > use a non-deprecated constructor in the examples. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-14637) Update CloudSolrClient examples
Andras Salamon created SOLR-14637: - Summary: Update CloudSolrClient examples Key: SOLR-14637 URL: https://issues.apache.org/jira/browse/SOLR-14637 Project: Solr Issue Type: Task Security Level: Public (Default Security Level. Issues are Public) Reporter: Andras Salamon CloudSolrClient.Builder() is deprecated ( SOLR-11629 ) but in the documentation we still use this constructor. I think it would be better to use a non-deprecated constructor in the examples. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] markharwood commented on a change in pull request #1541: RegExp - add case insensitive matching option
markharwood commented on a change in pull request #1541:
URL: https://github.com/apache/lucene-solr/pull/1541#discussion_r451581237
##
File path: lucene/core/src/java/org/apache/lucene/util/automaton/RegExp.java
##
@@ -499,10 +508,29 @@ public RegExp(String s) throws IllegalArgumentException {
* regular expression
*/
public RegExp(String s, int syntax_flags) throws IllegalArgumentException {
+this(s, syntax_flags, 0);
+ }
+ /**
+ * Constructs new RegExp from a string.
+ *
+ * @param s regexp string
+ * @param syntax_flags boolean 'or' of optional syntax constructs to be
+ * enabled
+ * @param match_flags boolean 'or' of match behavior options such as case
insensitivity
+ * @exception IllegalArgumentException if an error occurred while parsing the
+ * regular expression
+ */
+ public RegExp(String s, int syntax_flags, int match_flags) throws
IllegalArgumentException {
+// (for BWC reasons we don't validate invalid bits, just trim instead)
+syntax_flags = syntax_flags & 0xff;
Review comment:
As far as I can see, no. The change would be to remove that line and
replace with
if (syntax_flags > ALL) {
throw new IllegalArgumentException("Illegal syntax flag");
}
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14566) Record "NOW" on "coordinator" log messages
[
https://issues.apache.org/jira/browse/SOLR-14566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17153595#comment-17153595
]
ASF subversion and git services commented on SOLR-14566:
Commit ffb48947b34a38c0d5ce20049adee481cd29d37b in lucene-solr's branch
refs/heads/branch_8x from Jason Gerlowski
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=ffb4894 ]
SOLR-14566: Add request-ID to all distrib-search requests (#1574)
> Record "NOW" on "coordinator" log messages
> --
>
> Key: SOLR-14566
> URL: https://issues.apache.org/jira/browse/SOLR-14566
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Jason Gerlowski
>Assignee: Jason Gerlowski
>Priority: Minor
> Time Spent: 6h 10m
> Remaining Estimate: 0h
>
> Currently, in SolrCore.java we log each search request that comes through
> each core as it is finishing. This includes the path, query-params, QTime,
> and status. In the case of a distributed search both the "coordinator" node
> and each of the per-shard requests produce a log message.
> When Solr is fielding many identical queries, such as those created by a
> healthcheck or dashboard, it can be hard when examining logs to link the
> per-shard requests with the "cooordinator" request that came in upstream.
> One thing that would make this easier is if the {{NOW}} param added to
> per-shard requests is also included in the log message from the
> "coordinator". While {{NOW}} isn't unique strictly speaking, it often is in
> practice, and along with the query-params would allow debuggers to associate
> shard requests with coordinator requests a large majority of the time.
> An alternative approach would be to create a {{qid}} or {{query-uuid}} when
> the coordinator starts its work that can be logged everywhere. This provides
> a stronger expectation around uniqueness, but would require UUID generation
> on the coordinator, which may be non-negligible work at high QPS (maybe? I
> have no idea). It also loses the neatness of reusing data already present on
> the shard requests.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14566) Record "NOW" on "coordinator" log messages
[
https://issues.apache.org/jira/browse/SOLR-14566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17153597#comment-17153597
]
ASF subversion and git services commented on SOLR-14566:
Commit ffb48947b34a38c0d5ce20049adee481cd29d37b in lucene-solr's branch
refs/heads/branch_8x from Jason Gerlowski
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=ffb4894 ]
SOLR-14566: Add request-ID to all distrib-search requests (#1574)
> Record "NOW" on "coordinator" log messages
> --
>
> Key: SOLR-14566
> URL: https://issues.apache.org/jira/browse/SOLR-14566
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Jason Gerlowski
>Assignee: Jason Gerlowski
>Priority: Minor
> Time Spent: 6h 10m
> Remaining Estimate: 0h
>
> Currently, in SolrCore.java we log each search request that comes through
> each core as it is finishing. This includes the path, query-params, QTime,
> and status. In the case of a distributed search both the "coordinator" node
> and each of the per-shard requests produce a log message.
> When Solr is fielding many identical queries, such as those created by a
> healthcheck or dashboard, it can be hard when examining logs to link the
> per-shard requests with the "cooordinator" request that came in upstream.
> One thing that would make this easier is if the {{NOW}} param added to
> per-shard requests is also included in the log message from the
> "coordinator". While {{NOW}} isn't unique strictly speaking, it often is in
> practice, and along with the query-params would allow debuggers to associate
> shard requests with coordinator requests a large majority of the time.
> An alternative approach would be to create a {{qid}} or {{query-uuid}} when
> the coordinator starts its work that can be logged everywhere. This provides
> a stronger expectation around uniqueness, but would require UUID generation
> on the coordinator, which may be non-negligible work at high QPS (maybe? I
> have no idea). It also loses the neatness of reusing data already present on
> the shard requests.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14566) Record "NOW" on "coordinator" log messages
[
https://issues.apache.org/jira/browse/SOLR-14566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17153577#comment-17153577
]
ASF subversion and git services commented on SOLR-14566:
Commit 00203c292f8b61dd05281de5eed5bb8f711222a3 in lucene-solr's branch
refs/heads/master from Jason Gerlowski
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=00203c2 ]
SOLR-14566: Correct CHANGES.txt entry
> Record "NOW" on "coordinator" log messages
> --
>
> Key: SOLR-14566
> URL: https://issues.apache.org/jira/browse/SOLR-14566
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Jason Gerlowski
>Assignee: Jason Gerlowski
>Priority: Minor
> Time Spent: 6h 10m
> Remaining Estimate: 0h
>
> Currently, in SolrCore.java we log each search request that comes through
> each core as it is finishing. This includes the path, query-params, QTime,
> and status. In the case of a distributed search both the "coordinator" node
> and each of the per-shard requests produce a log message.
> When Solr is fielding many identical queries, such as those created by a
> healthcheck or dashboard, it can be hard when examining logs to link the
> per-shard requests with the "cooordinator" request that came in upstream.
> One thing that would make this easier is if the {{NOW}} param added to
> per-shard requests is also included in the log message from the
> "coordinator". While {{NOW}} isn't unique strictly speaking, it often is in
> practice, and along with the query-params would allow debuggers to associate
> shard requests with coordinator requests a large majority of the time.
> An alternative approach would be to create a {{qid}} or {{query-uuid}} when
> the coordinator starts its work that can be logged everywhere. This provides
> a stronger expectation around uniqueness, but would require UUID generation
> on the coordinator, which may be non-negligible work at high QPS (maybe? I
> have no idea). It also loses the neatness of reusing data already present on
> the shard requests.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14566) Record "NOW" on "coordinator" log messages
[
https://issues.apache.org/jira/browse/SOLR-14566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17153564#comment-17153564
]
ASF subversion and git services commented on SOLR-14566:
Commit 80f8ab717c01a124af176026ff36e5021fb1d8b2 in lucene-solr's branch
refs/heads/master from Jason Gerlowski
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=80f8ab7 ]
SOLR-14566: Add request-ID to all distrib-search requests (#1574)
> Record "NOW" on "coordinator" log messages
> --
>
> Key: SOLR-14566
> URL: https://issues.apache.org/jira/browse/SOLR-14566
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Jason Gerlowski
>Assignee: Jason Gerlowski
>Priority: Minor
> Time Spent: 6h 10m
> Remaining Estimate: 0h
>
> Currently, in SolrCore.java we log each search request that comes through
> each core as it is finishing. This includes the path, query-params, QTime,
> and status. In the case of a distributed search both the "coordinator" node
> and each of the per-shard requests produce a log message.
> When Solr is fielding many identical queries, such as those created by a
> healthcheck or dashboard, it can be hard when examining logs to link the
> per-shard requests with the "cooordinator" request that came in upstream.
> One thing that would make this easier is if the {{NOW}} param added to
> per-shard requests is also included in the log message from the
> "coordinator". While {{NOW}} isn't unique strictly speaking, it often is in
> practice, and along with the query-params would allow debuggers to associate
> shard requests with coordinator requests a large majority of the time.
> An alternative approach would be to create a {{qid}} or {{query-uuid}} when
> the coordinator starts its work that can be logged everywhere. This provides
> a stronger expectation around uniqueness, but would require UUID generation
> on the coordinator, which may be non-negligible work at high QPS (maybe? I
> have no idea). It also loses the neatness of reusing data already present on
> the shard requests.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] gerlowskija merged pull request #1574: SOLR-14566: Add request-ID to all distrib-search requests
gerlowskija merged pull request #1574: URL: https://github.com/apache/lucene-solr/pull/1574 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9411) Fail complation on warnings, 9x gradle-only
[ https://issues.apache.org/jira/browse/LUCENE-9411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17153514#comment-17153514 ] Dawid Weiss commented on LUCENE-9411: - Erick - Uwe means just explicitly listing all warning types Java 11 can support. > Fail complation on warnings, 9x gradle-only > --- > > Key: LUCENE-9411 > URL: https://issues.apache.org/jira/browse/LUCENE-9411 > Project: Lucene - Core > Issue Type: Improvement > Components: general/build >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Major > Labels: build > Fix For: master (9.0) > > Attachments: LUCENE-9411.patch, LUCENE-9411.patch, LUCENE-9411.patch, > LUCENE-9411.patch, annotations-warnings.patch > > > Moving this over here from SOLR-11973 since it's part of the build system and > affects Lucene as well as Solr. You might want to see the discussion there. > We have a clean compile for both Solr and Lucene, no rawtypes, unchecked, > try, etc. warnings. There are some peculiar warnings (things like > SuppressFBWarnings, i.e. FindBugs) that I'm not sure about at all, but let's > assume those are not a problem. Now I'd like to start failing the compilation > if people write new code that generates warnings. > From what I can tell, just adding the flag is easy in both the Gradle and Ant > builds. I still have to prove out that adding -Werrors does what I expect, > i.e. succeeds now and fails when I introduce warnings. > But let's assume that works. Are there objections to this idea generally? I > hope to have some data by next Monday. > FWIW, the Lucene code base had far fewer issues than Solr, but > common-build.xml is in Lucene. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9411) Fail complation on warnings, 9x gradle-only
[
https://issues.apache.org/jira/browse/LUCENE-9411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17153513#comment-17153513
]
Uwe Schindler commented on LUCENE-9411:
---
Anyways, the list of warnings did not really extend very much. So it's unlikely
that you miss some new warnings.
If we raise to a new Java version, we can just add the new warnings and fix the
build. That's not a desaster like you said.
So myrequest is simple and inline with other projects:
- enable all warnings explicit (the list can be found here):
https://docs.oracle.com/en/java/javase/11/tools/javac.html and not just
{{-Xlint:all}} like its now. Because {{all}} is likely break builds with future
java versions
- enable {{-Werror}}
> Fail complation on warnings, 9x gradle-only
> ---
>
> Key: LUCENE-9411
> URL: https://issues.apache.org/jira/browse/LUCENE-9411
> Project: Lucene - Core
> Issue Type: Improvement
> Components: general/build
>Reporter: Erick Erickson
>Assignee: Erick Erickson
>Priority: Major
> Labels: build
> Fix For: master (9.0)
>
> Attachments: LUCENE-9411.patch, LUCENE-9411.patch, LUCENE-9411.patch,
> LUCENE-9411.patch, annotations-warnings.patch
>
>
> Moving this over here from SOLR-11973 since it's part of the build system and
> affects Lucene as well as Solr. You might want to see the discussion there.
> We have a clean compile for both Solr and Lucene, no rawtypes, unchecked,
> try, etc. warnings. There are some peculiar warnings (things like
> SuppressFBWarnings, i.e. FindBugs) that I'm not sure about at all, but let's
> assume those are not a problem. Now I'd like to start failing the compilation
> if people write new code that generates warnings.
> From what I can tell, just adding the flag is easy in both the Gradle and Ant
> builds. I still have to prove out that adding -Werrors does what I expect,
> i.e. succeeds now and fails when I introduce warnings.
> But let's assume that works. Are there objections to this idea generally? I
> hope to have some data by next Monday.
> FWIW, the Lucene code base had far fewer issues than Solr, but
> common-build.xml is in Lucene.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9411) Fail complation on warnings, 9x gradle-only
[ https://issues.apache.org/jira/browse/LUCENE-9411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17153512#comment-17153512 ] Uwe Schindler commented on LUCENE-9411: --- Hi you misunderstood me: I dont want to disable the fail on warning, I just want a clear list of warnings that are enabled to fail the build. Everything else should just stay a warning. And the exercice could have been prevented if somebody checked Jenkins. We can for example instruct jenkins to fails the build when warnings are found. > Fail complation on warnings, 9x gradle-only > --- > > Key: LUCENE-9411 > URL: https://issues.apache.org/jira/browse/LUCENE-9411 > Project: Lucene - Core > Issue Type: Improvement > Components: general/build >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Major > Labels: build > Fix For: master (9.0) > > Attachments: LUCENE-9411.patch, LUCENE-9411.patch, LUCENE-9411.patch, > LUCENE-9411.patch, annotations-warnings.patch > > > Moving this over here from SOLR-11973 since it's part of the build system and > affects Lucene as well as Solr. You might want to see the discussion there. > We have a clean compile for both Solr and Lucene, no rawtypes, unchecked, > try, etc. warnings. There are some peculiar warnings (things like > SuppressFBWarnings, i.e. FindBugs) that I'm not sure about at all, but let's > assume those are not a problem. Now I'd like to start failing the compilation > if people write new code that generates warnings. > From what I can tell, just adding the flag is easy in both the Gradle and Ant > builds. I still have to prove out that adding -Werrors does what I expect, > i.e. succeeds now and fails when I introduce warnings. > But let's assume that works. Are there objections to this idea generally? I > hope to have some data by next Monday. > FWIW, the Lucene code base had far fewer issues than Solr, but > common-build.xml is in Lucene. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9411) Fail complation on warnings, 9x gradle-only
[ https://issues.apache.org/jira/browse/LUCENE-9411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17153505#comment-17153505 ] Erick Erickson commented on LUCENE-9411: I'm not sure how to balance your point .vs. building up a whole new backlog of warnings and then have to go through a similar exercise when we require, say, Java 14 as the minimum version. It'd be a major pain to require devs to fix all warnings for all versions of Java, that ain't gonna happen. And using new versions of Java is too important to hamper by failing on error warnings. It also would be a pain to go through parts of this exercise _again_ when we require Java 14 b/c we'd have to go through the process of finding all the warnings not flagged in Java 11. Admittedly the vast majority of the warnings were rawtypes and unchecked casts, so maybe it wouldn't be as big a deal as I fear. Or maybe the fact that I suppressed (and fixed a few) over 8,000 warnings that had built up over years is making me worry about it too much ;) I think what worries me is _unknowingly_ building up such a backlog again. We could maybe ameliorate that a bit by detecting the java version and selectively enabling/disabling the -Werror? > Fail complation on warnings, 9x gradle-only > --- > > Key: LUCENE-9411 > URL: https://issues.apache.org/jira/browse/LUCENE-9411 > Project: Lucene - Core > Issue Type: Improvement > Components: general/build >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Major > Labels: build > Fix For: master (9.0) > > Attachments: LUCENE-9411.patch, LUCENE-9411.patch, LUCENE-9411.patch, > LUCENE-9411.patch, annotations-warnings.patch > > > Moving this over here from SOLR-11973 since it's part of the build system and > affects Lucene as well as Solr. You might want to see the discussion there. > We have a clean compile for both Solr and Lucene, no rawtypes, unchecked, > try, etc. warnings. There are some peculiar warnings (things like > SuppressFBWarnings, i.e. FindBugs) that I'm not sure about at all, but let's > assume those are not a problem. Now I'd like to start failing the compilation > if people write new code that generates warnings. > From what I can tell, just adding the flag is easy in both the Gradle and Ant > builds. I still have to prove out that adding -Werrors does what I expect, > i.e. succeeds now and fails when I introduce warnings. > But let's assume that works. Are there objections to this idea generally? I > hope to have some data by next Monday. > FWIW, the Lucene code base had far fewer issues than Solr, but > common-build.xml is in Lucene. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-9037) ArrayIndexOutOfBoundsException due to repeated IOException during indexing
[ https://issues.apache.org/jira/browse/LUCENE-9037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ilan Ginzburg resolved LUCENE-9037. --- Resolution: Won't Fix > ArrayIndexOutOfBoundsException due to repeated IOException during indexing > -- > > Key: LUCENE-9037 > URL: https://issues.apache.org/jira/browse/LUCENE-9037 > Project: Lucene - Core > Issue Type: Bug > Components: core/index >Affects Versions: 7.1 >Reporter: Ilan Ginzburg >Priority: Minor > Attachments: TestIndexWriterTermsHashOverflow.java > > Time Spent: 40m > Remaining Estimate: 0h > > There is a limit to the number of tokens that can be held in memory by Lucene > when docs are indexed using DocumentsWriter, then bad things happen. The > limit can be reached by submitting a really large document, by submitting a > large number of documents without doing a commit (see LUCENE-8118) or by > repeatedly submitting documents that fail to get indexed in some specific > ways, leading to Lucene not cleaning up the in memory data structures that > eventually overflow. > The overflow is due to a 32 bit (signed) integer wrapping around to negative > territory, then causing an ArrayIndexOutOfBoundsException. > The failure path that we are reliably hitting is due to an IOException during > doc tokenization. A tokenizer implementing TokenStream throws an exception > from incrementToken() which causes indexing of that doc to fail. > The IOException bubbles back up to DocumentsWriter.updateDocument() (or > DocumentsWriter.updateDocuments() in some other cases) where it is not > treated as an AbortingException therefore it is not causing a reset of the > DocumentsWriterPerThread. On repeated failures (without any successful > indexing in between) if the upper layer (client via Solr) resubmits the doc > that fails again, DocumentsWriterPerThread will eventually cause > TermsHashPerField data structures to grow and overflow, leading to an > exception stack similar to the one in LUCENE-8118 (below stack trace copied > from a test run repro on 7.1): > java.lang.ArrayIndexOutOfBoundsException: > -65536java.lang.ArrayIndexOutOfBoundsException: -65536 > at __randomizedtesting.SeedInfo.seed([394FAB2B91B1D90A:C86FB3F3CE001AA8]:0) > at > org.apache.lucene.index.TermsHashPerField.writeByte(TermsHashPerField.java:198) > at > org.apache.lucene.index.TermsHashPerField.writeVInt(TermsHashPerField.java:221) > at > org.apache.lucene.index.FreqProxTermsWriterPerField.writeProx(FreqProxTermsWriterPerField.java:80) > at > org.apache.lucene.index.FreqProxTermsWriterPerField.addTerm(FreqProxTermsWriterPerField.java:171) > at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:185) > at > org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:792) > at > org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:430) > at > org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:392) > at > org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:239) > at > org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:481) > at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1717) > at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1462) > Using tokens composed only of lowercase letters, it takes less than > 130,000,000 different tokens (the shortest ones) to overflow > TermsHashPerField. > Using a single document (composed of the 20,000 shortest lowercase tokens) > submitted repeatedly for indexing requires 6352 submissions all failing with > an IOException on incrementToken() to trigger the > ArrayIndexOutOfBoundsException. > A proposed fix is to treat in DocumentsWriter.updateDocument() and > DocumentsWriter.updateDocuments() an IOException in the same way we treat an > AbortingException. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] murblanc closed pull request #998: LUCENE-9037: ArrayIndexOutOfBoundsException due to repeated IOExcepti…
murblanc closed pull request #998: URL: https://github.com/apache/lucene-solr/pull/998 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Assigned] (SOLR-14546) OverseerTaskProcessor can process messages out of order
[
https://issues.apache.org/jira/browse/SOLR-14546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ilan Ginzburg reassigned SOLR-14546:
Assignee: Ilan Ginzburg (was: Ilanm)
> OverseerTaskProcessor can process messages out of order
> ---
>
> Key: SOLR-14546
> URL: https://issues.apache.org/jira/browse/SOLR-14546
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Affects Versions: master (9.0)
>Reporter: Ilan Ginzburg
>Assignee: Ilan Ginzburg
>Priority: Major
> Labels: collection-api, correctness, overseer
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> *Summary:*
> Collection API and ConfigSet API messages can be processed out of order by
> the {{OverseerTaskProcessor}}/{{OverseerCollectionConfigSetProcessor}}
> because of the way locking is handled.
>
> *Some context:*
> The Overseer task processor dequeues messages from the Collection and
> ConfigSet Zookeeper queue at {{/overseer/collection-queue-work}} and hands
> processing to an executor pool with 100 max parallel tasks
> ({{ClusterStateUpdater}} on the other hand uses a single thread consuming and
> processing the {{/overseer/queue}} and does not suffer from the problem
> described here).
> Locking prevents tasks modifying the same or related data from running
> concurrently. For the Collection API, locking can be done at {{CLUSTER}},
> {{COLLECTION}}, {{SHARD}} or {{REPLICA}} level and each command has its
> locking level defined in {{CollectionParams.CollectionAction}}.
> Multiple tasks for the same or competing locking levels do not execute
> concurrently. Commands locking at {{COLLECTION}} level for the same
> collection (for example {{CREATE}} creating a collection, {{CREATEALIAS}}
> creating an alias for the collection and {{CREATESHARD}} creating a new shard
> for the collection) will be executed sequentially, and it is expected that
> they are executed in submission order. Commands locking at different levels
> are also serialized when needed (for example changes to a shard of a
> collection and changes to the collection itself).
>
> *The issue:*
> The way {{OverseerTaskProcessor.run()}} deals with messages that cannot be
> executed due to competing messages already running (i.e. lock unavailable) is
> not correct. The method basically iterates over the set of messages to
> execute, skips those that can’t be executed due to locking and executes those
> that can be (assuming enough remaining available executors).
> The issue of out of order execution occurs when at least 3 messages compete
> for a lock. The following scenario shows out of order execution (messages
> numbered in enqueue order):
> * Message 1 gets the lock and runs,
> * Message 2 can’t be executed because the lock is taken,
> * Message 1 finishes execution and releases the lock,
> * Message 3 can be executed, gets the lock and runs,
> * Message 2 eventually runs when Message 3 finishes and releases the lock.
> We therefore have execution order 1 - 3 - 2 when the expected execution order
> is 1 - 2 - 3. Order 1 - 3 - 2 might not make sense or result in a different
> final state from 1 - 2 - 3.
> (there’s a variant of this scenario in which after Message 2 can't be
> executed the number of max parallel tasks is reached, then remaining tasks in
> {{heads}} including Message 3 are put into the {{blockedTasks}} map. These
> tasks are the first ones considered for processing on the next iteration of
> the main {{while}} loop. The reordering is similar).
>
> Note that from the client perspective, there does not need to be 3
> outstanding tasks, 2 are sufficient. The third task required for observing
> reordering could be enqueued after the first one completed.
>
> *Impact of the issue:*
> This problem makes {{MultiThreadedOCPTest.testFillWorkQueue()}} fail because
> the test requires task execution in submission order (SOLR-14524).
>
> The messages required for reordering to happen are not necessarily messages
> at the same locking level: a message locking at {{SHARD}} or {{REPLICA}}
> level prevents execution of a {{COLLECTION}} level message for the same
> collection. Examples can be built of sequences of messages that lead to
> incorrect results or failures. For example {{ADDREPLICA}} followed by
> {{CREATEALIAS}} followed by {{DELETEALIAS}}, could see alias creation and
> deletion reordered, making no sense.
> I wasn’t able to come up with a realistic production example that would be
> impacted by this issue (doesn't mean one doesn't exist!).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: is
[jira] [Commented] (LUCENE-9411) Fail complation on warnings, 9x gradle-only
[
https://issues.apache.org/jira/browse/LUCENE-9411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17153461#comment-17153461
]
Uwe Schindler commented on LUCENE-9411:
---
Maybe we have to first check, if {{--release 11}} automatically only enables
warning of java 11. I have not verified this.
I reopened this issue to check this and either keep everything as-is, if we
know that {{--release 11}} only enabled warnings that were standard in Java 11,
or use explicit warning list.
> Fail complation on warnings, 9x gradle-only
> ---
>
> Key: LUCENE-9411
> URL: https://issues.apache.org/jira/browse/LUCENE-9411
> Project: Lucene - Core
> Issue Type: Improvement
> Components: general/build
>Reporter: Erick Erickson
>Assignee: Erick Erickson
>Priority: Major
> Labels: build
> Fix For: master (9.0)
>
> Attachments: LUCENE-9411.patch, LUCENE-9411.patch, LUCENE-9411.patch,
> LUCENE-9411.patch, annotations-warnings.patch
>
>
> Moving this over here from SOLR-11973 since it's part of the build system and
> affects Lucene as well as Solr. You might want to see the discussion there.
> We have a clean compile for both Solr and Lucene, no rawtypes, unchecked,
> try, etc. warnings. There are some peculiar warnings (things like
> SuppressFBWarnings, i.e. FindBugs) that I'm not sure about at all, but let's
> assume those are not a problem. Now I'd like to start failing the compilation
> if people write new code that generates warnings.
> From what I can tell, just adding the flag is easy in both the Gradle and Ant
> builds. I still have to prove out that adding -Werrors does what I expect,
> i.e. succeeds now and fails when I introduce warnings.
> But let's assume that works. Are there objections to this idea generally? I
> hope to have some data by next Monday.
> FWIW, the Lucene code base had far fewer issues than Solr, but
> common-build.xml is in Lucene.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9411) Fail complation on warnings, 9x gradle-only
[ https://issues.apache.org/jira/browse/LUCENE-9411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17153448#comment-17153448 ] Dawid Weiss commented on LUCENE-9411: - Makes sense to me. > Fail complation on warnings, 9x gradle-only > --- > > Key: LUCENE-9411 > URL: https://issues.apache.org/jira/browse/LUCENE-9411 > Project: Lucene - Core > Issue Type: Improvement > Components: general/build >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Major > Labels: build > Fix For: master (9.0) > > Attachments: LUCENE-9411.patch, LUCENE-9411.patch, LUCENE-9411.patch, > LUCENE-9411.patch, annotations-warnings.patch > > > Moving this over here from SOLR-11973 since it's part of the build system and > affects Lucene as well as Solr. You might want to see the discussion there. > We have a clean compile for both Solr and Lucene, no rawtypes, unchecked, > try, etc. warnings. There are some peculiar warnings (things like > SuppressFBWarnings, i.e. FindBugs) that I'm not sure about at all, but let's > assume those are not a problem. Now I'd like to start failing the compilation > if people write new code that generates warnings. > From what I can tell, just adding the flag is easy in both the Gradle and Ant > builds. I still have to prove out that adding -Werrors does what I expect, > i.e. succeeds now and fails when I introduce warnings. > But let's assume that works. Are there objections to this idea generally? I > hope to have some data by next Monday. > FWIW, the Lucene code base had far fewer issues than Solr, but > common-build.xml is in Lucene. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-9411) Fail complation on warnings, 9x gradle-only
[ https://issues.apache.org/jira/browse/LUCENE-9411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17153444#comment-17153444 ] Uwe Schindler edited comment on LUCENE-9411 at 7/8/20, 9:45 AM: By the way: Policeman Jenkins makes statistics on all warnings since beginning of all time - here on the 8.6 branch: https://jenkins.thetaphi.de/job/Lucene-Solr-8.6-Linux/java was (Author: thetaphi): By the way: Policeman Jenkins makes statistics on all warnings since beginning of all time - here on the 8.6 branch: https://jenkins.thetaphi.de/job/Lucene-Solr-8.6-Linux/1027/java/ > Fail complation on warnings, 9x gradle-only > --- > > Key: LUCENE-9411 > URL: https://issues.apache.org/jira/browse/LUCENE-9411 > Project: Lucene - Core > Issue Type: Improvement > Components: general/build >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Major > Labels: build > Fix For: master (9.0) > > Attachments: LUCENE-9411.patch, LUCENE-9411.patch, LUCENE-9411.patch, > LUCENE-9411.patch, annotations-warnings.patch > > > Moving this over here from SOLR-11973 since it's part of the build system and > affects Lucene as well as Solr. You might want to see the discussion there. > We have a clean compile for both Solr and Lucene, no rawtypes, unchecked, > try, etc. warnings. There are some peculiar warnings (things like > SuppressFBWarnings, i.e. FindBugs) that I'm not sure about at all, but let's > assume those are not a problem. Now I'd like to start failing the compilation > if people write new code that generates warnings. > From what I can tell, just adding the flag is easy in both the Gradle and Ant > builds. I still have to prove out that adding -Werrors does what I expect, > i.e. succeeds now and fails when I introduce warnings. > But let's assume that works. Are there objections to this idea generally? I > hope to have some data by next Monday. > FWIW, the Lucene code base had far fewer issues than Solr, but > common-build.xml is in Lucene. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9411) Fail complation on warnings, 9x gradle-only
[ https://issues.apache.org/jira/browse/LUCENE-9411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17153444#comment-17153444 ] Uwe Schindler commented on LUCENE-9411: --- By the way: Policeman Jenkins makes statistics on all warnings since beginning of all time - here on the 8.6 branch: https://jenkins.thetaphi.de/job/Lucene-Solr-8.6-Linux/1027/java/ > Fail complation on warnings, 9x gradle-only > --- > > Key: LUCENE-9411 > URL: https://issues.apache.org/jira/browse/LUCENE-9411 > Project: Lucene - Core > Issue Type: Improvement > Components: general/build >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Major > Labels: build > Fix For: master (9.0) > > Attachments: LUCENE-9411.patch, LUCENE-9411.patch, LUCENE-9411.patch, > LUCENE-9411.patch, annotations-warnings.patch > > > Moving this over here from SOLR-11973 since it's part of the build system and > affects Lucene as well as Solr. You might want to see the discussion there. > We have a clean compile for both Solr and Lucene, no rawtypes, unchecked, > try, etc. warnings. There are some peculiar warnings (things like > SuppressFBWarnings, i.e. FindBugs) that I'm not sure about at all, but let's > assume those are not a problem. Now I'd like to start failing the compilation > if people write new code that generates warnings. > From what I can tell, just adding the flag is easy in both the Gradle and Ant > builds. I still have to prove out that adding -Werrors does what I expect, > i.e. succeeds now and fails when I introduce warnings. > But let's assume that works. Are there objections to this idea generally? I > hope to have some data by next Monday. > FWIW, the Lucene code base had far fewer issues than Solr, but > common-build.xml is in Lucene. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-9411) Fail complation on warnings, 9x gradle-only
[
https://issues.apache.org/jira/browse/LUCENE-9411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17153440#comment-17153440
]
Uwe Schindler edited comment on LUCENE-9411 at 7/8/20, 9:43 AM:
Hi,
I was on vacation the last 2 weeks so I did not look into this. I have one
problem with failing on errors, but that's easy to fix! The problem is: Newer
java versions add new warning types to the code. As we say "work with Java 11"
it's hard to prevent a failure on later Java versions and it's also hard to
communicate with developers. I think you should be able to build Lucene/Solr
also with a later java version and not fail fatally with later versions!
Jenkins does not catch this at the moment because it still uses ANT.
So please, please only keep -Werror enabled if we explicitely say for which
warnings. As we fixed it for Java 11, just make the {{-Xlint}} be an explicit
list. I would just get the "enabled-by-default" -Xlint settings and list them
explicitely. In short:
- remove {{-Xlint}}
- add {{-Xlint:+foobar}} for each warning type that's the default in Java 11.
This is what other projects (also using GCC with Werror) are doing.
was (Author: thetaphi):
Hi,
I was on vacation the last 2 weeks so I did not look into this. I have one
problem with failing on errors, but that's easy to fix! The problem is: Newer
java versions add new warning types to the code. As we say "work with Java 11"
it's hard to prevent a failure on later Java versions and it's also hard to
communicate with developers. I think you should be able to build Lucene/Solr
also with a later java version and not fail fatally with later versions!
So please, please only keep -Werror enabled if we explicitely say for which
warnings. As we fixed it for Java 11, just make the {{-Xlint}} be an explicit
list. I would just get the "enabled-by-default" -Xlint settings and list them
explicitely. In short:
- remove {{-Xlint}}
- add {{-Xlint:+foobar}} for each warning type that's the default in Java 11.
This is what other projects (also using GCC with Werror) are doing.
> Fail complation on warnings, 9x gradle-only
> ---
>
> Key: LUCENE-9411
> URL: https://issues.apache.org/jira/browse/LUCENE-9411
> Project: Lucene - Core
> Issue Type: Improvement
> Components: general/build
>Reporter: Erick Erickson
>Assignee: Erick Erickson
>Priority: Major
> Labels: build
> Fix For: master (9.0)
>
> Attachments: LUCENE-9411.patch, LUCENE-9411.patch, LUCENE-9411.patch,
> LUCENE-9411.patch, annotations-warnings.patch
>
>
> Moving this over here from SOLR-11973 since it's part of the build system and
> affects Lucene as well as Solr. You might want to see the discussion there.
> We have a clean compile for both Solr and Lucene, no rawtypes, unchecked,
> try, etc. warnings. There are some peculiar warnings (things like
> SuppressFBWarnings, i.e. FindBugs) that I'm not sure about at all, but let's
> assume those are not a problem. Now I'd like to start failing the compilation
> if people write new code that generates warnings.
> From what I can tell, just adding the flag is easy in both the Gradle and Ant
> builds. I still have to prove out that adding -Werrors does what I expect,
> i.e. succeeds now and fails when I introduce warnings.
> But let's assume that works. Are there objections to this idea generally? I
> hope to have some data by next Monday.
> FWIW, the Lucene code base had far fewer issues than Solr, but
> common-build.xml is in Lucene.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Reopened] (LUCENE-9411) Fail complation on warnings, 9x gradle-only
[
https://issues.apache.org/jira/browse/LUCENE-9411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uwe Schindler reopened LUCENE-9411:
---
Hi,
I was on vacation the last 2 weeks so I did not look into this. I have one
problem with failing on errors, but that's easy to fix! The problem is: Newer
java versions add new warning types to the code. As we say "work with Java 11"
it's hard to prevent a failure on later Java versions and it's also hard to
communicate with developers. I think you should be able to build Lucene/Solr
also with a later java version and not fail fatally with later versions!
So please, please only keep -Werror enabled if we explicitely say for which
warnings. As we fixed it for Java 11, just make the {{-Xlint}} be an explicit
list. I would just get the "enabled-by-default" -Xlint settings and list them
explicitely. In short:
- remove {{-Xlint}}
- add {{-Xlint:+foobar}} for each warning type that's the default in Java 11.
This is what other projects (also using GCC with Werror) are doing.
> Fail complation on warnings, 9x gradle-only
> ---
>
> Key: LUCENE-9411
> URL: https://issues.apache.org/jira/browse/LUCENE-9411
> Project: Lucene - Core
> Issue Type: Improvement
> Components: general/build
>Reporter: Erick Erickson
>Assignee: Erick Erickson
>Priority: Major
> Labels: build
> Fix For: master (9.0)
>
> Attachments: LUCENE-9411.patch, LUCENE-9411.patch, LUCENE-9411.patch,
> LUCENE-9411.patch, annotations-warnings.patch
>
>
> Moving this over here from SOLR-11973 since it's part of the build system and
> affects Lucene as well as Solr. You might want to see the discussion there.
> We have a clean compile for both Solr and Lucene, no rawtypes, unchecked,
> try, etc. warnings. There are some peculiar warnings (things like
> SuppressFBWarnings, i.e. FindBugs) that I'm not sure about at all, but let's
> assume those are not a problem. Now I'd like to start failing the compilation
> if people write new code that generates warnings.
> From what I can tell, just adding the flag is easy in both the Gradle and Ant
> builds. I still have to prove out that adding -Werrors does what I expect,
> i.e. succeeds now and fails when I introduce warnings.
> But let's assume that works. Are there objections to this idea generally? I
> hope to have some data by next Monday.
> FWIW, the Lucene code base had far fewer issues than Solr, but
> common-build.xml is in Lucene.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
