[jira] [Commented] (LUCENE-6336) AnalyzingInfixSuggester needs duplicate handling
[
https://issues.apache.org/jira/browse/LUCENE-6336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17184969#comment-17184969
]
Tobias Graf commented on LUCENE-6336:
-
We hit the same issue. Would love to see this fixed.
> AnalyzingInfixSuggester needs duplicate handling
>
>
> Key: LUCENE-6336
> URL: https://issues.apache.org/jira/browse/LUCENE-6336
> Project: Lucene - Core
> Issue Type: Bug
>Affects Versions: 4.10.3, 5.0
>Reporter: Jan Høydahl
>Priority: Major
> Labels: lookup, suggester
> Attachments: LUCENE-6336.patch
>
>
> Spinoff from LUCENE-5833 but else unrelated.
> Using {{AnalyzingInfixSuggester}} which is backed by a Lucene index and
> stores payload and score together with the suggest text.
> I did some testing with Solr, producing the DocumentDictionary from an index
> with multiple documents containing the same text, but with random weights
> between 0-100. Then I got duplicate identical suggestions sorted by weight:
> {code}
> {
> "suggest":{"languages":{
> "engl":{
> "numFound":101,
> "suggestions":[{
> "term":"English",
> "weight":100,
> "payload":"0"},
> {
> "term":"English",
> "weight":99,
> "payload":"0"},
> {
> "term":"English",
> "weight":98,
> "payload":"0"},
> ---etc all the way down to 0---
> {code}
> I also reproduced the same behavior in AnalyzingInfixSuggester directly. So
> there is a need for some duplicate removal here, either while building the
> local suggest index or during lookup. Only the highest weight suggestion for
> a given term should be returned.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] atris opened a new pull request #1785: Update Circuit Breaker Configuration to Use Plugins
atris opened a new pull request #1785: URL: https://github.com/apache/lucene-solr/pull/1785 This commit refactors circuit breaker configurations to use PluginInfo and reduce the footprint on SolrConfig. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] linwenyao opened a new pull request #1784: LUCENE-9482: the description of "invalid deletion count" exception.
linwenyao opened a new pull request #1784:
URL: https://github.com/apache/lucene-solr/pull/1784
`throw new CorruptIndexException("invalid deletion count: " + softDelCount +
delCount + " vs maxDoc=" + info.maxDoc(), input);`
**It needs to be changed to:**
`throw new CorruptIndexException("invalid deletion count: " + (softDelCount
+ delCount) + " vs maxDoc=" + info.maxDoc(), input);`
The value of "softDelCount + delCount" must be calculated first.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] linwenyao opened a new pull request #1783: LUCENE-9482: There is a problem with the description of "invalid deletion count" exception.
linwenyao opened a new pull request #1783:
URL: https://github.com/apache/lucene-solr/pull/1783
`throw new CorruptIndexException("invalid deletion count: " + softDelCount +
delCount + " vs maxDoc=" + info.maxDoc(), input);`
**It needs to be changed to:**
`throw new CorruptIndexException("invalid deletion count: " + (softDelCount
+ delCount) + " vs maxDoc=" + info.maxDoc(), input);`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-9482) There is a problem with the description of "invalid deletion count" exception.
wenyao created LUCENE-9482:
--
Summary: There is a problem with the description of "invalid
deletion count" exception.
Key: LUCENE-9482
URL: https://issues.apache.org/jira/browse/LUCENE-9482
Project: Lucene - Core
Issue Type: Improvement
Components: core/index
Affects Versions: 7.7.3
Reporter: wenyao
class: org.apache.lucene.index.SegmentInfos#376
throw new CorruptIndexException("invalid deletion count: " + softDelCount +
delCount + " vs maxDoc=" + info.maxDoc(), input);
It needs to be changed to:
throw new CorruptIndexException("invalid deletion count: " + (softDelCount +
delCount) + " vs maxDoc=" + info.maxDoc(), input); // todo softDelCount +
delCount should merge
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] murblanc commented on pull request #1684: SOLR-14613: strongly typed initial proposal for placement plugin interface
murblanc commented on pull request #1684: URL: https://github.com/apache/lucene-solr/pull/1684#issuecomment-680467865 I have implemented the configuration part of the placement plugins. This happens in solr.xml that has been updated with an example. I was able to run collection creation and see the plugin code get called, compute placement and replicas get created. Property fetching (number of cores per node, metrics, system properties) is not yet implemented. That's the next one. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-14777) Chipping Away at Test Ignore in the Reference Impl Part 1
[ https://issues.apache.org/jira/browse/SOLR-14777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17184827#comment-17184827 ] Mark Robert Miller edited comment on SOLR-14777 at 8/26/20, 1:06 AM: - Dude, welcome aboard! Yeah man, this is janitor work, and a lot of talented people are not so interested. But also, some of the most talented people I have ever encountered are the best janitors out there. All the work of those talented non janitors ends up almost wasted without this work! And if this is your focus, you end being the one that actually knows what’s going on regardless of the experience and talents of others. So good luck for sure, and it sounds like you have the important stuff all figured out already. was (Author: markrmiller): Dude, welcome aboard! Yeah man, this is janitor work, and a lot of talented people are not so interested. But also, some of the most talented people I have ever encountered are the best janitors out there. All the work of those talented non janitors ends up almost wasted without this work! And if this is your focus, you end being the one that actually knows what’s going on regardless of the experience and tallest of others. So good luck for sure, and it sounds like you have the important stuff all figured out already. > Chipping Away at Test Ignore in the Reference Impl Part 1 > - > > Key: SOLR-14777 > URL: https://issues.apache.org/jira/browse/SOLR-14777 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Affects Versions: master (9.0) >Reporter: Marcus Eagan >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > I want to contribute to the future of Lucene and Solr :)! For me, one action > would be to help advance [~markrmil...@gmail.com]'s Reference Implementation > with the dirty work we all hate doing. Hopefully, I'll open up a few PRs > against his branch that improve test coverage. I'll keep my commits and my > PRs atomic to facilitate quicker review because I know we are all busy and > need to be able to understand what's happening quickly. I want to be a leader > in this community one day and it is incumbent on those who aspire to be > leaders to do the things that no one else uninspired (and probably more > talented) would want to do first. > Wish me luck and stay tuned! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14777) Chipping Away at Test Ignore in the Reference Impl Part 1
[ https://issues.apache.org/jira/browse/SOLR-14777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17184827#comment-17184827 ] Mark Robert Miller commented on SOLR-14777: --- Dude, welcome aboard! Yeah man, this is janitor work, and a lot of talented people are not so interested. But also, some of the most talented people I have ever encountered are the best janitors out there. All the work of those talented non janitors ends up almost wasted without this work! And if this is your focus, you end being the one that actually knows what’s going on regardless of the experience and tallest of others. So good luck for sure, and it sounds like you have the important stuff all figured out already. > Chipping Away at Test Ignore in the Reference Impl Part 1 > - > > Key: SOLR-14777 > URL: https://issues.apache.org/jira/browse/SOLR-14777 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Affects Versions: master (9.0) >Reporter: Marcus Eagan >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > I want to contribute to the future of Lucene and Solr :)! For me, one action > would be to help advance [~markrmil...@gmail.com]'s Reference Implementation > with the dirty work we all hate doing. Hopefully, I'll open up a few PRs > against his branch that improve test coverage. I'll keep my commits and my > PRs atomic to facilitate quicker review because I know we are all busy and > need to be able to understand what's happening quickly. I want to be a leader > in this community one day and it is incumbent on those who aspire to be > leaders to do the things that no one else uninspired (and probably more > talented) would want to do first. > Wish me luck and stay tuned! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14636) Provide a reference implementation for SolrCloud that is stable and fast.
[
https://issues.apache.org/jira/browse/SOLR-14636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17184819#comment-17184819
]
Mark Robert Miller commented on SOLR-14636:
---
Okay, I’m starting to get some solid help here. This is good news for anyone
following, because now I have collaborator(s), I cant be so wild and fun and
unconcerned for others. Now the focus has to be on group dynamics and
collaboration. So we are going to make some quick progress on these tests, I’m
going to work out some of the issues in the non test world, and we will be
ready to start demonstrating what’s going on here with some benchmarks.
> Provide a reference implementation for SolrCloud that is stable and fast.
> -
>
> Key: SOLR-14636
> URL: https://issues.apache.org/jira/browse/SOLR-14636
> Project: Solr
> Issue Type: Task
>Reporter: Mark Robert Miller
>Assignee: Mark Robert Miller
>Priority: Major
> Attachments: IMG_5575 (1).jpg, jenkins.png, solr-ref-branch.gif
>
>
> SolrCloud powers critical infrastructure and needs the ability to run quickly
> with stability. This reference implementation will allow for this.
> *location*: [https://github.com/apache/lucene-solr/tree/reference_impl]
> *status*: developer alpha, on the verge of developer beta
> *speed*: ludicrous
> *tests***:
> * *core*: {color:#00875a}*extremely stable*{color} with
> *{color:#de350b}ignores{color}*
> * *solrj*: {color:#00875a}*extremely stable*{color} with
> {color:#de350b}*ignores*{color}
> * *test-framework*: *extremely stable* with {color:#de350b}*ignores*{color}
> * *contrib/analysis-extras*: *extremely stable* with
> {color:#de350b}*ignores*{color}
> * *contrib/analytics*: {color:#00875a}*extremely stable*{color} with
> {color:#de350b}*ignores*{color}
> * *contrib/clustering*: {color:#00875a}*extremely stable*{color} with
> *{color:#de350b}ignores{color}*
> * *contrib/dataimporthandler*: {color:#00875a}*extremely stable*{color} with
> {color:#de350b}*ignores*{color}
> * *contrib/dataimporthandler-extras*: {color:#00875a}*extremely
> stable*{color} with *{color:#de350b}ignores{color}*
> * *contrib/extraction*: {color:#00875a}*extremely stable*{color} with
> {color:#de350b}*ignores*{color}
> * *contrib/jaegertracer-configurator*: {color:#00875a}*extremely
> stable*{color} with {color:#de350b}*ignores*{color}
> * *contrib/langid*: {color:#00875a}*extremely stable*{color} with
> {color:#de350b}*ignores*{color}
> * *contrib/prometheus-exporter*: {color:#00875a}*extremely stable*{color}
> with {color:#de350b}*ignores*{color}
> * *contrib/velocity*: {color:#00875a}*extremely stable*{color} with
> {color:#de350b}*ignores*{color}
> _* Running tests quickly and efficiently with strict policing will more
> frequently find bugs and requires a period of hardening._
> _** Non Nightly currently, Nightly comes last._
> h2. Getting Started with the Solr Reference Branch
> [AYoungLadysIllustratedPrimer|https://www.dropbox.com/sh/prnw0gpdoi0j9me/AABF2gPDoer6uL0ghXfEzbzja?dl=0]
> # Book One: [The 10 Minute
> Introduction|https://www.dropbox.com/s/x9k1m3zmjucc422/Book%20One%3A%20The%2010%20Minute%20Introduction.mp4?dl=0]
> # Book Two: A Brief, High Level Overview of the Changes
> <{color:#de350b}*_WORK IN PROGRESS_*{color}>
> ## Book Two: Process Intro: Section 1: [A Quick Look at the Practical
> Process|https://www.dropbox.com/s/96owizh57i5vd8w/Book%20Two%3A%20Process%20Intro%3A%20Section%201%3A%20A%20Quick%20Look%20at%20the%20Practical%20Process.m4v?dl=0]
> ## Book Two: Process Intro: Section 1-2: [A Quick Look at the Practical
> Process|https://www.dropbox.com/s/a1ipp0n3gtbf29m/Book%20Two%3A%20Process%20Intro%3A%20Section%202-2%3A%20A%20Quick%20Look%20at%20the%20Practical%20Process.m4v?dl=0]
> ## Book Two: Process Intro: Section 2: [A Brief Look at a Few Broad
> Problems|https://www.dropbox.com/s/kmkp1lu1tfits1p/Book%20Two%3A%20Process%20Intro%3A%20Section%202%3A%20A%20Brief%20Look%20at%20a%20Few%20Broad%20Problems.m4v?dl=0]
> ## Book Two: Process Intro: Section 3: [Die, Die, Die Solr
> XML|https://www.dropbox.com/s/hi452cpjjx1h880/Book%20Two%3A%20Process%20Intro%3A%20Section%203%3A%20Die%2C%20Die%2C%20Die%20Solr%20XML.m4v?dl=0]
> ## Book Two: Process Intro: Section 4: [Back to a Few Broad
> Problems|https://www.dropbox.com/s/tmeayi23p0rmgda/Book%20Two%3A%20Process%20Intro%3A%20Section%204%3A%20Back%20to%20a%20Few%20Broad%20Problems.m4v?dl=0]
> ## Book Two: Process Intro: Section 4-2: [Back to a Few Broad
> Problem|https://www.dropbox.com/s/1olr1v99t73uoml/Book%20Two%3A%20Process%20Intro%3A%20Section%204-2%3A%20Back%20to%20a%20Few%20Broad%20Problems.m4v?dl=0]
> ## Book Two: Process Intro: Section 6: [Step 5, Own the
> World|https://www.dropbox.com/s/usdki6wb1qy6
[GitHub] [lucene-solr] rishisankar commented on a change in pull request #1770: SOLR-14763 SolrJ HTTP/2 Async API using CompletableFuture
rishisankar commented on a change in pull request #1770:
URL: https://github.com/apache/lucene-solr/pull/1770#discussion_r476924185
##
File path:
solr/core/src/java/org/apache/solr/handler/component/HttpShardHandler.java
##
@@ -156,36 +155,31 @@ public void submit(final ShardRequest sreq, final String
shard, final Modifiable
return;
}
-// all variables that set inside this listener must be at least volatile
-responseCancellableMap.put(srsp, this.lbClient.asyncReq(lbReq, new
AsyncListener<>() {
- volatile long startTime = System.nanoTime();
-
- @Override
- public void onStart() {
-if (tracer != null && span != null) {
- tracer.inject(span.context(), Format.Builtin.HTTP_HEADERS, new
SolrRequestCarrier(req));
-}
-SolrRequestInfo requestInfo = SolrRequestInfo.getRequestInfo();
-if (requestInfo != null)
req.setUserPrincipal(requestInfo.getReq().getUserPrincipal());
- }
+long startTime = System.nanoTime();
+if (tracer != null && span != null) {
Review comment:
I think they're the same thread? In the previous implementation, the
same thread that calls `submit()` [calls the async request
method](https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/handler/component/HttpShardHandler.java#L160)
and then within the async request method [calls
`onStart()`](https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/LBHttp2SolrClient.java#L89)
- so it seems to me like previously the thread that called `submit()` and
`onStart()` are the same.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14778) Disabling UpdateLog leads to silently lost updates
[ https://issues.apache.org/jira/browse/SOLR-14778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17184758#comment-17184758 ] Megan Carey commented on SOLR-14778: The new replica states might not be _necessary_, I just prefer not to call a state "BUFFERING" if we are not, in fact, buffering :) but I'm content to leave it for the sake of simplicity. We can rely on logs/documentation to clarify. The flags attached to an update request to indicate a buffering state are not actually based on the replica state, but on things like if the update is forwarded from parent, and the index writer is locked, and other conditions... so I suppose the equivalent for blocking/rejecting would be checking a config like the one you propose. > Disabling UpdateLog leads to silently lost updates > -- > > Key: SOLR-14778 > URL: https://issues.apache.org/jira/browse/SOLR-14778 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud, update >Affects Versions: 8.6.1 >Reporter: Megan Carey >Priority: Minor > > Solr currently "supports" disabling the UpdateLog, though it is "required" > for NRT replicas (per the [docs|#transaction-log]]). However, when the update > log is disabled and a replica is in BUFFERING state (e.g. during MigrateCmd > or SplitShardCmd), updates are *lost silently*. While most users will likely > never consider disabling the updateLog, it seems pertinent to provide a > better support option. > Options as discussed in [ASF > Slack|[https://the-asf.slack.com/archives/CEKUCUNE9/p1598373062262300]]: > # No longer support disabling the updateLog as it is considered a pertinent > feature in SolrCloud. This might be undesirable for use cases where some data > loss is acceptable and the updateLog takes up too much space. > # Improve Solr documentation to explicitly outline the risks of disabling > the updateLog. > # Add logging to indicate when an update is swallowed in this state. > # _My preferred option:_ Support disabling the updateLog by providing > additional replica states besides BUFFERING, so that there is no data loss > when updateLog is disabled and replica goes offline for an operation like > split. Some ideas: > ## REJECTING: Fail updates so that the client can retry again once the > operation is complete. > ## BLOCKING: Stall update until operation is complete, and then execute > update. > Feedback is welcome; once we establish a path forward I'd be happy to pick it > up. If others are interested I can document my findings as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-14778) Disabling UpdateLog leads to silently lost updates
[ https://issues.apache.org/jira/browse/SOLR-14778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17184754#comment-17184754 ] David Smiley edited comment on SOLR-14778 at 8/25/20, 10:03 PM: It's possible to have durability without an updateLog provided that all update requests do a (hard) commit without necessarily opening a new searcher. If you mostly do big batch updates and few little one-off changes sporadically, then this is perfectly viable. I think when the updateLog is disabled, Solr should allow you to specify (a boolean) that updates should be durable via automatically having updates work as commit=true&openSearcher=false if the user specifies neither (thus the user can always override), -and furthermore disable autoSoftCommit & commitWithin-. Put differently, imagine a "durability strategy". Are we durable via the updateLog or via always/implicitly hard-committing? Different trade-offs. Another benefit is that updateLog disabling avoids this problem: SOLR-8030 (udpateLog doesn't store request params; problematic with custom URPs). I'm skeptical we actually need new "replica states" to express the distinction between BUFFERING, REJECT, and BLOCKING. Maybe that's what's best but there may be lots of code to change and compatibility concerns given that the current enum has been stable since at least 2015. Instead, I'm imagining a simple solrconfig.xml setting. I don't pretend to think the whole matter is simple though. was (Author: dsmiley): It's possible to have durability without an updateLog provided that all update requests do a (hard) commit without necessarily opening a new searcher. If you mostly do big batch updates and few little one-off changes sporadically, then this is perfectly viable. I think when the updateLog is disabled, Solr should allow you to specify (a boolean) that updates should be durable via automatically having updates work as commit=true&openSearcher=false if the user specifies neither (thus the user can always override), and furthermore disable autoSoftCommit & commitWithin. Put differently, imagine a "durability strategy". Are we durable via the updateLog or via always/implicitly hard-committing? Different trade-offs. Another benefit is that updateLog disabling avoids this problem: SOLR-8030 (udpateLog doesn't store request params; problematic with custom URPs). I'm skeptical we actually need new "replica states" to express the distinction between BUFFERING, REJECT, and BLOCKING. Maybe that's what's best but there may be lots of code to change and compatibility concerns given that the current enum has been stable since at least 2015. Instead, I'm imagining a simple solrconfig.xml setting. I don't pretend to think the whole matter is simple though. > Disabling UpdateLog leads to silently lost updates > -- > > Key: SOLR-14778 > URL: https://issues.apache.org/jira/browse/SOLR-14778 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud, update >Affects Versions: 8.6.1 >Reporter: Megan Carey >Priority: Minor > > Solr currently "supports" disabling the UpdateLog, though it is "required" > for NRT replicas (per the [docs|#transaction-log]]). However, when the update > log is disabled and a replica is in BUFFERING state (e.g. during MigrateCmd > or SplitShardCmd), updates are *lost silently*. While most users will likely > never consider disabling the updateLog, it seems pertinent to provide a > better support option. > Options as discussed in [ASF > Slack|[https://the-asf.slack.com/archives/CEKUCUNE9/p1598373062262300]]: > # No longer support disabling the updateLog as it is considered a pertinent > feature in SolrCloud. This might be undesirable for use cases where some data > loss is acceptable and the updateLog takes up too much space. > # Improve Solr documentation to explicitly outline the risks of disabling > the updateLog. > # Add logging to indicate when an update is swallowed in this state. > # _My preferred option:_ Support disabling the updateLog by providing > additional replica states besides BUFFERING, so that there is no data loss > when updateLog is disabled and replica goes offline for an operation like > split. Some ideas: > ## REJECTING: Fail updates so that the client can retry again once the > operation is complete. > ## BLOCKING: Stall update until operation is complete, and then execute > update. > Feedback is welcome; once we establish a path forward I'd be happy to pick it > up. If others are interested I can document my findings as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (SOLR-14778) Disabling UpdateLog leads to silently lost updates
[ https://issues.apache.org/jira/browse/SOLR-14778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17184754#comment-17184754 ] David Smiley commented on SOLR-14778: - It's possible to have durability without an updateLog provided that all update requests do a (hard) commit without necessarily opening a new searcher. If you mostly do big batch updates and few little one-off changes sporadically, then this is perfectly viable. I think when the updateLog is disabled, Solr should allow you to specify (a boolean) that updates should be durable via automatically having updates work as commit=true&openSearcher=false if the user specifies neither (thus the user can always override), and furthermore disable autoSoftCommit & commitWithin. Put differently, imagine a "durability strategy". Are we durable via the updateLog or via always/implicitly hard-committing? Different trade-offs. Another benefit is that updateLog disabling avoids this problem: SOLR-8030 (udpateLog doesn't store request params; problematic with custom URPs). I'm skeptical we actually need new "replica states" to express the distinction between BUFFERING, REJECT, and BLOCKING. Maybe that's what's best but there may be lots of code to change and compatibility concerns given that the current enum has been stable since at least 2015. Instead, I'm imagining a simple solrconfig.xml setting. I don't pretend to think the whole matter is simple though. > Disabling UpdateLog leads to silently lost updates > -- > > Key: SOLR-14778 > URL: https://issues.apache.org/jira/browse/SOLR-14778 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud, update >Affects Versions: 8.6.1 >Reporter: Megan Carey >Priority: Minor > > Solr currently "supports" disabling the UpdateLog, though it is "required" > for NRT replicas (per the [docs|#transaction-log]]). However, when the update > log is disabled and a replica is in BUFFERING state (e.g. during MigrateCmd > or SplitShardCmd), updates are *lost silently*. While most users will likely > never consider disabling the updateLog, it seems pertinent to provide a > better support option. > Options as discussed in [ASF > Slack|[https://the-asf.slack.com/archives/CEKUCUNE9/p1598373062262300]]: > # No longer support disabling the updateLog as it is considered a pertinent > feature in SolrCloud. This might be undesirable for use cases where some data > loss is acceptable and the updateLog takes up too much space. > # Improve Solr documentation to explicitly outline the risks of disabling > the updateLog. > # Add logging to indicate when an update is swallowed in this state. > # _My preferred option:_ Support disabling the updateLog by providing > additional replica states besides BUFFERING, so that there is no data loss > when updateLog is disabled and replica goes offline for an operation like > split. Some ideas: > ## REJECTING: Fail updates so that the client can retry again once the > operation is complete. > ## BLOCKING: Stall update until operation is complete, and then execute > update. > Feedback is welcome; once we establish a path forward I'd be happy to pick it > up. If others are interested I can document my findings as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14774) HealthCheckHandler shouldn't be an implicit SolrCore level handler and should be configurable
[
https://issues.apache.org/jira/browse/SOLR-14774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17184740#comment-17184740
]
Tomas Eduardo Fernandez Lobbe commented on SOLR-14774:
--
Thanks Ishan, I think that would require a whole lot of other changes and
discussions to be had (none of the top level components as they are now are
dynamic, the standalone case, etc).
[~noble.paul], let me know if you still have concerns, otherwise I'll merge
tomorrow to master and 8.x
> HealthCheckHandler shouldn't be an implicit SolrCore level handler and should
> be configurable
> -
>
> Key: SOLR-14774
> URL: https://issues.apache.org/jira/browse/SOLR-14774
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Tomas Eduardo Fernandez Lobbe
>Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> While trying to use the HealthCheckHandler I noticed that:
> * CoreContainer has some logic to read config from {{solr.xml}}, however,
> this is never used, and the handler is constructed explicitly inside
> {{InfoHandler}}. This means you can't plugin a different implementation.
> * Also noticed that it was added as an implicit plugin to the {{SolrCore}}.
> which means one could access the handler like
> {{//admin/health}}, however, this will never work, since
> it uses a parameterless constructor, which leaves the handler in an invalid
> state [1]. I honestly don't know why it's being added to the {{SolrCore}},
> this is supposed to be a node level handler, and for SolrCore one could use
> the {{PingRequestHandler}}
>
> [1]
> {noformat}
> 2020-08-21 21:14:06.094 ERROR (qtp599782425-18) [c:test s:shard1 r:core_node4
> x:test_shard1_replica_n2] o.a.s.s.HttpSolrCall
> null:org.apache.solr.common.SolrException: CoreContainer is either not
> initialized or shutting down
> at
> org.apache.solr.handler.admin.HealthCheckHandler.handleRequestBody(HealthCheckHandler.java:96)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:214)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2606)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:812)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:588)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:415)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1300)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1215)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221)
> at
> org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:177)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at org.eclipse.jetty.server.Server.handle(Server.java:500)
> at
> org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.
[jira] [Resolved] (SOLR-14751) Zookeeper Admin screen not working for old ZK versions
[
https://issues.apache.org/jira/browse/SOLR-14751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tomas Eduardo Fernandez Lobbe resolved SOLR-14751.
--
Fix Version/s: 8.6.2
Resolution: Fixed
> Zookeeper Admin screen not working for old ZK versions
> --
>
> Key: SOLR-14751
> URL: https://issues.apache.org/jira/browse/SOLR-14751
> Project: Solr
> Issue Type: Bug
> Components: Admin UI
>Affects Versions: 8.6
>Reporter: Jan Høydahl
>Assignee: Jan Høydahl
>Priority: Major
> Fix For: 8.7, 8.6.2
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> I tried viewing the ZK admin screen for Zookeeper 3.4.9, but it fails with
> stack:
> {noformat}
> 2020-08-13 12:05:43.317 INFO (qtp489411441-25) [ ] o.a.s.s.HttpSolrCall
> [admin] webapp=null path=/admin/zookeeper/status
> params={wt=json&_=1597320343286} status=500 QTime=2
> 2020-08-13 12:05:43.317 ERROR (qtp489411441-25) [ ] o.a.s.s.HttpSolrCall
> null:org.apache.solr.common.SolrException: Failed to get config from zookeeper
> at
> org.apache.solr.common.cloud.SolrZkClient.getConfig(SolrZkClient.java:777)
> at
> org.apache.solr.handler.admin.ZookeeperStatusHandler.handleRequestBody(ZookeeperStatusHandler.java:83)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:214)
> at
> org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:854)
> at
> org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:818)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:566)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:415)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1300)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1215)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221)
> at
> org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:177)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at org.eclipse.jetty.server.Server.handle(Server.java:500)
> at
> org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
> at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
> at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
> at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
> at
[jira] [Commented] (SOLR-14751) Zookeeper Admin screen not working for old ZK versions
[
https://issues.apache.org/jira/browse/SOLR-14751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17184737#comment-17184737
]
ASF subversion and git services commented on SOLR-14751:
Commit 016993b65e393b58246d54e8ddda9f56a453eb0e in lucene-solr's branch
refs/heads/branch_8_6 from Jan Høydahl
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=016993b ]
SOLR-14751: Zookeeper Admin screen not working for old ZK versions
> Zookeeper Admin screen not working for old ZK versions
> --
>
> Key: SOLR-14751
> URL: https://issues.apache.org/jira/browse/SOLR-14751
> Project: Solr
> Issue Type: Bug
> Components: Admin UI
>Affects Versions: 8.6
>Reporter: Jan Høydahl
>Assignee: Jan Høydahl
>Priority: Major
> Fix For: 8.7
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> I tried viewing the ZK admin screen for Zookeeper 3.4.9, but it fails with
> stack:
> {noformat}
> 2020-08-13 12:05:43.317 INFO (qtp489411441-25) [ ] o.a.s.s.HttpSolrCall
> [admin] webapp=null path=/admin/zookeeper/status
> params={wt=json&_=1597320343286} status=500 QTime=2
> 2020-08-13 12:05:43.317 ERROR (qtp489411441-25) [ ] o.a.s.s.HttpSolrCall
> null:org.apache.solr.common.SolrException: Failed to get config from zookeeper
> at
> org.apache.solr.common.cloud.SolrZkClient.getConfig(SolrZkClient.java:777)
> at
> org.apache.solr.handler.admin.ZookeeperStatusHandler.handleRequestBody(ZookeeperStatusHandler.java:83)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:214)
> at
> org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:854)
> at
> org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:818)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:566)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:415)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1300)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1215)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221)
> at
> org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:177)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at org.eclipse.jetty.server.Server.handle(Server.java:500)
> at
> org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
> at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
> at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
> at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.
[jira] [Updated] (SOLR-14778) Disabling UpdateLog leads to silently lost updates
[ https://issues.apache.org/jira/browse/SOLR-14778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Megan Carey updated SOLR-14778: --- Description: Solr currently "supports" disabling the UpdateLog, though it is "required" for NRT replicas (per the [docs|#transaction-log]]). However, when the update log is disabled and a replica is in BUFFERING state (e.g. during MigrateCmd or SplitShardCmd), updates are *lost silently*. While most users will likely never consider disabling the updateLog, it seems pertinent to provide a better support option. Options as discussed in [ASF Slack|[https://the-asf.slack.com/archives/CEKUCUNE9/p1598373062262300]]: # No longer support disabling the updateLog as it is considered a pertinent feature in SolrCloud. This might be undesirable for use cases where some data loss is acceptable and the updateLog takes up too much space. # Improve Solr documentation to explicitly outline the risks of disabling the updateLog. # Add logging to indicate when an update is swallowed in this state. # _My preferred option:_ Support disabling the updateLog by providing additional replica states besides BUFFERING, so that there is no data loss when updateLog is disabled and replica goes offline for an operation like split. Some ideas: ## REJECTING: Fail updates so that the client can retry again once the operation is complete. ## BLOCKING: Stall update until operation is complete, and then execute update. Feedback is welcome; once we establish a path forward I'd be happy to pick it up. If others are interested I can document my findings as well. was: Solr currently "supports" disabling the UpdateLog, though it is "required" for NRT replicas (per the [docs|[https://lucene.apache.org/solr/guide/8_6/updatehandlers-in-solrconfig.html#transaction-log]]). However, when the update log is disabled and a replica is in BUFFERING state (e.g. during MigrateCmd or SplitShardCmd), updates are *lost silently*. While most users will likely never consider disabling the updateLog, it seems pertinent to provide a better support option. Options as discussed in [ASF Slack|[https://the-asf.slack.com/archives/CEKUCUNE9/p1598373062262300]:] # No longer support disabling the updateLog as it is considered a pertinent feature in SolrCloud. This might be undesirable for use cases where some data loss is acceptable and the updateLog takes up too much space. # Improve Solr documentation to explicitly outline the risks of disabling the updateLog. # Add logging to indicate when an update is swallowed in this state. # _My preferred option:_ Support disabling the updateLog by providing additional replica states so that there is no data loss when updateLog is disabled and replica goes offline for an operation like split. ## REJECTING: Fail updates so that the client can retry again once the operation is complete. ## BLOCKING: Stall update until operation is complete, and then execute update. Feedback is welcome; once we establish a path forward I'll be happy to pick it up. If others are interested I can document my findings as well. > Disabling UpdateLog leads to silently lost updates > -- > > Key: SOLR-14778 > URL: https://issues.apache.org/jira/browse/SOLR-14778 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud, update >Affects Versions: 8.6.1 >Reporter: Megan Carey >Priority: Minor > > Solr currently "supports" disabling the UpdateLog, though it is "required" > for NRT replicas (per the [docs|#transaction-log]]). However, when the update > log is disabled and a replica is in BUFFERING state (e.g. during MigrateCmd > or SplitShardCmd), updates are *lost silently*. While most users will likely > never consider disabling the updateLog, it seems pertinent to provide a > better support option. > Options as discussed in [ASF > Slack|[https://the-asf.slack.com/archives/CEKUCUNE9/p1598373062262300]]: > # No longer support disabling the updateLog as it is considered a pertinent > feature in SolrCloud. This might be undesirable for use cases where some data > loss is acceptable and the updateLog takes up too much space. > # Improve Solr documentation to explicitly outline the risks of disabling > the updateLog. > # Add logging to indicate when an update is swallowed in this state. > # _My preferred option:_ Support disabling the updateLog by providing > additional replica states besides BUFFERING, so that there is no data loss > when updateLog is disabled and replica goes offline for an operation like > split. Some ideas: > ## REJECTING: Fail updates so that the client can retry again once the > operation is complete. > ## BLOCKING: Stall update until operation is complete, and then execute >
[jira] [Created] (SOLR-14778) Disabling UpdateLog leads to silently lost updates
Megan Carey created SOLR-14778: -- Summary: Disabling UpdateLog leads to silently lost updates Key: SOLR-14778 URL: https://issues.apache.org/jira/browse/SOLR-14778 Project: Solr Issue Type: Bug Security Level: Public (Default Security Level. Issues are Public) Components: SolrCloud, update Affects Versions: 8.6.1 Reporter: Megan Carey Solr currently "supports" disabling the UpdateLog, though it is "required" for NRT replicas (per the [docs|[https://lucene.apache.org/solr/guide/8_6/updatehandlers-in-solrconfig.html#transaction-log]]). However, when the update log is disabled and a replica is in BUFFERING state (e.g. during MigrateCmd or SplitShardCmd), updates are *lost silently*. While most users will likely never consider disabling the updateLog, it seems pertinent to provide a better support option. Options as discussed in [ASF Slack|[https://the-asf.slack.com/archives/CEKUCUNE9/p1598373062262300]:] # No longer support disabling the updateLog as it is considered a pertinent feature in SolrCloud. This might be undesirable for use cases where some data loss is acceptable and the updateLog takes up too much space. # Improve Solr documentation to explicitly outline the risks of disabling the updateLog. # Add logging to indicate when an update is swallowed in this state. # _My preferred option:_ Support disabling the updateLog by providing additional replica states so that there is no data loss when updateLog is disabled and replica goes offline for an operation like split. ## REJECTING: Fail updates so that the client can retry again once the operation is complete. ## BLOCKING: Stall update until operation is complete, and then execute update. Feedback is welcome; once we establish a path forward I'll be happy to pick it up. If others are interested I can document my findings as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8962) Can we merge small segments during refresh, for faster searching?
[
https://issues.apache.org/jira/browse/LUCENE-8962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17184735#comment-17184735
]
Michael McCandless commented on LUCENE-8962:
Yeah, +1 we should have opened a new issue for the {{getReader}} case (separate
from the {{commit}} case which already shipped in 8.6). It is a related but
different feature, and, yes, in a different Lucene 8.x feature release.
This was an exciting change, conceived originally by [~msfroh], contributed by
[~msoko...@gmail.com], undergoing many iterations after that, exotic test
failures, tons of help from [~simonw] to iterate the change to a clean solution
that would work for both {{commit}} and {{getReader}}, beasting, more
iterations, commits then reverts, etc. I think it would make a great blog post!
> Can we merge small segments during refresh, for faster searching?
> -
>
> Key: LUCENE-8962
> URL: https://issues.apache.org/jira/browse/LUCENE-8962
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/index
>Reporter: Michael McCandless
>Priority: Major
> Fix For: master (9.0), 8.6
>
> Attachments: LUCENE-8962_demo.png, failed-tests.patch,
> failure_log.txt, test.diff
>
> Time Spent: 31h
> Remaining Estimate: 0h
>
> With near-real-time search we ask {{IndexWriter}} to write all in-memory
> segments to disk and open an {{IndexReader}} to search them, and this is
> typically a quick operation.
> However, when you use many threads for concurrent indexing, {{IndexWriter}}
> will accumulate write many small segments during {{refresh}} and this then
> adds search-time cost as searching must visit all of these tiny segments.
> The merge policy would normally quickly coalesce these small segments if
> given a little time ... so, could we somehow improve {{IndexWriter'}}s
> refresh to optionally kick off merge policy to merge segments below some
> threshold before opening the near-real-time reader? It'd be a bit tricky
> because while we are waiting for merges, indexing may continue, and new
> segments may be flushed, but those new segments shouldn't be included in the
> point-in-time segments returned by refresh ...
> One could almost do this on top of Lucene today, with a custom merge policy,
> and some hackity logic to have the merge policy target small segments just
> written by refresh, but it's tricky to then open a near-real-time reader,
> excluding newly flushed but including newly merged segments since the refresh
> originally finished ...
> I'm not yet sure how best to solve this, so I wanted to open an issue for
> discussion!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8962) Can we merge small segments during refresh, for faster searching?
[
https://issues.apache.org/jira/browse/LUCENE-8962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17184702#comment-17184702
]
David Smiley commented on LUCENE-8962:
--
Then we need a new JIRA issue targeted at 8.7 discussing this. Once a release
ships, the issues it references (in CHANGES.txt) are read-only from a code
standpoint. If 8.6 hadn't shipped, then we could have iterated on it in this
issue. Although sadly git history is forever, the situation can be fixed up by
using a new issue and changing the CHANTES.txt to refer to it.
> Can we merge small segments during refresh, for faster searching?
> -
>
> Key: LUCENE-8962
> URL: https://issues.apache.org/jira/browse/LUCENE-8962
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/index
>Reporter: Michael McCandless
>Priority: Major
> Fix For: master (9.0), 8.6
>
> Attachments: LUCENE-8962_demo.png, failed-tests.patch,
> failure_log.txt, test.diff
>
> Time Spent: 31h
> Remaining Estimate: 0h
>
> With near-real-time search we ask {{IndexWriter}} to write all in-memory
> segments to disk and open an {{IndexReader}} to search them, and this is
> typically a quick operation.
> However, when you use many threads for concurrent indexing, {{IndexWriter}}
> will accumulate write many small segments during {{refresh}} and this then
> adds search-time cost as searching must visit all of these tiny segments.
> The merge policy would normally quickly coalesce these small segments if
> given a little time ... so, could we somehow improve {{IndexWriter'}}s
> refresh to optionally kick off merge policy to merge segments below some
> threshold before opening the near-real-time reader? It'd be a bit tricky
> because while we are waiting for merges, indexing may continue, and new
> segments may be flushed, but those new segments shouldn't be included in the
> point-in-time segments returned by refresh ...
> One could almost do this on top of Lucene today, with a custom merge policy,
> and some hackity logic to have the merge policy target small segments just
> written by refresh, but it's tricky to then open a near-real-time reader,
> excluding newly flushed but including newly merged segments since the refresh
> originally finished ...
> I'm not yet sure how best to solve this, so I wanted to open an issue for
> discussion!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8962) Can we merge small segments during refresh, for faster searching?
[
https://issues.apache.org/jira/browse/LUCENE-8962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17184699#comment-17184699
]
Michael Froh commented on LUCENE-8962:
--
8.6 added ability to merge small segments on commit.
The more recent changes add the ability to merge on getReader calls (which is
what the original issue was asking for -- merging on commit was a slightly
easier step on the way there).
> Can we merge small segments during refresh, for faster searching?
> -
>
> Key: LUCENE-8962
> URL: https://issues.apache.org/jira/browse/LUCENE-8962
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/index
>Reporter: Michael McCandless
>Priority: Major
> Fix For: master (9.0), 8.6
>
> Attachments: LUCENE-8962_demo.png, failed-tests.patch,
> failure_log.txt, test.diff
>
> Time Spent: 31h
> Remaining Estimate: 0h
>
> With near-real-time search we ask {{IndexWriter}} to write all in-memory
> segments to disk and open an {{IndexReader}} to search them, and this is
> typically a quick operation.
> However, when you use many threads for concurrent indexing, {{IndexWriter}}
> will accumulate write many small segments during {{refresh}} and this then
> adds search-time cost as searching must visit all of these tiny segments.
> The merge policy would normally quickly coalesce these small segments if
> given a little time ... so, could we somehow improve {{IndexWriter'}}s
> refresh to optionally kick off merge policy to merge segments below some
> threshold before opening the near-real-time reader? It'd be a bit tricky
> because while we are waiting for merges, indexing may continue, and new
> segments may be flushed, but those new segments shouldn't be included in the
> point-in-time segments returned by refresh ...
> One could almost do this on top of Lucene today, with a custom merge policy,
> and some hackity logic to have the merge policy target small segments just
> written by refresh, but it's tricky to then open a near-real-time reader,
> excluding newly flushed but including newly merged segments since the refresh
> originally finished ...
> I'm not yet sure how best to solve this, so I wanted to open an issue for
> discussion!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8962) Can we merge small segments during refresh, for faster searching?
[
https://issues.apache.org/jira/browse/LUCENE-8962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17184698#comment-17184698
]
David Smiley commented on LUCENE-8962:
--
I've been following along here (or _trying to_) and I'm confused by the recent
commits. LUCENE-8962 was fixed & released into 8.6 which shipped. Why are new
commits landing that reference it?
> Can we merge small segments during refresh, for faster searching?
> -
>
> Key: LUCENE-8962
> URL: https://issues.apache.org/jira/browse/LUCENE-8962
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/index
>Reporter: Michael McCandless
>Priority: Major
> Fix For: master (9.0), 8.6
>
> Attachments: LUCENE-8962_demo.png, failed-tests.patch,
> failure_log.txt, test.diff
>
> Time Spent: 31h
> Remaining Estimate: 0h
>
> With near-real-time search we ask {{IndexWriter}} to write all in-memory
> segments to disk and open an {{IndexReader}} to search them, and this is
> typically a quick operation.
> However, when you use many threads for concurrent indexing, {{IndexWriter}}
> will accumulate write many small segments during {{refresh}} and this then
> adds search-time cost as searching must visit all of these tiny segments.
> The merge policy would normally quickly coalesce these small segments if
> given a little time ... so, could we somehow improve {{IndexWriter'}}s
> refresh to optionally kick off merge policy to merge segments below some
> threshold before opening the near-real-time reader? It'd be a bit tricky
> because while we are waiting for merges, indexing may continue, and new
> segments may be flushed, but those new segments shouldn't be included in the
> point-in-time segments returned by refresh ...
> One could almost do this on top of Lucene today, with a custom merge policy,
> and some hackity logic to have the merge policy target small segments just
> written by refresh, but it's tricky to then open a near-real-time reader,
> excluding newly flushed but including newly merged segments since the refresh
> originally finished ...
> I'm not yet sure how best to solve this, so I wanted to open an issue for
> discussion!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dsmiley commented on pull request #1769: SOLR-11245: Absorb the docker-solr repo.
dsmiley commented on pull request #1769: URL: https://github.com/apache/lucene-solr/pull/1769#issuecomment-680224131 While SOLR_JETTY_HOST is a nice security feature for someone running Solr outside of Docker, isn't it pointless (needless pain) for anyone using Docker? I think so, thus the docker settings could set this so that you don't have to. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] madrob commented on pull request #1769: SOLR-11245: Absorb the docker-solr repo.
madrob commented on pull request #1769: URL: https://github.com/apache/lucene-solr/pull/1769#issuecomment-680219550 I was able to build this and did some very simple validation after launching containers using: ``` docker run --rm -d --name zk-for-solr -p 2181:2181 zookeeper docker run --rm -p 8983:8983 --link zk-for-solr:zk-for-solr -e SOLR_JETTY_HOST="0.0.0.0" apache/solr:9.0.0-SNAPSHOT -c -z zk-for-solr:2181 docker run --rm -p 8984:8984 --link zk-for-solr:zk-for-solr -e SOLR_JETTY_HOST="0.0.0.0" apache/solr:9.0.0-SNAPSHOT -c -z zk-for-solr:2181 -p 8984 ``` Can we make sure that user run instructions are included in the ref guide, or possibly the tutorial? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-14751) Zookeeper Admin screen not working for old ZK versions
[
https://issues.apache.org/jira/browse/SOLR-14751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17184685#comment-17184685
]
Tomas Eduardo Fernandez Lobbe edited comment on SOLR-14751 at 8/25/20, 7:02 PM:
This is a trivial bugfix, I'd like to backport this for 8.6.2
was (Author: tomasflobbe):
This is a trivial bugfix, I'd like to backport this for 8.6.1
> Zookeeper Admin screen not working for old ZK versions
> --
>
> Key: SOLR-14751
> URL: https://issues.apache.org/jira/browse/SOLR-14751
> Project: Solr
> Issue Type: Bug
> Components: Admin UI
>Affects Versions: 8.6
>Reporter: Jan Høydahl
>Assignee: Jan Høydahl
>Priority: Major
> Fix For: 8.7
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> I tried viewing the ZK admin screen for Zookeeper 3.4.9, but it fails with
> stack:
> {noformat}
> 2020-08-13 12:05:43.317 INFO (qtp489411441-25) [ ] o.a.s.s.HttpSolrCall
> [admin] webapp=null path=/admin/zookeeper/status
> params={wt=json&_=1597320343286} status=500 QTime=2
> 2020-08-13 12:05:43.317 ERROR (qtp489411441-25) [ ] o.a.s.s.HttpSolrCall
> null:org.apache.solr.common.SolrException: Failed to get config from zookeeper
> at
> org.apache.solr.common.cloud.SolrZkClient.getConfig(SolrZkClient.java:777)
> at
> org.apache.solr.handler.admin.ZookeeperStatusHandler.handleRequestBody(ZookeeperStatusHandler.java:83)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:214)
> at
> org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:854)
> at
> org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:818)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:566)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:415)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1300)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1215)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221)
> at
> org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:177)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at org.eclipse.jetty.server.Server.handle(Server.java:500)
> at
> org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
> at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
> at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
> at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
> at
> or
[jira] [Reopened] (SOLR-14751) Zookeeper Admin screen not working for old ZK versions
[
https://issues.apache.org/jira/browse/SOLR-14751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tomas Eduardo Fernandez Lobbe reopened SOLR-14751:
--
This is a trivial bugfix, I'd like to backport this for 8.6.1
> Zookeeper Admin screen not working for old ZK versions
> --
>
> Key: SOLR-14751
> URL: https://issues.apache.org/jira/browse/SOLR-14751
> Project: Solr
> Issue Type: Bug
> Components: Admin UI
>Affects Versions: 8.6
>Reporter: Jan Høydahl
>Assignee: Jan Høydahl
>Priority: Major
> Fix For: 8.7
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> I tried viewing the ZK admin screen for Zookeeper 3.4.9, but it fails with
> stack:
> {noformat}
> 2020-08-13 12:05:43.317 INFO (qtp489411441-25) [ ] o.a.s.s.HttpSolrCall
> [admin] webapp=null path=/admin/zookeeper/status
> params={wt=json&_=1597320343286} status=500 QTime=2
> 2020-08-13 12:05:43.317 ERROR (qtp489411441-25) [ ] o.a.s.s.HttpSolrCall
> null:org.apache.solr.common.SolrException: Failed to get config from zookeeper
> at
> org.apache.solr.common.cloud.SolrZkClient.getConfig(SolrZkClient.java:777)
> at
> org.apache.solr.handler.admin.ZookeeperStatusHandler.handleRequestBody(ZookeeperStatusHandler.java:83)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:214)
> at
> org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:854)
> at
> org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:818)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:566)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:415)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1300)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1215)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221)
> at
> org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:177)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at org.eclipse.jetty.server.Server.handle(Server.java:500)
> at
> org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
> at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
> at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
> at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
>
[jira] [Commented] (LUCENE-8962) Can we merge small segments during refresh, for faster searching?
[
https://issues.apache.org/jira/browse/LUCENE-8962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17184679#comment-17184679
]
Michael McCandless commented on LUCENE-8962:
Woot! Thank you [~simonw]!
> Can we merge small segments during refresh, for faster searching?
> -
>
> Key: LUCENE-8962
> URL: https://issues.apache.org/jira/browse/LUCENE-8962
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/index
>Reporter: Michael McCandless
>Priority: Major
> Fix For: master (9.0), 8.6
>
> Attachments: LUCENE-8962_demo.png, failed-tests.patch,
> failure_log.txt, test.diff
>
> Time Spent: 31h
> Remaining Estimate: 0h
>
> With near-real-time search we ask {{IndexWriter}} to write all in-memory
> segments to disk and open an {{IndexReader}} to search them, and this is
> typically a quick operation.
> However, when you use many threads for concurrent indexing, {{IndexWriter}}
> will accumulate write many small segments during {{refresh}} and this then
> adds search-time cost as searching must visit all of these tiny segments.
> The merge policy would normally quickly coalesce these small segments if
> given a little time ... so, could we somehow improve {{IndexWriter'}}s
> refresh to optionally kick off merge policy to merge segments below some
> threshold before opening the near-real-time reader? It'd be a bit tricky
> because while we are waiting for merges, indexing may continue, and new
> segments may be flushed, but those new segments shouldn't be included in the
> point-in-time segments returned by refresh ...
> One could almost do this on top of Lucene today, with a custom merge policy,
> and some hackity logic to have the merge policy target small segments just
> written by refresh, but it's tricky to then open a near-real-time reader,
> excluding newly flushed but including newly merged segments since the refresh
> originally finished ...
> I'm not yet sure how best to solve this, so I wanted to open an issue for
> discussion!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14636) Provide a reference implementation for SolrCloud that is stable and fast.
[
https://issues.apache.org/jira/browse/SOLR-14636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Robert Miller updated SOLR-14636:
--
Description:
SolrCloud powers critical infrastructure and needs the ability to run quickly
with stability. This reference implementation will allow for this.
*location*: [https://github.com/apache/lucene-solr/tree/reference_impl]
*status*: developer alpha, on the verge of developer beta
*speed*: ludicrous
*tests***:
* *core*: {color:#00875a}*extremely stable*{color} with
*{color:#de350b}ignores{color}*
* *solrj*: {color:#00875a}*extremely stable*{color} with
{color:#de350b}*ignores*{color}
* *test-framework*: *extremely stable* with {color:#de350b}*ignores*{color}
* *contrib/analysis-extras*: *extremely stable* with
{color:#de350b}*ignores*{color}
* *contrib/analytics*: {color:#00875a}*extremely stable*{color} with
{color:#de350b}*ignores*{color}
* *contrib/clustering*: {color:#00875a}*extremely stable*{color} with
*{color:#de350b}ignores{color}*
* *contrib/dataimporthandler*: {color:#00875a}*extremely stable*{color} with
{color:#de350b}*ignores*{color}
* *contrib/dataimporthandler-extras*: {color:#00875a}*extremely stable*{color}
with *{color:#de350b}ignores{color}*
* *contrib/extraction*: {color:#00875a}*extremely stable*{color} with
{color:#de350b}*ignores*{color}
* *contrib/jaegertracer-configurator*: {color:#00875a}*extremely
stable*{color} with {color:#de350b}*ignores*{color}
* *contrib/langid*: {color:#00875a}*extremely stable*{color} with
{color:#de350b}*ignores*{color}
* *contrib/prometheus-exporter*: {color:#00875a}*extremely stable*{color} with
{color:#de350b}*ignores*{color}
* *contrib/velocity*: {color:#00875a}*extremely stable*{color} with
{color:#de350b}*ignores*{color}
_* Running tests quickly and efficiently with strict policing will more
frequently find bugs and requires a period of hardening._
_** Non Nightly currently, Nightly comes last._
h2. Getting Started with the Solr Reference Branch
[AYoungLadysIllustratedPrimer|https://www.dropbox.com/sh/prnw0gpdoi0j9me/AABF2gPDoer6uL0ghXfEzbzja?dl=0]
# Book One: [The 10 Minute
Introduction|https://www.dropbox.com/s/x9k1m3zmjucc422/Book%20One%3A%20The%2010%20Minute%20Introduction.mp4?dl=0]
# Book Two: A Brief, High Level Overview of the Changes <{color:#de350b}*_WORK
IN PROGRESS_*{color}>
## Book Two: Process Intro: Section 1: [A Quick Look at the Practical
Process|https://www.dropbox.com/s/96owizh57i5vd8w/Book%20Two%3A%20Process%20Intro%3A%20Section%201%3A%20A%20Quick%20Look%20at%20the%20Practical%20Process.m4v?dl=0]
## Book Two: Process Intro: Section 1-2: [A Quick Look at the Practical
Process|https://www.dropbox.com/s/a1ipp0n3gtbf29m/Book%20Two%3A%20Process%20Intro%3A%20Section%202-2%3A%20A%20Quick%20Look%20at%20the%20Practical%20Process.m4v?dl=0]
## Book Two: Process Intro: Section 2: [A Brief Look at a Few Broad
Problems|https://www.dropbox.com/s/kmkp1lu1tfits1p/Book%20Two%3A%20Process%20Intro%3A%20Section%202%3A%20A%20Brief%20Look%20at%20a%20Few%20Broad%20Problems.m4v?dl=0]
## Book Two: Process Intro: Section 3: [Die, Die, Die Solr
XML|https://www.dropbox.com/s/hi452cpjjx1h880/Book%20Two%3A%20Process%20Intro%3A%20Section%203%3A%20Die%2C%20Die%2C%20Die%20Solr%20XML.m4v?dl=0]
## Book Two: Process Intro: Section 4: [Back to a Few Broad
Problems|https://www.dropbox.com/s/tmeayi23p0rmgda/Book%20Two%3A%20Process%20Intro%3A%20Section%204%3A%20Back%20to%20a%20Few%20Broad%20Problems.m4v?dl=0]
## Book Two: Process Intro: Section 4-2: [Back to a Few Broad
Problem|https://www.dropbox.com/s/1olr1v99t73uoml/Book%20Two%3A%20Process%20Intro%3A%20Section%204-2%3A%20Back%20to%20a%20Few%20Broad%20Problems.m4v?dl=0]
## Book Two: Process Intro: Section 6: [Step 5, Own the
World|https://www.dropbox.com/s/usdki6wb1qy6fy9/Book%20Two%3A%20Process%20Intro%3A%20Section%206%3A%20Step%205%2C%20Own%20World.m4v?dl=0]
# Book Three: Diving In: Section1: [LifeCycle, LifeCycle,
LifeCycle|https://www.dropbox.com/s/zno3qftzekyo3ac/Book%20Three%3A%20Diving%20In%3A%20Section%201%3A%20LifeCycle%2C%2CLifeCycle%2CLifeCycle.m4v?dl=0]
# Book Three: Diving In: Section 2-1: [Thread
Managment|https://www.dropbox.com/s/myds10jju7qfvi5/Book%20Three%3A%20Diving%20In%3A%20Section%202-1%3A%20Thread%20Managment.mp4?dl=0]
# Book Three: Diving In: Section 2-2: [Thread
Managment|https://www.dropbox.com/s/3gxwwriwaxvyn5i/Book%20Three%3A%20Diving%20In%3A%20Section%202-2%3A%20Thread%20Managment.m4v?dl=0]
# Book Three: Diving In: Section 3: [Code
Consistency|https://www.dropbox.com/s/s3e3d2b4ojdnmdu/Book%20Three%3A%20Diving%20In%3A%20Section%203%3A%20Code%20Consistency.m4v?dl=0]
# Book Four: Going Forward: Section 1: [A more positive
message|https://www.dropbox.com/s/d1dj9ean8jqb4j6/Book%20Four%3A%20Going%20Forward%3A%20Section%201%3A%20A%20more%20positive%20message.m4v?dl=0]
was:
SolrCloud powers critical infrastructure and
[jira] [Commented] (SOLR-14774) HealthCheckHandler shouldn't be an implicit SolrCore level handler and should be configurable
[
https://issues.apache.org/jira/browse/SOLR-14774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17184672#comment-17184672
]
Ishan Chattopadhyaya commented on SOLR-14774:
-
bq. Let me know if you still have concernes
Thanks for clarifying.
While I prefer all pluggable components to be centrally configured using APIs
for the sake of consistency across the cluster and also easy deployment
(without need for restarting), I defer to your judgement (if you want to use
solr.xml for this purpose).
> HealthCheckHandler shouldn't be an implicit SolrCore level handler and should
> be configurable
> -
>
> Key: SOLR-14774
> URL: https://issues.apache.org/jira/browse/SOLR-14774
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Tomas Eduardo Fernandez Lobbe
>Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> While trying to use the HealthCheckHandler I noticed that:
> * CoreContainer has some logic to read config from {{solr.xml}}, however,
> this is never used, and the handler is constructed explicitly inside
> {{InfoHandler}}. This means you can't plugin a different implementation.
> * Also noticed that it was added as an implicit plugin to the {{SolrCore}}.
> which means one could access the handler like
> {{//admin/health}}, however, this will never work, since
> it uses a parameterless constructor, which leaves the handler in an invalid
> state [1]. I honestly don't know why it's being added to the {{SolrCore}},
> this is supposed to be a node level handler, and for SolrCore one could use
> the {{PingRequestHandler}}
>
> [1]
> {noformat}
> 2020-08-21 21:14:06.094 ERROR (qtp599782425-18) [c:test s:shard1 r:core_node4
> x:test_shard1_replica_n2] o.a.s.s.HttpSolrCall
> null:org.apache.solr.common.SolrException: CoreContainer is either not
> initialized or shutting down
> at
> org.apache.solr.handler.admin.HealthCheckHandler.handleRequestBody(HealthCheckHandler.java:96)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:214)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2606)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:812)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:588)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:415)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1300)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1215)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221)
> at
> org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:177)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at org.eclipse.jetty.server.Server.handle(Server.java:500)
> at
> org.eclipse.jetty.server.HttpChannel.lambda$handl
[jira] [Commented] (SOLR-14688) First party package implementation design
[ https://issues.apache.org/jira/browse/SOLR-14688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17184669#comment-17184669 ] Ishan Chattopadhyaya commented on SOLR-14688: - Yes, that would be preferable. That will also give the user the flexibility to upgrade just a particular package (say streaming expressions) without upgrading entire Solr. > First party package implementation design > - > > Key: SOLR-14688 > URL: https://issues.apache.org/jira/browse/SOLR-14688 > Project: Solr > Issue Type: Improvement >Reporter: Noble Paul >Priority: Major > Labels: package, packagemanager > > Here's the design document for first party packages: > https://docs.google.com/document/d/1n7gB2JAdZhlJKFrCd4Txcw4HDkdk7hlULyAZBS-wXrE/edit?usp=sharing > Put differently, this is about package-ifying our "contribs". -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14688) First party package implementation design
[ https://issues.apache.org/jira/browse/SOLR-14688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17184662#comment-17184662 ] Tomas Eduardo Fernandez Lobbe commented on SOLR-14688: -- Thanks. I guess that means all first party packages will have to support back compatibility (at the code level) to all versions where we want to allow rolling upgrade from, right? > First party package implementation design > - > > Key: SOLR-14688 > URL: https://issues.apache.org/jira/browse/SOLR-14688 > Project: Solr > Issue Type: Improvement >Reporter: Noble Paul >Priority: Major > Labels: package, packagemanager > > Here's the design document for first party packages: > https://docs.google.com/document/d/1n7gB2JAdZhlJKFrCd4Txcw4HDkdk7hlULyAZBS-wXrE/edit?usp=sharing > Put differently, this is about package-ifying our "contribs". -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14688) First party package implementation design
[ https://issues.apache.org/jira/browse/SOLR-14688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17184660#comment-17184660 ] Ishan Chattopadhyaya commented on SOLR-14688: - bq. I'm just learning about the the package management, so this may be a dumb question, but what would be the process for a Solr rolling upgrade? Thanks for asking, Tomas. I'll work out an example and add proper user documentation around this. Beyond some very basic testing, I haven't tested all possible edge case scenarios for rolling upgrade, and I shall test it more before commenting on how properly it is working currently. However, in a nutshell, the idea is: # that if a package supports "solr 8" and rolling upgrade happens from 8.5 to 8.6, no change to package. # if a package supports "solr 8.5", then the user should install a latter version of the package which supports "solr 8.6" or "solr 8.6 - 9" *before* performing the upgrade. > First party package implementation design > - > > Key: SOLR-14688 > URL: https://issues.apache.org/jira/browse/SOLR-14688 > Project: Solr > Issue Type: Improvement >Reporter: Noble Paul >Priority: Major > Labels: package, packagemanager > > Here's the design document for first party packages: > https://docs.google.com/document/d/1n7gB2JAdZhlJKFrCd4Txcw4HDkdk7hlULyAZBS-wXrE/edit?usp=sharing > Put differently, this is about package-ifying our "contribs". -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Assigned] (SOLR-14616) Remove CDCR from 9.0
[ https://issues.apache.org/jira/browse/SOLR-14616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya reassigned SOLR-14616: --- Assignee: Anshum Gupta (was: Ishan Chattopadhyaya) > Remove CDCR from 9.0 > > > Key: SOLR-14616 > URL: https://issues.apache.org/jira/browse/SOLR-14616 > Project: Solr > Issue Type: Sub-task >Affects Versions: master (9.0) >Reporter: Ishan Chattopadhyaya >Assignee: Anshum Gupta >Priority: Blocker > Time Spent: 50m > Remaining Estimate: 0h > > This was deprecated in SOLR-14022 and should be removed in 9.0. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14616) Remove CDCR from 9.0
[ https://issues.apache.org/jira/browse/SOLR-14616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17184643#comment-17184643 ] Ishan Chattopadhyaya commented on SOLR-14616: - I've added an updated PR here: https://github.com/apache/lucene-solr/pull/1782 (earlier one was raised via origin/remove-cdcr, a branch name using wrong nomenclature and hence someone wiped up that branch). [~erickerickson], please review (no hurry, since Anshum, Tomas and Houston prefer that we wait to remove this). bq. We can always remove the code in 9.1. That way we maintain the back-compat, allow people to upgrade to 9.0 without losing any features, and then remove the already deprecated features in the new release. That will also give us time to come up with alternate options. [~anshum], I'm assigning this JIRA to you so you can take the best decision as to when to remove this. bq. I'm a bit concerned on the impact it may have on the users to remove this in 9.0. [~tflobbe], please feel free to remove this whenever you feel it is the right time. bq. I think it's very rushed to deprecate something in one of the last minor releases and remove it in the next major version. [~houston], if you feel we should wait until 10x to remove this, please assign this ticket to yourself and remove it then. > Remove CDCR from 9.0 > > > Key: SOLR-14616 > URL: https://issues.apache.org/jira/browse/SOLR-14616 > Project: Solr > Issue Type: Sub-task >Affects Versions: master (9.0) >Reporter: Ishan Chattopadhyaya >Assignee: Ishan Chattopadhyaya >Priority: Blocker > Time Spent: 50m > Remaining Estimate: 0h > > This was deprecated in SOLR-14022 and should be removed in 9.0. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] chatman opened a new pull request #1782: SOLR-14616: Remove CDCR from 9.0
chatman opened a new pull request #1782: URL: https://github.com/apache/lucene-solr/pull/1782 Here's the fresh PR after messing up the branch in previous PR: https://github.com/apache/lucene-solr/pull/1648 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14688) First party package implementation design
[ https://issues.apache.org/jira/browse/SOLR-14688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17184635#comment-17184635 ] Tomas Eduardo Fernandez Lobbe commented on SOLR-14688: -- I'm just learning about the the package management, so this may be a dumb question, but what would be the process for a Solr rolling upgrade? > First party package implementation design > - > > Key: SOLR-14688 > URL: https://issues.apache.org/jira/browse/SOLR-14688 > Project: Solr > Issue Type: Improvement >Reporter: Noble Paul >Priority: Major > Labels: package, packagemanager > > Here's the design document for first party packages: > https://docs.google.com/document/d/1n7gB2JAdZhlJKFrCd4Txcw4HDkdk7hlULyAZBS-wXrE/edit?usp=sharing > Put differently, this is about package-ifying our "contribs". -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] thelabdude commented on pull request #1781: SOLR-14777: Fix an ignore test that is helpful.
thelabdude commented on pull request #1781: URL: https://github.com/apache/lucene-solr/pull/1781#issuecomment-680171669 Precommit won't work on this branch as there are many `nocommit` in the code (obviously). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9481) Improve FST performance created by Builder
[
https://issues.apache.org/jira/browse/LUCENE-9481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17184601#comment-17184601
]
Michael McCandless commented on LUCENE-9481:
Wow, that is a fascinating result!
Maybe, instead of a paged {{byte[]}} storage, we should have a store that just
grows a single {{byte[]}} (e.g. using {{ArrayUtil.grow}}) up until reasonable
limits? Then the resulting FST could use the same {{ReversedBytesReader}}?
> Improve FST performance created by Builder
> --
>
> Key: LUCENE-9481
> URL: https://issues.apache.org/jira/browse/LUCENE-9481
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/FSTs
>Reporter: Anh Dung Bui
>Priority: Major
>
> Hi,
>
> We are using Lucene 8 and recently we observed a performance issue with FST.
> When FST is used directly from *org.apache.lucene.util.fst.Builder.finish()*,
> its about 40-50% slower than when it's first saved to a *DataOutput* (e.g
> filesystem or heap) and loaded again. The FST we used is about 1.6MB.
>
> Using VisualVM, we tracked the issue down to the *DataInput.readVInt* method,
> which in turn calls the readByte of the reversed BytesReader. When FST is
> loaded from filesystem/heap, it's using OnHeapFSTStore which use
> ReverseBytesReader, but when it's created by the Builder, it's backed by the
> BytesStore. The problem is, when the number of blocks of BytesStore is not 1,
> it'll use an [anonymous
> class|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/util/fst/BytesStore.java#L453]
> for reading bytes, and possibly because of JVM inlining, the
> ReverseBytesReader.readByte is inlined, while that of the anonymous class is
> not. We confirmed this by adding XX:+PrintInlining flag.
> {code:java}
> @ 80 org.apache.lucene.util.fst.BytesStore$2::readByte (68 bytes) too
> big{code}
> {code:java}
> @ 17 org.apache.lucene.util.fst.ReverseBytesReader::readByte (17 bytes)
> inline (hot){code}
>
> Currently we have 2 workarounds, which can also confirm the above theory:
> * Use a larger *bytesPageBits* in Builder, so that the block size is greater
> than the FST size.
> * Save and load the FST to heap after building, e.g using
> GrowableByteArrayDataOutput and ByteArrayDataInput.
> But I think it would be better if this optimization is done inside
> *Builder.finish()* instead, e.g by auto-adjust the block size of BytesStore
> or copy it to a FSTStore if possible, since none of the 2 workarounds above
> look great. We have many FST with largely different sizes, so with the first
> option we need to maintain the bytesPageBits for each dictionary, otherwise
> we risk wasting unnecessary heap or using insufficient block size. I also
> think this issue would affect other Lucene users too.
>
> For reference, the benchmark we setup is as below:
> * Warm up both FST for 10.000 iterations (10.000 seems to be the JVM
> inlining trigger threshold)
> * Run 100 tests, each test run 100.000 iterations
> * Each iteration is simply call *org.apache.lucene.util.fst.Util.get()* with
> the same 4-character word.
> It reports that 98 out of 100 tests, the one with save/load logic is 40-50%
> faster on average.
> Thanks
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9224) (ant) RAT report complains about ... solr/webapp rat-report.xml (from gradle)
[
https://issues.apache.org/jira/browse/LUCENE-9224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17184142#comment-17184142
]
ASF subversion and git services commented on LUCENE-9224:
-
Commit c88d2a7dcef4175c15a63b530284f8886a437728 in lucene-solr's branch
refs/heads/branch_8x from Simon Willnauer
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=c88d2a7 ]
LUCENE-9224: (ant) RAT report complains about ... solr/webapp rat-report.xml
> (ant) RAT report complains about ... solr/webapp rat-report.xml (from gradle)
> -
>
> Key: LUCENE-9224
> URL: https://issues.apache.org/jira/browse/LUCENE-9224
> Project: Lucene - Core
> Issue Type: Bug
>Reporter: Chris M. Hostetter
>Assignee: Erick Erickson
>Priority: Major
> Fix For: master (9.0)
>
> Attachments: LUCENE-9224.patch
>
>
> I'm not sure if this is due to some comflagration of mixing gradle work with
> ant work, but today i encountered the following failure after running "ant
> clean precommit" ...
> {noformat}
> rat-sources:
> [rat]
> [rat] *
> [rat] Summary
> [rat] ---
> [rat] Generated at: 2020-02-13T14:46:10-07:00
> [rat] Notes: 0
> [rat] Binaries: 1
> [rat] Archives: 0
> [rat] Standards: 95
> [rat]
> [rat] Apache Licensed: 75
> [rat] Generated Documents: 0
> [rat]
> [rat] JavaDocs are generated and so license header is optional
> [rat] Generated files do not required license headers
> [rat]
> [rat] 1 Unknown Licenses
> [rat]
> [rat] ***
> [rat]
> [rat] Unapproved licenses:
> [rat]
> [rat] /home/hossman/lucene/dev/solr/webapp/build/rat/rat-report.xml
> [rat]
> [rat] ***
> [rat]
> [rat] Archives:
> [rat]
> [rat] *
> [rat] Files with Apache License headers will be marked AL
> [rat] Binary files (which do not require AL headers) will be marked B
> [rat] Compressed archives will be marked A
> [rat] Notices, licenses etc will be marked N
> [rat] AL/home/hossman/lucene/dev/solr/webapp/build.xml
> [rat] !?
> /home/hossman/lucene/dev/solr/webapp/build/rat/rat-report.xml
> [rat] AL/home/hossman/lucene/dev/solr/webapp/web/WEB-INF/web.xml
> ...
> {noformat}
> RAT seems to be comlaining that there is no license header in it's own report
> file?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9224) (ant) RAT report complains about ... solr/webapp rat-report.xml (from gradle)
[
https://issues.apache.org/jira/browse/LUCENE-9224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17184140#comment-17184140
]
ASF subversion and git services commented on LUCENE-9224:
-
Commit c88d2a7dcef4175c15a63b530284f8886a437728 in lucene-solr's branch
refs/heads/branch_8x from Simon Willnauer
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=c88d2a7 ]
LUCENE-9224: (ant) RAT report complains about ... solr/webapp rat-report.xml
> (ant) RAT report complains about ... solr/webapp rat-report.xml (from gradle)
> -
>
> Key: LUCENE-9224
> URL: https://issues.apache.org/jira/browse/LUCENE-9224
> Project: Lucene - Core
> Issue Type: Bug
>Reporter: Chris M. Hostetter
>Assignee: Erick Erickson
>Priority: Major
> Fix For: master (9.0)
>
> Attachments: LUCENE-9224.patch
>
>
> I'm not sure if this is due to some comflagration of mixing gradle work with
> ant work, but today i encountered the following failure after running "ant
> clean precommit" ...
> {noformat}
> rat-sources:
> [rat]
> [rat] *
> [rat] Summary
> [rat] ---
> [rat] Generated at: 2020-02-13T14:46:10-07:00
> [rat] Notes: 0
> [rat] Binaries: 1
> [rat] Archives: 0
> [rat] Standards: 95
> [rat]
> [rat] Apache Licensed: 75
> [rat] Generated Documents: 0
> [rat]
> [rat] JavaDocs are generated and so license header is optional
> [rat] Generated files do not required license headers
> [rat]
> [rat] 1 Unknown Licenses
> [rat]
> [rat] ***
> [rat]
> [rat] Unapproved licenses:
> [rat]
> [rat] /home/hossman/lucene/dev/solr/webapp/build/rat/rat-report.xml
> [rat]
> [rat] ***
> [rat]
> [rat] Archives:
> [rat]
> [rat] *
> [rat] Files with Apache License headers will be marked AL
> [rat] Binary files (which do not require AL headers) will be marked B
> [rat] Compressed archives will be marked A
> [rat] Notices, licenses etc will be marked N
> [rat] AL/home/hossman/lucene/dev/solr/webapp/build.xml
> [rat] !?
> /home/hossman/lucene/dev/solr/webapp/build/rat/rat-report.xml
> [rat] AL/home/hossman/lucene/dev/solr/webapp/web/WEB-INF/web.xml
> ...
> {noformat}
> RAT seems to be comlaining that there is no license header in it's own report
> file?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9224) (ant) RAT report complains about ... solr/webapp rat-report.xml (from gradle)
[
https://issues.apache.org/jira/browse/LUCENE-9224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17184143#comment-17184143
]
ASF subversion and git services commented on LUCENE-9224:
-
Commit 5ab8c34fc4cf07e5c02b4b426c35f7185c3529b0 in lucene-solr's branch
refs/heads/branch_8_6 from Simon Willnauer
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=5ab8c34 ]
LUCENE-9224: (ant) RAT report complains about ... solr/webapp rat-report.xml
> (ant) RAT report complains about ... solr/webapp rat-report.xml (from gradle)
> -
>
> Key: LUCENE-9224
> URL: https://issues.apache.org/jira/browse/LUCENE-9224
> Project: Lucene - Core
> Issue Type: Bug
>Reporter: Chris M. Hostetter
>Assignee: Erick Erickson
>Priority: Major
> Fix For: master (9.0)
>
> Attachments: LUCENE-9224.patch
>
>
> I'm not sure if this is due to some comflagration of mixing gradle work with
> ant work, but today i encountered the following failure after running "ant
> clean precommit" ...
> {noformat}
> rat-sources:
> [rat]
> [rat] *
> [rat] Summary
> [rat] ---
> [rat] Generated at: 2020-02-13T14:46:10-07:00
> [rat] Notes: 0
> [rat] Binaries: 1
> [rat] Archives: 0
> [rat] Standards: 95
> [rat]
> [rat] Apache Licensed: 75
> [rat] Generated Documents: 0
> [rat]
> [rat] JavaDocs are generated and so license header is optional
> [rat] Generated files do not required license headers
> [rat]
> [rat] 1 Unknown Licenses
> [rat]
> [rat] ***
> [rat]
> [rat] Unapproved licenses:
> [rat]
> [rat] /home/hossman/lucene/dev/solr/webapp/build/rat/rat-report.xml
> [rat]
> [rat] ***
> [rat]
> [rat] Archives:
> [rat]
> [rat] *
> [rat] Files with Apache License headers will be marked AL
> [rat] Binary files (which do not require AL headers) will be marked B
> [rat] Compressed archives will be marked A
> [rat] Notices, licenses etc will be marked N
> [rat] AL/home/hossman/lucene/dev/solr/webapp/build.xml
> [rat] !?
> /home/hossman/lucene/dev/solr/webapp/build/rat/rat-report.xml
> [rat] AL/home/hossman/lucene/dev/solr/webapp/web/WEB-INF/web.xml
> ...
> {noformat}
> RAT seems to be comlaining that there is no license header in it's own report
> file?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9224) (ant) RAT report complains about ... solr/webapp rat-report.xml (from gradle)
[
https://issues.apache.org/jira/browse/LUCENE-9224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17184141#comment-17184141
]
ASF subversion and git services commented on LUCENE-9224:
-
Commit 5ab8c34fc4cf07e5c02b4b426c35f7185c3529b0 in lucene-solr's branch
refs/heads/branch_8_6 from Simon Willnauer
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=5ab8c34 ]
LUCENE-9224: (ant) RAT report complains about ... solr/webapp rat-report.xml
> (ant) RAT report complains about ... solr/webapp rat-report.xml (from gradle)
> -
>
> Key: LUCENE-9224
> URL: https://issues.apache.org/jira/browse/LUCENE-9224
> Project: Lucene - Core
> Issue Type: Bug
>Reporter: Chris M. Hostetter
>Assignee: Erick Erickson
>Priority: Major
> Fix For: master (9.0)
>
> Attachments: LUCENE-9224.patch
>
>
> I'm not sure if this is due to some comflagration of mixing gradle work with
> ant work, but today i encountered the following failure after running "ant
> clean precommit" ...
> {noformat}
> rat-sources:
> [rat]
> [rat] *
> [rat] Summary
> [rat] ---
> [rat] Generated at: 2020-02-13T14:46:10-07:00
> [rat] Notes: 0
> [rat] Binaries: 1
> [rat] Archives: 0
> [rat] Standards: 95
> [rat]
> [rat] Apache Licensed: 75
> [rat] Generated Documents: 0
> [rat]
> [rat] JavaDocs are generated and so license header is optional
> [rat] Generated files do not required license headers
> [rat]
> [rat] 1 Unknown Licenses
> [rat]
> [rat] ***
> [rat]
> [rat] Unapproved licenses:
> [rat]
> [rat] /home/hossman/lucene/dev/solr/webapp/build/rat/rat-report.xml
> [rat]
> [rat] ***
> [rat]
> [rat] Archives:
> [rat]
> [rat] *
> [rat] Files with Apache License headers will be marked AL
> [rat] Binary files (which do not require AL headers) will be marked B
> [rat] Compressed archives will be marked A
> [rat] Notices, licenses etc will be marked N
> [rat] AL/home/hossman/lucene/dev/solr/webapp/build.xml
> [rat] !?
> /home/hossman/lucene/dev/solr/webapp/build/rat/rat-report.xml
> [rat] AL/home/hossman/lucene/dev/solr/webapp/web/WEB-INF/web.xml
> ...
> {noformat}
> RAT seems to be comlaining that there is no license header in it's own report
> file?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (SOLR-13582) Ref guide build script: always check for and download public keys used to verify RVM distribution signature
[
https://issues.apache.org/jira/browse/SOLR-13582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cassandra Targett resolved SOLR-13582.
--
Resolution: Won't Fix
We removed the Jenkins script in LUCENE-9452, so this is no longer needed.
> Ref guide build script: always check for and download public keys used to
> verify RVM distribution signature
> ---
>
> Key: SOLR-13582
> URL: https://issues.apache.org/jira/browse/SOLR-13582
> Project: Solr
> Issue Type: Improvement
>Reporter: Steven Rowe
>Assignee: Steven Rowe
>Priority: Minor
>
> The 8.x ref guide builds are running on a Jenkins node they've never run on
> before: "websites2". As a result, the public keys used to verify the RVM
> distribution are not on the machine, and when the RVM bootstrap encounters
> this problem, it fails the build.
> In the past I've modified the Jenkins job config to download the public keys,
> triggered a run to download the keys, and then commented out the key download
> lines in the job config's build script.
> This issue is intended to permanently solve the problem by adding the
> following to our build script
> ({{dev-tools/scripts/jenkins.build.ref.guide.sh}}), to avoid manual
> intervention when new nodes are added in the future:
> * If {{gpg}} is installed:
> ** If the keys needed to verify RVM distribution signature are not available:
> *** Download the keys
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14752) Zookeeper status fails on zk 3.6.1 with prometheus enabled
[
https://issues.apache.org/jira/browse/SOLR-14752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17183984#comment-17183984
]
Philipp Trulson commented on SOLR-14752:
Sorry for the delay guys. I just finished my PR with an added test. It would be
awesome if we could squeeze it in the 8.6.2 release.
> Zookeeper status fails on zk 3.6.1 with prometheus enabled
> --
>
> Key: SOLR-14752
> URL: https://issues.apache.org/jira/browse/SOLR-14752
> Project: Solr
> Issue Type: Bug
> Components: Admin UI
>Affects Versions: 8.6
>Reporter: Philipp Trulson
>Assignee: Jan Høydahl
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> We specifically waited for SOLR-14463 to be released so it works fine with
> zookeeper 3.6, but unfortunately we still have problems. The reason could to
> be that in
> [ZookeeperStatusHandler.java|https://github.com/apache/lucene-solr/blob/releases/lucene-solr/8.6.0/solr/core/src/java/org/apache/solr/handler/admin/ZookeeperStatusHandler.java#L152]
> it explicitly expects integers to parse. But it seems that if you enable the
> [new prometheus
> metrics|https://zookeeper.apache.org/doc/r3.6.1/zookeeperMonitor.html#Prometheus]
> in zookeeper all the values are posted as floats instead and that makes your
> check fail.
> This is our output of 'echo mntr | nc zookeeper 2181'
>
> {code:java}
> zk_version3.6.1--104dcb3e3fb464b30c5186d229e00af9f332524b, built on
> 04/21/2020 15:01 GMT
> zk_server_state follower
> zk_peer_state following - broadcast
> zk_open_file_descriptor_count 69.0
> zk_prep_processor_queue_time_ms{quantile="0.5"} NaN
> zk_prep_processor_queue_time_ms{quantile="0.9"} NaN
> zk_prep_processor_queue_time_ms{quantile="0.99"} NaN
> zk_prep_processor_queue_time_ms_count 0.0
> zk_prep_processor_queue_time_ms_sum 0.0
> zk_jvm_memory_pool_allocated_bytes_total{pool="CodeHeap 'profiled nmethods'"}
> 5386112.0
> zk_jvm_memory_pool_allocated_bytes_total{pool="G1 Old Gen"} 5381768.0
> zk_jvm_memory_pool_allocated_bytes_total{pool="G1 Eden Space"}
> 9.437184E7
> zk_jvm_memory_pool_allocated_bytes_total{pool="CodeHeap 'non-profiled
> nmethods'"} 1023616.0
> zk_jvm_memory_pool_allocated_bytes_total{pool="G1 Survivor Space"}
> 7340032.0
> zk_jvm_memory_pool_allocated_bytes_total{pool="Compressed Class Space"}
> 2739816.0
> zk_jvm_memory_pool_allocated_bytes_total{pool="Metaspace"}2.6625024E7
> zk_jvm_memory_pool_allocated_bytes_total{pool="CodeHeap 'non-nmethods'"}
> 1187456.0
> zk_fsynctime{quantile="0.5"} NaN
> zk_fsynctime_count380.0
> zk_fsynctime_sum 48.0
> zk_reads_issued_from_session_queue{quantile="0.5"}NaN
> zk_reads_issued_from_session_queue_count 386.0
> zk_reads_issued_from_session_queue_sum4.0
> zk_packets_received 2893.0
> zk_commit_commit_proc_req_queued{quantile="0.5"} 0.0
> zk_commit_commit_proc_req_queued_count3275.0
> zk_commit_commit_proc_req_queued_sum 389.0
> zk_avg_latency0.1878
> zk_dead_watchers_queued 0.0
> zk_tls_handshake_exceeded 0.0
> zk_last_client_response_size 16.0
> zk_jvm_memory_bytes_used{area="heap"} 4.1033352E7
> zk_jvm_memory_bytes_used{area="nonheap"} 3.9164144E7
> zk_jvm_memory_bytes_committed{area="heap"}1.073741824E9
> zk_jvm_memory_bytes_committed{area="nonheap"} 4.3106304E7
> zk_jvm_memory_bytes_max{area="heap"} 1.073741824E9
> zk_jvm_memory_bytes_max{area="nonheap"} -1.0
> zk_jvm_memory_bytes_init{area="heap"} 1.073741824E9
> zk_jvm_memory_bytes_init{area="nonheap"} 7667712.0
> zk_jvm_memory_pool_bytes_used{pool="CodeHeap 'non-nmethods'"} 1187456.0
> zk_jvm_memory_pool_bytes_used{pool="Metaspace"} 2.7396232E7
> zk_jvm_memory_pool_bytes_used{pool="CodeHeap 'profiled nmethods'"}
> 6577408.0
> zk_jvm_memory_pool_bytes_used{pool="Compressed Class Space"} 2739816.0
> zk_jvm_memory_pool_bytes_used{pool="G1 Eden Space"} 3.145728E7
> zk_jvm_memory_pool_bytes_used{pool="G1 Old Gen"} 5381768.0
> zk_jvm_memory_pool_bytes_used{pool="G1 Survivor Space"} 4194304.0
> zk_jvm_memory_pool_bytes_used{pool="CodeHeap 'non-profiled nmethods'"}
> 1263232.0
> zk_jvm_memory_pool_bytes_committed{pool="CodeHeap 'non-nmethods'"}
> 2555904.0
> zk_jvm_memory_pool_bytes_committed{pool="Metaspace"} 2.8360704E7
> zk_jvm_memory_pool_bytes_committed{pool="CodeHeap 'profiled nmethods'"}
> 6619136.0
> zk_jvm_memory_pool_bytes_committed{pool="Compressed Class Space"}
> 3014656.0
> zk_jvm_memory_pool_bytes_committed{pool="G1 Eden Space"} 5.24288E7
> zk_jvm_memory_pool_bytes_committed{pool="G1 Old Gen"} 1.01711872E9
> zk_jvm_memory_pool_bytes_committed{pool="G1 Survivor Space"} 4194304.0
> zk_jvm_memory_pool_by
[GitHub] [lucene-solr] der-eismann commented on pull request #1771: SOLR-14752: Fix zk status with prometheus enabled zookeeper
der-eismann commented on pull request #1771: URL: https://github.com/apache/lucene-solr/pull/1771#issuecomment-679993953 Hey, sorry for the delay. I added a separate test case and a line for the changes in Solr 8.7.0. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14752) Zookeeper status fails on zk 3.6.1 with prometheus enabled
[
https://issues.apache.org/jira/browse/SOLR-14752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17183956#comment-17183956
]
Erick Erickson commented on SOLR-14752:
---
Actually, people are talking about cutting the RC more quickly, ping Ignacio
Vera and Simon Willnauer.
It really doesn't sound like this Jira will move fast enough to make it into
8.6.2 though, especially if it takes some time to verify.
> Zookeeper status fails on zk 3.6.1 with prometheus enabled
> --
>
> Key: SOLR-14752
> URL: https://issues.apache.org/jira/browse/SOLR-14752
> Project: Solr
> Issue Type: Bug
> Components: Admin UI
>Affects Versions: 8.6
>Reporter: Philipp Trulson
>Assignee: Jan Høydahl
>Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
> We specifically waited for SOLR-14463 to be released so it works fine with
> zookeeper 3.6, but unfortunately we still have problems. The reason could to
> be that in
> [ZookeeperStatusHandler.java|https://github.com/apache/lucene-solr/blob/releases/lucene-solr/8.6.0/solr/core/src/java/org/apache/solr/handler/admin/ZookeeperStatusHandler.java#L152]
> it explicitly expects integers to parse. But it seems that if you enable the
> [new prometheus
> metrics|https://zookeeper.apache.org/doc/r3.6.1/zookeeperMonitor.html#Prometheus]
> in zookeeper all the values are posted as floats instead and that makes your
> check fail.
> This is our output of 'echo mntr | nc zookeeper 2181'
>
> {code:java}
> zk_version3.6.1--104dcb3e3fb464b30c5186d229e00af9f332524b, built on
> 04/21/2020 15:01 GMT
> zk_server_state follower
> zk_peer_state following - broadcast
> zk_open_file_descriptor_count 69.0
> zk_prep_processor_queue_time_ms{quantile="0.5"} NaN
> zk_prep_processor_queue_time_ms{quantile="0.9"} NaN
> zk_prep_processor_queue_time_ms{quantile="0.99"} NaN
> zk_prep_processor_queue_time_ms_count 0.0
> zk_prep_processor_queue_time_ms_sum 0.0
> zk_jvm_memory_pool_allocated_bytes_total{pool="CodeHeap 'profiled nmethods'"}
> 5386112.0
> zk_jvm_memory_pool_allocated_bytes_total{pool="G1 Old Gen"} 5381768.0
> zk_jvm_memory_pool_allocated_bytes_total{pool="G1 Eden Space"}
> 9.437184E7
> zk_jvm_memory_pool_allocated_bytes_total{pool="CodeHeap 'non-profiled
> nmethods'"} 1023616.0
> zk_jvm_memory_pool_allocated_bytes_total{pool="G1 Survivor Space"}
> 7340032.0
> zk_jvm_memory_pool_allocated_bytes_total{pool="Compressed Class Space"}
> 2739816.0
> zk_jvm_memory_pool_allocated_bytes_total{pool="Metaspace"}2.6625024E7
> zk_jvm_memory_pool_allocated_bytes_total{pool="CodeHeap 'non-nmethods'"}
> 1187456.0
> zk_fsynctime{quantile="0.5"} NaN
> zk_fsynctime_count380.0
> zk_fsynctime_sum 48.0
> zk_reads_issued_from_session_queue{quantile="0.5"}NaN
> zk_reads_issued_from_session_queue_count 386.0
> zk_reads_issued_from_session_queue_sum4.0
> zk_packets_received 2893.0
> zk_commit_commit_proc_req_queued{quantile="0.5"} 0.0
> zk_commit_commit_proc_req_queued_count3275.0
> zk_commit_commit_proc_req_queued_sum 389.0
> zk_avg_latency0.1878
> zk_dead_watchers_queued 0.0
> zk_tls_handshake_exceeded 0.0
> zk_last_client_response_size 16.0
> zk_jvm_memory_bytes_used{area="heap"} 4.1033352E7
> zk_jvm_memory_bytes_used{area="nonheap"} 3.9164144E7
> zk_jvm_memory_bytes_committed{area="heap"}1.073741824E9
> zk_jvm_memory_bytes_committed{area="nonheap"} 4.3106304E7
> zk_jvm_memory_bytes_max{area="heap"} 1.073741824E9
> zk_jvm_memory_bytes_max{area="nonheap"} -1.0
> zk_jvm_memory_bytes_init{area="heap"} 1.073741824E9
> zk_jvm_memory_bytes_init{area="nonheap"} 7667712.0
> zk_jvm_memory_pool_bytes_used{pool="CodeHeap 'non-nmethods'"} 1187456.0
> zk_jvm_memory_pool_bytes_used{pool="Metaspace"} 2.7396232E7
> zk_jvm_memory_pool_bytes_used{pool="CodeHeap 'profiled nmethods'"}
> 6577408.0
> zk_jvm_memory_pool_bytes_used{pool="Compressed Class Space"} 2739816.0
> zk_jvm_memory_pool_bytes_used{pool="G1 Eden Space"} 3.145728E7
> zk_jvm_memory_pool_bytes_used{pool="G1 Old Gen"} 5381768.0
> zk_jvm_memory_pool_bytes_used{pool="G1 Survivor Space"} 4194304.0
> zk_jvm_memory_pool_bytes_used{pool="CodeHeap 'non-profiled nmethods'"}
> 1263232.0
> zk_jvm_memory_pool_bytes_committed{pool="CodeHeap 'non-nmethods'"}
> 2555904.0
> zk_jvm_memory_pool_bytes_committed{pool="Metaspace"} 2.8360704E7
> zk_jvm_memory_pool_bytes_committed{pool="CodeHeap 'profiled nmethods'"}
> 6619136.0
> zk_jvm_memory_pool_bytes_committed{pool="Compressed Class Space"}
> 3014656.0
> zk_jvm_memory_pool_bytes_committed{pool="G1 Eden Space"} 5.24288E7
> zk_jvm_memory_pool_bytes_committed{pool="G1 Old Gen"} 1.0171
[jira] [Updated] (LUCENE-9481) Improve FST performance created by Builder
[
https://issues.apache.org/jira/browse/LUCENE-9481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anh Dung Bui updated LUCENE-9481:
-
Description:
Hi,
We are using Lucene 8 and recently we observed a performance issue with FST.
When FST is used directly from *org.apache.lucene.util.fst.Builder.finish()*,
its about 40-50% slower than when it's first saved to a *DataOutput* (e.g
filesystem or heap) and loaded again. The FST we used is about 1.6MB.
Using VisualVM, we tracked the issue down to the *DataInput.readVInt* method,
which in turn calls the readByte of the reversed BytesReader. When FST is
loaded from filesystem/heap, it's using OnHeapFSTStore which use
ReverseBytesReader, but when it's created by the Builder, it's backed by the
BytesStore. The problem is, when the number of blocks of BytesStore is not 1,
it'll use an [anonymous
class|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/util/fst/BytesStore.java#L453]
for reading bytes, and possibly because of JVM inlining, the
ReverseBytesReader.readByte is inlined, while that of the anonymous class is
not. We confirmed this by adding XX:+PrintInlining flag.
{code:java}
@ 80 org.apache.lucene.util.fst.BytesStore$2::readByte (68 bytes) too big{code}
{code:java}
@ 17 org.apache.lucene.util.fst.ReverseBytesReader::readByte (17 bytes) inline
(hot){code}
Currently we have 2 workarounds, which can also confirm the above theory:
* Use a larger *bytesPageBits* in Builder, so that the block size is greater
than the FST size.
* Save and load the FST to heap after building, e.g using
GrowableByteArrayDataOutput and ByteArrayDataInput.
But I think it would be better if this optimization is done inside
*Builder.finish()* instead, e.g by auto-adjust the block size of BytesStore or
copy it to a FSTStore if possible, since none of the 2 workarounds above look
great. We have many FST with largely different sizes, so with the first option
we need to maintain the bytesPageBits for each dictionary, otherwise we risk
wasting unnecessary heap or using insufficient block size. I also think this
issue would affect other Lucene users too.
For reference, the benchmark we setup is as below:
* Warm up both FST for 10.000 iterations (10.000 seems to be the JVM inlining
trigger threshold)
* Run 100 tests, each test run 100.000 iterations
* Each iteration is simply call *org.apache.lucene.util.fst.Util.get()* with
the same 4-character word.
It reports that 98 out of 100 tests, the one with save/load logic is 40-50%
faster on average.
Thanks
was:
Hi,
We are using Lucene 8 and recently we observed a performance issue with FST.
When FST is used directly from *org.apache.lucene.util.fst.Builder.finish()*,
its about 40-50% slower than when it's first saved to a *DataOutput* (e.g
filesystem or heap) and loaded again. The FST we used is about 1.6MB.
Using VisualVM, we tracked the issue down to the *DataInput.readVInt* method,
which in turn calls the readByte of the reversed BytesReader. When FST is
loaded from filesystem/heap, it's using OnHeapFSTStore which use
ReverseBytesReader, but when it's created by the Builder, it's backed by the
BytesStore. The problem is, when the block size of BytesStore is not 1, it'll
use an [anonymous
class|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/util/fst/BytesStore.java#L453]
for reading bytes, and possibly because of JVM inlining, the
ReverseBytesReader.readByte is inlined, while that of the anonymous class is
not. We confirmed this by adding XX:+PrintInlining flag.
{code:java}
@ 80 org.apache.lucene.util.fst.BytesStore$2::readByte (68 bytes) too big{code}
{code:java}
@ 17 org.apache.lucene.util.fst.ReverseBytesReader::readByte (17 bytes) inline
(hot){code}
Currently we have 2 workarounds, which can also confirm the above theory:
* Use a larger *bytesPageBits* in Builder, so that the block size is greater
than the FST size.
* Save and load the FST to heap after building, e.g using
GrowableByteArrayDataOutput and ByteArrayDataInput.
But I think it would be better if this optimization is done inside
*Builder.finish()* instead, e.g by auto-adjust the block size of BytesStore or
copy it to a FSTStore if possible, since none of the 2 workarounds above look
great. We have many FST with largely different sizes, so with the first option
we need to maintain the bytesPageBits for each dictionary, otherwise we risk
wasting unnecessary heap or using insufficient block size. I also think this
issue would affect other Lucene users too.
For reference, the benchmark we setup is as below:
* Warm up both FST for 10.000 iterations (10.000 seems to be the JVM inlining
trigger threshold)
* Run 100 tests, each test run 100.000 iterations
* Each iteration is simply call *org.apache.lucene.util.fst.Util.get()* with
the same 4-character
[jira] [Comment Edited] (LUCENE-9474) Change Jenkins jobs to use Gradle for trunk
[
https://issues.apache.org/jira/browse/LUCENE-9474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17183926#comment-17183926
]
Uwe Schindler edited comment on LUCENE-9474 at 8/25/20, 10:50 AM:
--
Don't wonder about the names of env vars, mostly historic since JVM
randomization first implemented in 2012. The new one added by this issue is
only {{RUNTIME_JAVA_HOME}}. IMHO all the args should be renamed to
{{RUNTIME_}}, but that involves touching all jobs (Gradle, Ant).
was (Author: thetaphi):
Don't wonder about the names ov env vars, mostly historic since JVM
randomization first implemented in 2012. The new one added by this issue is
only {{RUNTIME_JAVA_HOME}}. IMHO all the args should be renamed to
{{RUNTIME_}}, but that involves touching all jobs (Gradle, Ant).
> Change Jenkins jobs to use Gradle for trunk
> ---
>
> Key: LUCENE-9474
> URL: https://issues.apache.org/jira/browse/LUCENE-9474
> Project: Lucene - Core
> Issue Type: Test
> Components: general/build
>Reporter: Erick Erickson
>Assignee: Uwe Schindler
>Priority: Major
> Attachments: screenshot-1.png, screenshot-2.png, screenshot-3.png,
> screenshot-4.png, screenshot-5.png, screenshot-6.png
>
>
> I rushed the gate and pushed LUCENE-9433 without coordinating, my apologies
> for the confusion.
> Meanwhile, Uwe has disabled Jenkins jobs for the weekend and we'll fix this
> up Real Soon Now.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9474) Change Jenkins jobs to use Gradle for trunk
[
https://issues.apache.org/jira/browse/LUCENE-9474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17183926#comment-17183926
]
Uwe Schindler commented on LUCENE-9474:
---
Don't wonder about the names ov env vars, mostly historic since JVM
randomization first implemented in 2012. The new one added by this issue is
only {{RUNTIME_JAVA_HOME}}. IMHO all the args should be renamed to
{{RUNTIME_}}, but that involves touching all jobs (Gradle, Ant).
> Change Jenkins jobs to use Gradle for trunk
> ---
>
> Key: LUCENE-9474
> URL: https://issues.apache.org/jira/browse/LUCENE-9474
> Project: Lucene - Core
> Issue Type: Test
> Components: general/build
>Reporter: Erick Erickson
>Assignee: Uwe Schindler
>Priority: Major
> Attachments: screenshot-1.png, screenshot-2.png, screenshot-3.png,
> screenshot-4.png, screenshot-5.png, screenshot-6.png
>
>
> I rushed the gate and pushed LUCENE-9433 without coordinating, my apologies
> for the confusion.
> Meanwhile, Uwe has disabled Jenkins jobs for the weekend and we'll fix this
> up Real Soon Now.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9474) Change Jenkins jobs to use Gradle for trunk
[
https://issues.apache.org/jira/browse/LUCENE-9474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17183921#comment-17183921
]
Uwe Schindler commented on LUCENE-9474:
---
On Policeman Jenkins, looks like this:
Prepare Environment for the run:
!screenshot-4.png|width=500!
The script is part of Jenkins installation and provides {{RUNTIME_JAVA_HOME}}
and {{TEST_JVM_ARGS}} environment variables, which are automatically consumed
by Gradle (thanks [~dweiss]). It also sets some other environment vars for
formatting the build@lao mails and the description:
!screenshot-5.png|width=500!
Here is the environment, {{JAVA_HOME}} comes from the default selection by
Jenkins UI (see above; colored green and used to run Gradle). The Blue ones are
for mails and Job description, the red ones are technical. The libeatmydata
preload is there to protect my NVME SSD raid array from fsyncs ran by (Solr)
tests all the time!
!screenshot-6.png|width=500!
> Change Jenkins jobs to use Gradle for trunk
> ---
>
> Key: LUCENE-9474
> URL: https://issues.apache.org/jira/browse/LUCENE-9474
> Project: Lucene - Core
> Issue Type: Test
> Components: general/build
>Reporter: Erick Erickson
>Assignee: Uwe Schindler
>Priority: Major
> Attachments: screenshot-1.png, screenshot-2.png, screenshot-3.png,
> screenshot-4.png, screenshot-5.png, screenshot-6.png
>
>
> I rushed the gate and pushed LUCENE-9433 without coordinating, my apologies
> for the confusion.
> Meanwhile, Uwe has disabled Jenkins jobs for the weekend and we'll fix this
> up Real Soon Now.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-9474) Change Jenkins jobs to use Gradle for trunk
[ https://issues.apache.org/jira/browse/LUCENE-9474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uwe Schindler updated LUCENE-9474: -- Attachment: screenshot-6.png > Change Jenkins jobs to use Gradle for trunk > --- > > Key: LUCENE-9474 > URL: https://issues.apache.org/jira/browse/LUCENE-9474 > Project: Lucene - Core > Issue Type: Test > Components: general/build >Reporter: Erick Erickson >Assignee: Uwe Schindler >Priority: Major > Attachments: screenshot-1.png, screenshot-2.png, screenshot-3.png, > screenshot-4.png, screenshot-5.png, screenshot-6.png > > > I rushed the gate and pushed LUCENE-9433 without coordinating, my apologies > for the confusion. > Meanwhile, Uwe has disabled Jenkins jobs for the weekend and we'll fix this > up Real Soon Now. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-9474) Change Jenkins jobs to use Gradle for trunk
[ https://issues.apache.org/jira/browse/LUCENE-9474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uwe Schindler updated LUCENE-9474: -- Attachment: screenshot-5.png > Change Jenkins jobs to use Gradle for trunk > --- > > Key: LUCENE-9474 > URL: https://issues.apache.org/jira/browse/LUCENE-9474 > Project: Lucene - Core > Issue Type: Test > Components: general/build >Reporter: Erick Erickson >Assignee: Uwe Schindler >Priority: Major > Attachments: screenshot-1.png, screenshot-2.png, screenshot-3.png, > screenshot-4.png, screenshot-5.png > > > I rushed the gate and pushed LUCENE-9433 without coordinating, my apologies > for the confusion. > Meanwhile, Uwe has disabled Jenkins jobs for the weekend and we'll fix this > up Real Soon Now. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-9474) Change Jenkins jobs to use Gradle for trunk
[ https://issues.apache.org/jira/browse/LUCENE-9474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uwe Schindler updated LUCENE-9474: -- Attachment: screenshot-4.png > Change Jenkins jobs to use Gradle for trunk > --- > > Key: LUCENE-9474 > URL: https://issues.apache.org/jira/browse/LUCENE-9474 > Project: Lucene - Core > Issue Type: Test > Components: general/build >Reporter: Erick Erickson >Assignee: Uwe Schindler >Priority: Major > Attachments: screenshot-1.png, screenshot-2.png, screenshot-3.png, > screenshot-4.png > > > I rushed the gate and pushed LUCENE-9433 without coordinating, my apologies > for the confusion. > Meanwhile, Uwe has disabled Jenkins jobs for the weekend and we'll fix this > up Real Soon Now. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9474) Change Jenkins jobs to use Gradle for trunk
[ https://issues.apache.org/jira/browse/LUCENE-9474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17183916#comment-17183916 ] Uwe Schindler commented on LUCENE-9474: --- Here the "Howto" for ASF Jenkins: The config file is placed in The "Lucene" Folder's config file store (not locally on the nodes for reasons seen above). One for both Jenkins nodes, we will soon make them "identical", as one node needs to be migrated into other data center anyways (INFRA-20657, [https://cwiki.apache.org/confluence/display/INFRA/Puppet+3+to+Puppet+6+Migration]): !screenshot-1.png|width=500! The Jobs are configured like this, applies to all of them (except the Refguide build, as it needs no special config). Example [https://ci-builds.apache.org/job/Lucene/job/Lucene-Solr-Check-master/configure]: !screenshot-2.png|width=500! Here is the gradle invocation (same like on Policeman Jenkins): !screenshot-3.png|width=500! I removed the Repro trigger and the "Ant build" and checkbox "Use ant". > Change Jenkins jobs to use Gradle for trunk > --- > > Key: LUCENE-9474 > URL: https://issues.apache.org/jira/browse/LUCENE-9474 > Project: Lucene - Core > Issue Type: Test > Components: general/build >Reporter: Erick Erickson >Assignee: Uwe Schindler >Priority: Major > Attachments: screenshot-1.png, screenshot-2.png, screenshot-3.png > > > I rushed the gate and pushed LUCENE-9433 without coordinating, my apologies > for the confusion. > Meanwhile, Uwe has disabled Jenkins jobs for the weekend and we'll fix this > up Real Soon Now. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-9474) Change Jenkins jobs to use Gradle for trunk
[ https://issues.apache.org/jira/browse/LUCENE-9474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uwe Schindler updated LUCENE-9474: -- Attachment: screenshot-3.png > Change Jenkins jobs to use Gradle for trunk > --- > > Key: LUCENE-9474 > URL: https://issues.apache.org/jira/browse/LUCENE-9474 > Project: Lucene - Core > Issue Type: Test > Components: general/build >Reporter: Erick Erickson >Assignee: Uwe Schindler >Priority: Major > Attachments: screenshot-1.png, screenshot-2.png, screenshot-3.png > > > I rushed the gate and pushed LUCENE-9433 without coordinating, my apologies > for the confusion. > Meanwhile, Uwe has disabled Jenkins jobs for the weekend and we'll fix this > up Real Soon Now. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-9474) Change Jenkins jobs to use Gradle for trunk
[ https://issues.apache.org/jira/browse/LUCENE-9474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uwe Schindler updated LUCENE-9474: -- Attachment: screenshot-2.png > Change Jenkins jobs to use Gradle for trunk > --- > > Key: LUCENE-9474 > URL: https://issues.apache.org/jira/browse/LUCENE-9474 > Project: Lucene - Core > Issue Type: Test > Components: general/build >Reporter: Erick Erickson >Assignee: Uwe Schindler >Priority: Major > Attachments: screenshot-1.png, screenshot-2.png, screenshot-3.png > > > I rushed the gate and pushed LUCENE-9433 without coordinating, my apologies > for the confusion. > Meanwhile, Uwe has disabled Jenkins jobs for the weekend and we'll fix this > up Real Soon Now. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-9474) Change Jenkins jobs to use Gradle for trunk
[ https://issues.apache.org/jira/browse/LUCENE-9474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uwe Schindler updated LUCENE-9474: -- Attachment: screenshot-1.png > Change Jenkins jobs to use Gradle for trunk > --- > > Key: LUCENE-9474 > URL: https://issues.apache.org/jira/browse/LUCENE-9474 > Project: Lucene - Core > Issue Type: Test > Components: general/build >Reporter: Erick Erickson >Assignee: Uwe Schindler >Priority: Major > Attachments: screenshot-1.png > > > I rushed the gate and pushed LUCENE-9433 without coordinating, my apologies > for the confusion. > Meanwhile, Uwe has disabled Jenkins jobs for the weekend and we'll fix this > up Real Soon Now. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] MarcusSorealheis commented on pull request #1781: SOLR-14777: Fix an ignore test that is helpful.
MarcusSorealheis commented on pull request #1781: URL: https://github.com/apache/lucene-solr/pull/1781#issuecomment-679910659 I'm inclined to believe that there is another test that is no longer so flakey that also extends `EmbeddedSolrServerTestBase` but I want to keep the scope limited for reviews because of _other_ challenging tests in the branch atm. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-14768) Error 500 on PDF extraction: java.lang.NoClassDefFoundError: org/eclipse/jetty/server/MultiParts
[
https://issues.apache.org/jira/browse/SOLR-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17183515#comment-17183515
]
Joe Doupnik edited comment on SOLR-14768 at 8/25/20, 9:00 AM:
--
My observations were described with pictures added, thus to see them please
refer to the user's list, 23 August. This include how to reproduce code
snippets.
I have added my report (from solr-users) as an email attachment to the main
item.
Joe D.
was (Author: jdoupnik):
My observations were described with pictures added, thus to see them please
refer to the user's list, 23 August. This include how to reproduce code
snippets.
Joe D.
> Error 500 on PDF extraction: java.lang.NoClassDefFoundError:
> org/eclipse/jetty/server/MultiParts
>
>
> Key: SOLR-14768
> URL: https://issues.apache.org/jira/browse/SOLR-14768
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: contrib - Solr Cell (Tika extraction)
>Affects Versions: 8.6, 8.6.1
>Reporter: Markus Kalkbrenner
>Priority: Major
> Attachments: Solr v8.6.x fails with multipart MIME in commands.eml
>
>
> See [https://www.mail-archive.com/solr-user@lucene.apache.org/msg152182.html]
> The integration tests of the solarium PHP library and the integration tests
> of the Search API Solr Drupal module both fail on PDF extraction if executed
> on Solr 8.6.
> They still work on Solr 8.5.1 an earlier versions.
> {quote}2020-08-20 12:30:35.279 INFO (qtp855700733-19) [ x:5f3e6ce2810ef]
> o.a.s.u.p.LogUpdateProcessorFactory [5f3e6ce2810ef] webapp=/solr
> path=/update/extract
> params=\{json.nl=flat&commitWithin=0&omitHeader=false&resource.name=testpdf.pdf&literal.id=extract-test&commit=true&extractOnly=false&uprefix=attr_&wt=json}{add=[extract-test
> (1675547519474466816)],commit=} 0 957
> solr8_1 | 2020-08-20 12:30:35.280 WARN (qtp855700733-19) [ ]
> o.e.j.s.HttpChannel /solr/5f3e6ce2810ef/update/extract =>
> java.lang.NoClassDefFoundError: org/eclipse/jetty/server/MultiParts
> solr8_1 | at
> org.apache.solr.servlet.SolrRequestParsers.cleanupMultipartFiles(SolrRequestParsers.java:624)
> solr8_1 | java.lang.NoClassDefFoundError: org/eclipse/jetty/server/MultiParts
> solr8_1 | at
> org.apache.solr.servlet.SolrRequestParsers.cleanupMultipartFiles(SolrRequestParsers.java:624)
> ~[?:?]
> solr8_1 | at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:443)
> ~[?:?]
> solr8_1 | at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)
> ~[?:?]
> solr8_1 | at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596)
> ~[jetty-servlet-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
> ~[jetty-servlet-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590)
> ~[jetty-security-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1300)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
> ~[jetty-servlet-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1215)
> ~[jetty-server-9.
[jira] [Updated] (SOLR-14768) Error 500 on PDF extraction: java.lang.NoClassDefFoundError: org/eclipse/jetty/server/MultiParts
[
https://issues.apache.org/jira/browse/SOLR-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joe Doupnik updated SOLR-14768:
---
Attachment: Solr v8.6.x fails with multipart MIME in commands.eml
> Error 500 on PDF extraction: java.lang.NoClassDefFoundError:
> org/eclipse/jetty/server/MultiParts
>
>
> Key: SOLR-14768
> URL: https://issues.apache.org/jira/browse/SOLR-14768
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: contrib - Solr Cell (Tika extraction)
>Affects Versions: 8.6, 8.6.1
>Reporter: Markus Kalkbrenner
>Priority: Major
> Attachments: Solr v8.6.x fails with multipart MIME in commands.eml
>
>
> See [https://www.mail-archive.com/solr-user@lucene.apache.org/msg152182.html]
> The integration tests of the solarium PHP library and the integration tests
> of the Search API Solr Drupal module both fail on PDF extraction if executed
> on Solr 8.6.
> They still work on Solr 8.5.1 an earlier versions.
> {quote}2020-08-20 12:30:35.279 INFO (qtp855700733-19) [ x:5f3e6ce2810ef]
> o.a.s.u.p.LogUpdateProcessorFactory [5f3e6ce2810ef] webapp=/solr
> path=/update/extract
> params=\{json.nl=flat&commitWithin=0&omitHeader=false&resource.name=testpdf.pdf&literal.id=extract-test&commit=true&extractOnly=false&uprefix=attr_&wt=json}{add=[extract-test
> (1675547519474466816)],commit=} 0 957
> solr8_1 | 2020-08-20 12:30:35.280 WARN (qtp855700733-19) [ ]
> o.e.j.s.HttpChannel /solr/5f3e6ce2810ef/update/extract =>
> java.lang.NoClassDefFoundError: org/eclipse/jetty/server/MultiParts
> solr8_1 | at
> org.apache.solr.servlet.SolrRequestParsers.cleanupMultipartFiles(SolrRequestParsers.java:624)
> solr8_1 | java.lang.NoClassDefFoundError: org/eclipse/jetty/server/MultiParts
> solr8_1 | at
> org.apache.solr.servlet.SolrRequestParsers.cleanupMultipartFiles(SolrRequestParsers.java:624)
> ~[?:?]
> solr8_1 | at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:443)
> ~[?:?]
> solr8_1 | at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)
> ~[?:?]
> solr8_1 | at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596)
> ~[jetty-servlet-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
> ~[jetty-servlet-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590)
> ~[jetty-security-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1300)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
> ~[jetty-servlet-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1215)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221)
> ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
> solr8_1 | at
> org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:177)
> ~[jetty-serve
[GitHub] [lucene-solr] MarcusSorealheis commented on pull request #1781: SOLR-14777: Fix an ignore test that is helpful.
MarcusSorealheis commented on pull request #1781: URL: https://github.com/apache/lucene-solr/pull/1781#issuecomment-679894298 I am seeing issues on pre-commit but I did not change any of those files. They should be fine: ``` BUILD FAILED /home/runner/work/lucene-solr/lucene-solr/build.xml:128: Found 489 violations in source files (invalid logger name [log, uses static class name, not specialized logger], invalid logging pattern [not private static final, uses static class name], nocommit). ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] MarcusSorealheis commented on pull request #1781: SOLR-14777: Fix an ignore test that is helpful.
MarcusSorealheis commented on pull request #1781: URL: https://github.com/apache/lucene-solr/pull/1781#issuecomment-679893584  This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] MarcusSorealheis commented on pull request #1781: SOLR-14777: Fix an ignore test that is helpful.
MarcusSorealheis commented on pull request #1781: URL: https://github.com/apache/lucene-solr/pull/1781#issuecomment-679892656 @markrmiller Is this the right idea for removing ignores? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] MarcusSorealheis opened a new pull request #1781: SOLR-14777: Fix an ignore test that is helpful.
MarcusSorealheis opened a new pull request #1781: URL: https://github.com/apache/lucene-solr/pull/1781 # Description There are a number of tests that are annotated `@Ignore` but we still need them. This is a first step to start to chip away at the ignores so that the team is no impacted. # Solution Fix an ignore test that is helpful. # Tests I am adding a test that was previously ignored. # Checklist Please review the following and check all that apply: - [ ] I have reviewed the guidelines for [How to Contribute](https://wiki.apache.org/solr/HowToContribute) and my code conforms to the standards described there to the best of my ability. - [ ] I have created a Jira issue and added the issue ID to my pull request title. - [ ] I have given Solr maintainers [access](https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork) to contribute to my PR branch. (optional but recommended) - [ ] I have developed this patch against the `master` branch. - [ ] I have run `ant precommit` and the appropriate test suite. - [ ] I have added tests for my changes. - [ ] I have added documentation for the [Ref Guide](https://github.com/apache/lucene-solr/tree/master/solr/solr-ref-guide) (for Solr changes only). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9481) Improve FST performance created by Builder
[
https://issues.apache.org/jira/browse/LUCENE-9481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17183859#comment-17183859
]
Anh Dung Bui commented on LUCENE-9481:
--
I realize
[SynonymMap|https://github.com/apache/lucene-solr/blob/master/lucene/analysis/common/src/java/org/apache/lucene/analysis/synonym/SynonymMap.java#L217]
also uses FST and might have the same problem, although Builder has been
renamed to FSTCompiler now.
> Improve FST performance created by Builder
> --
>
> Key: LUCENE-9481
> URL: https://issues.apache.org/jira/browse/LUCENE-9481
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/FSTs
>Reporter: Anh Dung Bui
>Priority: Major
>
> Hi,
>
> We are using Lucene 8 and recently we observed a performance issue with FST.
> When FST is used directly from *org.apache.lucene.util.fst.Builder.finish()*,
> its about 40-50% slower than when it's first saved to a *DataOutput* (e.g
> filesystem or heap) and loaded again. The FST we used is about 1.6MB.
>
> Using VisualVM, we tracked the issue down to the *DataInput.readVInt* method,
> which in turn calls the readByte of the reversed BytesReader. When FST is
> loaded from filesystem/heap, it's using OnHeapFSTStore which use
> ReverseBytesReader, but when it's created by the Builder, it's backed by the
> BytesStore. The problem is, when the block size of BytesStore is not 1, it'll
> use an [anonymous
> class|https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/util/fst/BytesStore.java#L453]
> for reading bytes, and possibly because of JVM inlining, the
> ReverseBytesReader.readByte is inlined, while that of the anonymous class is
> not. We confirmed this by adding XX:+PrintInlining flag.
> {code:java}
> @ 80 org.apache.lucene.util.fst.BytesStore$2::readByte (68 bytes) too
> big{code}
> {code:java}
> @ 17 org.apache.lucene.util.fst.ReverseBytesReader::readByte (17 bytes)
> inline (hot){code}
>
> Currently we have 2 workarounds, which can also confirm the above theory:
> * Use a larger *bytesPageBits* in Builder, so that the block size is greater
> than the FST size.
> * Save and load the FST to heap after building, e.g using
> GrowableByteArrayDataOutput and ByteArrayDataInput.
> But I think it would be better if this optimization is done inside
> *Builder.finish()* instead, e.g by auto-adjust the block size of BytesStore
> or copy it to a FSTStore if possible, since none of the 2 workarounds above
> look great. We have many FST with largely different sizes, so with the first
> option we need to maintain the bytesPageBits for each dictionary, otherwise
> we risk wasting unnecessary heap or using insufficient block size. I also
> think this issue would affect other Lucene users too.
>
> For reference, the benchmark we setup is as below:
> * Warm up both FST for 10.000 iterations (10.000 seems to be the JVM
> inlining trigger threshold)
> * Run 100 tests, each test run 100.000 iterations
> * Each iteration is simply call *org.apache.lucene.util.fst.Util.get()* with
> the same 4-character word.
> It reports that 98 out of 100 tests, the one with save/load logic is 40-50%
> faster on average.
> Thanks
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9474) Change Jenkins jobs to use Gradle for trunk
[ https://issues.apache.org/jira/browse/LUCENE-9474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17183851#comment-17183851 ] Dawid Weiss commented on LUCENE-9474: - Code coverage issue is here - LUCENE-9188. I haven't touched it. > Change Jenkins jobs to use Gradle for trunk > --- > > Key: LUCENE-9474 > URL: https://issues.apache.org/jira/browse/LUCENE-9474 > Project: Lucene - Core > Issue Type: Test > Components: general/build >Reporter: Erick Erickson >Assignee: Uwe Schindler >Priority: Major > > I rushed the gate and pushed LUCENE-9433 without coordinating, my apologies > for the confusion. > Meanwhile, Uwe has disabled Jenkins jobs for the weekend and we'll fix this > up Real Soon Now. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] chatman commented on pull request #1648: SOLR-14616: Remove CDCR from Solr 9.x
chatman commented on pull request #1648: URL: https://github.com/apache/lucene-solr/pull/1648#issuecomment-679868116 Unfortunately, thanks to some stupidity on my part, I lost the branch where I raised this PR from. I think all the changes need to be re-done now :-( I'm closed this PR for now, will raise a new one. Thanks and apologies @ErickErickson. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] chatman closed pull request #1648: SOLR-14616: Remove CDCR from Solr 9.x
chatman closed pull request #1648: URL: https://github.com/apache/lucene-solr/pull/1648 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14726) Streamline getting started experience
[
https://issues.apache.org/jira/browse/SOLR-14726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17183826#comment-17183826
]
Jan Høydahl commented on SOLR-14726:
When I do Solr training courses I like to get the student as quickly as
possible to some real value results, e.g. {{bin/post -c mycoll file.csv}}, and
then in an optional step explain the more verbose equivalent {{curl -XPOST -H
"Content-Type: text/csv" [http://localhost:8983/solr/mycoll/update?commit=true]
--data-binary @file.csv}}. Believe it or not, some students actually *type* in
those commands by hand instead of copy/paste :)
> Streamline getting started experience
> -
>
> Key: SOLR-14726
> URL: https://issues.apache.org/jira/browse/SOLR-14726
> Project: Solr
> Issue Type: Task
>Reporter: Ishan Chattopadhyaya
>Assignee: Alexandre Rafalovitch
>Priority: Major
> Labels: newdev
> Attachments: yasa-http.png
>
>
> The reference guide Solr tutorial is here:
> https://lucene.apache.org/solr/guide/8_6/solr-tutorial.html
> It needs to be simplified and easy to follow. Also, it should reflect our
> best practices, that should also be followed in production. I have following
> suggestions:
> # Make it less verbose. It is too long. On my laptop, it required 35 page
> downs button presses to get to the bottom of the page!
> # First step of the tutorial should be to enable security (basic auth should
> suffice).
> # {{./bin/solr start -e cloud}} <-- All references of -e should be removed.
> # All references of {{bin/solr post}} to be replaced with {{curl}}
> # Convert all {{bin/solr create}} references to curl of collection creation
> commands
> # Add docker based startup instructions.
> # Create a Jupyter Notebook version of the entire tutorial, make it so that
> it can be easily executed from Google Colaboratory. Here's an example:
> https://twitter.com/TheSearchStack/status/1289703715981496320
> # Provide downloadable Postman and Insomnia files so that the same tutorial
> can be executed from those tools. Except for starting Solr, all other steps
> should be possible to be carried out from those tools.
> # Use V2 APIs everywhere in the tutorial
> # Remove all example modes, sample data (films, tech products etc.),
> configsets from Solr's distribution (instead let the examples refer to them
> from github)
> # Remove the post tool from Solr, curl should suffice.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14752) Zookeeper status fails on zk 3.6.1 with prometheus enabled
[
https://issues.apache.org/jira/browse/SOLR-14752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17183824#comment-17183824
]
Jan Høydahl commented on SOLR-14752:
There'll be a 8.6.2 end of this week, perhaps we could piggy-back?
> Zookeeper status fails on zk 3.6.1 with prometheus enabled
> --
>
> Key: SOLR-14752
> URL: https://issues.apache.org/jira/browse/SOLR-14752
> Project: Solr
> Issue Type: Bug
> Components: Admin UI
>Affects Versions: 8.6
>Reporter: Philipp Trulson
>Assignee: Jan Høydahl
>Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
> We specifically waited for SOLR-14463 to be released so it works fine with
> zookeeper 3.6, but unfortunately we still have problems. The reason could to
> be that in
> [ZookeeperStatusHandler.java|https://github.com/apache/lucene-solr/blob/releases/lucene-solr/8.6.0/solr/core/src/java/org/apache/solr/handler/admin/ZookeeperStatusHandler.java#L152]
> it explicitly expects integers to parse. But it seems that if you enable the
> [new prometheus
> metrics|https://zookeeper.apache.org/doc/r3.6.1/zookeeperMonitor.html#Prometheus]
> in zookeeper all the values are posted as floats instead and that makes your
> check fail.
> This is our output of 'echo mntr | nc zookeeper 2181'
>
> {code:java}
> zk_version3.6.1--104dcb3e3fb464b30c5186d229e00af9f332524b, built on
> 04/21/2020 15:01 GMT
> zk_server_state follower
> zk_peer_state following - broadcast
> zk_open_file_descriptor_count 69.0
> zk_prep_processor_queue_time_ms{quantile="0.5"} NaN
> zk_prep_processor_queue_time_ms{quantile="0.9"} NaN
> zk_prep_processor_queue_time_ms{quantile="0.99"} NaN
> zk_prep_processor_queue_time_ms_count 0.0
> zk_prep_processor_queue_time_ms_sum 0.0
> zk_jvm_memory_pool_allocated_bytes_total{pool="CodeHeap 'profiled nmethods'"}
> 5386112.0
> zk_jvm_memory_pool_allocated_bytes_total{pool="G1 Old Gen"} 5381768.0
> zk_jvm_memory_pool_allocated_bytes_total{pool="G1 Eden Space"}
> 9.437184E7
> zk_jvm_memory_pool_allocated_bytes_total{pool="CodeHeap 'non-profiled
> nmethods'"} 1023616.0
> zk_jvm_memory_pool_allocated_bytes_total{pool="G1 Survivor Space"}
> 7340032.0
> zk_jvm_memory_pool_allocated_bytes_total{pool="Compressed Class Space"}
> 2739816.0
> zk_jvm_memory_pool_allocated_bytes_total{pool="Metaspace"}2.6625024E7
> zk_jvm_memory_pool_allocated_bytes_total{pool="CodeHeap 'non-nmethods'"}
> 1187456.0
> zk_fsynctime{quantile="0.5"} NaN
> zk_fsynctime_count380.0
> zk_fsynctime_sum 48.0
> zk_reads_issued_from_session_queue{quantile="0.5"}NaN
> zk_reads_issued_from_session_queue_count 386.0
> zk_reads_issued_from_session_queue_sum4.0
> zk_packets_received 2893.0
> zk_commit_commit_proc_req_queued{quantile="0.5"} 0.0
> zk_commit_commit_proc_req_queued_count3275.0
> zk_commit_commit_proc_req_queued_sum 389.0
> zk_avg_latency0.1878
> zk_dead_watchers_queued 0.0
> zk_tls_handshake_exceeded 0.0
> zk_last_client_response_size 16.0
> zk_jvm_memory_bytes_used{area="heap"} 4.1033352E7
> zk_jvm_memory_bytes_used{area="nonheap"} 3.9164144E7
> zk_jvm_memory_bytes_committed{area="heap"}1.073741824E9
> zk_jvm_memory_bytes_committed{area="nonheap"} 4.3106304E7
> zk_jvm_memory_bytes_max{area="heap"} 1.073741824E9
> zk_jvm_memory_bytes_max{area="nonheap"} -1.0
> zk_jvm_memory_bytes_init{area="heap"} 1.073741824E9
> zk_jvm_memory_bytes_init{area="nonheap"} 7667712.0
> zk_jvm_memory_pool_bytes_used{pool="CodeHeap 'non-nmethods'"} 1187456.0
> zk_jvm_memory_pool_bytes_used{pool="Metaspace"} 2.7396232E7
> zk_jvm_memory_pool_bytes_used{pool="CodeHeap 'profiled nmethods'"}
> 6577408.0
> zk_jvm_memory_pool_bytes_used{pool="Compressed Class Space"} 2739816.0
> zk_jvm_memory_pool_bytes_used{pool="G1 Eden Space"} 3.145728E7
> zk_jvm_memory_pool_bytes_used{pool="G1 Old Gen"} 5381768.0
> zk_jvm_memory_pool_bytes_used{pool="G1 Survivor Space"} 4194304.0
> zk_jvm_memory_pool_bytes_used{pool="CodeHeap 'non-profiled nmethods'"}
> 1263232.0
> zk_jvm_memory_pool_bytes_committed{pool="CodeHeap 'non-nmethods'"}
> 2555904.0
> zk_jvm_memory_pool_bytes_committed{pool="Metaspace"} 2.8360704E7
> zk_jvm_memory_pool_bytes_committed{pool="CodeHeap 'profiled nmethods'"}
> 6619136.0
> zk_jvm_memory_pool_bytes_committed{pool="Compressed Class Space"}
> 3014656.0
> zk_jvm_memory_pool_bytes_committed{pool="G1 Eden Space"} 5.24288E7
> zk_jvm_memory_pool_bytes_committed{pool="G1 Old Gen"} 1.01711872E9
> zk_jvm_memory_pool_bytes_committed{pool="G1 Survivor Space"} 4194304.0
> zk_jvm_memory_pool_bytes_committed{pool="CodeHeap 'non-profiled nmethods'"}
> 2555904.0
> zk_j
[jira] [Commented] (LUCENE-9474) Change Jenkins jobs to use Gradle for trunk
[ https://issues.apache.org/jira/browse/LUCENE-9474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17183804#comment-17183804 ] Uwe Schindler commented on LUCENE-9474: --- I used a static gradle.properties file in Config Store of ASF Jenkins . The check/test builds (including bad apples and nightly) were converted. Many Jobs around Maven and Clover already deleted and the reproducer disabled (no idea if we still need this and how to do this with Gradle; the reproduce with line does not help, maybe after the removal of ant). Still open are the artifact/javadocs builds. > Change Jenkins jobs to use Gradle for trunk > --- > > Key: LUCENE-9474 > URL: https://issues.apache.org/jira/browse/LUCENE-9474 > Project: Lucene - Core > Issue Type: Test > Components: general/build >Reporter: Erick Erickson >Assignee: Uwe Schindler >Priority: Major > > I rushed the gate and pushed LUCENE-9433 without coordinating, my apologies > for the confusion. > Meanwhile, Uwe has disabled Jenkins jobs for the weekend and we'll fix this > up Real Soon Now. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9479) Add forbiddenApis "commons-io-unsafe" bundled signatures to Gradle build
[ https://issues.apache.org/jira/browse/LUCENE-9479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17183800#comment-17183800 ] Uwe Schindler commented on LUCENE-9479: --- Thanks [~dweiss] for review! > Add forbiddenApis "commons-io-unsafe" bundled signatures to Gradle build > > > Key: LUCENE-9479 > URL: https://issues.apache.org/jira/browse/LUCENE-9479 > Project: Lucene - Core > Issue Type: Bug > Components: general/build >Affects Versions: master (9.0) >Reporter: Uwe Schindler >Assignee: Uwe Schindler >Priority: Major > Fix For: master (9.0) > > Time Spent: 20m > Remaining Estimate: 0h > > LUCENE-9077 added forbiddenapis also to Gradle. The signatures files were > selected based on the resolved artifacts. > Solr also uses commons-io (currently version 2.6)but in contrast to Ant > build, the signatures shipped with forbiddenapis are not applied. This was an > oversight as those signatures are special: we have to check the artifacts and > if commons-io is part of configuration, we add the bundled signature. > This can be done at the same place where the dynamic signatures for other > libraries are added. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-9479) Add forbiddenApis "commons-io-unsafe" bundled signatures to Gradle build
[ https://issues.apache.org/jira/browse/LUCENE-9479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uwe Schindler resolved LUCENE-9479. --- Fix Version/s: master (9.0) Resolution: Fixed > Add forbiddenApis "commons-io-unsafe" bundled signatures to Gradle build > > > Key: LUCENE-9479 > URL: https://issues.apache.org/jira/browse/LUCENE-9479 > Project: Lucene - Core > Issue Type: Bug > Components: general/build >Affects Versions: master (9.0) >Reporter: Uwe Schindler >Assignee: Uwe Schindler >Priority: Major > Fix For: master (9.0) > > Time Spent: 20m > Remaining Estimate: 0h > > LUCENE-9077 added forbiddenapis also to Gradle. The signatures files were > selected based on the resolved artifacts. > Solr also uses commons-io (currently version 2.6)but in contrast to Ant > build, the signatures shipped with forbiddenapis are not applied. This was an > oversight as those signatures are special: we have to check the artifacts and > if commons-io is part of configuration, we add the bundled signature. > This can be done at the same place where the dynamic signatures for other > libraries are added. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9479) Add forbiddenApis "commons-io-unsafe" bundled signatures to Gradle build
[ https://issues.apache.org/jira/browse/LUCENE-9479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17183792#comment-17183792 ] ASF subversion and git services commented on LUCENE-9479: - Commit 6abce32a3ecf661817ee9742fc4c69117c7e798b in lucene-solr's branch refs/heads/master from Uwe Schindler [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=6abce32 ] LUCENE-9479: Forbiddenapis & Gradle: Add commons-io bundled signatures (#1780) > Add forbiddenApis "commons-io-unsafe" bundled signatures to Gradle build > > > Key: LUCENE-9479 > URL: https://issues.apache.org/jira/browse/LUCENE-9479 > Project: Lucene - Core > Issue Type: Bug > Components: general/build >Affects Versions: master (9.0) >Reporter: Uwe Schindler >Assignee: Uwe Schindler >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > LUCENE-9077 added forbiddenapis also to Gradle. The signatures files were > selected based on the resolved artifacts. > Solr also uses commons-io (currently version 2.6)but in contrast to Ant > build, the signatures shipped with forbiddenapis are not applied. This was an > oversight as those signatures are special: we have to check the artifacts and > if commons-io is part of configuration, we add the bundled signature. > This can be done at the same place where the dynamic signatures for other > libraries are added. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] uschindler merged pull request #1780: LUCENE-9479: Forbiddenapis & Gradle: Add commons-io bundled signatures
uschindler merged pull request #1780: URL: https://github.com/apache/lucene-solr/pull/1780 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
