[jira] [Resolved] (LUCENE-9412) Repro Jenkins jobs failing to fetch build output
[
https://issues.apache.org/jira/browse/LUCENE-9412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mike Drob resolved LUCENE-9412.
---
Resolution: Won't Fix
> Repro Jenkins jobs failing to fetch build output
>
>
> Key: LUCENE-9412
> URL: https://issues.apache.org/jira/browse/LUCENE-9412
> Project: Lucene - Core
> Issue Type: Bug
> Components: general/build
>Reporter: Mike Drob
>Assignee: Mike Drob
>Priority: Major
>
> We get a python SSL error when trying to fetch the output of a previous job,
> https://builds.apache.org/job/Lucene-Solr-repro-Java11/1163/console
> It's an easy fix to not verify the certificate on the other end, but opening
> this for two reasons:
> 1) maybe there is something INFRA can do w.r.t. fixing either the cert for
> the Jenkins machines or the accepted CAs on the python install
> 2) document it here so that when somebody looks at the code later they'll
> have a reference for why it looks like we're doing something insecure.
> {noformat}
> Traceback (most recent call last):
> File "dev-tools/scripts/reproduceJenkinsFailures.py", line 309, in
> main()
> File "dev-tools/scripts/reproduceJenkinsFailures.py", line 250, in main
> tests = fetchAndParseJenkinsLog(config.url, numRetries = 2)
> File "dev-tools/scripts/reproduceJenkinsFailures.py", line 132, in
> fetchAndParseJenkinsLog
> raise RuntimeError('ERROR: fetching %s : %s' % (url, e))
> RuntimeError: ERROR: fetching
> https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/2228/consoleText
> : (_ssl.c:645)>
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-10457) LuceneTestCase.createTempDir could randomly return symbolic links
[ https://issues.apache.org/jira/browse/LUCENE-10457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Drob resolved LUCENE-10457. Lucene Fields: (was: New) Resolution: Won't Fix > LuceneTestCase.createTempDir could randomly return symbolic links > - > > Key: LUCENE-10457 > URL: https://issues.apache.org/jira/browse/LUCENE-10457 > Project: Lucene - Core > Issue Type: Task > Components: general/test >Reporter: Mike Drob >Priority: Major > > When we are creating temporary directories to use for other Lucene functions, > we could occasionally provide symbolic links instead of direct references to > directories. If the system running tests doesn't support symbolic links, then > we should ignore this option. > Providing links would be useful to test scenarios for example where users > have a symbolic link for the "current" index directory and then rotate that > over time but applications still use the same link. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-10625) addBackcompatIndexes.py on 9x can't handle 8x indices
Mike Drob created LUCENE-10625:
--
Summary: addBackcompatIndexes.py on 9x can't handle 8x indices
Key: LUCENE-10625
URL: https://issues.apache.org/jira/browse/LUCENE-10625
Project: Lucene - Core
Issue Type: Bug
Components: core/other, release wizard
Reporter: Mike Drob
When going through the release wizard for 8x, steps that involve running
{{python3 -u dev-tools/scripts/addBackcompatIndexes.py}} will fail because the
python script will try to run gradle commands inside of the 8x release to
generate the back combat indices.
We could maybe be more robust in detecting that the index already exists in the
temp directory? I'm not actually sure how to fix this, and it might not matter
much as I don't know how many additional 8.11 releases we will have, but
reporting for posterity.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-10564) SparseFixedBitSet#or doesn't update memory accounting
[
https://issues.apache.org/jira/browse/LUCENE-10564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mike Drob updated LUCENE-10564:
---
Fix Version/s: 8.11.2
> SparseFixedBitSet#or doesn't update memory accounting
> -
>
> Key: LUCENE-10564
> URL: https://issues.apache.org/jira/browse/LUCENE-10564
> Project: Lucene - Core
> Issue Type: Bug
>Reporter: Julie Tibshirani
>Priority: Minor
> Fix For: 8.11.2, 9.2
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> While debugging why a cache was using way more memory than expected, one of
> my colleagues noticed that {{SparseFixedBitSet#or}} doesn't update
> {{{}ramBytesUsed{}}}. Here's a unit test that demonstrates this:
> {code:java}
> public void testRamBytesUsed() throws IOException {
> BitSet bitSet = new SparseFixedBitSet(1000);
> long initialBytesUsed = bitSet.ramBytesUsed();
> DocIdSetIterator disi = DocIdSetIterator.all(1000);
> bitSet.or(disi);
> assertTrue(bitSet.ramBytesUsed() > initialBytesUsed);
> }
> {code}
> It also looks like we don't have any tests for {{SparseFixedBitSet}} memory
> accounting (unless I've missed them!) It'd be nice to add more coverage there
> too.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-10481) FacetsCollector does not need scores when not keeping them
[ https://issues.apache.org/jira/browse/LUCENE-10481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17538904#comment-17538904 ] Mike Drob commented on LUCENE-10481: The relevant results are part of https://github.com/filodb/FiloDB/pull/1357 btw. > FacetsCollector does not need scores when not keeping them > -- > > Key: LUCENE-10481 > URL: https://issues.apache.org/jira/browse/LUCENE-10481 > Project: Lucene - Core > Issue Type: Improvement > Components: modules/facet >Reporter: Mike Drob >Assignee: Mike Drob >Priority: Major > Fix For: 8.11.2, 9.2 > > Time Spent: 50m > Remaining Estimate: 0h > > FacetsCollector currently always specifies ScoreMode.COMPLETE, we could get > better performance by not requesting scores when we don't need them. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-10236) CombinedFieldsQuery to use fieldAndWeights.values() when constructing MultiNormsLeafSimScorer for scoring
[
https://issues.apache.org/jira/browse/LUCENE-10236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17538284#comment-17538284
]
Mike Drob commented on LUCENE-10236:
[~zacharymorn] is this still relevant for 8.11?
https://github.com/apache/lucene-solr/pull/2637
> CombinedFieldsQuery to use fieldAndWeights.values() when constructing
> MultiNormsLeafSimScorer for scoring
> -
>
> Key: LUCENE-10236
> URL: https://issues.apache.org/jira/browse/LUCENE-10236
> Project: Lucene - Core
> Issue Type: Improvement
> Components: modules/sandbox
>Reporter: Zach Chen
>Assignee: Zach Chen
>Priority: Minor
> Fix For: 9.1
>
> Time Spent: 6h 50m
> Remaining Estimate: 0h
>
> This is a spin-off issue from discussion in
> [https://github.com/apache/lucene/pull/418#issuecomment-967790816], for a
> quick fix in CombinedFieldsQuery scoring.
> Currently CombinedFieldsQuery would use a constructed
> [fields|https://github.com/apache/lucene/blob/3b914a4d73eea8923f823cbdb869de39213411dd/lucene/sandbox/src/java/org/apache/lucene/sandbox/search/CombinedFieldQuery.java#L420-L421]
> object to create a MultiNormsLeafSimScorer for scoring, but the fields
> object may contain duplicated field-weight pairs as it is [built from looping
> over
> fieldTerms|https://github.com/apache/lucene/blob/3b914a4d73eea8923f823cbdb869de39213411dd/lucene/sandbox/src/java/org/apache/lucene/sandbox/search/CombinedFieldQuery.java#L404-L414],
> resulting into duplicated norms being added during scoring calculation in
> MultiNormsLeafSimScorer.
> E.g. for CombinedFieldsQuery with two fields and two values matching a
> particular doc:
> {code:java}
> CombinedFieldQuery query =
> new CombinedFieldQuery.Builder()
> .addField("field1", (float) 1.0)
> .addField("field2", (float) 1.0)
> .addTerm(new BytesRef("foo"))
> .addTerm(new BytesRef("zoo"))
> .build(); {code}
> I would imagine the scoring to be based on the following:
> # Sum of freqs on doc = freq(field1:foo) + freq(field2:foo) +
> freq(field1:zoo) + freq(field2:zoo)
> # Sum of norms on doc = norm(field1) + norm(field2)
> but the current logic would use the following for scoring:
> # Sum of freqs on doc = freq(field1:foo) + freq(field2:foo) +
> freq(field1:zoo) + freq(field2:zoo)
> # Sum of norms on doc = norm(field1) + norm(field2) + norm(field1) +
> norm(field2)
>
> In addition, this differs from how MultiNormsLeafSimScorer is constructed
> from CombinedFieldsQuery explain function, which [uses
> fieldAndWeights.values()|https://github.com/apache/lucene/blob/3b914a4d73eea8923f823cbdb869de39213411dd/lucene/sandbox/src/java/org/apache/lucene/sandbox/search/CombinedFieldQuery.java#L387-L389]
> and does not contain duplicated field-weight pairs.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-10481) FacetsCollector does not need scores when not keeping them
[ https://issues.apache.org/jira/browse/LUCENE-10481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Drob updated LUCENE-10481: --- Fix Version/s: 8.11.2 > FacetsCollector does not need scores when not keeping them > -- > > Key: LUCENE-10481 > URL: https://issues.apache.org/jira/browse/LUCENE-10481 > Project: Lucene - Core > Issue Type: Improvement > Components: modules/facet >Reporter: Mike Drob >Assignee: Mike Drob >Priority: Major > Fix For: 8.11.2, 9.2 > > Time Spent: 50m > Remaining Estimate: 0h > > FacetsCollector currently always specifies ScoreMode.COMPLETE, we could get > better performance by not requesting scores when we don't need them. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-10481) FacetsCollector does not need scores when not keeping them
[ https://issues.apache.org/jira/browse/LUCENE-10481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17536218#comment-17536218 ] Mike Drob commented on LUCENE-10481: [~jpountz] - what would you think about a back port to 8.11.2? > FacetsCollector does not need scores when not keeping them > -- > > Key: LUCENE-10481 > URL: https://issues.apache.org/jira/browse/LUCENE-10481 > Project: Lucene - Core > Issue Type: Improvement > Components: modules/facet >Reporter: Mike Drob >Assignee: Mike Drob >Priority: Major > Fix For: 9.2 > > Time Spent: 0.5h > Remaining Estimate: 0h > > FacetsCollector currently always specifies ScoreMode.COMPLETE, we could get > better performance by not requesting scores when we don't need them. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-9651) Correct javadoc for benchmarks
[
https://issues.apache.org/jira/browse/LUCENE-9651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mike Drob resolved LUCENE-9651.
---
Fix Version/s: 10.0 (main)
Assignee: Mike Drob (was: Dawid Weiss)
Resolution: Fixed
> Correct javadoc for benchmarks
> ---
>
> Key: LUCENE-9651
> URL: https://issues.apache.org/jira/browse/LUCENE-9651
> Project: Lucene - Core
> Issue Type: Task
>Reporter: Dawid Weiss
>Assignee: Mike Drob
>Priority: Trivial
> Fix For: 10.0 (main)
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> Javadoc still mentions ant
> {code}
> Easiest way to run a benchmarks is using the predefined ant task:
> {code}
> It'd be good to correct it to use gradle (or direct java -jar... calls).
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-10481) FacetsCollector does not need scores when not keeping them
[ https://issues.apache.org/jira/browse/LUCENE-10481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Drob resolved LUCENE-10481. Fix Version/s: 9.2 Assignee: Mike Drob Resolution: Fixed > FacetsCollector does not need scores when not keeping them > -- > > Key: LUCENE-10481 > URL: https://issues.apache.org/jira/browse/LUCENE-10481 > Project: Lucene - Core > Issue Type: Improvement > Components: modules/facet >Reporter: Mike Drob >Assignee: Mike Drob >Priority: Major > Fix For: 9.2 > > Time Spent: 0.5h > Remaining Estimate: 0h > > FacetsCollector currently always specifies ScoreMode.COMPLETE, we could get > better performance by not requesting scores when we don't need them. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-10481) FacetsCollector does not need scores when not keeping them
[ https://issues.apache.org/jira/browse/LUCENE-10481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17510880#comment-17510880 ] Mike Drob commented on LUCENE-10481: Hmm... some slightly disappointing results - although we saw great improvement with this change, that doesn't seem to persist with Lucene 9.1 benchmarking that I'm trying to do right now. Possible that something else has taken care of this optimization in a different way. > FacetsCollector does not need scores when not keeping them > -- > > Key: LUCENE-10481 > URL: https://issues.apache.org/jira/browse/LUCENE-10481 > Project: Lucene - Core > Issue Type: Improvement > Components: modules/facet >Reporter: Mike Drob >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > FacetsCollector currently always specifies ScoreMode.COMPLETE, we could get > better performance by not requesting scores when we don't need them. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-10481) FacetsCollector does not need scores when not keeping them
[ https://issues.apache.org/jira/browse/LUCENE-10481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17510809#comment-17510809 ] Mike Drob commented on LUCENE-10481: I _think_ the use case would be to collect facets over only the top N documents that match a query and if we're doing that search without a TopDocsCollector? I'm not entirely certain what the motivation was. Our use case that is significantly improved by this was a boolean query with several filter clauses where we only want the facet counts and don't care about the documents themselves. > FacetsCollector does not need scores when not keeping them > -- > > Key: LUCENE-10481 > URL: https://issues.apache.org/jira/browse/LUCENE-10481 > Project: Lucene - Core > Issue Type: Improvement > Components: modules/facet >Reporter: Mike Drob >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > FacetsCollector currently always specifies ScoreMode.COMPLETE, we could get > better performance by not requesting scores when we don't need them. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-10481) FacetsCollector does not need scores when not keeping them
Mike Drob created LUCENE-10481: -- Summary: FacetsCollector does not need scores when not keeping them Key: LUCENE-10481 URL: https://issues.apache.org/jira/browse/LUCENE-10481 Project: Lucene - Core Issue Type: Improvement Components: modules/facet Reporter: Mike Drob FacetsCollector currently always specifies ScoreMode.COMPLETE, we could get better performance by not requesting scores when we don't need them. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-10479) Benchmark documentation referes to non-existent tasks
[
https://issues.apache.org/jira/browse/LUCENE-10479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mike Drob resolved LUCENE-10479.
Resolution: Duplicate
> Benchmark documentation referes to non-existent tasks
> -
>
> Key: LUCENE-10479
> URL: https://issues.apache.org/jira/browse/LUCENE-10479
> Project: Lucene - Core
> Issue Type: Task
>Reporter: Mike Drob
>Priority: Minor
>
> The Lucene benchmark package-info file has these instructions on how to run:
> {noformat}
> * To run the short version of the StandardBenchmarker, call "ant
> run-micro-standard". This
> * should take a minute or so to complete and give you a preliminary idea of
> how your change affects
> * the code.
> {noformat}
> Ant doesn't exist for us anymore, so we should replace these instructions
> with the Gradle equivalents. The intuitive replacements {{./gradlew
> run-micro-standard}} or {{runMicroStandard}} didn't work for me, so I'm not
> sure what the new way to run these benchmarks is. Maybe it still needs to be
> implemented? Or this comment deleted.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-10479) Benchmark documentation referes to non-existent tasks
Mike Drob created LUCENE-10479:
--
Summary: Benchmark documentation referes to non-existent tasks
Key: LUCENE-10479
URL: https://issues.apache.org/jira/browse/LUCENE-10479
Project: Lucene - Core
Issue Type: Task
Reporter: Mike Drob
The Lucene benchmark package-info file has these instructions on how to run:
{noformat}
* To run the short version of the StandardBenchmarker, call "ant
run-micro-standard". This
* should take a minute or so to complete and give you a preliminary idea of
how your change affects
* the code.
{noformat}
Ant doesn't exist for us anymore, so we should replace these instructions with
the Gradle equivalents. The intuitive replacements {{./gradlew
run-micro-standard}} or {{runMicroStandard}} didn't work for me, so I'm not
sure what the new way to run these benchmarks is. Maybe it still needs to be
implemented? Or this comment deleted.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-10457) LuceneTestCase.createTempDir could randomly return symbolic links
[
https://issues.apache.org/jira/browse/LUCENE-10457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17503016#comment-17503016
]
Mike Drob commented on LUCENE-10457:
> Sorry, a symbolic link is a whole different thing than a file. It has
> different semantics / platform specifics, etc. A symlink to a directory is
> not a directory.
In what ways? If you pass a symlink to a directory to {{Files.isDirectory}} it
will return true. You have to specifically ask Java for
{{Link_Option.NOFOLLOW}} to get the distinction.
> I'm a hard -1 to randomly doing this from LuceneTestCase.
Like I said earlier, LuceneTestCase isn't the only place that could handle
this. Would you be similarly opposed to a FilterFileSystem that does this?
> It isn't just fixing a test or two, its the ongoing costs of having confusing
> tests. It is literally 3 lines of code for you to do this in your
> symlink-using test.
Maybe some additional context for my motivation would be helpful? We had an
issue with symlinks in Solr; it wasn't a Lucene related problem; we fixed it;
everybody is happy and moved on. I started wondering what other places might be
susceptible to mishandling symlinks. Instead of trying to individually search
for every single place our code that could be affected, I started looking for
common areas that might yield high returns.
Switching LuceneTestCase.createTempDir to sometimes return symlinks to
directories broke exactly three places in Lucene code. Those are easy to
reproduce with three lines of code like you suggest. The concern is that I
wouldn't have found those three places without a larger, more general change
that sprayed symlinks all over the place. And we won't find similar issues in
the future, which is the point of the randomized testing framework in the first
place. Have a robust testing framework and find issues that we don't even know
exist.
For example, right now, running {{IndexSplitter
+}} doesn't work when {{destDir}} is a symlink to a directory. How
did I know? I wasn't looking for IndexSplitter issues. Should it work? I would
imagine yes. Or at the very least, give a better error than it currently does.
The core indexing and querying library never has to worry about symlinks, sure.
But we have plenty of operational tools that interact with the filesystem as
defined by users. And we don't control that.
> LuceneTestCase.createTempDir could randomly return symbolic links
> -
>
> Key: LUCENE-10457
> URL: https://issues.apache.org/jira/browse/LUCENE-10457
> Project: Lucene - Core
> Issue Type: Task
> Components: general/test
>Reporter: Mike Drob
>Priority: Major
>
> When we are creating temporary directories to use for other Lucene functions,
> we could occasionally provide symbolic links instead of direct references to
> directories. If the system running tests doesn't support symbolic links, then
> we should ignore this option.
> Providing links would be useful to test scenarios for example where users
> have a symbolic link for the "current" index directory and then rotate that
> over time but applications still use the same link.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-10457) LuceneTestCase.createTempDir could randomly return symbolic links
[ https://issues.apache.org/jira/browse/LUCENE-10457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17502663#comment-17502663 ] Mike Drob commented on LUCENE-10457: > we should suffer the complexity of fixing all tests to deal with symbolic > links when lucene doesn't itself create any symbolic links did this mean _we should not suffer_ because I'm having trouble understanding otherwise. I went through and the problems were ConnectionCostWriter in Ko/Ja analyzers, and IndexSplitter. The tests were actually all fine, but creating symbolic links in the tests helped uncover these issues. The problems in all three cases were paths derived from user input and could easily manifest when somebody used a symbolic link directory. We could fix these classes to work with output directories (which I have done in my branch already) or we could document that under no circumstances can users provide a symlink as a parameter, but that feels a little silly. I don't particularly care about whether we generate symbolic links from createTempDir or whether we inject that somewhere else. Lucene already has a lot of extension points for tests that do "extra things" like the various file system implementations, various buffers that come pre-loaded with offsets, I think there's code that arbitrarily corrupts some buffers for testing, etc. This doesn't feel too different from any of those. Using createTempDir felt good to me because he implementation should theoretically be black box - as a caller all that you know or have any right to know is that you've been given a directory. Sometimes it might just happen to be a symlink, or maybe it has whitespace, or maybe it is on a different filesystem root if that's available on the system. None of that matters, you got a directory and you should be able to deal with it. > LuceneTestCase.createTempDir could randomly return symbolic links > - > > Key: LUCENE-10457 > URL: https://issues.apache.org/jira/browse/LUCENE-10457 > Project: Lucene - Core > Issue Type: Task > Components: general/test >Reporter: Mike Drob >Priority: Major > > When we are creating temporary directories to use for other Lucene functions, > we could occasionally provide symbolic links instead of direct references to > directories. If the system running tests doesn't support symbolic links, then > we should ignore this option. > Providing links would be useful to test scenarios for example where users > have a symbolic link for the "current" index directory and then rotate that > over time but applications still use the same link. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-10457) LuceneTestCase.createTempDir could randomly return symbolic links
Mike Drob created LUCENE-10457: -- Summary: LuceneTestCase.createTempDir could randomly return symbolic links Key: LUCENE-10457 URL: https://issues.apache.org/jira/browse/LUCENE-10457 Project: Lucene - Core Issue Type: Task Components: general/test Reporter: Mike Drob When we are creating temporary directories to use for other Lucene functions, we could occasionally provide symbolic links instead of direct references to directories. If the system running tests doesn't support symbolic links, then we should ignore this option. Providing links would be useful to test scenarios for example where users have a symbolic link for the "current" index directory and then rotate that over time but applications still use the same link. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-9662) CheckIndex should be concurrent
[
https://issues.apache.org/jira/browse/LUCENE-9662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17485499#comment-17485499
]
Mike Drob edited comment on LUCENE-9662 at 2/1/22, 9:52 PM:
1) Changes here have been committed, should this be marked closed?
2) There are several usages of infoStream in CheckIndex that are not guarding
against null, should we fix those in this issue or a new one?
https://github.com/apache/lucene/pull/637
was (Author: mdrob):
1) Changes here have been committed, should this be marked closed?
2) There are several usages of infoStream in CheckIndex that are not guarding
against null, should we fix those in this issue or a new one?
> CheckIndex should be concurrent
> ---
>
> Key: LUCENE-9662
> URL: https://issues.apache.org/jira/browse/LUCENE-9662
> Project: Lucene - Core
> Issue Type: Bug
>Reporter: Michael McCandless
>Priority: Major
> Time Spent: 20h 50m
> Remaining Estimate: 0h
>
> I am watching a nightly benchmark run slowly run its {{CheckIndex}} step,
> using a single core out of the 128 cores the box has.
> It seems like this is an embarrassingly parallel problem, if the index has
> multiple segments, and would finish much more quickly on concurrent hardware
> if we did "thread per segment".
> If wanted to get even further concurrency, each part of the Lucene index that
> is checked is also independent, so it could be "thread per segment per part".
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9662) CheckIndex should be concurrent
[
https://issues.apache.org/jira/browse/LUCENE-9662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17485499#comment-17485499
]
Mike Drob commented on LUCENE-9662:
---
1) Changes here have been committed, should this be marked closed?
2) There are several usages of infoStream in CheckIndex that are not guarding
against null, should we fix those in this issue or a new one?
> CheckIndex should be concurrent
> ---
>
> Key: LUCENE-9662
> URL: https://issues.apache.org/jira/browse/LUCENE-9662
> Project: Lucene - Core
> Issue Type: Bug
>Reporter: Michael McCandless
>Priority: Major
> Time Spent: 20h 50m
> Remaining Estimate: 0h
>
> I am watching a nightly benchmark run slowly run its {{CheckIndex}} step,
> using a single core out of the 128 cores the box has.
> It seems like this is an embarrassingly parallel problem, if the index has
> multiple segments, and would finish much more quickly on concurrent hardware
> if we did "thread per segment".
> If wanted to get even further concurrency, each part of the Lucene index that
> is checked is also independent, so it could be "thread per segment per part".
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-9650) Errorprone on master/gradle no longer works with JDK-16
[
https://issues.apache.org/jira/browse/LUCENE-9650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17464782#comment-17464782
]
Mike Drob edited comment on LUCENE-9650 at 12/23/21, 8:27 PM:
--
We incidentally opened up some internal java modules for spotless plugin
already in LUCENE-10066, I think it would be reasonable to revisit this and
enable error-prone compiler again.
was (Author: mdrob):
We incidentally opened up some internal java modules for spotless plugin
already in LUCENE-9990, I think it would be reasonable to revisit this and
enable error-prone compiler again.
> Errorprone on master/gradle no longer works with JDK-16
> ---
>
> Key: LUCENE-9650
> URL: https://issues.apache.org/jira/browse/LUCENE-9650
> Project: Lucene - Core
> Issue Type: Bug
> Components: general/build
>Affects Versions: 9.0
>Reporter: Uwe Schindler
>Assignee: Dawid Weiss
>Priority: Major
> Fix For: 9.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> JDK-16 no longer allows access to internal classes of any module by default.
> It looks like errorprone tries to access some internals from the Java
> compiler. This now fails with Exception. You have to fully open the module or
> pass `--illegal-access=allow`.
> We have 3 options:
> - install an update of errorprone
> - disable error-prone if we detect a runtimeJdk with version >=16
> - run javac as a separate forked task (i think we do already) and pass
> `--illegal-access=allow` or open the internal compile module.
> Error message:
> {noformat}
> > Task :solr:solr-ref-guide:compileJava FAILED
> Exception in thread "main" java.lang.IllegalAccessError: class
> com.google.errorprone.ErrorProneJavacPlugin (in unnamed module @0x887af79)
> cannot access class com.sun.tools.javac.api.BasicJavacTask (in module
> jdk.compiler) because module jdk.compiler does not export
> com.sun.tools.javac.api to unnamed module @0x887af79
> at
> com.google.errorprone.ErrorProneJavacPlugin.init(ErrorProneJavacPlugin.java:38)
> at
> jdk.compiler/com.sun.tools.javac.api.BasicJavacTask.initPlugin(BasicJavacTask.java:254)
> at
> jdk.compiler/com.sun.tools.javac.api.BasicJavacTask.initPlugins(BasicJavacTask.java:228)
> at jdk.compiler/com.sun.tools.javac.main.Main.compile(Main.java:292)
> at jdk.compiler/com.sun.tools.javac.main.Main.compile(Main.java:176)
> at jdk.compiler/com.sun.tools.javac.Main.compile(Main.java:64)
> at jdk.compiler/com.sun.tools.javac.Main.main(Main.java:50)
> {noformat}
> Last failed build:
> https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/29129/
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-9650) Errorprone on master/gradle no longer works with JDK-16
[
https://issues.apache.org/jira/browse/LUCENE-9650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17464782#comment-17464782
]
Mike Drob edited comment on LUCENE-9650 at 12/23/21, 8:25 PM:
--
We incidentally opened up some internal java modules for spotless plugin
already in LUCENE-9990, I think it would be reasonable to revisit this and
enable error-prone compiler again.
was (Author: mdrob):
We incidentally opened up some internal java modules for spotless plugin
already in SOLR-15602, I think it would be reasonable to revisit this and
enable error-prone compiler again.
> Errorprone on master/gradle no longer works with JDK-16
> ---
>
> Key: LUCENE-9650
> URL: https://issues.apache.org/jira/browse/LUCENE-9650
> Project: Lucene - Core
> Issue Type: Bug
> Components: general/build
>Affects Versions: 9.0
>Reporter: Uwe Schindler
>Assignee: Dawid Weiss
>Priority: Major
> Fix For: 9.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> JDK-16 no longer allows access to internal classes of any module by default.
> It looks like errorprone tries to access some internals from the Java
> compiler. This now fails with Exception. You have to fully open the module or
> pass `--illegal-access=allow`.
> We have 3 options:
> - install an update of errorprone
> - disable error-prone if we detect a runtimeJdk with version >=16
> - run javac as a separate forked task (i think we do already) and pass
> `--illegal-access=allow` or open the internal compile module.
> Error message:
> {noformat}
> > Task :solr:solr-ref-guide:compileJava FAILED
> Exception in thread "main" java.lang.IllegalAccessError: class
> com.google.errorprone.ErrorProneJavacPlugin (in unnamed module @0x887af79)
> cannot access class com.sun.tools.javac.api.BasicJavacTask (in module
> jdk.compiler) because module jdk.compiler does not export
> com.sun.tools.javac.api to unnamed module @0x887af79
> at
> com.google.errorprone.ErrorProneJavacPlugin.init(ErrorProneJavacPlugin.java:38)
> at
> jdk.compiler/com.sun.tools.javac.api.BasicJavacTask.initPlugin(BasicJavacTask.java:254)
> at
> jdk.compiler/com.sun.tools.javac.api.BasicJavacTask.initPlugins(BasicJavacTask.java:228)
> at jdk.compiler/com.sun.tools.javac.main.Main.compile(Main.java:292)
> at jdk.compiler/com.sun.tools.javac.main.Main.compile(Main.java:176)
> at jdk.compiler/com.sun.tools.javac.Main.compile(Main.java:64)
> at jdk.compiler/com.sun.tools.javac.Main.main(Main.java:50)
> {noformat}
> Last failed build:
> https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/29129/
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9650) Errorprone on master/gradle no longer works with JDK-16
[
https://issues.apache.org/jira/browse/LUCENE-9650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17464782#comment-17464782
]
Mike Drob commented on LUCENE-9650:
---
We incidentally opened up some internal java modules for spotless plugin
already in SOLR-15602, I think it would be reasonable to revisit this and
enable error-prone compiler again.
> Errorprone on master/gradle no longer works with JDK-16
> ---
>
> Key: LUCENE-9650
> URL: https://issues.apache.org/jira/browse/LUCENE-9650
> Project: Lucene - Core
> Issue Type: Bug
> Components: general/build
>Affects Versions: 9.0
>Reporter: Uwe Schindler
>Assignee: Dawid Weiss
>Priority: Major
> Fix For: 9.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> JDK-16 no longer allows access to internal classes of any module by default.
> It looks like errorprone tries to access some internals from the Java
> compiler. This now fails with Exception. You have to fully open the module or
> pass `--illegal-access=allow`.
> We have 3 options:
> - install an update of errorprone
> - disable error-prone if we detect a runtimeJdk with version >=16
> - run javac as a separate forked task (i think we do already) and pass
> `--illegal-access=allow` or open the internal compile module.
> Error message:
> {noformat}
> > Task :solr:solr-ref-guide:compileJava FAILED
> Exception in thread "main" java.lang.IllegalAccessError: class
> com.google.errorprone.ErrorProneJavacPlugin (in unnamed module @0x887af79)
> cannot access class com.sun.tools.javac.api.BasicJavacTask (in module
> jdk.compiler) because module jdk.compiler does not export
> com.sun.tools.javac.api to unnamed module @0x887af79
> at
> com.google.errorprone.ErrorProneJavacPlugin.init(ErrorProneJavacPlugin.java:38)
> at
> jdk.compiler/com.sun.tools.javac.api.BasicJavacTask.initPlugin(BasicJavacTask.java:254)
> at
> jdk.compiler/com.sun.tools.javac.api.BasicJavacTask.initPlugins(BasicJavacTask.java:228)
> at jdk.compiler/com.sun.tools.javac.main.Main.compile(Main.java:292)
> at jdk.compiler/com.sun.tools.javac.main.Main.compile(Main.java:176)
> at jdk.compiler/com.sun.tools.javac.Main.compile(Main.java:64)
> at jdk.compiler/com.sun.tools.javac.Main.main(Main.java:50)
> {noformat}
> Last failed build:
> https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/29129/
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-10227) FilterPath.resolve(Path) doesn't work
[
https://issues.apache.org/jira/browse/LUCENE-10227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17442938#comment-17442938
]
Mike Drob commented on LUCENE-10227:
Was thinking about this more, and I'm not convinced that FilterPath benefits
from being so strict about what it accepts in {{resolve}}.
Consider the case where I'm copying some files from one file system to another.
Java doesn't have a recursive directory copy, but
[Files.copy|https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/nio/file/Files.html#copy(java.nio.file.Path,java.nio.file.Path,java.nio.file.CopyOption...)]
suggests using {{walkFileTree}} to handle that, and the
[examples|https://docs.oracle.com/javase/tutorial/displayCode.html?code=https://docs.oracle.com/javase/tutorial/essential/io/examples/Copy.java]
linked from the [official
tutorial|https://docs.oracle.com/javase/tutorial/essential/io/copy.html] call
{{target.resolve(source.relativize(file))}}. Does this work across file
systems? I haven't been able to test yet, but it would suggest to me that they
should. This situation will come up even without the bad use of {{Path.of}}.
> FilterPath.resolve(Path) doesn't work
> -
>
> Key: LUCENE-10227
> URL: https://issues.apache.org/jira/browse/LUCENE-10227
> Project: Lucene - Core
> Issue Type: Bug
>Reporter: Mike Drob
>Priority: Major
>
> Short reproducing sample that captures the error I encounter when trying to
> write a unit test:
> {code:java}
> package org.apache.lucene.mockfile;
> import org.apache.lucene.util.LuceneTestCase;
> import org.junit.Test;
> import java.nio.file.Path;
> public class TestFilterPath extends LuceneTestCase {
> @Test
> public void testMultiwrap() throws Exception {
> Path source = LuceneTestCase.createTempDir();
> Path child = source.resolve("child");
> assertEquals(source, child.getParent());
> child = source.resolve(Path.of("child")); // This line throws exception
> assertEquals(source, child.getParent());
> }
> }
> {code}
> When running this test (on my Mac) I get the following:
> {noformat}
> mismatch, expected: FilterPath, got: class sun.nio.fs.UnixPath
> java.nio.file.ProviderMismatchException: mismatch, expected: FilterPath, got:
> class sun.nio.fs.UnixPath
> at
> __randomizedtesting.SeedInfo.seed([318296D60FE95435:125A6A9E9053DCC3]:0)
> at org.apache.lucene.mockfile.FilterPath.toDelegate(FilterPath.java:292)
> at org.apache.lucene.mockfile.FilterPath.resolve(FilterPath.java:151)
> at
> org.apache.lucene.mockfile.TestFilterPath.testMultiwrap(TestFilterPath.java:16)
> {noformat}
> Part of the problem here might be that {{createTempDir()}} gives me a
> {{FilterPath}} wrapped four layers deep.
> Part of the problem might be that resolve calls toDelegate which is too
> strict about what it would accept?
> I'm not sure what the intent of all of these wrappings is, so I'm not
> completely sure how to fix it.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-10227) FilterPath.resolve(Path) doesn't work
[
https://issues.apache.org/jira/browse/LUCENE-10227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17442315#comment-17442315
]
Mike Drob commented on LUCENE-10227:
Switching the test to {{child =
source.resolve(source.getFileSystem().getPath("child"))}} did the trick. Thank
you, gents. I think it would be good to improve the exception message to
something like:
{noformat}
mismatch, expected: FilterPath, got: class sun.nio.fs.UnixPath
Did you use Path.of with the default FileSystem instead of using an existing
path as an anchor?
at
__randomizedtesting.SeedInfo.seed([318296D60FE95435:125A6A9E9053DCC3]:0)
at org.apache.lucene.mockfile.FilterPath.toDelegate(FilterPath.java:292)
{noformat}
I had read the javadoc for Path.of previously but failed to comprehend the full
implications of this subtlety. Happy to take input on better wording of the
message. I worry that adding JavaDoc to FilterPath would get lost since I
expect that most of the time people don't even know they're dealing with a
FilterPath.
> FilterPath.resolve(Path) doesn't work
> -
>
> Key: LUCENE-10227
> URL: https://issues.apache.org/jira/browse/LUCENE-10227
> Project: Lucene - Core
> Issue Type: Bug
>Reporter: Mike Drob
>Priority: Major
>
> Short reproducing sample that captures the error I encounter when trying to
> write a unit test:
> {code:java}
> package org.apache.lucene.mockfile;
> import org.apache.lucene.util.LuceneTestCase;
> import org.junit.Test;
> import java.nio.file.Path;
> public class TestFilterPath extends LuceneTestCase {
> @Test
> public void testMultiwrap() throws Exception {
> Path source = LuceneTestCase.createTempDir();
> Path child = source.resolve("child");
> assertEquals(source, child.getParent());
> child = source.resolve(Path.of("child")); // This line throws exception
> assertEquals(source, child.getParent());
> }
> }
> {code}
> When running this test (on my Mac) I get the following:
> {noformat}
> mismatch, expected: FilterPath, got: class sun.nio.fs.UnixPath
> java.nio.file.ProviderMismatchException: mismatch, expected: FilterPath, got:
> class sun.nio.fs.UnixPath
> at
> __randomizedtesting.SeedInfo.seed([318296D60FE95435:125A6A9E9053DCC3]:0)
> at org.apache.lucene.mockfile.FilterPath.toDelegate(FilterPath.java:292)
> at org.apache.lucene.mockfile.FilterPath.resolve(FilterPath.java:151)
> at
> org.apache.lucene.mockfile.TestFilterPath.testMultiwrap(TestFilterPath.java:16)
> {noformat}
> Part of the problem here might be that {{createTempDir()}} gives me a
> {{FilterPath}} wrapped four layers deep.
> Part of the problem might be that resolve calls toDelegate which is too
> strict about what it would accept?
> I'm not sure what the intent of all of these wrappings is, so I'm not
> completely sure how to fix it.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-10227) FilterPath.resolve(Path) doesn't work
[
https://issues.apache.org/jira/browse/LUCENE-10227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17441351#comment-17441351
]
Mike Drob commented on LUCENE-10227:
So all the wrapping is because the filter file system has a bunch of wrapping
to generate the paths -
{{ExtrasFS(HandleLimitFS(LeakFS(ShuffleFS(DisableFsyncFS(sun.nio.fs.MacOSXFileSystemProvider@62e280ab)}}
That still leaves me unsure of what I should be changing to fix this bug. I'd
like to use the {{resolve(Path)}} construct because in my code I'm actually
calling it with a path with multiple elements, so that seems cleaner. Should I
be figuring out how to call the String version instead?
> FilterPath.resolve(Path) doesn't work
> -
>
> Key: LUCENE-10227
> URL: https://issues.apache.org/jira/browse/LUCENE-10227
> Project: Lucene - Core
> Issue Type: Bug
>Reporter: Mike Drob
>Priority: Major
>
> Short reproducing sample that captures the error I encounter when trying to
> write a unit test:
> {code:java}
> package org.apache.lucene.mockfile;
> import org.apache.lucene.util.LuceneTestCase;
> import org.junit.Test;
> import java.nio.file.Path;
> public class TestFilterPath extends LuceneTestCase {
> @Test
> public void testMultiwrap() throws Exception {
> Path source = LuceneTestCase.createTempDir();
> Path child = source.resolve("child");
> assertEquals(source, child.getParent());
> child = source.resolve(Path.of("child")); // This line throws exception
> assertEquals(source, child.getParent());
> }
> }
> {code}
> When running this test (on my Mac) I get the following:
> {noformat}
> mismatch, expected: FilterPath, got: class sun.nio.fs.UnixPath
> java.nio.file.ProviderMismatchException: mismatch, expected: FilterPath, got:
> class sun.nio.fs.UnixPath
> at
> __randomizedtesting.SeedInfo.seed([318296D60FE95435:125A6A9E9053DCC3]:0)
> at org.apache.lucene.mockfile.FilterPath.toDelegate(FilterPath.java:292)
> at org.apache.lucene.mockfile.FilterPath.resolve(FilterPath.java:151)
> at
> org.apache.lucene.mockfile.TestFilterPath.testMultiwrap(TestFilterPath.java:16)
> {noformat}
> Part of the problem here might be that {{createTempDir()}} gives me a
> {{FilterPath}} wrapped four layers deep.
> Part of the problem might be that resolve calls toDelegate which is too
> strict about what it would accept?
> I'm not sure what the intent of all of these wrappings is, so I'm not
> completely sure how to fix it.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-10227) FilterPath.resolve(Path) doesn't work
Mike Drob created LUCENE-10227:
--
Summary: FilterPath.resolve(Path) doesn't work
Key: LUCENE-10227
URL: https://issues.apache.org/jira/browse/LUCENE-10227
Project: Lucene - Core
Issue Type: Bug
Reporter: Mike Drob
Short reproducing sample that captures the error I encounter when trying to
write a unit test:
{code:java}
package org.apache.lucene.mockfile;
import org.apache.lucene.util.LuceneTestCase;
import org.junit.Test;
import java.nio.file.Path;
public class TestFilterPath extends LuceneTestCase {
@Test
public void testMultiwrap() throws Exception {
Path source = LuceneTestCase.createTempDir();
Path child = source.resolve("child");
assertEquals(source, child.getParent());
child = source.resolve(Path.of("child")); // This line throws exception
assertEquals(source, child.getParent());
}
}
{code}
When running this test (on my Mac) I get the following:
{noformat}
mismatch, expected: FilterPath, got: class sun.nio.fs.UnixPath
java.nio.file.ProviderMismatchException: mismatch, expected: FilterPath, got:
class sun.nio.fs.UnixPath
at
__randomizedtesting.SeedInfo.seed([318296D60FE95435:125A6A9E9053DCC3]:0)
at org.apache.lucene.mockfile.FilterPath.toDelegate(FilterPath.java:292)
at org.apache.lucene.mockfile.FilterPath.resolve(FilterPath.java:151)
at
org.apache.lucene.mockfile.TestFilterPath.testMultiwrap(TestFilterPath.java:16)
{noformat}
Part of the problem here might be that {{createTempDir()}} gives me a
{{FilterPath}} wrapped four layers deep.
Part of the problem might be that resolve calls toDelegate which is too strict
about what it would accept?
I'm not sure what the intent of all of these wrappings is, so I'm not
completely sure how to fix it.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-10179) [release wizard] No longer check for release status on mirrors
[ https://issues.apache.org/jira/browse/LUCENE-10179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17429476#comment-17429476 ] Mike Drob commented on LUCENE-10179: FYI [~mayya] since you are in the middle of an 8.10.1 release, when you get to the "check mirrors" step, it will likely fail and you should perform the manual steps of checking for the release artifacts on the CDN and in maven central. > [release wizard] No longer check for release status on mirrors > -- > > Key: LUCENE-10179 > URL: https://issues.apache.org/jira/browse/LUCENE-10179 > Project: Lucene - Core > Issue Type: Task >Reporter: Mike Drob >Assignee: Jan Høydahl >Priority: Major > Fix For: main (9.0) > > > The ASF has moved to a CDN instead of the mirror system. We don't need to > update our upload steps, but checking mirrors is no longer necessary I > believe. > > https://blogs.apache.org/foundation/entry/apache-software-foundation-moves-to -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-10179) [release wizard] No longer check for release status on mirrors
[ https://issues.apache.org/jira/browse/LUCENE-10179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17429456#comment-17429456 ] Mike Drob commented on LUCENE-10179: We should back port this to 8x as well > [release wizard] No longer check for release status on mirrors > -- > > Key: LUCENE-10179 > URL: https://issues.apache.org/jira/browse/LUCENE-10179 > Project: Lucene - Core > Issue Type: Task >Reporter: Mike Drob >Assignee: Jan Høydahl >Priority: Major > Fix For: main (9.0) > > > The ASF has moved to a CDN instead of the mirror system. We don't need to > update our upload steps, but checking mirrors is no longer necessary I > believe. > > https://blogs.apache.org/foundation/entry/apache-software-foundation-moves-to -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-10179) [release wizard] No longer check for release status on mirrors
[
https://issues.apache.org/jira/browse/LUCENE-10179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17429337#comment-17429337
]
Mike Drob commented on LUCENE-10179:
{quote}We should probably also update the website Download page to use
dlcdn.apache.org directly instead of closer.lua? There are redirects, so not
urgent...
{quote}
I asked infra about this, they would prefer that we continue to use closer.lua
just in case the CDN is ever temporarily down. I don't know what specifically
will happen, but it sounds like they have thought about it and have a plan.
However, there's a redirect feature we can use and it will sort itself out
whether it needs to go to dlcdn, downloads, or archive.
[https://www.apache.org/dyn/closer.lua?action=download&filename=/lucene/java/8.10.0/lucene-8.10.0-src.tgz]
{quote}Do you know if this changes anything related to how we trust downloads
from CDN vs archive? I.e. could we now retrieve .asc and KEYS files from CDN?
We could not trust those from the mirrors.. I suppose the CDN infrastructure is
potentially vulnerable to attacks, so the same precaution is valid?
{quote}
Policy hasn't been updated yet, but word from INFRA is that it should be safe
for keys/sigs.
> [release wizard] No longer check for release status on mirrors
> --
>
> Key: LUCENE-10179
> URL: https://issues.apache.org/jira/browse/LUCENE-10179
> Project: Lucene - Core
> Issue Type: Task
>Reporter: Mike Drob
>Assignee: Jan Høydahl
>Priority: Major
>
> The ASF has moved to a CDN instead of the mirror system. We don't need to
> update our upload steps, but checking mirrors is no longer necessary I
> believe.
>
> https://blogs.apache.org/foundation/entry/apache-software-foundation-moves-to
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-10179) [release wizard] No longer check for release status on mirrors
[ https://issues.apache.org/jira/browse/LUCENE-10179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17429027#comment-17429027 ] Mike Drob commented on LUCENE-10179: cc: [~janhoy] > [release wizard] No longer check for release status on mirrors > -- > > Key: LUCENE-10179 > URL: https://issues.apache.org/jira/browse/LUCENE-10179 > Project: Lucene - Core > Issue Type: Task >Reporter: Mike Drob >Priority: Major > > The ASF has moved to a CDN instead of the mirror system. We don't need to > update our upload steps, but checking mirrors is no longer necessary I > believe. > > https://blogs.apache.org/foundation/entry/apache-software-foundation-moves-to -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-10179) [release wizard] No longer check for release status on mirrors
Mike Drob created LUCENE-10179: -- Summary: [release wizard] No longer check for release status on mirrors Key: LUCENE-10179 URL: https://issues.apache.org/jira/browse/LUCENE-10179 Project: Lucene - Core Issue Type: Task Reporter: Mike Drob The ASF has moved to a CDN instead of the mirror system. We don't need to update our upload steps, but checking mirrors is no longer necessary I believe. https://blogs.apache.org/foundation/entry/apache-software-foundation-moves-to -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9488) Update release process to work with Gradle.
[
https://issues.apache.org/jira/browse/LUCENE-9488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17424282#comment-17424282
]
Mike Drob commented on LUCENE-9488:
---
bq. The gradle build needs to assemble source package as well as zip and tgz.
Why can’t we have release wizard do {{git archive}}?
> Update release process to work with Gradle.
> ---
>
> Key: LUCENE-9488
> URL: https://issues.apache.org/jira/browse/LUCENE-9488
> Project: Lucene - Core
> Issue Type: Improvement
> Components: general/build
>Reporter: Erick Erickson
>Assignee: Mike Drob
>Priority: Major
> Time Spent: 14h 50m
> Remaining Estimate: 0h
>
> The release process needs to reflect using Gradle rather than Ant. I suspect
> this will be a significant task, thus it has its own JIRA
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9488) Update release process to work with Gradle.
[ https://issues.apache.org/jira/browse/LUCENE-9488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17422990#comment-17422990 ] Mike Drob commented on LUCENE-9488: --- I think [~hossman] did some work in the gpg signing, I’m not up to date on the latest unfortunately. > Update release process to work with Gradle. > --- > > Key: LUCENE-9488 > URL: https://issues.apache.org/jira/browse/LUCENE-9488 > Project: Lucene - Core > Issue Type: Improvement > Components: general/build >Reporter: Erick Erickson >Assignee: Mike Drob >Priority: Major > Time Spent: 14h 50m > Remaining Estimate: 0h > > The release process needs to reflect using Gradle rather than Ant. I suspect > this will be a significant task, thus it has its own JIRA -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-10107) Jar manifest Implementation Version is too long
[ https://issues.apache.org/jira/browse/LUCENE-10107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Drob resolved LUCENE-10107. Assignee: Mike Drob Resolution: Fixed Pushed to 8.10, can revisit if more iteration is needed. > Jar manifest Implementation Version is too long > --- > > Key: LUCENE-10107 > URL: https://issues.apache.org/jira/browse/LUCENE-10107 > Project: Lucene - Core > Issue Type: Task > Components: general/build >Affects Versions: 8.10 >Reporter: Mike Drob >Assignee: Mike Drob >Priority: Blocker > Fix For: 8.10 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > When we added an extra digit for 8.10 that caused the git commit she in the > manifest Implementation-Version to wrap, and now the smoke check fails > because of this. This issue is only for solving this in the 8.x branches > because those use ANT. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-9647) Add back ant precommit on PR for branch 8
[ https://issues.apache.org/jira/browse/LUCENE-9647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Drob resolved LUCENE-9647. --- Resolution: Duplicate This was fixed somewhere along the way. > Add back ant precommit on PR for branch 8 > - > > Key: LUCENE-9647 > URL: https://issues.apache.org/jira/browse/LUCENE-9647 > Project: Lucene - Core > Issue Type: Task > Components: general/build >Reporter: Mike Drob >Assignee: Mike Drob >Priority: Minor > Fix For: main (9.0) > > Time Spent: 1h 20m > Remaining Estimate: 0h > > When migrating everything to gradle only, we accidentally deleted our > branch_8x PR precommit action. > The file needs to be on master, but the branch specification should specify > branch_8x I believe. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-10108) Jar manifest Implementation Version will be too long
[ https://issues.apache.org/jira/browse/LUCENE-10108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Drob updated LUCENE-10108: --- Affects Version/s: (was: 8.10) Description: [Cloned from LUCENE-10107] In the future when we want to release 9.10 or 10.0 or 10.10, we will have too many digits and cause the git commit sha in the manifest Implementation-Version to wrap. This should be easy enough by replacing references to gitRev with gitRevShort was:When we added an extra digit for 8.10 that caused the git commit she in the manifest Implementation-Version to wrap, and now the smoke check fails because of this. This issue is only for solving this in the 8.x branches because those use ANT. Priority: Major (was: Blocker) > Jar manifest Implementation Version will be too long > > > Key: LUCENE-10108 > URL: https://issues.apache.org/jira/browse/LUCENE-10108 > Project: Lucene - Core > Issue Type: Task > Components: general/build >Reporter: Mike Drob >Priority: Major > > [Cloned from LUCENE-10107] > In the future when we want to release 9.10 or 10.0 or 10.10, we will have too > many digits and cause the git commit sha in the manifest > Implementation-Version to wrap. This should be easy enough by replacing > references to gitRev with gitRevShort -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-10107) Jar manifest Implementation Version is too long
[ https://issues.apache.org/jira/browse/LUCENE-10107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Drob updated LUCENE-10107: --- Fix Version/s: 8.10 > Jar manifest Implementation Version is too long > --- > > Key: LUCENE-10107 > URL: https://issues.apache.org/jira/browse/LUCENE-10107 > Project: Lucene - Core > Issue Type: Task > Components: general/build >Affects Versions: 8.10 >Reporter: Mike Drob >Priority: Blocker > Fix For: 8.10 > > > When we added an extra digit for 8.10 that caused the git commit she in the > manifest Implementation-Version to wrap, and now the smoke check fails > because of this. This issue is only for solving this in the 8.x branches > because those use ANT. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-10108) Jar manifest Implementation Version will be too long
Mike Drob created LUCENE-10108: -- Summary: Jar manifest Implementation Version will be too long Key: LUCENE-10108 URL: https://issues.apache.org/jira/browse/LUCENE-10108 Project: Lucene - Core Issue Type: Task Components: general/build Affects Versions: 8.10 Reporter: Mike Drob When we added an extra digit for 8.10 that caused the git commit she in the manifest Implementation-Version to wrap, and now the smoke check fails because of this. This issue is only for solving this in the 8.x branches because those use ANT. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-10107) Jar manifest Implementation Version is too long
Mike Drob created LUCENE-10107: -- Summary: Jar manifest Implementation Version is too long Key: LUCENE-10107 URL: https://issues.apache.org/jira/browse/LUCENE-10107 Project: Lucene - Core Issue Type: Task Components: general/build Affects Versions: 8.10 Reporter: Mike Drob When we added an extra digit for 8.10 that caused the git commit she in the manifest Implementation-Version to wrap, and now the smoke check fails because of this. This issue is only for solving this in the 8.x branches because those use ANT. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-10017) Do not check footers on Lucene7x segments when throwing IndexFormatTooOld
[ https://issues.apache.org/jira/browse/LUCENE-10017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Drob resolved LUCENE-10017. Fix Version/s: main (9.0) Assignee: Mike Drob Resolution: Fixed > Do not check footers on Lucene7x segments when throwing IndexFormatTooOld > - > > Key: LUCENE-10017 > URL: https://issues.apache.org/jira/browse/LUCENE-10017 > Project: Lucene - Core > Issue Type: Improvement > Components: core/index >Affects Versions: main (9.0) >Reporter: Mike Drob >Assignee: Mike Drob >Priority: Major > Fix For: main (9.0) > > Time Spent: 0.5h > Remaining Estimate: 0h > > Ran into this while investigating how to improve the logging situation in > SOLR-15516 > When we throw an IndexFormatTooOldException when attempting to read a > segment, we will read still attempt to read the footer since it is newer than > 7x (which AFAIUI was the oldest version to support these footers?) > We should update that to only check footers for versions newer than 8x, > otherwise we create a needless additional exception. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8638) Remove deprecated code in main
[ https://issues.apache.org/jira/browse/LUCENE-8638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17401029#comment-17401029 ] Mike Drob commented on LUCENE-8638: --- The assertions are because LuceneTestCase extends Assert and uses the inherited method a bunch instead of the migrated method from Hamcrest. That’s probably a good thing to clean up but nobody has done it because I don’t think it has real impact in the code, just a readability issue that we’re all used to by now. > Remove deprecated code in main > -- > > Key: LUCENE-8638 > URL: https://issues.apache.org/jira/browse/LUCENE-8638 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Alan Woodward >Priority: Blocker > Fix For: main (9.0) > > Time Spent: 3h > Remaining Estimate: 0h > > There are a number of deprecations in master that should be removed. This > issue is to keep track of deprecations as a whole, some individual > deprecations may require their own issues. > > Work on this issue should be pushed to the `master-deprecations` branch on > gitbox -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8682) Remove WordDelimiterFilter
[ https://issues.apache.org/jira/browse/LUCENE-8682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17376750#comment-17376750 ] Mike Drob commented on LUCENE-8682: --- There was a [recent conversation|https://lists.apache.org/thread.html/r64788fc974ac5f5763c184ed8d4ea16748a8adaf731952921571ef0d%40%3Cusers.solr.apache.org%3E] on the solr user list about this topic, which suggests to me that a naive find-and-replace in our code will not be sufficient. I think we need to pay attention to the replacement configurations because those are most likely to be copied by users when trying to get their own code to work. Possibly need to update docs? But I suppose we don't want the old WDF in our docs at all at this point. Tagging [~mgibney] for additional perspective as well, if available. > Remove WordDelimiterFilter > -- > > Key: LUCENE-8682 > URL: https://issues.apache.org/jira/browse/LUCENE-8682 > Project: Lucene - Core > Issue Type: Task >Affects Versions: main (9.0) >Reporter: Alan Woodward >Assignee: Alan Woodward >Priority: Major > Attachments: LUCENE-8682.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > > WordDelimiterFilter was deprecated a while back. We can remove it entirely > from the master branch. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-10017) Do not check footers on Lucene7x segments when throwing IndexFormatTooOld
[ https://issues.apache.org/jira/browse/LUCENE-10017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17373102#comment-17373102 ] Mike Drob commented on LUCENE-10017: It looks like the actual index versions in the footer don't let us differentiate between arbitrary 7x and an 8x, there's a [big jump|https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/index/SegmentInfos.java#L123] from 7.4 to 8.6 in the file metadata. One option is to just explicitly check for IFTOE before adding a suppressed corrupted index exception, which I'll put up as a PR. Maybe there's a better heuristic to use instead. > Do not check footers on Lucene7x segments when throwing IndexFormatTooOld > - > > Key: LUCENE-10017 > URL: https://issues.apache.org/jira/browse/LUCENE-10017 > Project: Lucene - Core > Issue Type: Improvement > Components: core/index >Affects Versions: main (9.0) >Reporter: Mike Drob >Priority: Major > > Ran into this while investigating how to improve the logging situation in > SOLR-15516 > When we throw an IndexFormatTooOldException when attempting to read a > segment, we will read still attempt to read the footer since it is newer than > 7x (which AFAIUI was the oldest version to support these footers?) > We should update that to only check footers for versions newer than 8x, > otherwise we create a needless additional exception. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-10017) Do not check footers on Lucene7x segments when throwing IndexFormatTooOld
[ https://issues.apache.org/jira/browse/LUCENE-10017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17373075#comment-17373075 ] Mike Drob commented on LUCENE-10017: I'm not sure if this is "blocker" level but I really would want this considered before we make a 9.0 release. > Do not check footers on Lucene7x segments when throwing IndexFormatTooOld > - > > Key: LUCENE-10017 > URL: https://issues.apache.org/jira/browse/LUCENE-10017 > Project: Lucene - Core > Issue Type: Improvement > Components: core/index >Affects Versions: main (9.0) >Reporter: Mike Drob >Priority: Major > > Ran into this while investigating how to improve the logging situation in > SOLR-15516 > When we throw an IndexFormatTooOldException when attempting to read a > segment, we will read still attempt to read the footer since it is newer than > 7x (which AFAIUI was the oldest version to support these footers?) > We should update that to only check footers for versions newer than 8x, > otherwise we create a needless additional exception. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-10017) Do not check footers on Lucene7x segments when throwing IndexFormatTooOld
Mike Drob created LUCENE-10017: -- Summary: Do not check footers on Lucene7x segments when throwing IndexFormatTooOld Key: LUCENE-10017 URL: https://issues.apache.org/jira/browse/LUCENE-10017 Project: Lucene - Core Issue Type: Improvement Components: core/index Affects Versions: main (9.0) Reporter: Mike Drob Ran into this while investigating how to improve the logging situation in SOLR-15516 When we throw an IndexFormatTooOldException when attempting to read a segment, we will read still attempt to read the footer since it is newer than 7x (which AFAIUI was the oldest version to support these footers?) We should update that to only check footers for versions newer than 8x, otherwise we create a needless additional exception. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8638) Remove deprecated code in master
[ https://issues.apache.org/jira/browse/LUCENE-8638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17372213#comment-17372213 ] Mike Drob commented on LUCENE-8638: --- What would be helpful here to move forward? Do you need reviewers? People to pick up the remaining deprecations in Alan's first comment? Merges from main into the working branch? Rebase from lucene-solr/master to lucene/main? Point me in a direction and I'm happy to roll up my sleeves and get dirty here. > Remove deprecated code in master > > > Key: LUCENE-8638 > URL: https://issues.apache.org/jira/browse/LUCENE-8638 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Alan Woodward >Priority: Blocker > Fix For: main (9.0) > > > There are a number of deprecations in master that should be removed. This > issue is to keep track of deprecations as a whole, some individual > deprecations may require their own issues. > > Work on this issue should be pushed to the `master-deprecations` branch on > gitbox -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-9905) Revise approach to specifying NN algorithm
[
https://issues.apache.org/jira/browse/LUCENE-9905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mike Drob updated LUCENE-9905:
--
Fix Version/s: main (9.0)
> Revise approach to specifying NN algorithm
> --
>
> Key: LUCENE-9905
> URL: https://issues.apache.org/jira/browse/LUCENE-9905
> Project: Lucene - Core
> Issue Type: Improvement
>Affects Versions: main (9.0)
>Reporter: Julie Tibshirani
>Priority: Blocker
> Fix For: main (9.0)
>
> Time Spent: 5h 50m
> Remaining Estimate: 0h
>
> In LUCENE-9322 we decided that the new vectors API shouldn’t assume a
> particular nearest-neighbor search data structure and algorithm. This
> flexibility is important since NN search is a developing area and we'd like
> to be able to experiment and evolve the algorithm. Right now we only have one
> algorithm (HNSW), but we want to maintain the ability to use another.
> Currently the algorithm to use is specified through {{SearchStrategy}}, for
> example {{SearchStrategy.EUCLIDEAN_HNSW}}. So a single format implementation
> is expected to handle multiple algorithms. Instead we could have one format
> implementation per algorithm. Our current implementation would be
> HNSW-specific like {{HnswVectorFormat}}, and to experiment with another
> algorithm you could create a new implementation like {{ClusterVectorFormat}}.
> This would be better aligned with the codec framework, and help avoid
> exposing algorithm details in the API.
> A concrete proposal (note many of these names will change when LUCENE-9855 is
> addressed):
> # Rename {{Lucene90VectorFormat}} to {{Lucene90HnswVectorFormat}}. Also add
> HNSW to name of {{Lucene90VectorWriter}} and {{Lucene90VectorReader}}.
> # Remove references to HNSW in {{SearchStrategy}}, so there is just
> {{SearchStrategy.EUCLIDEAN}}, etc. Rename {{SearchStrategy}} to something
> like {{SimilarityFunction}}.
> # Remove {{FieldType}} attributes related to HNSW parameters (maxConn and
> beamWidth). Instead make these arguments to {{Lucene90HnswVectorFormat}}.
> # Introduce {{PerFieldVectorFormat}} to allow a different NN approach or
> parameters to be configured per-field \(?\)
> One note: the current HNSW-based format includes logic for storing a numeric
> vector per document, as well as constructing + storing a HNSW graph. When
> adding another implementation, it’d be nice to be able to reuse logic for
> reading/ writing numeric vectors. I don’t think we need to design for this
> right now, but we can keep it in mind for the future?
> This issue is based on a thread [~jpountz] started:
> [https://mail-archives.apache.org/mod_mbox/lucene-dev/202103.mbox/%3CCAPsWd%2BOuQv5y2Vw39%3DXdOuqXGtDbM4qXx5-pmYiB1X4jPEdiFQ%40mail.gmail.com%3E]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-9908) Move VectorValues#search to VectorReader and LeafReader
[
https://issues.apache.org/jira/browse/LUCENE-9908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mike Drob updated LUCENE-9908:
--
Fix Version/s: main (9.0)
> Move VectorValues#search to VectorReader and LeafReader
> ---
>
> Key: LUCENE-9908
> URL: https://issues.apache.org/jira/browse/LUCENE-9908
> Project: Lucene - Core
> Issue Type: Improvement
>Affects Versions: main (9.0)
>Reporter: Adrien Grand
>Assignee: Julie Tibshirani
>Priority: Blocker
> Fix For: main (9.0)
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> As ANN search doesn't require a positioned iterator, we should move it from
> {{VectorValues}} to {{VectorReader}} and make it available from
> {{LeafReader}} via a new API, something like
> {{LeafReader#searchNearestNeighbors}}?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-9145) Address warnings found by static analysis
[ https://issues.apache.org/jira/browse/LUCENE-9145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Drob resolved LUCENE-9145. --- Fix Version/s: 9.0 Resolution: Fixed Fine to close, I think there's been work done in other places as well, like when we set up more of the error-prone stuff. > Address warnings found by static analysis > - > > Key: LUCENE-9145 > URL: https://issues.apache.org/jira/browse/LUCENE-9145 > Project: Lucene - Core > Issue Type: Sub-task >Reporter: Mike Drob >Assignee: Mike Drob >Priority: Major > Fix For: 9.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9334) Require consistency between data-structures on a per-field basis
[ https://issues.apache.org/jira/browse/LUCENE-9334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17326129#comment-17326129 ] Mike Drob commented on LUCENE-9334: --- Thanks David! I tried searching the dev list and for existing issues, but it looks like I started with the other end of the failing tests than you did. Thanks for being proactive! > Require consistency between data-structures on a per-field basis > > > Key: LUCENE-9334 > URL: https://issues.apache.org/jira/browse/LUCENE-9334 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Adrien Grand >Priority: Blocker > Fix For: main (9.0) > > Time Spent: 14.5h > Remaining Estimate: 0h > > Follow-up of > https://lists.apache.org/thread.html/r747de568afd7502008c45783b74cc3aeb31dab8aa60fcafaf65d5431%40%3Cdev.lucene.apache.org%3E. > We would like to start requiring consitency across data-structures on a > per-field basis in order to make it easier to do the right thing by default: > range queries can run faster if doc values are enabled, sorted queries can > run faster if points by indexed, etc. > This would be a big change, so it should be rolled out in a major. > Strict validation is tricky to implement, but we should still implement > best-effort validation: > - Documents all use the same data-structures, e.g. it is illegal for a > document to only enable points and another document to only enable doc values, > - When possible, check whether values are consistent too. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9334) Require consistency between data-structures on a per-field basis
[ https://issues.apache.org/jira/browse/LUCENE-9334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17326127#comment-17326127 ] Mike Drob commented on LUCENE-9334: --- I think this is causing SOLR-15360, but I can't say for certain. If there's any chance that somebody can come over and help us understand a bit more, that would be much appreciated. > Require consistency between data-structures on a per-field basis > > > Key: LUCENE-9334 > URL: https://issues.apache.org/jira/browse/LUCENE-9334 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Adrien Grand >Priority: Blocker > Fix For: main (9.0) > > Time Spent: 14.5h > Remaining Estimate: 0h > > Follow-up of > https://lists.apache.org/thread.html/r747de568afd7502008c45783b74cc3aeb31dab8aa60fcafaf65d5431%40%3Cdev.lucene.apache.org%3E. > We would like to start requiring consitency across data-structures on a > per-field basis in order to make it easier to do the right thing by default: > range queries can run faster if doc values are enabled, sorted queries can > run faster if points by indexed, etc. > This would be a big change, so it should be rolled out in a major. > Strict validation is tricky to implement, but we should still implement > best-effort validation: > - Documents all use the same data-structures, e.g. it is illegal for a > document to only enable points and another document to only enable doc values, > - When possible, check whether values are consistent too. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-9912) Disable HTML doclint in Lucene/Solr 8
[
https://issues.apache.org/jira/browse/LUCENE-9912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mike Drob updated LUCENE-9912:
--
Fix Version/s: (was: 8.8.2)
8.8.3
> Disable HTML doclint in Lucene/Solr 8
> -
>
> Key: LUCENE-9912
> URL: https://issues.apache.org/jira/browse/LUCENE-9912
> Project: Lucene - Core
> Issue Type: Task
> Components: general/build
>Affects Versions: 8.x, 8.8.1
> Environment: openjdk version "17-ea" 2021-09-14
> OpenJDK Runtime Environment (build 17-ea+16-1315)
> OpenJDK 64-Bit Server VM (build 17-ea+16-1315, mixed mode, sharing)
>Reporter: Uwe Schindler
>Assignee: Uwe Schindler
>Priority: Blocker
> Labels: Java17
> Fix For: 8.x, 8.8.3
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Due to LUCENE-9209 we fixed most of the Javadocs in Lucene / Solr master/main.
> But we did not do that in Lucene 8. Unfortunetaly later Java versions (JDK
> 17) now fail to compile or run javadoc, because the later java versions apply
> very strict HTML5 checks.
> The main problem again is thet we unconditinally enable {{-Werror}} at the
> same time with {{-Xdoclint:all}} (as said hundreds of times by me, we should
> really only enable warnings we know of!).
> To allow compilation with JDK-17 or later I will remove the HTML warnings
> from the config
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9912) Disable HTML doclint in Lucene/Solr 8
[
https://issues.apache.org/jira/browse/LUCENE-9912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17316428#comment-17316428
]
Mike Drob commented on LUCENE-9912:
---
[~uschindler] - I had already created RC1 for 8.8.2 before this was committed,
do you feel this is serious enough to warrant a new RC?
I don't fully understand the scope of the issue - I think it only impacts
compilation, but want to clarify that it does not affect running with Java 17
as well?
> Disable HTML doclint in Lucene/Solr 8
> -
>
> Key: LUCENE-9912
> URL: https://issues.apache.org/jira/browse/LUCENE-9912
> Project: Lucene - Core
> Issue Type: Task
> Components: general/build
>Affects Versions: 8.x, 8.8.1
> Environment: openjdk version "17-ea" 2021-09-14
> OpenJDK Runtime Environment (build 17-ea+16-1315)
> OpenJDK 64-Bit Server VM (build 17-ea+16-1315, mixed mode, sharing)
>Reporter: Uwe Schindler
>Assignee: Uwe Schindler
>Priority: Blocker
> Labels: Java17
> Fix For: 8.x, 8.8.2
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Due to LUCENE-9209 we fixed most of the Javadocs in Lucene / Solr master/main.
> But we did not do that in Lucene 8. Unfortunetaly later Java versions (JDK
> 17) now fail to compile or run javadoc, because the later java versions apply
> very strict HTML5 checks.
> The main problem again is thet we unconditinally enable {{-Werror}} at the
> same time with {{-Xdoclint:all}} (as said hundreds of times by me, we should
> really only enable warnings we know of!).
> To allow compilation with JDK-17 or later I will remove the HTML warnings
> from the config
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-9762) DoubleValuesSource.fromQuery can fail when doc visited twice
[
https://issues.apache.org/jira/browse/LUCENE-9762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mike Drob updated LUCENE-9762:
--
Fix Version/s: 8.8.2
> DoubleValuesSource.fromQuery can fail when doc visited twice
>
>
> Key: LUCENE-9762
> URL: https://issues.apache.org/jira/browse/LUCENE-9762
> Project: Lucene - Core
> Issue Type: Bug
>Affects Versions: 8.5
>Reporter: Chris M. Hostetter
>Assignee: David Smiley
>Priority: Major
> Fix For: 8.9, 8.8.2
>
> Attachments: LUCENE-9762.patch
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> As originally reported by Nicolás Lichtmaier on the java-user list, there are
> some trivial situations which can trigger an assertion error in the
> PostingsReader when enumerating PhrasePositions for a sloppy PhraseQuery...
> {noformat}
> Exception in thread "main" java.lang.AssertionError
> at
> org.apache.lucene.codecs.lucene84.Lucene84PostingsReader$EverythingEnum.nextPosition(Lucene84PostingsReader.java:940)
> at
> org.apache.lucene.search.PhrasePositions.nextPosition(PhrasePositions.java:57)
> at
> org.apache.lucene.search.PhrasePositions.firstPosition(PhrasePositions.java:46)
> at
> org.apache.lucene.search.SloppyPhraseMatcher.initSimple(SloppyPhraseMatcher.java:368)
> at
> org.apache.lucene.search.SloppyPhraseMatcher.initPhrasePositions(SloppyPhraseMatcher.java:356)
> at
> org.apache.lucene.search.SloppyPhraseMatcher.reset(SloppyPhraseMatcher.java:153)
> at org.apache.lucene.search.PhraseScorer$1.matches(PhraseScorer.java:49)
> at
> org.apache.lucene.search.DoubleValuesSource$WeightDoubleValuesSource$1.advanceExact(DoubleValuesSource.java:631)
> at
> org.apache.lucene.queries.function.FunctionScoreQuery$QueryBoostValuesSource$1.advanceExact(FunctionScoreQuery.java:343)
> at
> org.apache.lucene.search.DoubleValues$1.advanceExact(DoubleValues.java:53)
> at
> org.apache.lucene.search.DoubleValues$1.advanceExact(DoubleValues.java:53)
> at
> org.apache.lucene.queries.function.FunctionScoreQuery$MultiplicativeBoostValuesSource$1.advanceExact(FunctionScoreQuery.java:270)
> at
> org.apache.lucene.queries.function.FunctionScoreQuery$FunctionScoreWeight$1.score(FunctionScoreQuery.java:228)
> at
> org.apache.lucene.search.DoubleValuesSource$2.doubleValue(DoubleValuesSource.java:344)
> at
> org.apache.lucene.search.DoubleValues$1.doubleValue(DoubleValues.java:48)
> at
> org.apache.lucene.queries.function.FunctionScoreQuery$MultiplicativeBoostValuesSource$1.doubleValue(FunctionScoreQuery.java:265)
> at
> org.apache.lucene.queries.function.FunctionScoreQuery$FunctionScoreWeight$1.score(FunctionScoreQuery.java:229)
> at
> org.apache.lucene.search.TopScoreDocCollector$SimpleTopScoreDocCollector$1.collect(TopScoreDocCollector.java:76)
> at
> org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:276)
> at
> org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:232)
> at org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)
> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:659)
> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:443)
> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:572)
> at
> org.apache.lucene.search.IndexSearcher.searchAfter(IndexSearcher.java:419)
> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:430)
> at LuceneCrash.main(LuceneCrash.java:51)
> {noformat}
> http://mail-archives.apache.org/mod_mbox/lucene-java-user/202102.mbox/%3C177a65ec-5ec3-e1aa-99c3-b478e165d5e8%40wolfram.com%3E
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-9744) NPE on a degenerate query in MinimumShouldMatchIntervalsSource$MinimumMatchesIterator.getSubMatches()
[
https://issues.apache.org/jira/browse/LUCENE-9744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mike Drob updated LUCENE-9744:
--
Fix Version/s: 8.8.2
> NPE on a degenerate query in
> MinimumShouldMatchIntervalsSource$MinimumMatchesIterator.getSubMatches()
> -
>
> Key: LUCENE-9744
> URL: https://issues.apache.org/jira/browse/LUCENE-9744
> Project: Lucene - Core
> Issue Type: Bug
>Affects Versions: main (9.0), 8.7
>Reporter: Dawid Weiss
>Assignee: Alan Woodward
>Priority: Major
> Fix For: main (9.0), 8.9, 8.8.2
>
> Attachments: LUCENE-9744.patch, LUCENE-9744.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> This query:
> {code}
> Query q1 =
> new IntervalQuery(
> FLD_TEXT1,
> Intervals.ordered(
> Intervals.atLeast(1, Intervals.term("foo")),
> Intervals.atLeast(1, Intervals.term("bar";
> {code}
> produces an NPE when retrieving match regions. Note degenerate atLeast with
> only one argument - this may play a role but it's not signaled anywhere
> before (the query itself does select the right documents, NPE blows up when
> matches are retrieved).
> Stack:
> {code}
> java.lang.NullPointerException
> at
> __randomizedtesting.SeedInfo.seed([2DDAAECF04D90D2F:EB69410397F8761E]:0)
> at
> org.apache.lucene.queries.intervals.MinimumShouldMatchIntervalsSource$MinimumMatchesIterator.getSubMatches(MinimumShouldMatchIntervalsSource.java:389)
> at
> org.apache.lucene.queries.intervals.CachingMatchesIterator.cache(CachingMatchesIterator.java:39)
> at
> org.apache.lucene.queries.intervals.CachingMatchesIterator.next(CachingMatchesIterator.java:68)
> at
> org.apache.lucene.queries.intervals.IntervalMatches$2.nextInterval(IntervalMatches.java:137)
> at
> org.apache.lucene.queries.intervals.OrderedIntervalsSource$OrderedIntervalIterator.nextInterval(OrderedIntervalsSource.java:166)
> at
> org.apache.lucene.queries.intervals.ConjunctionIntervalsSource.matches(ConjunctionIntervalsSource.java:89)
> at
> org.apache.lucene.queries.intervals.IntervalQuery$IntervalWeight.lambda$matches$0(IntervalQuery.java:177)
> at org.apache.lucene.search.MatchesUtils.forField(MatchesUtils.java:115)
> at
> org.apache.lucene.queries.intervals.IntervalQuery$IntervalWeight.matches(IntervalQuery.java:174)
> at
> org.apache.lucene.search.matchhighlight.MatchRegionRetriever.highlightDocument(MatchRegionRetriever.java:210)
> at
> org.apache.lucene.search.matchhighlight.MatchRegionRetriever.highlightDocuments(MatchRegionRetriever.java:193)
> at
> org.apache.lucene.search.matchhighlight.MatchRegionRetriever.highlightDocuments(MatchRegionRetriever.java:146)
> at
> org.apache.lucene.search.matchhighlight.MatchHighlighter.highlight(MatchHighlighter.java:236)
> at
> org.apache.lucene.search.matchhighlight.TestMatchHighlighter.lambda$testNpeOnIntervalQuery$7(TestMatchHighlighter.java:540)
> at
> org.apache.lucene.search.matchhighlight.IndexBuilder.build(IndexBuilder.java:100)
> at
> org.apache.lucene.search.matchhighlight.TestMatchHighlighter.testNpeOnIntervalQuery(TestMatchHighlighter.java:510)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:564)
> {code}
> Repro test attached.
> [~romseygeek], help! :)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-9722) Aborted merge can leak readers if the output is empty
[ https://issues.apache.org/jira/browse/LUCENE-9722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Drob updated LUCENE-9722: -- Fix Version/s: (was: 8.8.2) 8.8.1 > Aborted merge can leak readers if the output is empty > - > > Key: LUCENE-9722 > URL: https://issues.apache.org/jira/browse/LUCENE-9722 > Project: Lucene - Core > Issue Type: Bug > Components: core/index >Affects Versions: main (9.0), 8.8 >Reporter: Nhat Nguyen >Assignee: Nhat Nguyen >Priority: Major > Fix For: main (9.0), 8.9, 8.8.1 > > Time Spent: 1h > Remaining Estimate: 0h > > We fail to close the merged readers of an aborted merge if its output segment > contains no document. > This bug was discovered by a test in Elasticsearch > ([elastic/elasticsearch#67884|https://github.com/elastic/elasticsearch/issues/67884]). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9836) Fix 8.x Maven Validation and publication to work with Maven Central and HTTPS again; remove pure Maven build (did not work anymore)
[
https://issues.apache.org/jira/browse/LUCENE-9836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17310834#comment-17310834
]
Mike Drob commented on LUCENE-9836:
---
[~uschindler] - do we need this back ported to 8.8.x bug fix branch? I'm
unclear on the impact of the issue - will it prevent a release from completing
or is this mostly addressing failures on the CI side?
> Fix 8.x Maven Validation and publication to work with Maven Central and HTTPS
> again; remove pure Maven build (did not work anymore)
> ---
>
> Key: LUCENE-9836
> URL: https://issues.apache.org/jira/browse/LUCENE-9836
> Project: Lucene - Core
> Issue Type: Improvement
> Components: general/build
>Affects Versions: 8.x, 8.9
>Reporter: Uwe Schindler
>Assignee: Uwe Schindler
>Priority: Major
> Fix For: 8.x, 8.9
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Currenty the Maven related stuff in 8.x completely fails, because
> Maven-Ant-Tasks is so outdated, that it has hardcoded Maven Central without
> HTTPS. This makes downloading fail.
> You can mostly fix this with an additional remote repository, so it can
> fallback to that one.
> I'd like to do the following on 8.x:
> - Remove the Ant-Support for Maven: {{ant run-maven-build}} (this no longer
> bootsraps, because Maven Ant Tasks can't download Maven, as here is no way to
> override hardcoded repo; I have a workaround in forbiddenapis, but that's too
> complicated, so I will simply remoe that task)
> - Fix the dependency checker: This works, but unfortunately there are some
> artifacts which itsself have "http:" in their POM file, those fail to
> download. Newer Maven versions have an hardcoded "fixer" in it, but Maven Ant
> Tasks again is missing this. I have no idea how to handle that.
> I already tried some heavy committing, but the only way to solve this is to
> replace maven-ant-tasks with the followup ant task. I am not sure if this
> worth the trouble!
> What do others think? Should we maybe simply disable the Maven Dependency
> checker?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9647) Add back ant precommit on PR for branch 8
[ https://issues.apache.org/jira/browse/LUCENE-9647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17304967#comment-17304967 ] Mike Drob commented on LUCENE-9647: --- Maybe the easiest way to implement this is to enable ant precommit on the lucene-solr repo, and leave only gradle on the lucene and solr repos. > Add back ant precommit on PR for branch 8 > - > > Key: LUCENE-9647 > URL: https://issues.apache.org/jira/browse/LUCENE-9647 > Project: Lucene - Core > Issue Type: Task > Components: general/build >Reporter: Mike Drob >Assignee: Mike Drob >Priority: Minor > Fix For: main (9.0) > > Time Spent: 1h 20m > Remaining Estimate: 0h > > When migrating everything to gradle only, we accidentally deleted our > branch_8x PR precommit action. > The file needs to be on master, but the branch specification should specify > branch_8x I believe. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-15245) Ref Guide missing "zk-read" permission
[ https://issues.apache.org/jira/browse/SOLR-15245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17299205#comment-17299205 ] Mike Drob commented on SOLR-15245: -- Should access to security.json require both zk-read and security-read or is just one sufficient? > Ref Guide missing "zk-read" permission > -- > > Key: SOLR-15245 > URL: https://issues.apache.org/jira/browse/SOLR-15245 > Project: Solr > Issue Type: Task > Security Level: Public(Default Security Level. Issues are Public) > Components: documentation >Reporter: Mike Drob >Priority: Major > > https://solr.apache.org/guide/8_8/rule-based-authorization-plugin.html#predefined-permissions > The docs do not explain what is the zk-read permission, found at > PermissionNameProvider. > Also, for some reason /cluster/zk/data/* uses the collection read permission > instead of ZK read permission, that should be either documented or corrected > as well. > cc: [~noble.paul] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-15245) Ref Guide missing "zk-read" permission
[ https://issues.apache.org/jira/browse/SOLR-15245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Drob updated SOLR-15245: - Issue Type: Task (was: Test) > Ref Guide missing "zk-read" permission > -- > > Key: SOLR-15245 > URL: https://issues.apache.org/jira/browse/SOLR-15245 > Project: Solr > Issue Type: Task > Security Level: Public(Default Security Level. Issues are Public) > Components: documentation >Reporter: Mike Drob >Priority: Major > > https://solr.apache.org/guide/8_8/rule-based-authorization-plugin.html#predefined-permissions > The docs do not explain what is the zk-read permission, found at > PermissionNameProvider. > Also, for some reason /cluster/zk/data/* uses the collection read permission > instead of ZK read permission, that should be either documented or corrected > as well. > cc: [~noble.paul] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-15245) Ref Guide missing "zk-read" permission
Mike Drob created SOLR-15245: Summary: Ref Guide missing "zk-read" permission Key: SOLR-15245 URL: https://issues.apache.org/jira/browse/SOLR-15245 Project: Solr Issue Type: Test Security Level: Public (Default Security Level. Issues are Public) Components: documentation Reporter: Mike Drob https://solr.apache.org/guide/8_8/rule-based-authorization-plugin.html#predefined-permissions The docs do not explain what is the zk-read permission, found at PermissionNameProvider. Also, for some reason /cluster/zk/data/* uses the collection read permission instead of ZK read permission, that should be either documented or corrected as well. cc: [~noble.paul] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-15244) Use NIO methods instead of IO when interacting with Lucene Directory
Mike Drob created SOLR-15244: Summary: Use NIO methods instead of IO when interacting with Lucene Directory Key: SOLR-15244 URL: https://issues.apache.org/jira/browse/SOLR-15244 Project: Solr Issue Type: Improvement Security Level: Public (Default Security Level. Issues are Public) Reporter: Mike Drob We have a lot of places where we are still using IO methods and File objects, and then doing conversions to Path for accessing Lucene Directory. We also have places where we are doing a bunch of conversions from Path to String and back to Path through SolrResourceLoader. Do a sweep of these and update at least the easy ones. There are a few places where File APIs might still make more sense or be too burdensome to convert, can leave those for another day. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-15234) PRS has issues with many concurrent creations
[
https://issues.apache.org/jira/browse/SOLR-15234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17298303#comment-17298303
]
Mike Drob commented on SOLR-15234:
--
This was in master, I’ll look for branch_8x tests shortly
> PRS has issues with many concurrent creations
> -
>
> Key: SOLR-15234
> URL: https://issues.apache.org/jira/browse/SOLR-15234
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: Mike Drob
>Priority: Major
>
> Running in a cloud environment, I am creating lots of 2x2 collections on a
> single node with 10 threads using PRS. These tests work fine without PRS.
> I see the following errors in my logs:
> {noformat}
> 2021-03-09 05:57:46.311 ERROR
> (OverseerStateUpdate-72069990450593800-hostname:8983_solr-n_02) [ ]
> o.a.s.c.c.PerReplicaStatesOps Multi-op exception: [core_node8:2:A:L,
> core_node6:4:D, core_node2:4:D, core_node4:2:A:L]
> {noformat}
> It would be good to include MDC logging on this if we can, so that I can know
> what collection the error describes.
> A little bit later:
> {noformat}
> 2021-03-09 05:57:46.694 INFO (qtp594858858-16) [
> x:collection-78_shard2_replica_n7] o.a.s.h.a.PrepRecoveryOp Going to wait for
> coreNodeName: core_node6, state: recovering, checkLive: true, onlyIfLeader:
> true, onlyIfLeaderActive: true
> 2021-03-09 05:57:46.695 ERROR (qtp594858858-16) [
> x:collection-78_shard2_replica_n7] o.a.s.h.RequestHandlerBase
> org.apache.solr.common.SolrException: core not
> found:collection-78_shard2_replica_n7
> at
> org.apache.solr.handler.admin.PrepRecoveryOp.execute(PrepRecoveryOp.java:71)
> at
> org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:367)
> at
> org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:397)
> at
> org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:181)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:214)
> at
> org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:836)
> at
> org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:800)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:545)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:518)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:432)
> at
> org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201)
> at
> org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1612)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1582)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191)
> at
> org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:177)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at org.eclipse.jetty.server.Server.handle(Server.java:516)
> at
> org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
[jira] [Commented] (SOLR-15234) PRS has issues with many concurrent creations
[
https://issues.apache.org/jira/browse/SOLR-15234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17298229#comment-17298229
]
Mike Drob commented on SOLR-15234:
--
Looking at the code, that first error seems like it could be totally benign?
> PRS has issues with many concurrent creations
> -
>
> Key: SOLR-15234
> URL: https://issues.apache.org/jira/browse/SOLR-15234
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: Mike Drob
>Priority: Major
>
> Running in a cloud environment, I am creating lots of 2x2 collections on a
> single node with 10 threads using PRS. These tests work fine without PRS.
> I see the following errors in my logs:
> {noformat}
> 2021-03-09 05:57:46.311 ERROR
> (OverseerStateUpdate-72069990450593800-hostname:8983_solr-n_02) [ ]
> o.a.s.c.c.PerReplicaStatesOps Multi-op exception: [core_node8:2:A:L,
> core_node6:4:D, core_node2:4:D, core_node4:2:A:L]
> {noformat}
> It would be good to include MDC logging on this if we can, so that I can know
> what collection the error describes.
> A little bit later:
> {noformat}
> 2021-03-09 05:57:46.694 INFO (qtp594858858-16) [
> x:collection-78_shard2_replica_n7] o.a.s.h.a.PrepRecoveryOp Going to wait for
> coreNodeName: core_node6, state: recovering, checkLive: true, onlyIfLeader:
> true, onlyIfLeaderActive: true
> 2021-03-09 05:57:46.695 ERROR (qtp594858858-16) [
> x:collection-78_shard2_replica_n7] o.a.s.h.RequestHandlerBase
> org.apache.solr.common.SolrException: core not
> found:collection-78_shard2_replica_n7
> at
> org.apache.solr.handler.admin.PrepRecoveryOp.execute(PrepRecoveryOp.java:71)
> at
> org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:367)
> at
> org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:397)
> at
> org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:181)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:214)
> at
> org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:836)
> at
> org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:800)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:545)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:518)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:432)
> at
> org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201)
> at
> org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1612)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1582)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191)
> at
> org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:177)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at org.eclipse.jetty.server.Server.handle(Server.java:516)
> at
> org.eclipse.jetty.server.HttpChannel.lambda$handle$1(Htt
[jira] [Created] (SOLR-15234) PRS has issues with many concurrent creations
Mike Drob created SOLR-15234:
Summary: PRS has issues with many concurrent creations
Key: SOLR-15234
URL: https://issues.apache.org/jira/browse/SOLR-15234
Project: Solr
Issue Type: Bug
Security Level: Public (Default Security Level. Issues are Public)
Components: SolrCloud
Reporter: Mike Drob
Running in a cloud environment, I am creating lots of 2x2 collections on a
single node with 10 threads using PRS. These tests work fine without PRS.
I see the following errors in my logs:
{noformat}
2021-03-09 05:57:46.311 ERROR
(OverseerStateUpdate-72069990450593800-hostname:8983_solr-n_02) [ ]
o.a.s.c.c.PerReplicaStatesOps Multi-op exception: [core_node8:2:A:L,
core_node6:4:D, core_node2:4:D, core_node4:2:A:L]
{noformat}
It would be good to include MDC logging on this if we can, so that I can know
what collection the error describes.
A little bit later:
{noformat}
2021-03-09 05:57:46.694 INFO (qtp594858858-16) [
x:collection-78_shard2_replica_n7] o.a.s.h.a.PrepRecoveryOp Going to wait for
coreNodeName: core_node6, state: recovering, checkLive: true, onlyIfLeader:
true, onlyIfLeaderActive: true
2021-03-09 05:57:46.695 ERROR (qtp594858858-16) [
x:collection-78_shard2_replica_n7] o.a.s.h.RequestHandlerBase
org.apache.solr.common.SolrException: core not
found:collection-78_shard2_replica_n7
at
org.apache.solr.handler.admin.PrepRecoveryOp.execute(PrepRecoveryOp.java:71)
at
org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:367)
at
org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:397)
at
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:181)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:214)
at
org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:836)
at
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:800)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:545)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:518)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:432)
at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201)
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1612)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1582)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191)
at
org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:177)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at
org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInt
[jira] [Commented] (SOLR-15215) SolrJ: Remove needless Netty dependency
[ https://issues.apache.org/jira/browse/SOLR-15215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17296971#comment-17296971 ] Mike Drob commented on SOLR-15215: -- Let’s start creating those dependencies at the same time that we remove them from solrj, and also a solrj-all dependency that people can use for minimal changes from their perspective > SolrJ: Remove needless Netty dependency > --- > > Key: SOLR-15215 > URL: https://issues.apache.org/jira/browse/SOLR-15215 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrJ >Reporter: David Smiley >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > SolrJ depends on Netty transitively via ZooKeeper. But ZooKeeper's Netty > dependency should be considered optional -- you have to opt-in. > BTW it's only needed in Solr-core because of Hadoop/HDFS which ought to move > to a contrib and take this dependency with it over there. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-15215) SolrJ: Remove needless Netty dependency
[ https://issues.apache.org/jira/browse/SOLR-15215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17296922#comment-17296922 ] Mike Drob commented on SOLR-15215: -- I was chatting with a few zookeeper folks and they made it sound like netty would be the preferred connection approach going forward. I don’t know about the performance of our http cluster state provider, so would like to have the ZK netty option still available. Can we add a new module that is just the right transitive dependencies for ZK-netty? Doesn’t need to be a package. > SolrJ: Remove needless Netty dependency > --- > > Key: SOLR-15215 > URL: https://issues.apache.org/jira/browse/SOLR-15215 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrJ >Reporter: David Smiley >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > SolrJ depends on Netty transitively via ZooKeeper. But ZooKeeper's Netty > dependency should be considered optional -- you have to opt-in. > BTW it's only needed in Solr-core because of Hadoop/HDFS which ought to move > to a contrib and take this dependency with it over there. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-15215) SolrJ: Remove needless Netty dependency
[ https://issues.apache.org/jira/browse/SOLR-15215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17296270#comment-17296270 ] Mike Drob commented on SOLR-15215: -- Agree with Jan, I'd like to see this available as a package before we remove it. > SolrJ: Remove needless Netty dependency > --- > > Key: SOLR-15215 > URL: https://issues.apache.org/jira/browse/SOLR-15215 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrJ >Reporter: David Smiley >Assignee: David Smiley >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > SolrJ depends on Netty transitively via ZooKeeper. But ZooKeeper's Netty > dependency should be considered optional -- you have to opt-in. > BTW it's only needed in Solr-core because of Hadoop/HDFS which ought to move > to a contrib and take this dependency with it over there. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-2852) SolrJ doesn't need woodstox jar
[ https://issues.apache.org/jira/browse/SOLR-2852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17296253#comment-17296253 ] Mike Drob commented on SOLR-2852: - Does this mean that SolrJ clients will not be able to read xml documents on disk for indexing? > SolrJ doesn't need woodstox jar > --- > > Key: SOLR-2852 > URL: https://issues.apache.org/jira/browse/SOLR-2852 > Project: Solr > Issue Type: Improvement > Components: clients - java >Reporter: David Smiley >Assignee: David Smiley >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > The /dist/solrj-lib/ directory contains wstx-asl-3.2.7.jar (Woodstox StAX > API). SolrJ doesn't actually have any type of dependency on this library. > The maven build doesn't have it as a dependency and the tests pass. Perhaps > Woodstox is faster than the JDK's StAX, I don't know, but I find that point > quite moot since SolrJ can use the efficient binary format. Woodstox is not > a small library either, weighting in at 524KB, and of course if someone > actually wants to use it, they can. > I propose woodstox be removed as a SolrJ dependency. I am *not* proposing it > be removed as a Solr WAR dependency since it is actually required there due > to an obscure XSLT issue. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9791) Monitor (aka Luwak) has concurrency issues related to BytesRefHash#find
[ https://issues.apache.org/jira/browse/LUCENE-9791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17290144#comment-17290144 ] Mike Drob commented on LUCENE-9791: --- > Maybe another clean way would be some method in BytesRefHash that returns a > read-only clone I see that Uwe already proposed this as well, I like this idea. > Monitor (aka Luwak) has concurrency issues related to BytesRefHash#find > --- > > Key: LUCENE-9791 > URL: https://issues.apache.org/jira/browse/LUCENE-9791 > Project: Lucene - Core > Issue Type: Bug > Components: core/other >Affects Versions: master (9.0), 8.7, 8.8 >Reporter: Paweł Bugalski >Priority: Major > Attachments: LUCENE-9791.patch > > Time Spent: 40m > Remaining Estimate: 0h > > _org.apache.lucene.monitor.Monitor_ can sometimes *NOT* match a document that > should be matched by one of registered queries if match operations are run > concurrently from multiple threads. > This is because sometimes in a concurrent environment > _TermFilteredPresearcher_ might not select a query that could later on match > one of documents being matched. > Internally _TermFilteredPresearcher_ is using a term acceptor: an instance of > _org.apache.lucene.monitor.QueryIndex.QueryTermFilter_. _QueryTermFilter_ is > correctly initialized under lock and its internal state (a map of > _org.apache.lucene.util.BytesRefHash_ instances) is correctly published. > Later one when those instances are used concurrently a problem with > _org.apache.lucene.util.BytesRefHash#find_ is triggered since it is not > thread safe. > _org.apache.lucene.util.BytesRefHash#find_ internally is using a private > _org.apache.lucene.util.BytesRefHash#equals_ method, which is using an > instance field _scratch1_ as a temporary buffer to compare its _ByteRef_ > parameter with contents of _ByteBlockPool_. This is not thread safe and can > cause incorrect answers as well as _ArrayOutOfBoundException_. > __ > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9791) Monitor (aka Luwak) has concurrency issues related to BytesRefHash#find
[
https://issues.apache.org/jira/browse/LUCENE-9791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17290140#comment-17290140
]
Mike Drob commented on LUCENE-9791:
---
>From the comments on the PR:
{quote}As a result it is now possible to call BytesRefHash#find concurrently as
long as there are
no concurrent modifications to BytesRefHash instance and it is correctly
published.
{quote}
My comment:
{quote}This feels like a trap waiting for somebody to fall in. I think a better
solution would be a pool of buffers and proper concurrency controls around
them, but don't know what the performance would look like.
At a minimum, we need to update the method javadoc to explain what is going on
and what proper usage looks like.
{quote}
Robert's response:
{quote}sorry I don't think it is a trap really. Everything else using this
thing is using it in a per-thread way. A lot of lucene works this way, it keeps
things fast.
I don't think we should slow down all the correct users and add complex
concurrency, just because we have one module abusing it. This would be
equivalent to making all java code use HashTable/StringBuffer when most code
only needs HashMap/StringBuilder. We could easily slow down important things
such as indexing. See my comments on the issue.
{quote}
I think we agree here. Specifically, I find Robert's earlier comment in this
JIRA very good - that we should not be trying to make this partially thread
safe. If we're going to be doing changes like this to make it thread safe, that
is the case where I would push for a full solution, but acknowledge that the
performance of that would likely be a high cost.
I think it is a problem if we start saying that you can use it concurrently
_this way_ but not concurrently _that way_. Eventually somebody gets confused.
Better to just draw a line in the sand and say this class is or is not thread
safe and be done with it.
Maybe another idea is to have a FrozenBytesRefHash that is explicitly safe for
concurrent lookup because it does not support modification?
> Monitor (aka Luwak) has concurrency issues related to BytesRefHash#find
> ---
>
> Key: LUCENE-9791
> URL: https://issues.apache.org/jira/browse/LUCENE-9791
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/other
>Affects Versions: master (9.0), 8.7, 8.8
>Reporter: Paweł Bugalski
>Priority: Major
> Attachments: LUCENE-9791.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> _org.apache.lucene.monitor.Monitor_ can sometimes *NOT* match a document that
> should be matched by one of registered queries if match operations are run
> concurrently from multiple threads.
> This is because sometimes in a concurrent environment
> _TermFilteredPresearcher_ might not select a query that could later on match
> one of documents being matched.
> Internally _TermFilteredPresearcher_ is using a term acceptor: an instance of
> _org.apache.lucene.monitor.QueryIndex.QueryTermFilter_. _QueryTermFilter_ is
> correctly initialized under lock and its internal state (a map of
> _org.apache.lucene.util.BytesRefHash_ instances) is correctly published.
> Later one when those instances are used concurrently a problem with
> _org.apache.lucene.util.BytesRefHash#find_ is triggered since it is not
> thread safe.
> _org.apache.lucene.util.BytesRefHash#find_ internally is using a private
> _org.apache.lucene.util.BytesRefHash#equals_ method, which is using an
> instance field _scratch1_ as a temporary buffer to compare its _ByteRef_
> parameter with contents of _ByteBlockPool_. This is not thread safe and can
> cause incorrect answers as well as _ArrayOutOfBoundException_.
> __
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-15178) Non-existent dependency listed in solr-core
[
https://issues.apache.org/jira/browse/SOLR-15178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17290072#comment-17290072
]
Mike Drob commented on SOLR-15178:
--
Ok, yea, I see what's going on now. The solr:server looks like a collection of
all the server dependencies, for running the Jetty container, but doesn't have
any java code itself. It has a handful of scripts and does the packaging for
the server tar. Given this, the change you proposed in the PR makes sense but
we'll need to verify that it doesn't mess up the contents of {{assemble}}.
More generally, I don't understand the distinction between when a dependency
needs to go in server or when it goes in core. I'm doing have time to do full
build testing this week, but will try to take a look next week.
> Non-existent dependency listed in solr-core
> ---
>
> Key: SOLR-15178
> URL: https://issues.apache.org/jira/browse/SOLR-15178
> Project: Solr
> Issue Type: Bug
>Affects Versions: master (9.0)
>Reporter: Bence Szabó
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Solr-core has a dependency, org.apache.solr:*server*, which fails to download.
> For testing I created a test project here:
> [https://github.com/bszabo97/solr_master_dep_test]
> If I run the command {{gradle -q dependencies --configuration solrCore}} the
> dependency org.apache.solr:server shows up and fails, though if I run
> {{gradle -q dependencies --configuration solrCore8}} it doesn't show up at
> all.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-15169) SolrPaths.assertPathAllowed normalization problem
[
https://issues.apache.org/jira/browse/SOLR-15169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289243#comment-17289243
]
Mike Drob commented on SOLR-15169:
--
I'm not sure that it is a good idea to start allowing directory traversal for
the allowed paths. I don't have a specific proof of concept for the kind of
escalation it would allow, but intuitively something about it feels wrong.
That said, if we are going to do this, we should normalize when we add the path
elements in the CoreContainer constructor and store those, not doing the
normalize on each check.
I would also move the unit test to TestCoreContainer, we already have some
similar tests there, instead of creating a new class.
Will need to think some more about whether this is safe, though. Maybe somebody
else will be able to chime in with additional perspective.
> SolrPaths.assertPathAllowed normalization problem
> -
>
> Key: SOLR-15169
> URL: https://issues.apache.org/jira/browse/SOLR-15169
> Project: Solr
> Issue Type: Bug
>Reporter: Andras Salamon
>Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In the {{SolrPaths.assertPathAllowed}} the {{normalize()}} method is only
> called for {{pathToAssert}} and not for the {{allowPaths}} elements, which
> means that the following call gives SolrException:
> {noformat}
> SolrPaths.assertPathAllowed(Path.of("/a/b/../b/d"),
> Set.of(Path.of("/a/b/../b"), Path.of("/c"))); {noformat}
> even if "a/b/..b/" is a prefix of "/a/b/../b/d".
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-15178) Non-existent dependency listed in solr-core
[
https://issues.apache.org/jira/browse/SOLR-15178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289180#comment-17289180
]
Mike Drob commented on SOLR-15178:
--
Thank you for reporting this. I am able to reproduce your issue.
I suspect that the correct change is to remove the dependency on solr:server,
which I think was added erroneously. [~dawid.weiss] - can you double check?
https://github.com/apache/lucene-solr/blob/master/solr/core/build.gradle#L54
> Non-existent dependency listed in solr-core
> ---
>
> Key: SOLR-15178
> URL: https://issues.apache.org/jira/browse/SOLR-15178
> Project: Solr
> Issue Type: Bug
>Affects Versions: master (9.0)
>Reporter: Bence Szabó
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Solr-core has a dependency, org.apache.solr:*server*, which fails to download.
> For testing I created a test project here:
> [https://github.com/bszabo97/solr_master_dep_test]
> If I run the command {{gradle -q dependencies --configuration solrCore}} the
> dependency org.apache.solr:server shows up and fails, though if I run
> {{gradle -q dependencies --configuration solrCore8}} it doesn't show up at
> all.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-15180) SearchHandler could prune uniqueKey queries
[
https://issues.apache.org/jira/browse/SOLR-15180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mike Drob updated SOLR-15180:
-
Description:
When doing a query that includes an {{id}}, we should be able to prune which
shards need to be distributed to.
This can be done explicitly by the user with {{_route\_}}, but I think Solr
should be able to automatically infer that value in some cases.
Other wish list items: we can potentially apply this to filter queries as well,
or to aliases.
The real-time-get handler already does intelligent routing to query only the
relevant shards, hopefully that logic can be re-used.
was:
When doing a query that includes an {{id}}, we should be able to prune which
shards need to be distributed to.
This can be done explicitly by the user with {{_route_}}, but I think Solr
should be able to automatically infer that value in some cases.
Other wish list items: we can potentially apply this to filter queries as well,
or to aliases.
The real-time-get handler already does intelligent routing to query only the
relevant shards, hopefully that logic can be re-used.
> SearchHandler could prune uniqueKey queries
> ---
>
> Key: SOLR-15180
> URL: https://issues.apache.org/jira/browse/SOLR-15180
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: search
>Reporter: Mike Drob
>Priority: Major
>
> When doing a query that includes an {{id}}, we should be able to prune which
> shards need to be distributed to.
> This can be done explicitly by the user with {{_route\_}}, but I think Solr
> should be able to automatically infer that value in some cases.
> Other wish list items: we can potentially apply this to filter queries as
> well, or to aliases.
> The real-time-get handler already does intelligent routing to query only the
> relevant shards, hopefully that logic can be re-used.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-15180) SearchHandler could prune uniqueKey queries
Mike Drob created SOLR-15180:
Summary: SearchHandler could prune uniqueKey queries
Key: SOLR-15180
URL: https://issues.apache.org/jira/browse/SOLR-15180
Project: Solr
Issue Type: Improvement
Security Level: Public (Default Security Level. Issues are Public)
Components: search
Reporter: Mike Drob
When doing a query that includes an {{id}}, we should be able to prune which
shards need to be distributed to.
This can be done explicitly by the user with {{_route_}}, but I think Solr
should be able to automatically infer that value in some cases.
Other wish list items: we can potentially apply this to filter queries as well,
or to aliases.
The real-time-get handler already does intelligent routing to query only the
relevant shards, hopefully that logic can be re-used.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-15135) Admin UI does not accurately reflect node PerReplicaStates status
[ https://issues.apache.org/jira/browse/SOLR-15135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17285317#comment-17285317 ] Mike Drob commented on SOLR-15135: -- [~ichattopadhyaya] [~thelabdude] Should this block a 8.8.1? > Admin UI does not accurately reflect node PerReplicaStates status > - > > Key: SOLR-15135 > URL: https://issues.apache.org/jira/browse/SOLR-15135 > Project: Solr > Issue Type: Bug > Components: Admin UI >Affects Versions: 8.8 >Reporter: Mike Drob >Priority: Blocker > > When creating collections using the new PRS, the admin UI will show them as > down because it only looks at state.json and not the correct PRS node. > cc: [~noble] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-15138) PerReplicaStates does not scale to large collections as well as state.json
[ https://issues.apache.org/jira/browse/SOLR-15138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284973#comment-17284973 ] Mike Drob commented on SOLR-15138: -- I've done some performance testing on master, here are comparisons on the same 4 node cluster after a sufficient warm up period. Small collections - 10 clients creating 10 collections in parallel, each collection 2x2 || ||Default||PRS|| |Median|4806ms|4779ms| |95%|6057ms|5434ms| Large collections - 1 client creating 10x10 sized collections || ||Default||PRS|| |Median|8972ms|13956ms| |95%|10294ms|17393ms| I didn't have a chance to take heap measurements or anything much more detailed here, but my observations are that while PRS is _slightly_ better for lots of small collections, looks like about 10% faster on the tails, and a fair bit worse for large collections overall, maybe 50-70% slower. I then sized my cluster up to 8 nodes, the small collection test performed almost identically (not sure if this is a good sign or not). The large collection test gave me these numbers: || ||Default||PRS|| |Median|8882ms|9016ms| |95%|8903ms|9246ms| A lot less impact from the slowest requests on the larger cluster, but PRS still does worse than the consolidated state.json implementation. Note that this was with the Distributed Overseer change in the code, but I did not have that enabled for this cluster. > PerReplicaStates does not scale to large collections as well as state.json > -- > > Key: SOLR-15138 > URL: https://issues.apache.org/jira/browse/SOLR-15138 > Project: Solr > Issue Type: Bug > Components: SolrCloud >Affects Versions: 8.8 >Reporter: Mike Drob >Assignee: Noble Paul >Priority: Major > Fix For: 8.9 > > Time Spent: 3h 20m > Remaining Estimate: 0h > > I was testing PRS collection creation with larger collections today > (previously I had tested with many small collections) and it seemed to be > having trouble keeping up. > > I was running a 4 node instance, each JVM with 4G Heap in k8s, and a single > zookeeper. > > With this cluster configuration, I am able to create several (at least 10) > collections with 11 shards and 11 replicas using the "old way" of keeping > state. These collections are created serially, waiting for all replicas to be > active before proceeding. > However, when attempting to do the same with PRS, the creation stalls on > collection 2 or 3, with several replicas stuck in a "down" state. Further, > when attempting to delete these collections using the regular API it > sometimes takes several attempts after getting stuck a few times as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-15153) Collection selector drop down does not sort collections
[
https://issues.apache.org/jira/browse/SOLR-15153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284852#comment-17284852
]
Mike Drob commented on SOLR-15153:
--
They're also not sorted at {{solr/#/~collections}} page, in the same order as
in the dropdown, so I imagine these are populated from the same place.
> Collection selector drop down does not sort collections
> ---
>
> Key: SOLR-15153
> URL: https://issues.apache.org/jira/browse/SOLR-15153
> Project: Solr
> Issue Type: Bug
> Components: Admin UI
>Reporter: Mike Drob
>Priority: Major
> Labels: newdev
>
> The collections selector drop down on the admin UI does not sort collections,
> making it harder to find the one that you care about.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-15153) Collection selector drop down does not sort collections
Mike Drob created SOLR-15153: Summary: Collection selector drop down does not sort collections Key: SOLR-15153 URL: https://issues.apache.org/jira/browse/SOLR-15153 Project: Solr Issue Type: Bug Security Level: Public (Default Security Level. Issues are Public) Components: Admin UI Reporter: Mike Drob The collections selector drop down on the admin UI does not sort collections, making it harder to find the one that you care about. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-15138) PerReplicaStates does not scale to large collections as well as state.json
[ https://issues.apache.org/jira/browse/SOLR-15138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17282042#comment-17282042 ] Mike Drob commented on SOLR-15138: -- For the shards which did not come up, I noticed that not all replicas had registered for leader election, and that there was no leader present for that shard. Maybe our creations timeouts need to take into account leaderVoteWait? > PerReplicaStates does not scale to large collections as well as state.json > -- > > Key: SOLR-15138 > URL: https://issues.apache.org/jira/browse/SOLR-15138 > Project: Solr > Issue Type: Bug > Components: SolrCloud >Affects Versions: 8.8 >Reporter: Mike Drob >Assignee: Noble Paul >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > I was testing PRS collection creation with larger collections today > (previously I had tested with many small collections) and it seemed to be > having trouble keeping up. > > I was running a 4 node instance, each JVM with 4G Heap in k8s, and a single > zookeeper. > > With this cluster configuration, I am able to create several (at least 10) > collections with 11 shards and 11 replicas using the "old way" of keeping > state. These collections are created serially, waiting for all replicas to be > active before proceeding. > However, when attempting to do the same with PRS, the creation stalls on > collection 2 or 3, with several replicas stuck in a "down" state. Further, > when attempting to delete these collections using the regular API it > sometimes takes several attempts after getting stuck a few times as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-15138) PerReplicaStates does not scale to large collections as well as state.json
[ https://issues.apache.org/jira/browse/SOLR-15138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17282040#comment-17282040 ] Mike Drob commented on SOLR-15138: -- This patch is an improvement over what we had previously, but I don't think it takes care of the situation completely. I was able to create 5 collections in my cluster, but #6 timed out. Although interestingly, #7 created just fine. Maybe there's a race condition somewhere then, if it's not related to the amount of existing or outstanding watches when subsequent collections continue to create. > PerReplicaStates does not scale to large collections as well as state.json > -- > > Key: SOLR-15138 > URL: https://issues.apache.org/jira/browse/SOLR-15138 > Project: Solr > Issue Type: Bug > Components: SolrCloud >Affects Versions: 8.8 >Reporter: Mike Drob >Assignee: Noble Paul >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > I was testing PRS collection creation with larger collections today > (previously I had tested with many small collections) and it seemed to be > having trouble keeping up. > > I was running a 4 node instance, each JVM with 4G Heap in k8s, and a single > zookeeper. > > With this cluster configuration, I am able to create several (at least 10) > collections with 11 shards and 11 replicas using the "old way" of keeping > state. These collections are created serially, waiting for all replicas to be > active before proceeding. > However, when attempting to do the same with PRS, the creation stalls on > collection 2 or 3, with several replicas stuck in a "down" state. Further, > when attempting to delete these collections using the regular API it > sometimes takes several attempts after getting stuck a few times as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-15138) PerReplicaStates does not scale to large collections as well as state.json
[
https://issues.apache.org/jira/browse/SOLR-15138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17281996#comment-17281996
]
Mike Drob commented on SOLR-15138:
--
Added a couple comments where I agreed with Ilan's review.
More generally, I don't know if that is the right place to be blocking? Why not
in AddReplicaCmd where we already have a check for user provided
\{{waitForFinalState}} parameter. Similarly, do we need to consider PRS state
in MoveReplicaCmd (or maybe other places?)
> PerReplicaStates does not scale to large collections as well as state.json
> --
>
> Key: SOLR-15138
> URL: https://issues.apache.org/jira/browse/SOLR-15138
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
>Affects Versions: 8.8
>Reporter: Mike Drob
>Assignee: Noble Paul
>Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
> I was testing PRS collection creation with larger collections today
> (previously I had tested with many small collections) and it seemed to be
> having trouble keeping up.
>
> I was running a 4 node instance, each JVM with 4G Heap in k8s, and a single
> zookeeper.
>
> With this cluster configuration, I am able to create several (at least 10)
> collections with 11 shards and 11 replicas using the "old way" of keeping
> state. These collections are created serially, waiting for all replicas to be
> active before proceeding.
> However, when attempting to do the same with PRS, the creation stalls on
> collection 2 or 3, with several replicas stuck in a "down" state. Further,
> when attempting to delete these collections using the regular API it
> sometimes takes several attempts after getting stuck a few times as well.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-15133) Document how to eliminate Failed to reserve shared memory warning
[
https://issues.apache.org/jira/browse/SOLR-15133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17281872#comment-17281872
]
Mike Drob commented on SOLR-15133:
--
I think [https://shipilev.net/jvm/anatomy-quarks/2-transparent-huge-pages/] is
a better explainer of what is going on, including the difference between
{{UseLargePages}} and {{UseTransparentHugePages}}.
Keeping this enabled for heaps beyond 1G (which is most Solr heaps IME),
appears to be beneficial, when the system supports it.
Now for the wrinkle... I believe both hugetlbfs and THP are reliant on kernel
settings/parameters, and docker images don't have kernels themselves. MacOS
doesn't support Large Pages
([https://bugs.openjdk.java.net/browse/JDK-8233062)] which suggests that
processes running in Docker for Mac wouldn't either. I don't know if this holds
true for Windows/Linux as well, or if the docker engines there are able to
delegate that memory management request.
> Document how to eliminate Failed to reserve shared memory warning
> -
>
> Key: SOLR-15133
> URL: https://issues.apache.org/jira/browse/SOLR-15133
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Docker, documentation
>Affects Versions: 8.7
>Reporter: David Eric Pugh
>Assignee: David Eric Pugh
>Priority: Minor
> Fix For: master (9.0)
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> inspired by a conversation on
> [https://github.com/docker-solr/docker-solr/issues/273,] it would be good to
> document how to get rid of shared memory warning in Docker setups.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-15133) Document how to eliminate Failed to reserve shared memory warning
[
https://issues.apache.org/jira/browse/SOLR-15133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17280033#comment-17280033
]
Mike Drob commented on SOLR-15133:
--
The origin seems to be from [~elyograg]'s wiki page, where he comments that:
{quote}A note about the UseLargePages option: Currently I do not have huge
pages allocated in my operating system. This option will not actually do
anything unless you allocate memory to huge pages. If you do so, memory usage
reporting with the “top” command will probably only show a few hundred MB of
resident memory used by your Solr process, even if it is in fact using several
gigabytes of heap. If you do enable huge pages in Linux, be aware that you
might wish to turn off an operating system feature called transparent huge
pages.{quote}
I don't understand enough of this quote to assert under what conditions large
pages are useful. Perhaps we can attempt to auto-detect the conditions based on
1) Specified Java Heap Size 2) Underlying system support and selectively enable
the flag then?
> Document how to eliminate Failed to reserve shared memory warning
> -
>
> Key: SOLR-15133
> URL: https://issues.apache.org/jira/browse/SOLR-15133
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Docker, documentation
>Affects Versions: 8.7
>Reporter: David Eric Pugh
>Assignee: David Eric Pugh
>Priority: Minor
> Fix For: master (9.0)
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> inspired by a conversation on
> [https://github.com/docker-solr/docker-solr/issues/273,] it would be good to
> document how to get rid of shared memory warning in Docker setups.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-15138) PerReplicaStates does not scale to large collections as well as state.json
Mike Drob created SOLR-15138: Summary: PerReplicaStates does not scale to large collections as well as state.json Key: SOLR-15138 URL: https://issues.apache.org/jira/browse/SOLR-15138 Project: Solr Issue Type: Bug Components: SolrCloud Affects Versions: 8.8 Reporter: Mike Drob I was testing PRS collection creation with larger collections today (previously I had tested with many small collections) and it seemed to be having trouble keeping up. I was running a 4 node instance, each JVM with 4G Heap in k8s, and a single zookeeper. With this cluster configuration, I am able to create several (at least 10) collections with 11 shards and 11 replicas using the "old way" of keeping state. These collections are created serially, waiting for all replicas to be active before proceeding. However, when attempting to do the same with PRS, the creation stalls on collection 2 or 3, with several replicas stuck in a "down" state. Further, when attempting to delete these collections using the regular API it sometimes takes several attempts after getting stuck a few times as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-15138) PerReplicaStates does not scale to large collections as well as state.json
[ https://issues.apache.org/jira/browse/SOLR-15138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17279234#comment-17279234 ] Mike Drob commented on SOLR-15138: -- cc: [~noble.paul] [~ichattopadhyaya] > PerReplicaStates does not scale to large collections as well as state.json > -- > > Key: SOLR-15138 > URL: https://issues.apache.org/jira/browse/SOLR-15138 > Project: Solr > Issue Type: Bug > Components: SolrCloud >Affects Versions: 8.8 >Reporter: Mike Drob >Priority: Major > > I was testing PRS collection creation with larger collections today > (previously I had tested with many small collections) and it seemed to be > having trouble keeping up. > > I was running a 4 node instance, each JVM with 4G Heap in k8s, and a single > zookeeper. > > With this cluster configuration, I am able to create several (at least 10) > collections with 11 shards and 11 replicas using the "old way" of keeping > state. These collections are created serially, waiting for all replicas to be > active before proceeding. > However, when attempting to do the same with PRS, the creation stalls on > collection 2 or 3, with several replicas stuck in a "down" state. Further, > when attempting to delete these collections using the regular API it > sometimes takes several attempts after getting stuck a few times as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-15137) Collection Overview Page could have shard replica info inline
Mike Drob created SOLR-15137: Summary: Collection Overview Page could have shard replica info inline Key: SOLR-15137 URL: https://issues.apache.org/jira/browse/SOLR-15137 Project: Solr Issue Type: Improvement Security Level: Public (Default Security Level. Issues are Public) Components: Admin UI Reporter: Mike Drob We could have some icons and indicators of various shard states on the collection overview page so that administrators don't have to expand each entry to see which one is the leader and which ones are down. This is possibly duplicative with the graph page? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-15135) Admin UI does not accurately reflect node PerReplicaStates status
[
https://issues.apache.org/jira/browse/SOLR-15135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17279004#comment-17279004
]
Mike Drob edited comment on SOLR-15135 at 2/4/21, 5:48 PM:
---
This is specific to the {{cloud?view=graph}} page.
{{Looking at ~/collections/}} and {{collection-overview}} show the correct
state. But many operators are used to looking at graph view for a quick
overview to find issues.
was (Author: mdrob):
This is specific to the {{/~cloud?view=graph}} page, {{~/collections/}} and
{{collection-overview}} show the correct state. But many operators are used to
looking at graph view for a quick overview to find issues.
> Admin UI does not accurately reflect node PerReplicaStates status
> -
>
> Key: SOLR-15135
> URL: https://issues.apache.org/jira/browse/SOLR-15135
> Project: Solr
> Issue Type: Bug
> Components: Admin UI
>Affects Versions: 8.8
>Reporter: Mike Drob
>Priority: Blocker
>
> When creating collections using the new PRS, the admin UI will show them as
> down because it only looks at state.json and not the correct PRS node.
> cc: [~noble]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-15135) Admin UI does not accurately reflect node PerReplicaStates status
[
https://issues.apache.org/jira/browse/SOLR-15135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17279004#comment-17279004
]
Mike Drob commented on SOLR-15135:
--
This is specific to the {{/~cloud?view=graph}} page, {{~/collections/}} and
{{collection-overview}} show the correct state. But many operators are used to
looking at graph view for a quick overview to find issues.
> Admin UI does not accurately reflect node PerReplicaStates status
> -
>
> Key: SOLR-15135
> URL: https://issues.apache.org/jira/browse/SOLR-15135
> Project: Solr
> Issue Type: Bug
> Components: Admin UI
>Affects Versions: 8.8
>Reporter: Mike Drob
>Priority: Blocker
>
> When creating collections using the new PRS, the admin UI will show them as
> down because it only looks at state.json and not the correct PRS node.
> cc: [~noble]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-15135) Admin UI does not accurately reflect node PerReplicaStates status
Mike Drob created SOLR-15135: Summary: Admin UI does not accurately reflect node PerReplicaStates status Key: SOLR-15135 URL: https://issues.apache.org/jira/browse/SOLR-15135 Project: Solr Issue Type: Bug Components: Admin UI Affects Versions: 8.8 Reporter: Mike Drob When creating collections using the new PRS, the admin UI will show them as down because it only looks at state.json and not the correct PRS node. cc: [~noble] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14928) Remove Overseer ClusterStateUpdater
[
https://issues.apache.org/jira/browse/SOLR-14928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17278475#comment-17278475
]
Mike Drob commented on SOLR-14928:
--
[~ilan] - those performance numbers are against a non PRS version, right? Can
you run a similar test with PRS and share the results?
> Remove Overseer ClusterStateUpdater
> ---
>
> Key: SOLR-14928
> URL: https://issues.apache.org/jira/browse/SOLR-14928
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: Ilan Ginzburg
>Assignee: Ilan Ginzburg
>Priority: Major
> Labels: cluster, collection-api, overseer
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Remove the Overseer {{ClusterStateUpdater}} thread and associated Zookeeper
> queue at {{<_chroot_>/overseer/queue}}.
> Change cluster state updates so that each (Collection API) command execution
> does the update directly in Zookeeper using optimistic locking (Compare and
> Swap on the {{state.json}} Zookeeper files).
> Following this change cluster state updates would still be happening only
> from the Overseer node (that's where Collection API commands are executing),
> but the code will be ready for distribution once such commands can be
> executed by any node (other work done in the context of parent task
> SOLR-14927).
> See the [Cluster State
> Updater|https://docs.google.com/document/d/1u4QHsIHuIxlglIW6hekYlXGNOP0HjLGVX5N6inkj6Ok/edit#heading=h.ymtfm3p518c]
> section in the Removing Overseer doc.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-15122) ClusterEventProducerTest.testEvents is unstable
[ https://issues.apache.org/jira/browse/SOLR-15122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17278465#comment-17278465 ] Mike Drob commented on SOLR-15122: -- Thanks to [~ab] for identifying where he suspected the issue was, and to [~ilan] for the thorough reviews. I'll follow AB's model here and continue to leave this open for another few days until we can confirm the fix. > ClusterEventProducerTest.testEvents is unstable > --- > > Key: SOLR-15122 > URL: https://issues.apache.org/jira/browse/SOLR-15122 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: Tests >Reporter: Mike Drob >Assignee: Andrzej Bialecki >Priority: Major > Fix For: master (9.0) > > Time Spent: 3.5h > Remaining Estimate: 0h > > This test looks to be unstable according to Jenkins since about Nov 5. I just > started seeing occasional failures locally when running the whole suite but > cannot reproduce when running in isolation. > https://lists.apache.org/thread.html/rf0c16b257bc3236ea414be51451806352b55f15d4949f4fd54a3b71a%40%3Cbuilds.lucene.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (SOLR-14253) Remove Sleeps from OverseerCollectionMessageHandler
[ https://issues.apache.org/jira/browse/SOLR-14253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Drob resolved SOLR-14253. -- Resolution: Fixed > Remove Sleeps from OverseerCollectionMessageHandler > --- > > Key: SOLR-14253 > URL: https://issues.apache.org/jira/browse/SOLR-14253 > Project: Solr > Issue Type: Bug > Components: Server >Reporter: Mike Drob >Assignee: Mike Drob >Priority: Major > Fix For: master (9.0) > > Time Spent: 3h 50m > Remaining Estimate: 0h > > From the conversations with Mark Miller a few months back - there are a lot > of places in the server code where we have hard sleeps instead of relying on > notifications and watchers to handle state. > I will begin to tackle these one at a time, starting with > OverseerCollectionMessageHandler. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-15122) ClusterEventProducerTest.testEvents is unstable
[ https://issues.apache.org/jira/browse/SOLR-15122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17278333#comment-17278333 ] Mike Drob commented on SOLR-15122: -- If we're not building our own concurrency logic, then I think it's fine to expose directly on the delegate classes. I'll add docs about the expected usage. > ClusterEventProducerTest.testEvents is unstable > --- > > Key: SOLR-15122 > URL: https://issues.apache.org/jira/browse/SOLR-15122 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: Tests >Reporter: Mike Drob >Assignee: Andrzej Bialecki >Priority: Major > Fix For: master (9.0) > > Time Spent: 2.5h > Remaining Estimate: 0h > > This test looks to be unstable according to Jenkins since about Nov 5. I just > started seeing occasional failures locally when running the whole suite but > cannot reproduce when running in isolation. > https://lists.apache.org/thread.html/rf0c16b257bc3236ea414be51451806352b55f15d4949f4fd54a3b71a%40%3Cbuilds.lucene.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Reopened] (SOLR-14253) Remove Sleeps from OverseerCollectionMessageHandler
[ https://issues.apache.org/jira/browse/SOLR-14253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Drob reopened SOLR-14253: -- Reopening for a test failure found by [~gerlowskija], to be addressed by the new PR. > Remove Sleeps from OverseerCollectionMessageHandler > --- > > Key: SOLR-14253 > URL: https://issues.apache.org/jira/browse/SOLR-14253 > Project: Solr > Issue Type: Bug > Components: Server >Reporter: Mike Drob >Assignee: Mike Drob >Priority: Major > Fix For: master (9.0) > > Time Spent: 3h 40m > Remaining Estimate: 0h > > From the conversations with Mark Miller a few months back - there are a lot > of places in the server code where we have hard sleeps instead of relying on > notifications and watchers to handle state. > I will begin to tackle these one at a time, starting with > OverseerCollectionMessageHandler. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-15122) ClusterEventProducerTest.testEvents is unstable
[ https://issues.apache.org/jira/browse/SOLR-15122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17277519#comment-17277519 ] Mike Drob commented on SOLR-15122: -- I was thinking about this some more, and think we should use a Phaser instead of rolling our own concurrency implementation. Thoughts? > ClusterEventProducerTest.testEvents is unstable > --- > > Key: SOLR-15122 > URL: https://issues.apache.org/jira/browse/SOLR-15122 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: Tests >Reporter: Mike Drob >Assignee: Andrzej Bialecki >Priority: Major > Fix For: master (9.0) > > Time Spent: 1h 40m > Remaining Estimate: 0h > > This test looks to be unstable according to Jenkins since about Nov 5. I just > started seeing occasional failures locally when running the whole suite but > cannot reproduce when running in isolation. > https://lists.apache.org/thread.html/rf0c16b257bc3236ea414be51451806352b55f15d4949f4fd54a3b71a%40%3Cbuilds.lucene.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-15122) ClusterEventProducerTest.testEvents is unstable
[ https://issues.apache.org/jira/browse/SOLR-15122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17277423#comment-17277423 ] Mike Drob commented on SOLR-15122: -- Refactored the code in anticipation of working over DelegatingPlacementPluginFactory as well, moved the wait logic to test sources with an int and an Object instead of Atomic Integer. Please take a look before I consolidate the other implementation > ClusterEventProducerTest.testEvents is unstable > --- > > Key: SOLR-15122 > URL: https://issues.apache.org/jira/browse/SOLR-15122 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: Tests >Reporter: Mike Drob >Assignee: Andrzej Bialecki >Priority: Major > Fix For: master (9.0) > > Time Spent: 50m > Remaining Estimate: 0h > > This test looks to be unstable according to Jenkins since about Nov 5. I just > started seeing occasional failures locally when running the whole suite but > cannot reproduce when running in isolation. > https://lists.apache.org/thread.html/rf0c16b257bc3236ea414be51451806352b55f15d4949f4fd54a3b71a%40%3Cbuilds.lucene.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-15122) ClusterEventProducerTest.testEvents is unstable
[ https://issues.apache.org/jira/browse/SOLR-15122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17277399#comment-17277399 ] Mike Drob commented on SOLR-15122: -- I'm pretty comfortable with that blanket statement. If you're using volatile, that means you expect multiple threads accessing, and if you have multiple threads writing then you shouldn't be using non-atomic postfix increment. If you can guarantee that you only have one writer, and the volatile is for the readers, then... maybe it's ok? It's still trappy and can lead to issues down the line. I used AtomicInteger because we don't have a mutable Integer and needed an object anyway for the sync block. I can rewrite that with an Object and an int if you prefer. > ClusterEventProducerTest.testEvents is unstable > --- > > Key: SOLR-15122 > URL: https://issues.apache.org/jira/browse/SOLR-15122 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: Tests >Reporter: Mike Drob >Assignee: Andrzej Bialecki >Priority: Major > Fix For: master (9.0) > > Time Spent: 40m > Remaining Estimate: 0h > > This test looks to be unstable according to Jenkins since about Nov 5. I just > started seeing occasional failures locally when running the whole suite but > cannot reproduce when running in isolation. > https://lists.apache.org/thread.html/rf0c16b257bc3236ea414be51451806352b55f15d4949f4fd54a3b71a%40%3Cbuilds.lucene.apache.org%3E -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
