Re: Lucene 9.4.0 release
> One thing is that for JIra, we needed to acquire special permissions in order to be able to add new release versions, but I don't think that is true for GitHub ... You do not need any special permissions to make new Milestones on GitHub. Every committer already has permission to create/close/delete Milestones, you can test it here. https://github.com/apache/lucene/milestones Tomoko 2022年9月1日(木) 9:57 Michael Sokolov : > I started to go through the release process, which has been scripted very > clearly by Jan and others, and found Jira-specific steps there. I think the > translations to GitHub will be pretty straightforward. One thing is that > for JIra, we needed to acquire special permissions in order to be able to > add new release versions, but I don't think that is true for GitHub ... Any > committer should be able to do that, right? > > On Wed, Aug 31, 2022, 8:42 PM Tomoko Uchida > wrote: > >> Hi Mike, >> thanks for your response. >> >> I'm actually not familiar with the whole release procedure that may be >> built on Jira. >> I hope other people help to interpret Jira-related things on the way into >> the language of GitHub issues. >> >> Tomoko >> >> 2022年9月1日(木) 3:40 Michael Sokolov : >> >>> Thanks for the links, Tomoko. I thought it would be helpful to ask on >>> the list in case this new system has not been fully internalized by >>> all the developers. And I confess I didn't know about that nice >>> milestone view! It makes this very easy, assuming of course that >>> everyone follows the guidelines and labels the issues with the >>> milestone. >>> >>> On Wed, Aug 31, 2022 at 11:16 AM Tomoko Uchida >>> wrote: >>> > >>> > Ok in short, you can find blockers for 9.4.0 here. >>> > https://github.com/apache/lucene/milestone/3 >>> > >>> > If there are other issues/PRs that are still not resolved but planned >>> to be delivered in 9.4.0, just set their Milestone as 9.4.0. Then they will >>> appear in the list. >>> > >>> > >>> > >>> > 2022年9月1日(木) 0:00 Tomoko Uchida : >>> >> >>> >> I believe we decided to use GitHub Milestone for release planning and >>> I wrote documentation about it. >>> >> >>> https://github.com/apache/lucene/blob/main/dev-docs/github-issues-howto.md >>> >> >>> >> Is this unclear to you? >>> >> >>> >> >>> >> 2022年8月31日(水) 23:13 Michael Sokolov : >>> >>> >>> >>> Hi, I'd like to start the ball rolling for a 9.4.0 release. We don't >>> >>> have a large number of things, but there are a few nice high-impact >>> >>> changes; see excerpt from CHANGE.txt below. I'll volunteer to be >>> >>> release manager. I'd just like to know if anybody has any outstanding >>> >>> work that is nearing completion and that they'd like to include, >>> >>> and/or if we have any known blocker issues*. >>> >>> >>> >>> * -- I am having a little trouble searching github for blocker >>> issues. >>> >>> I think there may not be any, but I tried >>> >>> >>> https://github.com/apache/lucene/issues?q=is%3Aissue+is%3Aopen+legacy-jira-priority%5C%3AMajor+ >>> >>> to find Major issues, for example, and it seems to only find one >>> Minor >>> >>> one? Does anyone have better github-search-fu? >>> >>> >>> >>> API Changes >>> >>> - >>> >>> * LUCENE-10577: Add VectorEncoding to enable byte-encoded HNSW >>> vectors >>> >>> (Michael Sokolov, Julie Tibshirani) >>> >>> >>> >>> New Features >>> >>> - >>> >>> * LUCENE-10654: Add new ShapeDocValuesField for LatLonShape and >>> >>> XYShape. (Nick Knize) >>> >>> >>> >>> * LUCENE-10629: Support match set filtering with a query in >>> >>> MatchingFacetSetCounts. (Stefan Vodita, Shai Erera) >>> >>> >>> >>> * LUCENE-10633: SortField#setOptimizeSortWithIndexedData and >>> >>> SortField#getOptimizeSortWithIndexedData were introduce to provide >>> >>> an option to disable sort optimization for various sort fields. >>> >>> (Mayya Sharipova) >>> >>> >>> >>> Improvements >>> >>> - >>> >>> * LUCENE-10592: Build HNSW Graph on indexing. (Mayya Sharipova, >>> Adrien >>> >>> Grand, Julie Tibshirani) >>> >>> >>> >>> * GITHUB#11715: Add Integer awareness to RamUsageEstimator.sizeOf >>> (Mike Drob) >>> >>> >>> >>> Optimizations >>> >>> - >>> >>> >>> >>> * LUCENE-10661: Reduce memory copy in BytesStore. (luyuncheng) >>> >>> >>> >>> * GITHUB#1020: Support #scoreSupplier and small optimizations to >>> >>> DocValuesRewriteMethod. (Greg Miller) >>> >>> >>> >>> * LUCENE-10633: Added support for dynamic pruning to queries sorted >>> by a string >>> >>> field that is indexed with terms and SORTED or SORTED_SET doc >>> values. >>> >>> (Adrien Grand) >>> >>> >>> >>> * LUCENE-10627: Using ByteBuffersDataInput reduce memory copy on >>> >>> compressing data. (luyuncheng) >>> >>> >>> >>> * GITHUB#1062: Optimize TermInSetQuery when a term is present that >>> >>> matches all docs in a segment. >>> >>> (Greg Miller) >>> >>> >>> >>> - >>> >>> To
Re: Lucene 9.4.0 release
I started to go through the release process, which has been scripted very clearly by Jan and others, and found Jira-specific steps there. I think the translations to GitHub will be pretty straightforward. One thing is that for JIra, we needed to acquire special permissions in order to be able to add new release versions, but I don't think that is true for GitHub ... Any committer should be able to do that, right? On Wed, Aug 31, 2022, 8:42 PM Tomoko Uchida wrote: > Hi Mike, > thanks for your response. > > I'm actually not familiar with the whole release procedure that may be > built on Jira. > I hope other people help to interpret Jira-related things on the way into > the language of GitHub issues. > > Tomoko > > 2022年9月1日(木) 3:40 Michael Sokolov : > >> Thanks for the links, Tomoko. I thought it would be helpful to ask on >> the list in case this new system has not been fully internalized by >> all the developers. And I confess I didn't know about that nice >> milestone view! It makes this very easy, assuming of course that >> everyone follows the guidelines and labels the issues with the >> milestone. >> >> On Wed, Aug 31, 2022 at 11:16 AM Tomoko Uchida >> wrote: >> > >> > Ok in short, you can find blockers for 9.4.0 here. >> > https://github.com/apache/lucene/milestone/3 >> > >> > If there are other issues/PRs that are still not resolved but planned >> to be delivered in 9.4.0, just set their Milestone as 9.4.0. Then they will >> appear in the list. >> > >> > >> > >> > 2022年9月1日(木) 0:00 Tomoko Uchida : >> >> >> >> I believe we decided to use GitHub Milestone for release planning and >> I wrote documentation about it. >> >> >> https://github.com/apache/lucene/blob/main/dev-docs/github-issues-howto.md >> >> >> >> Is this unclear to you? >> >> >> >> >> >> 2022年8月31日(水) 23:13 Michael Sokolov : >> >>> >> >>> Hi, I'd like to start the ball rolling for a 9.4.0 release. We don't >> >>> have a large number of things, but there are a few nice high-impact >> >>> changes; see excerpt from CHANGE.txt below. I'll volunteer to be >> >>> release manager. I'd just like to know if anybody has any outstanding >> >>> work that is nearing completion and that they'd like to include, >> >>> and/or if we have any known blocker issues*. >> >>> >> >>> * -- I am having a little trouble searching github for blocker issues. >> >>> I think there may not be any, but I tried >> >>> >> https://github.com/apache/lucene/issues?q=is%3Aissue+is%3Aopen+legacy-jira-priority%5C%3AMajor+ >> >>> to find Major issues, for example, and it seems to only find one Minor >> >>> one? Does anyone have better github-search-fu? >> >>> >> >>> API Changes >> >>> - >> >>> * LUCENE-10577: Add VectorEncoding to enable byte-encoded HNSW vectors >> >>> (Michael Sokolov, Julie Tibshirani) >> >>> >> >>> New Features >> >>> - >> >>> * LUCENE-10654: Add new ShapeDocValuesField for LatLonShape and >> >>> XYShape. (Nick Knize) >> >>> >> >>> * LUCENE-10629: Support match set filtering with a query in >> >>> MatchingFacetSetCounts. (Stefan Vodita, Shai Erera) >> >>> >> >>> * LUCENE-10633: SortField#setOptimizeSortWithIndexedData and >> >>> SortField#getOptimizeSortWithIndexedData were introduce to provide >> >>> an option to disable sort optimization for various sort fields. >> >>> (Mayya Sharipova) >> >>> >> >>> Improvements >> >>> - >> >>> * LUCENE-10592: Build HNSW Graph on indexing. (Mayya Sharipova, Adrien >> >>> Grand, Julie Tibshirani) >> >>> >> >>> * GITHUB#11715: Add Integer awareness to RamUsageEstimator.sizeOf >> (Mike Drob) >> >>> >> >>> Optimizations >> >>> - >> >>> >> >>> * LUCENE-10661: Reduce memory copy in BytesStore. (luyuncheng) >> >>> >> >>> * GITHUB#1020: Support #scoreSupplier and small optimizations to >> >>> DocValuesRewriteMethod. (Greg Miller) >> >>> >> >>> * LUCENE-10633: Added support for dynamic pruning to queries sorted >> by a string >> >>> field that is indexed with terms and SORTED or SORTED_SET doc >> values. >> >>> (Adrien Grand) >> >>> >> >>> * LUCENE-10627: Using ByteBuffersDataInput reduce memory copy on >> >>> compressing data. (luyuncheng) >> >>> >> >>> * GITHUB#1062: Optimize TermInSetQuery when a term is present that >> >>> matches all docs in a segment. >> >>> (Greg Miller) >> >>> >> >>> - >> >>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >> >>> For additional commands, e-mail: dev-h...@lucene.apache.org >> >>> >> >> - >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >> For additional commands, e-mail: dev-h...@lucene.apache.org >> >>
Re: Lucene 9.4.0 release
Hi Mike, thanks for your response. I'm actually not familiar with the whole release procedure that may be built on Jira. I hope other people help to interpret Jira-related things on the way into the language of GitHub issues. Tomoko 2022年9月1日(木) 3:40 Michael Sokolov : > Thanks for the links, Tomoko. I thought it would be helpful to ask on > the list in case this new system has not been fully internalized by > all the developers. And I confess I didn't know about that nice > milestone view! It makes this very easy, assuming of course that > everyone follows the guidelines and labels the issues with the > milestone. > > On Wed, Aug 31, 2022 at 11:16 AM Tomoko Uchida > wrote: > > > > Ok in short, you can find blockers for 9.4.0 here. > > https://github.com/apache/lucene/milestone/3 > > > > If there are other issues/PRs that are still not resolved but planned to > be delivered in 9.4.0, just set their Milestone as 9.4.0. Then they will > appear in the list. > > > > > > > > 2022年9月1日(木) 0:00 Tomoko Uchida : > >> > >> I believe we decided to use GitHub Milestone for release planning and I > wrote documentation about it. > >> > https://github.com/apache/lucene/blob/main/dev-docs/github-issues-howto.md > >> > >> Is this unclear to you? > >> > >> > >> 2022年8月31日(水) 23:13 Michael Sokolov : > >>> > >>> Hi, I'd like to start the ball rolling for a 9.4.0 release. We don't > >>> have a large number of things, but there are a few nice high-impact > >>> changes; see excerpt from CHANGE.txt below. I'll volunteer to be > >>> release manager. I'd just like to know if anybody has any outstanding > >>> work that is nearing completion and that they'd like to include, > >>> and/or if we have any known blocker issues*. > >>> > >>> * -- I am having a little trouble searching github for blocker issues. > >>> I think there may not be any, but I tried > >>> > https://github.com/apache/lucene/issues?q=is%3Aissue+is%3Aopen+legacy-jira-priority%5C%3AMajor+ > >>> to find Major issues, for example, and it seems to only find one Minor > >>> one? Does anyone have better github-search-fu? > >>> > >>> API Changes > >>> - > >>> * LUCENE-10577: Add VectorEncoding to enable byte-encoded HNSW vectors > >>> (Michael Sokolov, Julie Tibshirani) > >>> > >>> New Features > >>> - > >>> * LUCENE-10654: Add new ShapeDocValuesField for LatLonShape and > >>> XYShape. (Nick Knize) > >>> > >>> * LUCENE-10629: Support match set filtering with a query in > >>> MatchingFacetSetCounts. (Stefan Vodita, Shai Erera) > >>> > >>> * LUCENE-10633: SortField#setOptimizeSortWithIndexedData and > >>> SortField#getOptimizeSortWithIndexedData were introduce to provide > >>> an option to disable sort optimization for various sort fields. > >>> (Mayya Sharipova) > >>> > >>> Improvements > >>> - > >>> * LUCENE-10592: Build HNSW Graph on indexing. (Mayya Sharipova, Adrien > >>> Grand, Julie Tibshirani) > >>> > >>> * GITHUB#11715: Add Integer awareness to RamUsageEstimator.sizeOf > (Mike Drob) > >>> > >>> Optimizations > >>> - > >>> > >>> * LUCENE-10661: Reduce memory copy in BytesStore. (luyuncheng) > >>> > >>> * GITHUB#1020: Support #scoreSupplier and small optimizations to > >>> DocValuesRewriteMethod. (Greg Miller) > >>> > >>> * LUCENE-10633: Added support for dynamic pruning to queries sorted by > a string > >>> field that is indexed with terms and SORTED or SORTED_SET doc values. > >>> (Adrien Grand) > >>> > >>> * LUCENE-10627: Using ByteBuffersDataInput reduce memory copy on > >>> compressing data. (luyuncheng) > >>> > >>> * GITHUB#1062: Optimize TermInSetQuery when a term is present that > >>> matches all docs in a segment. > >>> (Greg Miller) > >>> > >>> - > >>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > >>> For additional commands, e-mail: dev-h...@lucene.apache.org > >>> > > - > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > >
Re: [lucene] branch main updated: SimpleText knn vectors; fix searchExhaustively and suppress a byte format test case (#11725)
thanks for fixing! On Wed, Aug 31, 2022 at 2:43 PM Michael Sokolov wrote: > > Oh -- sorry, I guess I forgot to backport. Thanks for tracking it down > - I'll push to branch_9x shortly > > On Wed, Aug 31, 2022 at 10:25 AM Robert Muir wrote: > > > > can we backport to 9.x if you get a chance? I'm still seeing this test > > trip in 9.x jenkins builds. > > > > > > On Mon, Aug 29, 2022 at 11:50 AM wrote: > > > > > > This is an automated email from the ASF dual-hosted git repository. > > > > > > sokolov pushed a commit to branch main > > > in repository https://gitbox.apache.org/repos/asf/lucene.git > > > > > > > > > The following commit(s) were added to refs/heads/main by this push: > > > new 61ef031f7fa SimpleText knn vectors; fix searchExhaustively and > > > suppress a byte format test case (#11725) > > > 61ef031f7fa is described below > > > > > > commit 61ef031f7fa3abdd7c8c2f36db71ad2289b66131 > > > Author: Michael Sokolov > > > AuthorDate: Mon Aug 29 11:49:52 2022 -0400 > > > > > > SimpleText knn vectors; fix searchExhaustively and suppress a byte > > > format test case (#11725) > > > --- > > > .../apache/lucene/codecs/simpletext/SimpleTextKnnVectorsReader.java | 6 > > > +++--- > > > lucene/core/src/test/org/apache/lucene/document/TestField.java | 4 > > > > > > 2 files changed, 7 insertions(+), 3 deletions(-) > > > > > > diff --git > > > a/lucene/codecs/src/java/org/apache/lucene/codecs/simpletext/SimpleTextKnnVectorsReader.java > > > > > > b/lucene/codecs/src/java/org/apache/lucene/codecs/simpletext/SimpleTextKnnVectorsReader.java > > > index e4b0ceb5916..10700f5de6f 100644 > > > --- > > > a/lucene/codecs/src/java/org/apache/lucene/codecs/simpletext/SimpleTextKnnVectorsReader.java > > > +++ > > > b/lucene/codecs/src/java/org/apache/lucene/codecs/simpletext/SimpleTextKnnVectorsReader.java > > > @@ -41,7 +41,6 @@ import > > > org.apache.lucene.store.BufferedChecksumIndexInput; > > > import org.apache.lucene.store.ChecksumIndexInput; > > > import org.apache.lucene.store.IOContext; > > > import org.apache.lucene.store.IndexInput; > > > -import org.apache.lucene.util.BitSet; > > > import org.apache.lucene.util.Bits; > > > import org.apache.lucene.util.BytesRef; > > > import org.apache.lucene.util.BytesRefBuilder; > > > @@ -187,8 +186,9 @@ public class SimpleTextKnnVectorsReader extends > > > KnnVectorsReader { > > >@Override > > >public TopDocs searchExhaustively( > > >String field, float[] target, int k, DocIdSetIterator acceptDocs) > > > throws IOException { > > > -int numDocs = (int) acceptDocs.cost(); > > > -return search(field, target, k, BitSet.of(acceptDocs, numDocs), > > > Integer.MAX_VALUE); > > > +FieldInfo info = readState.fieldInfos.fieldInfo(field); > > > +VectorSimilarityFunction vectorSimilarity = > > > info.getVectorSimilarityFunction(); > > > +return exhaustiveSearch(getVectorValues(field), acceptDocs, > > > vectorSimilarity, target, k); > > >} > > > > > >@Override > > > diff --git > > > a/lucene/core/src/test/org/apache/lucene/document/TestField.java > > > b/lucene/core/src/test/org/apache/lucene/document/TestField.java > > > index 781f2b613c6..6aa5518f33b 100644 > > > --- a/lucene/core/src/test/org/apache/lucene/document/TestField.java > > > +++ b/lucene/core/src/test/org/apache/lucene/document/TestField.java > > > @@ -20,6 +20,7 @@ import static > > > org.apache.lucene.search.DocIdSetIterator.NO_MORE_DOCS; > > > > > > import java.io.StringReader; > > > import java.nio.charset.StandardCharsets; > > > +import org.apache.lucene.codecs.Codec; > > > import org.apache.lucene.index.DirectoryReader; > > > import org.apache.lucene.index.IndexReader; > > > import org.apache.lucene.index.IndexWriter; > > > @@ -513,6 +514,9 @@ public class TestField extends LuceneTestCase { > > >} > > > > > >public void testKnnVectorField() throws Exception { > > > +if (Codec.getDefault().getName().equals("SimpleText")) { > > > + return; > > > +} > > > try (Directory dir = newDirectory(); > > > IndexWriter w = new IndexWriter(dir, newIndexWriterConfig())) { > > >Document doc = new Document(); > > > > > > > - > > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > > For additional commands, e-mail: dev-h...@lucene.apache.org > > > > - > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: [JENKINS] Lucene » Lucene-Check-main - Build # 6584 - Failure!
maybe the OOMKiller kicked in. On Wed, Aug 31, 2022 at 3:06 PM Dawid Weiss wrote: > > > I think Lucene tests killed the job runner. :) > > > Task :lucene:analysis:nori:spotlessJavaCheck > > Task :lucene:analysis:nori:spotlessCheck > FATAL: command execution failed > java.io.IOException: Backing channel 'lucene-solr-1' is disconnected. > [...] > > ERROR: lucene-solr-1 is offline; cannot locate jdk_17_latest > ERROR: lucene-solr-1 is offline; cannot locate jdk_17_latest > ERROR: lucene-solr-1 is offline; cannot locate jdk_17_latest > ERROR: lucene-solr-1 is offline; cannot locate jdk_17_latest > ERROR: lucene-solr-1 is offline; cannot locate jdk_17_latest > > > > On Wed, Aug 31, 2022 at 8:15 PM Apache Jenkins Server > wrote: >> >> Build: https://ci-builds.apache.org/job/Lucene/job/Lucene-Check-main/6584/ >> >> No tests ran. >> >> Build Log: >> [...truncated 959 lines...] >> FATAL: command execution failed >> java.io.IOException: Backing channel 'lucene-solr-1' is disconnected. >> at >> hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:215) >> at >> hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:285) >> at com.sun.proxy.$Proxy179.isAlive(Unknown Source) >> at >> hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1215) >> at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1207) >> at hudson.Launcher$ProcStarter.join(Launcher.java:524) >> at hudson.plugins.gradle.Gradle.perform(Gradle.java:317) >> at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) >> at >> hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:814) >> at hudson.model.Build$BuildExecution.build(Build.java:199) >> at hudson.model.Build$BuildExecution.doRun(Build.java:164) >> at >> hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:522) >> at hudson.model.Run.execute(Run.java:1896) >> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44) >> at >> hudson.model.ResourceController.execute(ResourceController.java:101) >> at hudson.model.Executor.run(Executor.java:442) >> Caused by: java.io.IOException: Pipe closed after 0 cycles >> at >> org.apache.sshd.common.channel.ChannelPipedInputStream.read(ChannelPipedInputStream.java:126) >> at >> org.apache.sshd.common.channel.ChannelPipedInputStream.read(ChannelPipedInputStream.java:105) >> at >> hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:94) >> at >> hudson.remoting.ChunkedInputStream.readHeader(ChunkedInputStream.java:75) >> at >> hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:105) >> at >> hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39) >> at >> hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) >> at >> hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:61) >> Build step 'Invoke Gradle script' changed build result to FAILURE >> Build step 'Invoke Gradle script' marked build as failure >> ERROR: Step ‘Archive the artifacts’ failed: no workspace for >> Lucene/Lucene-Check-main #6584 >> ERROR: Step ‘Publish JUnit test result report’ failed: no workspace for >> Lucene/Lucene-Check-main #6584 >> ERROR: lucene-solr-1 is offline; cannot locate jdk_17_latest >> Email was triggered for: Failure - Any >> Sending email for trigger: Failure - Any >> ERROR: lucene-solr-1 is offline; cannot locate jdk_17_latest >> ERROR: lucene-solr-1 is offline; cannot locate jdk_17_latest >> ERROR: lucene-solr-1 is offline; cannot locate jdk_17_latest >> ERROR: lucene-solr-1 is offline; cannot locate jdk_17_latest >> >> - >> To unsubscribe, e-mail: builds-unsubscr...@lucene.apache.org >> For additional commands, e-mail: builds-h...@lucene.apache.org - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: [JENKINS] Lucene » Lucene-Check-main - Build # 6584 - Failure!
I think Lucene tests killed the job runner. :) > Task :lucene:analysis:nori:spotlessJavaCheck > Task :lucene:analysis:nori:spotlessCheck FATAL: command execution failed java.io.IOException: Backing channel 'lucene-solr-1' is disconnected. [...] ERROR: lucene-solr-1 is offline; cannot locate jdk_17_latest ERROR: lucene-solr-1 is offline; cannot locate jdk_17_latest ERROR: lucene-solr-1 is offline; cannot locate jdk_17_latest ERROR: lucene-solr-1 is offline; cannot locate jdk_17_latest ERROR: lucene-solr-1 is offline; cannot locate jdk_17_latest On Wed, Aug 31, 2022 at 8:15 PM Apache Jenkins Server < jenk...@builds.apache.org> wrote: > Build: https://ci-builds.apache.org/job/Lucene/job/Lucene-Check-main/6584/ > > No tests ran. > > Build Log: > [...truncated 959 lines...] > FATAL: command execution failed > java.io.IOException: Backing channel 'lucene-solr-1' is disconnected. > at > hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:215) > at > hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:285) > at com.sun.proxy.$Proxy179.isAlive(Unknown Source) > at > hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1215) > at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1207) > at hudson.Launcher$ProcStarter.join(Launcher.java:524) > at hudson.plugins.gradle.Gradle.perform(Gradle.java:317) > at > hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) > at > hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:814) > at hudson.model.Build$BuildExecution.build(Build.java:199) > at hudson.model.Build$BuildExecution.doRun(Build.java:164) > at > hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:522) > at hudson.model.Run.execute(Run.java:1896) > at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44) > at > hudson.model.ResourceController.execute(ResourceController.java:101) > at hudson.model.Executor.run(Executor.java:442) > Caused by: java.io.IOException: Pipe closed after 0 cycles > at > org.apache.sshd.common.channel.ChannelPipedInputStream.read(ChannelPipedInputStream.java:126) > at > org.apache.sshd.common.channel.ChannelPipedInputStream.read(ChannelPipedInputStream.java:105) > at > hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:94) > at > hudson.remoting.ChunkedInputStream.readHeader(ChunkedInputStream.java:75) > at > hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:105) > at > hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39) > at > hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) > at > hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:61) > Build step 'Invoke Gradle script' changed build result to FAILURE > Build step 'Invoke Gradle script' marked build as failure > ERROR: Step ‘Archive the artifacts’ failed: no workspace for > Lucene/Lucene-Check-main #6584 > ERROR: Step ‘Publish JUnit test result report’ failed: no workspace for > Lucene/Lucene-Check-main #6584 > ERROR: lucene-solr-1 is offline; cannot locate jdk_17_latest > Email was triggered for: Failure - Any > Sending email for trigger: Failure - Any > ERROR: lucene-solr-1 is offline; cannot locate jdk_17_latest > ERROR: lucene-solr-1 is offline; cannot locate jdk_17_latest > ERROR: lucene-solr-1 is offline; cannot locate jdk_17_latest > ERROR: lucene-solr-1 is offline; cannot locate jdk_17_latest > > - > To unsubscribe, e-mail: builds-unsubscr...@lucene.apache.org > For additional commands, e-mail: builds-h...@lucene.apache.org
Re: [lucene] branch main updated: SimpleText knn vectors; fix searchExhaustively and suppress a byte format test case (#11725)
Oh -- sorry, I guess I forgot to backport. Thanks for tracking it down - I'll push to branch_9x shortly On Wed, Aug 31, 2022 at 10:25 AM Robert Muir wrote: > > can we backport to 9.x if you get a chance? I'm still seeing this test > trip in 9.x jenkins builds. > > > On Mon, Aug 29, 2022 at 11:50 AM wrote: > > > > This is an automated email from the ASF dual-hosted git repository. > > > > sokolov pushed a commit to branch main > > in repository https://gitbox.apache.org/repos/asf/lucene.git > > > > > > The following commit(s) were added to refs/heads/main by this push: > > new 61ef031f7fa SimpleText knn vectors; fix searchExhaustively and > > suppress a byte format test case (#11725) > > 61ef031f7fa is described below > > > > commit 61ef031f7fa3abdd7c8c2f36db71ad2289b66131 > > Author: Michael Sokolov > > AuthorDate: Mon Aug 29 11:49:52 2022 -0400 > > > > SimpleText knn vectors; fix searchExhaustively and suppress a byte > > format test case (#11725) > > --- > > .../apache/lucene/codecs/simpletext/SimpleTextKnnVectorsReader.java | 6 > > +++--- > > lucene/core/src/test/org/apache/lucene/document/TestField.java | 4 > > > > 2 files changed, 7 insertions(+), 3 deletions(-) > > > > diff --git > > a/lucene/codecs/src/java/org/apache/lucene/codecs/simpletext/SimpleTextKnnVectorsReader.java > > > > b/lucene/codecs/src/java/org/apache/lucene/codecs/simpletext/SimpleTextKnnVectorsReader.java > > index e4b0ceb5916..10700f5de6f 100644 > > --- > > a/lucene/codecs/src/java/org/apache/lucene/codecs/simpletext/SimpleTextKnnVectorsReader.java > > +++ > > b/lucene/codecs/src/java/org/apache/lucene/codecs/simpletext/SimpleTextKnnVectorsReader.java > > @@ -41,7 +41,6 @@ import org.apache.lucene.store.BufferedChecksumIndexInput; > > import org.apache.lucene.store.ChecksumIndexInput; > > import org.apache.lucene.store.IOContext; > > import org.apache.lucene.store.IndexInput; > > -import org.apache.lucene.util.BitSet; > > import org.apache.lucene.util.Bits; > > import org.apache.lucene.util.BytesRef; > > import org.apache.lucene.util.BytesRefBuilder; > > @@ -187,8 +186,9 @@ public class SimpleTextKnnVectorsReader extends > > KnnVectorsReader { > >@Override > >public TopDocs searchExhaustively( > >String field, float[] target, int k, DocIdSetIterator acceptDocs) > > throws IOException { > > -int numDocs = (int) acceptDocs.cost(); > > -return search(field, target, k, BitSet.of(acceptDocs, numDocs), > > Integer.MAX_VALUE); > > +FieldInfo info = readState.fieldInfos.fieldInfo(field); > > +VectorSimilarityFunction vectorSimilarity = > > info.getVectorSimilarityFunction(); > > +return exhaustiveSearch(getVectorValues(field), acceptDocs, > > vectorSimilarity, target, k); > >} > > > >@Override > > diff --git a/lucene/core/src/test/org/apache/lucene/document/TestField.java > > b/lucene/core/src/test/org/apache/lucene/document/TestField.java > > index 781f2b613c6..6aa5518f33b 100644 > > --- a/lucene/core/src/test/org/apache/lucene/document/TestField.java > > +++ b/lucene/core/src/test/org/apache/lucene/document/TestField.java > > @@ -20,6 +20,7 @@ import static > > org.apache.lucene.search.DocIdSetIterator.NO_MORE_DOCS; > > > > import java.io.StringReader; > > import java.nio.charset.StandardCharsets; > > +import org.apache.lucene.codecs.Codec; > > import org.apache.lucene.index.DirectoryReader; > > import org.apache.lucene.index.IndexReader; > > import org.apache.lucene.index.IndexWriter; > > @@ -513,6 +514,9 @@ public class TestField extends LuceneTestCase { > >} > > > >public void testKnnVectorField() throws Exception { > > +if (Codec.getDefault().getName().equals("SimpleText")) { > > + return; > > +} > > try (Directory dir = newDirectory(); > > IndexWriter w = new IndexWriter(dir, newIndexWriterConfig())) { > >Document doc = new Document(); > > > > - > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Lucene 9.4.0 release
Thanks for the links, Tomoko. I thought it would be helpful to ask on the list in case this new system has not been fully internalized by all the developers. And I confess I didn't know about that nice milestone view! It makes this very easy, assuming of course that everyone follows the guidelines and labels the issues with the milestone. On Wed, Aug 31, 2022 at 11:16 AM Tomoko Uchida wrote: > > Ok in short, you can find blockers for 9.4.0 here. > https://github.com/apache/lucene/milestone/3 > > If there are other issues/PRs that are still not resolved but planned to be > delivered in 9.4.0, just set their Milestone as 9.4.0. Then they will appear > in the list. > > > > 2022年9月1日(木) 0:00 Tomoko Uchida : >> >> I believe we decided to use GitHub Milestone for release planning and I >> wrote documentation about it. >> https://github.com/apache/lucene/blob/main/dev-docs/github-issues-howto.md >> >> Is this unclear to you? >> >> >> 2022年8月31日(水) 23:13 Michael Sokolov : >>> >>> Hi, I'd like to start the ball rolling for a 9.4.0 release. We don't >>> have a large number of things, but there are a few nice high-impact >>> changes; see excerpt from CHANGE.txt below. I'll volunteer to be >>> release manager. I'd just like to know if anybody has any outstanding >>> work that is nearing completion and that they'd like to include, >>> and/or if we have any known blocker issues*. >>> >>> * -- I am having a little trouble searching github for blocker issues. >>> I think there may not be any, but I tried >>> https://github.com/apache/lucene/issues?q=is%3Aissue+is%3Aopen+legacy-jira-priority%5C%3AMajor+ >>> to find Major issues, for example, and it seems to only find one Minor >>> one? Does anyone have better github-search-fu? >>> >>> API Changes >>> - >>> * LUCENE-10577: Add VectorEncoding to enable byte-encoded HNSW vectors >>> (Michael Sokolov, Julie Tibshirani) >>> >>> New Features >>> - >>> * LUCENE-10654: Add new ShapeDocValuesField for LatLonShape and >>> XYShape. (Nick Knize) >>> >>> * LUCENE-10629: Support match set filtering with a query in >>> MatchingFacetSetCounts. (Stefan Vodita, Shai Erera) >>> >>> * LUCENE-10633: SortField#setOptimizeSortWithIndexedData and >>> SortField#getOptimizeSortWithIndexedData were introduce to provide >>> an option to disable sort optimization for various sort fields. >>> (Mayya Sharipova) >>> >>> Improvements >>> - >>> * LUCENE-10592: Build HNSW Graph on indexing. (Mayya Sharipova, Adrien >>> Grand, Julie Tibshirani) >>> >>> * GITHUB#11715: Add Integer awareness to RamUsageEstimator.sizeOf (Mike >>> Drob) >>> >>> Optimizations >>> - >>> >>> * LUCENE-10661: Reduce memory copy in BytesStore. (luyuncheng) >>> >>> * GITHUB#1020: Support #scoreSupplier and small optimizations to >>> DocValuesRewriteMethod. (Greg Miller) >>> >>> * LUCENE-10633: Added support for dynamic pruning to queries sorted by a >>> string >>> field that is indexed with terms and SORTED or SORTED_SET doc values. >>> (Adrien Grand) >>> >>> * LUCENE-10627: Using ByteBuffersDataInput reduce memory copy on >>> compressing data. (luyuncheng) >>> >>> * GITHUB#1062: Optimize TermInSetQuery when a term is present that >>> matches all docs in a segment. >>> (Greg Miller) >>> >>> - >>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >>> For additional commands, e-mail: dev-h...@lucene.apache.org >>> - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Lucene 9.4.0 release
Ok in short, you can find blockers for 9.4.0 here. https://github.com/apache/lucene/milestone/3 If there are other issues/PRs that are still not resolved but planned to be delivered in 9.4.0, just set their Milestone as 9.4.0. Then they will appear in the list. 2022年9月1日(木) 0:00 Tomoko Uchida : > I believe we decided to use GitHub Milestone for release planning and I > wrote documentation about it. > https://github.com/apache/lucene/blob/main/dev-docs/github-issues-howto.md > > Is this unclear to you? > > > 2022年8月31日(水) 23:13 Michael Sokolov : > >> Hi, I'd like to start the ball rolling for a 9.4.0 release. We don't >> have a large number of things, but there are a few nice high-impact >> changes; see excerpt from CHANGE.txt below. I'll volunteer to be >> release manager. I'd just like to know if anybody has any outstanding >> work that is nearing completion and that they'd like to include, >> and/or if we have any known blocker issues*. >> >> * -- I am having a little trouble searching github for blocker issues. >> I think there may not be any, but I tried >> >> https://github.com/apache/lucene/issues?q=is%3Aissue+is%3Aopen+legacy-jira-priority%5C%3AMajor+ >> to find Major issues, for example, and it seems to only find one Minor >> one? Does anyone have better github-search-fu? >> >> API Changes >> - >> * LUCENE-10577: Add VectorEncoding to enable byte-encoded HNSW vectors >> (Michael Sokolov, Julie Tibshirani) >> >> New Features >> - >> * LUCENE-10654: Add new ShapeDocValuesField for LatLonShape and >> XYShape. (Nick Knize) >> >> * LUCENE-10629: Support match set filtering with a query in >> MatchingFacetSetCounts. (Stefan Vodita, Shai Erera) >> >> * LUCENE-10633: SortField#setOptimizeSortWithIndexedData and >> SortField#getOptimizeSortWithIndexedData were introduce to provide >> an option to disable sort optimization for various sort fields. >> (Mayya Sharipova) >> >> Improvements >> - >> * LUCENE-10592: Build HNSW Graph on indexing. (Mayya Sharipova, Adrien >> Grand, Julie Tibshirani) >> >> * GITHUB#11715: Add Integer awareness to RamUsageEstimator.sizeOf (Mike >> Drob) >> >> Optimizations >> - >> >> * LUCENE-10661: Reduce memory copy in BytesStore. (luyuncheng) >> >> * GITHUB#1020: Support #scoreSupplier and small optimizations to >> DocValuesRewriteMethod. (Greg Miller) >> >> * LUCENE-10633: Added support for dynamic pruning to queries sorted by a >> string >> field that is indexed with terms and SORTED or SORTED_SET doc values. >> (Adrien Grand) >> >> * LUCENE-10627: Using ByteBuffersDataInput reduce memory copy on >> compressing data. (luyuncheng) >> >> * GITHUB#1062: Optimize TermInSetQuery when a term is present that >> matches all docs in a segment. >> (Greg Miller) >> >> - >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >> For additional commands, e-mail: dev-h...@lucene.apache.org >> >>
Re: Lucene 9.4.0 release
I believe we decided to use GitHub Milestone for release planning and I wrote documentation about it. https://github.com/apache/lucene/blob/main/dev-docs/github-issues-howto.md Is this unclear to you? 2022年8月31日(水) 23:13 Michael Sokolov : > Hi, I'd like to start the ball rolling for a 9.4.0 release. We don't > have a large number of things, but there are a few nice high-impact > changes; see excerpt from CHANGE.txt below. I'll volunteer to be > release manager. I'd just like to know if anybody has any outstanding > work that is nearing completion and that they'd like to include, > and/or if we have any known blocker issues*. > > * -- I am having a little trouble searching github for blocker issues. > I think there may not be any, but I tried > > https://github.com/apache/lucene/issues?q=is%3Aissue+is%3Aopen+legacy-jira-priority%5C%3AMajor+ > to find Major issues, for example, and it seems to only find one Minor > one? Does anyone have better github-search-fu? > > API Changes > - > * LUCENE-10577: Add VectorEncoding to enable byte-encoded HNSW vectors > (Michael Sokolov, Julie Tibshirani) > > New Features > - > * LUCENE-10654: Add new ShapeDocValuesField for LatLonShape and > XYShape. (Nick Knize) > > * LUCENE-10629: Support match set filtering with a query in > MatchingFacetSetCounts. (Stefan Vodita, Shai Erera) > > * LUCENE-10633: SortField#setOptimizeSortWithIndexedData and > SortField#getOptimizeSortWithIndexedData were introduce to provide > an option to disable sort optimization for various sort fields. > (Mayya Sharipova) > > Improvements > - > * LUCENE-10592: Build HNSW Graph on indexing. (Mayya Sharipova, Adrien > Grand, Julie Tibshirani) > > * GITHUB#11715: Add Integer awareness to RamUsageEstimator.sizeOf (Mike > Drob) > > Optimizations > - > > * LUCENE-10661: Reduce memory copy in BytesStore. (luyuncheng) > > * GITHUB#1020: Support #scoreSupplier and small optimizations to > DocValuesRewriteMethod. (Greg Miller) > > * LUCENE-10633: Added support for dynamic pruning to queries sorted by a > string > field that is indexed with terms and SORTED or SORTED_SET doc values. > (Adrien Grand) > > * LUCENE-10627: Using ByteBuffersDataInput reduce memory copy on > compressing data. (luyuncheng) > > * GITHUB#1062: Optimize TermInSetQuery when a term is present that > matches all docs in a segment. > (Greg Miller) > > - > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > >
Re: [lucene] branch main updated: SimpleText knn vectors; fix searchExhaustively and suppress a byte format test case (#11725)
can we backport to 9.x if you get a chance? I'm still seeing this test trip in 9.x jenkins builds. On Mon, Aug 29, 2022 at 11:50 AM wrote: > > This is an automated email from the ASF dual-hosted git repository. > > sokolov pushed a commit to branch main > in repository https://gitbox.apache.org/repos/asf/lucene.git > > > The following commit(s) were added to refs/heads/main by this push: > new 61ef031f7fa SimpleText knn vectors; fix searchExhaustively and > suppress a byte format test case (#11725) > 61ef031f7fa is described below > > commit 61ef031f7fa3abdd7c8c2f36db71ad2289b66131 > Author: Michael Sokolov > AuthorDate: Mon Aug 29 11:49:52 2022 -0400 > > SimpleText knn vectors; fix searchExhaustively and suppress a byte format > test case (#11725) > --- > .../apache/lucene/codecs/simpletext/SimpleTextKnnVectorsReader.java | 6 > +++--- > lucene/core/src/test/org/apache/lucene/document/TestField.java | 4 > 2 files changed, 7 insertions(+), 3 deletions(-) > > diff --git > a/lucene/codecs/src/java/org/apache/lucene/codecs/simpletext/SimpleTextKnnVectorsReader.java > > b/lucene/codecs/src/java/org/apache/lucene/codecs/simpletext/SimpleTextKnnVectorsReader.java > index e4b0ceb5916..10700f5de6f 100644 > --- > a/lucene/codecs/src/java/org/apache/lucene/codecs/simpletext/SimpleTextKnnVectorsReader.java > +++ > b/lucene/codecs/src/java/org/apache/lucene/codecs/simpletext/SimpleTextKnnVectorsReader.java > @@ -41,7 +41,6 @@ import org.apache.lucene.store.BufferedChecksumIndexInput; > import org.apache.lucene.store.ChecksumIndexInput; > import org.apache.lucene.store.IOContext; > import org.apache.lucene.store.IndexInput; > -import org.apache.lucene.util.BitSet; > import org.apache.lucene.util.Bits; > import org.apache.lucene.util.BytesRef; > import org.apache.lucene.util.BytesRefBuilder; > @@ -187,8 +186,9 @@ public class SimpleTextKnnVectorsReader extends > KnnVectorsReader { >@Override >public TopDocs searchExhaustively( >String field, float[] target, int k, DocIdSetIterator acceptDocs) > throws IOException { > -int numDocs = (int) acceptDocs.cost(); > -return search(field, target, k, BitSet.of(acceptDocs, numDocs), > Integer.MAX_VALUE); > +FieldInfo info = readState.fieldInfos.fieldInfo(field); > +VectorSimilarityFunction vectorSimilarity = > info.getVectorSimilarityFunction(); > +return exhaustiveSearch(getVectorValues(field), acceptDocs, > vectorSimilarity, target, k); >} > >@Override > diff --git a/lucene/core/src/test/org/apache/lucene/document/TestField.java > b/lucene/core/src/test/org/apache/lucene/document/TestField.java > index 781f2b613c6..6aa5518f33b 100644 > --- a/lucene/core/src/test/org/apache/lucene/document/TestField.java > +++ b/lucene/core/src/test/org/apache/lucene/document/TestField.java > @@ -20,6 +20,7 @@ import static > org.apache.lucene.search.DocIdSetIterator.NO_MORE_DOCS; > > import java.io.StringReader; > import java.nio.charset.StandardCharsets; > +import org.apache.lucene.codecs.Codec; > import org.apache.lucene.index.DirectoryReader; > import org.apache.lucene.index.IndexReader; > import org.apache.lucene.index.IndexWriter; > @@ -513,6 +514,9 @@ public class TestField extends LuceneTestCase { >} > >public void testKnnVectorField() throws Exception { > +if (Codec.getDefault().getName().equals("SimpleText")) { > + return; > +} > try (Directory dir = newDirectory(); > IndexWriter w = new IndexWriter(dir, newIndexWriterConfig())) { >Document doc = new Document(); > - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Lucene 9.4.0 release
Hi, I'd like to start the ball rolling for a 9.4.0 release. We don't have a large number of things, but there are a few nice high-impact changes; see excerpt from CHANGE.txt below. I'll volunteer to be release manager. I'd just like to know if anybody has any outstanding work that is nearing completion and that they'd like to include, and/or if we have any known blocker issues*. * -- I am having a little trouble searching github for blocker issues. I think there may not be any, but I tried https://github.com/apache/lucene/issues?q=is%3Aissue+is%3Aopen+legacy-jira-priority%5C%3AMajor+ to find Major issues, for example, and it seems to only find one Minor one? Does anyone have better github-search-fu? API Changes - * LUCENE-10577: Add VectorEncoding to enable byte-encoded HNSW vectors (Michael Sokolov, Julie Tibshirani) New Features - * LUCENE-10654: Add new ShapeDocValuesField for LatLonShape and XYShape. (Nick Knize) * LUCENE-10629: Support match set filtering with a query in MatchingFacetSetCounts. (Stefan Vodita, Shai Erera) * LUCENE-10633: SortField#setOptimizeSortWithIndexedData and SortField#getOptimizeSortWithIndexedData were introduce to provide an option to disable sort optimization for various sort fields. (Mayya Sharipova) Improvements - * LUCENE-10592: Build HNSW Graph on indexing. (Mayya Sharipova, Adrien Grand, Julie Tibshirani) * GITHUB#11715: Add Integer awareness to RamUsageEstimator.sizeOf (Mike Drob) Optimizations - * LUCENE-10661: Reduce memory copy in BytesStore. (luyuncheng) * GITHUB#1020: Support #scoreSupplier and small optimizations to DocValuesRewriteMethod. (Greg Miller) * LUCENE-10633: Added support for dynamic pruning to queries sorted by a string field that is indexed with terms and SORTED or SORTED_SET doc values. (Adrien Grand) * LUCENE-10627: Using ByteBuffersDataInput reduce memory copy on compressing data. (luyuncheng) * GITHUB#1062: Optimize TermInSetQuery when a term is present that matches all docs in a segment. (Greg Miller) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org