Re: Deadlock between core reload and IndexFetcher

2024-05-24 Thread Joel Bernstein
k.run(java.base@17.0.10/Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.10/Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.10/Unknown Source) Joel Bernstein http://joelsolr.blogspot.com/ On Fri, May 24,

Deadlock between core reload and IndexFetcher

2024-05-24 Thread Joel Bernstein
I'll create a jira for this shortly but wanted to mention this on the dev list. We've run into a deadlock which occurs when a core reload occurs at the same time the IndexFetcher is running. The effect of this is that replication stops until Solr is restarted. There is a work around which is to

Re: NullPointerException in ReRankScaler.explain

2023-09-12 Thread Joel Bernstein
You are running into this bug: https://issues.apache.org/jira/browse/SOLR-16931 Currently ReRankScaler.explain requires the debugQuery=true parameter to work properly. I plan fixing this in the next release. Joel Bernstein http://joelsolr.blogspot.com/ On Tue, Sep 12, 2023 at 2:31 AM Anna

Re: Time for a 9.3 release?

2023-06-26 Thread Joel Bernstein
+1 Joel Bernstein http://joelsolr.blogspot.com/ On Mon, Jun 26, 2023 at 4:53 PM Tomás Fernández Löbbe wrote: > +1! I was thinking the same thing, thanks for volunteering. I do think we > should make sure we are running Lucene 9.7. I'm planning to work on the > upgrade unless some

Re: Hybrid scoring lexical / vector

2023-06-01 Thread Joel Bernstein
I'll also add an implementation for RRF to the ReRanker. https://weaviate.io/blog/hybrid-search-explained Joel Bernstein http://joelsolr.blogspot.com/ On Fri, May 26, 2023 at 4:58 PM Joel Bernstein wrote: > I'm going to create a ticket for adding Min/Max scaling to the ReRan

Re: Hybrid scoring lexical / vector

2023-05-26 Thread Joel Bernstein
I'm going to create a ticket for adding Min/Max scaling to the ReRanker. The ReRanker has access to all the topDocs so it should be pretty straightforward to min/max scale all the topDocs before ReRanking the topN. Joel Bernstein http://joelsolr.blogspot.com/ On Thu, May 25, 2023 at 5:18 AM

Re: Anyone using more obscure features of SimplePostTool?

2023-05-23 Thread Joel Bernstein
I have mixed feelings about this because I love command line tools. But I haven't actually used the bin/post tool in years. On Tue, May 23, 2023 at 10:13 AM David Smiley wrote: > +1 to deprecate bin/post. > > Eric, RE the issue title... it's better. Still, if I were advocating for > the work

Hybrid scoring lexical / vector

2023-05-23 Thread Joel Bernstein
One of the things that I'm focusing on is combining the Solr similarity score with the vector score in a consistent manner. My main concern is dealing with the unbounded nature of the Solr similarity score and how to balance that with a vector score. So my first question are there any mechanisms

Calling attention to SOLR-16807

2023-05-18 Thread Joel Bernstein
I wanted to bring up SOLR-16807 ( https://issues.apache.org/jira/browse/SOLR-16807) because its turning out to be a pretty big thorn in my side at the moment. Enough so that I'm considering proposing a bug fix release. The issue is that a catch all (*) copyfield is incompatible with an index that

Re: SolrCloud separating compute from storage

2023-04-28 Thread Joel Bernstein
. Joel Bernstein http://joelsolr.blogspot.com/ On Fri, Apr 28, 2023 at 4:03 PM David Smiley wrote: > To clarify the point to everyone: "separation of compute from storage" > allows infrastructure cost savings for when you have both large scale (many > shards in the cluster)

Re: [VOTE] Release Solr 9.2.1 RC1

2023-04-27 Thread Joel Bernstein
+1 (binding) SUCCESS! [0:43:48.160659] I tested out the assets as well and looked fine. Joel Bernstein http://joelsolr.blogspot.com/ On Thu, Apr 27, 2023 at 1:23 PM Jan Høydahl wrote: > +1 (binding) > > SUCCESS! [0:38:44.920838] > > Jan > > > 27. apr. 2023 kl. 16

Re: Cloud storage modules for backup/restore

2023-04-24 Thread Joel Bernstein
be indexing and merging lots of small segments etc ... Joel Bernstein http://joelsolr.blogspot.com/ On Fri, Apr 21, 2023 at 3:27 AM Ishan Chattopadhyaya < ichattopadhy...@gmail.com> wrote: > My colleague at SearchScale has tried S3FS, and running Solr indexes off > S3. We ca

Re: Cloud storage modules for backup/restore

2023-04-17 Thread Joel Bernstein
into a public maven repo and I can update this thread when that's done. Joel Bernstein http://joelsolr.blogspot.com/ On Mon, Apr 10, 2023 at 6:51 PM Ishan Chattopadhyaya < ichattopadhy...@gmail.com> wrote: > Oh thanks, Jan. I had missed it. It is a shame because it looks like a very >

Re: Welcome Marcus Eagan as Solr committer

2023-04-05 Thread Joel Bernstein
Welcome Marcus! Joel Bernstein http://joelsolr.blogspot.com/ On Mon, Apr 3, 2023 at 12:40 PM Tomás Fernández Löbbe wrote: > Welcome Marcus! > > On Mon, Apr 3, 2023 at 7:32 AM Houston Putman > wrote: > > > Congrats Marcus! > > > > - Houston > > >

Re: Logging KNN vector queries

2023-03-20 Thread Joel Bernstein
://issues.apache.org/jira/browse/SOLR-16710 On Mon, Mar 20, 2023 at 10:09 AM Joel Bernstein wrote: > I've run into an issue with KNN vector queries and the Solr logs. The > issue is for large vectors you can end up with 20K+ of just vector data in > the logs. This is because each floating po

Logging KNN vector queries

2023-03-20 Thread Joel Bernstein
h our logs. Even if we can configure longer log records I think we need the ability to specify parameters that we don't want to end up in the log. Any thoughts? I'll create a jira issue shortly as well. Joel Bernstein http://joelsolr.blogspot.com/

Re: [VOTE] Release Solr 9.1.0 RC1

2022-10-28 Thread Joel Bernstein
+1 SUCCESS! [1:17:27.016191] I kicked the tires on the assets as well. Created a 2 shard collection, loaded data, ran the SQL module, ran streaming expressions. Saw no problems. Joel Bernstein http://joelsolr.blogspot.com/ On Thu, Oct 27, 2022 at 10:14 AM Kevin Risden wrote: >

Github solr-committers group

2022-10-28 Thread Joel Bernstein
Looks like I (joel-bernstein) need to be added to: https://github.com/orgs/apache/teams/solr-committers/members?query=joel Is this something infra needs to do? Or does the PMC manage this? Joel Bernstein http://joelsolr.blogspot.com/

Re: FacetFieldProcessorByArrayDV recomputes ords for each request?

2022-09-01 Thread Joel Bernstein
The fact that its slow with 100 docs makes me wonder how many values are in the multi-value field? I'll load up some docs tomorrow with a multi-value field and see how it performs locally. Joel Bernstein http://joelsolr.blogspot.com/ On Thu, Sep 1, 2022 at 9:50 PM Michael Gibney wrote

ClientUtils.escapeQueryChars escaping whitespace?

2022-07-07 Thread Joel Bernstein
for whitespace to be escaped. Joel Bernstein http://joelsolr.blogspot.com/

Re: Welcome Markus Jelsma as Solr committer

2022-06-23 Thread Joel Bernstein
Welcome Markus! Joel Bernstein http://joelsolr.blogspot.com/ On Thu, Jun 23, 2022 at 8:47 AM Jason Gerlowski wrote: > Welcome Markus! > > On Wed, Jun 22, 2022 at 12:21 PM Ishan Chattopadhyaya > wrote: > > > > Welcome Markus. Looking forward to more help from you

Re: [VOTE] Release Solr 9.0.0 RC5

2022-05-07 Thread Joel Bernstein
+1 (binding) SUCCESS! [0:50:47.511601] Joel Bernstein http://joelsolr.blogspot.com/ On Fri, May 6, 2022 at 11:26 AM Eric Pugh wrote: > +1 (binding) > > SUCCESS! [1:05:27.496955] > > On May 6, 2022, at 11:16 AM, Timothy Potter wrote: > > +1 (binding) > > SUCCESS

Re: [VOTE] Release Solr 9.0.0 RC4

2022-05-02 Thread Joel Bernstein
It would be a pretty odd edge case. The enum method would typically not be used in high cardinality cases and limit:-1 means an overrequest is not needed. I personally don't think this is a blocker. Joel Bernstein http://joelsolr.blogspot.com/ On Mon, May 2, 2022 at 12:43 PM Michael Gibney

Re: [VOTE] Release Solr 9.0.0 RC4

2022-04-28 Thread Joel Bernstein
SUCCESS! [0:48:19.349163] +1 (binding) I also performed the following manual steps with the artifacts: - started Solr with the sql module - created a collection, - loaded data - queried from the sql admin screen - ran streaming expressions All worked as expected. Joel Bernstein http

Re: [VOTE] Release Solr 9.0.0 RC3

2022-04-06 Thread Joel Bernstein
SUCCESS! [0:44:43.026634] I have also manually verified that the SQL issues have been resolved in the artifacts. Here is my +1 Joel Bernstein http://joelsolr.blogspot.com/ On Tue, Apr 5, 2022 at 5:58 PM Jan Høydahl wrote: > Please vote for release candidate 3 for Solr 9.

Re: Solr github permissions

2022-04-01 Thread Joel Bernstein
After logging into id.apache.org I found my github id was indeed missing. After I filled it in I was invited to the apache committers github group. I haven't tested exactly what this will allow me to do but I suspect it will resolve the github issues. Joel Bernstein http://joelsolr.blogspot.com

Re: Solr github permissions

2022-04-01 Thread Joel Bernstein
Thanks! Joel Bernstein http://joelsolr.blogspot.com/ On Fri, Apr 1, 2022 at 9:44 AM Mike Drob wrote: > Reach out to ASF Infra for help troubleshooting either on slack #asfinfra > or through INFRA jira. > > Make sure your profile on id.apache.org is linked to your GitHub profile >

Solr github permissions

2022-04-01 Thread Joel Bernstein
My github permissions appear to be incorrect or missing for the Solr repository. I'm able to push to the apache gitbox repo, where my commits are mirrored to github. But my permissions on github seem to be non-existent for the project. My github account is joel-bernstein. Is there someone

Re: [VOTE] Release Solr 9.0.0 RC2

2022-03-31 Thread Joel Bernstein
Here is the jira https://issues.apache.org/jira/browse/SOLR-16131. My official vote for the RC: -1 Joel Bernstein http://joelsolr.blogspot.com/ On Thu, Mar 31, 2022 at 4:15 AM Jan Høydahl wrote: > Joel, > > I could reproduce the error > > Failed to l

Re: [VOTE] Release Solr 9.0.0 RC2

2022-03-30 Thread Joel Bernstein
rror-class","java.lang.ClassNotFoundException"], "msg":" Error loading class 'solr.SQLHandler'", Joel Bernstein http://joelsolr.blogspot.com/ On Wed, Mar 30, 2022 at 8:39 PM Joel Bernstein wrote: > I tried loading the sql module

Re: [VOTE] Release Solr 9.0.0 RC2

2022-03-30 Thread Joel Bernstein
I tried loading the sql module but ran into this error when running a sql query: Failed to load JDBC driver for 'org.apache.solr.handler.sql.CalciteSolrDriver' I loaded the module using system prop from startup command line: bin/solr start -c -Dsolr.modules=sql Joel Bernstein http

Re: [VOTE] Release Solr 9.0.0 RC2

2022-03-30 Thread Joel Bernstein
+1 SUCCESS! [1:26:27.731406] I also spent some time working with artifacts, creating collections, loading data, querying, streaming expressions. All looked good. Joel Bernstein http://joelsolr.blogspot.com/ On Wed, Mar 30, 2022 at 6:47 PM Houston Putman wrote: > +1 (binding) > >

Re: Release notes draft for Apache Solr 9.0

2022-03-29 Thread Joel Bernstein
Some nice improvements in the temporal graph queries: https://github.com/apache/solr/blob/main/solr/solr-ref-guide/modules/query-guide/pages/graph.adoc#temporal-graph-expressions Particularly useful for people building event correlation and SIEM products. Joel Bernstein http

Re: [VOTE] Release Solr 9.0.0 RC1

2022-03-29 Thread Joel Bernstein
SUCCESS! [1:24:30.108261] No need to vote on this one, but a success nonetheless. Joel Bernstein http://joelsolr.blogspot.com/ On Tue, Mar 29, 2022 at 4:44 PM Mike Drob wrote: > Never mind, I think that last one is because I ran from a src unpack and > not a bin unpack like I tho

Re: [Operator] [ANNOUNCE] Apache Solr Operator v0.5.1 released

2022-03-21 Thread Joel Bernstein
Ok thanks! Joel Bernstein http://joelsolr.blogspot.com/ On Mon, Mar 21, 2022 at 4:30 PM Houston Putman wrote: > Yes, its archived like all other apache releases. If you download using > closer.lua, it will automatically route to where the artifacts reside. It > works the same as

Re: [Operator] [ANNOUNCE] Apache Solr Operator v0.5.1 released

2022-03-21 Thread Joel Bernstein
It looks like https://downloads.apache.org/solr/solr-operator/v0.5.0/ was removed. Is that by design? Joel Bernstein http://joelsolr.blogspot.com/ On Mon, Mar 21, 2022 at 3:25 PM Houston Putman wrote: > The Apache Solr PMC is pleased to announce the release of the Apache Solr > Op

Thoughts on this overseer issue: SOLR-16013

2022-02-17 Thread Joel Bernstein
another test is firing ADDREPLICA requests in another test. In this situation the overseer leader being scaled down executes the ADDREPLICA and then the newly elected overseer executes the same ADDREPLICA. Joel Bernstein http://joelsolr.blogspot.com/

Re: Multi-thread search

2022-01-24 Thread Joel Bernstein
Thanks! I'll take a look. Joel Bernstein http://joelsolr.blogspot.com/ On Mon, Jan 24, 2022 at 3:33 PM Ishan Chattopadhyaya < ichattopadhy...@gmail.com> wrote: > SOLR-13350 > > On Tue, 25 Jan, 2022, 2:01 am Joel Bernstein, wrote: > >> I'm not sure if this topic

Multi-thread search

2022-01-24 Thread Joel Bernstein
I'm not sure if this topic has come up before, I'm interested in having Solr use Lucene's threaded search capability. There are situations where this could really help with performance I believe. Has anyone tried to tackle this before? Joel Bernstein http://joelsolr.blogspot.com/

Re: custom PostFilter

2022-01-24 Thread Joel Bernstein
block. I'm not sure if block indexing can be used in your use case or not though as it might be a grouping related feature. Joel Bernstein http://joelsolr.blogspot.com/ On Mon, Jan 24, 2022 at 10:09 AM Dan Rosher wrote: > Hi, > > We have a use case ( to calc commute tim

Re: Some 9.0 proposals of mine

2022-01-15 Thread Joel Bernstein
I'm quite excited about the vector search as well. This has a lot of interesting scientific and machine learning applications, for example large scale multi-variate knn regression. I'll start testing out the PR next week. Joel Bernstein http://joelsolr.blogspot.com/ On Fri, Jan 14, 2022 at 12

Re: New branch and feature freeze for Solr 9.0.0

2022-01-11 Thread Joel Bernstein
time widows this is a Fintech event correlation tool. Existing documentation for temporal graph queries are here: https://github.com/apache/solr/blob/main/solr/solr-ref-guide/src/graph.adoc#temporal-graph-expressions Joel Bernstein http://joelsolr.blogspot.com/ On Mon, Jan 10, 2022 at 5:21 PM

Re: Inventory updates via join query and caches

2021-12-19 Thread Joel Bernstein
. The downside to this is the pollution of the main index with ever changing inventory segments. Joel Bernstein http://joelsolr.blogspot.com/ On Sun, Dec 19, 2021 at 6:10 PM David Smiley wrote: > I'm not sure there is a clean/simple solution to this specific problem. > But I could i

Re: 8.11 release candidate

2021-11-03 Thread Joel Bernstein
I completed the backport this morning. Joel Bernstein http://joelsolr.blogspot.com/ On Tue, Nov 2, 2021 at 9:34 PM Joel Bernstein wrote: > Sorry for the lateness of this, but I'd like to get > https://issues.apache.org/jira/browse/SOLR-15761 committed tomorrow > (Wednesday) and

Re: 8.11 release candidate

2021-11-02 Thread Joel Bernstein
Sorry for the lateness of this, but I'd like to get https://issues.apache.org/jira/browse/SOLR-15761 committed tomorrow (Wednesday) and backport for 8.11. It's a very small change. Joel Bernstein http://joelsolr.blogspot.com/ On Tue, Nov 2, 2021 at 4:46 PM Adrien Grand wrote: > Hi Ja

Re: Leader election in Kube-land

2021-10-29 Thread Joel Bernstein
collections operator which creates collections using the Solr operator to allocate the Solr nodes. The collections operator is where all the intelligence resides for creating collections that maximize resiliency on kubernetes. Joel Bernstein http://joelsolr.blogspot.com/ On Thu, Oct 28, 2021 at 8

Leader election in Kube-land

2021-10-28 Thread Joel Bernstein
leader election in the code base, greatly decreases the amount of code complexity and allows committers to harden the eventually consistent model. Joel Bernstein http://joelsolr.blogspot.com/

Re: [Operator] Proposal to cut the v0.4.0 release soon

2021-07-23 Thread Joel Bernstein
sted I can share the python scripts I'm working on when they are ready next week. The scripts handle replica placement directly rather than relying on Solr for placement. Joel Bernstein http://joelsolr.blogspot.com/ On Wed, Jul 21, 2021 at 11:18 AM Timothy Potter wrote: > Sounds good Ho

Re: Testing out the solr-operator, snag with _default config

2021-06-22 Thread Joel Bernstein
A pod restart could be the issue as each Solr pod is showing 1 restart. But the logs are so clean that it didn't capture any sign of a restart if it happened within the one minute window between the initial start and failed collection creation. Joel Bernstein http://joelsolr.blogspot.com

Testing out the solr-operator, snag with _default config

2021-06-21 Thread Joel Bernstein
oud-headless.default:8983/solr: Error CREATEing SolrCore 'testcol_shard1_replica_n1': Unable to create core [testcol_shard1_replica_n1] Caused by: Can't find resource 'lang/contractions_it.txt' in classpath or '/configs/_default', cwd=/opt/solr-8.8.2/server Joel Bernstein http://joelsolr.blogspot.com/

Re: [NOTICE] 2021-06 Solr Committer Meeting on 9th June

2021-06-07 Thread Joel Bernstein
Please send me an invite as well. Joel Bernstein http://joelsolr.blogspot.com/ On Mon, Jun 7, 2021 at 5:26 AM Uwe Schindler wrote: > Hi, > > > > can you send me an invitation, too? > > > > Uwe > > > > - > > Uwe Schindler > > Achter

Re: Welcome Eric Pugh to the Apache Solr PMC

2021-04-30 Thread Joel Bernstein
Welcome Eric! Joel Bernstein http://joelsolr.blogspot.com/ On Fri, Apr 30, 2021 at 3:37 AM Dawid Weiss wrote: > Sorry to be late to the party but cordial congratulations, Eric! > > Dawid > > On Sat, Apr 24, 2021 at 2:19 AM Jan Høydahl > wrote: > > > > I am