Re: Getting CorruptIndexException after auto commit after integrating Solr encryption plugin

[Bruno Roustant]

>
> how to set COMMIT_ACTIVE_KEY and COMMIT_KEY_ID or PARAM_KEY_ID in
> SolrQueryRequest req dynamically ?
> If Keys are set to req object before calling
> EncryptionRequestHandler.handleRequestBody(SolrQueryRequest req,
> SolrQueryResponse rsp), this module will work.


Indeed the EncryptionRequestHandler is expected to be called with the key
id, and it will call your custom KeySupplier to get the key descriptor
(called cookie, maybe I'll rename key descriptor) and store it
automatically in the index. This key descriptor will later be passed as
parameter when calling periodically your KeySupplier to get the key secret.

If you started another approach, I will probably need a small project to
understand and elaborate. Maybe you have or you can build quickly a git
repo to reproduce your issue? It would help me to understand your approach
and needs. Or, better, you can submit a PR on the encryption module in
solr-sandbox with a test that reproduces the problem.

Bruno

Re: Re: Getting CorruptIndexException after auto commit after integrating Solr encryption plugin

[Bruno Roustant]

Could you avoid the "RE: Re:" prefix on the email title so that it's easier
to have a single thread mail please?

1- From you solrconfig.xml, I suppose
org.apache.solr.encryption.KmsKeySupplierFactory is your custom
KeySupplierFactory, right?
2- Are you able to run the encryption tests? Do they pass?
3- From your traces, I see "ref key  = 1071082519", which is surprising. Is
it keyRef or keyId? keyRef should be a small integer normally.
4- Could you output the content of Map commitUserData
inside EncryptionDirectory.getLatestCommitData()?
5- Could you output the IndexOutput param of each call to
EncryptionDirectory.maybeWrapOutput()? I would like to know if the
_0_Lucene90FieldsIndex-doc_ids_0.tmp is encrypted.
6- Can you confirm that, until you call the EncryptionRequestHandler, auto
commits are working, and the first auto commit after the
EncryptionRequestHandler call fails?

Re: Getting CorruptIndexException after auto commit after integrating Solr encryption plugin

[Bruno Roustant]

Hi Manish,

First, a disclaimer about the encryption module in solr-sandbox. It is in
sandbox, not integrated (yet) in the Solr project, so there can be some
road blocks. That said, I'll try to help.

1- Can you share the encryption-specific parts of your solrconfig.xml file?
2- Can you share the full stacktrace for the exception you mentioned?
3- How do you trigger the index encryption? Do you call the
EncryptionRequestHandler?
4- Does it happen at the first (auto)commit, or some time later?

Bruno

Re: Welcome Sanjay Dutt as Solr committer!

[Bruno Roustant]

Welcome Sanjay!

Le mar. 21 mai 2024 à 16:03, Gus Heck  a écrit :

> Welcome! Congratulations!
>
> On Tue, May 21, 2024 at 4:56 AM Jan Høydahl  wrote:
>
> > Welcome on board Sanjay, thrilled to see the community grow. Thanks also
> > for introducing yourself and for explaining Open Source to your Mom in
> such
> > a simple way 😁
> >
> > Jan
> >
> > > 20. mai 2024 kl. 18:23 skrev David Smiley :
> > >
> > > The Project Management Committee (PMC) for Apache Solr has invited
> > > Sanjay Dutt to become a committer and we are pleased to announce that
> > > they have accepted.
> > >
> > > Sanjay, the tradition is that new committers introduce themselves with
> > > a brief bio.
> > >
> > > Congratulations and welcome!
> > >
> > > -
> > > To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org
> > > For additional commands, e-mail: dev-h...@solr.apache.org
> > >
> >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org
> > For additional commands, e-mail: dev-h...@solr.apache.org
> >
> >
>
> --
> http://www.needhamsoftware.com (work)
> https://a.co/d/b2sZLD9 (my fantasy fiction book)
>

Re: [DISCUSS] Community Virtual Meetup, February 2024

[Bruno Roustant]

I vote for 2/23

Thanks Jason

Re: BackupRepository changes

[Bruno Roustant]

Separating the corruption check from the copy would be clean but I think it
would be intrusive in the public API, and currently making the check while
reading has better performance that I don't want to disturb for edge cases.
Making it toggle-able is feasible with the plugin init args. It would be
enabled by default, and could be disabled by a FilterBackupRepository (or
by config) if some specific logic is needed to verify the checksum. I gave
it a try and I created Jira
 and a proposal PR
.

1- Add an optional initialization argument to disable the checksum
verification (enabled by default) in each BackupRepository implementation.

2- Introduce an abstract FilterBackupDirectory to extend when we need to
perform different/additional logic on top of a delegate BackupDirectory.
For example, verify the checksum differently. Also, the
FilterBackupDirectory could inject some initialization arguments when the
BackupRepositoryFactory creates its delegate.
Also make the existing TrackingBackupRepository (for tests) extend
FilterBackupDirectory.

3- Add a test case to AbstractBackupRepositoryTest to ensure all
implementations support disabling checksum verification. And add a test for
LocalFileSystemDirectory (not tested currently) that also extends
AbstractBackupRepositoryTest.

Re: BackupRepository changes

[Bruno Roustant]

> Isn't the intent to ensure we don't waste time/space creating a
useless backup of something that is, I suppose, already corrupted?

That's right. And I didn't read the code enough; a clear effort has been
put here since the last time I read the code, to make all implementations
consistent to verify the checksum.

Hum, this is annoying for directory-based encryption. The only way becomes
to have an encryption extension for each and all different implementations.
Less clean than a FilterBackupRepository.

BackupRepository changes

[Bruno Roustant]

I noticed two points that I propose to change in some BackupRepository
implementations and in BackupRepositoryFactory:

1- Two implementations (S3BackupRepository, GCSBackupRepository) compute
and verify the checksum of index files being copied. Other implementations
do not.
=> I propose to remove this checksum verification, to simply copy the bytes.
First, because it seems to me it is useless to verify the checksum before
backup (checking it after restoring seems more useful).
Second, because for the encryption module in dev in solr-sandbox, I need to
ensure the bytes copied stay encrypted, and the checksum is not verifiable
in this case.

2- There is a special logic in BackupRepositoryFactory when creating the
TrackingBackupRepository. TrackingBackupRepository delegates to another
BackupDirectory, and it is only used for tests.
=> I propose to introduce FilterBackupRepository to delegate to a
BackupDirectory in a more generic way. TrackingBackupRepository would
become a FilterBackupRepository.
Again, I would use an EncryptionBackupRepository in solr-sandbox to
delegate to any BackupRepository, while ensuring copied bytes stay
encrypted.

I would appreciate any feedback :)

Re: forbidden-apis + assertThat = insanity

[Bruno Roustant]

I agree that having to write "MatcherAssert.assertThat" each time is
tedious and makes my code ugly. So finally I try to avoid this nice
construction. Not satisfying.

Le mer. 25 oct. 2023 à 22:44, Kevin Risden  a écrit :

> It came in as part of
> https://github.com/apache/solr/pull/947#issuecomment-1279651282
>
> I linked to one of the comments there since this was discussed on the PR.
> Basically it removes a ton of deprecated usages. If/when we upgrade Junit
> we should hopefully have much less to do.
>
> So to answer your question
>
> > - What is so terrible about the impl of Assert.assertThat?
> >
>
> Nothing just that it is deprecated and trying to stay on top of avoiding
> the use of deprecated code if we can.
>
> - What evil does it contain that makes it completely unssuitable for use?
> >
>
> Again same as above since it is the same question twice.
>
> I'll give you that its not ideal, but we have to draw the line somewhere.
>
> Kevin Risden
>
>
> On Wed, Oct 25, 2023 at 4:27 PM Chris Hostetter 
> wrote:
>
> >
> > Begin Rant...
> >
> > $ tail -6 ./gradle/validation/forbidden-apis/junit.junit.all.txt
> > junit.framework.TestCase @ All classes should derive from LuceneTestCase
> > junit.framework.Assert @ Use org.junit.Assert
> > junit.framework.** @ Use org.junit
> > org.junit.Assert#assertThat(**) @ Use
> > org.hamcrest.MatcherAssert.assertThat()
> > org.junit.matchers.JUnitMatchers @ Use org.hamcrest.CoreMatchers
> > org.junit.rules.ExpectedException @ Use Assert.assertThrows
> >
> > So all test classes should:
> >   - derive from LuceneTestCase (agreed)
> >   - use org.junit.Assert instead of junit.framework.Assert (fair enough)
> >   - use MatcherAssert.assertThat() instead of Assert.assertThat() ...
> >
> > ...ok, i guess?  ... except that LuceneTestCase extends Assert, giving us
> > Assert.assertThat() for free.  Kind of anoying to have to go out of our
> > way to import another class to do the same thing.
> >
> > But also ... you can't just use...
> >
> >   import static org.hamcrest.MatcherAssert.assertThat;
> >
> > ...because (assuming you're following best practices) you're already
> > inheriting from LuceneTestCase, so that static import is going to be
> > unused since it conflicts with the inherited 'assertThat' -- or at least
> > that's what ecjLintTest tells you ... i'm not 100% certain it's correct,
> > but either way you can't use it because it will fails the build.
> >
> > So your only recourse is to use...
> >
> >   import org.hamcrest.MatcherAssert;
> >
> > ...and now instead of lots of nice short 'assertThat(...)' method calls
> > you have to use the twice as long 'MatcherAssert.assertThat', which is
> > probably going to make spotless decide your assertions needs to span at
> > least one extra line (assuming you are using descriptive variable names)
> >
> > All because we don't want want devs to use a (deprecated) method
> > that's indirectly inherited from an ancestor of a baseclass we tell them
> > to use.
> >
> > Which begs the questions:
> >
> > - What is so terrible about the impl of Assert.assertThat?
> > - What evil does it contain that makes it completely unssuitable for use?
> >
> > Let's go find out what exactly we are asking forbidden-apis to
> > shield us from...
> >
> >
> >
> https://github.com/junit-team/junit4/blob/r4.13.2/src/main/java/org/junit/Assert.java#L962
> >
> >  public static  void assertThat(String reason, T actual,
> >  Matcher matcher) {
> >  MatcherAssert.assertThat(reason, actual, matcher);
> >  }
> >
> > ...oh dear lord ... what sweet horror is this??!?!?!!?
> >
> > Thank heavens we were wise enough to protect ourselves from this
> > unspeakably dangerous code!
> >
> >
> >
> > -Hoss
> > http://www.lucidworks.com/
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org
> > For additional commands, e-mail: dev-h...@solr.apache.org
> >
> >
>

PR to cleanup VersionBucket for the UpdateLog

[Bruno Roustant]

Hi All,

I have a PR[1] to remove the "highest" field from VersionBucket (highest
doc version of the bucket).
By reading the code, I understand that this field is used for an
optimization in the case of a distributed update and non-leader checking a
doc version for an update. But I wondered about the usefulness of this
optimization.
I ran the SolrIndexing benchmark after removing this field (and cleaning
all the code used to maintain it), and it shows no perf difference.

By removing this field (a long) from VersionBucket, we can save 500 MB of
RAM per Core, because there are 64K VersionBucket. In addition it
removes/simplifies code.

I plan to merge the PR soon. Please comment there if you have any concerns.

Thanks

Bruno

[1] https://github.com/apache/solr/pull/2021#issuecomment-1774115830

Re: Apache Solr Newsletter proposal September 2023

[Bruno Roustant]

+1 !

Le lun. 9 oct. 2023 à 16:54, Eric Pugh  a
écrit :

> I’d love to see it on mastodon/twitterX/linkedin too….   I bet ASF PR
> would love to see this as well ;-)
>
> > On Oct 8, 2023, at 5:36 PM, Alessandro Benedetti 
> wrote:
> >
> > +1 I think it's a wonderful idea!
> >
> > On Sun, 8 Oct 2023, 17:26 Arrieta, Alejandro, <
> aarri...@perrinsoftware.com>
> > wrote:
> >
> >> 0. Intro
> >> Welcome to the Apache Solr Newsletter proposal September 2023. As this
> is
> >> the first newsletter proposal of 2023 we include Solr Community news
> >> starting from January 2023:
> >>
> >> Welcome Alex Deparvu as Solr committer:
> >> https://lists.apache.org/thread/fx4o3y55ypqy1n06kdbq5k8h4pjxlowj
> >>
> >> Welcome Marcus Eagan as Solr committer:
> >> https://lists.apache.org/thread/2p2fjm18g39x2mknxqnbr3gfcx0q6rrx
> >>
> >> Welcome David Smiley as Solr's new PMC chair:
> >> https://lists.apache.org/thread/yrp2yll52r0fm2pbhw17b62oln63fp33
> >>
> >> Welcome Andy Webb as Solr committer:
> >> https://lists.apache.org/thread/k449v7jl4bwg6f7x707tvw5ch2dk8h3f
> >>
> >> Welcome Justin Sweeney as Solr committer:
> >> https://lists.apache.org/thread/g588106pohdwok7gpwwcsxpgvwxwwh41
> >>
> >> Welcome Colvin Cowie as Solr committer:
> >> https://lists.apache.org/thread/ngzjs0b3mb201d2shfvzdysdd5g6xqz1
> >>
> >> 1. Current versions
> >> Solr 9.3.0:
> >> https://solr.apache.org/downloads.html#solr-930
> >>
> >> Solr 8.11.2:
> >> https://solr.apache.org/downloads.html#solr-8112
> >>
> >> Solr Operator v0.7.1:
> >> https://solr.apache.org/operator/artifacts.html#solr-v071
> >>
> >> 2. Incoming versions
> >> Solr 9.4.0
> >> mailing list thread:
> >> https://lists.apache.org/thread/j44golmqp4sjf2rk74pqktkrsc1do58r
> >>
> >> Solr 8.11.3
> >> mailing list thread:
> >> https://lists.apache.org/thread/j44golmqp4sjf2rk74pqktkrsc1do58r
> >>
> >> 3. Solr Improvement Proposals News
> >> SIP-13: Cross Data Center Replication
> >> mailing list thread:
> >> https://lists.apache.org/thread/9y2qd7rs45049xbrt0gty40m4b5cv22y
> >> wiki link:
> >>
> >>
> https://cwiki.apache.org/confluence/display/SOLR/SIP-13%3A+Cross+Data+Center+Replication
> >>
> >> SIP-14 Embedded Zookeeper
> >> mailing list thread:
> >> https://lists.apache.org/thread/mch6y4wsbqykcmfy4r76movmx1wz0375
> >> wiki link:
> >>
> https://cwiki.apache.org/confluence/display/SOLR/SIP-14+Embedded+Zookeeper
> >>
> >> SIP-15 Node roles
> >> mailing list thread:
> >> https://lists.apache.org/thread/kwq609jxckpdqk54fvghxs66jtk92kof
> >> wiki link:
> >> https://cwiki.apache.org/confluence/display/SOLR/SIP-15+Node+roles
> >>
> >> SIP-19 Adopt JSR-330 dependency injection
> >> mailing list thread:
> >> https://lists.apache.org/thread/9y2qd7rs45049xbrt0gty40m4b5cv22y
> >> wiki link:
> >>
> >>
> https://cwiki.apache.org/confluence/display/SOLR/SIP-19+Adopt+JSR-330+dependency+injection
> >>
> >> 4. Links to blogs, videos, others related to Solr:
> >> Apache Solr Community Virtual Meetup, September 2023:
> >>
> >>
> https://www.meetup.com/apache-solr-virtual-community-meetups/events/296344421/
> >>
> >> DataImportHandler 9.3.0 released:
> >> https://lists.apache.org/thread/8z65tf4jr2ppdm4xo5r1bygjzt8z8r3f
> >>
> >> 5. Upcoming Events & meetings
> >> ASF Community over Code in Halifax, October 7-10 2023:
> >> https://communityovercode.org/
> >>
> >> Apache Solr Community Virtual Meetup, October 2023:
> >> https://www.meetup.com/apache-solr-virtual-community-meetups/events/
> >>
> >> 6. Connect with the Apache Solr Community
> >> Subscribe to Apache Solr Virtual Meetup for monthly meeting:
> >> https://www.meetup.com/apache-solr-virtual-community-meetups/
> >>
> >> Follow us on Twitter/X:
> >> https://twitter.com/apachesolr
> >>
> >> Visit our mailing list, jira, irc and slack:
> >> https://solr.apache.org/community.html
> >>
> >> How to contribute:
> >> https://solr.apache.org/community.html#how-to-contribute
> >>
> >> Apache Solr website:
> >> https://solr.apache.org/
> >>
> >> Apache Solr Operator website:
> >> https://solr.apache.org/operator/
> >>
> >> Note: This is a proposal and I will start working on the October 2023
> >> newsletter proposal and create a Jira to accept contributions. Maybe we
> can
> >> publish this on the Apache Solr web if accepted in the future.
> >>
> >> Kind Regards,
> >> Alejandro Arrieta
> >>
>
> ___
> Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 |
> http://www.opensourceconnections.com <
> http://www.opensourceconnections.com/> | My Free/Busy <
> http://tinyurl.com/eric-cal>
> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed <
> https://www.packtpub.com/big-data-and-business-intelligence/apache-solr-enterprise-search-server-third-edition-raw>
>
> This e-mail and all contents, including attachments, is considered to be
> Company Confidential unless explicitly stated otherwise, regardless of
> whether attachments are marked as such.
>
>

Re: Limiting Backup IO

[Bruno Roustant]

Nice +1

Le ven. 7 juil. 2023 à 16:07, David Smiley  a
écrit :

> I gave it a look.  I like it!
>
> ~ David
>
>
> On Thu, Jul 6, 2023 at 6:22 PM Pierre Salagnac 
> wrote:
>
> > Here is my POC to add a queue into CoreAdminHandler:
> > https://github.com/apache/solr/pull/1761
> >
> > It does the following:
> > - add a flag to core admin operations to be marked as expensive. For now,
> > only backup and restore are expensive, this may be extended.
> > - in CoreAdminHandler, we count the number of in-flight expensive
> > operations. If more than the limit (currently 5 by default) are
> > already in-flight, we don't submit any new ones to the thread pool, but
> we
> > add them into a queue.
> > - each time an expensive operation is completed, it starts the following
> > one from the queue, if any.
> >
> > Let me know what you think
> > Thanks
> >
> > Le jeu. 29 juin 2023 à 15:37, Pierre Salagnac  >
> > a
> > écrit :
> >
> > > Jason, I haven't done much scalability testing, so it's hard to give
> > > accurate numbers on when we start having issues.
> > > For the environment I looked in detail we run a 16 nodes cluster, and
> the
> > > collection I wasn't able to backup has about 1500 shards, ~1.5 GB each.
> > >
> > > Core backups/restores are expensive calls, compared to other admin
> calls
> > > like an empty core creation. I'm not sure of the full list of expensive
> > and
> > > non-expense operations, but we may also have a fairness issue when
> > > expensive operations block the non expensive ones for a while.
> > >
> > > I had a great discussion with David.
> > > I see two options for a long term fix:
> > >
> > > 1. Throttle core backups/restore at the top level (in the overseer).
> > > My approach was to not send too many concurrent requests from
> BackupCmd.
> > > I have a decent POC for backups, but it should be refactored to share
> > this
> > > mechanism for all admin operations.
> > > It will be harder to achieve in distributed mode (no overseer), since
> > > we'll need a central place to count how many backups are in-flight. We
> > may
> > > somehow lock in Zookeeper for this, so it may be over complex at the
> end.
> > >
> > >
> > > 2. Throttle in each node.
> > > - Currently, all async admin operations are handled by a
> > > ThreadPoolExecutor defined in CoreAdminHandler class. This pool is
> > > hardcoded with a size of 50, so if we receive more than 50 concurrent
> > > tasks, we add them in the queue of the executor. For a large collection
> > > backup, each node immediately starts 50 concurrent core snapshots.
> > > - Instead of immediately submitting to the executor, I think we should
> > > manage our own queue here for expensive operations. By counting the
> > number
> > > of in-flight backups/restores, we don't submit to the executor more
> than
> > > (lets say) 10 concurrent backups. Each time one is complete, we poll
> the
> > > queue and start the next one if appropriate.
> > > - This ensures fairness for expensive and non expensive operations.
> > > Non-expensive ones will always have at least 40 threads to be quickly
> > > handled. And this works well in distributed mode since the overseer is
> > not
> > > involved.
> > > - This could be extended to define more than one queue, but I'm not
> sure
> > > it is worth it.
> > >
> > > Pierre
> > >
> > >
> > > Le mar. 27 juin 2023 à 19:16, David Smiley  a
> écrit
> > :
> > >
> > >> Here's a POC: https://github.com/apache/solr/pull/1729
> > >>
> > >> ~ David Smiley
> > >> Apache Lucene/Solr Search Developer
> > >> http://www.linkedin.com/in/davidwsmiley
> > >>
> > >>
> > >> On Mon, Jun 19, 2023 at 3:36 PM David Smiley 
> > wrote:
> > >>
> > >> > Has anyone mitigated the potentially large IO impact of doing a
> backup
> > >> of
> > >> > a large collection or just in general?  If the collection is large
> > >> enough,
> > >> > there very well could be many shards on one host and it could
> saturate
> > >> the
> > >> > IO.  I wonder if there should be a rate limit mechanism or some
> other
> > >> > mechanism.
> > >> >
> > >> > Not the same but I know that at a segment level, the merges are rate
> > >> > limited -- ConcurrentMergeScheduler doesn't quite let you set it but
> > >> > adjusts itself automatically ("ioThrottle" boolean).
> > >> >
> > >> > ~ David Smiley
> > >> > Apache Lucene/Solr Search Developer
> > >> > http://www.linkedin.com/in/davidwsmiley
> > >> >
> > >>
> > >
> >
>

Re: Proposal of an encryption module in solr-sandbox

[Bruno Roustant]

Important notice: currently the encryption module does not encrypt TLogs
yet.

Le mer. 15 mars 2023 à 10:28, Ishan Chattopadhyaya <
ichattopadhy...@gmail.com> a écrit :

> HDFS*
>
> On Wed, 15 Mar, 2023, 2:56 pm Ishan Chattopadhyaya, <
> ichattopadhy...@gmail.com> wrote:
>
> > Btw, +1 to the initiative. I've heard of clients used encrypted HDFC for
> > these usecases. Direct support at Lucene/Solr level is much better.
> >
> > On Wed, 15 Mar, 2023, 2:52 pm Ishan Chattopadhyaya, <
> > ichattopadhy...@gmail.com> wrote:
> >
> >> Does it need to be a first party project?
> >>
> >> On Wed, 15 Mar, 2023, 2:46 pm Bruno Roustant, 
> >> wrote:
> >>
> >>> Hi,
> >>>
> >>> I pushed a PR <https://github.com/apache/solr-sandbox/pull/51> in
> >>> solr-sandbox <https://github.com/apache/solr-sandbox> to propose a
> >>> Java-level encryption for Solr.
> >>> This work is the follow up of LUCENE-9379
> >>> <https://issues.apache.org/jira/projects/LUCENE/issues/LUCENE-9379>.
> >>>
> >>> To give some details, here is the overview section of the ENCRYPTION.md
> >>> <
> >>>
> https://github.com/apache/solr-sandbox/blob/e422e3dd4febab54ba9a8d965189b38217552b46/ENCRYPTION.md
> >>> >
> >>> file in this PR:
> >>>
> >>> This solution provides the encryption of the Lucene index files at the
> >>> Java
> >>> level.
> >>> It encrypts all (or some) the files in a given index with a provided
> >>> encryption key.
> >>> It stores the id of the encryption key in the commit metadata (and
> >>> obviously the
> >>> key secret is never stored). It is possible to define a different key
> per
> >>> Solr Core.
> >>> This module also provides an EncryptionRequestHandler so that a client
> >>> can
> >>> trigger
> >>> the (re)encryption of a Solr Core index. The (re)encryption is done
> >>> concurrently
> >>> while the Solr Core can continue to serve update and query requests.
> >>>
> >>> Comparing with an OS-level encryption:
> >>>
> >>> - OS-level encryption [1][2] is more performant and more adapted to let
> >>> Lucene
> >>> leverage the OS memory cache. It can manage encryption at block or
> >>> filesystem
> >>> level in the OS. This makes it possible to encrypt with different keys
> >>> per-directory,
> >>> making multi-tenant use-cases possible.
> >>> If you can use OS-level encryption, prefer it and skip this Java-level
> >>> encryption.
> >>>
> >>> - Java-level encryption can be used when the OS-level encryption
> >>> management
> >>> is
> >>> not possible (e.g. host machine managed by a cloud provider). It has an
> >>> impact
> >>> on performance: expect -20% on most queries, -60% on multi-term
> queries.
> >>>
> >>> [1] https://wiki.archlinux.org/title/Fscrypt
> >>> [2] https://www.kernel.org/doc/html/latest/filesystems/fscrypt.html
> >>>
> >>> - Bruno
> >>>
> >>
>

Proposal of an encryption module in solr-sandbox

[Bruno Roustant]

Hi,

I pushed a PR  in
solr-sandbox  to propose a
Java-level encryption for Solr.
This work is the follow up of LUCENE-9379
.

To give some details, here is the overview section of the ENCRYPTION.md

file in this PR:

This solution provides the encryption of the Lucene index files at the Java
level.
It encrypts all (or some) the files in a given index with a provided
encryption key.
It stores the id of the encryption key in the commit metadata (and
obviously the
key secret is never stored). It is possible to define a different key per
Solr Core.
This module also provides an EncryptionRequestHandler so that a client can
trigger
the (re)encryption of a Solr Core index. The (re)encryption is done
concurrently
while the Solr Core can continue to serve update and query requests.

Comparing with an OS-level encryption:

- OS-level encryption [1][2] is more performant and more adapted to let
Lucene
leverage the OS memory cache. It can manage encryption at block or
filesystem
level in the OS. This makes it possible to encrypt with different keys
per-directory,
making multi-tenant use-cases possible.
If you can use OS-level encryption, prefer it and skip this Java-level
encryption.

- Java-level encryption can be used when the OS-level encryption management
is
not possible (e.g. host machine managed by a cloud provider). It has an
impact
on performance: expect -20% on most queries, -60% on multi-term queries.

[1] https://wiki.archlinux.org/title/Fscrypt
[2] https://www.kernel.org/doc/html/latest/filesystems/fscrypt.html

- Bruno

Re: Google Java Style: shall we declare intent to follow?

[Bruno Roustant]

+1
I find that a standard is more productive because I wouldn't have to
question anymore about what is the consistent naming/pattern to use.

Le mar. 7 mars 2023 à 19:05, Anshum Gupta  a écrit :

> I like the idea, David. Overall it makes for code that is easier to read
> and understand, specially for new contributors.
>
>
> On Wed, Mar 1, 2023 at 5:46 PM David Smiley  wrote:
>
> > I wish to standardize our use of casing in names/symbols.  And perhaps to
> > align with GJS more broadly.
> >
> > We use the google-java-format build plugin, which is obviously tightly
> > correlated with the Google Java Style.  I think we/Solr jumped on board
> > with auto code formatting but didn't necessarily declare an intent to
> align
> > ourselves with GJS more broadly.  I'd like us to do so now.  I'm not
> > advocating for sweeping changes to follow from this, just an intent to
> > align future decisions.  Maybe some previous things might get renamed as
> we
> > feel inclined.
> >
> > According to the Google Java Style[1], symbol names with acronyms should
> be
> > camel cased.  Thus RebalanceLeadersAPI should be RebalanceLeadersApi, for
> > example.
> >
> > FWIW I've followed this approach for a long time and I find that it
> > produces more predictable results with fewer wonky scenarios.  On the
> down
> > side, it's somewhat unsatisfying that acronyms aren't cased as would be
> > done in a natural (non-code) written form.  But such forms have space as
> a
> > delimiter, unlike code.
> >
> > [1] https://google.github.io/styleguide/javaguide.html#s5-naming
> >
> > ~ David Smiley
> > Apache Lucene/Solr Search Developer
> > http://www.linkedin.com/in/davidwsmiley
> >
>
>
> --
> Anshum Gupta
>

Re: Welcome Andy Webb as Solr committer

[Bruno Roustant]

Welcome!

Le mar. 7 mars 2023 à 21:14, Tomás Fernández Löbbe 
a écrit :

> Welcome Andy!
>
> On Tue, Mar 7, 2023 at 7:53 AM Gus Heck  wrote:
>
> > Welcome :)
> >
> > On Tue, Mar 7, 2023 at 10:37 AM Michael Gibney <
> mich...@michaelgibney.net>
> > wrote:
> >
> > > Welcome, and congrats!
> > >
> > > On Tue, Mar 7, 2023 at 6:19 AM Ilan Ginzburg 
> wrote:
> > > >
> > > > Welcome Andy and congrats!
> > > >
> > > > On Tue, Mar 7, 2023 at 7:52 AM Mikhail Khludnev 
> > wrote:
> > > >
> > > > > Welcome, Andy!
> > > > >
> > > > > On Mon, Mar 6, 2023 at 10:54 PM Andy Webb 
> > wrote:
> > > > >
> > > > > > hi all, thank you for the invitation and welcome messages - this
> > has
> > > been
> > > > > > an unexpected honour!
> > > > > >
> > > > > > I'm currently Technical Architect for Search at the BBC, based in
> > > > > > Manchester, UK. I've been using Solr since about 2012 and made a
> > few
> > > > > small
> > > > > > contributions over the last few years - thanks again to all the
> > > > > committers
> > > > > > who've helped me with those.
> > > > > >
> > > > > > I also love mountain biking and help to run a local club, often
> > > getting
> > > > > out
> > > > > > for rides in beautiful locations like the Peak District, Lake
> > > District,
> > > > > > Wales and Scotland.
> > > > > >
> > > > > > Andy
> > > > > >
> > > > > > On Mon, 6 Mar 2023, 11:16 Jan Høydahl, 
> > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I'm pleased to announce that Andy Webb has accepted the PMC's
> > > > > > > invitation to become a committer.
> > > > > > >
> > > > > > > Andy, the tradition is that new committers introduce themselves
> > > with a
> > > > > > > brief bio.
> > > > > > >
> > > > > > > Congratulations and welcome!
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Sincerely yours
> > > > > Mikhail Khludnev
> > > > > https://t.me/MUST_SEARCH
> > > > > A caveat: Cyrillic!
> > > > >
> > >
> > > -
> > > To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org
> > > For additional commands, e-mail: dev-h...@solr.apache.org
> > >
> > >
> >
> > --
> > http://www.needhamsoftware.com (work)
> > http://www.the111shift.com (play)
> >
>

preferredLeader is useless during election or node restart

[Bruno Roustant]

After many tests and deployments, it appears the preferredLeader flag
described in the RebalanceLeader command doc [1] is not useful.
It is taken into account only during the rebalance command. Afterwards, if
there is a leader election or some node restart, it is ignored.

Is this preferredLeader useless?
I thought to use it to make leadership kind of sticky, but in practice the
leadership assignment quickly returns to randomness. So, what was the
purpose of this flag for the rebalance command, really a one-shot leader
assignment, ignored after? Or is it a bug?

Indeed only the rebalance leader command doc talks about this replica
property. It is not mentioned elsewhere. But if it is ignored elsewhere,
it's not of a great help.

Should I enter a bug on preferredLeader property not respected during
leader elections?

[1]
https://solr.apache.org/guide/solr/latest/deployment-guide/collection-management.html#rebalanceleaders

Re: Welcome Justin Sweeney as Solr committer

[Bruno Roustant]

Welcome Justin!

Le jeu. 16 févr. 2023 à 18:04, Ilan Ginzburg  a écrit :

> Welcome Justin!
>
> On Tue, Feb 14, 2023 at 2:53 PM Kevin Risden  wrote:
>
> > Welcome Justin!
> >
> > Kevin Risden
> >
> >
> > On Mon, Feb 13, 2023 at 11:58 AM Houston Putman 
> > wrote:
> >
> > > Welcome Justin!
> > >
> > > - Houston
> > >
> > > On Sun, Feb 12, 2023 at 5:56 AM Alessandro Benedetti <
> > a.benede...@sease.io
> > > >
> > > wrote:
> > >
> > > > Welcome Justin!
> > > >
> > > > On Sat, 11 Feb 2023, 17:04 Jason Gerlowski, 
> > > wrote:
> > > >
> > > > > Congrats and welcome Justin!
> > > > >
> > > > > On Sat, Feb 11, 2023 at 9:19 AM Mikhail Khludnev 
> > > > wrote:
> > > > > >
> > > > > > Congratulations, Justin.
> > > > > >
> > > > > > On Fri, Feb 10, 2023 at 10:31 PM Justin Sweeney <
> > > > > justin.sweene...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi everyone,
> > > > > > >
> > > > > > > Thank you Ishan for the announcement. I'm excited to be joining
> > as
> > > a
> > > > > > > committer and continuing to be involved with and contributing
> to
> > > > Solr.
> > > > > > >
> > > > > > > I'm currently based outside of Boston, Massachusetts, although
> I
> > am
> > > > > soon
> > > > > > > moving to Maine.
> > > > > > > I've worked as a software engineer and managed engineering
> teams
> > > for
> > > > > the
> > > > > > > past 14 years. For roughly the past 8 years, I have been
> involved
> > > in
> > > > > the
> > > > > > > search space, first at ZoomInfo and currently at FullStory,
> doing
> > > > > > > development with and administering Solr clusters.
> > > > > > >
> > > > > > > Looking forward to continued involvement with the community and
> > > > working
> > > > > > > with all of you.
> > > > > > >
> > > > > > > Justin
> > > > > > >
> > > > > > > On Fri, Feb 10, 2023 at 2:00 PM Ishan Chattopadhyaya <
> > > > > > > ichattopadhy...@gmail.com> wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > I'm pleased to announce that Justin Sweeney has accepted the
> > > PMC's
> > > > > > > > invitation to become a committer.
> > > > > > > >
> > > > > > > > Justin, the tradition is that new committers introduce
> > themselves
> > > > > with a
> > > > > > > > brief bio.
> > > > > > > >
> > > > > > > > Congratulations and welcome!
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Sincerely yours
> > > > > > Mikhail Khludnev
> > > > > > https://t.me/MUST_SEARCH
> > > > > > A caveat: Cyrillic!
> > > > >
> > > > >
> -
> > > > > To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org
> > > > > For additional commands, e-mail: dev-h...@solr.apache.org
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: Welcome Colvin Cowie as Solr committer

[Bruno Roustant]

Welcome Colvin!

Le jeu. 16 févr. 2023 à 18:02, Ilan Ginzburg  a écrit :

> Congrats and welcome Colvin!
>
> On Wed, Feb 15, 2023 at 1:21 PM Jason Gerlowski 
> wrote:
>
> > Welcome Colvin!
> >
> > On Tue, Feb 14, 2023 at 2:34 PM Anshum Gupta 
> > wrote:
> > >
> > > Congratulations and welcome, Colvin!
> > >
> > > On Tue, Feb 14, 2023 at 1:57 AM Colvin Cowie <
> colvin.cowie@gmail.com
> > >
> > > wrote:
> > >
> > > > Thanks everyone,
> > > >
> > > > I'm based in Cambridge (UK). I've been a software engineer for about
> 10
> > > > years, most of which were at IBM until
> > > > my part of the company was spun off at the start of last year.
> > > > I've been working with Solr on and off since 2015, integrating it and
> > > > customizing it for our product and troubleshooting
> > > > customer deployments.
> > > >
> > > > I'm hoping to do more with Solr in our product, as it will give me
> more
> > > > reasons to contribute here :)
> > > >
> > > > Colvin
> > > >
> > > > On Tue, 14 Feb 2023 at 07:46, Noble Paul 
> wrote:
> > > >
> > > > > Congrats and welcome Colvin
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Feb 14, 2023 at 12:28 PM Gus Heck 
> > wrote:
> > > > >
> > > > > > Welcome Colvin :).
> > > > > >
> > > > > > On Mon, Feb 13, 2023 at 6:21 PM Jan Høydahl <
> jan@cominvent.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I'm pleased to announce that Colvin Cowie has accepted the
> PMC's
> > > > > > > invitation to become a committer.
> > > > > > >
> > > > > > > Colvin, the tradition is that new committers introduce
> themselves
> > > > with
> > > > > a
> > > > > > > brief bio.
> > > > > > >
> > > > > > > Congratulations and welcome!
> > > > > > >
> > > > > > >
> > -
> > > > > > > To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org
> > > > > > > For additional commands, e-mail: dev-h...@solr.apache.org
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > --
> > > > > > http://www.needhamsoftware.com (work)
> > > > > > http://www.the111shift.com (play)
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > -
> > > > > Noble Paul
> > > > >
> > > >
> > >
> > >
> > > --
> > > Anshum Gupta
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org
> > For additional commands, e-mail: dev-h...@solr.apache.org
> >
> >
>

Re: Outreachy

[Bruno Roustant]

Welcome Joshua!

And thanks David for this initiative!

Le mar. 6 déc. 2022 à 15:32, David Smiley  a écrit :

> Welcome Joshua!  I'm looking forward to these changes coming to
> fruition finally!
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Tue, Dec 6, 2022 at 7:38 AM Joshua Ouma  wrote:
>
> > Hi Everyone,
> >
> > I'm Joshua and have been accepted to intern with Solr through the
> Outreachy
> > internship programme. During the 12 weeks internship period I'll work
> > on *SOLR-11872
> > Refactor test infra to work with a managed SolrClient; ditch TestHarness
> > *with
> > David Smiley as my mentor.
> >
> > I'm grateful to be part of the community and looking forward to working
> > with you in meeting the project requirements.
> >
> > Regards,
> > Joshua
> >
> > On Mon, Oct 10, 2022 at 7:55 AM David Smiley  wrote:
> >
> > > Hey everyone,
> > >
> > > I am registered as an ASF mentor for Outreachy (kinda like GSOC).  I
> > > submitted a project proposal for SOLR-11872[1].  Thankfully, I'm seeing
> > > that a number of intern candidates are interested!  Candidates must
> > > contribute to the project in some small way by November 4th, after
> which
> > I
> > > will select the best candidate to do SOLR-11872.  At present,
> candidates
> > > are looking at "newdev" tasks and even volunteering.  Now would be a
> > great
> > > time to consider if issues you create are fitting of "newdev".  And of
> > > course, be supportive/helpful to contributors you've never seen/heard
> of
> > > before.  I think the JAX-RS conversion should have particularly good
> > > "newdev" issues now that the infrastructure is in place.
> > >
> > > [1]: https://issues.apache.org/jira/browse/SOLR-11872
> > > "Refactor test infra to work with a managed SolrClient; ditch
> > TestHarness"
> > >
> > > ~ David Smiley
> > > Apache Lucene/Solr Search Developer
> > > http://www.linkedin.com/in/davidwsmiley
> > >
> >
>

Re: JIT Shard leader design/proposal

[Bruno Roustant]

I don't know enough how the current leader election mechanism works, yet.
I miss a comparison between this proposal and the current mechanism.

B. With this proposal, each time we need the leader, we check it. What is
the cost of this check? Do we need to read the cluster state each time?

C.1.A "Strict use of ZkShardTerms". Does that mean that we wait for one
replica to become up to date? Does that mechanism already exist?
In the meantime, if the previous leader comes back, we can choose it.

Le jeu. 13 oct. 2022 à 16:24, David Smiley  a écrit :

> "JIT" is Just-In-Time; a way of looking at it.  Or call it on-demand leader
> elections.
>
> A property of my proposal that may not be obvious is that REBALANCELEADERS
> would be needless since preferredLeaders would become leaders automatically
> on-demand (when a leader is next needed).
>
> It's not clear how amenable Curator's election recipe is to preferredLeader
> and ZkShardTerms requirements; so using Curator for elections might simply
> not be an option?
>
> Here's another thematic concept that is tangentially related; maybe it's
> worth its own discussion:  Solr's view of state from ZK could become stale
> at any time.  Embrace sharing of ZK node versions between client and server
> to trigger the need for either side to update its state from ZK.  Watchers
> are nice but we shouldn't rely on them for correctness.  Put differently,
> SolrCloud should function (albeit perhaps slowly) if its use of ZK Watchers
> stopped working.  Today, CloudSolrClient passes _stateVer_ and it's parsed
> on the server by
> org.apache.solr.servlet.HttpSolrCall#checkStateVersionsAreValid.  This
> should be used more pervasively in most interactions, not merely a subset
> of uses of CloudSolrClient.  I think it might only be used to tell the
> client to get more up to date but ideally it'd be bidirectional.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Tue, Oct 11, 2022 at 6:34 PM David Smiley  wrote:
>
> > At work, I’ve attempted to troubleshoot issues relating to shard
> > leadership.  It’s quite possible that the root-causes may be related to
> > customizations in my fork of Solr; who knows.  The leadership
> > code/algorithm is so hard to debug/troubleshoot that it’s hard to say.
> > It’s no secret that Solr’s code here is a complicated puzzle[1].  Out of
> > this frustration, I began to ponder a fantasy of how I want leader
> election
> > to work, informed by my desire to scale to massive numbers of
> collections &
> > shards on a cluster.  Using Curator for elections would perhaps address
> > stability but not this scale.  I’d like to get input from you all on this
> > fantasy.  Surely I have overlooked things; please offer your insights!
> >
> > Thematic concept:  Don’t change/elect leaders until it’s actually
> > necessary.  In most cases where I work, the leader will return before we
> > truly need a leader.  Even when not true, I don’t think doing it lazily
> > should be a noticeable issue?  If so, it’s easy to imagine augmenting
> this
> > design with an optional eager leadership election.
> >
> > A. Only code paths that truly need a leader will do “leadership checks”,
> > resulting in a potential leader election.  This is principally on
> indexing
> > in DistributedZkUpdateProcessor but there are likely more spots.
> >
> > B. Leader check: Check if the shard’s leader is (a) known, and (b)
> > state=ACTIVE, and (c) on a “live” node, and (d) the preferredLeader
> > condition is satisfied.  Otherwise, try to elect a leader in a loop until
> > this set of conditions is achieved or a timeout is reached.
> > B.A: The preferredLeader condition means that either the leader is marked
> > as preferredLeader, or no replica with preferredLeader is eligible for
> > leadership.
> >
> > C. “Try to elect a leader”:   (The word “election” might not be the best
> > word for this algorithm, but whatever).
> > C.1.: A replica must be eligible to be a leader.  It must be live (on a
> > live node) and have an ACTIVE state.  And, very important, eligibility
> > should be governed by ZkShardTerms which knows which replicas have the
> most
> > up-to-date state.
> > C.1.A: Strict use of ZkShardTerms is designed to ensure that there is no
> > data loss.  That said “forceLeader” remains in the toolbox of Solr admins
> > (which monkey’s with ZkShardTerms to cheat).  We may need a new optional
> > mechanisms to be closer to what we have today — to basically ignore
> > ZkShardTerms after a configured period of time?
> > C.1.B. I am assuming that replicas will become eligible on their own
> (e.g.
> > as nodes re-join) instead of this algorithm needing to initiate/tell any
> to
> > get into this state somehow.
> > C.2: If there are no leader-eligible replicas, complain with useful
> > information to diagnose why no leader was found.  Don’t log this if we
> > already logged this same message in our leadership check loop.  Sleep
> > per

Re: preferredLeader

[Bruno Roustant]

It should also be possible to set preferred leaders during shard split.
Currently, shard split always keeps the current host as the leader of the
sub-shards. And if we split again sub-shards, this can lead to a very
unbalanced situation where the same host becomes the leader for a whole set
of shards.
I created a Jira issue SOLR-16438
 to support a
setPreferredLeaders on the split command (and to rebalance the leaders on
the split is complete).

Le ven. 23 sept. 2022 à 15:26, David Smiley  a écrit :

> I'm not yet sure if preferredLeader is honored after a restart of the node
> that has this state; though it'd be easy to see.  I plan to observe this
> empirically soon.  If memory serves, I recall leader checks occur for each
> replica on a node's start.  One would think this would respect
> preferredLeader already?  If it doesn't, then the solution, I think, would
> be to fix it so that it does respect it?  We'll see.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Thu, Sep 22, 2022 at 7:02 AM Jan Høydahl  wrote:
>
> > We have the new Replica Placement API, which can react to various events
> > like nodeUp, nodeDown etc.
> > Would it make sense to auto rebalance leaders after a certain delay after
> > the last nodeDown/Up event occurred?
> >
> > Jan
> >
> > > 22. sep. 2022 kl. 05:23 skrev David Smiley :
> > >
> > > SolrCloud's choice of a leader isn't "sticky"; it can change and won't
> > > necessarily change back on its own if the original choice is eligible
> > > again.  Setting "preferredLeader" on a replica and calling the
> > > REBALANCELEADERS API is how to make it sticky.[1].  Without this, a
> > rolling
> > > restart (common-place in a cloud-native world?) will leave the last
> node
> > > without any leaders.  That can lead to a resource usage imbalance,
> > > particularly with TLOG due to the higher differentiation of activity a
> > > leader does vs a non-leader.  Make sense?
> > >
> > > Would it make sense for collection creation to have a
> setPreferredLeader
> > > option or some-such boolean name?  Or just let the caller/client do so
> if
> > > it pleases, and maybe mention this as advice in the reference guide?
> Or
> > > just do this by default for TLOG collection creation... although I
> wonder
> > > if it'd interfere with how some users manage their clusters?
> > >
> > > [1]
> > >
> >
> https://solr.apache.org/guide/solr/latest/deployment-guide/collection-management.html#rebalanceleaders
> > >
> > > ~ David Smiley
> > > Apache Lucene/Solr Search Developer
> > > http://www.linkedin.com/in/davidwsmiley
> >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org
> > For additional commands, e-mail: dev-h...@solr.apache.org
> >
> >
>

MoveReplicaCmd with concurrent updates

[Bruno Roustant]

Hi all,
I'm playing with the MoveReplicaCmd. Internally it calls addReplica and
then deleteReplica. And it seems that, if there are concurrent updates, the
call to deleteReplica makes the concurrent updates fail because they do not
find the replica at the original host.
Details:
The test moves mytest_shard1_0_replica_n5 from 127.0.0.1:61086_solr to
127.0.0.1:61084_solr.
And it fails just after the MoveReplicaCmd deletes the source replica with
the error message:
Error 404 Can not find: /solr/mytest_shard1_0_replica_n5/update

Is there a way to first route requests to the replica from the original
host to the target host, before we can delete the original replica?

Best,
Bruno

Re: Welcome Markus Jelsma as Solr committer

[Bruno Roustant]

Welcome Markus!

Le mar. 21 juin 2022 à 20:13, Christine Poerschke (BLOOMBERG/ LONDON) <
cpoersc...@bloomberg.net> a écrit :

> Hello everyone.
>
> On behalf of the Apache Solr PMC, I'm pleased to announce that Markus
> Jelsma has accepted the invitation to become a Solr committer.
>
> Markus - it's a tradition that you introduce yourself with a brief bio, if
> you wish.
>
> Congratulations and welcome!
>
> --
> Christine
>

Re: Quarterly Committer Meetings

[Bruno Roustant]

+1
I'll try to make it, probably with some lateness.

Bruno

Le jeu. 13 janv. 2022 à 03:34, Shawn Heisey  a écrit :

> On 1/12/2022 3:46 AM, Alessandro Benedetti wrote:
> > +1, the time works for me and I am in!
> >
> > Do we have a calendar event already?
>
> Looking forward to seeing a Google Meet link!  I blocked out some time
> on my calendar for this.
>
> Thanks,
> Shawn
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org
> For additional commands, e-mail: dev-h...@solr.apache.org
>
>

Re: [DISCUSS] Mark Rate Limiting as Experimental for Solr 9.0

[Bruno Roustant]

+1
It will allow folks to try it (me included?)

Bruno

Le mer. 12 janv. 2022 à 17:00, Mike Drob  a écrit :

> Howdy folks,
>
> After finding two quick bugs in the rate limiting code[1][2], I suspect
> that this feature has largely gone untested in our code up until now. I
> also looked at TestRequestRateLimiter and at least locally I'm getting a
> lot of EofException in the logs, which I think is unintended. This might be
> easy to fix as well, but I don't know what the next bug will be if we keep
> going down this path.
>
> I think Atri has mentioned that he was using a version of this feature
> with 8.x, but there's probably been some divergence between the two
> implementations. Maybe we can get interested folks to try it with 9.0 and
> then I have high confidence we can get it stable in time for 9.1? I do not
> think reverting the feature is necessary, and it should stay in the code.
>
> Thoughts?
> Mike
>
> [1]: https://github.com/apache/solr/pull/516
> [2]: https://github.com/apache/solr/pull/505
>