DivyanshIITB commented on PR #14335:
URL: https://github.com/apache/lucene/pull/14335#issuecomment-2710583795
Thank you for the review, @jpountz!
I see your concern regarding equal resource distribution across IndexWriter
instances potentially leading to inefficiencies when some writers have a
significantly heavier workload than others. My initial goal was to ensure
fairness in multi-tenant environments, but I agree that a static 1/N allocation
might not be ideal.
To address this, I’m considering an adaptive resource allocation approach,
where:
- Recent Merge Activity Tracking: Each IndexWriter's recent merge activity
is monitored.
- Dynamic Thread Allocation: Instead of an equal split, threads are assigned
based on recent workload.
- Merge Throttling Balance: Prevents a single heavy writer from starving
others while ensuring underutilized writers don’t hold
excessive resources.
Would this approach align better with Lucene’s design goals?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
