sgup432 opened a new issue, #16398: URL: https://github.com/apache/lucene/issues/16398
### Description While working on performance optimization features in lucene, I see we usually end up writing our own JMH micro-benchmark(we have a lot of them now) or run existing luceneutil benchmarks locally. And regressions are often only caught later during nightlies. I was wondering if we could provide a way for users to trigger lightweight benchmarks directly from a PR? Someone could write a command like below, and benchmarks would automatically run in the background and post results on the PR comparing against the baseline: ``` /benchmark jmh <filter> # Run an existing JMH benchmark /benchmark jmh "Write a benchmark to test my PR changes" # passing a prompt to AI model, # it writes a temporary benchmark that doesn't need to checked in /benchmark search <task-filter> # luceneutil search-only (on top of existing data) /benchmark full <task-filter> # luceneutil index + search (default 1M/10M docs?) /benchmark full <task-filter> --data wikimedium5m # explicit corpus size ``` This would make it easy for anyone(author or reviewer) to run benchmarks and catch regressions before merge. I know this requires a dedicated machine to run the benchmarks, so I'm not sure how that part would work. Maybe something similar to what we already have for the luceneutil nightlies or a self-hosted GitHub Actions runner? I'm not sure if folks have already thought about this, but I'd love to hear opinions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
