Hi Jan, Thanks for taking the time to go over the projects. And apologies for the delayed response. To better understand your proposal, I went over the general mechanics of the solr-orbit and solr-orbit-workload projects. I think it makes perfect sense for solr-datagen to be another workload in solr-orbit-workload! It can be a great addition for benchmarking BYOC (Bring your own configset) scenarios ;)
I can see solr-orbit needing some work around understanding a workload.json when the dataset doesn't already exist for download (as in this case), but presumably that should be fairly doable. Also as you pointed out, solr-datagen will need another mode (in addition to "index" and "reindex") to only generate data and handover to solr-orbit. I will try to take a stab at this, although might take a while before I get to it. In the meantime, for anyone interested, pull requests on solr-datagen to that effect are welcome! Thanks, Rahul On Mon, Jun 22, 2026 at 2:24 PM Jan Høydahl <[email protected]> wrote: > Hi, > > Cool with helpful tools like this. Have you thought of writing blog posts > about them > and publishing on solr's blog? > > Idea - could there perhaps be some synergy between our new solr-orbit tool > and your solr-datagen tool? I mean, we currently have two workloads > available > at https://github.com/apache/solr-orbit-workloads - could perhaps the > datagen > be a new workload type, but instead of pulling a static dataset, it > generates one? > I suppose it can generate with a seed so you get same output each time? And > your tool is python, so perhaps it can be adapted as a workload, i.e. the > data > generation logic only, not the ingest logic which is handled by orbit > itself? > > Jan > > > 22. juni 2026 kl. 17:51 skrev Rahul Goswami <[email protected]>: > > > > Hello, > > I commonly find myself needing to cook up data for Solr or > > (not-so-commonly) needing to validate data integrity (detect > > corruption/data deviation scenarios). To that effect, I recently vibe > coded > > a couple of repos over several iterations of testing and > feature-additions. > > > > solr-datagen <https://github.com/rahulgoswami/solr-datagen> : Generate > and > > index realistic documents into Apache Solr at scale, or reindex existing > > documents in-place. > > Given a Solr URL (with collection/core name), solr-datagen can either > > generate synthetic documents with realistic data across all field types > and > > index them in parallel batches (index mode), or read existing documents > > from the collection and write them back in-place (reindex mode) > > > > index_validator <https://github.com/rahulgoswami/index_validator> : > Detect > > field-level data loss and value corruption by comparing two Solr indexes > > (or a pre-operation snapshot against a live index) document-by-document. > > > > I have found them useful for scale testing and data validation and so > > thought of sharing with the community if it helps anyone. These > originated > > as scratch-work tools and gradually evolved to handle scale and support > > more features as needed. So I'm sure there are a lot of ways these can be > > improved (eg: solr-datagen doesn't necessarily have to be specific to > Solr > > and can be decoupled to work with other engines). Sharing here in their > > present form nevertheless if it helps others. > > > > Best, > > Rahul > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
