Hi Srikanth, David, +1, I like this. One concrete gap it would close: solr-mcp's get-schema <https://github.com/apache/solr-mcp/blob/83ab54b7a42cbe1de1c958b0e6f3fe6156da7253/src/main/java/org/apache/solr/mcp/server/schema/SchemaService.java#L241-L248> only returns defined fields, so dynamic fields that actually hold data are invisible to an agent today.
A few use cases I'd add: - Onboarding a collection someone else built, which is routine for platform teams like the one at my company that run search for many product teams. The same goes for multi-tenant collections where doc types carry different fields - Data quality after ingestion: a field's fill rate dropping from 98% to 40% usually means a broken pipeline. I deal with this almost weekly with other data stores, and a cheap way to see it would help a lot - Auditing dynamic field sprawl before a schema cleanup or migration On David's point about checking cardinality first, Druid is a useful reference. I've used it for metrics OLAP, and its segmentMetadata <https://druid.apache.org/docs/latest/querying/segmentmetadataquery/> query reports per-column cardinality and min/max from metadata it already stores, without scanning data. If "/discover" can answer "is this field safe to facet" from stats the index already keeps, the safety gate stays cheap and only borderline fields need the heavier approximate check. Will closely follow this development. Amrit On Sun, Nov 15, 2026 05:58 AM, Eric Pugh via dev <[email protected]> wrote: > > > On 2026/11/14 17:25:37 Jan Høydahl via dev wrote: > > Hi > > > > Sounds like something that could be useful, although it would be a bit > uncommon > > to have a bunch of data in Solr already without already knowing your > fields? > > > > The proposal reminds me of OpenRefine https://openrefine.org which is a > great > > tool to use *before* you start indexing a dataset to Solr. It will eat a > CSV, XMl or > > whatever, and let you discover what's in the data, cardinality, data > quality etc. > > > > A different use case, but just wanted to x-ref it. > > > > Jan > > > > > 13. nov. 2026 kl. 19:38 skrev David Smiley via dev < > [email protected]>: > > > > > > (disclaimer: I worked with Srikanth on this proposal) > > > > > > I've seen a number of situations where we fumble in the absence of > this or > > > things are built similar outside of Solr. The experts here may not > fully > > > appreciate this as we are experts; Solr newbies and sometimes agents, > are > > > not. I think there is growing utility for this with LLM-based > > > interrogation of what's in Solr, and I could definitely see Solr's MCP > > > using this. > > > > > > Safety: This handler is only intended for use by a single person/LLM > at a > > > time to protect the "production" load assumed to be happening > concurrently > > > on the cluster. Of course this aspect could be toggled off if desired > > > (e.g. a "dev" cluster). Assume a timeAllowed/cpuAllowed. Maybe even > rate > > > limit to no more than one or two requests at a time (Solr has code for > > > this). > > > > > > There will be a slippery slope & tension on what /discover should do > versus > > > leave to existing/standard Solr methods. In my view, /discover > shouldn't > > > return entire documents -- just use /select if that is needed. > > > > > > I think it'd be helpful for the response structure to mirror /select > where > > > it fits. Certainly for facets & stats, which is mostly what /discover > > > exposes. Much of the logic of /discover is just applying safe > defaults on > > > what facet/stats to return (possibly blocking "unsafe" requests), plus > a > > > rather new field coverage component. That's powered by > > > "facet.query=field:*" ultimately, but still. In order to determine if > it's > > > "safe" to get the facet values for a field, an approximate cardinality > > > check may be necessary first. > > > > > > On Wed, Nov 11, 2026 at 4:40 PM Srikanth Tumati (BLOOMBERG/ 120 PARK) > via > > > dev <[email protected]> wrote: > > > > > >> Hi all, > > >> > > >> I'd like your thoughts on adding a new data discovery handler to > Solr. The > > >> design would revolve around two principles, simplicity and safety. It > > >> should be easy for anyone to use, and it should have protections > built in > > >> so that exploring your data never puts the cluster at risk. In some > cases, > > >> that may mean returning approximate results instead of exact ones. > > >> Questions, comments and concerns are all welcome. > > >> > > >> Problem > > >> People often come to Solr with a question but no idea how the data is > > >> structured. Which fields exist, which ones are populated, and what do > they > > >> contain? Developers, analysts and (increasingly) LLM agents all run > into > > >> this, and Solr makes these questions hard to answer. Additionally, > dynamic > > >> fields mean the schema doesn't show which fields actually exist, and > > >> documents in the same collection can often have different fields. > > >> Performing a very naive data discovery has the potential to execute > very > > >> expensive Solr queries (e.g. faceting on a high cardinality field, > > >> returning a significant number of rows, etc…). > > >> > > >> What exists today > > >> *Schema XML/Schema API shows the fields that are defined, but not the > > >> dynamic fields that actually exist in the index. It also has no > statistics, > > >> so clients are unable to tell which fields have data. > > >> *Luke (/admin/luke) has per-field stats, but you can't narrow them > down > > >> with filter queries. Newer versions of Solr can run it across shards, > but > > >> the detailed term stats are still per-shard. Finally, it's also an > advanced > > >> tool and hard for beginners to use. > > >> *Facets can answer the question, but only if you already know how to > write > > >> a good Solr query. Every client has to build a facet for each field > and > > >> batch them, which isn't obvious to someone new to Solr. The concern of > > >> expensive queries is directly related to utilizing facets. > > >> > > >> Proposal > > >> A new DiscoveryHandler that takes filter queries (fq) and supports: > > >> *Field coverage: how many matching documents have each field > > >> *Field statistics/facets: distinct count, top values, and min/max for > > >> numbers and dates. For low cardinality fields, the handler would > return the > > >> top facets. Where an exact answer would be too expensive, these may be > > >> provided as approximate values. > > >> *Pivot field: optionally compute stats and coverage by the values of a > > >> field, such as the document type. Each value-group then gets its own > > >> numbers. > > >> Future Plans > > >> Once the basic stats are available, there's a lot more the handler > could > > >> return: > > >> *Field relationships: field coverage inter-relationships, such as > fields > > >> that only appear when another field is present or has a certain value. > > >> *Visualization: the Admin UI could eventually be updated to display > this > > >> information. How exactly is out of scope for this proposal. > > >> The aim is for discovery to eventually grow from "what fields are > here" > > >> into a real picture of how a user’s data is structured. > > >> > > >> Thank you, > > >> Srikanth > > >> > > >> > > >> > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > > > This lines up with the idea that in the future, we actually expose (in a > safe-ish manner) all the capabilities of our search environment to the > agents so they build a "perfect" one off query. Isntead of a narrow "we > accept just a query and maybe some filters" and then expand everything out, > we flip that scirpt and say "here are all the tools, build your perfect > query" and we then provide all the validation/guard rails/workload > management to support that approach! > > I've looked at https://llmstxt.org/ and the idea of having a > http://localhost:8983/llms.txt that told you about the cluster, what > collections are available based on who you are and your permissions. And > then imagine we had https://localhost:8983/solr/techproducts/llms.txt > that was specific to that collection? > > I think there is some overlap here with what solr-mcp might provide in > being a "safer" way of exposing primitives to Agents.... > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
