Andrzej Bialecki created SOLR-18674:
---------------------------------------

             Summary: Extend Solr schema with field-level metadata
                 Key: SOLR-18674
                 URL: https://issues.apache.org/jira/browse/SOLR-18674
             Project: Solr
          Issue Type: Improvement
            Reporter: Andrzej Bialecki


h2. Problem

Schema fields carry only indexing and storage properties. There is no supported 
way to record what a field is {*}for{*}, such as a human description, its 
purpose, owner, or semantic role. Teams keep this in wikis or in {{_comment}} 
XML comments, which are lost on Schema API round-trips. Admin UI users, 
tooling, and LLM/agent clients that inspect a schema get no semantic context.

Especially for LLM/agent clients this additional information can reduce the 
effort (and error rate) of data discovery.
h2. Current behavior
 * {{SchemaField}} keeps unrecognized attributes in its {{args}} map 
({{{}SchemaField.java:49{}}}).
 * {{getNamedPropertyValues(false)}} coerces every value to a boolean 
({{{}SchemaField.java:472-478{}}}). An arbitrary string attribute such as 
{{description="..."}} is returned by the Schema API as {{{}false{}}}.
 * Nothing is documented or validated, and {{{}copyField{}}}, dynamic fields 
and field types have no equivalent.

h2. Proposal

Add an optional, non-functional {{metadata}} concept to fields, dynamic fields, 
and field types:
 * *First-class {{{}description{}}}:* a free-text string attribute. Also 
support {{{}purpose{}}}, a short free-text string.
 * *Generic {{metadata}} map:* string keys to string values, for user-defined 
entries such as {{{}owner{}}}, {{{}pii=true{}}}, or {{{}source-system{}}}. Keys 
are namespaced by convention, e.g. {{{}team.owner{}}}.
 * *Schema API:* {{{}add-field{}}}, {{replace-field}} and the other field 
commands accept these attributes. {{GET /schema/fields}} returns them as 
strings. A new {{includeMetadata}} parameter, default true, controls this.
 * *Managed schema XML:* serialized as attributes, or as a child {{<metadata>}} 
element for the map. Classic {{schema.xml}} parsing accepts both forms.
 * *Semantics:* metadata never affects indexing, query behavior, or the 
schema's compatibility checks. Changing only metadata should not require a 
reindex, and replace-field should say so in its response.
 * *Constraints:* maximum key and value lengths, plus a total size cap, to keep 
ZooKeeper nodes small.

h2. Design notes
 # Fix the boolean coercion first. Store metadata in a dedicated field on 
{{SchemaField}} rather than reusing {{{}args{}}}.
 # Reserve the {{metadata}} and {{description}} names so they are not treated 
as field properties.
 # SolrJ gets typed accessors in {{SchemaRepresentation}} and the schema 
request classes. Admin UI shows the description as a tooltip and column.
 # Ref guide: document the attributes in the field and Schema API pages. Add a 
changelog entry once a JIRA or PR exists.

h2. Compatibility and risks
 * *Backward compatible:* old schemas load unchanged. Older nodes that see the 
new attributes already ignore them, but the boolean-coercion bug may break 
their Schema API output. That matters for rolling upgrades and needs a test.
 * *Size growth in ZooKeeper:* mitigated by the length caps above.
 * *Overlap:* other attributes might be better handled separately, such as 
{{useDocValuesAsStored}} and similar functional flags. Keep this proposal 
strictly descriptive.

h2. Open questions
 * Should metadata also be allowed on {{copyField}} and on the schema as a 
whole?
 * Fixed set of keys, such as {{purpose}} as an enum, or a free-form map only?
 * Common set of recommended keys that follow some existing recommendation? 
Examples:
 ** [https://mlcommons.org/working-groups/data/croissant/]
 ** 
[https://blog.open-metadata.org/announcing-openmetadata-2-0-the-open-context-layer-for-ai-agents-83b8ce8b9dde]
 ** [https://docs.databricks.com/aws/en/uc-semantics/agent-metadata]
 * Should {{description}} support localization, or just one language?

h2. Plan
 # Fix coercion, add the {{SchemaField}} metadata storage and XML round-trip, 
with tests.
 # Schema API read and write support, plus SolrJ.
 # Dynamic fields, field types, and Admin UI.
 # Docs and changelog.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to