[
https://issues.apache.org/jira/browse/SOLR-18674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18148339#comment-18148339
]
Andrzej Bialecki commented on SOLR-18674:
-----------------------------------------
[~isaric] I created sub-tasks, hopefully I will be able to draft the Part 1
next week.
> Extend Solr schema with field-level metadata
> --------------------------------------------
>
> Key: SOLR-18674
> URL: https://issues.apache.org/jira/browse/SOLR-18674
> Project: Solr
> Issue Type: Improvement
> Reporter: Andrzej Bialecki
> Assignee: Andrzej Bialecki
> Priority: Major
>
> h2. Problem
> Schema fields carry only indexing and storage properties. There is no
> supported way to record what a field is {*}for{*}, such as a human
> description, its purpose, owner, or semantic role. Teams keep this in wikis
> or in {{_comment}} XML comments, which are lost on Schema API round-trips.
> Admin UI users, tooling, and LLM/agent clients that inspect a schema get no
> semantic context.
> Especially for LLM/agent clients this additional information can reduce the
> effort (and error rate) of data discovery.
> h2. Current behavior
> * {{SchemaField}} keeps unrecognized attributes in its {{args}} map
> ({{{}SchemaField.java:49{}}}).
> * {{getNamedPropertyValues(false)}} coerces every value to a boolean
> ({{{}SchemaField.java:472-478{}}}). An arbitrary string attribute such as
> {{description="..."}} is returned by the Schema API as {{{}false{}}}.
> * Nothing is documented or validated, and {{{}copyField{}}}, dynamic fields
> and field types have no equivalent.
> h2. Proposal
> Add an optional, non-functional {{metadata}} concept to fields, dynamic
> fields, and field types:
> * *First-class {{{}description{}}}:* a free-text string attribute. Also
> support {{{}purpose{}}}, a short free-text string.
> * *Generic {{metadata}} map:* string keys to string values, for user-defined
> entries such as {{{}owner{}}}, {{{}pii=true{}}}, or {{{}source-system{}}}.
> Keys are namespaced by convention, e.g. {{{}team.owner{}}}.
> * *Schema API:* {{{}add-field{}}}, {{replace-field}} and the other field
> commands accept these attributes. {{GET /schema/fields}} returns them as
> strings. A new {{includeMetadata}} parameter, default true, controls this.
> * *Managed schema XML:* serialized as attributes, or as a child
> {{<metadata>}} element for the map. Classic {{schema.xml}} parsing accepts
> both forms.
> * *Semantics:* metadata never affects indexing, query behavior, or the
> schema's compatibility checks. Changing only metadata should not require a
> reindex, and replace-field should say so in its response.
> * *Constraints:* maximum key and value lengths, plus a total size cap, to
> keep ZooKeeper nodes small.
> h2. Design notes
> # Fix the boolean coercion first. Store metadata in a dedicated field on
> {{SchemaField}} rather than reusing {{{}args{}}}.
> # Reserve the {{metadata}} and {{description}} names so they are not treated
> as field properties.
> # SolrJ gets typed accessors in {{SchemaRepresentation}} and the schema
> request classes. Admin UI shows the description as a tooltip and column.
> # Ref guide: document the attributes in the field and Schema API pages. Add
> a changelog entry once a JIRA or PR exists.
> h2. Compatibility and risks
> * *Backward compatible:* old schemas load unchanged. Older nodes that see
> the new attributes already ignore them, but the boolean-coercion bug may
> break their Schema API output. That matters for rolling upgrades and needs a
> test.
> * *Size growth in ZooKeeper:* mitigated by the length caps above.
> * *Overlap:* other attributes might be better handled separately, such as
> {{useDocValuesAsStored}} and similar functional flags. Keep this proposal
> strictly descriptive.
> h2. Open questions
> * Should metadata also be allowed on {{copyField}} and on the schema as a
> whole?
> * Fixed set of keys, such as {{purpose}} as an enum, or a free-form map only?
> * Common set of recommended keys that follow some existing recommendation?
> Examples:
> ** [https://mlcommons.org/working-groups/data/croissant/]
> **
> [https://blog.open-metadata.org/announcing-openmetadata-2-0-the-open-context-layer-for-ai-agents-83b8ce8b9dde]
> ** [https://docs.databricks.com/aws/en/uc-semantics/agent-metadata]
> * Should {{description}} support localization, or just one language?
> h2. Plan
> # Fix coercion, add the {{SchemaField}} metadata storage and XML round-trip,
> with tests.
> # Schema API read and write support, plus SolrJ.
> # Dynamic fields, field types, and Admin UI.
> # Docs and changelog.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]