[ 
https://issues.apache.org/jira/browse/SOLR-18674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18146886#comment-18146886
 ] 

Ivan Saric commented on SOLR-18674:
-----------------------------------

Thanks for writing this up. One point in the current-behavior section differs 
from main, and it changes the compatibility picture.

*Unknown field attributes are rejected, not kept.* 
[parseProperties|https://github.com/apache/solr/blob/8b2e99721e28c66b96654d0dff440d74785febb2/solr/core/src/java/org/apache/solr/schema/FieldProperties.java#L133]
 runs every attribute through {{propertyNameToInt}}, which [throws "Invalid 
field 
property"|https://github.com/apache/solr/blob/8b2e99721e28c66b96654d0dff440d74785febb2/solr/core/src/java/org/apache/solr/schema/FieldProperties.java#L102]
 for any name outside the known properties and a [three-name ignore 
list|https://github.com/apache/solr/blob/8b2e99721e28c66b96654d0dff440d74785febb2/solr/core/src/java/org/apache/solr/schema/FieldProperties.java#L108].
 Both the [Schema 
API|https://github.com/apache/solr/blob/8b2e99721e28c66b96654d0dff440d74785febb2/solr/core/src/java/org/apache/solr/schema/ManagedIndexSchema.java#L1314]
 and [schema.xml 
loading|https://github.com/apache/solr/blob/8b2e99721e28c66b96654d0dff440d74785febb2/solr/core/src/java/org/apache/solr/schema/IndexSchema.java#L704]
 go through it, so the [boolean 
coercion|https://github.com/apache/solr/blob/8b2e99721e28c66b96654d0dff440d74785febb2/solr/core/src/java/org/apache/solr/schema/SchemaField.java#L479]
 is unreachable for new keys today. Step one is to stop rejecting.

*Rolling upgrade is therefore the other way round.* A managed schema with 
{{description="..."}} as an XML attribute fails core loading on every node not 
yet upgraded. The loader reads only the attributes of {{<field>}} and ignores 
child elements, so a child {{<metadata>}} element is upgrade-safe and 
attributes are not. I would persist everything, {{description}} included, as 
the child element.

*Field types already behave the way fields should.* 
[getNonFieldPropertyArgs|https://github.com/apache/solr/blob/8b2e99721e28c66b96654d0dff440d74785febb2/solr/core/src/java/org/apache/solr/schema/FieldType.java#L1351]
 returns a type's extra arguments as strings; carving {{description}} and 
{{metadata}} out before {{parseProperties}} would make fields consistent with 
that.

Smaller notes:
* SolrJ already round-trips a nested {{metadata}} object 
([AddField|https://github.com/apache/solr/blob/8b2e99721e28c66b96654d0dff440d74785febb2/solr/solrj/src/java/org/apache/solr/client/solrj/request/schema/SchemaRequest.java#L122],
 
[getField()|https://github.com/apache/solr/blob/8b2e99721e28c66b96654d0dff440d74785febb2/solr/solrj/src/java/org/apache/solr/client/solrj/response/schema/SchemaResponse.java#L307]);
 in the response it should be a {{Map}} so it renders as an object regardless 
of {{json.nl}}.
* The Admin UI Schema Browser reads [Luke's 
flags|https://github.com/apache/solr/blob/8b2e99721e28c66b96654d0dff440d74785febb2/solr/core/src/java/org/apache/solr/handler/admin/LukeRequestHandler.java#L786],
 not {{/schema/fields}}, so the tooltip needs a Luke change or a data-source 
switch.
* Elasticsearch's [field 
meta|https://www.elastic.co/docs/reference/elasticsearch/mapping-reference/mapping-field-meta]
 is a capped string-to-string map with no separate description attribute. Worth 
asking whether {{description}} is just a conventional key in {{metadata}}.

Happy to take the SolrJ part once the server side settles.

> Extend Solr schema with field-level metadata
> --------------------------------------------
>
>                 Key: SOLR-18674
>                 URL: https://issues.apache.org/jira/browse/SOLR-18674
>             Project: Solr
>          Issue Type: Improvement
>            Reporter: Andrzej Bialecki
>            Assignee: Andrzej Bialecki
>            Priority: Major
>
> h2. Problem
> Schema fields carry only indexing and storage properties. There is no 
> supported way to record what a field is {*}for{*}, such as a human 
> description, its purpose, owner, or semantic role. Teams keep this in wikis 
> or in {{_comment}} XML comments, which are lost on Schema API round-trips. 
> Admin UI users, tooling, and LLM/agent clients that inspect a schema get no 
> semantic context.
> Especially for LLM/agent clients this additional information can reduce the 
> effort (and error rate) of data discovery.
> h2. Current behavior
>  * {{SchemaField}} keeps unrecognized attributes in its {{args}} map 
> ({{{}SchemaField.java:49{}}}).
>  * {{getNamedPropertyValues(false)}} coerces every value to a boolean 
> ({{{}SchemaField.java:472-478{}}}). An arbitrary string attribute such as 
> {{description="..."}} is returned by the Schema API as {{{}false{}}}.
>  * Nothing is documented or validated, and {{{}copyField{}}}, dynamic fields 
> and field types have no equivalent.
> h2. Proposal
> Add an optional, non-functional {{metadata}} concept to fields, dynamic 
> fields, and field types:
>  * *First-class {{{}description{}}}:* a free-text string attribute. Also 
> support {{{}purpose{}}}, a short free-text string.
>  * *Generic {{metadata}} map:* string keys to string values, for user-defined 
> entries such as {{{}owner{}}}, {{{}pii=true{}}}, or {{{}source-system{}}}. 
> Keys are namespaced by convention, e.g. {{{}team.owner{}}}.
>  * *Schema API:* {{{}add-field{}}}, {{replace-field}} and the other field 
> commands accept these attributes. {{GET /schema/fields}} returns them as 
> strings. A new {{includeMetadata}} parameter, default true, controls this.
>  * *Managed schema XML:* serialized as attributes, or as a child 
> {{<metadata>}} element for the map. Classic {{schema.xml}} parsing accepts 
> both forms.
>  * *Semantics:* metadata never affects indexing, query behavior, or the 
> schema's compatibility checks. Changing only metadata should not require a 
> reindex, and replace-field should say so in its response.
>  * *Constraints:* maximum key and value lengths, plus a total size cap, to 
> keep ZooKeeper nodes small.
> h2. Design notes
>  # Fix the boolean coercion first. Store metadata in a dedicated field on 
> {{SchemaField}} rather than reusing {{{}args{}}}.
>  # Reserve the {{metadata}} and {{description}} names so they are not treated 
> as field properties.
>  # SolrJ gets typed accessors in {{SchemaRepresentation}} and the schema 
> request classes. Admin UI shows the description as a tooltip and column.
>  # Ref guide: document the attributes in the field and Schema API pages. Add 
> a changelog entry once a JIRA or PR exists.
> h2. Compatibility and risks
>  * *Backward compatible:* old schemas load unchanged. Older nodes that see 
> the new attributes already ignore them, but the boolean-coercion bug may 
> break their Schema API output. That matters for rolling upgrades and needs a 
> test.
>  * *Size growth in ZooKeeper:* mitigated by the length caps above.
>  * *Overlap:* other attributes might be better handled separately, such as 
> {{useDocValuesAsStored}} and similar functional flags. Keep this proposal 
> strictly descriptive.
> h2. Open questions
>  * Should metadata also be allowed on {{copyField}} and on the schema as a 
> whole?
>  * Fixed set of keys, such as {{purpose}} as an enum, or a free-form map only?
>  * Common set of recommended keys that follow some existing recommendation? 
> Examples:
>  ** [https://mlcommons.org/working-groups/data/croissant/]
>  ** 
> [https://blog.open-metadata.org/announcing-openmetadata-2-0-the-open-context-layer-for-ai-agents-83b8ce8b9dde]
>  ** [https://docs.databricks.com/aws/en/uc-semantics/agent-metadata]
>  * Should {{description}} support localization, or just one language?
> h2. Plan
>  # Fix coercion, add the {{SchemaField}} metadata storage and XML round-trip, 
> with tests.
>  # Schema API read and write support, plus SolrJ.
>  # Dynamic fields, field types, and Admin UI.
>  # Docs and changelog.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to