sdf-jkl opened a new issue, #11627:
URL: https://github.com/apache/arrow-rs/issues/11627
### Is your feature request related to a problem or challenge?
VariantPath has evolved toward Spark/Databricks-style singular access, with
RFC 9535 used to resolve individual syntax and evaluation questions. We
should
document the intended subset and how compatibility exceptions are handled.
The same accepted string can currently select different keys across engines.
Arrow interprets `['a\nb']` as anb and `['a\]b']` as a]b. Spark currently
retains
those backslashes. Proposed Spark PR apache/spark#59097 decodes recognized
escapes such as
`\n`, but retains unknown escapes such as `\]`. RFC 9535 rejects unknown
escapes.
These examples are literal path segments; Spark and RFC paths start with $.
### Describe the solution you'd like
Agree on and document VariantPath's grammar and compatibility target:
- Is the baseline Spark Variant paths, an RFC 9535 subset, or an Arrow
dialect?
- Specify quote, backslash, Unicode, control-character, and malformed-escape
rules, including whether legacy accepted paths retain their interpretation.
- Specify root spelling, shorthand field names, and index syntax/ranges.
- Distinguish the grammar accepted by VariantPath from consumer restrictions:
shredding schema paths support `[*]`, while `variant_get` accepts singular
paths.
- Add a small compatibility matrix and conformance tests after agreement.
### Describe alternatives you've considered
- Follow Spark/Databricks Variant paths, documenting Arrow-specific
extensions
such as schema wildcards and any version-dependent compatibility choices.
- Adopt the relevant subset of RFC 9535, explicitly documenting differences
from Spark/Databricks.
- Preserve the current Arrow dialect and document its differences from both.
Continuing to decide syntax case by case leaves callers without a stable
compatibility contract. Any change to existing escape interpretation needs an
explicit compatibility decision.
### Additional context
Lineage:
- #7893: kernel proposal based on Databricks `variant_get`.
https://github.com/apache/arrow-rs/issues/7893
- #8166 / #8280: dot splitting introduced; scovich warned that escaping
needed
a defined contract.
https://github.com/apache/arrow-rs/pull/8166#discussion_r2286247849
- #8954: escaping request; Databricks/RFC differences explicitly raised.
https://github.com/apache/arrow-rs/issues/8954#issuecomment-3615598189
- #9012 / #9050: array indexing and a planned Databricks-style bracket
subset.
https://github.com/apache/arrow-rs/pull/9012#discussion_r2640273680
- #9276: arbitrary backslash escaping introduced, initially for unquoted
bracket fields; its description explicitly shows `[a\xb]` selecting axb.
https://github.com/apache/arrow-rs/pull/9276
- #9479: require quoted fields and accept both quote styles, consulting RFC
and Databricks documentation; existing escape decoding remained unchanged.
https://github.com/apache/arrow-rs/pull/9479#discussion_r2854728284
- #9606 / #9676: evaluation mismatches return missing results, independently
of casting; explicit Spark compatibility goal. This did not settle
escaping.
https://github.com/apache/arrow-rs/issues/9606
https://github.com/apache/arrow-rs/pull/8354#issuecomment-4104436183
- #10635: `[*]` for schema construction, rejected by singular `variant_get`.
https://github.com/apache/arrow-rs/pull/10635#discussion_r3980950307
- #11228 / #11231: quote-aware bracket boundaries; explicitly preserve `\]`
and defer broader escape decisions.
https://github.com/apache/arrow-rs/issues/11228
- Proposed Spark escape handling:
https://github.com/apache/spark/pull/59097
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
