adriangb commented on issue #18717:
URL: https://github.com/apache/datafusion/issues/18717#issuecomment-3555074557
> I don't think I understand how `->` syntax relates to hashable union
arrays or cross-type comparisons, sorry?
I don't know how variant plans on handling this in general but imaging you
have the data `[{"id":123},{"id":"some-uuid"}]`. This can happen if e.g. two
different systems send data to the same data store and they don't agree on what
a super generic field like `id` means. Then if you want to do a query such as
"the id is equal to 123" what you want is `[true, false]`. How we write that
query with variant I'm not sure but in Postgres JSON syntax that would be
`col->'id' = 123`. For our implementation of json stuff this returns a Union
array. Again not sure how variant plans on handling this but I'd imagine a
Union array makes sense.
> Meanwhile tho -- I would think it's an engine decision whether
variant-type columns should support the same sugar as JSON pathing (`->` here).
In spark SQL, for example, the [`:`
sugar](https://docs.databricks.com/aws/en/sql/language-manual/functions/colonsign)
can be used to path into both variant and JSON columns and
`variant_column:a.b.c::INT` would be equivalent to `variant_get(variant_column,
"a.b.c", Int32)`. Without a `::` (= cast), it just extracts a sub-object of the
same type as the input (JSON or variant).
I agree that what syntax is used as sugar for variant access should be up
the engine, but I think multiple of them will end up facing the same problem,
besides solving the low level problem of making UnionArray more usable seems
like a good thing in general for the ecosystem that probably has other use
cases outside of variant/json.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
