wjones127 opened a new issue, #10987: URL: https://github.com/apache/datafusion/issues/10987
I've been starting to experiment with implementing the Open Variant Type [^1] in Rust / DataFusion. There is a specification and Java library for this, and Spark will release this type in 4.0. There are also plans to integrate this into table formats such as Delta Lake [^3] and Iceberg [^4]. This would be a high-performance data type for semi-structured data, designed for better OLAP performance than JSON or BSON (discussed in #7845). I've discussed a little bit in the Arrow repo about it's potential as an Arrow extension type [^2]. I'm working on creating an extension similar to [datafusion-functions-json](https://github.com/datafusion-contrib/datafusion-functions-json). If we could create a new repo `datafusion-functions-variant`, I'd be happy to develop that in the open. [^1]: https://github.com/apache/spark/tree/master/common/variant [^2]: https://github.com/apache/arrow/issues/42069 [^3]: https://www.databricks.com/blog/introducing-open-variant-data-type-delta-lake-and-apache-spark [^4]: https://lists.apache.org/thread/xnyo1k66dxh0ffpg7j9f04xgos0kwc34 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
