All, the only documentation about the File Metadata ( hidden_metadata struct) I can seem to find is on the databricks website https://docs.databricks.com/en/ingestion/file-metadata-column.html#file-metadata-column for reference here is the struct:
As far as I can tell this feature was released as part of spark 3.20 based on this stack overflow post unfortunately I wasn’t able to locate this in the release notes. Though I may have missed it somehow. So I have the following questions and seeking guidance from the list at how to best approach this
Would both of these be issues that should be opened in JIRA? Both of these seem like simple and useful things to add but are above my ability to submit PR’s for without some guidance. I’m happy to help especially with a documentation PR’ if someone can confirm and get me started in the right direction. I don’t really have the java / scala skills needed to implement the feature. Thanks for any pointers --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org |