That sounds potentially Spark-related to me. Feel free to open a JIRA
issue in Apache Spark if it is happening in Spark. If you can see the
schema metadata in Arrow then the issue is probably there
On Thu, Oct 10, 2019 at 5:11 PM Isaac Myers
wrote:
>
> Thanks for the quick response. When I use py
Thanks for the quick response. When I use pyspark to read a parquet file
written by arrow, I can't see even file-level metadata. Is that also a known
issue? (Note: I searched the JIRA issues and couldn't find any info.)
Sent with ProtonMail Secure Email.
‐‐‐ Original Message ‐‐‐
On Thu
We haven't implemented storing field-level metadata in Parquet files
yet. It's somewhat tricky. See
https://issues.apache.org/jira/browse/ARROW-4359
On Thu, Oct 10, 2019 at 11:51 AM Isaac Myers
wrote:
>
> I can write both field- and schema-level metadata and read the values back
> from schema o
I can write both field- and schema-level metadata and read the values back from
schema or relevant field. I write the schema and table described by the schema
to a local parquet file. Upon reading the table or schema from the parquet
file, only schema metadata are present and field metadata are
