thijsheijden opened a new issue, #13218:
URL: https://github.com/apache/iceberg/issues/13218
### Query engine
Spark
### Question
Hi everyone! I am attempting to create an Iceberg table using Spark with an
existing group of Parquet files. For my usecase it is important that data
remains in the same files, and the ordering of rows within files remains the
same. This meant 'add_files' was perfect. However, when using the command
```
spark.sql(f"""
CALL iceberg_catalog.system.add_files(
table => 'default.{args.table}',
source_table => '`parquet`.`{batch_dirs[i]}`'
)
""")
```
the generated manifest files contain only the number of rows and the file
size, not the upper and lower bounds which I need to be able to benchmark and
test filtering performance. Is this intended behaviour, or am I writing/reading
my Parquet files wrong. I understand that Pandas does not write row-level
metadata to Parquet files, so I switched to using pyarrow. When using
parquet-tools I can see that the file has valid metadata.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
