Kontinuation commented on issue #1296:
URL: https://github.com/apache/sedona/issues/1296#issuecomment-2029885038
The `geo` metadata in the parquet footers may not be the same for all
written geoparquet files, especially the bbox field, this makes the default
parquet footer metadata merging process fail with the following exception:
```
java.lang.RuntimeException: could not merge metadata: key geo has
conflicting values:
[{"version":"1.0.0","primary_column":"geom","columns":{"geom":{"encoding":"WKB","geometry_types":["Polygon"],"bbox":[1.0,1.0,9998.0,9998.0],"crs":null}}},
{"version":"1.0.0","primary_column":"geom","columns":{"geom":{"encoding":"WKB","geometry_types":["Polygon"],"bbox":[0.0,0.0,10000.0,10000.0],"crs":null}}}]
at
org.apache.parquet.hadoop.metadata.StrictKeyValueMetadataMergeStrategy.merge(StrictKeyValueMetadataMergeStrategy.java:36)
at
org.apache.parquet.hadoop.metadata.GlobalMetaData.merge(GlobalMetaData.java:106)
at
org.apache.parquet.hadoop.ParquetFileWriter.mergeFooters(ParquetFileWriter.java:1451)
at
org.apache.parquet.hadoop.ParquetFileWriter.mergeFooters(ParquetFileWriter.java:1422)
at
org.apache.parquet.hadoop.ParquetFileWriter.writeMetadataFile(ParquetFileWriter.java:1383)
at
org.apache.parquet.hadoop.ParquetOutputCommitter.writeMetaDataFile(ParquetOutputCommitter.java:84)
at
org.apache.parquet.hadoop.ParquetOutputCommitter.commitJob(ParquetOutputCommitter.java:50)
at
org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.commitJob(HadoopMapReduceCommitProtocol.scala:192)
```
We have to implement an output committer for GeoParquet to merge `geo`
metadata properly. If your usecase do not need to read the geo metadata from
_common_metadata or _metadata file, we can simply ignore geo metadata when
generating such files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@sedona.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
