prodeezy commented on issue #2962: URL: https://github.com/apache/iceberg/issues/2962#issuecomment-898091999
Sample data written using Iceberg v0.9.0 (with parquet 1.11.0) : https://drive.google.com/drive/folders/1GckUAYZpOYJQtkTFtavKzpqxKzXkCbqy?usp=sharing Fails when reading using code from tag: apache-iceberg-0.10.0 ``` spark-shell --jars spark-runtime/build/libs/iceberg-spark-runtime-0.10.0.jar > spark.read.format("iceberg").load("iceberg_issue_2962_parquet_1_11_0").show(false) [Stage 0:> (0 + 1) / 1]2021-08-12 17:58:28 ERROR Executor:91 - Exception in task 0.0 in stage 0.0 (TID 0) java.lang.IllegalArgumentException: [segmentMembership, map, key] required binary key (STRING) = 2 is not in the store: [] 8 ``` If i switch to parquet 1.11.0 and rebuild, it starts reading just fine. ``` scala> spark.read.format("iceberg").load("/tools/iceberg_issue_2962_parquet_1_11_0").show(false) +------------------------------------------------------------------------------------------------+ |segmentMembership | +------------------------------------------------------------------------------------------------+ |[AEPSegmentMatch -> []] | |[AEPSegmentMatch -> []] | |[AEPSegmentMatch -> []] | |[AEPSegmentMatch -> []] | |[AEPSegmentMatch -> [cea3dda5-5604-4548-a031-148867e425b5 -> [, existing, 2021-04-28 03:34:36]]]| |[AEPSegmentMatch -> [cea3dda5-5604-4548-a031-148867e425b5 -> [, existing, 2021-04-28 03:34:36]]]| |[AEPSegmentMatch -> [cea3dda5-5604-4548-a031-148867e425b5 -> [, existing, 2021-04-28 03:34:37]]]| |[AEPSegmentMatch -> [cea3dda5-5604-4548-a031-148867e425b5 -> [, existing, 2021-04-28 03:34:37]]]| +------------------------------------------------------------------------------------------------+ ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
