sdf-jkl commented on PR #117: URL: https://github.com/apache/parquet-testing/pull/117#issuecomment-4960945584
No. I checked the implementations I could find that use this corpus, and they either consume it as a git submodule (arrow-rs, arrow-go, arrow-dotnet, arrow-cpp, datafusion) or pin it to an explicit revision. iceberg-go pins a `REF` in `gen_fixtures.sh`, elasticsearch fetches from a pinned commit hash and excludes `shredded_variant/` anyway, and parquet-go vendors a subset into its own testdata. For all of those, merging is a no-op until they choose to bump. And when they bump, they are fine. `-INVALID` is an existing convention (cases 43, 84 and 125 already use it) that consumers key off generically rather than by case number. arrow-go skips any case whose `parquet_file` contains `-INVALID` and prints the `notes` ([`variant_test.go#L229`](https://github.com/apache/arrow-go/blob/main/parquet/pqarrow/variant_test.go#L229)), and arrow-dotnet does the same in its conformance suite. Both read the filenames from `cases.json`, which this PR updates in lockstep. No cases are added, removed or reordered, and no file contents change, so positional indexing into the array (as arrow-rs does) still works, and implementations that choose to leniently read the shredded value still can. The one project I found that this does affect is [hardwood](https://github.com/hardwood-hq/hardwood). It shallow-clones parquet-testing master at HEAD (unpinned), enumerates `shredded_variant/*.parquet` by directory walk rather than from `cases.json`, and keys its skip-lists on exact filenames, so cases 41, 131 and 138 would stop being skipped. Small fix on their side, and I have opened an issue there: hardwood-hq/hardwood#783. I cannot claim to have found every consumer. parquet-testing has no dependency graph, so this is not exhaustively enumerable. If any implementer pins to master, please shout. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
