thisisnic commented on issue #20145: URL: https://github.com/apache/arrow/issues/20145#issuecomment-5868332630
_This comment was written by Claude (an AI assistant) at @thisisnic's request, after investigating the issue against current `main`._ This isn't how Hive partitioning works in general; it's an R-specific bug, and it still reproduces on current `main`. The C++ `FileSystemDatasetFactory` handles this case correctly: it sets `partition_base_dir` to the selector's base directory (https://github.com/apache/arrow/blob/65974f5316ab9404605e7f5a2fa6c890e9c42888/cpp/src/arrow/dataset/discovery.cc#L180-L182) and strips it from every path before inferring partitions (https://github.com/apache/arrow/blob/65974f5316ab9404605e7f5a2fa6c890e9c42888/cpp/src/arrow/dataset/discovery.cc#L257). So `foo=bar` in the root directory name is never seen as a partition segment on the C++ side. The problem is the R-side validation in `handle_partitioning()`, added in ARROW-10485. When `partitioning` is a character vector or a `Schema` and `hive_style` is `NA` or `TRUE`, it lists every path under the root with `fs$ls(path, recursive = TRUE)`, which returns full paths, and passes them straight to `HivePartitioningFactory$Inspect()` without stripping the base directory (https://github.com/apache/arrow/blob/65974f5316ab9404605e7f5a2fa6c890e9c42888/r/R/dataset-factory.R#L90-L95). `Inspect()` then sees `.../foo=bar/am=0/part-0.parquet` and detects `c("foo", "am")`, which fails the comparison against `"am"`. Consistent with that, the code paths that skip the R check all work on the reprex above: `open_dataset(subdir)` with the default `hive_partition()`, `open_dataset(subdir, partitioning = hive_partition(am = int32()))`, and `open_dataset(subdir, partitioning = "am", hive_style = FALSE)` all give a dataset with only `am` as the partition column. Only `partitioning = "am"` and `partitioning = schema(am = int32())` error. The fix is to make the R pre-check inspect paths relative to the dataset root, matching what C++ does, by stripping `path_and_fs$path` from the front of the paths returned by `fs$ls()` before calling `Inspect()`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
