HwangDongJun commented on code in PR #12833:
URL: https://github.com/apache/gluten/pull/12833#discussion_r4144868231
##########
gluten-iceberg/src/main/scala/org/apache/iceberg/spark/source/GlutenIcebergSourceUtil.scala:
##########
@@ -115,6 +115,32 @@ object GlutenIcebergSourceUtil {
)
}
+ /**
+ * Returns the table-level root path(s) so callers can validate the
underlying filesystem
+ * scheme(s) without enumerating every data file.
+ *
+ * We deliberately read this from table metadata (Table.location() plus the
write.data.path /
+ * write.folder-storage.path properties, when set) rather than from the
actual planned scan tasks
+ * (task.file().path()): resolving the real per-file paths requires planning
the Iceberg scan's
+ * input partitions, but doing so eagerly -- before Spark has pushed down
its dynamic partition
+ * pruning runtime filters -- breaks DPP's subquery-reuse detection for
SupportsRuntimeV2Filtering
+ * scans (see https://github.com/apache/gluten/issues/12712).
+ *
+ * This mirrors how the non-Iceberg
BatchScanExecTransformer.getRootPathsInternal resolves root
+ * paths purely from FileIndex metadata (fileScan.fileIndex.rootPaths)
without planning any input
+ * partitions. Just like that metadata-only approach, this does not reflect
files relocated by an
+ * entirely custom LocationProvider that ignores both properties above, but
it is a strict
+ * improvement over unconditionally returning Seq.empty.
+ */
+ def getRootPaths(table: Table): Seq[String] = {
+ val properties = table.properties()
+ Seq(
+ Option(table.location()),
+ Option(properties.get(TableProperties.WRITE_DATA_LOCATION)),
+ Option(properties.get(TableProperties.WRITE_FOLDER_STORAGE_LOCATION))
Review Comment:
Went through the Table/LocationProvider API surface again — there isn't one
that returns a single "the" path for a table, since
LocationProvider.newDataLocation(...) needs a filename to compute a location
(and something like the default ObjectStoreLocationProvider hashes the filename
into the path, so there's no meaningful answer without one). That's actually
part of why I moved to reading the path straight off the planned scan tasks
(per infvg's suggestion above) instead of relying on table-location metadata —
it sidesteps needing a "one true path" API entirely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
