unikdahal opened a new issue, #5319:
URL: https://github.com/apache/datafusion-comet/issues/5319
### What is the problem the feature request solves?
#### What is the problem the feature request solves?
When running Copy-On-Write (CoW) table operations such as MERGE INTO,
UPDATE, or DELETE against Iceberg tables in Spark Iceberg's Spark planning
extension creates a
SparkCopyOnWriteScan
(org.apache.iceberg.spark.source.SparkCopyOnWriteScan) for scanning the target
table data files.
Found the gap while testing native iceberg writes #4487
Currently, IcebergReflection.ICEBERG_SCAN_CLASSES only includes:
• org.apache.iceberg.spark.source.SparkBatchQueryScan
• org.apache.iceberg.spark.source.SparkStagedScan
Because SparkCopyOnWriteScan is missing from ICEBERG_SCAN_CLASSES,
IcebergReflection.isIcebergScanClass(...) returns false when CometScanRule
evaluates a physical plan
containing a SparkCopyOnWriteScan. As a result, Comet fails to recognize
SparkCopyOnWriteScan as a supported Iceberg scan and falls back to JVM Spark
execution for the scan
phase of Copy-On-Write queries.
### Describe the potential solution
Add SparkCopyOnWriteScan
("org.apache.iceberg.spark.source.SparkCopyOnWriteScan") to
IcebergReflection.ClassNames and include it in
IcebergReflection.ICEBERG_SCAN_CLASSES.
// In IcebergReflection.scala
object ClassNames {
...
val SPARK_COPY_ON_WRITE_SCAN =
"org.apache.iceberg.spark.source.SparkCopyOnWriteScan"
}
val ICEBERG_SCAN_CLASSES: Set[String] =
Set(
ClassNames.SPARK_BATCH_QUERY_SCAN,
ClassNames.SPARK_STAGED_SCAN,
ClassNames.SPARK_COPY_ON_WRITE_SCAN)
This enables CometScanRule to recognize SparkCopyOnWriteScan instances and
convert them to native Comet scans during Iceberg Copy-On-Write MERGE, UPDATE,
and DELETE
operations.
### Additional context
SparkCopyOnWriteScan extends Iceberg's standard Spark scan primitives
(SparkBatchQueryScan / SparkScan) and behaves identically with respect to file
tasks, table schemas, and
filter pushdowns during data file reads.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
