sandugood opened a new issue, #4774: URL: https://github.com/apache/datafusion-comet/issues/4774
### Describe the bug When running the same query (as presented in the #4723) just from logs it can be seen that Comet skips a rather big stage. Default Spark has both of these stages at the beginning of the execution: [Stage 0:> (0 + 0) / 7350][Stage 1:> (0 + 0) / 7350] Comet has only one stage with 7350 tasks. It might be relative to FULL OUTER join. Because when we are forming features with a 30-day, 180-day, 365-day windows everything seems fine and resulting values are the same across both engines. However when performing FULL OUTER join for the end result - we get significantly smaller values on Comet's side. ### Steps to reproduce _No response_ ### Expected behavior Almost identical (or identical down to floating-point precision) results between default Spark and Comet ### Additional context 1. Iceberg V2 2. Positional (row-level) deletes. 3. MOR tables 4. Hive for metastore We are using explicit keys with `spark.sql.catalog.<cat>.s3.access-key-id` and also a path-style access (`spark.sql.catalog.iceberg.s3.path-style-access`). Here is full catalog config for the SparkSession that I am testing Comet on. spark.sql.catalog.iceberg.s3.secret-access-key `secret_key` spark.sql.catalog.iceberg.type `hive` spark.sql.catalog.iceberg `org.apache.iceberg.spark.SparkCatalog` spark.sql.catalog.iceberg.s3.path-style-access `true` spark.sql.catalog.iceberg.io-impl `org.apache.iceberg.aws.s3.S3FileIO` spark.sql.catalog.iceberg.s3.endpoint `https://endpoint:port/` spark.sql.catalog.iceberg.client.region `dummy` spark.sql.catalog.iceberg.uri `thrift://hms_endpoint:port` spark.sql.catalog.iceberg.s3.separator `/` spark.sql.catalog.iceberg.s3.access-key-id `access_key` Tested the same query agains default Spark3.5/4.0/4.1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
