andygrove opened a new issue, #7150: URL: https://github.com/apache/datafusion-comet/issues/7150
### What is the problem the feature request solves? `ParquetWriteWithMagicCommitterSuite`, added in #6940, is the only suite that runs native Parquet writes through S3A's magic committer against a real S3 API. Like the other MinIO-backed suites, it needs Docker for testcontainers, so it is on the manual list in `dev/ci/check-suites.py` and no CI job runs it. A change that breaks the magic committer route, such as the close ordering, the abort of a failed file, or which committer configurations the planning gate admits, would pass CI. The other manual MinIO suites have the same gap: `ParquetReadFromS3Suite`, `CsvReadFromS3Suite`, `IcebergReadFromS3Suite`, `CometS3CredentialBridgeSuite` and `HadoopS3ACredentialProviderAdapterBridgeSuite`. ### Describe the potential solution Add a job like `contrib-delta-s3` in `.github/workflows/delta_contrib_test.yml`. It runs a MinIO-backed suite on a plain `ubuntu-24.04` runner, where testcontainers can start MinIO. Running `ParquetWriteWithMagicCommitterSuite` on Spark 3.5 and 4.1 would cover both native writers (`CometNativeWriteExec` and `CometWriteFilesExec`) and both Hadoop lines (3.3.4 and 3.4). The suite takes about 20 seconds once Comet is built. The read suites could join the same job. Two things to keep in mind: - `CometS3TestBase` builds its bucket setup client without a region, so the job needs `AWS_REGION=us-east-1` on a runner with no AWS config. - The suite sets `fs.s3a.committer.magic.enabled=true` for its own session, and the shared test configuration turns the magic committer off for every other suite. The job should run the suite on its own. ### Additional context Raised by @parthchandra in the review of #6940, as a follow-up. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
