comphead commented on issue #6957: URL: https://github.com/apache/datafusion-comet/issues/6957#issuecomment-6779501161
Hit this on a Spark 4.0 + Iceberg 1.9 job on EKS: a `HadoopFileIO` table on S3 with IRSA credentials. Every Iceberg write planned as `IcebergWrite` instead of `CometIcebergWrite`. The effective Hadoop configuration has four keys that the S3A allowlist on `main` rejects: - `fs.s3a.aws.credentials.provider`, set to the SDK v2 `EnvironmentVariableCredentialsProvider,WebIdentityTokenFileCredentialsProvider`, which S3A needs to use IRSA - `fs.s3a.connection.maximum` and `fs.s3a.threads.max` - `fs.s3a.region`, which is not a Hadoop key but comes from the cluster's Spark defaults The native Iceberg scan runs with the same settings, and `main` already has a native IRSA provider (`native/core/src/cloud/s3/web_identity.rs`). So mapping the environment and web-identity providers (part 2) and admitting the pool sizes (part 1) would unblock this deployment. For context, in a `rewrite_data_files` benchmark over 24,000 small files, Iceberg's procedure with only the scan native was about 8x faster than Spark. Each rewrite task is now CPU-bound, and the native scan takes about 0.65 s of each ~5 s task. The rest is `CometColumnarToRow` and Iceberg's Java writer. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
