jackylee-ch opened a new pull request, #12495: URL: https://github.com/apache/gluten/pull/12495
## What changes are proposed in this pull request? This PR adds an explicit, credential-safe path for forwarding user-provided Hadoop filesystem configuration from Spark to each native Velox runtime. Spark's SQL configuration propagation does not carry bare `fs.*` settings, so executor-side native reads and writes can otherwise miss the filesystem configuration used by the originating Spark session. - Introduce a Hadoop configuration contributor SPI and collector that keeps user-provided filesystem settings, normalizes `fs.*` and `spark.hadoop.fs.*` keys, and caches an immutable snapshot once per `SparkSession`. - Carry that snapshot through whole-stage RDD execution and native writer/schema paths as dedicated serializable state, without embedding credentials in Substrait plans. - Overlay filesystem settings when each Velox runtime is created, keep Gluten control keys authoritative, and configure Hive, Iceberg, Delta, CUDF, and GCS writer connectors per runtime. - Derive runtime identity without credential values, extend filesystem and UGI redaction, and fail closed when GCS credential configuration is inconsistent. - Preserve existing Scala and C++ overloads, constructors, and default delegation so existing backend integrations remain source and binary compatible. ClickHouse source paths are unchanged. ## How was this patch tested? - Spark 3.5: the 10-module reactor install and a clean Velox backend compile/test completed successfully. - Spark 3.3 with Delta: the 10-module clean `test-compile` completed successfully. - Targeted Scala suites passed 43/43 tests: `HadoopConfCollectorSuite`, `HadoopConfTransportSuite`, `RuntimesSuite`, `GlutenRuntimeConfigSuite`, `VeloxHadoopConfTransportSuite`, and `VeloxHadoopConfWriterSuite`. This includes a real Spark closure serializer round trip. - Native tests passed 7/7 cases across `GlutenConfigTest` and `RuntimeTest`. - `VeloxParquetWriteHadoopConfSuite` passed 1/1 using the real native Parquet writer. - Scala formatting, clang-format 15 validation, license-header validation for all changed source files, and `git diff --check` passed. Known local validation gaps: - The local native build disabled S3, GCS, ABFS, and HDFS support, and no real cloud credentials were available. Credential-backed cloud read/write end-to-end tests and conditional cloud-native branches were therefore not run locally. - `VeloxParquetWriteForHiveSuite` passed 2/16 tests; 14 tests failed with an existing `TableWrite` `std::bad_cast` in the local `velox_ep_pr12331_runsetup` build. An A/B run using the previous connector-registration path produced the same failures, which excludes the new per-runtime connector overlay as the cause, but this suite is not green in the current local environment. ## Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex GPT-5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
