HwangDongJun commented on PR #12831: URL: https://github.com/apache/gluten/pull/12831#issuecomment-5363818614
@zhouyuan Good question. Short answer: no SDK modification needed for either — both official SDKs already expose the same kind of "override CA bundle path" option that AWS SDK C++'s `caFile` gave us for S3: - **Azure (ABFS)**: Azure SDK C++'s `Azure::Core::Http::CurlTransportOptions` has `CAInfo`/`CAPath` fields. Velox's `AzureClientProviderImpl` currently constructs `BlobClient`/`DataLakeFileClient` without passing any custom `BlobClientOptions`/`DataLakeClientOptions`, so it just falls through to libcurl's compiled-in default — same underlying issue as S3. Fix would be the same shape: build a `CurlTransportOptions` with `CAInfo` set from a new `fs.azure.ssl.ca-file` config (or `SSL_CERT_FILE` fallback) and pass it into the client options. - **GCS**: google-cloud-cpp has `google::cloud::CARootsFilePathOption`, settable directly on the `google::cloud::Options` object Velox's `GcsFileSystem::Impl::initializeClient()` already builds. Just needs `options.set<gcs::CARootsFilePathOption>(path)` added there. On the Gluten side: ABFS config passthrough (`getAbfsHiveConfig`) already forwards all `spark.hadoop.fs.azure.*` configs verbatim, so a new `fs.azure.ssl.ca-file` config would work with zero changes there. GCS passthrough is an explicit allowlist, so it'd need one new entry. One caveat I want to be upfront about: unlike S3, we don't have Azure Blob Storage or GCS test infrastructure on our end, so we can't reproduce/verify this the same way (real hang → real fix, A/B tested). We could still write the patches following the same pattern and the SDKs' documented options, but it'd be compile-verified only, not runtime-verified against live storage. Given that, would you prefer this as a follow-up to this PR (once the S3 fix here is settled), or a separate issue/PR so it doesn't hold up review here? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
