HwangDongJun opened a new issue, #12711: URL: https://github.com/apache/gluten/issues/12711
### Description ## Summary This is a follow-up to #10670. We independently hit the exact same symptom (Gluten native S3 access hangs with no error, works fine when Gluten is disabled) on our own environment, and were able to identify the root cause and a reproducible fix. Since #10670 was closed without a concrete resolution, and the same issue is very likely to keep recurring for anyone running official/nightly Gluten binaries on a non-RHEL-family container image, we'd like to share the findings here and propose a small documentation improvement. ## Environment where the issue was reproduced - Gluten versions: 1.6.0, 1.7.0-SNAPSHOT (both official/nightly binaries, no custom build) - Spark: 4.0.x, Velox backend - Runtime container base image: Ubuntu (via `apache/spark` official Docker image with `-ubuntu` tag) - Storage: S3-compatible object storage (non-AWS, MinIO-based) - Symptom: with `spark.gluten.sql.columnar.batchscan=true`, queries hang indefinitely with no error/exception in driver or executor logs. Disabling Gluten (`spark.gluten.enabled=false`) makes the exact same query complete normally. ## Root cause Gluten's native library (`libvelox.so`) is compiled on CentOS/Rocky Linux as part of the official build/release pipeline. The native AWS SDK (used via curl) that Velox links against resolves the CA bundle from the RHEL-family standard path `/etc/pki/tls/certs/ca-bundle.crt`, which is hardcoded at build/link time. On a non-RHEL-family runtime (Ubuntu/Debian, where the CA bundle instead lives at `/etc/ssl/certs/ca-certificates.crt`), this path does not exist. The native curl call then fails the TLS handshake (`curlCode: 77`), and an internal exponential-backoff retry loop kicks in — from the Spark driver/executor's point of view this looks exactly like a silent hang, with no exception surfaced to Spark/JVM logs. This matches the disclaimer on the official [Downloads page](https://gluten.apache.org/downloads/): > Please note that the binary files were compiled on CentOS 7 using static linking... Performance and runtime compatibility are not guaranteed on other operating systems or hardware platforms. We confirmed: - The problem reproduces identically regardless of which Gluten jar is used (official release build, nightly build, or a jar built via the project's own CentOS/Rocky-based CI pipeline) — the jar's origin doesn't matter, since all of them are compiled on the same OS family. - Setting standard environment variables such as `SSL_CERT_FILE` or `CURL_CA_BUNDLE` has **no effect** — these are not honored by the native curl call inside the Velox/AWS SDK layer. This is a common first thing people try (understandably, since it works for many other tools), so it's worth explicitly documenting that it does *not* work here. ## Fix No Gluten source code or jar rebuild is required. Adding a single symlink at container image build time (as root) is sufficient: ```dockerfile RUN mkdir -p /etc/pki/tls/certs && \ ln -sf /etc/ssl/certs/ca-certificates.crt /etc/pki/tls/certs/ca-bundle.crt ``` After applying this, `batchscan=true` (native scan) works correctly with no hang, and we further confirmed it is consistently faster than the `batchscan=false` (JVM fallback) workaround across multiple query patterns and submission methods — typically in the range of roughly 1.3x–2.2x, with identical query results in all cases. ## Proposed enhancement Since this isn't something Gluten can unilaterally "fix" for every possible runtime OS (as the Downloads page already acknowledges), we'd like to propose adding a short note to the relevant getting-started documentation (e.g. `docs/get-started/VeloxS3.md` or similar), along the lines of: > If you deploy the official Gluten binaries on a non-RHEL-family base image (e.g. Ubuntu/Debian), native S3 access may silently hang due to a hardcoded CA bundle path expectation. Symlinking your distribution's CA bundle to `/etc/pki/tls/certs/ca-bundle.crt` resolves this. Note that `SSL_CERT_FILE`/`CURL_CA_BUNDLE` environment variables are not effective for this. This would save future users (especially anyone deploying on Debian/Ubuntu based Spark images, which are quite common) significant debugging time, as the current symptom (silent hang, no errors) makes this very hard to diagnose from first principles — as evidenced by #10670 remaining unresolved for months. Happy to help draft the doc PR if this is welcome. --- *This issue was written with the assistance of AI (used to help organize and phrase the investigation notes).* ### Gluten version main branch -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
