HwangDongJun commented on code in PR #12831: URL: https://github.com/apache/gluten/pull/12831#discussion_r4359549691
########## cpp/velox/compute/VeloxBackend.cc: ########## @@ -191,6 +192,9 @@ void VeloxBackend::init( hiveConnectorConfig_ = createHiveConnectorConfig(backendConf_); + // Make sure the statically linked OpenSSL can find trusted certificates before any filesystem creates an HTTP client. + ensureTlsTrustStore(); Review Comment: Copilot points out that `setenv` can race with other threads reading the environment. Not changed: the call is made once, early in `VeloxBackend::init`, before any filesystem is registered, and `NativeBackendInitializer` guards the init with `compareAndSet`, so it runs once per JVM. I cannot rule out other threads (JVM, Spark) reading the environment at that moment. The alternative, setting `caFile` on each client, would only cover S3, not GCS and ABFS, which use the same libcurl. This is a trade-off for the maintainers to decide. ########## cpp/velox/utils/TlsTrustStore.cc: ########## @@ -0,0 +1,126 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "utils/TlsTrustStore.h" + +#include <glog/logging.h> +#include <cctype> +#include <cstdlib> +#include <filesystem> + +namespace gluten { + +namespace { + +namespace fs = std::filesystem; + +bool isRegularFile(const std::string& path) { + std::error_code ec; + return fs::is_regular_file(path, ec) && !ec; +} + +// OpenSSL finds certificates in a directory only through files named <8 hex digits>.<digit>, created by +// `openssl rehash`. +bool isHashedCertificateName(const std::string& name) { + if (name.size() < 10 || name[8] != '.') { + return false; + } + for (size_t i = 0; i < 8; ++i) { + if (!std::isxdigit(static_cast<unsigned char>(name[i]))) { + return false; + } + } + for (size_t i = 9; i < name.size(); ++i) { + if (!std::isdigit(static_cast<unsigned char>(name[i]))) { + return false; + } + } + return true; +} + +bool hasHashedCertificate(const std::string& dir) { + std::error_code ec; + fs::directory_iterator it(dir, fs::directory_options::skip_permission_denied, ec); + if (ec) { + return false; + } + for (const auto& entry : it) { + if (isHashedCertificateName(entry.path().filename().string())) { + return true; + } + } + return false; +} + +std::optional<std::string> getEnv(const char* name) { + const char* value = std::getenv(name); + return value != nullptr ? std::optional<std::string>(value) : std::nullopt; +} + +} // namespace + +const TlsSearchPaths& systemTlsSearchPaths() { + static const TlsSearchPaths paths{ + "/etc/ssl/cert.pem", + "/etc/ssl/certs", + { + "/etc/ssl/certs/ca-certificates.crt", // Debian, Ubuntu, Alpine + "/etc/pki/tls/certs/ca-bundle.crt", // RHEL, CentOS, Rocky, Alma, Fedora + "/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem", // RHEL family, what the link above points to + "/etc/ssl/ca-bundle.pem", // openSUSE + }}; + return paths; +} Review Comment: Copilot suggests reading the defaults with `X509_get_default_cert_file()` / `X509_get_default_cert_dir()`. Not changed: no Gluten source includes the OpenSSL headers today, and `find_package(OpenSSL)` is only set up for `VCPKG_DYNAMIC_OPENSSL=ON`, so I could not confirm the include path is available in every build configuration. The vcpkg openssl port uses `--openssldir=/etc/ssl`, and `OPENSSLDIR: "/etc/ssl"` is in the built `libvelox.so`. Added a comment in 6d8af40a8. Can switch to the OpenSSL functions if you prefer; I would need a CI build to confirm it compiles everywhere. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
