Copilot commented on code in PR #12405:
URL: https://github.com/apache/gluten/pull/12405#discussion_r3559487188
##########
cpp/velox/utils/ConfigExtractor.cc:
##########
@@ -294,6 +294,30 @@ std::string getConfigValue(
return got->second;
}
+std::shared_ptr<facebook::velox::config::ConfigBase> mergeWithSessionOverrides(
+ const std::shared_ptr<facebook::velox::config::ConfigBase>& baseConf,
+ const std::unordered_map<std::string, std::string>& sessionConf) {
+ auto merged = baseConf->rawConfigs(); // copy — we must not mutate the
shared base
+
+ // ── Step 1: forward all per-account and generic fs.azure.* / fs.s3a.* /
fs.gs.*
+ // keys from the session config into the merged map. Session value always
wins.
+ static const std::vector<std::string_view> kForwardPrefixes = {
+ "fs.azure.",
+ "fs.s3a.",
+ "fs.gs.",
+ };
+ for (const auto& [k, v] : sessionConf) {
+ for (const auto& prefix : kForwardPrefixes) {
+ if (k.starts_with(prefix)) {
+ merged[k] = v;
+ break;
+ }
+ }
+ }
Review Comment:
mergeWithSessionOverrides currently only forwards session keys that already
start with "fs.azure." / "fs.s3a." / "fs.gs.". However Spark session configs
are typically provided as "spark.hadoop.fs.*" (see
getAbfsHiveConfig/createHiveConnectorConfig stripping logic). As a result,
session overrides like "spark.hadoop.fs.azure.account.key..." won’t be
propagated into the merged connector config, so the delayed connector
registration still won’t pick up per-session ABFS/S3/GCS settings.
##########
cpp/velox/compute/VeloxRuntime.cc:
##########
@@ -284,23 +283,26 @@ void VeloxRuntime::initializeExecutors() {
void VeloxRuntime::registerConnectors() {
auto* backend = VeloxBackend::get();
+
+ auto merged = mergeWithSessionOverrides(backend->getStaticConnectorConfig(),
veloxCfg_->rawConfigs());
+
Review Comment:
For ABFS, registering connectors with a merged config is usually not enough
by itself: the ABFS client provider is registered separately via
registerAzureClientProvider, and in this codebase it’s registered using the
hive config (see VeloxParquetDataSourceABFS::initSink). If we don’t
(re)register the Azure client provider with the merged config here,
session-level ABFS credentials/settings may still not take effect for reads.
##########
cpp/velox/compute/WholeStageResultIterator.cc:
##########
@@ -141,6 +143,11 @@ WholeStageResultIterator::WholeStageResultIterator(
GLUTEN_CHECK(fileSystem != nullptr, "File System for spilling is null!");
fileSystem->mkdir(spillDir);
+ // Prepare for the session level configurations and pass to connectors
+
+ // register the hive connectors
+ veloxRuntime_->registerConnectors();
Review Comment:
WholeStageResultIterator unconditionally dereferences veloxRuntime_. Since
this is a raw pointer passed in via the constructor, add an explicit check
before use to avoid a hard crash if a future call site passes nullptr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
