pratham76 opened a new issue, #13645:
URL: https://github.com/apache/gluten/issues/13645
### Backend
VL (Velox)
### Bug description
Gluten translates Spark's per-bucket S3A options
(`spark.hadoop.fs.s3a.bucket.<bucket>.<option>`) into Velox per-bucket keys
(`s3.bucket.<bucket>.<option>`) in `getS3HiveConfig`
(`cpp/velox/utils/ConfigExtractor.cc`). The parser assumes the bucket name ends
at the first `.` after the prefix:
```cpp
auto remaining = skey.substr(kSparkHadoopS3BucketPrefix.size());
int dot = remaining.find(".");
auto bucketName = remaining.substr(0, dot);
auto suffix = remaining.substr(dot + 1);
```
S3 bucket names may contain dots (e.g. `my.data.bucket`), and Hadoop S3A
supports per-bucket options for them, because `S3AUtils.propagateBucketOptions`
matches the `fs.s3a.bucket.<bucket>.` prefix using the real bucket name.
For example, with:
```
spark.hadoop.fs.s3a.bucket.my.data.bucket.endpoint=http://...
```
Gluten parses the bucket as `my` and the option as `data.bucket.endpoint`.
That option is not recognized, so the setting is **silently dropped**. The
native scan then falls back to the global `fs.s3a.*` values (endpoint,
credentials, region, IAM role, ...), while vanilla Spark correctly uses the
per-bucket ones.
A minor related issue: the `size_t` result of `find()` is stored in an `int`.
### Gluten version
main branch
### Spark version
Spark-3.5.x
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
```bash
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]