ccciudatu opened a new issue, #50: URL: https://github.com/apache/arrow-rs-object-store/issues/50
**Describe the bug** <!-- A clear and concise description of what the bug is. --> Client options (such as `ALLOW_HTTP`, `CONNECT_TIMEOUT`, etc.) cannot be passed from env vars to the underlying HTTP client of object stores. Each object store builder has a `from_env()` factory method that will populate config keys using env vars. Each of these factories selects only the env vars that start with the corresponding object store prefix (`AWS_`, `AZURE_` etc.) and parses them into the corresponding config keys. For unknown config keys, all the parsers fall back to treating them as potential client options (for the underlying HTTP client). However, the client options are not object-store specific, so they don't have an object-store prefix. Therefore, an env var like `AWS_CONNECT_TIMEOUT` will result in a parsing error, as it doesn't correspond to an aws config key and cannot be parsed as a client option either (since that one expects just `CONNECT_TIMEOUT`). On the other hand, `CONNECT_TIMEOUT` will be filtered out by all the `from_env()` factories, since it doesn't have any object store prefix. **To Reproduce** <!-- Steps to reproduce the behavior: --> Setting the env var `PROXY_URL` to an invalid location will have no effect on any of the object stores. Setting (e.g.) `AWS_PROXY_URL` (or `GOOGLE_PROXY_URL`, or `AZURE_PROXY_URL`) will result in an `UnknownConfigurationKey` when trying to build the corresponding object store and will be ignored, so it also has no effect. **Expected behavior** <!-- A clear and concise description of what you expected to happen. --> The expectation is for the underlying HTTP client to be customisable via env vars; there are several ways to achieve this (and most likely there's no "best way" to do it). One approach would be to explicitly look for all the unprefixed client options (like `PROXY_URL`) when building any object store `from_env`. A downside of this would be that when multiple object stores are used, these settings cannot be individualized per object store. An arguably better approach would be to allow prefixed client options (like `AWS_PROXY_URL`, `AZURE_PROXY_URL`) to be set as env vars (and each parser will make sure to drop the prefix before attempting to read any unknown key as a client option). (A PR will follow for this second approach). **Additional context** <!-- Add any other context about the problem here. --> https://github.com/apache/arrow-rs/blob/master/object_store/src/aws/builder.rs#L442 https://github.com/apache/arrow-rs/blob/master/object_store/src/aws/builder.rs#L406 https://github.com/apache/arrow-rs/blob/master/object_store/src/azure/builder.rs#L456 https://github.com/apache/arrow-rs/blob/master/object_store/src/azure/builder.rs#L412 https://github.com/apache/arrow-rs/blob/master/object_store/src/gcp/builder.rs#L245 https://github.com/apache/arrow-rs/blob/master/object_store/src/gcp/builder.rs#L189 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
