GitHub user gitsult4n added a comment to the discussion:
pd.read_parquet("gs://...") fails with CURL error 56 when HTTPS_PROXY is set,
pyarrow 23 C++ GCS client ignores NO_PROXY
Two separate facts stack up here
**NO_PROXY really is dropped on the C++ path**
Arrow's GcsFileSystem goes through google-cloud-cpp and its curl layer
unconditionally does
```cpp
handle_.SetOption(CURLOPT_NOPROXY, "metadata.google.internal");
```
Once `CURLOPT_NOPROXY` is set explicitly libcurl skips its own
no_proxy/NO_PROXY env lookup entirely `https_proxy` detection still runs
separately so every GCS request is forced through the proxy hence curl error 56
`GcsOptions` in `gcsfs.h` and the pyarrow `GcsFileSystem` binding both have
zero proxy fields so there is no pyarrow side override for this
**storage_options={} did not fix pyarrow it swapped the backend**
Pandas only takes the native arrow path when the option is exactly `None`
```python
if storage_options is None:
fs, path_or_handle = pa_fs.FileSystem.from_uri(path)
```
`{}` is not `None` so pandas falls through to `fsspec.core.url_to_fs(...)`
which is gcsfs a pure python client that honors NO_PROXY normally You were
never fixing arrow you were routing around it
So
- yes known and structural not a bug you can flag
- no env var or pyarrow option restores it
- the supported fix is exactly what you found just be explicit about it
```python
import gcsfs
pd.read_parquet("gs://bucket/path", filesystem=gcsfs.GCSFileSystem())
```
Passing the filesystem directly is more stable than relying on the
`storage_options={}` side effect since a future pandas release could change
that branching
GitHub link:
https://github.com/apache/arrow/discussions/49979#discussioncomment-18069090
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
