dosubot[bot] commented on issue #6857: URL: https://github.com/apache/opendal/issues/6857#issuecomment-3606020941
<!-- Greeting --> Hi @gaius-qi! I'm [Dosu](https://go.dosu.dev/dosubot) and I’m helping the opendal team. <!-- Answer --> This is a known issue: enabling gzip, brotli, zstd, or deflate in reqwest causes automatic decompression of HTTP responses, which leads to mismatches between the Content-Length reported by OpenDAL and the actual size of the OSS object. This happens because reqwest modifies the response body and Content-Length when these features are enabled, so OpenDAL receives the decompressed size instead of the original object size. This problem has been discussed in several issues, including [#5070](https://github.com/apache/opendal/issues/5070), [#5897](https://github.com/apache/opendal/issues/5897), [#3810](https://github.com/apache/opendal/issues/3810), and [#730](https://github.com/apache/opendal/issues/730). A common workaround is to configure your reqwest client with `.no_gzip()` and set the `Accept-Encoding` header explicitly (for example, to `"gzip"`), which disables automatic decompression and ensures you get the raw data and correct content length. Here’s an example pattern (adapt for OSS): ```rust let mut default_headers = reqwest::header::HeaderMap::new(); default_headers.insert(reqwest::header::ACCEPT_ENCODING, reqwest::header::HeaderValue::from_static("gzip")); let client = reqwest::Client::builder() .default_headers(default_headers) .no_gzip() .build() .unwrap(); ``` Then pass this client to OpenDAL as you are already doing. Note that this workaround may not be possible in WASM/browser environments, where the browser always decompresses responses and you cannot control the `Accept-Encoding` header. There is ongoing discussion about whether OpenDAL should handle content encoding itself or rely on the HTTP client, but for now, using a custom reqwest client as above is the recommended approach for native targets. <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/383cd5b0-82d2-453f-bf05-544e261d9d1e?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/383cd5b0-82d2-453f-bf05-544e261d9d1e?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/383cd5b0-82d2-453f-bf05-544e261d9d1e?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/383cd5b0-82d2-453f-bf05-544e261d9d1e?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/383cd5b0-82d2-453f-bf05-544e261d9d1e?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/383cd5b0-82d2-453f-bf05-544e261d9d1e?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/383cd5b0-82d2-453f-bf05-544e261d9d1e?feedback_type=other)</sup> [](https://app.dosu.dev/997a2c5a-0c54-497b-989d-44651635dd36/ask?utm_source=github) [](https://cursor.com/link/prompt?text=This%20is%20a%20known%20issue%3A%20enabling%20gzip%2C%20brotli%2C%20zstd%2C%20or%20deflate%20in%20reqwest%20causes%20automatic%20decompression%20of%20HTTP%20responses%2C%20which%20leads%20to%20mismatches%20between%20the%20Content-Length%20reported%20by%20OpenDAL%20and%20the%20actual%20size%20of%20the%20OSS%20object.%20This%20happens%20because%20reqwest%20modifies%20the%20response%20body%20and%20Content-Length%20when%20these%20features%20are%20enabled%2C%20so%20OpenDAL%20receives%20the%20decompressed%20size%20instead%20of%20the%20original%20object%20size.%20This%20problem%20has%20been%20discussed%20in%20several%20issues%2C%20including%20%5B%235070%5D%28https%3A//github.com/apache/opendal/issues/5070%29%2C%20%5B%235897%5D%28https%3A//github.com/apach e/opendal/issues/5897%29%2C%20%5B%233810%5D%28https%3A//github.com/apache/opendal/issues/3810%29%2C%20and%20%5B%23730%5D%28https%3A//github.com/apache/opendal/issues/730%29.%0A%0AA%20common%20workaround%20is%20to%20configure%20your%20reqwest%20client%20with%20%60.no_gzip%28%29%60%20and%20set%20the%20%60Accept-Encoding%60%20header%20explicitly%20%28for%20example%2C%20to%20%60%22gzip%22%60%29%2C%20which%20disables%20automatic%20decompression%20and%20ensures%20you%20get%20the%20raw%20data%20and%20correct%20content%20length.%20Here%E2%80%99s%20an%20example%20pattern%20%28adapt%20for%20OSS%29%3A%0A%0A%60%60%60rust%0Alet%20mut%20default_headers%20%3D%20reqwest%3A%3Aheader%3A%3AHeaderMap%3A%3Anew%28%29%3B%0Adefault_headers.insert%28reqwest%3A%3Aheader%3A%3AACCEPT_ENCODING%2C%20reqwest%3A%3Aheader%3A%3AHeaderValue%3A%3Afrom_static%28%22gzip%22%29%29%3B%0A%0Alet%20client%20%3D%20reqwest%3A%3AClient%3A%3Abuilder%28%29%0A%20%20%20%20.default_headers%28default_headers%29%0A%20%20%20%20.no_gzip% 28%29%0A%20%20%20%20.build%28%29%0A%20%20%20%20.unwrap%28%29%3B%0A%60%60%60%0A%0AThen%20pass%20this%20client%20to%20OpenDAL%20as%20you%20are%20already%20doing.%20Note%20that%20this%20workaround%20may%20not%20be%20possible%20in%20WASM/browser%20environments%2C%20where%20the%20browser%20always%20decompresses%20responses%20and%20you%20cannot%20control%20the%20%60Accept-Encoding%60%20header.%0A%0AThere%20is%20ongoing%20discussion%20about%20whether%20OpenDAL%20should%20handle%20content%20encoding%20itself%20or%20rely%20on%20the%20HTTP%20client%2C%20but%20for%20now%2C%20using%20a%20custom%20reqwest%20client%20as%20above%20is%20the%20recommended%20approach%20for%20native%20targets.) [](https://go.dosu.dev/discord-bot) [](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/a pache/opendal/issues/6857) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
