devinjdangelo commented on code in PR #7435:
URL: https://github.com/apache/arrow-datafusion/pull/7435#discussion_r1312307794
##########
datafusion/core/src/dataframe.rs:
##########
@@ -2292,4 +2365,41 @@ mod tests {
Ok(())
}
+
+ #[tokio::test]
+ async fn write_parquet_with_compression() -> Result<()> {
+ let test_df = test_table().await?;
+ let tmp_dir = TempDir::new()?;
+ let local = Arc::new(LocalFileSystem::new_with_prefix(&tmp_dir)?);
+ let local_url = Url::parse("file://local").unwrap();
+ let ctx = &test_df.session_state;
+ ctx.runtime_env().register_object_store(&local_url, local);
+
+ let output_path = "file://local/test.parquet";
+ test_df
+ .write_parquet(
+ output_path,
+ DataFrameWriteOptions::new().with_single_file_output(true),
+ Some(
+ WriterProperties::builder()
+ .set_compression(parquet::basic::Compression::SNAPPY)
Review Comment:
I expanded the test to include all supported compression codecs. We
unfortunately cannot test all compression levels for those codecs that support
levels, since in general they do not include the used compression level in the
file metadata. The parquet crate reader always reports the compression level as
the "default" level.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
