wiedld commented on code in PR #11345:
URL: https://github.com/apache/datafusion/pull/11345#discussion_r1671386105
##########
datafusion/core/tests/memory_limit/mod.rs:
##########
@@ -323,6 +323,31 @@ async fn oom_recursive_cte() {
.await
}
+#[tokio::test]
+async fn oom_parquet_sink() {
+ let file = tempfile::Builder::new()
+ .suffix(".parquet")
+ .tempfile()
+ .unwrap();
+
+ TestCase::new()
+ .with_query(format!(
+ "
+ COPY (select * from t)
+ TO '{}'
+ STORED AS PARQUET OPTIONS (compression 'uncompressed');
+ ",
+ file.path().to_string_lossy()
+ ))
+ .with_expected_errors(vec![
+ // TODO: update error handling in ParquetSink
+ "Unable to send array to writer!",
Review Comment:
The parallelized writes have vectors for channels, and vectors for the
spawned tasks. This error we are hitting (on memory limit reached) is for the
closed channel.
I believe we want to be surfacing errors for the tasks, which **should**
exit due to the memory reservation error. Need to poke around a bit more.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
