jayshrivastava commented on code in PR #27589:
URL: https://github.com/apache/datafusion/pull/27589#discussion_r4632266750
##########
datafusion/proto/tests/cases/plans/dynamic_filters.rs:
##########
@@ -1111,3 +1111,36 @@ fn extension_codec_expr_participates_in_deduplication()
-> Result<()> {
Ok(())
}
+
+/// <https://github.com/apache/datafusion/issues/27543>
Review Comment:
```suggestion
/// Ensure that dynamic filters are reset by `reset_plan_states` so plans
can be re-executed after
/// serialization.
///
/// See <https://github.com/apache/datafusion/issues/27543>
```
##########
datafusion/physical-plan/src/execution_plan.rs:
##########
@@ -2010,6 +2010,12 @@ pub fn check_not_null_constraints(
/// However, if the data of the left table is derived from the work table, it
will become outdated
/// as the work table changes. When the next iteration executes this plan
again, we must clear the left table.
///
+/// # Serializing an executed plan
+///
+/// Serialization (for example with `datafusion-proto`) keeps the current
state of
+/// dynamic filters. To run a decoded plan from the start, call this function
+/// before you serialize a plan that already executed.
Review Comment:
In general, serializing preserves a lot of state right? Not just dynamic
filters?
##########
datafusion/proto/tests/cases/plans/dynamic_filters.rs:
##########
@@ -1111,3 +1111,36 @@ fn extension_codec_expr_participates_in_deduplication()
-> Result<()> {
Ok(())
}
+
+/// <https://github.com/apache/datafusion/issues/27543>
+#[tokio::test]
+async fn test_dynamic_filter_roundtrip_after_execution_and_reset() ->
Result<()> {
+ let dir = tempfile::tempdir()?;
+ let path = dir.path().to_str().unwrap();
+ let ctx = SessionContext::new();
+ for (name, end) in [("big", 201), ("small", 4)] {
+ ctx.sql(&format!("COPY (SELECT value AS id FROM range(1, {end})) TO
'{path}/{name}/t.parquet' STORED AS PARQUET"))
+ .await?
+ .collect()
+ .await?;
+ ctx.register_parquet(name, format!("{path}/{name}"),
Default::default())
+ .await?;
+ }
+
+ for (sql, expected) in [
+ ("SELECT * FROM big ORDER BY id LIMIT 1", 1),
+ ("SELECT * FROM big ORDER BY id DESC LIMIT 3", 3),
+ ("SELECT big.id FROM small JOIN big ON small.id = big.id", 3),
+ ] {
+ let plan = ctx.sql(sql).await?.create_physical_plan().await?;
+ datafusion::physical_plan::collect(Arc::clone(&plan),
ctx.task_ctx()).await?;
+ let plan =
datafusion::physical_plan::execution_plan::reset_plan_states(plan)?;
+ let bytes = datafusion_proto::bytes::physical_plan_to_bytes(plan)?;
Review Comment:
Nit: assert that there was in fact a dynamic filter in the plan at some
point to reset?
##########
datafusion/proto/tests/cases/plans/dynamic_filters.rs:
##########
@@ -1111,3 +1111,36 @@ fn extension_codec_expr_participates_in_deduplication()
-> Result<()> {
Ok(())
}
+
+/// <https://github.com/apache/datafusion/issues/27543>
+#[tokio::test]
+async fn test_dynamic_filter_roundtrip_after_execution_and_reset() ->
Result<()> {
+ let dir = tempfile::tempdir()?;
+ let path = dir.path().to_str().unwrap();
+ let ctx = SessionContext::new();
+ for (name, end) in [("big", 201), ("small", 4)] {
+ ctx.sql(&format!("COPY (SELECT value AS id FROM range(1, {end})) TO
'{path}/{name}/t.parquet' STORED AS PARQUET"))
+ .await?
+ .collect()
+ .await?;
+ ctx.register_parquet(name, format!("{path}/{name}"),
Default::default())
+ .await?;
+ }
+
+ for (sql, expected) in [
+ ("SELECT * FROM big ORDER BY id LIMIT 1", 1),
+ ("SELECT * FROM big ORDER BY id DESC LIMIT 3", 3),
Review Comment:
Consider adding an aggregate case so we cover min/max filters, not just TopK
and joins.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]