adriangb commented on code in PR #20416:
URL: https://github.com/apache/datafusion/pull/20416#discussion_r2958092872
##########
datafusion/proto/src/physical_plan/to_proto.rs:
##########
@@ -256,6 +258,47 @@ pub fn serialize_physical_expr_with_converter(
codec: &dyn PhysicalExtensionCodec,
proto_converter: &dyn PhysicalProtoConverterExtension,
) -> Result<protobuf::PhysicalExprNode> {
+ // Check for DynamicFilterPhysicalExpr before snapshotting.
+ // We need to handle it before snapshot_physical_expr because snapshot()
+ // replaces the DynamicFilterPhysicalExpr with its inner expression.
+ if let Some(df) =
value.as_any().downcast_ref::<DynamicFilterPhysicalExpr>() {
Review Comment:
Sorry for the absence.
A bit of context: the idea of the `snapshot()` method was that it would (1)
allow us to get a simplified predicate that `PruningPredicate` knows how to
handle (so that we didn't have to add another downcast into
`DynamicFilterPhysicalExpr`) and (2) give us an expression we can serialize
across the wire.
I agree it hasn't panned out well as a design and is a good candidate for
removal.
IMO one of the warts with DataFusion's serialization machinery is that
generally there is no method on these traits for them to decide how to
serialize themselves -> the serialization code has to try downcasting into
every known type. In an ideal world `DynamicFilterPhysicalExpr` could decide
how to serialize and deserialize itself.
> What we want in this PR, is to serialize the actual
`DynamicFilterPhysicalExpr`, with all it's details, so probably `
snapshot_physical_expr(Arc::clone(value))?;` should not be getting called at
all for any expression, and we should be able to treat the dynamic filter as
just another serializable `PhysicalExpr`
> I've kept the `snapshot()` method, but I removed its usage in the
serializer.
👍🏻 yep this was always the loose plan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
