gabotechs commented on code in PR #18192:
URL: https://github.com/apache/datafusion/pull/18192#discussion_r2446787526
##########
datafusion/proto/src/physical_plan/from_proto.rs:
##########
@@ -207,10 +212,19 @@ where
/// * `codec` - An extension codec used to decode custom UDFs.
pub fn parse_physical_expr(
proto: &protobuf::PhysicalExprNode,
- ctx: &TaskContext,
+ decode_ctx: &DecodeContext,
input_schema: &Schema,
codec: &dyn PhysicalExtensionCodec,
) -> Result<Arc<dyn PhysicalExpr>> {
+ // Check cache first if an ID is present
+ if let Some(id) = proto.id {
+ if let Some(cached) = decode_ctx.get_cached_expr(id) {
+ return Ok(cached);
+ }
+ }
+
Review Comment:
This looks like a very elegant solution!
It does solve the problem at hand, but I wonder how could this be extended
for the case where the producer part of a dynamic filter is deserialized in one
machine, and the consumer part in deserialized in other different machine,
which is almost always going to be the case in a distributed context.
For example, the idea that powered
https://github.com/gabotechs/datafusion/pull/7, is that users can subscribe to
changes to the dynamic filters in the global registry, and send/produce updates
over the wire, something like:
```rust
let ctx = SessionContext::new();
let registry = ctx
.task_ctx()
.session_config()
.get_extension::<DynamicFiltersRegistry>();
registry.subscribe_to_updates();
registry.push_updates();
let plan: Arc<dyn ExecutionPlan>;
execute_stream(plan, ctx.task_ctx());
```
I really would love to see something like this happening without recurring
to a "global place" for reading/writing updates to dynamic filters, but I
cannot come up with other ideas.
Do you think there's a chance we can do that with something simpler like
what this PR proposes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
