Re: [PR] Expose `ExecutionPlan` statistics across the FFI boundary [datafusion]

2026-05-19 Thread via GitHub 


timsaucer merged PR #22157:
URL: https://github.com/apache/datafusion/pull/22157


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Re: [PR] Expose `ExecutionPlan` statistics across the FFI boundary [datafusion]

2026-05-18 Thread via GitHub 


mailmindlin commented on PR #22157:
URL: https://github.com/apache/datafusion/pull/22157#issuecomment-4478612143

   I wasn't sure that `usize` was correct for FFI because I didn't want to 
exhaustively check `dlopen()`-style implementations across platforms but it 
looks like Datafusion already uses `FFI_Option` elsewhere so that's a 
moot point.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Re: [PR] Expose `ExecutionPlan` statistics across the FFI boundary [datafusion]

2026-05-18 Thread via GitHub 


mailmindlin commented on code in PR #22157:
URL: https://github.com/apache/datafusion/pull/22157#discussion_r3259475305


##
datafusion/ffi/src/execution_plan.rs:
##
@@ -74,6 +75,15 @@ pub struct FFI_ExecutionPlan {
 /// underlying [`ExecutionPlan::metrics`] returned `None`.
 pub metrics: unsafe extern "C" fn(plan: &Self) -> 
FFI_Option,
 
+/// Snapshot partition statistics. `partition == None` corresponds to
+/// statistics over all partitions; `Some(idx)` corresponds to a specific
+/// partition. The returned bytes are a prost-encoded
+/// `datafusion_proto_common::Statistics`.
+pub partition_statistics: unsafe extern "C" fn(
+plan: &Self,
+partition: FFI_Option,

Review Comment:
   I wonder if it's a HTML sanitizer bug. It was a deliberate choice to use 
`u64` because I wasn't sure that `FFI_Option` was safe for FFI, in case 
usize is different on both sides. Let me check how the FFI negotiation works 
real quick first.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Re: [PR] Expose `ExecutionPlan` statistics across the FFI boundary [datafusion]

2026-05-18 Thread via GitHub 


timsaucer commented on PR #22157:
URL: https://github.com/apache/datafusion/pull/22157#issuecomment-4478403518

   @mailmindlin would you mind trying the skill in 
https://github.com/apache/datafusion/pull/22327 against this?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Re: [PR] Expose `ExecutionPlan` statistics across the FFI boundary [datafusion]

2026-05-15 Thread via GitHub 


timsaucer commented on code in PR #22157:
URL: https://github.com/apache/datafusion/pull/22157#discussion_r3243790822


##
datafusion/ffi/src/execution_plan.rs:
##
@@ -74,6 +75,15 @@ pub struct FFI_ExecutionPlan {
 /// underlying [`ExecutionPlan::metrics`] returned `None`.
 pub metrics: unsafe extern "C" fn(plan: &Self) -> 
FFI_Option,
 
+/// Snapshot partition statistics. `partition == None` corresponds to
+/// statistics over all partitions; `Some(idx)` corresponds to a specific
+/// partition. The returned bytes are a prost-encoded
+/// `datafusion_proto_common::Statistics`.
+pub partition_statistics: unsafe extern "C" fn(
+plan: &Self,
+partition: FFI_Option,

Review Comment:
   why `FFI_Option` instead of `FFI_Option`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Re: [PR] Expose `ExecutionPlan` statistics across the FFI boundary [datafusion]

2026-05-15 Thread via GitHub 


timsaucer commented on code in PR #22157:
URL: https://github.com/apache/datafusion/pull/22157#discussion_r3248696693


##
datafusion/ffi/src/execution_plan.rs:
##
@@ -74,6 +75,15 @@ pub struct FFI_ExecutionPlan {
 /// underlying [`ExecutionPlan::metrics`] returned `None`.
 pub metrics: unsafe extern "C" fn(plan: &Self) -> 
FFI_Option,
 
+/// Snapshot partition statistics. `partition == None` corresponds to
+/// statistics over all partitions; `Some(idx)` corresponds to a specific
+/// partition. The returned bytes are a prost-encoded
+/// `datafusion_proto_common::Statistics`.
+pub partition_statistics: unsafe extern "C" fn(
+plan: &Self,
+partition: FFI_Option,

Review Comment:
   Fascinating. If you type in `FFI_Option` without the back ticks then it 
doesn't show the ``. added the ticks



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Re: [PR] Expose `ExecutionPlan` statistics across the FFI boundary [datafusion]

2026-05-14 Thread via GitHub 


mailmindlin commented on code in PR #22157:
URL: https://github.com/apache/datafusion/pull/22157#discussion_r3243827298


##
datafusion/ffi/src/execution_plan.rs:
##
@@ -74,6 +75,15 @@ pub struct FFI_ExecutionPlan {
 /// underlying [`ExecutionPlan::metrics`] returned `None`.
 pub metrics: unsafe extern "C" fn(plan: &Self) -> 
FFI_Option,
 
+/// Snapshot partition statistics. `partition == None` corresponds to
+/// statistics over all partitions; `Some(idx)` corresponds to a specific
+/// partition. The returned bytes are a prost-encoded
+/// `datafusion_proto_common::Statistics`.
+pub partition_statistics: unsafe extern "C" fn(
+plan: &Self,
+partition: FFI_Option,

Review Comment:
   I'm not sure I understand what you mean



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Re: [PR] Expose `ExecutionPlan` statistics across the FFI boundary [datafusion]

2026-05-14 Thread via GitHub 


mailmindlin commented on code in PR #22157:
URL: https://github.com/apache/datafusion/pull/22157#discussion_r3243827298


##
datafusion/ffi/src/execution_plan.rs:
##
@@ -74,6 +75,15 @@ pub struct FFI_ExecutionPlan {
 /// underlying [`ExecutionPlan::metrics`] returned `None`.
 pub metrics: unsafe extern "C" fn(plan: &Self) -> 
FFI_Option,
 
+/// Snapshot partition statistics. `partition == None` corresponds to
+/// statistics over all partitions; `Some(idx)` corresponds to a specific
+/// partition. The returned bytes are a prost-encoded
+/// `datafusion_proto_common::Statistics`.
+pub partition_statistics: unsafe extern "C" fn(
+plan: &Self,
+partition: FFI_Option,

Review Comment:
   huh?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Re: [PR] Expose `ExecutionPlan` statistics across the FFI boundary [datafusion]

2026-05-14 Thread via GitHub 


timsaucer commented on code in PR #22157:
URL: https://github.com/apache/datafusion/pull/22157#discussion_r3243790822


##
datafusion/ffi/src/execution_plan.rs:
##
@@ -74,6 +75,15 @@ pub struct FFI_ExecutionPlan {
 /// underlying [`ExecutionPlan::metrics`] returned `None`.
 pub metrics: unsafe extern "C" fn(plan: &Self) -> 
FFI_Option,
 
+/// Snapshot partition statistics. `partition == None` corresponds to
+/// statistics over all partitions; `Some(idx)` corresponds to a specific
+/// partition. The returned bytes are a prost-encoded
+/// `datafusion_proto_common::Statistics`.
+pub partition_statistics: unsafe extern "C" fn(
+plan: &Self,
+partition: FFI_Option,

Review Comment:
   why FFI_Option instead of FFI_Option?



##
datafusion/ffi/src/execution_plan.rs:
##
@@ -454,10 +476,24 @@ impl ExecutionPlan for ForeignExecutionPlan {
 unsafe { (self.plan.metrics)(&self.plan) }.into();
 ffi.map(MetricsSet::from)
 }
+
+fn partition_statistics(&self, partition: Option) -> 
Result> {
+let bytes = df_result!(unsafe {
+(self.plan.partition_statistics)(
+&self.plan,
+partition.map(|p| p as u64).into(),
+)
+})?;
+Ok(Arc::new(deserialize_statistics(bytes.as_slice())?))
+}
 }
 
 #[cfg(any(test, feature = "integration-tests"))]
 pub mod tests {
+#[cfg(test)]
+use datafusion_common::stats::Precision;
+#[cfg(test)]
+use datafusion_common::{ColumnStatistics, ScalarValue};

Review Comment:
   If this is here because you get clippy warnings otherwise, then it's 
marginally cleaner to remove the `use` statements and just fully qualify these 
below.



##
datafusion/ffi/src/table_provider.rs:
##
@@ -179,6 +188,16 @@ unsafe extern "C" fn schema_fn_wrapper(provider: 
&FFI_TableProvider) -> WrappedS
 provider.inner().schema().into()
 }
 
+unsafe extern "C" fn statistics_fn_wrapper(
+provider: &FFI_TableProvider,
+) -> FFI_Result>> {
+let serialized: Option> = provider
+.inner()
+.statistics()
+.map(|s| serialize_statistics(&s).into_iter().collect());
+FFI_Result::Ok(serialized.into())

Review Comment:
   Why return a result? It looks infallible unless I missed something



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Re: [PR] Expose `ExecutionPlan` statistics across the FFI boundary [datafusion]

2026-05-13 Thread via GitHub 


github-actions[bot] commented on PR #22157:
URL: https://github.com/apache/datafusion/pull/22157#issuecomment-4445391574

   
   Thank you for opening this pull request!
   
   Reviewer note: 
[cargo-semver-checks](https://github.com/obi1kenobi/cargo-semver-checks) 
reported the current version number is not SemVer-compatible with the changes 
in this pull request (compared against the base branch).
   
   
   Details
   
   ```
Cloning apache/main
   Building datafusion-ffi v53.1.0 (current)
  Built [  65.038s] (current)
Parsing datafusion-ffi v53.1.0 (current)
 Parsed [   0.060s] (current)
   Building datafusion-ffi v53.1.0 (baseline)
  Built [  59.992s] (baseline)
Parsing datafusion-ffi v53.1.0 (baseline)
 Parsed [   0.056s] (baseline)
   Checking datafusion-ffi v53.1.0 -> v53.1.0 (no change; assume patch)
Checked [   0.259s] 222 checks: 220 pass, 1 fail, 1 warn, 30 skip
   
   --- failure constructible_struct_adds_field: externally-constructible struct 
adds field ---
   
   Description:
   A pub struct constructible with a struct literal has a new pub field. 
Existing struct literals must be updated to include the new field.
   ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
  impl: 
https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.47.0/src/lints/constructible_struct_adds_field.ron
   
   Failed in:
 field FFI_ExecutionPlan.metrics in 
/home/runner/work/datafusion/datafusion/datafusion/ffi/src/execution_plan.rs:76
 field FFI_ExecutionPlan.partition_statistics in 
/home/runner/work/datafusion/datafusion/datafusion/ffi/src/execution_plan.rs:82
   
   --- warning repr_c_plain_struct_fields_reordered: struct fields reordered in 
repr(C) struct ---
   
   Description:
   A public repr(C) struct had its fields reordered. This can change the 
struct's memory layout, possibly breaking FFI use cases that depend on field 
position and order.
   ref: 
https://doc.rust-lang.org/reference/type-layout.html#reprc-structs
  impl: 
https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.47.0/src/lints/repr_c_plain_struct_fields_reordered.ron
   
   Failed in:
 FFI_ExecutionPlan.clone moved from position 7 to 9, in 
/home/runner/work/datafusion/datafusion/datafusion/ffi/src/execution_plan.rs:89
 FFI_ExecutionPlan.release moved from position 8 to 10, in 
/home/runner/work/datafusion/datafusion/datafusion/ffi/src/execution_plan.rs:92
 FFI_ExecutionPlan.private_data moved from position 9 to 11, in 
/home/runner/work/datafusion/datafusion/datafusion/ffi/src/execution_plan.rs:96
 FFI_ExecutionPlan.library_marker_id moved from position 10 to 12, in 
/home/runner/work/datafusion/datafusion/datafusion/ffi/src/execution_plan.rs:101
 FFI_TableProvider.logical_codec moved from position 6 to 7, in 
/home/runner/work/datafusion/datafusion/datafusion/ffi/src/table_provider.rs:145
 FFI_TableProvider.version moved from position 9 to 10, in 
/home/runner/work/datafusion/datafusion/datafusion/ffi/src/table_provider.rs:155
 FFI_TableProvider.library_marker_id moved from position 11 to 12, in 
/home/runner/work/datafusion/datafusion/datafusion/ffi/src/table_provider.rs:164
   
Summary semver requires new major version: 1 major and 0 minor checks 
failed
Warning produced 1 major and 0 minor level warnings
   Finished [ 127.127s] datafusion-ffi
   Building datafusion-physical-expr-common v53.1.0 (current)
  Built [  20.866s] (current)
Parsing datafusion-physical-expr-common v53.1.0 (current)
 Parsed [   0.020s] (current)
   Building datafusion-physical-expr-common v53.1.0 (baseline)
  Built [  20.631s] (baseline)
Parsing datafusion-physical-expr-common v53.1.0 (baseline)
 Parsed [   0.021s] (baseline)
   Checking datafusion-physical-expr-common v53.1.0 -> v53.1.0 (no change; 
assume patch)
Checked [   0.200s] 222 checks: 222 pass, 30 skip
Summary no semver update required
   Finished [  42.563s] datafusion-physical-expr-common
   ```
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]