gabotechs commented on code in PR #26890:
URL: https://github.com/apache/datafusion/pull/26890#discussion_r4522571413
##########
datafusion/substrait/src/logical_plan/consumer/substrait_consumer.rs:
##########
@@ -211,82 +191,120 @@ pub trait SubstraitConsumer: Send + Sync + Sized {
/// All [Rel]s to be converted pass through this method.
/// You can provide your own implementation if you wish to customize the
conversion behaviour.
- async fn consume_rel(&self, rel: &Rel) ->
datafusion::common::Result<LogicalPlan> {
- from_substrait_rel(self, rel).await
+ async fn consume_rel(
+ &self,
+ rel: &Rel,
+ context: &SubstraitConsumerContext,
+ ) -> datafusion::common::Result<LogicalPlan> {
+ from_substrait_rel(self, rel, context).await
+ }
+
+ /// Consumes independent inputs of a JOIN or UNION, returning plans in
input order.
+ ///
+ /// Implementations must return exactly one plan per input, in the same
order.
+ /// On error, implementations must drop any unfinished input futures.
+ ///
+ /// The default consumes inputs sequentially, stopping at the first error.
+ /// Overrides can provide a different scheduling policy, but must account
for
+ /// any additional mutable traversal state in the custom consumer.
+ async fn consume_rel_inputs(
Review Comment:
I'm still not convinced about this change. For the sake of this PR, I'd
remove it, and we can discuss in a follow up
##########
datafusion/substrait/src/logical_plan/consumer/substrait_consumer.rs:
##########
@@ -828,73 +792,105 @@ mod tests {
use super::*;
use crate::logical_plan::consumer::utils::tests::test_consumer;
use datafusion::arrow::datatypes::{Field, Schema};
+ use datafusion::logical_expr::ExprSchemable;
- fn make_schema(fields: &[(&str, DataType)]) -> Arc<DFSchema> {
- let arrow_fields: Vec<Field> = fields
- .iter()
- .map(|(name, dt)| Field::new(*name, dt.clone(), true))
- .collect();
+ fn schema(name: &str, data_type: DataType) -> Arc<DFSchema> {
Arc::new(
- DFSchema::try_from(Schema::new(arrow_fields))
- .expect("failed to create schema"),
+ DFSchema::try_from(Schema::new(vec![Field::new(name, data_type,
true)]))
+ .unwrap(),
)
}
#[test]
- fn test_get_outer_schema_empty_stack() {
- let consumer = test_consumer();
-
- // No schemas pushed — any steps_out should return None
- assert!(consumer.get_outer_schema(0).is_none());
- assert!(consumer.get_outer_schema(1).is_none());
- assert!(consumer.get_outer_schema(2).is_none());
+ fn extensions_inherit_and_override_without_changing_other_scopes() {
Review Comment:
:thinking: these tests were pretty coupled to implementation details, and
now they are getting in the way rather than helping.
No action required in this PR, but if you find yourself in the position of
adding new test, rather than contributing to these ones, I'd try to just test
through the public API.
##########
datafusion/substrait/src/logical_plan/consumer/substrait_consumer.rs:
##########
@@ -580,18 +570,11 @@ pub trait SubstraitConsumer: Send + Sync + Sized {
pub struct DefaultSubstraitConsumer<'a> {
pub(super) extensions: &'a Extensions,
pub(super) state: &'a SessionState,
- outer_schemas: RwLock<Vec<Arc<DFSchema>>>,
- lambda_consumer: DefaultSubstraitLambdaConsumer,
}
Review Comment:
Nice!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
