rizzabhdwaj opened a new issue, #300: URL: https://github.com/apache/datafusion-spark/issues/300
### Background Follow-up to the review discussion in [#98](https://github.com/apache/datafusion-spark/pull/98). That PR documents the required call order (`with_config()` before `with_spark_features()`); this issue tracks addressing the underlying configuration-loss behavior separately. ## Description `SessionStateBuilderSpark::with_spark_features()` registers a `SparkOptions` configuration extension. However, calling `SessionStateBuilder::with_config()` afterward replaces the entire session configuration, removing that extension unless the replacement configuration also contains it. This makes Spark configuration registration dependent on builder call order. The current documentation recommends calling `with_config()` before `with_spark_features()`, but this issue tracks a longer-term solution. ## Reproduction ```rust use datafusion::execution::SessionStateBuilder; use datafusion::prelude::SessionConfig; use datafusion_spark::{SessionStateBuilderSpark, SparkOptions}; let state = SessionStateBuilder::new() .with_spark_features() .with_config(SessionConfig::new()) .build(); // The Spark configuration extension has been dropped. assert!( state .config_options() .extensions .get::<SparkOptions>() .is_none() ); ``` ## Current Behavior Spark functions remain registered, but the `SparkOptions` extension is absent. As a result, `SET spark.map_key_dedup_policy = 'LAST_WIN'` cannot configure the extension, and any previously configured extension values are lost. ## Desired Outcome Provide a clear, safe way to combine `with_config()` and `with_spark_features()` without silently losing Spark configuration. The solution should define how existing Spark options interact with a replacement configuration, including which values take precedence when both configurations contain `SparkOptions`. It should respect the general replacement semantics of `with_config()` rather than implicitly changing them without consideration. ## Current Workaround Call `with_config()` first: ```rust let state = SessionStateBuilder::new() .with_config(SessionConfig::new()) .with_spark_features() .build(); ``` To preserve custom Spark options, include them in the configuration passed to `with_config()` before calling `with_spark_features()`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
