rizzabhdwaj opened a new issue, #300:
URL: https://github.com/apache/datafusion-spark/issues/300

   ### Background
   
   Follow-up to the review discussion in 
[#98](https://github.com/apache/datafusion-spark/pull/98). That PR documents 
the required
   call order (`with_config()` before `with_spark_features()`); this issue 
tracks
   addressing the underlying configuration-loss behavior separately.
   
   ## Description
   
   `SessionStateBuilderSpark::with_spark_features()` registers a `SparkOptions` 
configuration extension. However, calling `SessionStateBuilder::with_config()` 
afterward replaces the entire session configuration, removing that extension 
unless the replacement configuration also contains it.
   
   This makes Spark configuration registration dependent on builder call order. 
The current documentation recommends calling `with_config()` before 
`with_spark_features()`, but this issue tracks a longer-term solution.
   
   ## Reproduction
   
   ```rust
   use datafusion::execution::SessionStateBuilder;
   use datafusion::prelude::SessionConfig;
   use datafusion_spark::{SessionStateBuilderSpark, SparkOptions};
   
   let state = SessionStateBuilder::new()
       .with_spark_features()
       .with_config(SessionConfig::new())
       .build();
   
   // The Spark configuration extension has been dropped.
   assert!(
       state
           .config_options()
           .extensions
           .get::<SparkOptions>()
           .is_none()
   );
   ```
   
   ## Current Behavior
   
   Spark functions remain registered, but the `SparkOptions` extension is 
absent. As a result, `SET spark.map_key_dedup_policy = 'LAST_WIN'` cannot 
configure the extension, and any previously configured extension values are 
lost.
   
   ## Desired Outcome
   
   Provide a clear, safe way to combine `with_config()` and 
`with_spark_features()` without silently losing Spark configuration.
   
   The solution should define how existing Spark options interact with a 
replacement configuration, including which values take precedence when both 
configurations contain `SparkOptions`. It should respect the general 
replacement semantics of `with_config()` rather than implicitly changing them 
without consideration.
   
   ## Current Workaround
   
   Call `with_config()` first:
   
   ```rust
   let state = SessionStateBuilder::new()
       .with_config(SessionConfig::new())
       .with_spark_features()
       .build();
   ```
   
   To preserve custom Spark options, include them in the configuration passed 
to `with_config()` before calling `with_spark_features()`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to