Re: [I] Make TaskContext wrap SessionState [datafusion]
tustvold commented on issue #10631: URL: https://github.com/apache/datafusion/issues/10631#issuecomment-2131201943 IIRC SessionConfig is the static configuration used to create a SessionContext, which is an interior mutable wrapper around SessionState. The idea was a query is planned against an immutable SessionState to avoiding inconsistency during planning. RuntimeEnv is then for things shared across multiple sessions. As for TaskContext, I believe it was added for Ballista where the execution takes place on different nodes to planning. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
Re: [I] Make TaskContext wrap SessionState [datafusion]
alamb commented on issue #10631: URL: https://github.com/apache/datafusion/issues/10631#issuecomment-2131197268 > I see TaskContext is just part of SessionState. Could we just make TaskContext wrap SessionState? I can't remember why TaskContext doesn't wrap SessionState -- maybe @tustvold does 樂 At some point `SessionState` was in `datafusion` (the core crate) so it couldn't be referenced by subcrates. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
Re: [I] Make TaskContext wrap SessionState [datafusion]
crepererum commented on issue #10631: URL: https://github.com/apache/datafusion/issues/10631#issuecomment-2129296188 I would suggest a rather larger refactoring? We have: - `SessionState` - `SessionConfig` - `SessionContext` - `TaskContext` - `RuntimeConfig` - `RuntimeEnv` - `ConfigOptions` So it seems that someone tried to model a hierarchy runtime->session->task but TBH the types are pretty inconsistent and I don't know what we would need a "task" for. I kinda see why someone would differentiate between a "runtime" (long-lived) and a "session" (request/query-scoped). I also don't understand why there is a "state" and a "context". -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
[I] Make TaskContext wrap SessionState [datafusion]
lewiszlw opened a new issue, #10631: URL: https://github.com/apache/datafusion/issues/10631 ### Is your feature request related to a problem or challenge? I have a user defined table function `FlattenTableFunc` whice needs reading data from other table provider. To serialize execution plan `FlattenExec`, I can't put `SessionState` in it directly. So I have to reconstruct `SessionState` in `FlattenExec::execute` method. ### Describe the solution you'd like I see `TaskContext` is just part of `SessionState`. Could we just make `TaskContext` wrap `SessionState`? ### Describe alternatives you've considered _No response_ ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
