yjshen commented on a change in pull request #811:
URL: https://github.com/apache/arrow-datafusion/pull/811#discussion_r682256093
##########
File path: datafusion/src/execution/context.rs
##########
@@ -125,12 +127,26 @@ pub struct ExecutionContext {
pub state: Arc<Mutex<ExecutionContextState>>,
}
+lazy_static! {
Review comment:
I made a registry (a hashmap) for storing all available protocol
handlers (or ObjectStore as you suggested above) in the context at runtime and
registering the LocalFS handler by default. In this way, I intended to make
DataFusion’s core protocol agnostic and get different handlers according to the
scheme of a path when users are creating new tables at runtime (based on their
previous registrations), and get the suitable handler for each table.
And at the same time, I thought a global/sharable context is useful when we
consider controlling the total amount of memory Datafusion could use. All
physical operators should register their memory usage with a global context,
and try to allocate more memory or spill itself if used up all allowed memory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
