akshaychitneni commented on code in PR #2889:
URL:
https://github.com/apache/datafusion-ballista/pull/2889#discussion_r4634119997
##########
ballista/scheduler/src/state/cache_linker.rs:
##########
@@ -694,6 +756,51 @@ impl<T: 'static + AsLogicalPlan, U: 'static +
AsExecutionPlan> SchedulerState<T,
}
}
+ /// Drops the `DataFrame::cache()` data of `session_id`, which is being
+ /// removed. Its built caches are gone at once for new queries, and their
+ /// data is cleaned up once no running query reads it. Its builds still in
+ /// flight are cancelled, and what waits on them fails. Using the session
+ /// again builds its caches anew.
+ pub(crate) async fn remove_session_caches(
Review Comment:
Good catch, thanks. Fixed by failing the reader: removing a session now
marks the queries reading its caches, and a cache one of them loses afterwards
is not rebuilt. The query fails with `lost DataFrame::cache() data after its
session was closed ... retry the query`, so a retry on the session rebuilds as
usual, while queries started after the removal still rebuild normally. I did
not drop the rebuilt entry instead, as a rebuild can also rebuild the caches
inside it. Added
`removing_a_session_fails_a_later_rebuild_for_its_running_reads` (your
suggested test plus `lose_executor`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]