akshaychitneni commented on PR #2633:
URL:
https://github.com/apache/datafusion-ballista/pull/2633#issuecomment-6304334897
> Thanks for this. Reusing pinned shuffle output and the stage rollback path
is a good fit. Inline comments below. The main one is a possible hang in the
static graph when a cache's build-stage id collides with an own input id
(traced by reading, not run). The rest are mostly simplification and per-job
overhead.
>
Thanks for the thorough review. I have addressed comments. fix the
correctness bug you reported and will add more chaos test cases in the follow
up pr to keep this short. Also remove cachestorage api for now. It was added to
work on in-memory cache in future but will work on it later.
> * After a scheduler restart, does a read job held as `Queued` stay
`Queued`, since its pending state lives only in the process-local `CacheLinker`?
Currently, JobState backend is in-mem only, a restart drops job. Only a
durable backend could leave job stuck queued. I will look into it as a followup
> Pinned caches are only removed by executor TTL cleanup. Is that acceptable
until eviction lands?
Yes and next read rebuilds it. I plan to work on cleanup on remove_session
and also ttl based eviction tasks next after this pr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
