hi, I'm learning spark, and wonder when to delete shuffle data, I find the ContextCleaner class which clean the shuffle data when shuffle dependency is GC-ed. Based on source code, the shuffle dependency is gc-ed only when active job finish, but i'm not sure, Could you explain the life cycle of a Shuffle-Dependency?
- Fwd: the life cycle shuffle Dependency yang chen
- Re: the life cycle shuffle Dependency murat migdisoglu
