Re: the life cycle shuffle Dependency

Hello, why would you like to delete the shuffle data yourself in the first place? On Thu, Dec 28, 2023, 10:08 yang chen wrote: > > hi, I'm learning spark, and wonder when to delete shuffle data, I find the > ContextCleaner class which clean the shuffle data when shuffle dependency > is GC-ed.

Fwd: the life cycle shuffle Dependency

hi, I'm learning spark, and wonder when to delete shuffle data, I find the ContextCleaner class which clean the shuffle data when shuffle dependency is GC-ed. Based on source code, the shuffle dependency is gc-ed only when active job finish, but i'm not sure, Could you explain the life cycle of