Oh well, never mind. The problem is that ResultTask's stageId is immutable and is used to construct the Task superclass. Anyway, my solution now is to use this.id for the rddId and to gather all rddIds using a spark listener on stage completed to clean up for any activity registered for those rdds. I could use TaskContext's hook but I'd have to add some more messaging so I can clear state that may live on a different executor than the one my partition is on, but since I don't know that the executor will succeed, this is not safe.
-- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/TaskContext-stageId-0-tp9152p9162.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
