its the latter -- after spark gets to the end of the iterator (or if it
hits an exception)
so your example is good, that is exactly what it is intended for.
On Fri, Apr 17, 2015 at 12:23 PM, Akshat Aranya <aara...@gmail.com> wrote:
> Hi,
>
> I'm trying to figure out when TaskCompletionListeners are called -- are
> they called at the end of the RDD's compute() method, or after the
> iteration through the iterator of the compute() method is completed.
>
> To put it another way, is this OK:
>
> class DatabaseRDD[T] extends RDD[T] {
>
> def compute(...): Iterator[T] = {
> val session = // acquire a DB session
> TaskContext.get.addTaskCompletionListener { (context) =>
> session.release()
> }
>
> val iterator = session.query(...)
> iterator
> }
> }
>
