FYI, I am already using QueryExecutionListener which satisfies the requirements.
But that only works for dataframe APIs. If someone does df.rdd().someAction(), QueryExecutionListener is never invoked. I want something like QueryExecutionListener works in case of df.rdd().someAction() too. I explored SparkListener#onJobEnd, but then how to propagate some state from DataSourceReader to SparkListener? On Wed, Jun 12, 2019 at 2:22 PM Shubham Chaurasia <shubh.chaura...@gmail.com> wrote: > Hi All, > > Is there any way to receive some event that a DataSourceReader is > finished? > I want to do some clean up after all the DataReaders are finished reading > and hence need some kind of cleanUp() mechanism at DataSourceReader(Driver) > level. > > How to achieve this? > > For instance, in DataSourceWriter we can rely on commit() and abort() > methods to know that all the DataWriters are finished. > > Thanks, > Shubham >
