I have a spark application which creates multiple sessions. Each of these sessions can run jobs in parallel. I want to log some details about the execution of these jobs, but want to the tag them with the session they were called from.
I tried creating a listener from within each session (instantiations of the same class, but with session specific information), but whenever an event is emmited, it is intercepted by all the listeners (which makes sense since the listeners are attached to the shared context). Is there a way I can ensure that the events are only intercepted by the listener from the same session. If not, then is it possible to add any identifiers to the events which can be used to figure the session which triggered a job? Thanks Naved
