Hi community, I would like to collect information about the execution of a Spark job while it is running. Could I define some kind of application metrics (such as a counter that would be incremented in my code) that I could retrieve regularly while the job is running?
Thank you for help, Aurelien
