Hi, I have a spark application for batch processing in standalone cluster. The job is to query the database and then do some transformation, aggregation, and several actions such as indexing the result into the elasticsearch.
If I dont call the sc.stop(), the spark application wont stop and take will keep the resource used by the application. In the other hand, if I call the sc.stop(), the spark app will be stopped before it query the database and do further processing. Can anyone help me to give best practice in stopping the spark application when the job is complete? Thanks, Imre
