Hi I am developing python applications. I use kubernetes to containerise my applications.
I want to set up a spark cluster in kubernetes. I only want specific spark jobs to be processed by my spark cluster (for example large data ETL processes that would take long using python alone). I don’t want to use spark-submit and submit the entire application. I also don’t want to use a yaml file pointing to the mainApplication that needs to be submitted. How do I set up a spark cluster in kubernetes which then can be accessed to run specific spark jobs? Would a SparkSession be used instead? Please advise. Thanks!
