Hello, I newly joined this group and I went through the archive to see if any discussion exists on submitting Beam pipelines to a SparkRunner on k8s.
I run my Spark jobs on a k8s cluster in the cluster mode. Would like to deploy my beam pipeline on a SparkRunner with k8s underneath. The Beam documentation: https://beam.apache.org/documentation/runners/spark/ does not discuss about k8s (though there is mention of Mesos and YARN). Can someone please point me to relevant material in this regard? Or, provide the steps for running my beam pipeline in this configuration? Thank you, Regards, Buvana
