Hello,
I've been having an incredibly difficult time running an apache beam pipeline on a spark cluster in aws EMR. Spark on EMR is configured for yarn, which appears to be the primary source of my issues, the documentation here: https://beam.apache.org/documentation/runners/spark/ only seems to describe how to run beam on a cluster in Standalone mode. I've tried different beam pipeline arguments, but it seems to run in local mode every time after it can't find a spark master url. Has anyone run into this issue, and or does anyone have suggestions or examples of how to get this to work. Thank you for your help.
