[ https://issues.apache.org/jira/browse/BEAM-1802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15940415#comment-15940415 ]
Ismaël Mejía commented on BEAM-1802: ------------------------------------ [~amitsela] I am precisely talking about this case on batch jobs. But I will do as Aviem suggested. I saw your comments on his PR and I understand now, it makes sense (however I confess I am a bit lost about why I don't need to do something like this with other runners). [~aviemzur] stop is protected so I cannot use it, are you guys going to make this one public? > Spark Runner does not shutdown correctly when executing multiple pipelines in > sequence > -------------------------------------------------------------------------------------- > > Key: BEAM-1802 > URL: https://issues.apache.org/jira/browse/BEAM-1802 > Project: Beam > Issue Type: Bug > Components: runner-spark > Reporter: Ismaël Mejía > Assignee: Aviem Zur > > I found this while running the Nexmark queries in sequence in local mode. I > had the correct configuration but it didn't seem to work. > 17/03/24 12:07:49 WARN org.apache.spark.SparkContext: Multiple running > SparkContexts detected in the same JVM! > org.apache.spark.SparkException: Only one SparkContext may be running in this > JVM (see SPARK-2243). To ignore this error, set > spark.driver.allowMultipleContexts = true. The currently running SparkContext > was created at: > org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:59) > org.apache.beam.runners.spark.translation.SparkContextFactory.createSparkContext(SparkContextFactory.java:100) > org.apache.beam.runners.spark.translation.SparkContextFactory.getSparkContext(SparkContextFactory.java:69) > org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:206) > org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:91) > org.apache.beam.sdk.Pipeline.run(Pipeline.java:266) > org.apache.beam.integration.nexmark.NexmarkRunner.run(NexmarkRunner.java:1233) > org.apache.beam.integration.nexmark.NexmarkDriver.runAll(NexmarkDriver.java:69) > org.apache.beam.integration.nexmark.drivers.NexmarkSparkDriver.main(NexmarkSparkDriver.java:46) > at > org.apache.spark.SparkContext$$anonfun$assertNoOtherContextIsRunning$1.apply(SparkContext.scala:2257) > at > org.apache.spark.SparkContext$$anonfun$assertNoOtherContextIsRunning$1.apply(SparkContext.scala:2239) -- This message was sent by Atlassian JIRA (v6.3.15#6346)