[ https://issues.apache.org/jira/browse/SPARK-8896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14618764#comment-14618764 ]
Aniket Bhatnagar commented on SPARK-8896: ----------------------------------------- As per the documentation (scala docs), only one SparkContext per JVM is allowed. However, no such documentation exists for StreamingContext. SparkContext makes an effort to ensure only one SparkContext instance exists in the JVM using contextBeingConstructed variable. However, no such effort is made in StreamingContext. This lead me to believe that multiple StreamingContexts are allowed in a JVM. I can understand why only one SparkContext is allowed per JVM (global state, et al), but I don't think thats true for StreamingContext. There maybe genuine use cases to have 2 StreamingContext instances using the same SparkContext to leverage the same workers and have different batch intervals. > StreamingSource should choose a unique name > ------------------------------------------- > > Key: SPARK-8896 > URL: https://issues.apache.org/jira/browse/SPARK-8896 > Project: Spark > Issue Type: Bug > Components: Streaming > Affects Versions: 1.4.0 > Reporter: Aniket Bhatnagar > > If 2 instances of StreamingContext are created and run using the same > SparkContext, it results the following exception in the logs and causes the > latter StreamingContext's metrics to go unreported. > [ForkJoinPool-2-worker-7] [info] o.a.s.m.MetricsSystem -Metrics already > registered > java.lang.IllegalArgumentException: A metric named > <AppName>.StreamingMetrics.streaming.lastReceivedBatch_processingEndTime > already exists > at > com.codahale.metrics.MetricRegistry.register(MetricRegistry.java:91)~[metrics-core-3.1.0.jar:3.1.0] > at > com.codahale.metrics.MetricRegistry.registerAll(MetricRegistry.java:385)~[metrics-core-3.1.0.jar:3.1.0] > at > com.codahale.metrics.MetricRegistry.register(MetricRegistry.java:85)~[metrics-core-3.1.0.jar:3.1.0] > at > org.apache.spark.metrics.MetricsSystem.registerSource(MetricsSystem.scala:148) > ~[spark-core_2.11-1.4.0.jar:1.4.0] > at > org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:199) > [spark-streaming_2.11-1.4.0.jar:1.4.0] > at > org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:71) > [spark-streaming_2.11-1.4.0.jar:1.4.0] -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org