[ https://issues.apache.org/jira/browse/SPARK-22975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-22975: ------------------------------------ Assignee: Apache Spark > MetricsReporter producing NullPointerException when there was no progress > reported > ---------------------------------------------------------------------------------- > > Key: SPARK-22975 > URL: https://issues.apache.org/jira/browse/SPARK-22975 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.2.0, 2.2.1 > Reporter: Yuriy Bondaruk > Assignee: Apache Spark > > The exception occurs in MetricsReporter when it tries to register gauges > using lastProgress of each stream. > {code:java} > registerGauge("inputRate-total", () => > stream.lastProgress.inputRowsPerSecond) > registerGauge("processingRate-total", () => > stream.lastProgress.inputRowsPerSecond) > registerGauge("latency", () => > stream.lastProgress.durationMs.get("triggerExecution").longValue()) > {code} > In case if a stream doesn't have any progress reported than following > exception occurs: > {noformat} > 18/01/05 17:45:57 ERROR ScheduledReporter: RuntimeException thrown from > CloudwatchReporter#report. Exception was suppressed. > java.lang.NullPointerException > at > org.apache.spark.sql.execution.streaming.MetricsReporter$$anonfun$1.apply$mcD$sp(MetricsReporter.scala:42) > at > org.apache.spark.sql.execution.streaming.MetricsReporter$$anonfun$1.apply(MetricsReporter.scala:42) > at > org.apache.spark.sql.execution.streaming.MetricsReporter$$anonfun$1.apply(MetricsReporter.scala:42) > at > org.apache.spark.sql.execution.streaming.MetricsReporter$$anon$1.getValue(MetricsReporter.scala:49) > at > amazon.nexus.spark.metrics.cloudwatch.CloudwatchReporter.lambda$createNumericGaugeMetricDatumStream$0(CloudwatchReporter.java:146) > at > java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174) > at > java.util.Collections$UnmodifiableMap$UnmodifiableEntrySet.lambda$entryConsumer$0(Collections.java:1575) > at > java.util.TreeMap$EntrySpliterator.forEachRemaining(TreeMap.java:2969) > at > java.util.Collections$UnmodifiableMap$UnmodifiableEntrySet$UnmodifiableEntrySetSpliterator.forEachRemaining(Collections.java:1600) > at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) > at > java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) > at > java.util.stream.StreamSpliterators$WrappingSpliterator.forEachRemaining(StreamSpliterators.java:312) > at > java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742) > at > java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742) > at > java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742) > at > java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742) > at > java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742) > at > java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742) > at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) > at > java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) > at > java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) > at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) > at > java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:510) > at > amazon.nexus.spark.metrics.cloudwatch.CloudwatchReporter.partitionIntoSublists(CloudwatchReporter.java:390) > at > amazon.nexus.spark.metrics.cloudwatch.CloudwatchReporter.report(CloudwatchReporter.java:137) > at > com.codahale.metrics.ScheduledReporter.report(ScheduledReporter.java:162) > at > com.codahale.metrics.ScheduledReporter$1.run(ScheduledReporter.java:117) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org