[
https://issues.apache.org/jira/browse/SPARK-27420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kengo Seki updated SPARK-27420:
-------------------------------
Summary: KinesisInputDStream should expose a way to configure CloudWatch
metrics (was: KinesisInputDStream should expose a way to disable CloudWatch
metrics)
> KinesisInputDStream should expose a way to configure CloudWatch metrics
> -----------------------------------------------------------------------
>
> Key: SPARK-27420
> URL: https://issues.apache.org/jira/browse/SPARK-27420
> Project: Spark
> Issue Type: Improvement
> Components: DStreams, Input/Output
> Affects Versions: 2.3.3
> Reporter: Jerome Gagnon
> Priority: Major
>
> KinesisInputDStream currently does not provide a way to disable CloudWatch
> metrics push. Kinesis client library (KCL) which is used under the hood
> provide the ability through `withMetrics` methods.
> To make things worse the default level is "DETAILED" which pushes 10s of
> metrics every 10 seconds. When dealing with multiple streaming jobs this add
> up pretty quickly, leading to thousands of dollar in cost.
> Exposing a way to disable/set the proper level of monitoring is critical to
> us. We had to send invalid credentials and suppress log as a less-than-ideal
> workaround : see
> [https://stackoverflow.com/questions/41811039/disable-cloudwatch-for-aws-kinesis-at-spark-streaming/55599002#55599002]
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
