Jerome Gagnon created SPARK-27420:
-------------------------------------
Summary: KinesisInputDStream should expose a way to disable
CloudWatch metrics
Key: SPARK-27420
URL: https://issues.apache.org/jira/browse/SPARK-27420
Project: Spark
Issue Type: Improvement
Components: DStreams, Input/Output
Affects Versions: 2.3.3
Reporter: Jerome Gagnon
KinesisInputDStream currently does not provide a way to disable CloudWatch
metrics push. Kinesis client library (KCL) which is used under the hood provide
the ability through `withMetrics` methods.
To make things worse the default level is "DETAILED" which pushes 10s of
metrics every 10 seconds. When dealing with multiple streaming jobs this add up
pretty quickly, leading to thousands of dollar in cost.
Exposing a way to disable/set the proper level of monitoring is critical to us.
We had to send invalid credentials and suppress log as a less-than-ideal
workaround : see
[https://stackoverflow.com/questions/41811039/disable-cloudwatch-for-aws-kinesis-at-spark-streaming/55599002#55599002]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
