[
https://issues.apache.org/jira/browse/SPARK-38881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Khaitman updated SPARK-38881:
----------------------------------
Description:
This relates to https://issues.apache.org/jira/browse/SPARK-27420 which was
merged as part of Spark 3.0.0
This change is desirable as it further exposes the metricsLevel config
parameter that was added for the Scala/Java Spark APIs when working with the
Kinesis Streaming integration, and makes it available to the PySpark API as
well.
This change passes all tests, and local testing was done with a development
Kinesis stream in AWS, in order to confirm that metrics were no longer being
reported to CloudWatch after specifying MetricsLevel.NONE in the PySpark
Kinesis streaming context creation, and also worked as it does today when
leaving the MetricsLevel parameter out, which would result in a default of
DETAILED, with CloudWatch metrics appearing again.
https://github.com/apache/spark/pull/36201
was:
This relates to https://issues.apache.org/jira/browse/SPARK-27420 which was
merged as part of Spark 3.0.0
This change is desirable as it further exposes the metricsLevel config
parameter that was added for the Scala/Java Spark APIs when working with the
Kinesis Streaming integration, and makes it available to the PySpark API as
well.
This change passes all tests, and local testing was done with a development
Kinesis stream in AWS, in order to confirm that metrics were no longer being
reported to CloudWatch after specifying MetricsLevel.NONE in the PySpark
Kinesis streaming context creation, and also worked as it does today when
leaving the MetricsLevel parameter out, which would result in a default of
DETAILED, with CloudWatch metrics appearing again.
https://github.com/apache/spark/pull/36166
> PySpark Kinesis Streaming should expose metricsLevel CloudWatch config that
> is already supported in the Scala/Java APIs
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-38881
> URL: https://issues.apache.org/jira/browse/SPARK-38881
> Project: Spark
> Issue Type: Improvement
> Components: DStreams, Input/Output, PySpark
> Affects Versions: 3.2.1
> Reporter: Mark Khaitman
> Priority: Major
>
> This relates to https://issues.apache.org/jira/browse/SPARK-27420 which was
> merged as part of Spark 3.0.0
> This change is desirable as it further exposes the metricsLevel config
> parameter that was added for the Scala/Java Spark APIs when working with the
> Kinesis Streaming integration, and makes it available to the PySpark API as
> well.
> This change passes all tests, and local testing was done with a development
> Kinesis stream in AWS, in order to confirm that metrics were no longer being
> reported to CloudWatch after specifying MetricsLevel.NONE in the PySpark
> Kinesis streaming context creation, and also worked as it does today when
> leaving the MetricsLevel parameter out, which would result in a default of
> DETAILED, with CloudWatch metrics appearing again.
> https://github.com/apache/spark/pull/36201
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
