Hello.
We’re successfully exporting technical metrics to prometheus using built-in
capabilities of Spark 3, but we need to add custom business metrics as well
using python. Seems like there’s no documentation for that.
Thanks.
just curious, where to write?
Anil Dasari wrote:
We are upgrading spark from 2.4.7 to 3.0.1. we use spark sql (hive) to
checkpoint data frames (intermediate data). DF write is very slow in
3.0.1 compared to 2.4.7.
-
To
Maybe you can give a look at this?
https://github.com/banzaicloud/spark-metrics
regards
Xinyu Luan wrote:
Can I get any suggestion or some examples for how to get the metrics
correctly.
-
To unsubscribe e-mail:
Hi, I am working on getting elastic search metrics(bytes_sent, bytes_accepted and bytes_retries) whenever we read/write data from elastic search in spark job. We tried to use spark listener, but still did get 0 byte by creating a listener class in the following attachment.package com.moesif.spark
The suggestion is to check:
1. Used format for write
2. Used parallelism
On Thu, Apr 14, 2022 at 7:13 PM Anil Dasari wrote:
> Hello,
>
>
>
> We are upgrading spark from 2.4.7 to 3.0.1. we use spark sql (hive) to
> checkpoint data frames (intermediate data). DF write is very slow in 3.0.1
>
Hello,
We are upgrading spark from 2.4.7 to 3.0.1. we use spark sql (hive) to
checkpoint data frames (intermediate data). DF write is very slow in 3.0.1
compared to 2.4.7.
Have read the release notes and there were no major changes except managed
tables and adaptive scheduling. We are not
This was bug in scala-library:2.12.11 and it is fixed in 2.12.15
https://github.com/scala/bug/issues/12419
https://github.com/scala/scala/pull/9676
spark-sql_2.12 v3.2.1 uses scala-library v2.12.15, but somehow an older
version of scala-library v2.12.11 which causes a problem and throws the
