I am confused with your question. Are you running a the Spark cluster
on AWS EMR and trying to output the result to a Prometheus instance
running on your localhost? Isn't your localhost behind the firewall
and not accessible by AWS? What does it mean "have prometheus available
in executors"? Apparently you need to have a Prometheus instance
running on AWS so your EMR cluster can access easily.
Directing Spark output/sink to Prometheus would be difficult. The ideal
integration scenario would be to write a Spark customer connector that
uses the Prometheus client library to populate your Spark processing
result directly in Prometheus' database. Hope this helps...
-- ND
On 9/28/20 3:21 AM, Christine Gong wrote:
What should i do to expose my own custom prometheus metrics for
cluster mode spark streaming job?
I want to run a spark streaming job to read from kafka , do some
calculations and write to localhost prometheus on port 9111.
https://github.com/jaegertracing/jaeger-analytics-java/blob/master/spark/src/main/java/io/jaegertracing/analytics/spark/SparkRunner.java#L47
is it possible to have the prometheus available in executors? I tried
both emr cluster as well as k8s, only local mode works (the metrics
are available on driver's 9111 only)
Looks like the prometheus servlet sink is my best option? Any advice
would be much appreciated!!
Thanks,
Christine
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscr...@spark.apache.org
