Hi, In the spark job, it exports to prometheus localhost http server, to be later scraped by prometheus service. (https://github.com/prometheus/client_java#http) The problem here is when ssh to the emr instances themselves, only can see the metrics on (e.g. curl localhost:9111) driver in local mode. If i run the spark job in cluster mode, the localhost:9111 is still available for curl but no data, executors i can not curl at all. Same scenario when running in kubernetes, i also check if the metrics are there or not by execing into the containers. Hope that makes the question clear here Prometheus supports pushgateway but thats for short batch job, not sure if thats applicable for my long running streaming job Thanks!
-- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
