Hi Saisai, As I mentioned,
I am getting very less information using the /metrics/json URI and *also the /metrics/master/json and /metrics/applications/json URIs do not seem to be working*. Please verify the following at your end. The metrics dumped using /metrics/json is as folllows: { "version": "3.0.0", "gauges": { "DAGScheduler.job.activeJobs": { "value": 2 }, "DAGScheduler.job.allJobs": { "value": 8 }, "DAGScheduler.stage.failedStages": { "value": 0 }, "DAGScheduler.stage.runningStages": { "value": 2 }, "DAGScheduler.stage.waitingStages": { "value": 0 }, "app-20151116150148-0001.driver.BlockManager.disk.diskSpaceUsed_MB": { "value": 0 }, "app-20151116150148-0001.driver.BlockManager.memory.maxMem_MB": { "value": 1590 }, "app-20151116150148-0001.driver.BlockManager.memory.memUsed_MB": { "value": 0 }, "app-20151116150148-0001.driver.BlockManager.memory.remainingMem_MB": { "value": 1590 }, "app-20151116150148-0001.driver.NetworkWordCount.StreamingMetrics.streaming.lastCompletedBatch_processingDelay": { "value": 142 }, "app-20151116150148-0001.driver.NetworkWordCount.StreamingMetrics.streaming.lastCompletedBatch_processingEndTime": { "value": 1447666380153 }, "app-20151116150148-0001.driver.NetworkWordCount.StreamingMetrics.streaming.lastCompletedBatch_processingStartTime": { "value": 1447666380011 }, "app-20151116150148-0001.driver.NetworkWordCount.StreamingMetrics.streaming.lastCompletedBatch_schedulingDelay": { "value": 0 }, "app-20151116150148-0001.driver.NetworkWordCount.StreamingMetrics.streaming.lastCompletedBatch_submissionTime": { "value": 1447666380011 }, "app-20151116150148-0001.driver.NetworkWordCount.StreamingMetrics.streaming.lastCompletedBatch_totalDelay": { "value": 142 }, "app-20151116150148-0001.driver.NetworkWordCount.StreamingMetrics.streaming.lastReceivedBatch_processingEndTime": { "value": 1447666380153 }, "app-20151116150148-0001.driver.NetworkWordCount.StreamingMetrics.streaming.lastReceivedBatch_processingStartTime": { "value": 1447666380011 }, "app-20151116150148-0001.driver.NetworkWordCount.StreamingMetrics.streaming.lastReceivedBatch_records": { "value": 2 }, "app-20151116150148-0001.driver.NetworkWordCount.StreamingMetrics.streaming.lastReceivedBatch_submissionTime": { "value": 1447666380011 }, "app-20151116150148-0001.driver.NetworkWordCount.StreamingMetrics.streaming.receivers": { "value": 2 }, "app-20151116150148-0001.driver.NetworkWordCount.StreamingMetrics.streaming.retainedCompletedBatches": { "value": 2 }, "app-20151116150148-0001.driver.NetworkWordCount.StreamingMetrics.streaming.runningBatches": { "value": 0 }, "app-20151116150148-0001.driver.NetworkWordCount.StreamingMetrics.streaming.totalCompletedBatches": { "value": 2 }, "app-20151116150148-0001.driver.NetworkWordCount.StreamingMetrics.streaming.totalProcessedRecords": { "value": 2 }, "app-20151116150148-0001.driver.NetworkWordCount.StreamingMetrics.streaming.totalReceivedRecords": { "value": 2 }, "app-20151116150148-0001.driver.NetworkWordCount.StreamingMetrics.streaming.unprocessedBatches": { "value": 0 }, "app-20151116150148-0001.driver.NetworkWordCount.StreamingMetrics.streaming.waitingBatches": { "value": 0 } }, "counters": { }, "histograms": { }, "meters": { }, "timers": { "DAGScheduler.messageProcessingTime": { "count": 161, "max": 213.21269999999998, "mean": 2.848791141751413, "min": 0.044309999999999995, "p50": 0.268713, "p75": 0.37620899999999996, "p95": 13.734098, "p98": 26.988401, "p99": 34.595124, "p999": 213.21269999999998, "stddev": 16.63943615966855, "m15_rate": 27.1366618014181, "m1_rate": 4.237092045329674, "m5_rate": 20.796710761839634, "mean_rate": 1.267708945789378, "duration_units": "milliseconds", "rate_units": "calls/second" } } } Thanks and Regards, Rakesh On Mon, Nov 16, 2015 at 3:18 PM, Saisai Shao <sai.sai.s...@gmail.com> wrote: > it should worked. I tested in my local environment with "curl > http://localhost:4040/metrics/json/", there's metrics dumped. For cluster > metrics, you have to change your base url to point to cluster manager. > > Thanks > Jerry > > On Mon, Nov 16, 2015 at 5:42 PM, ihavethepotential < > ihavethepotent...@gmail.com> wrote: > >> Hi all, >> >> I am trying to get the metrics using the MetricsServlet sink(that I guess >> is >> enabled by default) as mentioned in the Spark documentation: >> >> "5. MetricsServlet is added by default as a sink in master, worker and >> client >> # driver, you can send http request "/metrics/json" to get a snapshot of >> all >> the >> # registered metrics in json format. For master, requests >> "/metrics/master/json" and >> # "/metrics/applications/json" can be sent seperately to get metrics >> snapshot of >> # instance master and applications. MetricsServlet may not be configured >> by >> self." >> >> I am getting very less information using the /metrics/json URI and also >> the >> /metrics/master/json and /metrics/applications/json URIs do not seem to be >> working. >> >> My intent is to get the metrics for >> 1. Cluster summary >> 2. Apps running and completed >> 3. Executors summary of the apps >> 4. Job summary of the apps >> 5. DAGs metrics for the app jobs >> >> Help for the same is greatly appreciated. >> >> Thanks and Regards, >> Rakesh >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-enable-MetricsServlet-sink-in-Spark-1-5-0-tp25392.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> >> >