Spark REST APIs not working when application deployed in CLUSTER mode
Hi all, I am able to make the REST API calls in client mode but when I run the application in Cluster mode, I am getting the below exception in driver logs: java.lang.IncompatibleClassChangeError: com.sun.jersey.json.impl.provider.entity.JSONRootElementProvider and com.sun.jersey.json.impl.provider.entity.JSONRootElementProvider$Wadl disagree on InnerClasses attribute Seems the jersey container is not able to initialize due to jar conflicts. Please help. Thanks and Regards, Rakesh
Unable to get json for application jobs in spark 1.5.0
Hi all, I am trying to get the json for all the jobs running within my application whose UI port is 4040. I am making an HTTP GET request at the following URI: http://172.26.32.143:4040/api/v1/applications/gauravpipe/jobs But getting the following service unavailable exception: Error 503 Service Unavailable HTTP ERROR 503 Problem accessing /api/v1/applications/gauravpipe/jobs. Reason: Service UnavailableCaused by:org.spark-project.jetty.servlet.ServletHolder$1: java.lang.reflect.InvocationTargetException at org.spark-project.jetty.servlet.ServletHolder.makeUnavailable(ServletHolder.java:496) at org.spark-project.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:543) at org.spark-project.jetty.servlet.ServletHolder.getServlet(ServletHolder.java:415) at org.spark-project.jetty.servlet.ServletHolder.handle(ServletHolder.java:657) at org.spark-project.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501) at org.spark-project.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) at org.spark-project.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428) at org.spark-project.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) at org.spark-project.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.spark-project.jetty.server.handler.GzipHandler.handle(GzipHandler.java:264) at org.spark-project.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255) at org.spark-project.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.spark-project.jetty.server.Server.handle(Server.java:370) at org.spark-project.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494) at org.spark-project.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971) at org.spark-project.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033) at org.spark-project.jetty.http.HttpParser.parseNext(HttpParser.java:644) at org.spark-project.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) at org.spark-project.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) at org.spark-project.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667) at org.spark-project.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) at org.spark-project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.spark-project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at com.sun.jersey.spi.container.servlet.WebComponent.createResourceConfig(WebComponent.java:728) at com.sun.jersey.spi.container.servlet.WebComponent.createResourceConfig(WebComponent.java:678) at com.sun.jersey.spi.container.servlet.WebComponent.init(WebComponent.java:203) at com.sun.jersey.spi.container.servlet.ServletContainer.init(ServletContainer.java:373) at com.sun.jersey.spi.container.servlet.ServletContainer.init(ServletContainer.java:556) at javax.servlet.GenericServlet.init(GenericServlet.java:241) at org.spark-project.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:532) ... 22 more Caused by: java.lang.IncompatibleClassChangeError: Implementing class at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:800) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) at java.net.URLClassLoader.access$100(URLClassLoader.java:71) at java.net.URLClassLoader$1.run(URLClassLoader.java:361) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) at com.sun.jersey.api.core.ScanningResourceConfig.init(ScanningResourceConfig.java:79) at
Spark Monitoring to get Spark GCs and records processed
Hi all, I want to monitor Spark to get the following: 1. All the GC stats for Spark JVMs 2. Records successfully processed in a batch 3. Records failed in a batch 4. Getting historical data for batches,jobs,stages,tasks,etc, Please let me know how can I get these information in Spark. Regards, Rakesh
Re: How to enable MetricsServlet sink in Spark 1.5.0?
Hi Saisai,
As I mentioned,
I am getting very less information using the /metrics/json URI and
*also the /metrics/master/json and /metrics/applications/json URIs do not
seem to be working*.
Please verify the following at your end.
The metrics dumped using /metrics/json is as folllows:
{
"version": "3.0.0",
"gauges":
{
"DAGScheduler.job.activeJobs":
{
"value": 2
},
"DAGScheduler.job.allJobs":
{
"value": 8
},
"DAGScheduler.stage.failedStages":
{
"value": 0
},
"DAGScheduler.stage.runningStages":
{
"value": 2
},
"DAGScheduler.stage.waitingStages":
{
"value": 0
},
"app-20151116150148-0001.driver.BlockManager.disk.diskSpaceUsed_MB":
{
"value": 0
},
"app-20151116150148-0001.driver.BlockManager.memory.maxMem_MB":
{
"value": 1590
},
"app-20151116150148-0001.driver.BlockManager.memory.memUsed_MB":
{
"value": 0
},
"app-20151116150148-0001.driver.BlockManager.memory.remainingMem_MB":
{
"value": 1590
},
"app-20151116150148-0001.driver.NetworkWordCount.StreamingMetrics.streaming.lastCompletedBatch_processingDelay":
{
"value": 142
},
"app-20151116150148-0001.driver.NetworkWordCount.StreamingMetrics.streaming.lastCompletedBatch_processingEndTime":
{
"value": 1447666380153
},
"app-20151116150148-0001.driver.NetworkWordCount.StreamingMetrics.streaming.lastCompletedBatch_processingStartTime":
{
"value": 1447666380011
},
"app-20151116150148-0001.driver.NetworkWordCount.StreamingMetrics.streaming.lastCompletedBatch_schedulingDelay":
{
"value": 0
},
"app-20151116150148-0001.driver.NetworkWordCount.StreamingMetrics.streaming.lastCompletedBatch_submissionTime":
{
"value": 1447666380011
},
"app-20151116150148-0001.driver.NetworkWordCount.StreamingMetrics.streaming.lastCompletedBatch_totalDelay":
{
"value": 142
},
"app-20151116150148-0001.driver.NetworkWordCount.StreamingMetrics.streaming.lastReceivedBatch_processingEndTime":
{
"value": 1447666380153
},
"app-20151116150148-0001.driver.NetworkWordCount.StreamingMetrics.streaming.lastReceivedBatch_processingStartTime":
{
"value": 1447666380011
},
"app-20151116150148-0001.driver.NetworkWordCount.StreamingMetrics.streaming.lastReceivedBatch_records":
{
"value": 2
},
"app-20151116150148-0001.driver.NetworkWordCount.StreamingMetrics.streaming.lastReceivedBatch_submissionTime":
{
"value": 1447666380011
},
"app-20151116150148-0001.driver.NetworkWordCount.StreamingMetrics.streaming.receivers":
{
"value": 2
},
"app-20151116150148-0001.driver.NetworkWordCount.StreamingMetrics.streaming.retainedCompletedBatches":
{
"value": 2
},
"app-20151116150148-0001.driver.NetworkWordCount.StreamingMetrics.streaming.runningBatches":
{
"value": 0
},
"app-20151116150148-0001.driver.NetworkWordCount.StreamingMetrics.streaming.totalCompletedBatches":
{
"value": 2
},
"app-20151116150148-0001.driver.NetworkWordCount.StreamingMetrics.streaming.totalProcessedRecords":
{
"value": 2
},
"app-20151116150148-0001.driver.NetworkWordCount.StreamingMetrics.streaming.totalReceivedRecords":
{
"value": 2
},
"app-20151116150148-0001.driver.NetworkWordCount.StreamingMetrics.streaming.unprocessedBatches":
{
"value": 0
},
"app-20151116150148-0001.driver.NetworkWordCount.StreamingMetrics.streaming.waitingBatches":
{
"value": 0
}
},
"counters":
{
},
"histograms":
{
},
"meters":
{
},
"timers":
{
"DAGScheduler.messageProcessingTime":
{
"count": 161,
"max": 213.212698,
"mean": 2.848791141751413,
"min": 0.044305,
"p50": 0.268713,
"p75": 0.376208996,
"p95": 13.734098,
"p98": 26.988401,
"p99": 34.595124,
"p999": 213.212698,
"stddev": 16.63943615966855,
"m15_rate": 27.1366618014181,
"m1_rate":
