Hello,
I have a question: is there a way to extend Spark REST API with higher-level,
application-specific handlers? (exposing additional information, like
app-specific metrics, but also taking some actions within an app)
Yes, I could host my own REST endpoint within an app, but then I have to solve
the problem of advertising that endpoint to the client. The app will be
deployed on YARN, so the port cannot be fixed (since multiple AMs can run on a
single machine). Spark's REST API endpoint is already "advertised" by YARN RM
to the client, so I was hoping I could just add to it.
I tried calling sparkContext.metricsSystem.getServletHandlers, but the Jetty
dependencies are shaded (I guess so the application could use a different
version of Jetty), so its de-facto signature is as follows:
public org.spark-project.jetty.servlet.ServletContextHandler[]
getServletHandlers();
(note the org.spark-project package)
...so when I add the Jetty dependency (org.eclipse.jetty...) to my project, it
compiles but during runtime it throws:
java.lang.NoSuchMethodError:
org.apache.spark.metrics.MetricsSystem.getServletHandlers()[Lorg/eclipse/jetty/servlet/ServletContextHandler;
So it seems to me that getServletHandlers is not intended for public use -
correct?
Is there any other way to extended Spark's REST API?
I would appreciate your help!
Thanks,
Sebastian
