We're doing similar thing to lunch spark job in tomcat, and I opened a JIRA for this. There are couple technical discussions there.
https://issues.apache.org/jira/browse/SPARK-2100 In this end, we realized that spark uses jetty not only for Spark WebUI, but also for distributing the jars and tasks, so it really hard to remove the web dependency in Spark. In the end, we lunch our spark job in yarn-cluster mode, and in the runtime, the only dependency in our web application is spark-yarn which doesn't contain any spark web stuff. PS, upgrading the spark jetty 8.x to 9.x in spark may not be straightforward by just changing the version in spark build script. Jetty 9.x required Java 7 since the servlet api (servlet 3.1) requires Java 7. Sincerely, DB Tsai ------------------------------------------------------- My Blog: https://www.dbtsai.com LinkedIn: https://www.linkedin.com/in/dbtsai On Tue, Jul 8, 2014 at 8:43 AM, Koert Kuipers <ko...@tresata.com> wrote: > do you control your cluster and spark deployment? if so, you can try to > rebuild with jetty 9.x > > > On Tue, Jul 8, 2014 at 9:39 AM, Martin Gammelsæter > <martingammelsae...@gmail.com> wrote: >> >> Digging a bit more I see that there is yet another jetty instance that >> is causing the problem, namely the BroadcastManager has one. I guess >> this one isn't very wise to disable... It might very well be that the >> WebUI is a problem as well, but I guess the code doesn't get far >> enough. Any ideas on how to solve this? Spark seems to use jetty >> 8.1.14, while dropwizard uses jetty 9.0.7, so that might be the source >> of the problem. Any ideas? >> >> On Tue, Jul 8, 2014 at 2:58 PM, Martin Gammelsæter >> <martingammelsae...@gmail.com> wrote: >> > Hi! >> > >> > I am building a web frontend for a Spark app, allowing users to input >> > sql/hql and get results back. When starting a SparkContext from within >> > my server code (using jetty/dropwizard) I get the error >> > >> > java.lang.NoSuchMethodError: >> > org.eclipse.jetty.server.AbstractConnector: method <init>()V not found >> > >> > when Spark tries to fire up its own jetty server. This does not happen >> > when running the same code without my web server. This is probably >> > fixable somehow(?) but I'd like to disable the webUI as I don't need >> > it, and ideally I would like to access that information >> > programatically instead, allowing me to embed it in my own web >> > application. >> > >> > Is this possible? >> > >> > -- >> > Best regards, >> > Martin Gammelsæter >> >> >> >> -- >> Mvh. >> Martin Gammelsæter >> 92209139 > >
