Re: Replacing Jetty with TomCat
Hi Sean, The issue we have here is that all our products are based on a single platform and we try to make all our products coherent with our platform as much as possible. so, having two web services in one instance would not be a very elegant solution. That is why we were seeking a way to switch it to Tomcat. But as I understand, it is not readily supported, hence we will have to accept it as it is. If we are not using the Spark UIs, is it possible to disable the UIs and prevent the jetty server from starting, but yet use the core spark functionality? Hi Corey, thank you for your ideas. Our biggest concern here was that it starts a new webserver inside spark. opening up new ports etc. might be seen as security threats when it comes to commercial distributions. cheers On Wed, Feb 18, 2015 at 3:25 PM, Sean Owen so...@cloudera.com wrote: I do not think it makes sense to make the web server configurable. Mostly because there's no real problem in running an HTTP service internally based on Netty while you run your own HTTP service based on something else like Tomcat. What's the problem? On Wed, Feb 18, 2015 at 3:14 AM, Niranda Perera niranda.per...@gmail.com wrote: Hi Sean, The main issue we have is, running two web servers in a single product. we think it would not be an elegant solution. Could you please point me to the main areas where jetty server is tightly coupled or extension points where I could plug tomcat instead of jetty? If successful I could contribute it to the spark project. :-) cheers On Mon, Feb 16, 2015 at 4:51 PM, Sean Owen so...@cloudera.com wrote: There's no particular reason you have to remove the embedded Jetty server, right? it doesn't prevent you from using it inside another app that happens to run in Tomcat. You won't be able to switch it out without rewriting a fair bit of code, no, but you don't need to. On Mon, Feb 16, 2015 at 5:08 AM, Niranda Perera niranda.per...@gmail.com wrote: Hi, We are thinking of integrating Spark server inside a product. Our current product uses Tomcat as its webserver. Is it possible to switch the Jetty webserver in Spark to Tomcat off-the-shelf? Cheers -- Niranda -- Niranda -- Niranda
Re: Replacing Jetty with TomCat
To add to Sean and Reynold's point: Please correct me if I'm wrong, but Spark depends on hadoop-common which also uses jetty in the HttpServer2 code. So even if you remove jetty from Spark by making it an optional dependency, it will be pulled in by Hadoop. So you'll still see that your program that depends on hypothetical Spark-Tomcat will still pull in jetty jars. -Ewan On 19/02/15 10:23, Sean Owen wrote: Sure, but you are not using Netty at all. It's invisible to you. It's not as if you have to set up and maintain a Jetty container. I don't think your single platform for your apps is relevant. You can turn off the UI, but as Reynold said, the HTTP servers are also part of the core data transport functionality and you can't turn that off. It's not merely unsupported to swap this out with an arbitrary container, it's not clear it would work with Tomcat without re-integrating with its behavior and tuning. But it also shouldn't matter to anyone. On Thu, Feb 19, 2015 at 8:11 AM, Niranda Perera niranda.per...@gmail.com wrote: Hi Sean, The issue we have here is that all our products are based on a single platform and we try to make all our products coherent with our platform as much as possible. so, having two web services in one instance would not be a very elegant solution. That is why we were seeking a way to switch it to Tomcat. But as I understand, it is not readily supported, hence we will have to accept it as it is. If we are not using the Spark UIs, is it possible to disable the UIs and prevent the jetty server from starting, but yet use the core spark functionality? Hi Corey, thank you for your ideas. Our biggest concern here was that it starts a new webserver inside spark. opening up new ports etc. might be seen as security threats when it comes to commercial distributions. cheers On Wed, Feb 18, 2015 at 3:25 PM, Sean Owen so...@cloudera.com wrote: I do not think it makes sense to make the web server configurable. Mostly because there's no real problem in running an HTTP service internally based on Netty while you run your own HTTP service based on something else like Tomcat. What's the problem? On Wed, Feb 18, 2015 at 3:14 AM, Niranda Perera niranda.per...@gmail.com wrote: Hi Sean, The main issue we have is, running two web servers in a single product. we think it would not be an elegant solution. Could you please point me to the main areas where jetty server is tightly coupled or extension points where I could plug tomcat instead of jetty? If successful I could contribute it to the spark project. :-) cheers On Mon, Feb 16, 2015 at 4:51 PM, Sean Owen so...@cloudera.com wrote: There's no particular reason you have to remove the embedded Jetty server, right? it doesn't prevent you from using it inside another app that happens to run in Tomcat. You won't be able to switch it out without rewriting a fair bit of code, no, but you don't need to. On Mon, Feb 16, 2015 at 5:08 AM, Niranda Perera niranda.per...@gmail.com wrote: Hi, We are thinking of integrating Spark server inside a product. Our current product uses Tomcat as its webserver. Is it possible to switch the Jetty webserver in Spark to Tomcat off-the-shelf? Cheers -- Niranda -- Niranda -- Niranda - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: Replacing Jetty with TomCat
Sure, but you are not using Netty at all. It's invisible to you. It's not as if you have to set up and maintain a Jetty container. I don't think your single platform for your apps is relevant. You can turn off the UI, but as Reynold said, the HTTP servers are also part of the core data transport functionality and you can't turn that off. It's not merely unsupported to swap this out with an arbitrary container, it's not clear it would work with Tomcat without re-integrating with its behavior and tuning. But it also shouldn't matter to anyone. On Thu, Feb 19, 2015 at 8:11 AM, Niranda Perera niranda.per...@gmail.com wrote: Hi Sean, The issue we have here is that all our products are based on a single platform and we try to make all our products coherent with our platform as much as possible. so, having two web services in one instance would not be a very elegant solution. That is why we were seeking a way to switch it to Tomcat. But as I understand, it is not readily supported, hence we will have to accept it as it is. If we are not using the Spark UIs, is it possible to disable the UIs and prevent the jetty server from starting, but yet use the core spark functionality? Hi Corey, thank you for your ideas. Our biggest concern here was that it starts a new webserver inside spark. opening up new ports etc. might be seen as security threats when it comes to commercial distributions. cheers On Wed, Feb 18, 2015 at 3:25 PM, Sean Owen so...@cloudera.com wrote: I do not think it makes sense to make the web server configurable. Mostly because there's no real problem in running an HTTP service internally based on Netty while you run your own HTTP service based on something else like Tomcat. What's the problem? On Wed, Feb 18, 2015 at 3:14 AM, Niranda Perera niranda.per...@gmail.com wrote: Hi Sean, The main issue we have is, running two web servers in a single product. we think it would not be an elegant solution. Could you please point me to the main areas where jetty server is tightly coupled or extension points where I could plug tomcat instead of jetty? If successful I could contribute it to the spark project. :-) cheers On Mon, Feb 16, 2015 at 4:51 PM, Sean Owen so...@cloudera.com wrote: There's no particular reason you have to remove the embedded Jetty server, right? it doesn't prevent you from using it inside another app that happens to run in Tomcat. You won't be able to switch it out without rewriting a fair bit of code, no, but you don't need to. On Mon, Feb 16, 2015 at 5:08 AM, Niranda Perera niranda.per...@gmail.com wrote: Hi, We are thinking of integrating Spark server inside a product. Our current product uses Tomcat as its webserver. Is it possible to switch the Jetty webserver in Spark to Tomcat off-the-shelf? Cheers -- Niranda -- Niranda -- Niranda - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: Replacing Jetty with TomCat
I do not think it makes sense to make the web server configurable. Mostly because there's no real problem in running an HTTP service internally based on Netty while you run your own HTTP service based on something else like Tomcat. What's the problem? On Wed, Feb 18, 2015 at 3:14 AM, Niranda Perera niranda.per...@gmail.com wrote: Hi Sean, The main issue we have is, running two web servers in a single product. we think it would not be an elegant solution. Could you please point me to the main areas where jetty server is tightly coupled or extension points where I could plug tomcat instead of jetty? If successful I could contribute it to the spark project. :-) cheers On Mon, Feb 16, 2015 at 4:51 PM, Sean Owen so...@cloudera.com wrote: There's no particular reason you have to remove the embedded Jetty server, right? it doesn't prevent you from using it inside another app that happens to run in Tomcat. You won't be able to switch it out without rewriting a fair bit of code, no, but you don't need to. On Mon, Feb 16, 2015 at 5:08 AM, Niranda Perera niranda.per...@gmail.com wrote: Hi, We are thinking of integrating Spark server inside a product. Our current product uses Tomcat as its webserver. Is it possible to switch the Jetty webserver in Spark to Tomcat off-the-shelf? Cheers -- Niranda -- Niranda - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: Replacing Jetty with TomCat
Hey Niranda, It seems to me a lot of effort to support multiple libraries inside of Spark like this, so I'm not sure that's a great solution. If you are building an application that embeds Spark, is it not possible for you to continue to use Jetty for Spark's internal servers and use tomcat for your own server's? I would guess that many complex applications end up embedding multiple server libraries in various places (Spark itself has different transport mechanisms, etc.) - Patrick On Tue, Feb 17, 2015 at 7:14 PM, Niranda Perera niranda.per...@gmail.com wrote: Hi Sean, The main issue we have is, running two web servers in a single product. we think it would not be an elegant solution. Could you please point me to the main areas where jetty server is tightly coupled or extension points where I could plug tomcat instead of jetty? If successful I could contribute it to the spark project. :-) cheers On Mon, Feb 16, 2015 at 4:51 PM, Sean Owen so...@cloudera.com wrote: There's no particular reason you have to remove the embedded Jetty server, right? it doesn't prevent you from using it inside another app that happens to run in Tomcat. You won't be able to switch it out without rewriting a fair bit of code, no, but you don't need to. On Mon, Feb 16, 2015 at 5:08 AM, Niranda Perera niranda.per...@gmail.com wrote: Hi, We are thinking of integrating Spark server inside a product. Our current product uses Tomcat as its webserver. Is it possible to switch the Jetty webserver in Spark to Tomcat off-the-shelf? Cheers -- Niranda -- Niranda - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: Replacing Jetty with TomCat
Hi Sean, The main issue we have is, running two web servers in a single product. we think it would not be an elegant solution. Could you please point me to the main areas where jetty server is tightly coupled or extension points where I could plug tomcat instead of jetty? If successful I could contribute it to the spark project. :-) cheers On Mon, Feb 16, 2015 at 4:51 PM, Sean Owen so...@cloudera.com wrote: There's no particular reason you have to remove the embedded Jetty server, right? it doesn't prevent you from using it inside another app that happens to run in Tomcat. You won't be able to switch it out without rewriting a fair bit of code, no, but you don't need to. On Mon, Feb 16, 2015 at 5:08 AM, Niranda Perera niranda.per...@gmail.com wrote: Hi, We are thinking of integrating Spark server inside a product. Our current product uses Tomcat as its webserver. Is it possible to switch the Jetty webserver in Spark to Tomcat off-the-shelf? Cheers -- Niranda -- Niranda
Re: Replacing Jetty with TomCat
Niranda, I'm not sure if I'd say Spark's use of Jetty to expose its UI monitoring layer constitutes a use of two web servers in a single product. Hadoop uses Jetty as well as do many other applications today that need embedded http layers for serving up their monitoring UI to users. This is completely aside from any web container an application developer would use to interact with Spark and Hadoop and service domain-specific content to users. The two are disjoint. Many applications use Thrift as a means of establishing socket connections between clients and across servers. One alternative to Thrift is Protobuf. You wouldn't say I want to swap out thrift for protobuf in Cassandra because I want to use protobuf in my application and there shouldn't be two different socket layer abstractions on my cluster. I could understand wanting to do this if you were being forced to deploy a war file to a web container in order to do the monitoring but Spark's UI is embedded within the code. If you are worried about having the Jetty libraries on your classpath, you can exclude the Jetty dependencies from your servlet code if you want to interact with a SparkContext in Tomcat. On Tue, Feb 17, 2015 at 10:22 PM, Patrick Wendell pwend...@gmail.com wrote: Hey Niranda, It seems to me a lot of effort to support multiple libraries inside of Spark like this, so I'm not sure that's a great solution. If you are building an application that embeds Spark, is it not possible for you to continue to use Jetty for Spark's internal servers and use tomcat for your own server's? I would guess that many complex applications end up embedding multiple server libraries in various places (Spark itself has different transport mechanisms, etc.) - Patrick On Tue, Feb 17, 2015 at 7:14 PM, Niranda Perera niranda.per...@gmail.com wrote: Hi Sean, The main issue we have is, running two web servers in a single product. we think it would not be an elegant solution. Could you please point me to the main areas where jetty server is tightly coupled or extension points where I could plug tomcat instead of jetty? If successful I could contribute it to the spark project. :-) cheers On Mon, Feb 16, 2015 at 4:51 PM, Sean Owen so...@cloudera.com wrote: There's no particular reason you have to remove the embedded Jetty server, right? it doesn't prevent you from using it inside another app that happens to run in Tomcat. You won't be able to switch it out without rewriting a fair bit of code, no, but you don't need to. On Mon, Feb 16, 2015 at 5:08 AM, Niranda Perera niranda.per...@gmail.com wrote: Hi, We are thinking of integrating Spark server inside a product. Our current product uses Tomcat as its webserver. Is it possible to switch the Jetty webserver in Spark to Tomcat off-the-shelf? Cheers -- Niranda -- Niranda - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: Replacing Jetty with TomCat
There's no particular reason you have to remove the embedded Jetty server, right? it doesn't prevent you from using it inside another app that happens to run in Tomcat. You won't be able to switch it out without rewriting a fair bit of code, no, but you don't need to. On Mon, Feb 16, 2015 at 5:08 AM, Niranda Perera niranda.per...@gmail.com wrote: Hi, We are thinking of integrating Spark server inside a product. Our current product uses Tomcat as its webserver. Is it possible to switch the Jetty webserver in Spark to Tomcat off-the-shelf? Cheers -- Niranda - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: Replacing Jetty with TomCat
Most likely no. We are using the embedded mode of Jetty, rather than using servlets. Even if it is possible, you probably wouldn't want to embed Spark in your application server ... On Sun, Feb 15, 2015 at 9:08 PM, Niranda Perera niranda.per...@gmail.com wrote: Hi, We are thinking of integrating Spark server inside a product. Our current product uses Tomcat as its webserver. Is it possible to switch the Jetty webserver in Spark to Tomcat off-the-shelf? Cheers -- Niranda
Re: Replacing Jetty with TomCat
Hi Reynold, Thank you for the response. Could you please clarify the need of Jetty server inside Spark? Is it used for Spark core functionality or is it there for Spark jobs UI purposes? cheers On Mon, Feb 16, 2015 at 10:47 AM, Reynold Xin r...@databricks.com wrote: Most likely no. We are using the embedded mode of Jetty, rather than using servlets. Even if it is possible, you probably wouldn't want to embed Spark in your application server ... On Sun, Feb 15, 2015 at 9:08 PM, Niranda Perera niranda.per...@gmail.com wrote: Hi, We are thinking of integrating Spark server inside a product. Our current product uses Tomcat as its webserver. Is it possible to switch the Jetty webserver in Spark to Tomcat off-the-shelf? Cheers -- Niranda -- Niranda
Re: Replacing Jetty with TomCat
Mostly UI. However, we are also using Jetty as a file server I believe (for distributing files from the driver to workers). On Sun, Feb 15, 2015 at 9:24 PM, Niranda Perera niranda.per...@gmail.com wrote: Hi Reynold, Thank you for the response. Could you please clarify the need of Jetty server inside Spark? Is it used for Spark core functionality or is it there for Spark jobs UI purposes? cheers On Mon, Feb 16, 2015 at 10:47 AM, Reynold Xin r...@databricks.com wrote: Most likely no. We are using the embedded mode of Jetty, rather than using servlets. Even if it is possible, you probably wouldn't want to embed Spark in your application server ... On Sun, Feb 15, 2015 at 9:08 PM, Niranda Perera niranda.per...@gmail.com wrote: Hi, We are thinking of integrating Spark server inside a product. Our current product uses Tomcat as its webserver. Is it possible to switch the Jetty webserver in Spark to Tomcat off-the-shelf? Cheers -- Niranda -- Niranda
