Re: mod_perl in larger scale environments
Brad Van Sickle bvs7...@gmail.com wrote: What are you guys using to cluster the LVS servers? Is that functionality native to LVS? The keepalived daemon provides this. Can be triggered manually (I'm doing this for a master-master mysql setup) or automatically, for our backend LVS system. This is achieved by specific incantations in the keepalived.conf config file. Shout if you want more details. -- Cosimo
Re: mod_perl in larger scale environments
On Thu, Apr 15, 2010 at 8:17 PM, Brad Van Sickle bvs7...@gmail.com wrote: What are you guys using to cluster the LVS servers? Is that functionality native to LVS? There's a heartbeat system you can use for HA. If you look at the docs for LVS you'll see info on it. Also, I've come across fastcgi a lot in researching this and I'm having trouble understanding exactly what it is. FastCGI is a protocol for sending web requests to a daemon. It is implemented by multiple web servers and the daemon can be in any language. There's plenty of perl support for FastCGI and apache supports it, as do nginx and varnish and many others. The fastcgi website makes it sound like it's almost a mod_perl replacement and another approach to executing perl code in a web environment Correct, it's an alternative to mod_perl. You can find discussions in the list archives about the tradeoffs. Basically, you can't use any of the apache API stuff, like filters or the auth system. - Perrin
Re: mod_perl in larger scale environments
In data 15 aprile 2010 alle ore 05:11:15, Brad Van Sickle bvs7...@gmail.com ha scritto: LVS does sound interesting but in your infrastructure layout aren't your single LVS load balancers single points of failure? I simplified a bit too much :) Every LVS machine has a hot-spare, and you can perform manual or automated failover. Automated failover is said to keep your connections running while migrating them over to the backup lvs. We have never had a failure, just manual failover due to upgrades, etc... I'm ignorant there. What advantages exactly? I'l be brief here because this is a mod_perl list :) [...] Thanks :) -- Cosimo
Re: mod_perl in larger scale environments
On Thu, Apr 15, 2010 at 4:38 PM, Cosimo Streppone cos...@streppone.it wrote: In data 15 aprile 2010 alle ore 05:11:15, Brad Van Sickle bvs7...@gmail.com ha scritto: LVS does sound interesting but in your infrastructure layout aren't your single LVS load balancers single points of failure? I simplified a bit too much :) Every LVS machine has a hot-spare, and you can perform manual or automated failover. Automated failover is said to keep your connections running while migrating them over to the backup lvs. We have never had a failure, just manual failover due to upgrades, etc... We use LVS to load balance our reverse proxies as well as our app servers. - 2 LVS servers using heartbeat for automatic failover (we are looking to switch to keepalived instead of heartbeat in the future), - 3 nginx servers which do content compression and ssl offloading as well as caching (we don't need 3 of them but we like the redundancy and the ability to drop one without impacting performance) - 5 app servers running apache and mod_perl We have just switched to nginx from squid in the last few months and have been very happy with it. nginx can also deliver static content directly or act as a FastCGI frontend (relaying the requests to backend app servers) as well as many other things. But our main reason for switching to nginx was the ability to offload SSL requests and remove that complexity from the app servers (we previously used squid as our reverse proxy which can't do SSL offloading). nginx can do it's own load balancing as well but we preferred to use our existing LVS infrastructure to handle that for us. As an added bonus, LVS also load balances our mail cluster... Cheers, Cees Hek
Re: mod_perl in larger scale environments
What are you guys using to cluster the LVS servers? Is that functionality native to LVS? Also, I've come across fastcgi a lot in researching this and I'm having trouble understanding exactly what it is. The fastcgi website makes it sound like it's almost a mod_perl replacement and another approach to executing perl code in a web environment, however a lot of the webservers that implement fastcgi keep referring to it almost like it's a protocol that proxy servers use to talk to their back end app servers... I'm a bit confused about what it is, what situations it's used in, and if it provides me any benefit if I'm already married to mod_perl. I know this probably isn't the best place to ask this question, but can someone clarify this for me? On 4/15/2010 8:04 PM, Cees Hek wrote: On Thu, Apr 15, 2010 at 4:38 PM, Cosimo Strepponecos...@streppone.it wrote: In data 15 aprile 2010 alle ore 05:11:15, Brad Van Sickle bvs7...@gmail.com ha scritto: LVS does sound interesting but in your infrastructure layout aren't your single LVS load balancers single points of failure? I simplified a bit too much :) Every LVS machine has a hot-spare, and you can perform manual or automated failover. Automated failover is said to keep your connections running while migrating them over to the backup lvs. We have never had a failure, just manual failover due to upgrades, etc... We use LVS to load balance our reverse proxies as well as our app servers. - 2 LVS servers using heartbeat for automatic failover (we are looking to switch to keepalived instead of heartbeat in the future), - 3 nginx servers which do content compression and ssl offloading as well as caching (we don't need 3 of them but we like the redundancy and the ability to drop one without impacting performance) - 5 app servers running apache and mod_perl We have just switched to nginx from squid in the last few months and have been very happy with it. nginx can also deliver static content directly or act as a FastCGI frontend (relaying the requests to backend app servers) as well as many other things. But our main reason for switching to nginx was the ability to offload SSL requests and remove that complexity from the app servers (we previously used squid as our reverse proxy which can't do SSL offloading). nginx can do it's own load balancing as well but we preferred to use our existing LVS infrastructure to handle that for us. As an added bonus, LVS also load balances our mail cluster... Cheers, Cees Hek
mod_perl in larger scale environments
Hello I have a lot of experience in large scale web applications using Java and Websphere, but I now find myself needing to scale a web application built on mod_perl, and I have some questions about best practices for doing that since I don't have any sort of deployment manager or an intelligent HTTP plugin. I currently have an application set up in the standard 3 tiered model: Apache web layer - Apache/mod_perl app layer - MySQL DB layer. Right now I have one app layer node, but traffic is dictating that I need to expand capacity there soon and I plan on adding more hosts to that layer. My first question relates to quality of service and load balancing: I'm currently using mod_proxy on the web layer, and I know I can set that up to load balance requests to multiple app layer nodes, but to the best of my knowledge mod_proxy is not able to provide any quality of service. So if a node in the app layer had a problem (or was shut down for maintenance) mod_proxy would be unaware of that and would still send requests to that node. How are situations like this normally handled? Is there something I can use other than mod_proxy that is intelligent enough to mark a host as down? I'd rather not use a hardware load balancer here if I can avoid it. My second question deals with management of multiple mod_perl nodes: At some point, if you have enough app layer nodes, managing the code deployments, apache configs and server restarts becomes very cumbersome if you're doing it all manually. Are there any tools that can make these tasks easier or give me one management view? Again, I'm used to working in web applications with a full blown app server. I love working with mod_perl, but I do find myself missing the advantages an app server gives me sometimes. Hopefully someone can offer me some suggestions here. Thanks
Re: mod_perl in larger scale environments
On Wed, Apr 14, 2010 at 1:57 PM, Brad Van Sickle bvs7...@gmail.com wrote: My first question relates to quality of service and load balancing: I'm currently using mod_proxy on the web layer, and I know I can set that up to load balance requests to multiple app layer nodes, but to the best of my knowledge mod_proxy is not able to provide any quality of service. So if a node in the app layer had a problem (or was shut down for maintenance) mod_proxy would be unaware of that and would still send requests to that node. How are situations like this normally handled? Is there something I can use other than mod_proxy that is intelligent enough to mark a host as down? I'd rather not use a hardware load balancer here if I can avoid it. Check out perlbal - http://search.cpan.org/dist/Perlbal/ The load balancing is really nice, and you can handle 10k's of clients with junkyard hardware. My second question deals with management of multiple mod_perl nodes: At some point, if you have enough app layer nodes, managing the code deployments, apache configs and server restarts becomes very cumbersome if you're doing it all manually. Are there any tools that can make these tasks easier or give me one management view? Puppet and cfmanager are general purpose systems administration tools that serve well here. In addition, you can manage the rest of the system with these tools.
Re: mod_perl in larger scale environments
On Wed, Apr 14, 2010 at 5:33 PM, Brad Van Sickle bvs7...@gmail.com wrote: I didn't find much info on perlbal after a quick glance, I'll certainly give it a closer look, but my inital reaction is that I'm leary of replacing Apache on my web layer. I'm doing a few things with a few other modules ( mod_rewrite for example) in addition to mod_proxy, and from what I was able to find in my initial look, I didn't see any support for some of those types of things. There are many full-featured proxy servers these days. There's even mod_proxy_balancer for apache, but that doesn't do high-availability, which you're looking for. Check out some of these for reverse-proxying if you don't like perlbal: - nginx - lighttpd - varnish - pound All of those can serve as mod_perl frontends. - Perrin
Re: mod_perl in larger scale environments
So it sounds like Apache is simply not going to meet my needs. In the event that I do need to replace Apache, hopefully you can save me some research time and recommend me one of the listed options that fulfills my needs (or confirm that perlbal does) I need the following features: 1) provides support for named virtual hosts 2) supports SSL to the client 3) supports URL rewriting (similar to mod_rewrite) 4) knows the availability of pool members and provides high availability. 5) the ability to serve static content itself I guess 5 isn't strictly neccessary, but it would be nice to serve static content (css/js/images/etc...) from the same piece of technology without proxying those requests to another Apache instance running on the same host (or something) Thanks for all the help! On 4/14/2010 5:48 PM, Perrin Harkins wrote: On Wed, Apr 14, 2010 at 5:33 PM, Brad Van Sicklebvs7...@gmail.com wrote: I didn't find much info on perlbal after a quick glance, I'll certainly give it a closer look, but my inital reaction is that I'm leary of replacing Apache on my web layer. I'm doing a few things with a few other modules ( mod_rewrite for example) in addition to mod_proxy, and from what I was able to find in my initial look, I didn't see any support for some of those types of things. There are many full-featured proxy servers these days. There's even mod_proxy_balancer for apache, but that doesn't do high-availability, which you're looking for. Check out some of these for reverse-proxying if you don't like perlbal: - nginx - lighttpd - varnish - pound All of those can serve as mod_perl frontends. - Perrin
Re: mod_perl in larger scale environments
On Wed, Apr 14, 2010 at 3:15 PM, Brad Van Sickle bvs7...@gmail.com wrote: So it sounds like Apache is simply not going to meet my needs. In the event that I do need to replace Apache, hopefully you can save me some research time and recommend me one of the listed options that fulfills my needs (or confirm that perlbal does) You may want to try Apache with the event mpm using mod_proxy. I haven't used that, I ended up using Perlbal mostly so I could customize the code without having to deal with intermittent segfaults like I did when I patched apache and got something wrong. I need the following features: 1) provides support for named virtual hosts 2) supports SSL to the client 3) supports URL rewriting (similar to mod_rewrite) 4) knows the availability of pool members and provides high availability. 5) the ability to serve static content itself I think perlbal does all of this. Getting it configured to do all of this may not be straightforward, but the perlbal list is very helpful for that. That being said, I've heard really great things about Varnish, so I'd try that if I didn't have perlbal. I guess 5 isn't strictly neccessary, but it would be nice to serve static content (css/js/images/etc...) from the same piece of technology without proxying those requests to another Apache instance running on the same host (or something) Thanks for all the help! On 4/14/2010 5:48 PM, Perrin Harkins wrote: On Wed, Apr 14, 2010 at 5:33 PM, Brad Van Sicklebvs7...@gmail.com wrote: I didn't find much info on perlbal after a quick glance, I'll certainly give it a closer look, but my inital reaction is that I'm leary of replacing Apache on my web layer. I'm doing a few things with a few other modules ( mod_rewrite for example) in addition to mod_proxy, and from what I was able to find in my initial look, I didn't see any support for some of those types of things. There are many full-featured proxy servers these days. There's even mod_proxy_balancer for apache, but that doesn't do high-availability, which you're looking for. Check out some of these for reverse-proxying if you don't like perlbal: - nginx - lighttpd - varnish - pound All of those can serve as mod_perl frontends. - Perrin
Re: mod_perl in larger scale environments
In data 14 aprile 2010 alle ore 22:57:06, Brad Van Sickle
bvs7...@gmail.com ha scritto:
My first question relates to quality of service and load balancing:
Hi Brad,
we're using LVS (http://en.wikipedia.org/wiki/Linux_Virtual_Server),
and I find it very useful and reliable.
Our infrastructure for the modperl application, simplifying,
consists of:
a) 1 main front lvs load balancer
b) 2 web frontends
c) 1 back lvs load balancer
d) 12 apache/modperl backends
e) 5 db servers
f) ...other stuff... :)
a) load balances incoming traffic between the 2 web frontends.
b) rewrites backend requests to a single address, wlb (web load balancer)
which is handled by c)
c) takes incoming requests for several different virtual hostnames, as
in:
wlb.domain.com = {weighted round robin to} = (back1,back2,...,back12)
mlb.domain.com = {wrr} = (db1, db2, ..., db5) (mysql load balancer)
s-mlb.domain.com = {wrr} = (search-db1, search-db2, ...) (search
mysql lb)
d) app servers are stateless, so we don't need sticky sessions
This architecture can be simplified, and we're trying to do it.
So, I'm not saying this is the best practice or not even sane. :)
LVS performs health checking via HTTP requests,
with or without md5 checksum of the responses,
or direct TCP connections to the port you specify (f.ex. for db servers).
I'm currently using mod_proxy on the web layer, and I know I can set
that up to load balance requests to multiple app layer nodes, but to the
best of my knowledge mod_proxy is not able to provide any quality of
service. So if a node in the app layer had a problem (or was shut down
for maintenance) mod_proxy would be unaware of that and would still send
requests to that node.
That's where LVS is useful.
LVS can do direct routing or tcp handoff IIRC, and we're using it.
The client and servers talk directly to each other,
without taking up too much resources on the LVS machine itself.
I'd rather not use a hardware load balancer here if I can avoid it.
LVS usually runs on our older less powerful machines.
My second question deals with management of multiple mod_perl nodes:
At some point, if you have enough app layer nodes, managing the code
deployments, apache configs and server restarts becomes very cumbersome
if you're doing it all manually.
We're using a simple but limited in-house tool that basically uses
rsync, ssh, and keeps list of hosts w/ roles.
Currently for a pilot project I used puppet for config management
and fabric as last mile deployment tool. So far I'm happy
with the result.
Are there any tools that can make these tasks easier or give me one
management view?
I don't know. Everything we've done is command line based,
so it's not very friendly. Actually I'm currently looking into
higher level tools to integrate what we've done.
I also looked at ControlTier, but it feels too heavy for me.
Needs its own (powerful) machine. I'd be glad to hear experiences on it.
Again, I'm used to working in web applications with a full blown app
server. I love working with mod_perl, but I do find myself missing the
advantages an app server gives me sometimes.
I'm ignorant there. What advantages exactly?
--
Cosimo
Re: mod_perl in larger scale environments
I concur with LVS. I have LVS running on a $10 piece of hardware (300
MHz CPU,
128MB memory) that acts as a load balancer for 15+ web servers. I use
keepalive
to monitor the systems.
Dzuy
Cosimo Streppone wrote:
In data 14 aprile 2010 alle ore 22:57:06, Brad Van Sickle
bvs7...@gmail.com ha scritto:
My first question relates to quality of service and load balancing:
Hi Brad,
we're using LVS (http://en.wikipedia.org/wiki/Linux_Virtual_Server),
and I find it very useful and reliable.
Our infrastructure for the modperl application, simplifying,
consists of:
a) 1 main front lvs load balancer
b) 2 web frontends
c) 1 back lvs load balancer
d) 12 apache/modperl backends
e) 5 db servers
f) ...other stuff... :)
a) load balances incoming traffic between the 2 web frontends.
b) rewrites backend requests to a single address, wlb (web load
balancer)
which is handled by c)
c) takes incoming requests for several different virtual hostnames,
as in:
wlb.domain.com = {weighted round robin to} =
(back1,back2,...,back12)
mlb.domain.com = {wrr} = (db1, db2, ..., db5) (mysql load balancer)
s-mlb.domain.com = {wrr} = (search-db1, search-db2, ...) (search
mysql lb)
d) app servers are stateless, so we don't need sticky sessions
This architecture can be simplified, and we're trying to do it.
So, I'm not saying this is the best practice or not even sane. :)
LVS performs health checking via HTTP requests,
with or without md5 checksum of the responses,
or direct TCP connections to the port you specify (f.ex. for db servers).
I'm currently using mod_proxy on the web layer, and I know I can set
that up to load balance requests to multiple app layer nodes, but to
the best of my knowledge mod_proxy is not able to provide any quality
of service. So if a node in the app layer had a problem (or was shut
down for maintenance) mod_proxy would be unaware of that and would
still send requests to that node.
That's where LVS is useful.
LVS can do direct routing or tcp handoff IIRC, and we're using it.
The client and servers talk directly to each other,
without taking up too much resources on the LVS machine itself.
I'd rather not use a hardware load balancer here if I can avoid it.
LVS usually runs on our older less powerful machines.
My second question deals with management of multiple mod_perl nodes:
At some point, if you have enough app layer nodes, managing the code
deployments, apache configs and server restarts becomes very
cumbersome if you're doing it all manually.
We're using a simple but limited in-house tool that basically uses
rsync, ssh, and keeps list of hosts w/ roles.
Currently for a pilot project I used puppet for config management
and fabric as last mile deployment tool. So far I'm happy
with the result.
Are there any tools that can make these tasks easier or give me one
management view?
I don't know. Everything we've done is command line based,
so it's not very friendly. Actually I'm currently looking into
higher level tools to integrate what we've done.
I also looked at ControlTier, but it feels too heavy for me.
Needs its own (powerful) machine. I'd be glad to hear experiences on it.
Again, I'm used to working in web applications with a full blown app
server. I love working with mod_perl, but I do find myself missing
the advantages an app server gives me sometimes.
I'm ignorant there. What advantages exactly?
Re: mod_perl in larger scale environments
LVS does sound interesting but in your infrastructure layout aren't your
single LVS load balancers single points of failure? Especially if they
are running on older hardware? Maybe that isn't important in your
environment? However, it seems like that negates a lot of the high
availability goal of load balancing.
It still may be a possibility for me, possibly running on the same
host as my existing web layer apache instance and using a localhost
connection... I will definitely look into it.
Again, I'm used to working in web applications with a full blown app
server. I love working with mod_perl, but I do find myself missing the
advantages an app server gives me sometimes.
I'm ignorant there. What advantages exactly?
I'l be brief here because this is a mod_perl list :)
The specific product I'm used to working with is IBM Websphere, which
allows you to cluster your individual app servers and then manage them
all from one administration tool. So settting or config changes, code
deployments, etc... are snynced across all nodes. It makes managing app
clusters extremely easy.It also provides a plugin to IBM's http
server that handles proxying back to the application servers and
provides load balancing/high availability,
Those are the two advantages that address my original questions
directly. App servers provide a lot of other benefits such as allowing
you to leverage things like shared memory and shared DB and messaging
connections/buses... many of these can be simulated in mod_perl.
(Apache::DBI, etc...)
On 4/14/2010 6:27 PM, Cosimo Streppone wrote:
In data 14 aprile 2010 alle ore 22:57:06, Brad Van Sickle
bvs7...@gmail.com ha scritto:
My first question relates to quality of service and load balancing:
Hi Brad,
we're using LVS (http://en.wikipedia.org/wiki/Linux_Virtual_Server),
and I find it very useful and reliable.
Our infrastructure for the modperl application, simplifying,
consists of:
a) 1 main front lvs load balancer
b) 2 web frontends
c) 1 back lvs load balancer
d) 12 apache/modperl backends
e) 5 db servers
f) ...other stuff... :)
a) load balances incoming traffic between the 2 web frontends.
b) rewrites backend requests to a single address, wlb (web load
balancer)
which is handled by c)
c) takes incoming requests for several different virtual hostnames,
as in:
wlb.domain.com = {weighted round robin to} =
(back1,back2,...,back12)
mlb.domain.com = {wrr} = (db1, db2, ..., db5) (mysql load balancer)
s-mlb.domain.com = {wrr} = (search-db1, search-db2, ...) (search
mysql lb)
d) app servers are stateless, so we don't need sticky sessions
This architecture can be simplified, and we're trying to do it.
So, I'm not saying this is the best practice or not even sane. :)
LVS performs health checking via HTTP requests,
with or without md5 checksum of the responses,
or direct TCP connections to the port you specify (f.ex. for db servers).
I'm currently using mod_proxy on the web layer, and I know I can set
that up to load balance requests to multiple app layer nodes, but to
the best of my knowledge mod_proxy is not able to provide any quality
of service. So if a node in the app layer had a problem (or was shut
down for maintenance) mod_proxy would be unaware of that and would
still send requests to that node.
That's where LVS is useful.
LVS can do direct routing or tcp handoff IIRC, and we're using it.
The client and servers talk directly to each other,
without taking up too much resources on the LVS machine itself.
I'd rather not use a hardware load balancer here if I can avoid it.
LVS usually runs on our older less powerful machines.
My second question deals with management of multiple mod_perl nodes:
At some point, if you have enough app layer nodes, managing the code
deployments, apache configs and server restarts becomes very
cumbersome if you're doing it all manually.
We're using a simple but limited in-house tool that basically uses
rsync, ssh, and keeps list of hosts w/ roles.
Currently for a pilot project I used puppet for config management
and fabric as last mile deployment tool. So far I'm happy
with the result.
Are there any tools that can make these tasks easier or give me one
management view?
I don't know. Everything we've done is command line based,
so it's not very friendly. Actually I'm currently looking into
higher level tools to integrate what we've done.
I also looked at ControlTier, but it feels too heavy for me.
Needs its own (powerful) machine. I'd be glad to hear experiences on it.
Again, I'm used to working in web applications with a full blown app
server. I love working with mod_perl, but I do find myself missing
the advantages an app server gives me sometimes.
I'm ignorant there. What advantages exactly?
