Re: OT: Bezeqint made me "poof... he's gone"
On 09.06.2009 Shachar Shemesh wrote: > Oron Peled wrote: > > ... > > Few days ago I accidentally discovered that my hosted homepage wasn't > > accessible -- further tests + ~1 hour on the phone (navigating through > > Bezeqint support structure) revealed the unbelievable > > > > THE FREAKING BASTARDS PULLED THE PLUG ON THE DOMAINS WITHOUT EVEN TELLING > > ANYBODY. > > > > I'm now in damage control mode (formal faxes to customer support, etc.) > > Anybody else? > > > Are you sure your email still works? So far... ;-) Other interesting facts: Someone on the list mentioned that users.actcom.co.il/~oron is still there. I checked and it's and amazingly still there. But they left it as an isolated island: - The host users.actcom.co.il is not accessible (they probably just redirected some urls) - Who is allowed to crawl it? Nobody. $ wget -qO - http://users.actcom.co.il/robots.txt User-agent: * Disallow: / - The www.actcom.co.il is totally down with no redirection. which means many broken links. To be continued... -- Oron Peled Voice: +972-4-8228492 o...@actcom.co.il http://www.actcom.co.il/~oron "Debugging is at least twice as hard as writing the program in the first place. So if your code is as clever as you can possibly make it, then by definition you're not smart enough to debug it." -- Brian Kernighan ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
CFS
Your suggestion to check nginx incoming vs. outgoing gave me another > idea - I'll try to find whether I can get such stats from the LVS > server, though LVS itself could be dropping connections due to lack of > space to track all of them too (which closes the loop - how can I tell > whether nginx' own server doesn't drop incoming connections which > nginx itself doesn't know about?). > > Thanks, > > --Amos > > > > -- > > Message: 2 > Date: Tue, 9 Jun 2009 21:42:08 +1000 > From: Amos Shapira > Subject: Re: How to count dropped connections > To: Noam Rathaus > Cc: linux-il > Message-ID: > <9c2cca270906090442y5a3aff8ai36f66e56cf2ce...@mail.gmail.com> > Content-Type: text/plain; charset=ISO-8859-1 > > 2009/6/9 Noam Rathaus : >> Amos, >> >> What are you trying to count? I hope I understood you correctly, you want to >> know how many HTTP requests are being handled, against those that couldn't >> be handled due to lack of connections. > > Yes. "How many connections from customers have reached our servers but > failed to complete the TCP hand shake and send a request?". > >> >> netstat is a very bad counting devices, unless you are counting packets. > > I know. I try to use it as a tool to find counters which might exist > in the kernel. For instance - it can't tell me which port or IP > address the connections failed on. > >> >> If you want to count "requests" I would count incoming connection requests >> (SYN) vs apache log of requests >> >> The incoming connections should be counted using tcpdump or similar > > I just read during my googl'ing that tcpdump is not reliable - it > could report packets more than once, e.g. packets which haven't been > sent or count packets more than once. Also it slows down the network > for time-stamping. > > Maybe a clever iptables rule can count incoming SYN packets on the > relevant ports (we listen on about 4-5 different ports) and then I can > compare it against Apache access log for same period. > >> >> while apache log should be easily achievable by grep > > If the TCP-level connection is dropped before an HTTP request is > received then I'm not sure Apache's log will show it (just tried this > on a Ubuntu desktop, don't know how much it indicates for CentOS 5). > > Thanks, > > --Amos > > > > -- > > Message: 3 > Date: Tue, 09 Jun 2009 15:13:43 +0300 > From: Shachar Shemesh > Subject: Re: How to count dropped connections > To: Amos Shapira > Cc: linux-il > Message-ID: <4a2e51f7.4040...@shemesh.biz> > Content-Type: text/plain; charset="iso-8859-1"; Format="flowed" > > Amos Shapira wrote: >> >> Maybe a clever iptables rule can count incoming SYN packets on the >> relevant ports (we listen on about 4-5 different ports) and then I can >> compare it against Apache access log for same period. >> > No need for anything special. Just do "iptables -L -v" to see how many > hits on each rule. iptables even has command option that give you the > stats and atomically zero the counters. All you need in addition is > grep, and you're almost set. >> >>> while apache log should be easily achievable by grep >>> >> >> If the TCP-level connection is dropped before an HTTP request is >> received then I'm not sure Apache's log will show it (just tried this >> on a Ubuntu desktop, don't know how much it indicates for CentOS 5). >> > Do you count that as a successful connection? It sounds to me like it is > not, which means that apache not listing it is actually a good thing. > > What I would be worried about (not very, mind you) is SYN floods and > other stuff. Some failed TCP connections should not be counted (SYN is > invalid, three way handshake did not complete due to client > considerations, retransmitted SYNs etc.). The only way I can think of to > find those is a sniffer (I don't know of any tcpdump rules that can > match those, and I wouldn't trust its performance anyway, so I think a > dedicated one would work best). > > Shachar > > -- > Shachar Shemesh > Lingnu Open Source Consulting Ltd. > http://www.lingnu.com > > -- next part -- > An HTML attachment was scrubbed... > URL: > <http://mailman.cs.huji.ac.il/pipermail/linux-il/attachments/20090609/b651a960/attachment-0001.html> > > -- > > Message: 4 > Date: Tue, 9 Jun 2009 15:22:58 +0300 > From: shimi > Subject:
Re: [Off topic] Engraving Hebrew on US laptop keyboard
On Tue, 2 Jun 2009 14:31:45 +0300 Dotan Cohen wrote: > > 1. Engraving Hebrew on keyboard > > I actually just painted the keyboard letter keys black, so there are > no English or Hebrew letters! Now my laptop is less usable to thieves, > and my typing speed has increased dramatically. > > > 2. Service/warranty > > > > Dell told me that I would have no warranty on my Inspiron, and in fact > I did not try to get the laptop serviced at all despite having some > problems. I think that IBM (now Lenovo) had a worldwide warranty > program once, designed for business travelers. > On T and X models you can expand you standard 1 year warranty to an international 3 year one. Some places already sell it with the international warranty, they are not always aware of it though and where it is honored depends on the specific model (you can check on the lenovo site based on model number and serial number). Didn't try to service a model supposedly not covered in Israel to see if they still honor it though. By the way, if you are buying a thinkpad in the states and have a about 3-5 weeks advance notice, it can be a lot cheaper ordering directly from lenovo as you can customize your laptop quite a bit and cut costs on things that you don't want. ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: [Off topic] Engraving Hebrew on US laptop keyboard
On Tue, 2 Jun 2009 05:44:33 +0300 Alexander Indenbaum wrote: > Hello, > > This is off topic, still probably crowd here has relevant experience, so ... > > I'm considering bringing Sony laptop from US. My main concerns are: > 1. Engraving Hebrew on keyboard Asked at some stores at the time. Around 140nis. didn't do it as I blind type hebrew and not so much at that. Annoys my wife though, but so does the linux on it ... > 2. Service/warranty > Horrible, terrible, horrendous. Unless things changed any, I brought a Sony from the states a few years back, the cpu burned (bad cooling or something). I took it to ישפאר, don't remembered how much they charged for checking the problem, I do remember that their prices were higher than anyone else though, but they didn't know how to repair it and told me that I have to send it to the states. In any case at least at the time they weren't official importers according to them and Sony and thus didn't honor international warranty, only theirs. I ended up sending the laptop to the states. The support was no use at all, officials bounced me around and didn't bother returning promised replies. They ended up charging 250$ to fix it under warranty + 140$ ups + vat. > Does anyone has a pointer or recommendation about (1). How could > provide such service in Israel? > Does anyone has an experience servicing laptops purchased in US in > Israel (probably by ישפאר)? Like I said, hoping it's a model ישפאר knows they will charge you an arm and a leg for it and won't honor international warranty. Otherwise they will tell you to send it back to the states. I don't know if mac camera cover sony, I used them for the thinkpad I brought a year or so ago (from B&H at the time, who don't use them any more I'm afraid). Tried using them on some problem with the thinkpad but it turns out that it has an international warranty (they claimed it doesn't) and that IBM Israel honors the international warranty, took them a week but they repaired both problems with no fuss. Current thinkpads are not as good as they used to be, but if you want warranty and reliability then they are a much better option then Sony. Go only for the T or X series though. If I'm not mistaken apples are also good for that purpose (I can ask again but a if I'm not mistaken a friend of mine got some spares for his power supply under the international warranty with no issues) Not sure if I would bother with anthing other then macs and thinkpads these days (again, note that thinkpads, not other lenvos, specifically only t and x models) > > ~baum > > ___ > Linux-il mailing list > Linux-il@cs.huji.ac.il > http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: A question about CFS
Shachar Shemesh writes: > Yes. One of the things said about CFS was that with it things that > previously required real time priority now can run at nice -20 just > as well. That same text (I don't remember right now where I read > it, nor who wrote it) said that CFS fixed a nice anomality that used > to exist with O(1). I am not sure what this refers to, so no comment. > Apparently, two adjacent nice levels are supposed to have a more or > less fixed CPU spread between them, no matter what the absolute nice > level is (the article suggested that if the system has just two > processes, one at nice level X and another at nice X+1, then the CPU > should be divided 45-55 between them). OK, first, we know that niceness and priority are one and the same thing, niceness being just priority relative to default priority (120). So, how is CPU time divided between processes of different priority/niceness (with SCHED_NORMAL=SCHED_OTHER=SCHED_FAIR policy, not real-time)? In O(1) the size of a time slice given to a process scaled with priority. This seems to be the effect you are describing. See below. > It was suggested that O(1) had non-linear nice graph, due to the > tricks played to make nice -20 approximate what it should do. I recall that there were multiple reasons why this scaling should be non-linear. One reason was precisely what you point to: a fixed ratio between CPU utilization between processes at adjacent nice levels. In O(1), the timeslice values were fixed, leading to a non-constant (or, rather, priority-dependent) CPU utilization ratios. In other words, the utilization ratio between a process at priority P and a process at priority P+1 depended on P. This was considered a weakness. I actually think that there was more controversy about nice +19 than about nice -20: what should the minimal timeslice be? how much CPU should a process at nice +19 get? was it too much? does it get a different *percentage* of CPU time at different CPU speeds and loads? There were similar/mirror complaints about nice -20 - were those processes given enough CPU? CFS decouples priorities from the clock and gets rid of timeslices. This makes it possible to 1) give nice +19 processes a fixed percentage of CPU time; 2) make the CPU time split between P and P+1 independent of P (I think this is where 55%/45% comes from); 3) processes with negative niceness get enough CPU. All of the above works because everything is relative now, independently of HZ, timeslices, etc. Hope it helps, -- Oleg Goldshmidt | p...@goldshmidt.org ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: A question about CFS
Oleg Goldshmidt wrote: Yes, as in O(1). The implementation of the runqueue has changed in CFS. You are hinting that between the run queues nothing has changed. That seem unlikely to me. If so, how do the relative priorities happen? I am not sure what you mean by "relative priorities". Do you mean, "niceness"? Yes. One of the things said about CFS was that with it things that previously required real time priority now can run at nice -20 just as well. That same text (I don't remember right now where I read it, nor who wrote it) said that CFS fixed a nice anomality that used to exist with O(1). Apparently, two adjacent nice levels are supposed to have a more or less fixed CPU spread between them, no matter what the absolute nice level is (the article suggested that if the system has just two processes, one at nice level X and another at nice X+1, then the CPU should be divided 45-55 between them). It was suggested that O(1) had non-linear nice graph, due to the tricks played to make nice -20 approximate what it should do. Only ready or running processes are scheduled - others don't even want the CPU. Being as it is that the load average on my system rarely goes above 4, I can see why nobody is worried about O(log n). Thanks, Shachar -- Shachar Shemesh Lingnu Open Source Consulting Ltd. http://www.lingnu.com ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: A question about CFS
Shachar Shemesh writes: > Hi all, > I'm trying to understand Linux's Completely Fair Scheduler better Hi, Disclaimer: i've never had a chance to look at the CFS nearly as closely as the previous incarnation ["O(1)"]. I'll take a shot, though. >1. How do the different priorities factor in? My understanding is that CFS is within a single priority level. > Obviously, the real time priorities do not (their scheduling > methods are well defined), but how about the nice values? The > "fair clock" is a property of "rq" (run queue) > - does that mean that each nice value has its own queue? Yes, as in O(1). The implementation of the runqueue has changed in CFS. > If so, how do the relative priorities happen? I am not sure what you mean by "relative priorities". Do you mean, "niceness"? If so, there is a simple mapping of niceness to priority in kernel/sched.c (unchanged from O(1)): /* * Convert user-nice values [ -20 ... 0 ... 19 ] * to static priority [ MAX_RT_PRIO..MAX_PRIO-1 ], * and back. */ #define NICE_TO_PRIO(nice) (MAX_RT_PRIO + (nice) + 20) #define PRIO_TO_NICE(prio) ((prio) - MAX_RT_PRIO - 20) #define TASK_NICE(p)PRIO_TO_NICE((p)->static_prio) >2. Are those "all processes in the system", or just "ready >processes in the system"? Only ready or running processes are scheduled - others don't even want the CPU. > Wikipedia says that CFS treats wait time due to "sorry, no CPU > for you" and wait time due to "I don't need the CPU right now" > the same, so this suggests all processes. I suspect you are a bit confused here, though it sounds like you've made the right guess below. Accounting for sleep time when deciding whom to give the CPU the soonest is related to recognizing interactive processes: they typically sleep a lot waiting for user input and such. In the O(1) scheduler such processes are allocated additional time slices in the current epoch, in CFS their "fair share" of the CPU is increased accordingly. >3. The articles say that CFS gives extra priority for >interactive processes, but does not mention how. Is this just >a by product of the wall time - wait time calculation (makes >sense), or is there any additional tweaking going on that >does that? See above. -- Oleg Goldshmidt | p...@goldshmidt.org ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: OT: Bezeqint made me "poof... he's gone"
looks like www.actcom.co.il doesn't respond to requests any longer. users.actcom.co.il does respond to requests. it looks like this breaks my site as well, since i've used www.actcom.co.il in the links between my pages :0 i guess i'll modify the site for a new URL and upload a new version over the weekend. in fact, i think i would better buy my own domain finally and move the site elsewhere i've postponed this for more then a decade ;) --guy Oron Peled wrote: Good day (cross-posted, check when replying). A a previous customer of Actcom I continued with Bezeqint under the same terms (including a contract renewal ~1 year ago). Few days ago I accidentally discovered that my hosted homepage wasn't accessible -- further tests + ~1 hour on the phone (navigating through Bezeqint support structure) revealed the unbelievable THE FREAKING BASTARDS PULLED THE PLUG ON THE DOMAINS WITHOUT EVEN TELLING ANYBODY. I'm now in damage control mode (formal faxes to customer support, etc.) Anybody else? ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: How to count dropped connections
On Tuesday 09 June 2009 15:13:43 Shachar Shemesh wrote: > > If the TCP-level connection is dropped before an HTTP request is > > received then I'm not sure Apache's log will show it (just tried this > > on a Ubuntu desktop, don't know how much it indicates for CentOS 5). > > > > Do you count that as a successful connection? It sounds to me like it is > not, which means that apache not listing it is actually a good thing. > > What I would be worried about (not very, mind you) is SYN floods and > other stuff. Some failed TCP connections should not be counted (SYN is > invalid, three way handshake did not complete due to client > considerations, retransmitted SYNs etc.). The only way I can think of to > find those is a sniffer (I don't know of any tcpdump rules that can > match those, and I wouldn't trust its performance anyway, so I think a > dedicated one would work best). How about using iptables to count the TCP packets containing SYN's and comparing it to the access_log entries? There are a couple of pitfalls here that needs to be addressed (like retransmition of SYN packets), but this could probably be avoided by using parsing script, which would eliminate the duplicates. ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: How to count dropped connections
On Tue, Jun 9, 2009 at 2:40 PM, Amos Shapira wrote > 1. It means that we'll have another application-level proxy where > right now we are very happy with LVS's performance, transparency and > handling of lots of other traffic going on (we also use it for > internal VIP forwarding among the various components of the system). > I.e. we'll need yet another technology to maintain in addition to LVS, > which we are very happy with. Can't help with that, but I will mention that nginx has a load balancing feature as well; > > 2. One of the applications does lots of TCP/IP-level connection > sniffing so it can't be used behind an application-level proxy, it > must have a direct connection to the browser (LVS works for us since > it acts like a bridge - doesn't touch anything inside the packet > except for the destination MAC address). You mean that it connects back to the origin and then run stuff on it? If that's the problem, nginx has an option to forward the originating IP address via an HTTP header, which you can then use in your application. > > > Your suggestion to check nginx incoming vs. outgoing gave me another > idea - I'll try to find whether I can get such stats from the LVS > server, though LVS itself could be dropping connections due to lack of > space to track all of them too (which closes the loop - how can I tell > whether nginx' own server doesn't drop incoming connections which > nginx itself doesn't know about?). > I think that you can't (side of sniffing). However nginx is designed for tens of thousands of simultaneous keep-alive sessions, with a *very* small footprint. This is a very high limit. I think that above it, you'ld need something which is hardware assisted (like F5 BigIP) >From my experience, nginx allowed to reduce the number of running Apache threads (apache+mod_php5 was the setup). I don't know how much of your content can be served directly by nginx (I dumped Apache completely, because nginx can do PHP via FastCGI). How much you'll reduce depends on what volume of the requests is going to your proprietary module :) But all static content handling can be done by nginx. When number of Apache threads go down, your CPU load goes down, and your memory usage goes WAY down. My numbers were 10-20 loadavg with 6GB RAM, down to loadavg of 1 with a few tens of MBs of RAM... my hardware problem became a bandwidth problem... our uplink couldn't sustain what we now been able to push... of course, that's because I was pure-PHP - so your case is different. But really, setting it up is a breeze, and very easy to test. I'll do give you a tip, though. nginx has static buffers set up for everything; And it tests them and returns an HTTP error if a request is larger than the buffers. So if your requests are bigger than very plain access (large cookies, file uploads, etc) - be advised to set up the relevant buffers in nginx.conf with values larger than the default (the default is VERY light on resources...) -- Shimi ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: How to count dropped connections
Amos Shapira wrote: Maybe a clever iptables rule can count incoming SYN packets on the relevant ports (we listen on about 4-5 different ports) and then I can compare it against Apache access log for same period. No need for anything special. Just do "iptables -L -v" to see how many hits on each rule. iptables even has command option that give you the stats and atomically zero the counters. All you need in addition is grep, and you're almost set. while apache log should be easily achievable by grep If the TCP-level connection is dropped before an HTTP request is received then I'm not sure Apache's log will show it (just tried this on a Ubuntu desktop, don't know how much it indicates for CentOS 5). Do you count that as a successful connection? It sounds to me like it is not, which means that apache not listing it is actually a good thing. What I would be worried about (not very, mind you) is SYN floods and other stuff. Some failed TCP connections should not be counted (SYN is invalid, three way handshake did not complete due to client considerations, retransmitted SYNs etc.). The only way I can think of to find those is a sniffer (I don't know of any tcpdump rules that can match those, and I wouldn't trust its performance anyway, so I think a dedicated one would work best). Shachar -- Shachar Shemesh Lingnu Open Source Consulting Ltd. http://www.lingnu.com ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: How to count dropped connections
2009/6/9 Noam Rathaus : > Amos, > > What are you trying to count? I hope I understood you correctly, you want to > know how many HTTP requests are being handled, against those that couldn't > be handled due to lack of connections. Yes. "How many connections from customers have reached our servers but failed to complete the TCP hand shake and send a request?". > > netstat is a very bad counting devices, unless you are counting packets. I know. I try to use it as a tool to find counters which might exist in the kernel. For instance - it can't tell me which port or IP address the connections failed on. > > If you want to count "requests" I would count incoming connection requests > (SYN) vs apache log of requests > > The incoming connections should be counted using tcpdump or similar I just read during my googl'ing that tcpdump is not reliable - it could report packets more than once, e.g. packets which haven't been sent or count packets more than once. Also it slows down the network for time-stamping. Maybe a clever iptables rule can count incoming SYN packets on the relevant ports (we listen on about 4-5 different ports) and then I can compare it against Apache access log for same period. > > while apache log should be easily achievable by grep If the TCP-level connection is dropped before an HTTP request is received then I'm not sure Apache's log will show it (just tried this on a Ubuntu desktop, don't know how much it indicates for CentOS 5). Thanks, --Amos ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: How to count dropped connections
2009/6/9 shimi : > Look, > > Basically, if your HTTP connection handler handles many connections well - > your module (backend) processing would become the bottleneck. That's how it > usually happens. So if you wrap the requests by a frontend proxy (again, I > recommend nginx) - and just put an error log for the relevant vhosts there, > every time nginx cannot pass a request to backend processing, would be > logged. Then you only need to look at the log, and that's it! Thanks. I looked at nginx a while ago and it looks like a great performance booster. We might be able to use it for some of the applications if/when it comes to that. I'll consider it for this problem for most applications. On the other hand: 1. It means that we'll have another application-level proxy where right now we are very happy with LVS's performance, transparency and handling of lots of other traffic going on (we also use it for internal VIP forwarding among the various components of the system). I.e. we'll need yet another technology to maintain in addition to LVS, which we are very happy with. 2. One of the applications does lots of TCP/IP-level connection sniffing so it can't be used behind an application-level proxy, it must have a direct connection to the browser (LVS works for us since it acts like a bridge - doesn't touch anything inside the packet except for the destination MAC address). Your suggestion to check nginx incoming vs. outgoing gave me another idea - I'll try to find whether I can get such stats from the LVS server, though LVS itself could be dropping connections due to lack of space to track all of them too (which closes the loop - how can I tell whether nginx' own server doesn't drop incoming connections which nginx itself doesn't know about?). Thanks, --Amos ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: How to count dropped connections
On Tue, Jun 9, 2009 at 2:21 PM, Amos Shapira wrote: > 2009/6/9 shimi : > > At what layer do you define "dropping a request" ? Not accepting a TCP > > connection (4) ? Failure to complete the request from the reverse proxy > to > > the backend servers (HTTP error) (assuming you have backend servers - the > > network structure is not obvious from your original message)? > > We use Linux Virtual Server (LVS) in DR mode (i.e. packets come in to > a virtual IP through the LVS, outgoing replies are sent directly from > the Real Servers) in load-balancing (i.e. multiple servers accept and > handle requests in parallel). > > Some of the Real Servers are configured as "persistent", i.e. all > requests from same client within a two-minute period will be handled > by the same Real Server, due to functional requirements, others don't > have this requirement. > > > > > What answers the TCP requests to port 80? > > All requests are handled by Apache 2.2 modules written in C++. > > > > > Do you use efficient HTTP handlers already, e.g. Lighttpd or even better, > > nginx? :) > > We looked at it (lighttpd) and back then didn't see a justification to > make the switch (already had in-house knowledge to write Apache > modules vs. another learning period to take with lighttpd, plus > Apache's flexibility back when we didn't know what we'll need was a > factor, e.g. we also had php, perl and CGI code running around at the > beginning). > > As far as we can tell the bulk of the load is inside our own > home-grown module, not inside Apache. > > What I'd like to know is how many failed connections clients receive, > be it "connection refused" or time outs. > Look, Basically, if your HTTP connection handler handles many connections well - your module (backend) processing would become the bottleneck. That's how it usually happens. So if you wrap the requests by a frontend proxy (again, I recommend nginx) - and just put an error log for the relevant vhosts there, every time nginx cannot pass a request to backend processing, would be logged. Then you only need to look at the log, and that's it! -- Shimi ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: How to count dropped connections
Amos Shapira wrote: 2009/6/9 shimi : At what layer do you define "dropping a request" ? Not accepting a TCP connection (4) ? Failure to complete the request from the reverse proxy to the backend servers (HTTP error) (assuming you have backend servers - the network structure is not obvious from your original message)? We use Linux Virtual Server (LVS) in DR mode (i.e. packets come in to a virtual IP through the LVS, outgoing replies are sent directly from the Real Servers) in load-balancing (i.e. multiple servers accept and handle requests in parallel). Some of the Real Servers are configured as "persistent", i.e. all requests from same client within a two-minute period will be handled by the same Real Server, due to functional requirements, others don't have this requirement. What answers the TCP requests to port 80? All requests are handled by Apache 2.2 modules written in C++. Do you use efficient HTTP handlers already, e.g. Lighttpd or even better, nginx? :) We looked at it (lighttpd) and back then didn't see a justification to make the switch (already had in-house knowledge to write Apache modules vs. another learning period to take with lighttpd, plus Apache's flexibility back when we didn't know what we'll need was a factor, e.g. we also had php, perl and CGI code running around at the beginning). As far as we can tell the bulk of the load is inside our own home-grown module, not inside Apache. What I'd like to know is how many failed connections clients receive, be it "connection refused" or time outs. If it's at the TCP level you want to know this, a sniffer is probably your best approach. I'm not aware of any ready made one, but writing a dedicated one (even without libpcap) that is capable of living up to the load you describe should not be too big a project. Contact me if you want a quote :-) Shachar -- Shachar Shemesh Lingnu Open Source Consulting Ltd. http://www.lingnu.com ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: How to count dropped connections
2009/6/9 shimi : > At what layer do you define "dropping a request" ? Not accepting a TCP > connection (4) ? Failure to complete the request from the reverse proxy to > the backend servers (HTTP error) (assuming you have backend servers - the > network structure is not obvious from your original message)? We use Linux Virtual Server (LVS) in DR mode (i.e. packets come in to a virtual IP through the LVS, outgoing replies are sent directly from the Real Servers) in load-balancing (i.e. multiple servers accept and handle requests in parallel). Some of the Real Servers are configured as "persistent", i.e. all requests from same client within a two-minute period will be handled by the same Real Server, due to functional requirements, others don't have this requirement. > > What answers the TCP requests to port 80? All requests are handled by Apache 2.2 modules written in C++. > > Do you use efficient HTTP handlers already, e.g. Lighttpd or even better, > nginx? :) We looked at it (lighttpd) and back then didn't see a justification to make the switch (already had in-house knowledge to write Apache modules vs. another learning period to take with lighttpd, plus Apache's flexibility back when we didn't know what we'll need was a factor, e.g. we also had php, perl and CGI code running around at the beginning). As far as we can tell the bulk of the load is inside our own home-grown module, not inside Apache. What I'd like to know is how many failed connections clients receive, be it "connection refused" or time outs. Hope this answers the questions. Thanks, --Amos ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: How to count dropped connections
Amos, What are you trying to count? I hope I understood you correctly, you want to know how many HTTP requests are being handled, against those that couldn't be handled due to lack of connections. netstat is a very bad counting devices, unless you are counting packets. If you want to count "requests" I would count incoming connection requests (SYN) vs apache log of requests The incoming connections should be counted using tcpdump or similar while apache log should be easily achievable by grep On Tue, Jun 9, 2009 at 1:44 PM, Amos Shapira wrote: > Hello, > > We are running a cluster of web servers of various functions on top of > CentOS 5, and would like to know whether and how many requests we > might be dropping because of insufficient provisioning. The cluster > could be hit by a a few millions requests per day soon and we'd like > the stats to be there both now (in case we miss something obvious) and > when the crowd comes charging. > > Googl'ing around for explanations of "netstat -s" and the files under > /proc/net didn't come up with anything which looks like the definite > answer, except maybe "packets pruned from receive queue because of > socket buffer overrun", but I'm not sure what it counts. > > Does anyone know how to get the stats I want? > > Thanks, > > --Amos > > ___ > Linux-il mailing list > Linux-il@cs.huji.ac.il > http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il > > ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: How to count dropped connections
On Tue, Jun 9, 2009 at 1:44 PM, Amos Shapira wrote: > Hello, > > We are running a cluster of web servers of various functions on top of > CentOS 5, and would like to know whether and how many requests we > might be dropping because of insufficient provisioning. The cluster > could be hit by a a few millions requests per day soon and we'd like > the stats to be there both now (in case we miss something obvious) and > when the crowd comes charging. > > Googl'ing around for explanations of "netstat -s" and the files under > /proc/net didn't come up with anything which looks like the definite > answer, except maybe "packets pruned from receive queue because of > socket buffer overrun", but I'm not sure what it counts. > > Does anyone know how to get the stats I want? > At what layer do you define "dropping a request" ? Not accepting a TCP connection (4) ? Failure to complete the request from the reverse proxy to the backend servers (HTTP error) (assuming you have backend servers - the network structure is not obvious from your original message)? What answers the TCP requests to port 80? Do you use efficient HTTP handlers already, e.g. Lighttpd or even better, nginx? :) -- Shimi ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
How to count dropped connections
Hello, We are running a cluster of web servers of various functions on top of CentOS 5, and would like to know whether and how many requests we might be dropping because of insufficient provisioning. The cluster could be hit by a a few millions requests per day soon and we'd like the stats to be there both now (in case we miss something obvious) and when the crowd comes charging. Googl'ing around for explanations of "netstat -s" and the files under /proc/net didn't come up with anything which looks like the definite answer, except maybe "packets pruned from receive queue because of socket buffer overrun", but I'm not sure what it counts. Does anyone know how to get the stats I want? Thanks, --Amos ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
A question about CFS
Hi all, I'm trying to understand Linux's Completely Fair Scheduler better, here is what I got so far (sources - wikipedia, Gilad's slides on tuxology): Each process is assigned a "wall clock", which is how much CPU wall time it should get if the system is completely fair. The system also tracks how much time it spends waiting. The difference between the two (wall time - waiting time) is the sort key for all the processes in the system (exact details irrelevant for this discussion). The process with the biggest difference (i.e. - greatest waiting time) is the next one to receive the CPU. My questions: 1. How do the different priorities factor in? Obviously, the real time priorities do not (their scheduling methods are well defined), but how about the nice values? The "fair clock" is a property of "rq" (run queue) - does that mean that each nice value has its own queue? If so, how do the relative priorities happen? 2. Are those "all processes in the system", or just "ready processes in the system"? Wikipedia says that CFS treats wait time due to "sorry, no CPU for you" and wait time due to "I don't need the CPU right now" the same, so this suggests all processes. Then again, if this really is all processes, the O(log n) must really hurt. 3. The articles say that CFS gives extra priority for interactive processes, but does not mention how. Is this just a by product of the wall time - wait time calculation (makes sense), or is there any additional tweaking going on that does that? Thanks, Shachar -- Shachar Shemesh Lingnu Open Source Consulting Ltd. http://www.lingnu.com ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
Re: OT: Bezeqint made me "poof... he's gone"
Oron Peled wrote: Good day (cross-posted, check when replying). A a previous customer of Actcom I continued with Bezeqint under the same terms (including a contract renewal ~1 year ago). Few days ago I accidentally discovered that my hosted homepage wasn't accessible -- further tests + ~1 hour on the phone (navigating through Bezeqint support structure) revealed the unbelievable THE FREAKING BASTARDS PULLED THE PLUG ON THE DOMAINS WITHOUT EVEN TELLING ANYBODY. I'm now in damage control mode (formal faxes to customer support, etc.) Anybody else? Are you sure your email still works? Shachar -- Shachar Shemesh Lingnu Open Source Consulting Ltd. http://www.lingnu.com ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
OT: Bezeqint made me "poof... he's gone"
Good day (cross-posted, check when replying). A a previous customer of Actcom I continued with Bezeqint under the same terms (including a contract renewal ~1 year ago). Few days ago I accidentally discovered that my hosted homepage wasn't accessible -- further tests + ~1 hour on the phone (navigating through Bezeqint support structure) revealed the unbelievable THE FREAKING BASTARDS PULLED THE PLUG ON THE DOMAINS WITHOUT EVEN TELLING ANYBODY. I'm now in damage control mode (formal faxes to customer support, etc.) Anybody else? -- Oron Peled Voice: +972-4-8228492 o...@actcom.co.il http://www.actcom.co.il/~oron Promises are like babies: fun to make, but hell to deliver. -- Nadav Har'El ___ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
