Re: number of bytes/packets sent/received
On Thu, Oct 9, 2008 at 15:54, Andrej van der Zee [EMAIL PROTECTED] wrote: Hi, No, the Content-Length doesn't include the size of the headers. Yes of course, but you could add the length of all r-headers_in. It's not very reliable. In headers_out you have things like key and value, while the resulting header is key: value\r\n. Then, I am not sure if the ap_http_header_filter that transforms the apr_table_t *headers_out to strings changes/inserts headers. Then there are two other filters, the byte-range and the content-length filter. The content-length filter sets the value of the Content-Length header while I don't know what the byte-range filter does (maybe it is applicable in the case of chunked transport). You would need to count the size of the headers_out _after_ such filters alter the headers. So... too hairy. I just wondered if the content-type reflects the actual size, for example if the request is compressed. If the request is compressed, Content-Length indicates the length of the _compressed_ body. Content-Type should stay unmodified regardless if the body is compressed or not. However, a compressed body is flagged by the Content-Encoding: gzip header (or deflate). S
Re: number of bytes/packets sent/received
On Thu, Oct 9, 2008 at 15:44, Andrej van der Zee [EMAIL PROTECTED] wrote: Thanks for your reply! Giving it a second thought, the problem looks complicated. Such filter-based counters would count the traffic of _one_ apache process. When you increment the counter, you have to protect it from concurrent access by other threads running in the same apache child process. Then, before the apache child process exits, you need to add the counter to a global counter shared among apache child processes (which has to be protected against race conditions as well). It's really hairy. Don't do it in apache. I forgot to mention, but I need the number of packets / bytes per HTTP request and log it to a database with other information. So I guess that would take case of all the hairy stuff. I think I will give it a go as you recommend with an input and output filter that only counts the bytes and just passes on the data, unless somebody comes with a better idea (what about the content-length for example, can I trust this?). I understood from your email that the headers are counted with it (I never wrote a filter before) and that's exactly what I need. No, the Content-Length doesn't include the size of the headers. While I was typing the message, Dave Ingram's message arrived. I think the solution he proposes is much better. Sorin Sorin
Re: number of bytes/packets sent/received
On Thu, Oct 9, 2008 at 14:40, Andrej van der Zee [EMAIL PROTECTED] wrote: Hi, I was wondering what would be the right way to get the number of bytes / packets that are sent / received in an Apache module for httpd-2.1 and higher (including the HTTP headers). Check server/core.c and server/core_filters.c in the apache sources. There are two filters there, ap_core_input_filter and ap_core_output_filter. They are the first input/last output filters. They are responsible of packaging the byte stream from the socket into bucket brigades and of writing the contents of bucket brigades to the socket. So, the output of ap_core_input_filter is really was comes from the socket, and the input of ap_core_output_filter is really was goes to the socket. So you'll need to write your own filters that count the bytes and insert them immediately after/before ap_core_input/output_filter. This, if you want to count the headers. If not, for output it would be easier to parse the access logs. For input, I don't know. Wouldn't it be easier if you don't do this in apache but with some sort of network monitoring tools? iptables, snmp, or something like that? I'm not an expert, but I suppose some smart tools must be out there, allowing you to filter by source IP, port, etc. Sorin
Re: number of bytes/packets sent/received
On Thu, Oct 9, 2008 at 15:11, Sorin Manolache [EMAIL PROTECTED] wrote: On Thu, Oct 9, 2008 at 14:40, Andrej van der Zee [EMAIL PROTECTED] wrote: Hi, I was wondering what would be the right way to get the number of bytes / packets that are sent / received in an Apache module for httpd-2.1 and higher (including the HTTP headers). Check server/core.c and server/core_filters.c in the apache sources. There are two filters there, ap_core_input_filter and ap_core_output_filter. They are the first input/last output filters. They are responsible of packaging the byte stream from the socket into bucket brigades and of writing the contents of bucket brigades to the socket. So, the output of ap_core_input_filter is really was comes from the socket, and the input of ap_core_output_filter is really was goes to the socket. So you'll need to write your own filters that count the bytes and insert them immediately after/before ap_core_input/output_filter. This, if you want to count the headers. If not, for output it would be easier to parse the access logs. For input, I don't know. Wouldn't it be easier if you don't do this in apache but with some sort of network monitoring tools? iptables, snmp, or something like that? I'm not an expert, but I suppose some smart tools must be out there, allowing you to filter by source IP, port, etc. Sorin Giving it a second thought, the problem looks complicated. Such filter-based counters would count the traffic of _one_ apache process. When you increment the counter, you have to protect it from concurrent access by other threads running in the same apache child process. Then, before the apache child process exits, you need to add the counter to a global counter shared among apache child processes (which has to be protected against race conditions as well). It's really hairy. Don't do it in apache. Sorin
Re: number of bytes/packets sent/received
Thanks for your reply! Giving it a second thought, the problem looks complicated. Such filter-based counters would count the traffic of _one_ apache process. When you increment the counter, you have to protect it from concurrent access by other threads running in the same apache child process. Then, before the apache child process exits, you need to add the counter to a global counter shared among apache child processes (which has to be protected against race conditions as well). It's really hairy. Don't do it in apache. I forgot to mention, but I need the number of packets / bytes per HTTP request and log it to a database with other information. So I guess that would take case of all the hairy stuff. I think I will give it a go as you recommend with an input and output filter that only counts the bytes and just passes on the data, unless somebody comes with a better idea (what about the content-length for example, can I trust this?). I understood from your email that the headers are counted with it (I never wrote a filter before) and that's exactly what I need. Cheers, Andrej
Re: number of bytes/packets sent/received
I forgot to mention, but I need the number of packets / bytes per HTTP request and log it to a database with other information. So I guess that would take case of all the hairy stuff. I think I will give it a go as you recommend with an input and output filter that only counts the bytes and just passes on the data, unless somebody comes with a better idea (what about the content-length for example, can I trust this?). I understood from your email that the headers are counted with it (I never wrote a filter before) and that's exactly what I need. Would mod_logio be at all helpful? http://httpd.apache.org/docs/2.2/mod/mod_logio.html I haven't looked into it, but the source might give you some ideas. You may even have a direct solution if you use a module that logs directly to a database. I must admit that I don't know offhand if such a module exists. Dave
Re: number of bytes/packets sent/received
Hi Dave, Would mod_logio be at all helpful? http://httpd.apache.org/docs/2.2/mod/mod_logio.html I haven't looked into it, but the source might give you some ideas. You may even have a direct solution if you use a module that logs directly to a database. I must admit that I don't know offhand if such a module exists. You really made it easy for me huh? Thanks alot. Cheers, Andrej
Re: number of bytes/packets sent/received
Hi, No, the Content-Length doesn't include the size of the headers. Yes of course, but you could add the length of all r-headers_in. I just wondered if the content-type reflects the actual size, for example if the request is compressed. While I was typing the message, Dave Ingram's message arrived. I think the solution he proposes is much better. Yes, thanks alot for your time anyway. Cheers, Andrej
Re: number of bytes/packets sent/received
Would mod_logio be at all helpful? You really made it easy for me huh? Thanks alot. No use in doing more work than you need to ;-) To be honest, I only found out about it today while researching some obscure logging options. A quick search reveals that there are some modules out there that do log to databases, but I don't know how well-maintained they are (mod_log_sql only mentions Apache 2.0, for example) or how much work they would be to use (an O'Reilly page http://www.onlamp.com/pub/a/apache/2005/02/10/database_logs.html?page=2 suggests that Apache's default mod_log_config needs to be rebuilt). Dave
Re: number of bytes/packets sent/received
Hi, A quick search reveals that there are some modules out there that do log to databases, but I don't know how well-maintained they are (mod_log_sql only mentions Apache 2.0, for example) or how much work they would be to use (an O'Reilly page http://www.onlamp.com/pub/a/apache/2005/02/10/database_logs.html?page=2 suggests that Apache's default mod_log_config needs to be rebuilt). Since Apache 2.1 there is mod_dbd that makes it pretty easy to log to a MySQL, sqlite2/3 and Postgres database. It also takes care of connection pooling. Cheers, Andrej
Re: number of bytes/packets sent/received
A quick search reveals that there are some modules out there that do log to databases, but I don't know how well-maintained they are (mod_log_sql only mentions Apache 2.0, for example) or how much work they would be to use (an O'Reilly page http://www.onlamp.com/pub/a/apache/2005/02/10/database_logs.html?page=2 suggests that Apache's default mod_log_config needs to be rebuilt). Since Apache 2.1 there is mod_dbd that makes it pretty easy to log to a MySQL, sqlite2/3 and Postgres database. It also takes care of connection pooling. Ah. I'm planning to use mod_dbd myself for a project, but I wasn't aware it provided any logging ability. Dave
