Re: [users@httpd] Apache Randomly Dropping POST Data
On Wed, Feb 23, 2011 at 1:18 PM, Ursa Polaris polaris.u...@gmail.com wrote: So I have had my Apache 2.2 server randomly dropping POST data (all or nothing, it doesn't just drop bits or pieces) Doesn't IE sometimes forget to send the POST data when it re-connects after an error? - The official User-To-User support forum of the Apache HTTP Server Project. See URL:http://httpd.apache.org/userslist.html for more info. To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org from the digest: users-digest-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
Re: [users@httpd] Apache Randomly Dropping POST Data
On Wed, Feb 23, 2011 at 1:51 PM, Eric Covener cove...@gmail.com wrote: On Wed, Feb 23, 2011 at 1:18 PM, Ursa Polaris polaris.u...@gmail.com wrote: So I have had my Apache 2.2 server randomly dropping POST data (all or nothing, it doesn't just drop bits or pieces) Doesn't IE sometimes forget to send the POST data when it re-connects after an error? yes a few things to look for to try to matc to IE-missing-post-body: 1. client is IE (from user-agent) 2. request fails after Timeout seconds 3. the first request on the connection (log via %k) - The official User-To-User support forum of the Apache HTTP Server Project. See URL:http://httpd.apache.org/userslist.html for more info. To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org from the digest: users-digest-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
Re: [users@httpd] Apache Randomly Dropping POST Data
I guess I forgot to mention that we have verified using WireShark that Chrome, Firefox and IE are all correctly sending the POST data over the network in these cases. It's not a browser issue. On Wed, Feb 23, 2011 at 11:30 AM, Jeff Trawick traw...@gmail.com wrote: On Wed, Feb 23, 2011 at 1:51 PM, Eric Covener cove...@gmail.com wrote: On Wed, Feb 23, 2011 at 1:18 PM, Ursa Polaris polaris.u...@gmail.com wrote: So I have had my Apache 2.2 server randomly dropping POST data (all or nothing, it doesn't just drop bits or pieces) Doesn't IE sometimes forget to send the POST data when it re-connects after an error? yes a few things to look for to try to matc to IE-missing-post-body: 1. client is IE (from user-agent) 2. request fails after Timeout seconds 3. the first request on the connection (log via %k) - The official User-To-User support forum of the Apache HTTP Server Project. See URL:http://httpd.apache.org/userslist.html for more info. To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org from the digest: users-digest-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org - The official User-To-User support forum of the Apache HTTP Server Project. See URL:http://httpd.apache.org/userslist.html for more info. To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org from the digest: users-digest-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
Re: [users@httpd] Apache Randomly Dropping POST Data
On Wed, Feb 23, 2011 at 4:01 PM, Ursa Polaris polaris.u...@gmail.com wrote: I guess I forgot to mention that we have verified using WireShark that Chrome, Firefox and IE are all correctly sending the POST data over the network in these cases. It's not a browser issue. that's great info (note that you have to be a little bit careful when looking at the packet trace; you may see a TCP connection with the entire request header and POST body sent yet it receives no response; then the browser opens a new TCP connection, sends the request header without body, and after Timeout seconds gets an error response) unless you have any single sign-on modules loaded (always a good target to blame) or other third-party modules which can process the request body, no other ideas here; possibly the timing of the body being sent vs. error being triggered would narrow down the possibilities - The official User-To-User support forum of the Apache HTTP Server Project. See URL:http://httpd.apache.org/userslist.html for more info. To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org from the digest: users-digest-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
Re: [users@httpd] Apache Randomly Dropping POST Data
We have tested using a remote browser with a user sitting at a browser to repro the issue (hours of hitting the enter key, holy cow), and we have also used scripts that make LWP POST requests on the same netblock, as well as running the LWP POST from localhost - all versions saw about a 1% occurrence of POST data being dropped at the Perl/CGI runtime (except for the user testing with browser, which took about 40-50 thousand reloads to get a repro, no joke). It's obvious that Apache is consuming the POST data at some point prior to getting to the Perl/CGI level, but I'm a Perl-head and not an Apache-head, so I don't know how to debug Apache to find a trace of where it's being consumed and removed prior to getting to Perl/CGI. Our virtualhost configuration is very minimal and simple for a basic web apps server with some static/SSI content. There's a cgi-bin and a handful of mod_perl2 handlers, but the scripts having the issues are the ones in the cgi-bin path and just use plain Perl with CGI GET/POST parameter parsing. Here's a list of the modules we have enabled on the server, all of them are default config (from Ubuntu's stable packages), nothing custom: /etc/apache2/mods-enabled/alias.load /etc/apache2/mods-enabled/auth_basic.load /etc/apache2/mods-enabled/authn_file.load /etc/apache2/mods-enabled/authz_default.load /etc/apache2/mods-enabled/authz_groupfile.load /etc/apache2/mods-enabled/authz_host.load /etc/apache2/mods-enabled/authz_user.load /etc/apache2/mods-enabled/autoindex.load /etc/apache2/mods-enabled/cgi.load /etc/apache2/mods-enabled/dir.load /etc/apache2/mods-enabled/env.load /etc/apache2/mods-enabled/include.load /etc/apache2/mods-enabled/mime.load /etc/apache2/mods-enabled/mod-wsgi.load /etc/apache2/mods-enabled/negotiation.load /etc/apache2/mods-enabled/perl.load /etc/apache2/mods-enabled/php5.load /etc/apache2/mods-enabled/rewrite.load /etc/apache2/mods-enabled/setenvif.load /etc/apache2/mods-enabled/status.load Hopefully that helps give more details and insight as well. I'm totally at a loss with this issue, but it's really bothering myself and my users something fierce. =/ I'm hoping not to be forced to move to nginx, but the lack of support from Apache folks is discouraging (not to mention the AWFUL documentation for mod_perl2)... On Wed, Feb 23, 2011 at 1:46 PM, Jeff Trawick traw...@gmail.com wrote: On Wed, Feb 23, 2011 at 4:01 PM, Ursa Polaris polaris.u...@gmail.com wrote: I guess I forgot to mention that we have verified using WireShark that Chrome, Firefox and IE are all correctly sending the POST data over the network in these cases. It's not a browser issue. that's great info (note that you have to be a little bit careful when looking at the packet trace; you may see a TCP connection with the entire request header and POST body sent yet it receives no response; then the browser opens a new TCP connection, sends the request header without body, and after Timeout seconds gets an error response) unless you have any single sign-on modules loaded (always a good target to blame) or other third-party modules which can process the request body, no other ideas here; possibly the timing of the body being sent vs. error being triggered would narrow down the possibilities - The official User-To-User support forum of the Apache HTTP Server Project. See URL:http://httpd.apache.org/userslist.html for more info. To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org from the digest: users-digest-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org - The official User-To-User support forum of the Apache HTTP Server Project. See URL:http://httpd.apache.org/userslist.html for more info. To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org from the digest: users-digest-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
Re: [users@httpd] Apache Randomly Dropping POST Data
On Wed, Feb 23, 2011 at 5:23 PM, Ursa Polaris polaris.u...@gmail.com wrote: Hopefully that helps give more details and insight as well. I'm totally at a loss with this issue, but it's really bothering myself and my users something fierce. =/ I'm hoping not to be forced to move to nginx, but the lack of support from Apache folks is discouraging (not to mention the AWFUL documentation for mod_perl2)... Don't take this the wrong way, but within ~4 hours you've had responses from two different Apache devs representing many years of experience supporting customers on this technology. That's not so shabby. Note that this is potentially as time consuming for others as it has been to you already unless you happen to catch the attention of someone who has solved this type of issue before and there's a match with their solution and your observations. Changing some/all of the technology will presumably help, if you're willing to risk other problems that have to be solved. Maybe that is cheap to experiment with. It may be possible not to move totally to nginx to eliminate some parts of the technology (see below). So real CGIs are the ones with the problem. If you're using a threaded MPM, are you using mod_cgid? (you should be) mod_dumpio can log the request body/POST data at some intermediate point between the network and the CGI. strace of CGI children (if practical to do that) would give another data point. You could temporarily re-implement the most crucial part of the CGI in C just to get Perl/CPAN out of the picture -- doesn't have to perform the real function; just has to confirm that the POST data got to the CGI process. (I doubt Perl/CPAN is the problem, but then every individual part looks unlikely as well.) Good luck! - The official User-To-User support forum of the Apache HTTP Server Project. See URL:http://httpd.apache.org/userslist.html for more info. To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org from the digest: users-digest-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
Re: [users@httpd] Apache Randomly Dropping POST Data
This is the first I've heard of mod_cgid, so no we're not using it. Apparently it IS in our mods-available folder and ready to use though. I'm using Prefork mode which is the default for Ubuntu's stable Apache package. Is that a threaded MPM that should make use of mod_cgid? Also, do you think it could be beneficial for us to build our own Apache from source rather than relying on what Ubuntu's placed in it's stable package library? I notice that it's often several minor versions behind, not sure if that could be an issue here though since this issue has been going on for a long time now. =/ On Wed, Feb 23, 2011 at 2:43 PM, Jeff Trawick traw...@gmail.com wrote: On Wed, Feb 23, 2011 at 5:23 PM, Ursa Polaris polaris.u...@gmail.com wrote: Hopefully that helps give more details and insight as well. I'm totally at a loss with this issue, but it's really bothering myself and my users something fierce. =/ I'm hoping not to be forced to move to nginx, but the lack of support from Apache folks is discouraging (not to mention the AWFUL documentation for mod_perl2)... Don't take this the wrong way, but within ~4 hours you've had responses from two different Apache devs representing many years of experience supporting customers on this technology. That's not so shabby. Note that this is potentially as time consuming for others as it has been to you already unless you happen to catch the attention of someone who has solved this type of issue before and there's a match with their solution and your observations. Changing some/all of the technology will presumably help, if you're willing to risk other problems that have to be solved. Maybe that is cheap to experiment with. It may be possible not to move totally to nginx to eliminate some parts of the technology (see below). So real CGIs are the ones with the problem. If you're using a threaded MPM, are you using mod_cgid? (you should be) mod_dumpio can log the request body/POST data at some intermediate point between the network and the CGI. strace of CGI children (if practical to do that) would give another data point. You could temporarily re-implement the most crucial part of the CGI in C just to get Perl/CPAN out of the picture -- doesn't have to perform the real function; just has to confirm that the POST data got to the CGI process. (I doubt Perl/CPAN is the problem, but then every individual part looks unlikely as well.) Good luck! - The official User-To-User support forum of the Apache HTTP Server Project. See URL:http://httpd.apache.org/userslist.html for more info. To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org from the digest: users-digest-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org - The official User-To-User support forum of the Apache HTTP Server Project. See URL:http://httpd.apache.org/userslist.html for more info. To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org from the digest: users-digest-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
Re: [users@httpd] Apache Randomly Dropping POST Data
On Wed, Feb 23, 2011 at 6:53 PM, Ursa Polaris polaris.u...@gmail.com wrote: This is the first I've heard of mod_cgid, so no we're not using it. Apparently it IS in our mods-available folder and ready to use though. I'm using Prefork mode which is the default for Ubuntu's stable Apache package. Is that a threaded MPM that should make use of mod_cgid? Since you're using prefork, mod_cgi is fine. Forget about mod_cgid. Also, do you think it could be beneficial for us to build our own Apache from source rather than relying on what Ubuntu's placed in it's stable package library? I notice that it's often several minor versions behind, not sure if that could be an issue here though since this issue has been going on for a long time now. =/ I doubt that 2.2.latest would have a fix to solve this, but it is one of those things which isn't too hard to do and could at the very least rule out one possibility. On Wed, Feb 23, 2011 at 2:43 PM, Jeff Trawick traw...@gmail.com wrote: On Wed, Feb 23, 2011 at 5:23 PM, Ursa Polaris polaris.u...@gmail.com wrote: Hopefully that helps give more details and insight as well. I'm totally at a loss with this issue, but it's really bothering myself and my users something fierce. =/ I'm hoping not to be forced to move to nginx, but the lack of support from Apache folks is discouraging (not to mention the AWFUL documentation for mod_perl2)... Don't take this the wrong way, but within ~4 hours you've had responses from two different Apache devs representing many years of experience supporting customers on this technology. That's not so shabby. Note that this is potentially as time consuming for others as it has been to you already unless you happen to catch the attention of someone who has solved this type of issue before and there's a match with their solution and your observations. Changing some/all of the technology will presumably help, if you're willing to risk other problems that have to be solved. Maybe that is cheap to experiment with. It may be possible not to move totally to nginx to eliminate some parts of the technology (see below). So real CGIs are the ones with the problem. If you're using a threaded MPM, are you using mod_cgid? (you should be) mod_dumpio can log the request body/POST data at some intermediate point between the network and the CGI. strace of CGI children (if practical to do that) would give another data point. You could temporarily re-implement the most crucial part of the CGI in C just to get Perl/CPAN out of the picture -- doesn't have to perform the real function; just has to confirm that the POST data got to the CGI process. (I doubt Perl/CPAN is the problem, but then every individual part looks unlikely as well.) Good luck! - The official User-To-User support forum of the Apache HTTP Server Project. See URL:http://httpd.apache.org/userslist.html for more info. To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org from the digest: users-digest-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org -- Born in Roswell... married an alien... - The official User-To-User support forum of the Apache HTTP Server Project. See URL:http://httpd.apache.org/userslist.html for more info. To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org from the digest: users-digest-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
