requests possibly not reaching the log phase

[John ORourke]

(apologies if this is a dupe, I originally sent from the wrong address)

Hi there,

I have an unusual problem - a large e-commerce site integrated with
Authorize.net for card payments which appears to be failing to log some
requests.

The Authorize.net system makes HTTP POST requests to our server, and
about 1 in every 500 transactions, the Authorize.net system reports a
timeout and there's no trace of the request in our logs.  Authorize.net
won't investigate in any detail because their system is reporting that
the request simply timed out.

I'm using mod_perl but not hooking into the logging phase, and using a
mod_log CustomLog directive which outputs the usual stuff, plus request
time, connection state, and PID.

So for now, I'm assuming a request is being sent but the log handler
phase isn't running.  The only way I can make this happen in a test
environment is by opening a TCP connection and then closing it without
sending any data.  Are there any other reasons the log phase wouldn't be
run?

My next steps are to add simple request logging during the Trans phase,
and failing that, packet sniffing, but this is a live high-traffic
server so I'm trying to avoid that if possible.

cheers
John

Re: requests possibly not reaching the log phase

[Graham Leggett]

John ORourke wrote:

> I have an unusual problem - a large e-commerce site integrated with
> Authorize.net for card payments which appears to be failing to log some
> requests.
> 
> The Authorize.net system makes HTTP POST requests to our server, and
> about 1 in every 500 transactions, the Authorize.net system reports a
> timeout and there's no trace of the request in our logs.  Authorize.net
> won't investigate in any detail because their system is reporting that
> the request simply timed out.

If httpd isn't logging anything, the most likely explanation is that the
request isn't reaching httpd at all.

How reliable is your network between their system and yours? Are there
any load balancing devices or other network magic in the way that could
potentially be misconfigured?

> My next steps are to add simple request logging during the Trans phase,
> and failing that, packet sniffing, but this is a live high-traffic
> server so I'm trying to avoid that if possible.

Packet sniffing will answer the question "was there any evidence of a
request", and would probably be the least invasive way of measuring
this. It's always useful to see what the network is actually doing,
rather than what you think it's doing.

Regards,
Graham
--

Re: requests possibly not reaching the log phase

[John ORourke]

Graham Leggett wrote:

If httpd isn't logging anything, the most likely explanation is that the
request isn't reaching httpd at all.

How reliable is your network between their system and yours? Are there
any load balancing devices or other network magic in the way that could
potentially be misconfigured?
  

That gives me a few new places to hunt down the issue, thanks.


Packet sniffing will answer the question "was there any evidence of a
request", and would probably be the least invasive way of measuring
this. It's always useful to see what the network is actually doing,
rather than what you think it's doing.
  


Definitely - I've decided to bite the bullet and run tshark in the 
background, hopefully that will turn up some new information.  Debugging 
an error which happens at random a few times a month is an interesting 
challenge!


cheers
John

Re: [mod_fcgid] Feedback / Suggestions

[Barry Scott]

Jeff Trawick wrote:

On Tue, Nov 24, 2009 at 10:05 AM, Edgar Frank  wrote:
  


In the interim, is mod_fastcgi really that bad?

  
mod_fastcgi is fine for handling GET/POST requests, but it fails to 
implement

Authorization or Authenication.

So yes mod_fastcgi is really bad.

mod_fcgid is a very welcome as  a supported httpd module.

Barry

Re: requests possibly not reaching the log phase

[Paweł Sypek]

You might also take a look into global apache's error log. There are very
rare cases when PHP (or, probably, other modules) causes segmentation faults
in apache's child processes which prevents any request info to be written to
access log. The last time I had problems with this issue it was caused by
Oracle PHP extension (oci).

2009/12/3 John ORourke 

> That gives me a few new places to hunt down the issue, thanks.
>

Regards,
Paul

Re: requests possibly not reaching the log phase

[Nick Kew]

On 3 Dec 2009, at 08:43, John ORourke wrote:

> So for now, I'm assuming a request is being sent but the log handler
> phase isn't running.

Seems improbable, though mod_perl adds an extra layer.  Graham's
suggestion looks more plausible.

> My next steps are to add simple request logging during the Trans phase,

You can do that by adding a match-and-do-nothing RewriteRule and
setting RewriteLogLevel to log it.  Or the third-party mod_security
will log a great deal more for you.

btw, this should be on the users@ list.

-- 
Nick Kew

Re: requests possibly not reaching the log phase

[Dan Poirier]

John ORourke  writes:

> The Authorize.net system makes HTTP POST requests to our server, and
> about 1 in every 500 transactions, the Authorize.net system reports a
> timeout and there's no trace of the request in our logs.  Authorize.net
> won't investigate in any detail because their system is reporting that
> the request simply timed out.

This could happen in Apache 2 if the web server timed out before reading
the complete request.  (Apache 1.3 logs a 408; Apache 2 doesn't; Apache
trunk will log a 4xx of some sort depending on how much of the request
was read.)

If that's what's happening, it would indicate a problem outside of
Apache.  You could confirm it by backporting the fix from
https://issues.apache.org/bugzilla/show_bug.cgi?id=39785 so logging
would occur.  Then you might need to resort to packet tracing or
something like that to figure out exactly what's going on.

Dan

Re: [mod_fcgid] Feedback / Suggestions

[Eric Covener]

On Thu, Dec 3, 2009 at 5:57 AM, Barry Scott  wrote:
> Jeff Trawick wrote:
>>
>> On Tue, Nov 24, 2009 at 10:05 AM, Edgar Frank  wrote:
>>
>> In the interim, is mod_fastcgi really that bad?
>>
>
> mod_fastcgi is fine for handling GET/POST requests, but it fails to
> implement
> Authorization or Authenication.
>
> So yes mod_fastcgi is really bad.

mod_fastcgi has supported this for many years:

http://www.fastcgi.com/drupal/node/25#FastCgiAuthorizer
http://www.fastcgi.com/drupal/node/25#FastCgiAuthenticator

-- 
Eric Covener
cove...@gmail.com

Re: svn commit: r885606 - /httpd/httpd/trunk/build/rpm/httpd.init

[William A. Rowe Jr.]

Graham Leggett wrote:
> 
> On a Redhat machine (Fedora/RHEL/Centos), search
> /etc/rc.d/init.d/functions for functions called "daemon", "status" and
> "killproc". These functions provide similar but incompatible
> functionality to that provided by apachectl, and only exist on Redhat
> derived machines.

Ok, so they want to roll their own.  Sounds like a maintainer issue.  What
does this say for using our httpd rpm for an Ubuntu or other distribution
of linux?

> The startup script is far too trivial to justify jumping through hoops
> to try and make apachectl work like Redhat's init. It's caused us enough
> grief already, thus the fix.

IMHO, if it is fundamentally incompatible with apachectl and non-redhat
distributions, let the the packagers tweak and take the zany customizations
out from under our problem set.

Re: svn commit: r885606 - /httpd/httpd/trunk/build/rpm/httpd.init

[William A. Rowe Jr.]

Gregg L. Smith wrote:
> Original Message ---
>> Finally, I have yet to see any feedback on the pcre mandatory 
>> dependency issue.  Comments?
> 
> Personally, I thought your Monopoly metaphor was quite on target.
> 
> libz, openssl, lua = batteries not included
> apr, apu, pcre = drive train not included.
> 
>> And what is passing for an excuse for a local PCRE install 
>> these days probably doesn't look like 7.8 or later, with 
>> various fixes we are vulnerable to.
> 
> This does not leave me with a warm and fuzzy feeling. As a user, is the pcre 
> 8.0 I've built going to expose me to risks that your maintained 7.8 does not? 
> If yes, then I'd prefer your maintained one. After all, who knows better than 
> you what will interact with your code to produce problems. Regardless of 
> merit, who will ultimately get blamed in the end? Could your reputation be 
> tarnished? Can you completely divorce yourself from something your software 
> requires to run?

I'm referring to pre v7 chaos.  And mostly not referring to modern
linux distros.

> The 'Jump Ship' factor;
> 
> To me, and I'm probably wrong, it sounds like Mr. Felt's comment was an 
> ultimatum of sorts as 'indefinitely' is a pretty strong word. With this issue 
> you have created a deal with it or jump ship ultimatum which could very well 
> leave some people scrambling to get off. Each person is going to inevitably 
> weigh the pain factor, the pain of dealing with it over the pain of jumping 
> ship. I consider myself lucky that my second attempt to deal with it was 
> successful, or so it seems so far anyway, but I never know from day to day.

Agreed that ease-of-adoption is going to be the usual, first barrier to
anyone jumping aboard 2.4 from 2.2, 2.0, or even still from 1.3.

> I may be wrong but as an outsider looking in, I see you wanting to stop 
> maintaining/including the gear box and are instead spending the time on 
> adding more optional gadgets to choose from (some of the third party modules 
> you've taken over). In the end, I'd prefer having a reverse gear over the 
> rear window defogger. You are also loosing all control of a required piece of 
> equipment, this has got to make some of you at least cringe a little.

I'm not 100% sure I understand what you are saying here.  Drop the
gearbox and let them assemble their own transmission?  Or distribute
a most modern transmission that the user can ignore or swap out if they
want to install their own?

> Sorry for the outburst, but you opened the door for, and I've said what I've 
> wanted to for some time now, thanks for listening. Corrections and daggers 
> welcomed.

No problems, thanks for chiming in.

[RESULTS] [VOTE] Release httpd 2.3.4-alpha

[Paul Querna]

Vote Results:
   +1 (binding): Sander Temme, Paul Querna, Joe Orton,  Niklas Edmundsson,
   +1: Gregg Smith
 +/-0: Rainer Jung
-1: William A. Rowe, Jr.

Vote passes.

I'll push out the tarballs to start getting mirrors, and hack on an
announcement email for tomorrow.

On Wed, Nov 25, 2009 at 2:43 PM, Paul Querna  wrote:
> Test tarballs for Apache httpd 2.3.4-alpha are available at:
>   
>
> Your votes please;
>
>  +/- 1
>  [  ]  Release httpd-2.3.4 as Alpha
>
> Vote closes at 18:00 UTC on Monday November 30 2009.
>
> May your Thanksgiving be filled with Turkey and httpd testing,
>
> Thanks,
>
> Paul
>

Re: [RESULTS] [VOTE] Release httpd 2.3.4-alpha

[William A. Rowe Jr.]

Paul Querna wrote:
> Vote Results:
>+1 (binding): Sander Temme, Paul Querna, Joe Orton,  Niklas Edmundsson,
>+1: Gregg Smith
>  +/-0: Rainer Jung
> -1: William A. Rowe, Jr.
> 
> Vote passes.

I'm sorry.  I explicitly insisted on a vote on the -deps package seperately
from the 2.3.4 package, because it was entirely reasonable that Sander Temme,
Paul Querna, Joe Orton, Niklas Edmundsson, or Gregg Smith reviewed -only- the
httpd-2.3.4-alpha.tar.xx package alone.

Gentlemen, if you could verify that your vote was for *both* the core package
and the -deps.tar.xx package, that might keep Paul on his schedule (or fall
against the proposed -deps.tar.xx package).

Remember your -deps vote is to approve the release of apr 1.4.0-dev and the
apr-util 1.4.0 dev, and the API versioning rules will bind from that release
forwards.

Re: [VOTE] Release httpd 2.3.4-alpha

[William A. Rowe Jr.]

Paul Querna wrote:
> 
> I don't agree that we can't release a bundled unreleased version of
> APR, we did this for many versions of httpd 2.0.x and 2.1.x.  It
> definitely isn't preferred, but that's the APR project's problem.

Look, your argument simply doesn't fly.

In httpd 2.0 timeframe we were only shipping apr-0.9.x - it did NOT
have the same API/ABI constraints (some of them, but not all).  All
of those intermediary releases kept the ABI rules of APR.

Now that you have shipped immediately while ignoring my objection,
I'll treat all +1's as binding no matter if they approved both of
the pieces or not, and have tagged 1.4.0 of both apr and apr-util.

We have no alternative, or else all author's VERSION_MAJOR/MINOR
tests are invalid.

It becomes up to the APR project if this aught to be 1.4.0 or burn
a number and move on.  For 1.4 initial release, I want to pick up
Branko's fix, so I plan to label this 1.4.1.

No intention of tagging apr-util yet till we decide if it can be
API frozen, so if we dislike the current includes/ tree, it will
end up being deprecated interfaces and version 1.5.0 already.

Re: [RESULTS] [VOTE] Release httpd 2.3.4-alpha

[Jeff Trawick]

On Thu, Dec 3, 2009 at 6:21 PM, William A. Rowe Jr.  wrote:

> Remember your -deps vote is to approve the release of apr 1.4.0-dev and the
> apr-util 1.4.0 dev, and the API versioning rules will bind from that release
> forwards.

The APR versioning rules are hopelessly broken if a tarball snapshot
of the 1.4.x branch before a GA release casts the API in stone.

Surely I misunderstood you.

Re: svn commit: r885606 - /httpd/httpd/trunk/build/rpm/httpd.init

[Graham Leggett]

William A. Rowe Jr. wrote:

> Ok, so they want to roll their own.  Sounds like a maintainer issue.  What
> does this say for using our httpd rpm for an Ubuntu or other distribution
> of linux?

Ubuntu is Debian based, and uses the .deb packaging format, and startup
scripts derived from the Debian layout. If someone was to donate debian
packaging for httpd, I would expect one or two files to appear below
build/deb, and that would be all that would be needed.

> IMHO, if it is fundamentally incompatible with apachectl and non-redhat
> distributions, let the the packagers tweak and take the zany customizations
> out from under our problem set.

Apachectl is archaic and largely broken for most people - it made sense
ten years ago, it makes a lot less sense today.

The pattern followed by most modern unix based packaging is for an
application to drop a snippet of config contained in a discrete file in
a directory ending in ".d". So you have
/etc/httpd/conf.d/.conf, instead of a manual edit to
/etc/httpd/conf/httpd.conf, and your httpd startup goes within an
optional script called /etc/sysconfig/httpd instead of in a script file
in a bin directory as apachectl wants. I understand Debian has different
naming conventions, but otherwise the underlying principles are the same.

In our case, we package up config files within standalone RPMs all of
their own (suitably tagged and versioned), or we generate the config
file using puppet. Editing a file in place is always painful and error
prone, it is far less painful to provide a discrete file that can be
dropped in and removed cleanly. Apachectl doesn't give us this - you
need to edit apachectl directly to modify the command line parameters
passed to httpd.

As for the packagers tweaking and making zany customisations, that is
exactly what they do now. For us it makes the their packaging unsuitable
for our needs, simply because we tweak and make zany customisations for
needs of our own. It is far less painful to take a vanilla RPM published
by the ASF, and then tweak it for our needs, than it is to take a Fedora
RPM, untweak all their customisations, and then retweak it with ours.

Regards,
Graham
--

Re: svn commit: r885606 - /httpd/httpd/trunk/build/rpm/httpd.init

[Gregg L. Smith]

William A. Rowe Jr. wrote:

I may be wrong but as an outsider looking in, I see you wanting to stop 
maintaining/including the gear box and are instead spending the time on adding 
more optional gadgets to choose from (some of the third party modules you've 
taken over). In the end, I'd prefer having a reverse gear over the rear window 
defogger. You are also loosing all control of a required piece of equipment, 
this has got to make some of you at least cringe a little.


I'm not 100% sure I understand what you are saying here.  Drop the
gearbox and let them assemble their own transmission?  Or distribute
a most modern transmission that the user can ignore or swap out if they
want to install their own?



Bill,

The latter in your statement, distribute the most modern transmission 
that the user can ignore or if they wish or swap out with something they 
prefer.


Basically it's easy. If it must be there to build the darn thing, pass 
GO and collect the $$$, then it should be there. If not, then it becomes 
an option and that is left up to the user/builder ergo openssl zlib and 
lua.


so +1 to including pcre in -deps (or srclib/pcre), basically as it is 
now in the other branches.


In reality I probably shouldn't say anything anymore as I adapted and 
overcame, begrudgingly. It sure was a sore subject back between 2.3.1 
and 2.3.2 and I guess it just held on and I could not resist making noise.


Sorry this got into the wrong thread, I blame my webmail :)

Gregg

Re: [RESULTS] [VOTE] Release httpd 2.3.4-alpha

[William A. Rowe Jr.]

Jeff Trawick wrote:
> On Thu, Dec 3, 2009 at 6:21 PM, William A. Rowe Jr.  
> wrote:
> 
>> Remember your -deps vote is to approve the release of apr 1.4.0-dev and the
>> apr-util 1.4.0 dev, and the API versioning rules will bind from that release
>> forwards.
> 
> The APR versioning rules are hopelessly broken if a tarball snapshot
> of the 1.4.x branch before a GA release casts the API in stone.
> 
> Surely I misunderstood you.

Is there a README indicating that the MAJOR/MINOR version tests for this
particular tarball are not relevant/complete?  No.

This is not a snapshot.  It is labeled httpd-2.3.4-alpha.tar.xx release.
You surely don't misunderstand what I said.

As for broken versioning rules, please take that to APR.

Perhaps in retrospect, APR would consider an even/odds approach as httpd
has for adding (even eliminating) interfaces during a development cycle.

Re: svn commit: r885606 - /httpd/httpd/trunk/build/rpm/httpd.init

[William A. Rowe Jr.]

Graham Leggett wrote:
> William A. Rowe Jr. wrote:
> 
>> Ok, so they want to roll their own.  Sounds like a maintainer issue.  What
>> does this say for using our httpd rpm for an Ubuntu or other distribution
>> of linux?
> 
> Ubuntu is Debian based, and uses the .deb packaging format, and startup
> scripts derived from the Debian layout.

The last I heard, the 'rpm' project is open source, free to be adopted by
any platform.

As for the rest of your comments, if we solve the general problem, I'm all
for including it in the httpd tree.  If we are solving specific packagers
problems, I'm ok with placing this in the httpd.a.o domain, but we should
move this nonsense outside of the development tree into a packaging tree.