Re: [PLUG] Can anyone recommend intligent way to debug not executing CGI scripts on Apache2?

[Tomas K]

Just in case, someone still think about this topic.
For the lack of other ideas - I resolved the issue by tearing my config
file templates apart and putting them back together from the working
deployed parts. Diffing the old (broken config) and new working one
produces identity, so do access logs. The only difference I can observe
is that in one Apache instance spawns process and executes the .cgi -
returning generated response, while in the broken instance returns .cgi
as text.
I resolved this independently of posting my question - I was hoping to
learn some way of observing/querying what configuration Apache actually
applies internally. That would have allowed me to compare differences
from Apache's point of view, not mine, of the config files.
It would have made a great difference - the next time I have the
same/similar problem. It is akin of knowing how to competently fish
rather than just randomly poking into the water with a spear -
thankfully Apache is not as smart as a fish looking at me from the
water.
Anyway - .cgi execution is one of those areas without log coverage.
Tomas
On Mon, 2019-01-07 at 13:52 -0800, Tomas Kuchta wrote:
> The problem is 99.9% in the Apache config - the same application
> runs on one server not the other. No matter which way I synchronize
> the code, it executes in one place, not the other.
> The only difference are Apache config files, produced mostly with
> Jinja templates and search and replace with a few config lines added
> related to host aliases to flip things to production.
> 
> Often, the whole thing ends up working after a few days of tearing it
> apart and putting it back together. I have a feeling that it may be
> something like not printable characters or a white space typo, 
> In any case it is not producing any load errors. Diffing config files
> produces identity with the exception of the host name and intentional
> differences.
> 
> I tried dos2unix, etc too. ... All I can come up with is to take it
> apart randomly and put it back together until I get lucky and it all
> works. No, errors, no messages, no hints. Web is full of similar
> (Apache does not execute my code) issues with no suggestion how to go
> after it other than eye balling the configs.
> 
> On Mon, Jan 7, 2019, 1:31 PM Paul Heinlein  wrote:
> > On Mon, 7 Jan 2019, Tomas Kuchta wrote:
> > 
> > 
> > 
> > > Yes, the file serving part works, and is observable in the logs.
> > There is
> > 
> > > good visibility on client side with curl or browsers. When Apache
> > executes
> > 
> > > it, I get the output html/json/ When it doesn't work, I get
> > to download
> > 
> > > the code.
> > 
> > >
> > 
> > > I usually build couple of identical machines and synchronize
> > their
> > 
> > > config/content. Still, when one machine in the pair is stubbornly
> > refusing
> > 
> > > to run some stuff, it takes a lot of diffing, renaming,
> > rebuilding, ...
> > 
> > >
> > 
> > > What makes Apache to choose to execute some files and not the
> > others, is
> > 
> > > the $M question.
> > 
> > 
> > 
> > There are a few possible answers.
> > 
> > 
> > 
> > First, the non-executing scripts may live in a directory not 
> > 
> > identified by a ScriptAlias directive or in a directory that
> > doesn't 
> > 
> > have "SetHandler cgi-script" and "Options +ExecCGI" configured.
> > 
> > 
> > 
> > Second, if the CGI script is handled by an interpreter
> > (/usr/bin/perl, 
> > 
> > for example, or /usr/bin/python) the #! line may be incorrect or
> > the 
> > 
> > file may not be marked as executable in the filesystem.
> > 
> > 
> > 
> > Third, the CGI script fails to set the Content-Type header of the 
> > 
> > return text correctly. Use "curl -I <>" to see.
> > In 
> > 
> > cases where it's failing, you're probably getting "text/plain";
> > but 
> > 
> > where it's successful you're seeing "application/json".
> > 
> > 
> > 
> > Those are the most likely cases I can imagine.
> > 
> > 
> > 
> > 
___
PLUG mailing list
PLUG@pdxlinux.org
http://lists.pdxlinux.org/mailman/listinfo/plug

Re: [PLUG] Can anyone recommend intligent way to debug not executing CGI scripts on Apache2?

[Larry Brigman]

I would suspect that there are conditionals in those jinja2 templates that
is making things different.


On Mon, Jan 7, 2019, 6:28 PM Bill Barry  On Tue, Jan 8, 2019 at 2:02 AM Bill Barry  wrote:
>
> >
> >
> > On Mon, Jan 7, 2019 at 9:52 PM Tomas Kuchta <
> tomas.kuchta.li...@gmail.com>
> > wrote:
> >
> >> The problem is 99.9% in the Apache config - the same application
> runs
> >> on one server not the other. No matter which way I synchronize the code,
> >> it
> >> executes in one place, not the other.
> >>
> >
> >
> >
> >>
> >> The only difference are Apache config files, produced mostly with Jinja
> >> templates and search and replace with a few config lines added related
> to
> >> host aliases to flip things to production.
> >>
> >>
> > That does sound like a good source for the problem,  maybe you can create
> > the config files in a completely different way, say static files that
> read
> > from environment variables.
> >
> > Bill
> >
>
> On the other hand that does not really explain the symptoms.
>
> Bill
> ___
> PLUG mailing list
> PLUG@pdxlinux.org
> http://lists.pdxlinux.org/mailman/listinfo/plug
>
___
PLUG mailing list
PLUG@pdxlinux.org
http://lists.pdxlinux.org/mailman/listinfo/plug

Re: [PLUG] Can anyone recommend intligent way to debug not executing CGI scripts on Apache2?

[Bill Barry]

On Tue, Jan 8, 2019 at 2:02 AM Bill Barry  wrote:

>
>
> On Mon, Jan 7, 2019 at 9:52 PM Tomas Kuchta 
> wrote:
>
>> The problem is 99.9% in the Apache config - the same application runs
>> on one server not the other. No matter which way I synchronize the code,
>> it
>> executes in one place, not the other.
>>
>
>
>
>>
>> The only difference are Apache config files, produced mostly with Jinja
>> templates and search and replace with a few config lines added related to
>> host aliases to flip things to production.
>>
>>
> That does sound like a good source for the problem,  maybe you can create
> the config files in a completely different way, say static files that read
> from environment variables.
>
> Bill
>

On the other hand that does not really explain the symptoms.

Bill
___
PLUG mailing list
PLUG@pdxlinux.org
http://lists.pdxlinux.org/mailman/listinfo/plug

Re: [PLUG] Can anyone recommend intligent way to debug not executing CGI scripts on Apache2?

[Bill Barry]

On Mon, Jan 7, 2019 at 9:52 PM Tomas Kuchta 
wrote:

> The problem is 99.9% in the Apache config - the same application runs
> on one server not the other. No matter which way I synchronize the code, it
> executes in one place, not the other.
>



>
> The only difference are Apache config files, produced mostly with Jinja
> templates and search and replace with a few config lines added related to
> host aliases to flip things to production.
>
>
That does sound like a good source for the problem,  maybe you can create
the config files in a completely different way, say static files that read
from environment variables.

Bill
___
PLUG mailing list
PLUG@pdxlinux.org
http://lists.pdxlinux.org/mailman/listinfo/plug

Re: [PLUG] Can anyone recommend intligent way to debug not executing CGI scripts on Apache2?

[Richard England]

On 1/7/19 11:12 AM, Tomas Kuchta wrote:

Hi,

I am having difficulty to debug why CGI scripts are not executed by
Apache2, again.

Can someone recommend some iteligent, algorithmical and converging way to
debug this?

There is nothing visible in the logs with CGIs, no error messages, no
config errors, no obvious file ownership, permission issues.

I am standing up the servers using Ansible, one like the other, repeatably.
Still, some servers work, some do not. Most of the time, the script execute
fine upon provisionning, some times the do not. When they do not run, it
takes days of almost random rebuilds to resolve. Some times .cgi works,
sometimes .py/php does not. Often helloWorld.cgi runs but not some other
random.cgi. Usually, what I want/need does not run. I am sick of sinking
days in this unproductive trial and error...
Is Ansible really creating the exact same environment on each system? 
Protections and all? Any differences in the "ssh" related environment on 
the systems you are provisioning?

___
PLUG mailing list
PLUG@pdxlinux.org
http://lists.pdxlinux.org/mailman/listinfo/plug

Re: [PLUG] Can anyone recommend intligent way to debug not executing CGI scripts on Apache2?

[Tomas Kuchta]

The problem is 99.9% in the Apache config - the same application runs
on one server not the other. No matter which way I synchronize the code, it
executes in one place, not the other.

The only difference are Apache config files, produced mostly with Jinja
templates and search and replace with a few config lines added related to
host aliases to flip things to production.

Often, the whole thing ends up working after a few days of tearing it apart
and putting it back together. I have a feeling that it may be something
like not printable characters or a white space typo,  In any case it is
not producing any load errors. Diffing config files produces identity with
the exception of the host name and intentional differences.

I tried dos2unix, etc too. ... All I can come up with is to take it apart
randomly and put it back together until I get lucky and it all works. No,
errors, no messages, no hints. Web is full of similar (Apache does not
execute my code) issues with no suggestion how to go after it other than
eye balling the configs.

On Mon, Jan 7, 2019, 1:31 PM Paul Heinlein  On Mon, 7 Jan 2019, Tomas Kuchta wrote:
>
> > Yes, the file serving part works, and is observable in the logs. There is
> > good visibility on client side with curl or browsers. When Apache
> executes
> > it, I get the output html/json/ When it doesn't work, I get to
> download
> > the code.
> >
> > I usually build couple of identical machines and synchronize their
> > config/content. Still, when one machine in the pair is stubbornly
> refusing
> > to run some stuff, it takes a lot of diffing, renaming, rebuilding, ...
> >
> > What makes Apache to choose to execute some files and not the others, is
> > the $M question.
>
> There are a few possible answers.
>
> First, the non-executing scripts may live in a directory not
> identified by a ScriptAlias directive or in a directory that doesn't
> have "SetHandler cgi-script" and "Options +ExecCGI" configured.
>
> Second, if the CGI script is handled by an interpreter (/usr/bin/perl,
> for example, or /usr/bin/python) the #! line may be incorrect or the
> file may not be marked as executable in the filesystem.
>
> Third, the CGI script fails to set the Content-Type header of the
> return text correctly. Use "curl -I <>" to see. In
> cases where it's failing, you're probably getting "text/plain"; but
> where it's successful you're seeing "application/json".
>
> Those are the most likely cases I can imagine.
>
> --
> Paul Heinlein
> heinl...@madboa.com
> 45°38' N, 122°6' W___
> PLUG mailing list
> PLUG@pdxlinux.org
> http://lists.pdxlinux.org/mailman/listinfo/plug
>
___
PLUG mailing list
PLUG@pdxlinux.org
http://lists.pdxlinux.org/mailman/listinfo/plug

Re: [PLUG] Can anyone recommend intligent way to debug not executing CGI scripts on Apache2?

[Louis Kowolowski]

On Jan 7, 2019, at 3:30 PM, Paul Heinlein  wrote:
> 
> On Mon, 7 Jan 2019, Tomas Kuchta wrote:
> 
>> Yes, the file serving part works, and is observable in the logs. There is
>> good visibility on client side with curl or browsers. When Apache executes
>> it, I get the output html/json/ When it doesn't work, I get to download
>> the code.
>> 
>> I usually build couple of identical machines and synchronize their
>> config/content. Still, when one machine in the pair is stubbornly refusing
>> to run some stuff, it takes a lot of diffing, renaming, rebuilding, ...
>> 
>> What makes Apache to choose to execute some files and not the others, is
>> the $M question.
> 
> There are a few possible answers.
> 
> First, the non-executing scripts may live in a directory not identified by a 
> ScriptAlias directive or in a directory that doesn't have "SetHandler 
> cgi-script" and "Options +ExecCGI" configured.
> 
> Second, if the CGI script is handled by an interpreter (/usr/bin/perl, for 
> example, or /usr/bin/python) the #! line may be incorrect or the file may not 
> be marked as executable in the filesystem.
> 
> Third, the CGI script fails to set the Content-Type header of the return text 
> correctly. Use "curl -I <>" to see. In cases where it's 
> failing, you're probably getting "text/plain"; but where it's successful 
> you're seeing "application/json".
> 
> Those are the most likely cases I can imagine.
> 
Don't forget the script extension for CGI, and also file permissions of the 
script itself.

--
Louis Kowolowskilou...@cryptomonkeys.org 

Cryptomonkeys:   http://www.cryptomonkeys.com/ 


Making life more interesting for people since 1977

___
PLUG mailing list
PLUG@pdxlinux.org
http://lists.pdxlinux.org/mailman/listinfo/plug

Re: [PLUG] Can anyone recommend intligent way to debug not executing CGI scripts on Apache2?

[Paul Heinlein]

On Mon, 7 Jan 2019, Tomas Kuchta wrote:


Yes, the file serving part works, and is observable in the logs. There is
good visibility on client side with curl or browsers. When Apache executes
it, I get the output html/json/ When it doesn't work, I get to download
the code.

I usually build couple of identical machines and synchronize their
config/content. Still, when one machine in the pair is stubbornly refusing
to run some stuff, it takes a lot of diffing, renaming, rebuilding, ...

What makes Apache to choose to execute some files and not the others, is
the $M question.


There are a few possible answers.

First, the non-executing scripts may live in a directory not 
identified by a ScriptAlias directive or in a directory that doesn't 
have "SetHandler cgi-script" and "Options +ExecCGI" configured.


Second, if the CGI script is handled by an interpreter (/usr/bin/perl, 
for example, or /usr/bin/python) the #! line may be incorrect or the 
file may not be marked as executable in the filesystem.


Third, the CGI script fails to set the Content-Type header of the 
return text correctly. Use "curl -I <>" to see. In 
cases where it's failing, you're probably getting "text/plain"; but 
where it's successful you're seeing "application/json".


Those are the most likely cases I can imagine.

--
Paul Heinlein
heinl...@madboa.com
45°38' N, 122°6' W___
PLUG mailing list
PLUG@pdxlinux.org
http://lists.pdxlinux.org/mailman/listinfo/plug

Re: [PLUG] Can anyone recommend intligent way to debug not executing CGI scripts on Apache2?

[Tomas Kuchta]

Yes, the file serving part works, and is observable in the logs. There is
good visibility on client side with curl or browsers. When Apache executes
it, I get the output html/json/ When it doesn't work, I get to download
the code.

I usually build couple of identical machines and synchronize their
config/content. Still, when one machine in the pair is stubbornly refusing
to run some stuff, it takes a lot of diffing, renaming, rebuilding, ...

What makes Apache to choose to execute some files and not the others, is
the $M question.

On Mon, Jan 7, 2019, 11:34 AM Bill Barry  On Mon, Jan 7, 2019 at 7:12 PM Tomas Kuchta 
> wrote:
>
> > Hi,
> >
> > I am having difficulty to debug why CGI scripts are not executed by
> > Apache2, again.
> >
> > Can someone recommend some iteligent, algorithmical and converging way to
> > debug this?
> >
> > There is nothing visible in the logs with CGIs, no error messages, no
> > config errors, no obvious file ownership, permission issues.
> >
> > I am standing up the servers using Ansible, one like the other,
> repeatably.
> > Still, some servers work, some do not. Most of the time, the script
> execute
> > fine upon provisionning, some times the do not. When they do not run, it
> > takes days of almost random rebuilds to resolve. Some times .cgi works,
> > sometimes .py/php does not. Often helloWorld.cgi runs but not some other
> > random.cgi. Usually, what I want/need does not run. I am sick of sinking
> > days in this unproductive trial and error...
> >
> > Any sensible and calm advice?
> >
> > Thanks, Tomas
> >
> > Can you confirm that Apache is receiving the request? If not is it an
> upstream DNS/networking problem?
>
> Bill
> ___
> PLUG mailing list
> PLUG@pdxlinux.org
> http://lists.pdxlinux.org/mailman/listinfo/plug
>
___
PLUG mailing list
PLUG@pdxlinux.org
http://lists.pdxlinux.org/mailman/listinfo/plug

Re: [PLUG] Can anyone recommend intligent way to debug not executing CGI scripts on Apache2?

[Bill Barry]

On Mon, Jan 7, 2019 at 7:12 PM Tomas Kuchta 
wrote:

> Hi,
>
> I am having difficulty to debug why CGI scripts are not executed by
> Apache2, again.
>
> Can someone recommend some iteligent, algorithmical and converging way to
> debug this?
>
> There is nothing visible in the logs with CGIs, no error messages, no
> config errors, no obvious file ownership, permission issues.
>
> I am standing up the servers using Ansible, one like the other, repeatably.
> Still, some servers work, some do not. Most of the time, the script execute
> fine upon provisionning, some times the do not. When they do not run, it
> takes days of almost random rebuilds to resolve. Some times .cgi works,
> sometimes .py/php does not. Often helloWorld.cgi runs but not some other
> random.cgi. Usually, what I want/need does not run. I am sick of sinking
> days in this unproductive trial and error...
>
> Any sensible and calm advice?
>
> Thanks, Tomas
>
> Can you confirm that Apache is receiving the request? If not is it an
upstream DNS/networking problem?

Bill
___
PLUG mailing list
PLUG@pdxlinux.org
http://lists.pdxlinux.org/mailman/listinfo/plug

Re: [PLUG] Can anyone recommend intligent way to debug not executing CGI scripts on Apache2?

[Russell Senior]

I'm a big fan of strace.

On Mon, Jan 7, 2019 at 11:14 AM Tomas Kuchta 
wrote:

> Hi,
>
> I am having difficulty to debug why CGI scripts are not executed by
> Apache2, again.
>
> Can someone recommend some iteligent, algorithmical and converging way to
> debug this?
>
> There is nothing visible in the logs with CGIs, no error messages, no
> config errors, no obvious file ownership, permission issues.
>
> I am standing up the servers using Ansible, one like the other, repeatably.
> Still, some servers work, some do not. Most of the time, the script execute
> fine upon provisionning, some times the do not. When they do not run, it
> takes days of almost random rebuilds to resolve. Some times .cgi works,
> sometimes .py/php does not. Often helloWorld.cgi runs but not some other
> random.cgi. Usually, what I want/need does not run. I am sick of sinking
> days in this unproductive trial and error...
>
> Any sensible and calm advice?
>
> Thanks, Tomas
> ___
> PLUG mailing list
> PLUG@pdxlinux.org
> http://lists.pdxlinux.org/mailman/listinfo/plug
>
___
PLUG mailing list
PLUG@pdxlinux.org
http://lists.pdxlinux.org/mailman/listinfo/plug

[PLUG] Can anyone recommend intligent way to debug not executing CGI scripts on Apache2?

[Tomas Kuchta]

Hi,

I am having difficulty to debug why CGI scripts are not executed by
Apache2, again.

Can someone recommend some iteligent, algorithmical and converging way to
debug this?

There is nothing visible in the logs with CGIs, no error messages, no
config errors, no obvious file ownership, permission issues.

I am standing up the servers using Ansible, one like the other, repeatably.
Still, some servers work, some do not. Most of the time, the script execute
fine upon provisionning, some times the do not. When they do not run, it
takes days of almost random rebuilds to resolve. Some times .cgi works,
sometimes .py/php does not. Often helloWorld.cgi runs but not some other
random.cgi. Usually, what I want/need does not run. I am sick of sinking
days in this unproductive trial and error...

Any sensible and calm advice?

Thanks, Tomas
___
PLUG mailing list
PLUG@pdxlinux.org
http://lists.pdxlinux.org/mailman/listinfo/plug