Re: [web2py] Re: Randomly stuck web2py

2016-11-07 Thread Wasim Baig 
Just to update, we moved to nginx+uwsgi and things seem to be just fine
(lot faster too).

Will continue monitoring and report if anything seems wrong.

Thanks for the advice.

-wasim


On Tue, 25 Oct 2016 at 22:44 Michele Comitini 
wrote:

You can try to log/debug something meaningful using web2py's loggin
framework:
http://web2py.com/books/default/chapter/29/04/the-core?search=logger#Logging

BTW the CTRL-C (SIGINT) seems to show that the process is in the middle of
some I/O operation.
Try to use the strace command
strace -o logfile  -f python web2py.py ...

you will see a LOT of cruft when everything goes well, but it will give you
some hint on which system call python gets stuck.


2016-10-25 17:01 GMT+02:00 Wasim Baig :

Hi Niphlod/Michele:

Thanks for the suggestions.

Will put a rate limiter in front of Rocket just to be sure.

I wasn't able to see anything in the logs, actually, all I have is the
httpserver.log.
How can I get web2py to log more what its doing internally? -D 100 didn't
do anything, or maybe I didn't know where to look. If web2py could tell me
where its stuck that would be most helpful.

Concur on the SSL, we've noticed that this happens mostly on the SSL part.
Also, I've noticed it happens a lot more if we have multiple interfaces on
the same web2py process.  Interestingly, on a different installation, this
happens too, and hangs more often if I edit a file. If I don't touch them
much, they seems happier :(
This is leading me to believe that its something I'm doing in the
application that is causing this behavior. But the apps themselves are
generally fairly simple ones, forms, grids and mysql.

As a temporary hack, for now I run multiple web2py processes, and haproxy
in front of them, and have monit monitor each of the individual web2py
processes and kill/restart them if non-responsive, but this isn't fixing
the problem, just bandaging over it.

Again, thanks for the input.

-wasim


On Tue, 25 Oct 2016 at 14:11 Michele Comitini 
wrote:

@Wasim are you leaving some web2py shell open with a db transaction going?

Check uptime of your machine to see if load average is high or your run in
a dead-lock.
I see you are using certificates, I found that python ssl sockets have some
quirks depending on python vs openssl version.

Of course follow niphlod suggestions and keep in mind that it is easy to be
under DDos attack, again it is better to use a dedicated SSL fronted server
instead of python in the open internet...



2016-10-24 12:44 GMT+02:00 Niphlod :

see the log to trace any DDos attempts (or stuck requests )


On Friday, October 21, 2016 at 10:54:32 PM UTC+2, Wasim Baig wrote:

Been using web2py for sometime now, thank you all for making life so easy.

We use stock web2py src on debian 8, with Rocket and mysql and start them
in screen.

Frequently my web2py process get stuck. Ctrl-C won't kill it, we end up
Ctrl-Z it, and then kill -9 pid.

For example,

root 13387  0.8  0.4 1131328 156496 pts/63 Sl+  Oct18   7:12 python2.7
web2py.py -d /var/run/web2py_localhost_8000.pid -a XXX --interfaces=
127.0.0.1:8000
root 13421  0.0  0.4 1109424 143336 pts/8  Sl   Oct18   0:39 python2.7
web2py.py -d /var/run/web2py_8000.pid -a XXX
--interfaces=148.251.132.230:8000:/etc/ssl/private/batuni_pk.key:/etc/ssl/private/batuni_pk.crt
--ca-cert=/etc/ssl/private/AddTrustExternalCARoot.crt
--ca-cert=/etc/ssl/private/COMODORSAAddTrustCA.crt
--ca-cert=/etc/ssl/private/COMODORSADomainValidationSecureServerCA.crt
--ca-cert=/etc/ssl/private/batuni_pk.ca-bundle


Of these two processes, the one on localhost continues to respond fine, but
the one on public IP was stuck.
Nothing in the logs that I can see point to a problem.

What can I do to troubleshoot this?

-wasim

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.




-- 
Michele Comitini
Glisco s.n.c.
tel: +39 335 66 71 336

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
-

Re: [web2py] Re: Randomly stuck web2py

2016-10-25 Thread Michele Comitini 
You can try to log/debug something meaningful using web2py's loggin
framework:
http://web2py.com/books/default/chapter/29/04/the-core?search=logger#Logging

BTW the CTRL-C (SIGINT) seems to show that the process is in the middle of
some I/O operation.
Try to use the strace command
strace -o logfile  -f python web2py.py ...

you will see a LOT of cruft when everything goes well, but it will give you
some hint on which system call python gets stuck.


2016-10-25 17:01 GMT+02:00 Wasim Baig :

> Hi Niphlod/Michele:
>
> Thanks for the suggestions.
>
> Will put a rate limiter in front of Rocket just to be sure.
>
> I wasn't able to see anything in the logs, actually, all I have is the
> httpserver.log.
> How can I get web2py to log more what its doing internally? -D 100 didn't
> do anything, or maybe I didn't know where to look. If web2py could tell me
> where its stuck that would be most helpful.
>
> Concur on the SSL, we've noticed that this happens mostly on the SSL part.
> Also, I've noticed it happens a lot more if we have multiple interfaces on
> the same web2py process.  Interestingly, on a different installation, this
> happens too, and hangs more often if I edit a file. If I don't touch them
> much, they seems happier :(
> This is leading me to believe that its something I'm doing in the
> application that is causing this behavior. But the apps themselves are
> generally fairly simple ones, forms, grids and mysql.
>
> As a temporary hack, for now I run multiple web2py processes, and haproxy
> in front of them, and have monit monitor each of the individual web2py
> processes and kill/restart them if non-responsive, but this isn't fixing
> the problem, just bandaging over it.
>
> Again, thanks for the input.
>
> -wasim
>
>
> On Tue, 25 Oct 2016 at 14:11 Michele Comitini 
> wrote:
>
>> @Wasim are you leaving some web2py shell open with a db transaction going?
>>
>> Check uptime of your machine to see if load average is high or your run
>> in a dead-lock.
>> I see you are using certificates, I found that python ssl sockets have
>> some quirks depending on python vs openssl version.
>>
>> Of course follow niphlod suggestions and keep in mind that it is easy to
>> be under DDos attack, again it is better to use a dedicated SSL fronted
>> server instead of python in the open internet...
>>
>>
>>
>> 2016-10-24 12:44 GMT+02:00 Niphlod :
>>
>> see the log to trace any DDos attempts (or stuck requests )
>>
>>
>> On Friday, October 21, 2016 at 10:54:32 PM UTC+2, Wasim Baig wrote:
>>
>> Been using web2py for sometime now, thank you all for making life so easy.
>>
>> We use stock web2py src on debian 8, with Rocket and mysql and start them
>> in screen.
>>
>> Frequently my web2py process get stuck. Ctrl-C won't kill it, we end up
>> Ctrl-Z it, and then kill -9 pid.
>>
>> For example,
>>
>> root 13387  0.8  0.4 1131328 156496 pts/63 Sl+  Oct18   7:12
>> python2.7 web2py.py -d /var/run/web2py_localhost_8000.pid -a XXX
>> --interfaces=127.0.0.1:8000
>> root 13421  0.0  0.4 1109424 143336 pts/8  Sl   Oct18   0:39
>> python2.7 web2py.py -d /var/run/web2py_8000.pid -a XXX --interfaces=
>> 148.251.132.230:8000:/etc/ssl/private/batuni_pk.key:/etc/ssl/private/batuni_pk.crt
>> --ca-cert=/etc/ssl/private/AddTrustExternalCARoot.crt
>> --ca-cert=/etc/ssl/private/COMODORSAAddTrustCA.crt
>> --ca-cert=/etc/ssl/private/COMODORSADomainValidationSecureServerCA.crt
>> --ca-cert=/etc/ssl/private/batuni_pk.ca-bundle
>>
>>
>> Of these two processes, the one on localhost continues to respond fine,
>> but the one on public IP was stuck.
>> Nothing in the logs that I can see point to a problem.
>>
>> What can I do to troubleshoot this?
>>
>> -wasim
>>
>> --
>> Resources:
>> - http://web2py.com
>> - http://web2py.com/book (Documentation)
>> - http://github.com/web2py/web2py (Source code)
>> - https://code.google.com/p/web2py/issues/list (Report Issues)
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "web2py-users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to web2py+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>>
>>
>> --
>> Michele Comitini
>> Glisco s.n.c.
>> tel: +39 335 66 71 336
>>
>> --
>> Resources:
>> - http://web2py.com
>> - http://web2py.com/book (Documentation)
>> - http://github.com/web2py/web2py (Source code)
>> - https://code.google.com/p/web2py/issues/list (Report Issues)
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "web2py-users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to web2py+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
> --
> Resources:
> - http://web2py.com
> - http://web2py.com/book (Documentation)
> - http://github.com/web2py/web2py (Source code)
> -

Re: [web2py] Re: Randomly stuck web2py

2016-10-25 Thread Wasim Baig 
Hi Niphlod/Michele:

Thanks for the suggestions.

Will put a rate limiter in front of Rocket just to be sure.

I wasn't able to see anything in the logs, actually, all I have is the
httpserver.log.
How can I get web2py to log more what its doing internally? -D 100 didn't
do anything, or maybe I didn't know where to look. If web2py could tell me
where its stuck that would be most helpful.

Concur on the SSL, we've noticed that this happens mostly on the SSL part.
Also, I've noticed it happens a lot more if we have multiple interfaces on
the same web2py process.  Interestingly, on a different installation, this
happens too, and hangs more often if I edit a file. If I don't touch them
much, they seems happier :(
This is leading me to believe that its something I'm doing in the
application that is causing this behavior. But the apps themselves are
generally fairly simple ones, forms, grids and mysql.

As a temporary hack, for now I run multiple web2py processes, and haproxy
in front of them, and have monit monitor each of the individual web2py
processes and kill/restart them if non-responsive, but this isn't fixing
the problem, just bandaging over it.

Again, thanks for the input.

-wasim


On Tue, 25 Oct 2016 at 14:11 Michele Comitini 
wrote:

> @Wasim are you leaving some web2py shell open with a db transaction going?
>
> Check uptime of your machine to see if load average is high or your run in
> a dead-lock.
> I see you are using certificates, I found that python ssl sockets have
> some quirks depending on python vs openssl version.
>
> Of course follow niphlod suggestions and keep in mind that it is easy to
> be under DDos attack, again it is better to use a dedicated SSL fronted
> server instead of python in the open internet...
>
>
>
> 2016-10-24 12:44 GMT+02:00 Niphlod :
>
> see the log to trace any DDos attempts (or stuck requests )
>
>
> On Friday, October 21, 2016 at 10:54:32 PM UTC+2, Wasim Baig wrote:
>
> Been using web2py for sometime now, thank you all for making life so easy.
>
> We use stock web2py src on debian 8, with Rocket and mysql and start them
> in screen.
>
> Frequently my web2py process get stuck. Ctrl-C won't kill it, we end up
> Ctrl-Z it, and then kill -9 pid.
>
> For example,
>
> root 13387  0.8  0.4 1131328 156496 pts/63 Sl+  Oct18   7:12 python2.7
> web2py.py -d /var/run/web2py_localhost_8000.pid -a XXX --interfaces=
> 127.0.0.1:8000
> root 13421  0.0  0.4 1109424 143336 pts/8  Sl   Oct18   0:39 python2.7
> web2py.py -d /var/run/web2py_8000.pid -a XXX 
> --interfaces=148.251.132.230:8000:/etc/ssl/private/batuni_pk.key:/etc/ssl/private/batuni_pk.crt
> --ca-cert=/etc/ssl/private/AddTrustExternalCARoot.crt
> --ca-cert=/etc/ssl/private/COMODORSAAddTrustCA.crt
> --ca-cert=/etc/ssl/private/COMODORSADomainValidationSecureServerCA.crt
> --ca-cert=/etc/ssl/private/batuni_pk.ca-bundle
>
>
> Of these two processes, the one on localhost continues to respond fine,
> but the one on public IP was stuck.
> Nothing in the logs that I can see point to a problem.
>
> What can I do to troubleshoot this?
>
> -wasim
>
> --
> Resources:
> - http://web2py.com
> - http://web2py.com/book (Documentation)
> - http://github.com/web2py/web2py (Source code)
> - https://code.google.com/p/web2py/issues/list (Report Issues)
> ---
> You received this message because you are subscribed to the Google Groups
> "web2py-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to web2py+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
> --
> Michele Comitini
> Glisco s.n.c.
> tel: +39 335 66 71 336
>
> --
> Resources:
> - http://web2py.com
> - http://web2py.com/book (Documentation)
> - http://github.com/web2py/web2py (Source code)
> - https://code.google.com/p/web2py/issues/list (Report Issues)
> ---
> You received this message because you are subscribed to the Google Groups
> "web2py-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to web2py+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [web2py] Re: Randomly stuck web2py

2016-10-25 Thread Michele Comitini 
@Wasim are you leaving some web2py shell open with a db transaction going?

Check uptime of your machine to see if load average is high or your run in
a dead-lock.
I see you are using certificates, I found that python ssl sockets have some
quirks depending on python vs openssl version.

Of course follow niphlod suggestions and keep in mind that it is easy to be
under DDos attack, again it is better to use a dedicated SSL fronted server
instead of python in the open internet...



2016-10-24 12:44 GMT+02:00 Niphlod :

> see the log to trace any DDos attempts (or stuck requests )
>
>
> On Friday, October 21, 2016 at 10:54:32 PM UTC+2, Wasim Baig wrote:
>>
>> Been using web2py for sometime now, thank you all for making life so easy.
>>
>> We use stock web2py src on debian 8, with Rocket and mysql and start them
>> in screen.
>>
>> Frequently my web2py process get stuck. Ctrl-C won't kill it, we end up
>> Ctrl-Z it, and then kill -9 pid.
>>
>> For example,
>>
>> root 13387  0.8  0.4 1131328 156496 pts/63 Sl+  Oct18   7:12
>> python2.7 web2py.py -d /var/run/web2py_localhost_8000.pid -a XXX
>> --interfaces=127.0.0.1:8000
>> root 13421  0.0  0.4 1109424 143336 pts/8  Sl   Oct18   0:39
>> python2.7 web2py.py -d /var/run/web2py_8000.pid -a XXX --interfaces=
>> 148.251.132.230:8000:/etc/ssl/private/batuni_pk
>> .key:/etc/ssl/private/batuni_pk.crt 
>> --ca-cert=/etc/ssl/private/AddTrustExternalCARoot.crt
>> --ca-cert=/etc/ssl/private/COMODORSAAddTrustCA.crt
>> --ca-cert=/etc/ssl/private/COMODORSADomainValidationSecureServerCA.crt
>> --ca-cert=/etc/ssl/private/batuni_pk.ca-bundle
>>
>>
>> Of these two processes, the one on localhost continues to respond fine,
>> but the one on public IP was stuck.
>> Nothing in the logs that I can see point to a problem.
>>
>> What can I do to troubleshoot this?
>>
>> -wasim
>>
> --
> Resources:
> - http://web2py.com
> - http://web2py.com/book (Documentation)
> - http://github.com/web2py/web2py (Source code)
> - https://code.google.com/p/web2py/issues/list (Report Issues)
> ---
> You received this message because you are subscribed to the Google Groups
> "web2py-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to web2py+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Michele Comitini
Glisco s.n.c.
tel: +39 335 66 71 336

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[web2py] Re: Randomly stuck web2py

2016-10-24 Thread Niphlod 
see the log to trace any DDos attempts (or stuck requests )

On Friday, October 21, 2016 at 10:54:32 PM UTC+2, Wasim Baig wrote:
>
> Been using web2py for sometime now, thank you all for making life so easy.
>
> We use stock web2py src on debian 8, with Rocket and mysql and start them 
> in screen.
>
> Frequently my web2py process get stuck. Ctrl-C won't kill it, we end up 
> Ctrl-Z it, and then kill -9 pid.
>
> For example, 
>
> root 13387  0.8  0.4 1131328 156496 pts/63 Sl+  Oct18   7:12 python2.7 
> web2py.py -d /var/run/web2py_localhost_8000.pid -a XXX --interfaces=
> 127.0.0.1:8000 
> root 13421  0.0  0.4 1109424 143336 pts/8  Sl   Oct18   0:39 python2.7 
> web2py.py -d /var/run/web2py_8000.pid -a XXX 
> --interfaces=148.251.132.230:8000:/etc/ssl/private/batuni_pk.key:/etc/ssl/private/batuni_pk.crt
>  
> --ca-cert=/etc/ssl/private/AddTrustExternalCARoot.crt 
> --ca-cert=/etc/ssl/private/COMODORSAAddTrustCA.crt 
> --ca-cert=/etc/ssl/private/COMODORSADomainValidationSecureServerCA.crt 
> --ca-cert=/etc/ssl/private/batuni_pk.ca-bundle
>
>
> Of these two processes, the one on localhost continues to respond fine, 
> but the one on public IP was stuck.
> Nothing in the logs that I can see point to a problem. 
>
> What can I do to troubleshoot this?
>
> -wasim 
>

-- 
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
--- 
You received this message because you are subscribed to the Google Groups 
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.