Re: [web2py] Re: web2py scheduler with huge output or return values always timeout (even if the task finishes)

[Niphlod]

aw hell, it's true. huge outputs are effectively blocking. don't know 
if it's fixable though. I don't see any obvious error in the implementation 
that could prevent it

On Monday, February 29, 2016 at 9:26:38 PM UTC+1, Niphlod wrote:
>
> uhm, not sure about that though. the order of operations in the scheduler 
> respects the docs, and even if the bugreport has the 128k statement, the 
> docs don't report it let me try.
>
> On Monday, February 29, 2016 at 9:14:41 PM UTC+1, Boris Aramis Aguilar 
> Rodríguez wrote:
>>
>> It happends not only while printing but also when returning values on a 
>> function as noted on the example :) so yes i do avoid printing but this 
>> also happens when returning more than 128kb of data (as noted by the bug 
>> reports) due to the fact (it seems) that the max data a process can 
>> comunicate to other via pipes is around that limit so that is the 
>> underlying reason.
>> El feb 29, 2016 2:07 PM, "Niphlod"  escribió:
>>
>>> don't know if it's documented on the book but nevertheless there has 
>>> been a few times it popped up on this group. As output is buffered, 
>>> especially on Windows but on unixes flavoured OSes as well, the "printed" 
>>> output should be limited. 
>>> Of course it never has been a real issue because if you need to return 
>>> something you shouldn't NEVER EVER use print. 
>>> And because you should be already accustomed to NEVER EVER using print 
>>> in any web2py application, first because print doesn't get you anything and 
>>> second because if you need huge prints you're probably sidestepping a 
>>> correct usage of the logging library.
>>> Anyhow, I'll submit right away a PR on the book with a solid note about 
>>> it.
>>>
>>> On Monday, February 29, 2016 at 7:41:41 PM UTC+1, Boris Aramis Aguilar 
>>> Rodríguez wrote:

 Hi, there is an issue driving me crazy with the web2py scheduler:

 If you return something that has a huge size then it will always 
 timeout; even if the scheduler task correctly finishes. Let me explain 
 with 
 an example:

 def small_test():
 s = 's'*1256018
 another_s = s
 #print s
 #print another_s
 #print 'FINISHED PROCESS'
 return dict(s = s, another_s = another_s, f = 'finished')

 small_test is the function to execute, as you can see a string full of 
 's' 1256018 times is. Simple

 So when you enqueue the scheduler every time the output is the same: 
 http://prnt.sc/a9iarj (screenshot of the TIMEOUT)

 As you can see from the screenshot, the process actually finished; 
 while logging the scheduler output shows the following:

 DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:   work to do 1405
 DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:new scheduler_run 
 record
 INFO:web2py.scheduler.PRTALONENETLAPP-SRV#24475:new task 1405 
 "small_test" portal/default.small_test
 DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475: new task allocated: 
 portal/default.small_test
 DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:   task starting
 DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:task started
 DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:new task report: 
 COMPLETED
 DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:   result: {"s": 
 "ss$
 DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording 
 heartbeat (RUNNING)
 DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording 
 heartbeat (RUNNING)
 DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording 
 heartbeat (RUNNING)
 DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording 
 heartbeat (RUNNING)
 DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording 
 heartbeat (RUNNING)
 DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:freeing workers 
 that have not sent heartbeat
 DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording 
 heartbeat (RUNNING)
 DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording 
 heartbeat (RUNNING)
 DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording 
 heartbeat (RUNNING)
 DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording 
 heartbeat (RUNNING)
 DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording 
 heartbeat (RUNNING)
 DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:freeing workers 
 that have not sent heartbeat
 DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording 
 heartbeat (RUNNING)
 DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording 
 heartbeat (RUNNING)
 DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording 
 heartbeat (RUNNING)
 

Re: [web2py] Re: web2py scheduler with huge output or return values always timeout (even if the task finishes)

[Boris Aramis Aguilar Rodríguez]

Please do try :) I might be wrong :P and it could be something else; but I
tried to narrow it down as much as I could.
El feb 29, 2016 2:26 PM, "Niphlod"  escribió:

> uhm, not sure about that though. the order of operations in the scheduler
> respects the docs, and even if the bugreport has the 128k statement, the
> docs don't report it let me try.
>
> On Monday, February 29, 2016 at 9:14:41 PM UTC+1, Boris Aramis Aguilar
> Rodríguez wrote:
>>
>> It happends not only while printing but also when returning values on a
>> function as noted on the example :) so yes i do avoid printing but this
>> also happens when returning more than 128kb of data (as noted by the bug
>> reports) due to the fact (it seems) that the max data a process can
>> comunicate to other via pipes is around that limit so that is the
>> underlying reason.
>> El feb 29, 2016 2:07 PM, "Niphlod"  escribió:
>>
>>> don't know if it's documented on the book but nevertheless there has
>>> been a few times it popped up on this group. As output is buffered,
>>> especially on Windows but on unixes flavoured OSes as well, the "printed"
>>> output should be limited.
>>> Of course it never has been a real issue because if you need to return
>>> something you shouldn't NEVER EVER use print.
>>> And because you should be already accustomed to NEVER EVER using print
>>> in any web2py application, first because print doesn't get you anything and
>>> second because if you need huge prints you're probably sidestepping a
>>> correct usage of the logging library.
>>> Anyhow, I'll submit right away a PR on the book with a solid note about
>>> it.
>>>
>>> On Monday, February 29, 2016 at 7:41:41 PM UTC+1, Boris Aramis Aguilar
>>> Rodríguez wrote:

 Hi, there is an issue driving me crazy with the web2py scheduler:

 If you return something that has a huge size then it will always
 timeout; even if the scheduler task correctly finishes. Let me explain with
 an example:

 def small_test():
 s = 's'*1256018
 another_s = s
 #print s
 #print another_s
 #print 'FINISHED PROCESS'
 return dict(s = s, another_s = another_s, f = 'finished')

 small_test is the function to execute, as you can see a string full of
 's' 1256018 times is. Simple

 So when you enqueue the scheduler every time the output is the same:
 http://prnt.sc/a9iarj (screenshot of the TIMEOUT)

 As you can see from the screenshot, the process actually finished;
 while logging the scheduler output shows the following:

 DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:   work to do 1405
 DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:new scheduler_run
 record
 INFO:web2py.scheduler.PRTALONENETLAPP-SRV#24475:new task 1405
 "small_test" portal/default.small_test
 DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475: new task allocated:
 portal/default.small_test
 DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:   task starting
 DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:task started
 DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:new task report:
 COMPLETED
 DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:   result: {"s":
 "ss$
 DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording
 heartbeat (RUNNING)
 DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording
 heartbeat (RUNNING)
 DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording
 heartbeat (RUNNING)
 DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording
 heartbeat (RUNNING)
 DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording
 heartbeat (RUNNING)
 DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:freeing workers
 that have not sent heartbeat
 DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording
 heartbeat (RUNNING)
 DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording
 heartbeat (RUNNING)
 DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording
 heartbeat (RUNNING)
 DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording
 heartbeat (RUNNING)
 DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording
 heartbeat (RUNNING)
 DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:freeing workers
 that have not sent heartbeat
 DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording
 heartbeat (RUNNING)
 DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording
 heartbeat (RUNNING)
 DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording
 heartbeat (RUNNING)
 DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording
 heartbeat (RUNNING)
 

Re: [web2py] Re: web2py scheduler with huge output or return values always timeout (even if the task finishes)

[Niphlod]

uhm, not sure about that though. the order of operations in the scheduler 
respects the docs, and even if the bugreport has the 128k statement, the 
docs don't report it let me try.

On Monday, February 29, 2016 at 9:14:41 PM UTC+1, Boris Aramis Aguilar 
Rodríguez wrote:
>
> It happends not only while printing but also when returning values on a 
> function as noted on the example :) so yes i do avoid printing but this 
> also happens when returning more than 128kb of data (as noted by the bug 
> reports) due to the fact (it seems) that the max data a process can 
> comunicate to other via pipes is around that limit so that is the 
> underlying reason.
> El feb 29, 2016 2:07 PM, "Niphlod"  
> escribió:
>
>> don't know if it's documented on the book but nevertheless there has been 
>> a few times it popped up on this group. As output is buffered, especially 
>> on Windows but on unixes flavoured OSes as well, the "printed" output 
>> should be limited. 
>> Of course it never has been a real issue because if you need to return 
>> something you shouldn't NEVER EVER use print. 
>> And because you should be already accustomed to NEVER EVER using print in 
>> any web2py application, first because print doesn't get you anything and 
>> second because if you need huge prints you're probably sidestepping a 
>> correct usage of the logging library.
>> Anyhow, I'll submit right away a PR on the book with a solid note about 
>> it.
>>
>> On Monday, February 29, 2016 at 7:41:41 PM UTC+1, Boris Aramis Aguilar 
>> Rodríguez wrote:
>>>
>>> Hi, there is an issue driving me crazy with the web2py scheduler:
>>>
>>> If you return something that has a huge size then it will always 
>>> timeout; even if the scheduler task correctly finishes. Let me explain with 
>>> an example:
>>>
>>> def small_test():
>>> s = 's'*1256018
>>> another_s = s
>>> #print s
>>> #print another_s
>>> #print 'FINISHED PROCESS'
>>> return dict(s = s, another_s = another_s, f = 'finished')
>>>
>>> small_test is the function to execute, as you can see a string full of 
>>> 's' 1256018 times is. Simple
>>>
>>> So when you enqueue the scheduler every time the output is the same: 
>>> http://prnt.sc/a9iarj (screenshot of the TIMEOUT)
>>>
>>> As you can see from the screenshot, the process actually finished; while 
>>> logging the scheduler output shows the following:
>>>
>>> DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:   work to do 1405
>>> DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:new scheduler_run 
>>> record
>>> INFO:web2py.scheduler.PRTALONENETLAPP-SRV#24475:new task 1405 
>>> "small_test" portal/default.small_test
>>> DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475: new task allocated: 
>>> portal/default.small_test
>>> DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:   task starting
>>> DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:task started
>>> DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:new task report: 
>>> COMPLETED
>>> DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:   result: {"s": 
>>> "ss$
>>> DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording 
>>> heartbeat (RUNNING)
>>> DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording 
>>> heartbeat (RUNNING)
>>> DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording 
>>> heartbeat (RUNNING)
>>> DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording 
>>> heartbeat (RUNNING)
>>> DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording 
>>> heartbeat (RUNNING)
>>> DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:freeing workers 
>>> that have not sent heartbeat
>>> DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording 
>>> heartbeat (RUNNING)
>>> DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording 
>>> heartbeat (RUNNING)
>>> DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording 
>>> heartbeat (RUNNING)
>>> DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording 
>>> heartbeat (RUNNING)
>>> DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording 
>>> heartbeat (RUNNING)
>>> DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:freeing workers 
>>> that have not sent heartbeat
>>> DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording 
>>> heartbeat (RUNNING)
>>> DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording 
>>> heartbeat (RUNNING)
>>> DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording 
>>> heartbeat (RUNNING)
>>> DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording 
>>> heartbeat (RUNNING)
>>> DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording 
>>> heartbeat (RUNNING)
>>> DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:freeing workers 
>>> that have not sent heartbeat
>>> 

Re: [web2py] Re: web2py scheduler with huge output or return values always timeout (even if the task finishes)

[Boris Aramis Aguilar Rodríguez]

It happends not only while printing but also when returning values on a
function as noted on the example :) so yes i do avoid printing but this
also happens when returning more than 128kb of data (as noted by the bug
reports) due to the fact (it seems) that the max data a process can
comunicate to other via pipes is around that limit so that is the
underlying reason.
El feb 29, 2016 2:07 PM, "Niphlod"  escribió:

> don't know if it's documented on the book but nevertheless there has been
> a few times it popped up on this group. As output is buffered, especially
> on Windows but on unixes flavoured OSes as well, the "printed" output
> should be limited.
> Of course it never has been a real issue because if you need to return
> something you shouldn't NEVER EVER use print.
> And because you should be already accustomed to NEVER EVER using print in
> any web2py application, first because print doesn't get you anything and
> second because if you need huge prints you're probably sidestepping a
> correct usage of the logging library.
> Anyhow, I'll submit right away a PR on the book with a solid note about it.
>
> On Monday, February 29, 2016 at 7:41:41 PM UTC+1, Boris Aramis Aguilar
> Rodríguez wrote:
>>
>> Hi, there is an issue driving me crazy with the web2py scheduler:
>>
>> If you return something that has a huge size then it will always timeout;
>> even if the scheduler task correctly finishes. Let me explain with an
>> example:
>>
>> def small_test():
>> s = 's'*1256018
>> another_s = s
>> #print s
>> #print another_s
>> #print 'FINISHED PROCESS'
>> return dict(s = s, another_s = another_s, f = 'finished')
>>
>> small_test is the function to execute, as you can see a string full of
>> 's' 1256018 times is. Simple
>>
>> So when you enqueue the scheduler every time the output is the same:
>> http://prnt.sc/a9iarj (screenshot of the TIMEOUT)
>>
>> As you can see from the screenshot, the process actually finished; while
>> logging the scheduler output shows the following:
>>
>> DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:   work to do 1405
>> DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:new scheduler_run
>> record
>> INFO:web2py.scheduler.PRTALONENETLAPP-SRV#24475:new task 1405
>> "small_test" portal/default.small_test
>> DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475: new task allocated:
>> portal/default.small_test
>> DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:   task starting
>> DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:task started
>> DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:new task report:
>> COMPLETED
>> DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:   result: {"s":
>> "ss$
>> DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording
>> heartbeat (RUNNING)
>> DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording
>> heartbeat (RUNNING)
>> DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording
>> heartbeat (RUNNING)
>> DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording
>> heartbeat (RUNNING)
>> DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording
>> heartbeat (RUNNING)
>> DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:freeing workers
>> that have not sent heartbeat
>> DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording
>> heartbeat (RUNNING)
>> DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording
>> heartbeat (RUNNING)
>> DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording
>> heartbeat (RUNNING)
>> DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording
>> heartbeat (RUNNING)
>> DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording
>> heartbeat (RUNNING)
>> DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:freeing workers
>> that have not sent heartbeat
>> DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording
>> heartbeat (RUNNING)
>> DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording
>> heartbeat (RUNNING)
>> DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording
>> heartbeat (RUNNING)
>> DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording
>> heartbeat (RUNNING)
>> DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording
>> heartbeat (RUNNING)
>> DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:freeing workers
>> that have not sent heartbeat
>> DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:recording
>> heartbeat (RUNNING)
>> DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475:task timeout
>> DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475: recording task report
>> in db (TIMEOUT)
>> DEBUG:web2py.scheduler.PRTALONENETLAPP-SRV#24475: status TIMEOUT,
>> stop_time 2016-02-29 11:56:52.393706, run_result {"s":
>> "sss$
>> INFO:web2py.scheduler.PRTALONENETLAPP-SRV#24475:task completed (TIMEOUT)
>>