Oh, I didnt know that. Basically, I am trying to run several processes and
then monitor their output files periodically and import their content to
DB, so that all the fancy DB stuff can be done on top of them. From my
knowledge, periodic task that runs for the whole test period would block
the web2py execution, which is what I am trying to prevent by threading.
Jan
Dne pátek, 18. května 2012 16:56:24 UTC+2 Massimo Di Pierro napsal(a):
>
> Rule #1. You cannot use threading in a model or controller.
>
> When an http request arrives the web server spawns a new thread which runs
> your models, controller and view.
> This call is wrapped into a transaction which connects to the database (or
> pick connection from pool), then commits and closes the database. These
> threads are started and stopped by the web server, not the application.
>
> If you make your own threads from within a model, they may continue run
> after the database is closed. Even if if you try prevent it
> by explicitly joining, the web server may kill one of the threads at will
> with unpredictable results. This can cause of all kind of trouble from
> unpredictable behavior to memory leaks to the error you see.
>
> Moreover threads in Python just make the application slower because the
> interpreter is not parallel. In this case you would not gain anything in
> concurrency either.
>
> What are you trying to achieve? I think you need to use the scheduler or a
> background process.
>
>
> On Friday, 18 May 2012 08:44:30 UTC-5, Jan Rozhon wrote:
>>
>> Sure, here is part of my model. Jan
>>
>> # coding: utf8
>>
>> import re
>> import ast
>> import applications.SIP_Tester.modules.functions as functions
>> import os
>> import shutil
>> import subprocess
>> import threading
>> import time
>> from datetime import datetime
>>
>> tb = DAL('sqlite://tb.sqlite') # test database
>> tb.define_table('tests',
>> Field('test_id', 'integer', requires=IS_INT_IN_RANGE(0,
>> 65535), notnull=True, required=True, label='Test ID'),
>> Field('ts', 'datetime',
>> requires=IS_DATETIME(format=T('%Y-%m-%d
>> %H:%M:%S'), error_message='must be YYYY-MM-DD HH:MM:SS!'), label=
>> 'Timestamp'),
>> Field('args', 'text', requires=IS_LENGTH(65536),
>> label='Test
>> Arguments'),
>> format='%(test_id)s')
>> tb.define_table('counts',
>> Field('test_id', tb.tests, ondelete='CASCADE', notnull=
>> True, requires=IS_IN_DB(tb, tb.tests.test_id, '%(test_id)s')),
>> Field('test_pid', 'integer', requires=IS_INT_IN_RANGE(0,
>> 65535), notnull=True, required=True, label='Test PID'),
>> Field('ts', 'datetime',
>> requires=IS_DATETIME(format=T('%Y-%m-%d
>> %H:%M:%S'), error_message='must be YYYY-MM-DD HH:MM:SS!'), label=
>> 'Timestamp'),
>> Field('msg_idx', 'integer', requires=IS_INT_IN_RANGE(0,
>> 10000), label='Message Index'),
>> Field('msg_name', 'string', requires=IS_LENGTH(100),label
>> ='Message Name'),
>> Field('msg_mode', 'string', requires=IS_LENGTH(10),label
>> ='Message Mode'),
>> Field('msg_value', 'integer', requires=IS_INT_IN_RANGE(0
>> ,1e100), label='Message Value'),
>> )
>>
>> counts_tab_fields=('test_id','test_pid','ts', 'msg_idx', 'msg_name',
>> 'msg_mode', 'msg_value') #global table fields tuple to be
>> zipped for w2p inserts
>>
>>
>> class SIP_test(object):
>> def __init__(self, tb_id):
>> self.uas = False
>> self.tb_id = tb_id
>> test_id_row = tb(tb.tests.id==self.tb_id).select(tb.tests.test_id
>> ).first()
>> self.test_id = test_id_row.test_id
>> self.path='applications/SIP_Tester/tests/'+str(self.test_id)
>> os.makedirs(self.path)
>> args_row = tb(tb.tests.id==self.tb_id).select(tb.tests.args).
>> first()
>> args = ast.literal_eval(args_row.args)
>> self.server_ip = args['server_ip']
>> self.uac_scenario = args['uac_scenario']
>> self.rate = args['rate']
>> self.max_calls = args['max_calls']
>> self.sim_calls = args['sim_calls']
>> self.uac_number_count = args['uac_number_count']
>> self.uac_printf_string = args['uac_printf_string']
>> uac_local_ip = args['local_ip_uac']
>> self.uac_local_ip_len, self.uac_local_ip = functions.
>> ip_input_type(uac_local_ip)
>> self.uac_cli_list = self.get_cli_list('uac')
>> if self.uas == True:
>> self.uas_cli_list = self.get_cli_list('uas')
>> ### run processes ###
>> uac_proc_list = self.runproc(self.uac_cli_list)
>> # Some other methods ....
>>
>>
>> class resmon(threading.Thread):
>> def __init__(self, tb_id, path):
>> threading.Thread.__init__(self)
>> self.proclist=session.p
>> self.tb_id = tb_id
>> self.path = path
>> self._stop = False
>> self.counts_index_dict=None
>> test_id_row=tb(tb.tests.id==self.tb_id).select(tb.tests.test_id).
>> first()
>> self.test_id=test_id_row.test_id
>>
>>
>> def stop(self):
>> self._stop = True
>>
>> #Some other methods
>>
>> def read_counts(self,pid,cfile):
>> """Parses counts files so the results can be imported to DB.
>> Sets class counts_index_dict variable"""
>> line=cfile.readline()
>> while line:
>> if re.match('^CurrentTime;.*', line) and not self.
>> counts_index_dict:
>> index_dict={}
>> item_list=line.strip().strip(';').split(';')
>> for item in item_list:
>> index_dict[item_list.index(item)]=item.strip().split(
>> '_')
>> self.counts_index_dict=index_dict
>> elif re.match('^CurrentTime;.*', line) and self.
>> counts_index_dict:pass
>> else:
>> cnt=0
>> ts=0
>> value_list=line.strip().strip(';').split(';')
>> for item in value_list:
>> if cnt==0:
>> print item
>> unix_ts=re.findall('\d+\.\d+', item)[0]
>> ts=datetime.fromtimestamp(float(unix_ts)).
>> strftime('%Y-%m-%d %H:%M:%S')
>> elif cnt!=1:
>> vals_for_insert=tuple([self.test_id, pid, ts] +
>> self.counts_index_dict[cnt] + [item])
>> key_vals=dict(zip(counts_tab_fields,vals_for_insert
>> ))
>> #!!!!!!!!!!!!!!!!!!!!!!! Here it throws an ERROR
>> !!!!!!!!!!!!!!!!!!!
>> tb.counts.insert(**key_vals)
>> #print key_vals
>> cnt+=1
>> line=cfile.readline()
>> time.sleep(0.2)
>> return
>>
>> def run(self):
>> current_file_dict={}
>> known_files=[]
>> while not self._stop:
>> proccount = int(process_count()['cnt'])
>> current_file_dict, known_files=self.getflist(proccount,
>> current_file_dict,known_files)
>> for pid, flist in current_file_dict.iteritems():
>> if flist[0]:
>> self.read_counts(pid,flist[0])
>>
>> if proccount == 0:
>> self.stop()
>> time.sleep(0.5)
>> print 'konec resmonu'
>>
>>
>>
>> Dne pátek, 18. května 2012 15:12:50 UTC+2 Richard napsal(a):
>>>
>>> Maybe showing a bit of code could help!
>>>
>>> Richard
>>>
>>> On Fri, May 18, 2012 at 9:10 AM, Jan Rozhon <jan.roz...@gmail.com>wrote:
>>>
>>>> Hi group,
>>>>
>>>> I am stuck with a database problem in my project and I am unable to
>>>> figure out, what is wrong. Basically, in *models* I define database ("*
>>>> tb*") and two *classes* ("*sip_test*" and "*resmon*"). The first class
>>>> only runs several shell commands, but the second class i overseeing the
>>>> outcome of the first class and therefore I run it in *separate thread*so
>>>> it does not block the whole w2p with its periodic tasks. In this second
>>>> class I am trying to *insert some values into database* ("tb") but I
>>>> get "*Cannot operate on a closed database*." error. If I try to insert
>>>> same values from controller it works fine, but not in any instance of this
>>>> class. Could you point me to direction where the problem could be and
>>>> possibly how to solve it?
>>>>
>>>> Thanks in advance.
>>>>
>>>> PS. I googled several thinks about threads and this db error, but they
>>>> confused me even more, so sorry, if the answer is in there and I dont see
>>>> it.
>>>>
>>>
>>>
