OK. Finally I have it :)
It was only the amount of memory it was consumed each time. I tried it with
a smaller but a significant number of records and finally I figured out
that there is a limit and after that memory consumption is ok. I had only
reach that count of browser request :-D
Sorry for your time and thanks for help. =-<>-=
2012. május 11., péntek 9:30:28 UTC+2 időpontban szimszon a következőt írta:
>
> The problem is not that if I call the controller function if consumes
> memory. My problem is that there is something that still reference
> something after the execution is finished so the consumed memory never get
> released / reused...
>
> 2012. május 10., csütörtök 23:27:32 UTC+2 időpontban Martin.Mulone a
> következőt írta:
>>
>> The problem is fetchall(), is getting all the records to memory.
>> If you want to iterate a large dataset or not so heavy, but have a table
>> with many fields, perhaps you can do it by blocks.
>>
>> For example using raw sql with dal: example: this fetch by 1000 records
>>
>> queryraw = "select * from mytable %(limit_query)s"
>> def records(query):
>> rows = db.executesql(query)
>> if rows:
>> return True
>> else:
>> return False
>>
>> rec = 0
>> stage = 0
>> block = 1000while True:
>> offset = stage * block
>> query = queryraw % {'limitquery': 'LIMIT %s, %s' % (offset, block)}
>> if not records(query):
>> break
>>
>> rows = db.executesql(query)
>> for row in rows:
>> rec += 1
>> # do something
>>
>> stage += 1
>>
>> This is not fast but keep memory low.
>>
>>
>> 2012/5/10 szimszon <szims...@gmail.com>
>>
>>> Could it be related?
>>> https://groups.google.com/forum/#!topic/web2py/hmsupVHdDHo/discussion
>>> (Memory
>>> leak in standalone DAL (issue #731), can you please help test?)
>>>
>>> 2012. május 10., csütörtök 22:32:53 UTC+2 időpontban szimszon a
>>> következőt írta:
>>>
>>>> Okay. It's clear.
>>>>
>>>> I'm only puzzled about why the memory didn't get freed or reused after
>>>> execution is finished. And if I execute the controller function in 1-2min
>>>> interval mem is still not reused.
>>>>
>>>> So I understand it can eat up the memory but why is all memory locked
>>>> forever and didn't get reused - in my understanding (it's not much) in my
>>>> case GC do not free mem at all in python level. So some reference is still
>>>> intact after function is finished.
>>>>
>>>>
>>>> 2012. május 10., csütörtök 21:53:06 UTC+2 időpontban Richard a
>>>> következőt írta:
>>>>>
>>>>> Ok, you don't need it to works all the time.
>>>>>
>>>>> Did you get it to update your records?
>>>>>
>>>>> If not, and if as you said it is a one trip, you can just treat the
>>>>> whole records batch by batch...
>>>>>
>>>>> Look here : http://web2py.com/books/**default/chapter/29/14#**
>>>>> Populating-database-with-**dummy-data<http://web2py.com/books/default/chapter/29/14#Populating-database-with-dummy-data>
>>>>>
>>>>> for i in range(10):
>>>>>
>>>>>
>>>>> populate(db.mytable,100)
>>>>>
>>>>>
>>>>> db.commit()
>>>>>
>>>>>
>>>>> In this example only 100 records are populated at a time between
>>>>> db.commit()... So maybe you just have to wrap for loop that will
>>>>> db.commit() a couples of time during your processing.
>>>>>
>>>>> Richard
>>>>>
>>>>>
>>>>> On Thu, May 10, 2012 at 3:32 PM, szimszon <szims...@gmail.com> wrote:
>>>>>
>>>>>> I had to store files and a lot of properties for it. It was in csv.
>>>>>> But after I processed it we figured out that not all value was correct
>>>>>> in
>>>>>> csv but it was a bit redundant. So I can correct it by go through all
>>>>>> the
>>>>>> records row by row. So that was a one time trip.
>>>>>>
>>>>>> I just realized after the process I had no memory left. So now I'm
>>>>>> investigating what happened...
>>>>>>
>>>>>> 2012. május 10., csütörtök 21:00:05 UTC+2 időpontban Richard a
>>>>>> következőt írta:
>>>>>>>
>>>>>>> Yes but in this case it is not for the entire reecords...
>>>>>>>
>>>>>>> Why would you return a full list of all the records?
>>>>>>>
>>>>>>> I don't understand what is the purpose of listar that you return in
>>>>>>> the view under a html table, why do you need to return all the 100000+
>>>>>>> entries?
>>>>>>>
>>>>>>> Richard
>>>>>>>
>>>>>>> On Thu, May 10, 2012 at 2:56 PM, szimszon <szims...@gmail.com>wrote:
>>>>>>>
>>>>>>>> In book it is a recommended way to iterate over sql results:
>>>>>>>>
>>>>>>>> http://web2py.com/books/**defaul**t/chapter/29/6<http://web2py.com/books/default/chapter/29/6>
>>>>>>>>
>>>>>>>> You can do all the steps in one statement:
>>>>>>>>
>>>>>>>> 1.
>>>>>>>> 2.
>>>>>>>> 3.
>>>>>>>>
>>>>>>>> >>> for row in db(db.person.name=='Alex').sel****ect():
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> print row.name
>>>>>>>>
>>>>>>>>
>>>>>>>> Alex
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> 2012. május 10., csütörtök 20:42:22 UTC+2 időpontban Bruce Wade a
>>>>>>>> következőt írta:
>>>>>>>>
>>>>>>>>> Sorry, you really need to read more about how python works. If you
>>>>>>>>> learn how for loops work and memory you will understand the problem.
>>>>>>>>>
>>>>>>>>> One solution do the query before the for loop then loop through
>>>>>>>>> the objects. This may help at bit. Research xrange vs range
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, May 10, 2012 at 11:30 AM, szimszon <szims...@gmail.com>wrote:
>>>>>>>>>
>>>>>>>>>> Sorry I don't understand. What do you mean "achieve with join"?
>>>>>>>>>>
>>>>>>>>>> There is an empty for loop with db.executesql() without join. And
>>>>>>>>>> it is eating up the memory. :(
>>>>>>>>>>
>>>>>>>>>> 2012. május 10., csütörtök 19:12:30 UTC+2 időpontban Richard a
>>>>>>>>>> következőt írta:
>>>>>>>>>>
>>>>>>>>>>> You can't manage what you want to achieve with join?
>>>>>>>>>>>
>>>>>>>>>>> Richard
>>>>>>>>>>>
>>>>>>>>>>> On Thu, May 10, 2012 at 10:48 AM, szimszon
>>>>>>>>>>> <szims...@gmail.com>wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Sorry for my dumbness but if something is wrong with my code
>>>>>>>>>>>> please point me the right line. I'm not so good in English if it
>>>>>>>>>>>> comes to
>>>>>>>>>>>> "object instance count" and so. Yeah I know I should go and do
>>>>>>>>>>>> some
>>>>>>>>>>>> milkmaid job :) but I'm curious.
>>>>>>>>>>>>
>>>>>>>>>>>> I'm just define some variable:
>>>>>>>>>>>>
>>>>>>>>>>>> lista = list()
>>>>>>>>>>>> last_row = None
>>>>>>>>>>>> next_page_number = 0
>>>>>>>>>>>>
>>>>>>>>>>>> Go in a for loop that just assign the db query result
>>>>>>>>>>>> one-by-one to row variable.
>>>>>>>>>>>>
>>>>>>>>>>>> After that I assign the TABLE() helper to a list variable. That
>>>>>>>>>>>> is it.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 2012. május 10., csütörtök 16:34:49 UTC+2 időpontban Bruce Wade
>>>>>>>>>>>> a következőt írta:
>>>>>>>>>>>>
>>>>>>>>>>>>> Using direct sql query or DAL is going to cause the exact same
>>>>>>>>>>>>> problem in this situation.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, May 10, 2012 at 7:28 AM, szimszon
>>>>>>>>>>>>> <szims...@gmail.com>wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> It's postgres:// in a
>>>>>>>>>>>>>> Version 1.99.7 (2012-04-23 11:26:23) dev of web2py, and
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Python 2.7.3 (default, Apr 20 2012, 22:44:07)
>>>>>>>>>>>>>> [GCC 4.6.3] on linux2
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> python-psycopg2 2.4.5-1
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2012. május 10., csütörtök 15:40:36 UTC+2 időpontban
>>>>>>>>>>>>>> rochacbruno a következőt írta:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Just for curiosity, what happens if you do it in pure sql?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> for row in db.executesql("**sele**********ct * from
>>>>>>>>>>>>>>> file_properties where id > 0"):
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> # do something
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Does it have a lower memory usage?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Thu, May 10, 2012 at 4:14 AM, Bruce Wade
>>>>>>>>>>>>>>> <<bruce.w...@gmail.com>
>>>>>>>>>>>>>>> > wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> for row in db( db.file_**propert**********ies.id > 0 ).
>>>>>>>>>>>>>>>> select(
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Bruno Rocha
>>>>>>>>>>>>>>> [http://rochacbruno.com.br]
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>> Bruce Wade
>>>>>>>>>>>>> http://ca.linkedin.com/in/**bruc********elwade<http://ca.linkedin.com/in/brucelwade>
>>>>>>>>>>>>> http://www.wadecybertech.com
>>>>>>>>>>>>> http://www.fittraineronline.**co********m<http://www.fittraineronline.com>-
>>>>>>>>>>>>> Fitness Personal Trainers Online
>>>>>>>>>>>>> http://www.warplydesigned.com
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> --
>>>>>>>>> Regards,
>>>>>>>>> Bruce Wade
>>>>>>>>> http://ca.linkedin.com/in/**bruc****elwade<http://ca.linkedin.com/in/brucelwade>
>>>>>>>>> http://www.wadecybertech.com
>>>>>>>>> http://www.fittraineronline.**co****m<http://www.fittraineronline.com>-
>>>>>>>>> Fitness Personal Trainers Online
>>>>>>>>> http://www.warplydesigned.com
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>
>>
>>
>> --
>> http://www.tecnodoc.com.ar
>>
>>
