Could it be related? https://groups.google.com/forum/#!topic/web2py/hmsupVHdDHo/discussion (Memory leak in standalone DAL (issue #731), can you please help test?)
2012. május 10., csütörtök 22:32:53 UTC+2 időpontban szimszon a következőt írta: > > Okay. It's clear. > > I'm only puzzled about why the memory didn't get freed or reused after > execution is finished. And if I execute the controller function in 1-2min > interval mem is still not reused. > > So I understand it can eat up the memory but why is all memory locked > forever and didn't get reused - in my understanding (it's not much) in my > case GC do not free mem at all in python level. So some reference is still > intact after function is finished. > > > 2012. május 10., csütörtök 21:53:06 UTC+2 időpontban Richard a következőt > írta: >> >> Ok, you don't need it to works all the time. >> >> Did you get it to update your records? >> >> If not, and if as you said it is a one trip, you can just treat the whole >> records batch by batch... >> >> Look here : >> http://web2py.com/books/default/chapter/29/14#Populating-database-with-dummy-data >> >> for i in range(10): >> >> populate(db.mytable,100) >> >> db.commit() >> >> >> In this example only 100 records are populated at a time between >> db.commit()... So maybe you just have to wrap for loop that will >> db.commit() a couples of time during your processing. >> >> Richard >> >> >> On Thu, May 10, 2012 at 3:32 PM, szimszon <szims...@gmail.com> wrote: >> >>> I had to store files and a lot of properties for it. It was in csv. But >>> after I processed it we figured out that not all value was correct in csv >>> but it was a bit redundant. So I can correct it by go through all the >>> records row by row. So that was a one time trip. >>> >>> I just realized after the process I had no memory left. So now I'm >>> investigating what happened... >>> >>> 2012. május 10., csütörtök 21:00:05 UTC+2 időpontban Richard a >>> következőt írta: >>>> >>>> Yes but in this case it is not for the entire reecords... >>>> >>>> Why would you return a full list of all the records? >>>> >>>> I don't understand what is the purpose of listar that you return in the >>>> view under a html table, why do you need to return all the 100000+ entries? >>>> >>>> Richard >>>> >>>> On Thu, May 10, 2012 at 2:56 PM, szimszon <szims...@gmail.com> wrote: >>>> >>>>> In book it is a recommended way to iterate over sql results: >>>>> >>>>> http://web2py.com/books/**default/chapter/29/6<http://web2py.com/books/default/chapter/29/6> >>>>> >>>>> You can do all the steps in one statement: >>>>> >>>>> 1. >>>>> 2. >>>>> 3. >>>>> >>>>> >>> for row in db(db.person.name=='Alex').sel**ect(): >>>>> >>>>> >>>>> print row.name >>>>> >>>>> Alex >>>>> >>>>> >>>>> >>>>> >>>>> 2012. május 10., csütörtök 20:42:22 UTC+2 időpontban Bruce Wade a >>>>> következőt írta: >>>>> >>>>>> Sorry, you really need to read more about how python works. If you >>>>>> learn how for loops work and memory you will understand the problem. >>>>>> >>>>>> One solution do the query before the for loop then loop through the >>>>>> objects. This may help at bit. Research xrange vs range >>>>>> >>>>>> >>>>>> On Thu, May 10, 2012 at 11:30 AM, szimszon <szims...@gmail.com>wrote: >>>>>> >>>>>>> Sorry I don't understand. What do you mean "achieve with join"? >>>>>>> >>>>>>> There is an empty for loop with db.executesql() without join. And it >>>>>>> is eating up the memory. :( >>>>>>> >>>>>>> 2012. május 10., csütörtök 19:12:30 UTC+2 időpontban Richard a >>>>>>> következőt írta: >>>>>>> >>>>>>>> You can't manage what you want to achieve with join? >>>>>>>> >>>>>>>> Richard >>>>>>>> >>>>>>>> On Thu, May 10, 2012 at 10:48 AM, szimszon <szims...@gmail.com>wrote: >>>>>>>> >>>>>>>>> Sorry for my dumbness but if something is wrong with my code >>>>>>>>> please point me the right line. I'm not so good in English if it >>>>>>>>> comes to >>>>>>>>> "object instance count" and so. Yeah I know I should go and do some >>>>>>>>> milkmaid job :) but I'm curious. >>>>>>>>> >>>>>>>>> I'm just define some variable: >>>>>>>>> >>>>>>>>> lista = list() >>>>>>>>> last_row = None >>>>>>>>> next_page_number = 0 >>>>>>>>> >>>>>>>>> Go in a for loop that just assign the db query result one-by-one >>>>>>>>> to row variable. >>>>>>>>> >>>>>>>>> After that I assign the TABLE() helper to a list variable. That is >>>>>>>>> it. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> 2012. május 10., csütörtök 16:34:49 UTC+2 időpontban Bruce Wade a >>>>>>>>> következőt írta: >>>>>>>>> >>>>>>>>>> Using direct sql query or DAL is going to cause the exact same >>>>>>>>>> problem in this situation. >>>>>>>>>> >>>>>>>>>> On Thu, May 10, 2012 at 7:28 AM, szimszon <szims...@gmail.com>wrote: >>>>>>>>>> >>>>>>>>>>> It's postgres:// in a >>>>>>>>>>> Version 1.99.7 (2012-04-23 11:26:23) dev of web2py, and >>>>>>>>>>> >>>>>>>>>>> Python 2.7.3 (default, Apr 20 2012, 22:44:07) >>>>>>>>>>> [GCC 4.6.3] on linux2 >>>>>>>>>>> >>>>>>>>>>> python-psycopg2 2.4.5-1 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> 2012. május 10., csütörtök 15:40:36 UTC+2 időpontban rochacbruno >>>>>>>>>>> a következőt írta: >>>>>>>>>>> >>>>>>>>>>>> Just for curiosity, what happens if you do it in pure sql? >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> for row in db.executesql("**sele********ct * from >>>>>>>>>>>> file_properties where id > 0"): >>>>>>>>>>>> >>>>>>>>>>>> # do something >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Does it have a lower memory usage? >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Thu, May 10, 2012 at 4:14 AM, Bruce Wade <<bruce.w...@gmail.com> >>>>>>>>>>>> > wrote: >>>>>>>>>>>> >>>>>>>>>>>>> for row in db( db.file_**propert********ies.id > 0 ).select( >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> >>>>>>>>>>>> Bruno Rocha >>>>>>>>>>>> [http://rochacbruno.com.br] >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> -- >>>>>>>>>> Regards, >>>>>>>>>> Bruce Wade >>>>>>>>>> http://ca.linkedin.com/in/**bruc******elwade<http://ca.linkedin.com/in/brucelwade> >>>>>>>>>> http://www.wadecybertech.com >>>>>>>>>> http://www.fittraineronline.**co******m<http://www.fittraineronline.com>- >>>>>>>>>> Fitness Personal Trainers Online >>>>>>>>>> http://www.warplydesigned.com >>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> -- >>>>>> Regards, >>>>>> Bruce Wade >>>>>> http://ca.linkedin.com/in/**bruc**elwade<http://ca.linkedin.com/in/brucelwade> >>>>>> http://www.wadecybertech.com >>>>>> http://www.fittraineronline.**co**m <http://www.fittraineronline.com>- >>>>>> Fitness Personal Trainers Online >>>>>> http://www.warplydesigned.com >>>>>> >>>>>> >>>> >>
