Re: [PHP] memory leak when referencing associative arrays
Hi Jochem, Thank you very much for your reply! Your reply lend me this morning to had a look again at the code, and I have solved more than half of the trouble caused by a leak on a PhpGtk function! :-) Sorry for the more long reply than it should be, but I hope you'll have the time to read it. Unfortunately, I have not fount an easyier way to explain the thing and get to something with meaning. Perhaps the more comprimising memory leak happened when my application is active, receiving and processing the data received from one or several devices in real-time -well, it does a 300ms. delay until the main next loop iteration starts-, and this trouble was limiting a lot the amount of time the app. could be running before the OS have to start swapping pages to disk, caused by the memory being leaked on each iteration, -of course, it depends on the size or number of the incoming data per minute, etc. etc.-, in wich situation I have to finish the app. and restart it again for clean state, but in a nomal case use test, it was consuming about a megabite each minute or two, when each round is completed and data is received and processed from all devices. The improvement it is as simpler as not using styles for every node insert as some GtkCtrees are populated when receiving and processing data) and now it only takes about 300kb when before was taking one meg. It is a only a PHP-Gtk issue as the leak is produced when copying a widget style, a must if you want to change the style of the nodes being inserted-updated, and not doing this helped a lot (I was doing this about 200 times (one time per node) in minute or two wich lead to a high amount of mem leaked...) I really did not expect to find such a leak on a simple $style = $astyle-copy() where $astyle = gtk::rc_get_style($ctree); !!-. Anyway a problem persist, as 300kb per minute or two are a lot of kilobytes if you though in that time I only add about 100-125 items and update another 50 (each item have from 50 to 200 bytes of data- to a pair of global associative arrays. And I really don't know exactly where it is leaking, I don't know if it leaks on every iteration, or maybe the amount of memory used while the application runs is what it should, but then later when you stop the actual processing, save the data, and want to start a new run -all variables are unset-, the memory used stays there... and rewriting to the same global variables with -probably- the same keys -but with different data- when receiving data and processing from the devices again, does not reuse the previouly reserved memory... it continues eating more and more. Or if you load previously saved data from disk -it only saves the incoming data and all items are re-calculated always- the same happens, more and more memory used and nothing is reused or freed. And if I do some runs, or load some data files, the application gets so many megs. and I need to restart it. I have no circular references on the arrays, and not a lot of references at all. And removing some caused the app. to get more memory than before, as I experienced some months ago. Only some associative arrays being referenced with or just generated and processed from within three objects. I though I could ask for some help regards the garbage collector, so I see better what is happening. I haven't got yet the time to look at the sources and try to understand how does it work and that... :-) it could be really a pain. And I can't update to a newer php version... as I think it could help. But on the other way, the actual trouble was almost solved some months ago when I did some tests -very succesful I'll say regards the not reused memory between application runs, so the memory used in a run was used again for the new data the next time the receiving/processing started, or when loading saved data-, but as it didn't help much in the memory used while processing, that not helped much for the main trouble, and felt lost 'cause though it was likely the same issue and could not improve more the code. I achieved that by refactoring a big piece of code and doing all the main processing of global arrays from some plain functions, outside the objects where it was/is done, but sadly that lend to some bugs -i did it in hurry- and I had not the time to workaround them, and have to get back to the previous version to continue developement with more stable code. Now that I have found where the most memory was being leaked when the application is active receiving data -that was really the more compromising thing- I'll have to find the time to refactor carefully the main processing loop to plain functions -a 170kb. code class- and not get caught by bugs... again :-) Just a matter of processing a lot of global data within objects ? Best Regards, Gonzalo. Jochem Maas escribió: Gonzalo MC wrote: Hi all, I'm having some pain with this process. I'll try
Re: [PHP] memory leak when referencing associative arrays
Hi Jochem, Thank you very much for your reply! Your reply lend me this morning to had a look again at the code, and I have solved more than half of the trouble caused by a leak on a PhpGtk function! :-) Sorry for the more long reply than it should be, but I hope you'll have the time to read it. Unfortunately, I have not fount an easyier way to explain the thing and get to something with meaning. Perhaps the more comprimising memory leak happened when my application is active, receiving and processing the data received from one or several devices in real-time -well, it does a 300ms. delay until the main next loop iteration starts-, and this trouble was limiting a lot the amount of time the app. could be running before the OS have to start swapping pages to disk, caused by the memory being leaked on each iteration, -of course, it depends on the size or number of the incoming data per minute, etc. etc.-, in wich situation I have to finish the app. and restart it again for clean state, but in a nomal case use test, it was consuming about a megabite each minute or two, when each round is completed and data is received and processed from all devices. The improvement it is as simpler as not using styles for every node insert as some GtkCtrees are populated when receiving and processing data) and now it only takes about 300kb when before was taking one meg. It is a only a PHP-Gtk issue as the leak is produced when copying a widget style, a must if you want to change the style of the nodes being inserted-updated, and not doing this helped a lot (I was doing this about 200 times (one time per node) in minute or two wich lead to a high amount of mem leaked...) I really did not expect to find such a leak on a simple $style = $astyle-copy() where $astyle = gtk::rc_get_style($ctree); !!-. Anyway a problem persist, as 300kb per minute or two are a lot of kilobytes if you though in that time I only add about 100-125 items and update another 50 (each item have from 50 to 200 bytes of data- to a pair of global associative arrays. And I really don't know exactly where it is leaking, I don't know if it leaks on every iteration, or maybe the amount of memory used while the application runs is what it should, but then later when you stop the actual processing, save the data, and want to start a new run -all variables are unset-, the memory used stays there... and rewriting to the same global variables with -probably- the same keys -but with different data- when receiving data and processing from the devices again, does not reuse the previouly reserved memory... it continues eating more and more. Or if you load previously saved data from disk -it only saves the incoming data and all items are re-calculated always- the same happens, more and more memory used and nothing is reused or freed. And if I do some runs, or load some data files, the application gets so many megs. and I need to restart it. I have no circular references on the arrays, and not a lot of references at all. And removing some caused the app. to get more memory than before, as I experienced some months ago. Only some associative arrays being referenced with or just generated and processed from within three objects. I though I could ask for some help regards the garbage collector, so I see better what is happening. I haven't got yet the time to look at the sources and try to understand how does it work and that... :-) it could be really a pain. And I can't update to a newer php version... as I think it could help. But on the other way, the actual trouble was almost solved some months ago when I did some tests -very succesful I'll say regards the not reused memory between application runs, so the memory used in a run was used again for the new data the next time the receiving/processing started, or when loading saved data-, but as it didn't help much in the memory used while processing, that not helped much for the main trouble, and felt lost 'cause though it was likely the same issue and could not improve more the code. I achieved that by refactoring a big piece of code and doing all the main processing of global arrays from some plain functions, outside the objects where it was/is done, but sadly that lend to some bugs -i did it in hurry- and I had not the time to workaround them, and have to get back to the previous version to continue developement with more stable code. Now that I have found where the most memory was being leaked when the application is active receiving data -that was really the more compromising thing- I'll have to find the time to refactor carefully the main processing loop to plain functions -a 170kb. code class- and not get caught by bugs... again :-) Just a matter of processing a lot of global data within objects ? Best Regards, Gonzalo. Jochem Maas escribió: Gonzalo MC wrote: Hi all, I'm having some pain with this process. I'll try
Re: [PHP] memory leak when referencing associative arrays
Jochem Maas escribió: Gonzalo MC wrote: Hi Jochem, Thank you very much for your reply! no problem :-) Your reply lend me this morning to had a look again at the code, and I have solved more than half of the trouble caused by a leak on a PhpGtk function! :-) Sorry for the more long reply than it should be, but I hope you'll have the time to read it. Unfortunately, I have not fount an easyier way to explain the thing and get to something with meaning. well I read it, but I didn't understand all of it. not to worry. as a sidenote - reading about the objects (some gtk style wotsit) in your codebase I got the impression that the unfreed memory is possibly [mostly] to do with those - my memory usage problems are always to do with objects (remember in php that object variables basically work semantically like scalar values as opposed to php5 where object variables are handles to objectsa and effectively[, always,] work like [php] references of normal variables) I'd concentrate on getting those objects 'tuned' - generous but highly controlled use of the '' symbol to force objects to be passed by reference where possible/relevant (e.g. when using the 'new MyObject;' syntax) can also help. ... Hi again, Yeah you're right the memory usage problems always have to do with objects on php4, and using '' is of great help, but in my case, with the actual implementation, I've tested almost all the ways in the exact points where data is handled, and fount no reliable way for reusing reserved memory from the interpreter other than going to functional style programming and avoid setting object properties, etc. Fortunately, my application almost does not create any object other than in the beginning/initialization of and for the GUI, for what I have some helper classes to abstract interfaces, but they are all instantiated on the program startup and the only objects I do create on the run time are perhaps the new style copyes according to each new insert or update to the existing widgets on screen, and they are almost all referenced byref using when adding pixmaps and such - note the $astyle = gtk::rc_get_style($ctree); line per example is only done once before all items loop and it does not take any memory, not the case when creating or transfering a widget from a way to another where the really matters- but the trouble is when you call for each node in the loop to the $astyle-copy(); method per example-. All the data I generate, update or read are arrays with numeric or alfanumeric keys, in wich I place another array with some alfanumeric keys. The data is generated by an object. It is stored in global variables, and some references to these are keept on that or other object properties, when processing or updating data based on the last data received. I think the trick was to put these references to the global array items, outside these objects, in the global namespace too, but only worked when the data was processed from plain functions, -actually the processing work is done from a class instance that extends another root class-. Maybe it haves something to do with the processing and gui update loops, fired by gtk_timeout's from inside objects calling object methods... Maybe on refactoring, now I have more knowledge about whats going on, I will notice of. I expect to get soon the time needed to refactor again this and I'll post the concrete workaround for my case, perhaps could help somebody to get a more clear idea of ways to avoid these memory usage issues on php4 / PhpGtk1. Until then, I'll though a good plan so when refactoring going in stages of migration so I could really find the real nature of the issue -object properties are references to some globals / processing of globals fired by gtk_timeouts calling object methods by reference / whatever it could be-. And thank you again for your response. I know perhaps it does not matter much but... :-) thats it. Best Regards, Gonzalo. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] memory leak when referencing associative arrays
Gonzalo MC escribió: Jochem Maas escribió: Gonzalo MC wrote: Hi Jochem, Thank you very much for your reply! no problem :-) Your reply lend me this morning to had a look again at the code, and I have solved more than half of the trouble caused by a leak on a PhpGtk function! :-) Sorry for the more long reply than it should be, but I hope you'll have the time to read it. Unfortunately, I have not fount an easyier way to explain the thing and get to something with meaning. well I read it, but I didn't understand all of it. not to worry. as a sidenote - reading about the objects (some gtk style wotsit) in your codebase I got the impression that the unfreed memory is possibly [mostly] to do with those - my memory usage problems are always to do with objects (remember in php that object variables basically work semantically like scalar values as opposed to php5 where object variables are handles to objectsa and effectively[, always,] work like [php] references of normal variables) I'd concentrate on getting those objects 'tuned' - generous but highly controlled use of the '' symbol to force objects to be passed by reference where possible/relevant (e.g. when using the 'new MyObject;' syntax) can also help. ... Hi again, Yeah you're right the memory usage problems always have to do with objects on php4, and using '' is of great help, but in my case, with the actual implementation, I've tested almost all the ways in the exact points where data is handled, and fount no reliable way for reusing reserved memory from the interpreter other than going to functional style programming and avoid setting object properties, etc. Fortunately, my application almost does not create any object other than in the beginning/initialization of and for the GUI, for what I have some helper classes to abstract interfaces, but they are all instantiated on the program startup and the only objects I do create on the run time are perhaps the new style copyes according to each new insert or update to the existing widgets on screen, and they are almost all referenced byref using when adding pixmaps and such - note the $astyle = gtk::rc_get_style($ctree); line per example is only done once before all items loop and it does not take any memory, not the case when creating or transfering a widget from a way to another where the really matters- but the trouble is when you call for each node in the loop to the $astyle-copy(); method per example-. All the data I generate, update or read are arrays with numeric or alfanumeric keys, in wich I place another array with some alfanumeric keys. The data is generated by an object. It is stored in global variables, and some references to these are keept on that or other object properties, when processing or updating data based on the last data received. I think the trick was to put these references to the global array items, outside these objects, in the global namespace too, but only worked when the data was processed from plain functions, -actually the processing work is done from a class instance that extends another root class-. Maybe it haves something to do with the processing and gui update loops, fired by gtk_timeout's from inside objects calling object methods... Maybe on refactoring, now I have more knowledge about whats going on, I will notice of. I expect to get soon the time needed to refactor again this and I'll post the concrete workaround for my case, perhaps could help somebody to get a more clear idea of ways to avoid these memory usage issues on php4 / PhpGtk1. Until then, I'll though a good plan so when refactoring going in stages of migration so I could really find the real nature of the issue -object properties are references to some globals / processing of globals fired by gtk_timeouts calling object methods by reference / whatever it could be-. And thank you again for your response. I know perhaps it does not matter much but... :-) thats it. Best Regards, Gonzalo. Sorry but I forget to say that when loading data from a file, -so a big process is done without updating the GUI where is the only place objects are perhaps more relevant- the program gets a lots of megs of ram, only with the processing of the global arrays from within an object... So it has to do with the processing of that data from an object. For a moment I though the data I'd willing to reuse is maybe leak in some place... not in the process itself, but on the other way I do remember clearly how I did get improved this part of the process -loading from file, processing, and finally do a GUI update with results, and do it again with another file, and have only a small amount extra of memory used- while doing it from an object lends to a lot of memory not being reused -or being really leak-. Regards, Gonzalo -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
[PHP] memory leak when referencing associative arrays
Hi all, I'm having some pain with this process. I'll try to explain it well, my english is little bad... :-) I've got two main loops, handled by timeouts. One loop calls to an object method every half second to do a computation. It reads from external device, creates helper arrays for calculations, and then creates/updates another array where the final calculations are stored. In the other loop, fired by events or some circunstances, I loop thru the calculations array and get the new state for some values. This is not done on the same object. I'm having some memory leak trouble while doing that from within objects using global variables to store the associative arraya, so it seems to me that some references are keep by php to these data I do generate and destroy or update on each computation loop, and that memory it is not freed anytime until program execution ends. If I do the same computation, but I don't do the data lookups to the associative arrays, the memory is normaly fred as expected -I noticed the same trouble with this that depends on how the arrays are handled or accesed before deleting them, when trying to test for that problem-. But if I do the array lookups (i mean with this to do a loop to the array to get all item updates ...) to data is being processed / updated from other objects, the trouble I explained begin... It is somewhat complex to give an example to reproduce the whole thing... Notice I use php4.4.1, and sadly, it is a long running script as it handles some phpgtk interface. I noticed the same results when converting these global variables to local object properties, sadly. It is possible on PHP4 to call on runtime to the garbage collector? possibly via a php extension / C code? Does the php garbage collector keep track of uncollectable objects / zvals as python gc does? It is some way to freed some memory I know it should be really unreferenced? Some thing like a manual force to the garbage collection of a zval -associative array at php level-? Perhaps should I use a different mechanism for storing the associative arrays so I could not suffer this issue? If my trouble is not very clear, please don't hesitate to ask me. Thanks in advance. Regards, Gonzalo. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
