Re: Possible Memory Leak
I don't know about that. Years ago people might have said PHP isn't made for command-line scripting, but then PHP 4.3 came along and gave us a command-line PHP SAPI and it works great. PHP may have been initially designed for serving web pages, but it's a fine general-purpose language that can do most anything you want it to, and it is of great benefit when you can use as few programming languages as possible. Sure, this particular task might benefit from being done in parallel, but why not parallel PHP processes? I am in the process of writing a daemon in PHP, as a CakePHP shell, and I too have the expectation that there will not be memory leaks (though I have not measured it yet). The term isn't quite right, of course -- an interpreted garbage-collected language like PHP can't have memory leaks in the traditional sense (since you're not allocating and deallocating memory yourself) unless there is a bug in the PHP language itself that's causing this, which I kind of doubt given its popularity and age. Rather, I'd assume at this point that CakePHP is caching or logging some information within itself (perhaps collecting all the SQL queries in an array or something), under the assumption that all scripts will be short-lived and it won't matter; if so, CakePHP might want to reconsider that assumption. Or perhaps this is already governed by the debug config setting, or another config setting that we need to discover. On Feb 22, 2011, at 00:19, Walther wrote: This sounds like exactly the sort of task that PHP isn't made for. Right tool for the job and all that. On Feb 22, 5:30 am, Dr. Tarique Sani tariques...@gmail.com wrote: Web servers are simply not designed to have such long single requests The best would be to use shell with short php scripts and some sort of a queue system, which allows you to stop and resume your task, you should also look at parallelizing the task Being a bit presumptive here your simulation looks like a perfect candidate for using map-reduce -- Our newest site for the community: CakePHP Video Tutorials http://tv.cakephp.org Check out the new CakePHP Questions site http://ask.cakephp.org and help others with their CakePHP related questions. To unsubscribe from this group, send email to cake-php+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/cake-php
Re: Possible Memory Leak
On Feb 22, 1:07 am, ProFire profir...@hotmail.com wrote: I use a single http request. Why are you using a http request at all? I'd suggest: * use the cli - a month long http request? you're just asking for trouble * write your process such that it's restartable * daemonize it - a manager process that doesn't do anything except restart your worker process if/when it stops. * OR more generally use a queue system (job x registers jobs x1, x2, x3 ... before it ends) Ryan's guess about the query logging is probably a good place to look (look in dbo_source for cacheMethod), but it doesn't warrant any action from the team - CakePHP might want to reconsider that assumption - the assumption that php/http requests are being used as designed? Come on, which side of the 80/20 rule are batch processes in cake which last longer than a few hours? How hard is it to identify these memory leaks and eliminate them? hth, AD -- Our newest site for the community: CakePHP Video Tutorials http://tv.cakephp.org Check out the new CakePHP Questions site http://ask.cakephp.org and help others with their CakePHP related questions. To unsubscribe from this group, send email to cake-php+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/cake-php
Re: Possible Memory Leak
On Feb 22, 2011, at 02:57, AD7six wrote: On Feb 22, 1:07 am, ProFire profir...@hotmail.com wrote: I use a single http request. Why are you using a http request at all? I'd suggest: * use the cli - a month long http request? you're just asking for trouble * write your process such that it's restartable * daemonize it - a manager process that doesn't do anything except restart your worker process if/when it stops. * OR more generally use a queue system (job x registers jobs x1, x2, x3 ... before it ends) Ryan's guess about the query logging is probably a good place to look (look in dbo_source for cacheMethod), but it doesn't warrant any action from the team - CakePHP might want to reconsider that assumption - the assumption that php/http requests are being used as designed? Come on, which side of the 80/20 rule are batch processes in cake which last longer than a few hours? How hard is it to identify these memory leaks and eliminate them? I had assumed he was using a CakePHP shell script. I agree trying to run a month-long PHP HTTP request is not considered normal and should not really be expected to work. But there's nothing in the design of the PHP language that would make a month-long or longer PHP command line script improper, and I think that usage should be supported, even within CakePHP. The CakePHP book's sole example of what one might want to do with a shell script is to run a report [1]; that's exactly what this thread is about -- though admittedly it's a very large report. I don't know how hard it is to identify and eliminate this unwanted memory usage; I have not attempted to identify it. [1] http://book.cakephp.org/#!/view/1108/Creating-Your-Own-Shells -- Our newest site for the community: CakePHP Video Tutorials http://tv.cakephp.org Check out the new CakePHP Questions site http://ask.cakephp.org and help others with their CakePHP related questions. To unsubscribe from this group, send email to cake-php+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/cake-php
Re: Possible Memory Leak
Hi, I use a single http request. Ah and that is your problem. PHP and web servers in general are not designed to execute long running tasks. I do batch processing in cakephp using a javascript timer that invokes an ajax call to cake every 2 seconds. Cake then executes a small portion of the enormous batch task and logs that. Since this is the type of task you don't want to rush it works very well for me. It is sure is a matter of taste whether or not you think such a solution is pretty and I don't know whether or not it is appropriate in your case. Regards, Maurits On 02/22/2011 01:07 AM, ProFire wrote: I use a single http request. On Feb 21, 11:32 pm, Maurits van der Scheemaur...@vdschee.nl wrote: Hi, Do you use a single http request or multiple invoked by a javascript timer? Regards, Maurits On 02/21/2011 02:49 PM, ProFire wrote: Hi fellowcakephpers, I've been a long time user ofCakePHPand I've been satisfied since I first tried it. In fact, there's no more turning back for me ever since I started it. No framework matchesCakePHPwhen it comes to ease of development. However, very recently, I've encountered a problem I can't figure out the source. I'm dealing with Financial Data and being in finance, my application often have to crunch huge sets of data. I've always been very careful with how my application has handled the data as the data involve is huge and I could run into a memory leak if I don't clear those unused variables. This year, I was tasked to run a very heavy simulation on the financial data that involves possible 100 million mysql queries in a single run. As such, I'm very prepared to let the simulation run over a period of 1 month. However, within 2 days, the application threw a memory exhausted error. What really puzzled me was I had been very careful not to store any unused data in memory. In my algorithm, after I query the data, I store them in a temporary variable. At its final usage, I unset the variable despite knowing that the next iteration the data will be overwritten. That's just to be sure. After each round of simulation, the variables used are stored in the database. After the last $model-save(), I clear every variable used in the simulation, even if the data will be overwritten at the next iteration. All other persistent data throughout the iteration are either integers, floats or unchanging arrays. As such, there's no way these persistent data could be the cause of memory leak. I've debugged as much as I could to pin point the source of the memory leak in my controllers and models, but without any luck. I ran a smaller simulation and monitor the memory size each iteration, I noticed that the memory either stays the same or gets bigger. I still put my faith inCakePHPand I need expert advise on where this memory leak could be. -- Our newest site for the community: CakePHP Video Tutorials http://tv.cakephp.org Check out the new CakePHP Questions site http://ask.cakephp.org and help others with their CakePHP related questions. To unsubscribe from this group, send email to cake-php+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/cake-php
Re: Possible Memory Leak
I totally agree with other guys about demonizing it and making the process being able to pause, maybe restart the computer and contine from where you have paused. So I am writting to express wory about a precision of your results. I am not an old cake user and I haven't had a chance to do complex math functions in cake, but having in mind you have so many results you need to go through a simple 0.0001 mistake could multiply to something huge. Esepecialy if you have numbers with a lot of figures eg 123456789123.123456789123456789 Now in some cases floating numbers here just start to loose last figures and you don't even get that reported about that. I think it's cool you are trying to do this in cake because you will proove that anything is possible but I would without a hesitation check the results in math lab - software which actually is programmed , optimased and tested for such calculations and also does them more quickly. Also I believe - not sure - that you can rent supercomputers with matlab or smth like that like cloud computing and get your job done even quicker. I hope i didn't bother you too much :) All the best, Milos On Feb 22, 7:51 am, Maurits van der Schee maur...@vdschee.nl wrote: Hi, I use a single http request. Ah and that is your problem. PHP and web servers in general are not designed to execute long running tasks. I do batch processing in cakephp using a javascript timer that invokes an ajax call to cake every 2 seconds. Cake then executes a small portion of the enormous batch task and logs that. Since this is the type of task you don't want to rush it works very well for me. It is sure is a matter of taste whether or not you think such a solution is pretty and I don't know whether or not it is appropriate in your case. Regards, Maurits On 02/22/2011 01:07 AM, ProFire wrote: I use a single http request. On Feb 21, 11:32 pm, Maurits van der Scheemaur...@vdschee.nl wrote: Hi, Do you use a single http request or multiple invoked by a javascript timer? Regards, Maurits On 02/21/2011 02:49 PM, ProFire wrote: Hi fellowcakephpers, I've been a long time user ofCakePHPand I've been satisfied since I first tried it. In fact, there's no more turning back for me ever since I started it. No framework matchesCakePHPwhen it comes to ease of development. However, very recently, I've encountered a problem I can't figure out the source. I'm dealing with Financial Data and being in finance, my application often have to crunch huge sets of data. I've always been very careful with how my application has handled the data as the data involve is huge and I could run into a memory leak if I don't clear those unused variables. This year, I was tasked to run a very heavy simulation on the financial data that involves possible 100 million mysql queries in a single run. As such, I'm very prepared to let the simulation run over a period of 1 month. However, within 2 days, the application threw a memory exhausted error. What really puzzled me was I had been very careful not to store any unused data in memory. In my algorithm, after I query the data, I store them in a temporary variable. At its final usage, I unset the variable despite knowing that the next iteration the data will be overwritten. That's just to be sure. After each round of simulation, the variables used are stored in the database. After the last $model-save(), I clear every variable used in the simulation, even if the data will be overwritten at the next iteration. All other persistent data throughout the iteration are either integers, floats or unchanging arrays. As such, there's no way these persistent data could be the cause of memory leak. I've debugged as much as I could to pin point the source of the memory leak in my controllers and models, but without any luck. I ran a smaller simulation and monitor the memory size each iteration, I noticed that the memory either stays the same or gets bigger. I still put my faith inCakePHPand I need expert advise on where this memory leak could be. -- Our newest site for the community: CakePHP Video Tutorials http://tv.cakephp.org Check out the new CakePHP Questions site http://ask.cakephp.org and help others with their CakePHP related questions. To unsubscribe from this group, send email to cake-php+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/cake-php
Re: Possible Memory Leak
Really? You're using PHP--and through a webserver--to do month-long simulation on the financial data (is that the same as Financial Data?) that could result in 100 million DB queries? Pardon me if I'm skeptical of this post. On Mon, Feb 21, 2011 at 8:49 AM, ProFire profir...@hotmail.com wrote: Hi fellow cakephpers, I've been a long time user of CakePHP and I've been satisfied since I first tried it. In fact, there's no more turning back for me ever since I started it. No framework matches CakePHP when it comes to ease of development. However, very recently, I've encountered a problem I can't figure out the source. I'm dealing with Financial Data and being in finance, my application often have to crunch huge sets of data. I've always been very careful with how my application has handled the data as the data involve is huge and I could run into a memory leak if I don't clear those unused variables. This year, I was tasked to run a very heavy simulation on the financial data that involves possible 100 million mysql queries in a single run. As such, I'm very prepared to let the simulation run over a period of 1 month. However, within 2 days, the application threw a memory exhausted error. What really puzzled me was I had been very careful not to store any unused data in memory. In my algorithm, after I query the data, I store them in a temporary variable. At its final usage, I unset the variable despite knowing that the next iteration the data will be overwritten. That's just to be sure. After each round of simulation, the variables used are stored in the database. After the last $model-save(), I clear every variable used in the simulation, even if the data will be overwritten at the next iteration. All other persistent data throughout the iteration are either integers, floats or unchanging arrays. As such, there's no way these persistent data could be the cause of memory leak. I've debugged as much as I could to pin point the source of the memory leak in my controllers and models, but without any luck. I ran a smaller simulation and monitor the memory size each iteration, I noticed that the memory either stays the same or gets bigger. I still put my faith in CakePHP and I need expert advise on where this memory leak could be. -- Our newest site for the community: CakePHP Video Tutorials http://tv.cakephp.org Check out the new CakePHP Questions site http://ask.cakephp.org and help others with their CakePHP related questions. To unsubscribe from this group, send email to cake-php+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/cake-php -- Our newest site for the community: CakePHP Video Tutorials http://tv.cakephp.org Check out the new CakePHP Questions site http://ask.cakephp.org and help others with their CakePHP related questions. To unsubscribe from this group, send email to cake-php+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/cake-php
Re: Possible Memory Leak
Hi, Do you use a single http request or multiple invoked by a javascript timer? Regards, Maurits On 02/21/2011 02:49 PM, ProFire wrote: Hi fellow cakephpers, I've been a long time user of CakePHP and I've been satisfied since I first tried it. In fact, there's no more turning back for me ever since I started it. No framework matches CakePHP when it comes to ease of development. However, very recently, I've encountered a problem I can't figure out the source. I'm dealing with Financial Data and being in finance, my application often have to crunch huge sets of data. I've always been very careful with how my application has handled the data as the data involve is huge and I could run into a memory leak if I don't clear those unused variables. This year, I was tasked to run a very heavy simulation on the financial data that involves possible 100 million mysql queries in a single run. As such, I'm very prepared to let the simulation run over a period of 1 month. However, within 2 days, the application threw a memory exhausted error. What really puzzled me was I had been very careful not to store any unused data in memory. In my algorithm, after I query the data, I store them in a temporary variable. At its final usage, I unset the variable despite knowing that the next iteration the data will be overwritten. That's just to be sure. After each round of simulation, the variables used are stored in the database. After the last $model-save(), I clear every variable used in the simulation, even if the data will be overwritten at the next iteration. All other persistent data throughout the iteration are either integers, floats or unchanging arrays. As such, there's no way these persistent data could be the cause of memory leak. I've debugged as much as I could to pin point the source of the memory leak in my controllers and models, but without any luck. I ran a smaller simulation and monitor the memory size each iteration, I noticed that the memory either stays the same or gets bigger. I still put my faith in CakePHP and I need expert advise on where this memory leak could be. -- Our newest site for the community: CakePHP Video Tutorials http://tv.cakephp.org Check out the new CakePHP Questions site http://ask.cakephp.org and help others with their CakePHP related questions. To unsubscribe from this group, send email to cake-php+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/cake-php
Re: Possible Memory Leak
I use a single http request. On Feb 21, 11:32 pm, Maurits van der Schee maur...@vdschee.nl wrote: Hi, Do you use a single http request or multiple invoked by a javascript timer? Regards, Maurits On 02/21/2011 02:49 PM, ProFire wrote: Hi fellowcakephpers, I've been a long time user ofCakePHPand I've been satisfied since I first tried it. In fact, there's no more turning back for me ever since I started it. No framework matchesCakePHPwhen it comes to ease of development. However, very recently, I've encountered a problem I can't figure out the source. I'm dealing with Financial Data and being in finance, my application often have to crunch huge sets of data. I've always been very careful with how my application has handled the data as the data involve is huge and I could run into a memory leak if I don't clear those unused variables. This year, I was tasked to run a very heavy simulation on the financial data that involves possible 100 million mysql queries in a single run. As such, I'm very prepared to let the simulation run over a period of 1 month. However, within 2 days, the application threw a memory exhausted error. What really puzzled me was I had been very careful not to store any unused data in memory. In my algorithm, after I query the data, I store them in a temporary variable. At its final usage, I unset the variable despite knowing that the next iteration the data will be overwritten. That's just to be sure. After each round of simulation, the variables used are stored in the database. After the last $model-save(), I clear every variable used in the simulation, even if the data will be overwritten at the next iteration. All other persistent data throughout the iteration are either integers, floats or unchanging arrays. As such, there's no way these persistent data could be the cause of memory leak. I've debugged as much as I could to pin point the source of the memory leak in my controllers and models, but without any luck. I ran a smaller simulation and monitor the memory size each iteration, I noticed that the memory either stays the same or gets bigger. I still put my faith inCakePHPand I need expert advise on where this memory leak could be. -- Our newest site for the community: CakePHP Video Tutorials http://tv.cakephp.org Check out the new CakePHP Questions site http://ask.cakephp.org and help others with their CakePHP related questions. To unsubscribe from this group, send email to cake-php+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/cake-php
Re: Possible Memory Leak
Web servers are simply not designed to have such long single requests The best would be to use shell with short php scripts and some sort of a queue system, which allows you to stop and resume your task, you should also look at parallelizing the task Being a bit presumptive here your simulation looks like a perfect candidate for using map-reduce HTH Tarique On Tue, Feb 22, 2011 at 5:37 AM, ProFire profir...@hotmail.com wrote: I use a single http request. On Feb 21, 11:32 pm, Maurits van der Schee maur...@vdschee.nl wrote: Hi, Do you use a single http request or multiple invoked by a javascript timer? Regards, Maurits On 02/21/2011 02:49 PM, ProFire wrote: Hi fellowcakephpers, I've been a long time user ofCakePHPand I've been satisfied since I first tried it. In fact, there's no more turning back for me ever since I started it. No framework matchesCakePHPwhen it comes to ease of development. However, very recently, I've encountered a problem I can't figure out the source. I'm dealing with Financial Data and being in finance, my application often have to crunch huge sets of data. I've always been very careful with how my application has handled the data as the data involve is huge and I could run into a memory leak if I don't clear those unused variables. This year, I was tasked to run a very heavy simulation on the financial data that involves possible 100 million mysql queries in a single run. As such, I'm very prepared to let the simulation run over a period of 1 month. However, within 2 days, the application threw a memory exhausted error. What really puzzled me was I had been very careful not to store any unused data in memory. In my algorithm, after I query the data, I store them in a temporary variable. At its final usage, I unset the variable despite knowing that the next iteration the data will be overwritten. That's just to be sure. After each round of simulation, the variables used are stored in the database. After the last $model-save(), I clear every variable used in the simulation, even if the data will be overwritten at the next iteration. All other persistent data throughout the iteration are either integers, floats or unchanging arrays. As such, there's no way these persistent data could be the cause of memory leak. I've debugged as much as I could to pin point the source of the memory leak in my controllers and models, but without any luck. I ran a smaller simulation and monitor the memory size each iteration, I noticed that the memory either stays the same or gets bigger. I still put my faith inCakePHPand I need expert advise on where this memory leak could be. -- Our newest site for the community: CakePHP Video Tutorials http://tv.cakephp.org Check out the new CakePHP Questions site http://ask.cakephp.org and help others with their CakePHP related questions. To unsubscribe from this group, send email to cake-php+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/cake-php -- = PHP for E-Biz: http://sanisoft.com = -- Our newest site for the community: CakePHP Video Tutorials http://tv.cakephp.org Check out the new CakePHP Questions site http://ask.cakephp.org and help others with their CakePHP related questions. To unsubscribe from this group, send email to cake-php+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/cake-php
Re: Possible Memory Leak
Maybe write it as a console script? Or find some way to batch through the requests using multiple http requests. On Feb 21, 7:30 pm, Dr. Tarique Sani tariques...@gmail.com wrote: Web servers are simply not designed to have such long single requests The best would be to use shell with short php scripts and some sort of a queue system, which allows you to stop and resume your task, you should also look at parallelizing the task Being a bit presumptive here your simulation looks like a perfect candidate for using map-reduce HTH Tarique On Tue, Feb 22, 2011 at 5:37 AM, ProFire profir...@hotmail.com wrote: I use a single http request. On Feb 21, 11:32 pm, Maurits van der Schee maur...@vdschee.nl wrote: Hi, Do you use a single http request or multiple invoked by a javascript timer? Regards, Maurits On 02/21/2011 02:49 PM, ProFire wrote: Hi fellowcakephpers, I've been a long time user ofCakePHPand I've been satisfied since I first tried it. In fact, there's no more turning back for me ever since I started it. No framework matchesCakePHPwhen it comes to ease of development. However, very recently, I've encountered a problem I can't figure out the source. I'm dealing with Financial Data and being in finance, my application often have to crunch huge sets of data. I've always been very careful with how my application has handled the data as the data involve is huge and I could run into a memory leak if I don't clear those unused variables. This year, I was tasked to run a very heavy simulation on the financial data that involves possible 100 million mysql queries in a single run. As such, I'm very prepared to let the simulation run over a period of 1 month. However, within 2 days, the application threw a memory exhausted error. What really puzzled me was I had been very careful not to store any unused data in memory. In my algorithm, after I query the data, I store them in a temporary variable. At its final usage, I unset the variable despite knowing that the next iteration the data will be overwritten. That's just to be sure. After each round of simulation, the variables used are stored in the database. After the last $model-save(), I clear every variable used in the simulation, even if the data will be overwritten at the next iteration. All other persistent data throughout the iteration are either integers, floats or unchanging arrays. As such, there's no way these persistent data could be the cause of memory leak. I've debugged as much as I could to pin point the source of the memory leak in my controllers and models, but without any luck. I ran a smaller simulation and monitor the memory size each iteration, I noticed that the memory either stays the same or gets bigger. I still put my faith inCakePHPand I need expert advise on where this memory leak could be. -- Our newest site for the community: CakePHP Video Tutorials http://tv.cakephp.org Check out the new CakePHP Questions sitehttp://ask.cakephp.organd help others with their CakePHP related questions. To unsubscribe from this group, send email to cake-php+unsubscr...@googlegroups.com For more options, visit this group athttp://groups.google.com/group/cake-php -- = PHP for E-Biz:http://sanisoft.com = -- Our newest site for the community: CakePHP Video Tutorials http://tv.cakephp.org Check out the new CakePHP Questions site http://ask.cakephp.org and help others with their CakePHP related questions. To unsubscribe from this group, send email to cake-php+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/cake-php
Re: Possible Memory Leak
Oops - Tarique beat me to it. On Feb 21, 7:30 pm, Dr. Tarique Sani tariques...@gmail.com wrote: Web servers are simply not designed to have such long single requests The best would be to use shell with short php scripts and some sort of a queue system, which allows you to stop and resume your task, you should also look at parallelizing the task Being a bit presumptive here your simulation looks like a perfect candidate for using map-reduce HTH Tarique On Tue, Feb 22, 2011 at 5:37 AM, ProFire profir...@hotmail.com wrote: I use a single http request. On Feb 21, 11:32 pm, Maurits van der Schee maur...@vdschee.nl wrote: Hi, Do you use a single http request or multiple invoked by a javascript timer? Regards, Maurits On 02/21/2011 02:49 PM, ProFire wrote: Hi fellowcakephpers, I've been a long time user ofCakePHPand I've been satisfied since I first tried it. In fact, there's no more turning back for me ever since I started it. No framework matchesCakePHPwhen it comes to ease of development. However, very recently, I've encountered a problem I can't figure out the source. I'm dealing with Financial Data and being in finance, my application often have to crunch huge sets of data. I've always been very careful with how my application has handled the data as the data involve is huge and I could run into a memory leak if I don't clear those unused variables. This year, I was tasked to run a very heavy simulation on the financial data that involves possible 100 million mysql queries in a single run. As such, I'm very prepared to let the simulation run over a period of 1 month. However, within 2 days, the application threw a memory exhausted error. What really puzzled me was I had been very careful not to store any unused data in memory. In my algorithm, after I query the data, I store them in a temporary variable. At its final usage, I unset the variable despite knowing that the next iteration the data will be overwritten. That's just to be sure. After each round of simulation, the variables used are stored in the database. After the last $model-save(), I clear every variable used in the simulation, even if the data will be overwritten at the next iteration. All other persistent data throughout the iteration are either integers, floats or unchanging arrays. As such, there's no way these persistent data could be the cause of memory leak. I've debugged as much as I could to pin point the source of the memory leak in my controllers and models, but without any luck. I ran a smaller simulation and monitor the memory size each iteration, I noticed that the memory either stays the same or gets bigger. I still put my faith inCakePHPand I need expert advise on where this memory leak could be. -- Our newest site for the community: CakePHP Video Tutorials http://tv.cakephp.org Check out the new CakePHP Questions sitehttp://ask.cakephp.organd help others with their CakePHP related questions. To unsubscribe from this group, send email to cake-php+unsubscr...@googlegroups.com For more options, visit this group athttp://groups.google.com/group/cake-php -- = PHP for E-Biz:http://sanisoft.com = -- Our newest site for the community: CakePHP Video Tutorials http://tv.cakephp.org Check out the new CakePHP Questions site http://ask.cakephp.org and help others with their CakePHP related questions. To unsubscribe from this group, send email to cake-php+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/cake-php
Re: Possible Memory Leak
This sounds like exactly the sort of task that PHP isn't made for. Right tool for the job and all that. On Feb 22, 5:30 am, Dr. Tarique Sani tariques...@gmail.com wrote: Web servers are simply not designed to have such long single requests The best would be to use shell with short php scripts and some sort of a queue system, which allows you to stop and resume your task, you should also look at parallelizing the task Being a bit presumptive here your simulation looks like a perfect candidate for using map-reduce HTH Tarique On Tue, Feb 22, 2011 at 5:37 AM, ProFire profir...@hotmail.com wrote: I use a single http request. On Feb 21, 11:32 pm, Maurits van der Schee maur...@vdschee.nl wrote: Hi, Do you use a single http request or multiple invoked by a javascript timer? Regards, Maurits On 02/21/2011 02:49 PM, ProFire wrote: Hi fellowcakephpers, I've been a long time user ofCakePHPand I've been satisfied since I first tried it. In fact, there's no more turning back for me ever since I started it. No framework matchesCakePHPwhen it comes to ease of development. However, very recently, I've encountered a problem I can't figure out the source. I'm dealing with Financial Data and being in finance, my application often have to crunch huge sets of data. I've always been very careful with how my application has handled the data as the data involve is huge and I could run into a memory leak if I don't clear those unused variables. This year, I was tasked to run a very heavy simulation on the financial data that involves possible 100 million mysql queries in a single run. As such, I'm very prepared to let the simulation run over a period of 1 month. However, within 2 days, the application threw a memory exhausted error. What really puzzled me was I had been very careful not to store any unused data in memory. In my algorithm, after I query the data, I store them in a temporary variable. At its final usage, I unset the variable despite knowing that the next iteration the data will be overwritten. That's just to be sure. After each round of simulation, the variables used are stored in the database. After the last $model-save(), I clear every variable used in the simulation, even if the data will be overwritten at the next iteration. All other persistent data throughout the iteration are either integers, floats or unchanging arrays. As such, there's no way these persistent data could be the cause of memory leak. I've debugged as much as I could to pin point the source of the memory leak in my controllers and models, but without any luck. I ran a smaller simulation and monitor the memory size each iteration, I noticed that the memory either stays the same or gets bigger. I still put my faith inCakePHPand I need expert advise on where this memory leak could be. -- Our newest site for the community: CakePHP Video Tutorials http://tv.cakephp.org Check out the new CakePHP Questions sitehttp://ask.cakephp.organd help others with their CakePHP related questions. To unsubscribe from this group, send email to cake-php+unsubscr...@googlegroups.com For more options, visit this group athttp://groups.google.com/group/cake-php -- = PHP for E-Biz:http://sanisoft.com = -- Our newest site for the community: CakePHP Video Tutorials http://tv.cakephp.org Check out the new CakePHP Questions site http://ask.cakephp.org and help others with their CakePHP related questions. To unsubscribe from this group, send email to cake-php+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/cake-php
