Re: [symfony-users] Re: memory leaks... just keep on leaking
Dennis... create a simple loop of approx 10,000 iterations and create a criteria inside the loop... and echo the php's memory usage... I this the issue is quite obvious... On Mon, Apr 12, 2010 at 4:33 PM, Richard U richard@gmail.com wrote: I am uploading a csv file.. i require looping over each row... the data in each column has to be processed individually and allocated to the right table... On Mon, Apr 12, 2010 at 3:56 PM, Dennis Benkert spinecras...@googlemail.com wrote: Although I am pretty sure you are frustrated there is no reason to show disrespect to other people. Christian's point is that we cannot help you without getting some information about your specific problem. Maybe you can provide some code sample that show us your memory problems. Also it's interesting to know what you want to do with the code (are you inserting thousands of entries, looping over many entries and saving them again, etc.). - Dennis Am 12.04.2010 um 07:41 schrieb Slavka richard@gmail.com: Christian have you just started using symfony? if not then i am sure you are aware of the memory leaks i am talking about... Cheers On Apr 12, 3:30 pm, Christian Schaefer cae...@gmail.com wrote: Hi Slavka, from your tiny description it is not clear whether the leak you experience is a result of Propel or symfony or your usage of them. You should know that symfony 1.2 is no longer supported but replaced by symfony 1.4. The favoured ORM nowadays is Doctrine although Propel is still supported. But most importantly: you posted a complaint rather than a question. This might work with a licenced software from some company but with an open source community a certain netiquette should apply. Cheers /Christian On 12 Apr., 05:51, Slavka richard@gmail.com wrote: It seems to be pretty clear the symfony 1.2 and propel are still full of memory leaks... I have a simple for loop selecting the same object into the same variable... in theory this should not lead to increase memory use for every loop however that is what is happening... Has anyone every looked into this... if so what are options to handle the leaks... , if there is no solution we are going to have to consider just using mysql functions... Regards -- If you want to report a vulnerability issue on symfony, please send it to security at symfony-project.com You received this message because you are subscribed to the Google Groups symfony users group. To post to this group, send email to symfony-users@googlegroups.com To unsubscribe from this group, send email to symfony-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/symfony-users?hl=en To unsubscribe, reply using remove me as the subject. -- If you want to report a vulnerability issue on symfony, please send it to security at symfony-project.com You received this message because you are subscribed to the Google Groups symfony users group. To post to this group, send email to symfony-users@googlegroups.com To unsubscribe from this group, send email to symfony-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/symfony-users?hl=en -- If you want to report a vulnerability issue on symfony, please send it to security at symfony-project.com You received this message because you are subscribed to the Google Groups symfony users group. To post to this group, send email to symfony-users@googlegroups.com To unsubscribe from this group, send email to symfony-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/symfony-users?hl=en
Re: [symfony-users] Re: memory leaks... just keep on leaking
So, how big are these CSV files? How many lines are you processing and can you tell us how many entries you're inserting to your DB using Propel? It would be best if you could show us the code that implements what you described in your last mail. Investing maybe 30 minutes to write an email that desribes the problem you have, what part of your code suffers from it and the data you're using will most likely keep you away from writing 5 or more emails desribing everything. Plus it will stop me asking stupid questions because I have to digg into your problem this way. :) - Dennis 2010/4/12 Richard U richard@gmail.com: I am uploading a csv file.. i require looping over each row... the data in each column has to be processed individually and allocated to the right table... On Mon, Apr 12, 2010 at 3:56 PM, Dennis Benkert spinecras...@googlemail.com wrote: Although I am pretty sure you are frustrated there is no reason to show disrespect to other people. Christian's point is that we cannot help you without getting some information about your specific problem. Maybe you can provide some code sample that show us your memory problems. Also it's interesting to know what you want to do with the code (are you inserting thousands of entries, looping over many entries and saving them again, etc.). - Dennis Am 12.04.2010 um 07:41 schrieb Slavka richard@gmail.com: Christian have you just started using symfony? if not then i am sure you are aware of the memory leaks i am talking about... Cheers On Apr 12, 3:30 pm, Christian Schaefer cae...@gmail.com wrote: Hi Slavka, from your tiny description it is not clear whether the leak you experience is a result of Propel or symfony or your usage of them. You should know that symfony 1.2 is no longer supported but replaced by symfony 1.4. The favoured ORM nowadays is Doctrine although Propel is still supported. But most importantly: you posted a complaint rather than a question. This might work with a licenced software from some company but with an open source community a certain netiquette should apply. Cheers /Christian On 12 Apr., 05:51, Slavka richard@gmail.com wrote: It seems to be pretty clear the symfony 1.2 and propel are still full of memory leaks... I have a simple for loop selecting the same object into the same variable... in theory this should not lead to increase memory use for every loop however that is what is happening... Has anyone every looked into this... if so what are options to handle the leaks... , if there is no solution we are going to have to consider just using mysql functions... Regards -- If you want to report a vulnerability issue on symfony, please send it to security at symfony-project.com You received this message because you are subscribed to the Google Groups symfony users group. To post to this group, send email to symfony-users@googlegroups.com To unsubscribe from this group, send email to symfony-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/symfony-users?hl=en To unsubscribe, reply using remove me as the subject. -- If you want to report a vulnerability issue on symfony, please send it to security at symfony-project.com You received this message because you are subscribed to the Google Groups symfony users group. To post to this group, send email to symfony-users@googlegroups.com To unsubscribe from this group, send email to symfony-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/symfony-users?hl=en -- If you want to report a vulnerability issue on symfony, please send it to security at symfony-project.com You received this message because you are subscribed to the Google Groups symfony users group. To post to this group, send email to symfony-users@googlegroups.com To unsubscribe from this group, send email to symfony-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/symfony-users?hl=en -- If you want to report a vulnerability issue on symfony, please send it to security at symfony-project.com You received this message because you are subscribed to the Google Groups symfony users group. To post to this group, send email to symfony-users@googlegroups.com To unsubscribe from this group, send email to symfony-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/symfony-users?hl=en
Re: [symfony-users] Re: memory leaks... just keep on leaking
Except from the fact that I haven't read your code, what you describe is a problem with PHP versions prior to 5.3 The problem is in the GC of the engine I'm not blaming it on PHP, but you could try that. If you can't upgrade to the new PHP, then I'm sorry, but what you try to do may not work at all. And you can forget about blaming Propel or symfony for the problem. Try to instantiate a big class inside that loop, a class created by your own code and you will see the problem appear. Maybe it can take longer, because Propel classes tend to be quite huge, but that will fail at some point. My 2 cents, Alvaro On Apr 12, 2010, at 3:28 PM, Dennis Benkert wrote: So, how big are these CSV files? How many lines are you processing and can you tell us how many entries you're inserting to your DB using Propel? It would be best if you could show us the code that implements what you described in your last mail. Investing maybe 30 minutes to write an email that desribes the problem you have, what part of your code suffers from it and the data you're using will most likely keep you away from writing 5 or more emails desribing everything. Plus it will stop me asking stupid questions because I have to digg into your problem this way. :) - Dennis 2010/4/12 Richard U richard@gmail.com: I am uploading a csv file.. i require looping over each row... the data in each column has to be processed individually and allocated to the right table... On Mon, Apr 12, 2010 at 3:56 PM, Dennis Benkert spinecras...@googlemail.com wrote: Although I am pretty sure you are frustrated there is no reason to show disrespect to other people. Christian's point is that we cannot help you without getting some information about your specific problem. Maybe you can provide some code sample that show us your memory problems. Also it's interesting to know what you want to do with the code (are you inserting thousands of entries, looping over many entries and saving them again, etc.). - Dennis Am 12.04.2010 um 07:41 schrieb Slavka richard@gmail.com: Christian have you just started using symfony? if not then i am sure you are aware of the memory leaks i am talking about... Cheers On Apr 12, 3:30 pm, Christian Schaefer cae...@gmail.com wrote: Hi Slavka, from your tiny description it is not clear whether the leak you experience is a result of Propel or symfony or your usage of them. You should know that symfony 1.2 is no longer supported but replaced by symfony 1.4. The favoured ORM nowadays is Doctrine although Propel is still supported. But most importantly: you posted a complaint rather than a question. This might work with a licenced software from some company but with an open source community a certain netiquette should apply. Cheers /Christian On 12 Apr., 05:51, Slavka richard@gmail.com wrote: It seems to be pretty clear the symfony 1.2 and propel are still full of memory leaks... I have a simple for loop selecting the same object into the same variable... in theory this should not lead to increase memory use for every loop however that is what is happening... Has anyone every looked into this... if so what are options to handle the leaks... , if there is no solution we are going to have to consider just using mysql functions... Regards -- If you want to report a vulnerability issue on symfony, please send it to security at symfony-project.com You received this message because you are subscribed to the Google Groups symfony users group. To post to this group, send email to symfony-users@googlegroups.com To unsubscribe from this group, send email to symfony-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/symfony-users?hl=en To unsubscribe, reply using remove me as the subject. -- If you want to report a vulnerability issue on symfony, please send it to security at symfony-project.com You received this message because you are subscribed to the Google Groups symfony users group. To post to this group, send email to symfony-users@googlegroups.com To unsubscribe from this group, send email to symfony-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/symfony-users?hl=en -- If you want to report a vulnerability issue on symfony, please send it to security at symfony-project.com You received this message because you are subscribed to the Google Groups symfony users group. To post to this group, send email to symfony-users@googlegroups.com To unsubscribe from this group, send email to symfony-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/symfony-users?hl=en -- If you want to report a vulnerability issue on symfony, please send it to security at symfony-project.com You received this message because you are
Re: [symfony-users] Re: memory leaks... just keep on leaking
I thought that was the issue... , the thing is how can i work around it? On Mon, Apr 12, 2010 at 5:38 PM, Alvaro Videla harryjek...@gmail.com wrote: Except from the fact that I haven't read your code, what you describe is a problem with PHP versions prior to 5.3 The problem is in the GC of the engine I'm not blaming it on PHP, but you could try that. If you can't upgrade to the new PHP, then I'm sorry, but what you try to do may not work at all. And you can forget about blaming Propel or symfony for the problem. Try to instantiate a big class inside that loop, a class created by your own code and you will see the problem appear. Maybe it can take longer, because Propel classes tend to be quite huge, but that will fail at some point. My 2 cents, Alvaro On Apr 12, 2010, at 3:28 PM, Dennis Benkert wrote: So, how big are these CSV files? How many lines are you processing and can you tell us how many entries you're inserting to your DB using Propel? It would be best if you could show us the code that implements what you described in your last mail. Investing maybe 30 minutes to write an email that desribes the problem you have, what part of your code suffers from it and the data you're using will most likely keep you away from writing 5 or more emails desribing everything. Plus it will stop me asking stupid questions because I have to digg into your problem this way. :) - Dennis 2010/4/12 Richard U richard@gmail.com: I am uploading a csv file.. i require looping over each row... the data in each column has to be processed individually and allocated to the right table... On Mon, Apr 12, 2010 at 3:56 PM, Dennis Benkert spinecras...@googlemail.com wrote: Although I am pretty sure you are frustrated there is no reason to show disrespect to other people. Christian's point is that we cannot help you without getting some information about your specific problem. Maybe you can provide some code sample that show us your memory problems. Also it's interesting to know what you want to do with the code (are you inserting thousands of entries, looping over many entries and saving them again, etc.). - Dennis Am 12.04.2010 um 07:41 schrieb Slavka richard@gmail.com: Christian have you just started using symfony? if not then i am sure you are aware of the memory leaks i am talking about... Cheers On Apr 12, 3:30 pm, Christian Schaefer cae...@gmail.com wrote: Hi Slavka, from your tiny description it is not clear whether the leak you experience is a result of Propel or symfony or your usage of them. You should know that symfony 1.2 is no longer supported but replaced by symfony 1.4. The favoured ORM nowadays is Doctrine although Propel is still supported. But most importantly: you posted a complaint rather than a question. This might work with a licenced software from some company but with an open source community a certain netiquette should apply. Cheers /Christian On 12 Apr., 05:51, Slavka richard@gmail.com wrote: It seems to be pretty clear the symfony 1.2 and propel are still full of memory leaks... I have a simple for loop selecting the same object into the same variable... in theory this should not lead to increase memory use for every loop however that is what is happening... Has anyone every looked into this... if so what are options to handle the leaks... , if there is no solution we are going to have to consider just using mysql functions... Regards -- If you want to report a vulnerability issue on symfony, please send it to security at symfony-project.com You received this message because you are subscribed to the Google Groups symfony users group. To post to this group, send email to symfony-users@googlegroups.com To unsubscribe from this group, send email to symfony-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/symfony-users?hl=en To unsubscribe, reply using remove me as the subject. -- If you want to report a vulnerability issue on symfony, please send it to security at symfony-project.com You received this message because you are subscribed to the Google Groups symfony users group. To post to this group, send email to symfony-users@googlegroups.com To unsubscribe from this group, send email to symfony-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/symfony-users?hl=en -- If you want to report a vulnerability issue on symfony, please send it to security at symfony-project.com You received this message because you are subscribed to the Google Groups symfony users group. To post to this group, send email to symfony-users@googlegroups.com To unsubscribe from this group, send email to symfony-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/symfony-users?hl=en -- If you want to report a
Re: [symfony-users] Re: memory leaks... just keep on leaking
Dennis thats the function if you want to have a look at it...
public function executeIndex(sfWebRequest $request) {
ini_set('memory_limit','512M');
set_time_limit(60*5);
sfConfig::set('sf_logging_enabled', false);
sfConfig::set('sf_web_debug', false);
$timer = new Timer;
$timer-start_time();
$this-submited_bulk_stats =
$request-getParameterHolder()-get('bulk_stats');
$this-form = new BulkAddStatsForm();
$this-count_updated_records = 0;
$this-count_inserted_records = 0;
if ($request-isMethod('post'))
{
$this-form-bind($request-getParameter('bulk_stats'),
$request-getFiles('bulk_stats'));
if ($this-form-isValid())
{
$values = $this-form-getValues();
$file = $this-form-getValue('file');
$file_name =
str_replace($file-getOriginalExtension(), '',
$file-getOriginalName());
$save_to =
sfConfig::get('sf_upload_dir').DIRECTORY_SEPARATOR.'uploaded_'.$file_name.'_'.date(dmYHis).$file-getOriginalExtension();
$file-save($save_to);
echo brFile saved: $save_to;
echo brFile size: .filesize($save_to);
// get accounts
$accounts = AccountsPeer::doSelect(new
Criteria());
$re_indexed_accounts =
$re_indexed_accounts_by_id = array();
foreach ($accounts as $tmp_account) {
$re_indexed_accounts[$tmp_account-getAffName().$tmp_account-getAffUser()]
= $tmp_account;
$re_indexed_accounts_by_id[$tmp_account-getId()] = $tmp_account;
}
// get parking programs
$parking_programs =
ParkingProgramPeer::doSelect(new Criteria());
$re_indexed_parking_programs = array();
foreach ($parking_programs as $pp) {
$re_indexed_parking_programs[$pp-getId()] = $pp;
}
// Process the uploaded file
$objReader = new PHPExcel_Reader_Excel2007;
$objPHPExcel = $objReader-load($save_to);
// Define allowed columns
$allowed_columns =
array('domain','views','clicks','revenue','currency','date','park','account');
$re_indexed_data = array();
$data =
$objPHPExcel-getActiveSheet()-toArray();
// Get db connections since we are using two
different db's here
$connection_cdb = Propel::getConnection();
$connection_stats =
Propel::getConnection('cdb_stats');
// get domains
$domains_re_indexed = array();
$query = 'SELECT * FROM domains';
$statement = $connection_cdb-prepare($query);
$statement-execute();
while ($dObj = $statement-fetch(PDO::FETCH_OBJ))
{
$domains_re_indexed[$dObj-dn] =
array('domain_id'=$dObj-domain_ID, 'deal_id'=$dObj-deal_id);
}
try {
$validated_data = array();
$sfLogger =
sfContext::getInstance()-getLogger();
foreach($data as $row_number = $row) {
$batch_options = array();
// Check its not an empty row
if (!strlen(implode(, $row)))
{
continue;
}
Re: [symfony-users] Re: memory leaks... just keep on leaking
Thanks Dennis, awesome feedback!
On Mon, Apr 12, 2010 at 5:53 PM, Richard U richard@gmail.com wrote:
Dennis thats the function if you want to have a look at it...
public function executeIndex(sfWebRequest $request) {
ini_set('memory_limit','512M');
set_time_limit(60*5);
sfConfig::set('sf_logging_enabled', false);
sfConfig::set('sf_web_debug', false);
$timer = new Timer;
$timer-start_time();
$this-submited_bulk_stats =
$request-getParameterHolder()-get('bulk_stats');
$this-form = new BulkAddStatsForm();
$this-count_updated_records = 0;
$this-count_inserted_records = 0;
if ($request-isMethod('post'))
{
$this-form-bind($request-getParameter('bulk_stats'),
$request-getFiles('bulk_stats'));
if ($this-form-isValid())
{
$values = $this-form-getValues();
$file = $this-form-getValue('file');
$file_name =
str_replace($file-getOriginalExtension(), '',
$file-getOriginalName());
$save_to =
sfConfig::get('sf_upload_dir').DIRECTORY_SEPARATOR.'uploaded_'.$file_name.'_'.date(dmYHis).$file-getOriginalExtension();
$file-save($save_to);
echo brFile saved: $save_to;
echo brFile size: .filesize($save_to);
// get accounts
$accounts = AccountsPeer::doSelect(new
Criteria());
$re_indexed_accounts =
$re_indexed_accounts_by_id = array();
foreach ($accounts as $tmp_account) {
$re_indexed_accounts[$tmp_account-getAffName().$tmp_account-getAffUser()]
= $tmp_account;
$re_indexed_accounts_by_id[$tmp_account-getId()] = $tmp_account;
}
// get parking programs
$parking_programs =
ParkingProgramPeer::doSelect(new Criteria());
$re_indexed_parking_programs = array();
foreach ($parking_programs as $pp) {
$re_indexed_parking_programs[$pp-getId()] = $pp;
}
// Process the uploaded file
$objReader = new PHPExcel_Reader_Excel2007;
$objPHPExcel = $objReader-load($save_to);
// Define allowed columns
$allowed_columns =
array('domain','views','clicks','revenue','currency','date','park','account');
$re_indexed_data = array();
$data =
$objPHPExcel-getActiveSheet()-toArray();
// Get db connections since we are using two
different db's here
$connection_cdb = Propel::getConnection();
$connection_stats =
Propel::getConnection('cdb_stats');
// get domains
$domains_re_indexed = array();
$query = 'SELECT * FROM domains';
$statement = $connection_cdb-prepare($query);
$statement-execute();
while ($dObj = $statement-fetch(PDO::FETCH_OBJ))
{
$domains_re_indexed[$dObj-dn] =
array('domain_id'=$dObj-domain_ID, 'deal_id'=$dObj-deal_id);
}
try {
$validated_data = array();
$sfLogger =
sfContext::getInstance()-getLogger();
foreach($data as $row_number = $row) {
$batch_options = array();
// Check its not an empty row
if (!strlen(implode(,
$row))) {
continue;
}
foreach($row as $col_number =
$col) {
echo brMEMORY:
.memory_get_usage();
if ($row_number == 1) {
$col =
strtolower($col);
Re: [symfony-users] Re: memory leaks... just keep on leaking
Ok, just to clarify some things. I do this in my spare time (as most
of the people on this mailing list). Having this said you may realize
that I cannot (and will not) give you feedback a few minutes after you
replied to my mail. If you might take a look at other threads on this
mailing list other people had to wait for answers on their questions,
too. If you can't life with the fact that you have to wait for answers
on a mailing list for an open source software you might have to
consider buying support by someone like Sensio.
Also I don't see why you have to reply to people who want to help you
with such a disrespect.
Talking about the code you posted. After looking at it for 5 minutes
it looks to me that you have a really fat controller whith a huge
amount of code which I might consider to move into some service
classes. Also, as Alvaro already stated using ORMs to manipulate huge
amounts of data will lead to memory problems after some time.
Switching down to raw sql might be an option to solve your problem
(again, I only had a look on your code for about 5 minutes).
- Dennis
2010/4/12 Richard U richard@gmail.com:
Thanks Dennis, awesome feedback!
On Mon, Apr 12, 2010 at 5:53 PM, Richard U richard@gmail.com wrote:
Dennis thats the function if you want to have a look at it...
public function executeIndex(sfWebRequest $request) {
ini_set('memory_limit','512M');
set_time_limit(60*5);
sfConfig::set('sf_logging_enabled', false);
sfConfig::set('sf_web_debug', false);
$timer = new Timer;
$timer-start_time();
$this-submited_bulk_stats =
$request-getParameterHolder()-get('bulk_stats');
$this-form = new BulkAddStatsForm();
$this-count_updated_records = 0;
$this-count_inserted_records = 0;
if ($request-isMethod('post'))
{
$this-form-bind($request-getParameter('bulk_stats'),
$request-getFiles('bulk_stats'));
if ($this-form-isValid())
{
$values = $this-form-getValues();
$file = $this-form-getValue('file');
$file_name =
str_replace($file-getOriginalExtension(), '',
$file-getOriginalName());
$save_to =
sfConfig::get('sf_upload_dir').DIRECTORY_SEPARATOR.'uploaded_'.$file_name.'_'.date(dmYHis).$file-getOriginalExtension();
$file-save($save_to);
echo brFile saved: $save_to;
echo brFile size: .filesize($save_to);
// get accounts
$accounts = AccountsPeer::doSelect(new
Criteria());
$re_indexed_accounts =
$re_indexed_accounts_by_id = array();
foreach ($accounts as $tmp_account) {
$re_indexed_accounts[$tmp_account-getAffName().$tmp_account-getAffUser()]
= $tmp_account;
$re_indexed_accounts_by_id[$tmp_account-getId()] = $tmp_account;
}
// get parking programs
$parking_programs =
ParkingProgramPeer::doSelect(new Criteria());
$re_indexed_parking_programs = array();
foreach ($parking_programs as $pp) {
$re_indexed_parking_programs[$pp-getId()] = $pp;
}
// Process the uploaded file
$objReader = new PHPExcel_Reader_Excel2007;
$objPHPExcel = $objReader-load($save_to);
// Define allowed columns
$allowed_columns =
array('domain','views','clicks','revenue','currency','date','park','account');
$re_indexed_data = array();
$data =
$objPHPExcel-getActiveSheet()-toArray();
// Get db connections since we are using two
different db's here
$connection_cdb = Propel::getConnection();
$connection_stats =
Propel::getConnection('cdb_stats');
// get domains
$domains_re_indexed = array();
$query = 'SELECT * FROM domains';
$statement = $connection_cdb-prepare($query);
$statement-execute();
while ($dObj = $statement-fetch(PDO::FETCH_OBJ))
{
Re: [symfony-users] Re: memory leaks... just keep on leaking
Hi Richard Model::clearInstancePool() is not only mechanism that tries to solve memory leaks issue in Propel - check this out,http://bugfeatures.com/blog/comments.php?y=09m=08entry=entry090809-055809#entry090809-055809_comment090809-134908 Also, you should take into consideration manually destroying objects and variables used inside of your loops. As my predecessors wrote: PHPs Garbage Collector is very unreliable thing and carrying at least part of its work by yourself should give measurable benefits. Cheers, Alan -- If you want to report a vulnerability issue on symfony, please send it to security at symfony-project.com You received this message because you are subscribed to the Google Groups symfony users group. To post to this group, send email to symfony-users@googlegroups.com To unsubscribe from this group, send email to symfony-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/symfony-users?hl=en To unsubscribe, reply using remove me as the subject.
Re: [symfony-users] Re: memory leaks... just keep on leaking
Ok guys... thanks for your patience with me...
Don't know if you guys had a chance to check out the code i pasted but
if you didn't i am already doing most of the things stated in the
replays... and i really think that this is the a php issue... not
symfony/propel issue..
Having said that... i have started working on a solution where the
file upload is done in a separate action... which then forwards the
user to another action that processes the data in the uploaded file x
amount of records at a time... refreshing the page and passing the
pager's current status as a request parameter, so processing a file
with 1 1000 records at time would mean 10 page refreshes...
Anyway if anyone gets a chance let me know what you think about this...
Regards
Richard
Btw... to anyone who i offended earlier... my deepest apologies...
sometimes frustration gets the better of me...
On Mon, Apr 12, 2010 at 10:01 PM, Tony Piper tpi...@tpiper.com wrote:
Just to chip in with another possible cause:
I see you're doing all of this in a controller (and others have talked
about whether that's a good idea). Have you got the web debug toolbar
enabled? If so, it could be keeping a log of all DB queries executed
in your action. I had something similar with a batch job in 1.3/1.4.
There 's a trac ticket which gives some workarounds:
http://trac.symfony-project.org/ticket/3039
Hope this helps.
cheers,
Tony.
On Apr 12, 8:53 am, Richard U richard@gmail.com wrote:
Dennis thats the function if you want to have a look at it...
public function executeIndex(sfWebRequest $request) {
ini_set('memory_limit','512M');
set_time_limit(60*5);
sfConfig::set('sf_logging_enabled', false);
sfConfig::set('sf_web_debug', false);
$timer = new Timer;
$timer-start_time();
$this-submited_bulk_stats =
$request-getParameterHolder()-get('bulk_stats');
$this-form = new BulkAddStatsForm();
$this-count_updated_records = 0;
$this-count_inserted_records = 0;
if ($request-isMethod('post'))
{
$this-form-bind($request-getParameter('bulk_stats'),
$request-getFiles('bulk_stats'));
if ($this-form-isValid())
{
$values = $this-form-getValues();
$file = $this-form-getValue('file');
$file_name =
str_replace($file-getOriginalExtension(), '',
$file-getOriginalName());
$save_to =
sfConfig::get('sf_upload_dir').DIRECTORY_SEPARATOR.'uploaded_'.$file_name.'
_'.date(dmYHis).$file-getOriginalExtension();
$file-save($save_to);
echo brFile saved: $save_to;
echo brFile size: .filesize($save_to);
// get accounts
$accounts = AccountsPeer::doSelect(new
Criteria());
$re_indexed_accounts =
$re_indexed_accounts_by_id = array();
foreach ($accounts as $tmp_account) {
$re_indexed_accounts[$tmp_account-getAffName().$tmp_account-getAffUser()]
= $tmp_account;
$re_indexed_accounts_by_id[$tmp_account-getId()] = $tmp_account;
}
// get parking programs
$parking_programs =
ParkingProgramPeer::doSelect(new Criteria());
$re_indexed_parking_programs = array();
foreach ($parking_programs as $pp) {
$re_indexed_parking_programs[$pp-getId()] = $pp;
}
// Process the uploaded file
$objReader = new PHPExcel_Reader_Excel2007;
$objPHPExcel = $objReader-load($save_to);
// Define allowed columns
$allowed_columns =
array('domain','views','clicks','revenue','currency','date','park','account
');
$re_indexed_data = array();
$data =
$objPHPExcel-getActiveSheet()-toArray();
// Get db connections since we are using two
different db's here
$connection_cdb = Propel::getConnection();
$connection_stats =
Propel::getConnection('cdb_stats');
// get domains
$domains_re_indexed = array();
Re: [symfony-users] Re: memory leaks... just keep on leaking
That sounds ideal, but i think in my situation the user would most
likely be waiting at the screen... i guess your design can still be
applied in the same way with some ajax displaying to the user the
processing as it proceeds...
On Mon, Apr 12, 2010 at 11:16 PM, Tony Piper tpi...@tpiper.com wrote:
Unless you absolutely require the upload to be processed with the user
present I would handle loading this data via a background task, so I
can split it up into as small chunks as I like/need without annoying
the user; What if they want to upload 50,000 records? What happens if
their browser dies on page 3?
In my controller I'd move the uploaded file to a safe place and then
create an entry in a processing queue table with details of that file.
I'd then have a background task that runs every 5 minutes via cron and
processes any new files, notifying the user when their file has been
processed. If anything bad happens when processing the file the cron
job would notify the user and/or the web developer.
cheers,
Tony.
On Apr 12, 1:32 pm, Richard U richard@gmail.com wrote:
Ok guys... thanks for your patience with me...
Don't know if you guys had a chance to check out the code i pasted but
if you didn't i am already doing most of the things stated in the
replays... and i really think that this is the a php issue... not
symfony/propel issue..
Having said that... i have started working on a solution where the
file upload is done in a separate action... which then forwards the
user to another action that processes the data in the uploaded file x
amount of records at a time... refreshing the page and passing the
pager's current status as a request parameter, so processing a file
with 1 1000 records at time would mean 10 page refreshes...
Anyway if anyone gets a chance let me know what you think about this...
Regards
Richard
Btw... to anyone who i offended earlier... my deepest apologies...
sometimes frustration gets the better of me...
On Mon, Apr 12, 2010 at 10:01 PM, Tony Piper tpi...@tpiper.com wrote:
Just to chip in with another possible cause:
I see you're doing all of this in a controller (and others have talked
about whether that's a good idea). Have you got the web debug toolbar
enabled? If so, it could be keeping a log of all DB queries executed
in your action. I had something similar with a batch job in 1.3/1.4.
There 's a trac ticket which gives some workarounds:
http://trac.symfony-project.org/ticket/3039
Hope this helps.
cheers,
Tony.
On Apr 12, 8:53 am, Richard U richard@gmail.com wrote:
Dennis thats the function if you want to have a look at it...
public function executeIndex(sfWebRequest $request) {
ini_set('memory_limit','512M');
set_time_limit(60*5);
sfConfig::set('sf_logging_enabled', false);
sfConfig::set('sf_web_debug', false);
$timer = new Timer;
$timer-start_time();
$this-submited_bulk_stats =
$request-getParameterHolder()-get('bulk_stats');
$this-form = new BulkAddStatsForm();
$this-count_updated_records = 0;
$this-count_inserted_records = 0;
if ($request-isMethod('post'))
{
$this-form-bind($request-getParameter('bulk_stats'),
$request-getFiles('bulk_stats'));
if ($this-form-isValid())
{
$values = $this-form-getValues();
$file = $this-form-getValue('file');
$file_name =
str_replace($file-getOriginalExtension(), '',
$file-getOriginalName());
$save_to =
sfConfig::get('sf_upload_dir').DIRECTORY_SEPARATOR.'uploaded_'.$file_name.'
_'.date(dmYHis).$file-getOriginalExtension();
$file-save($save_to);
echo brFile saved: $save_to;
echo brFile size: .filesize($save_to);
// get accounts
$accounts = AccountsPeer::doSelect(new
Criteria());
$re_indexed_accounts =
$re_indexed_accounts_by_id = array();
foreach ($accounts as $tmp_account) {
$re_indexed_accounts[$tmp_account-getAffName().$tmp_account-getAffUser()]
= $tmp_account;
$re_indexed_accounts_by_id[$tmp_account-getId()] = $tmp_account;
}
// get parking programs
$parking_programs =
ParkingProgramPeer::doSelect(new Criteria());
Re: [symfony-users] Re: memory leaks... just keep on leaking
Although I am pretty sure you are frustrated there is no reason to show disrespect to other people. Christian's point is that we cannot help you without getting some information about your specific problem. Maybe you can provide some code sample that show us your memory problems. Also it's interesting to know what you want to do with the code (are you inserting thousands of entries, looping over many entries and saving them again, etc.). - Dennis Am 12.04.2010 um 07:41 schrieb Slavka richard@gmail.com: Christian have you just started using symfony? if not then i am sure you are aware of the memory leaks i am talking about... Cheers On Apr 12, 3:30 pm, Christian Schaefer cae...@gmail.com wrote: Hi Slavka, from your tiny description it is not clear whether the leak you experience is a result of Propel or symfony or your usage of them. You should know that symfony 1.2 is no longer supported but replaced by symfony 1.4. The favoured ORM nowadays is Doctrine although Propel is still supported. But most importantly: you posted a complaint rather than a question. This might work with a licenced software from some company but with an open source community a certain netiquette should apply. Cheers /Christian On 12 Apr., 05:51, Slavka richard@gmail.com wrote: It seems to be pretty clear the symfony 1.2 and propel are still full of memory leaks... I have a simple for loop selecting the same object into the same variable... in theory this should not lead to increase memory use for every loop however that is what is happening... Has anyone every looked into this... if so what are options to handle the leaks... , if there is no solution we are going to have to consider just using mysql functions... Regards -- If you want to report a vulnerability issue on symfony, please send it to security at symfony-project.com You received this message because you are subscribed to the Google Groups symfony users group. To post to this group, send email to symfony-users@googlegroups.com To unsubscribe from this group, send email to symfony-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/symfony-users?hl=en To unsubscribe, reply using remove me as the subject. -- If you want to report a vulnerability issue on symfony, please send it to security at symfony-project.com You received this message because you are subscribed to the Google Groups symfony users group. To post to this group, send email to symfony-users@googlegroups.com To unsubscribe from this group, send email to symfony-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/symfony-users?hl=en
