A ran into memory issues today when I had to modify every record in a table after a Doctrine Migration. The problem is that PHP doesn't decrease the reference count of a child/parent object, and therefore doesn't free the memory. The problem is described http://paul-m-jones.com/archives/262 and http://www.alexatnet.com/content/optimize-php-memory-usage-eliminate-circular-references (and many more places). The userspace solution is to manually unset references to other objects before an object gets out of scope.
I don't know about Propel, but a solution for Doctrine is posted here: http://trac.doctrine-project.org/ticket/710 When dealing with large datasets you should call $obj->free() before the object gets out of scope. Example code: for ($i = 0; $i < 5000; $i++) { $u = new User(); $u->name = "User " . $i; $u->save(); $u->free(true); } This solved my problem, as memory consumption doesn't grow anymore. I bet a similar solution is possible for Propel. Cheers, Tijmen On Apr 12, 11:14 am, Dennis Benkert <spinecras...@googlemail.com> wrote: > Ok, just to clarify some things. I do this in my spare time (as most > of the people on this mailing list). Having this said you may realize > that I cannot (and will not) give you feedback a few minutes after you > replied to my mail. If you might take a look at other threads on this > mailing list other people had to wait for answers on their questions, > too. If you can't life with the fact that you have to wait for answers > on a mailing list for an open source software you might have to > consider buying support by someone like Sensio. > > Also I don't see why you have to reply to people who want to help you > with such a disrespect. > > Talking about the code you posted. After looking at it for 5 minutes > it looks to me that you have a really fat controller whith a huge > amount of code which I might consider to move into some service > classes. Also, as Alvaro already stated using ORMs to manipulate huge > amounts of data will lead to memory problems after some time. > Switching down to raw sql might be an option to solve your problem > (again, I only had a look on your code for about 5 minutes). > > - Dennis > > 2010/4/12 Richard U <richard....@gmail.com>: > > > Thanks Dennis, awesome feedback! > > > On Mon, Apr 12, 2010 at 5:53 PM, Richard U <richard....@gmail.com> wrote: > >> Dennis thats the function if you want to have a look at it... > > >> public function executeIndex(sfWebRequest $request) { > > >> ini_set('memory_limit','512M'); > >> set_time_limit(60*5); > >> sfConfig::set('sf_logging_enabled', false); > >> sfConfig::set('sf_web_debug', false); > > >> $timer = new Timer; > >> $timer->start_time(); > > >> $this->submited_bulk_stats = > >> $request->getParameterHolder()->get('bulk_stats'); > > >> $this->form = new BulkAddStatsForm(); > > >> $this->count_updated_records = 0; > >> $this->count_inserted_records = 0; > > >> if ($request->isMethod('post')) > >> { > >> > >> $this->form->bind($request->getParameter('bulk_stats'), > >> $request->getFiles('bulk_stats')); > >> if ($this->form->isValid()) > >> { > >> $values = $this->form->getValues(); > > >> $file = $this->form->getValue('file'); > >> $file_name = > >> str_replace($file->getOriginalExtension(), '', > >> $file->getOriginalName()); > >> $save_to = > >> sfConfig::get('sf_upload_dir').DIRECTORY_SEPARATOR.'uploaded_'.$file_name.'_'.date("dmYHis").$file->getOriginalExtension(); > >> $file->save($save_to); > > >> echo "<br>File saved: $save_to"; > >> echo "<br>File size: ".filesize($save_to); > > >> // get accounts > >> $accounts = AccountsPeer::doSelect(new > >> Criteria()); > >> $re_indexed_accounts = > >> $re_indexed_accounts_by_id = array(); > >> foreach ($accounts as $tmp_account) { > >> > >> $re_indexed_accounts[$tmp_account->getAffName().$tmp_account->getAffUser()] > >> = $tmp_account; > >> > >> $re_indexed_accounts_by_id[$tmp_account->getId()] = $tmp_account; > >> } > > >> // get parking programs > >> $parking_programs = > >> ParkingProgramPeer::doSelect(new Criteria()); > >> $re_indexed_parking_programs = array(); > >> foreach ($parking_programs as $pp) { > >> > >> $re_indexed_parking_programs[$pp->getId()] = $pp; > >> } > > >> // Process the uploaded file > >> $objReader = new PHPExcel_Reader_Excel2007; > >> $objPHPExcel = $objReader->load($save_to); > > >> // Define allowed columns > >> $allowed_columns = > >> array('domain','views','clicks','revenue','currency','date','park','account'); > >> $re_indexed_data = array(); > >> $data = > >> $objPHPExcel->getActiveSheet()->toArray(); > > >> // Get db connections since we are using > >> two different db's here > >> $connection_cdb = Propel::getConnection(); > >> $connection_stats = > >> Propel::getConnection('cdb_stats'); > > >> // get domains > >> $domains_re_indexed = array(); > >> $query = 'SELECT * FROM domains'; > >> $statement = $connection_cdb->prepare($query); > >> $statement->execute(); > >> while ($dObj = > >> $statement->fetch(PDO::FETCH_OBJ)) > >> { > >> $domains_re_indexed[$dObj->dn] = > >> array('domain_id'=>$dObj->domain_ID, 'deal_id'=>$dObj->deal_id); > >> } > > >> try { > >> $validated_data = array(); > >> $sfLogger = > >> sfContext::getInstance()->getLogger(); > > >> foreach($data as $row_number => > >> $row) { > >> $batch_options = array(); > > >> // Check its not an empty > >> row > >> if (!strlen(implode("", > >> $row))) { > >> continue; > >> } > > >> foreach($row as $col_number > >> => $col) { > >> echo "<br>MEMORY: > >> ".memory_get_usage(); > >> if ($row_number == > >> 1) { > >> $col = > >> strtolower($col); > >> $key = > >> array_search($col, $allowed_columns); > >> if ($key > >> !== false) { > >> > >> $data_indexes[$col_number] = $allowed_columns[$key]; > >> } > >> } else { > >> if > >> (isset($data_indexes[$col_number]) && > >> strlen($data_indexes[$col_number]) && strlen($col)) { > >> if > >> ($data_indexes[$col_number] == "date") { > >> > >> $col = date("Y-m-d", strtotime(str_replace("/", "-", $col))); > >> } > >> > >> $batch_options[$data_indexes[$col_number]] = $col; > >> } > >> } > >> } > > >> if ($row_number > 1) { > > >> // Park info now > >> comes from the form > >> if > >> (!isset($values['parking_program']) || > >> !strlen($values['parking_program'])) { > >> throw new > >> Exception('Missing or invalid parking program'); > >> } else { > >> > >> $batch_options['aff_name'] = > >> strtolower($re_indexed_parking_programs[$values['parking_program']]->getName()); > >> > > ... > > read more » -- If you want to report a vulnerability issue on symfony, please send it to security at symfony-project.com You received this message because you are subscribed to the Google Groups "symfony users" group. To post to this group, send email to symfony-users@googlegroups.com To unsubscribe from this group, send email to symfony-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/symfony-users?hl=en To unsubscribe, reply using "remove me" as the subject.
