Unless you absolutely require the upload to be processed with the user present I would handle loading this data via a background task, so I can split it up into as small chunks as I like/need without annoying the user; What if they want to upload 50,000 records? What happens if their browser dies on page 3?
In my controller I'd move the uploaded file to a safe place and then create an entry in a processing queue table with details of that file. I'd then have a background task that runs every 5 minutes via cron and processes any new files, notifying the user when their file has been processed. If anything bad happens when processing the file the cron job would notify the user and/or the web developer. cheers, Tony. On Apr 12, 1:32 pm, Richard U <richard....@gmail.com> wrote: > Ok guys... thanks for your patience with me... > Don't know if you guys had a chance to check out the code i pasted but > if you didn't i am already doing most of the things stated in the > replays... and i really think that this is the a php issue... not > symfony/propel issue.. > > Having said that... i have started working on a solution where the > file upload is done in a separate action... which then forwards the > user to another action that processes the data in the uploaded file x > amount of records at a time... refreshing the page and passing the > pager's current status as a request parameter, so processing a file > with 10000 1000 records at time would mean 10 page refreshes... > > Anyway if anyone gets a chance let me know what you think about this... > > Regards > Richard > > Btw... to anyone who i offended earlier... my deepest apologies... > sometimes frustration gets the better of me... > > > > On Mon, Apr 12, 2010 at 10:01 PM, Tony Piper <tpi...@tpiper.com> wrote: > > Just to chip in with another possible cause: > > > I see you're doing all of this in a controller (and others have talked > > about whether that's a good idea). Have you got the web debug toolbar > > enabled? If so, it could be keeping a log of all DB queries executed > > in your action. I had something similar with a batch job in 1.3/1.4. > > > There 's a trac ticket which gives some workarounds: > >http://trac.symfony-project.org/ticket/3039 > > > Hope this helps. > > > cheers, > > > Tony. > > > On Apr 12, 8:53 am, Richard U <richard....@gmail.com> wrote: > >> Dennis thats the function if you want to have a look at it... > > >> public function executeIndex(sfWebRequest $request) { > > >> ini_set('memory_limit','512M'); > >> set_time_limit(60*5); > >> sfConfig::set('sf_logging_enabled', false); > >> sfConfig::set('sf_web_debug', false); > > >> $timer = new Timer; > >> $timer->start_time(); > > >> $this->submited_bulk_stats = > >> $request->getParameterHolder()->get('bulk_stats'); > > >> $this->form = new BulkAddStatsForm(); > > >> $this->count_updated_records = 0; > >> $this->count_inserted_records = 0; > > >> if ($request->isMethod('post')) > >> { > >> > >> $this->form->bind($request->getParameter('bulk_stats'), > >> $request->getFiles('bulk_stats')); > >> if ($this->form->isValid()) > >> { > >> $values = $this->form->getValues(); > > >> $file = $this->form->getValue('file'); > >> $file_name = > >> str_replace($file->getOriginalExtension(), '', > >> $file->getOriginalName()); > >> $save_to = > >> sfConfig::get('sf_upload_dir').DIRECTORY_SEPARATOR.'uploaded_'.$file_name.' > >> _'.date("dmYHis").$file->getOriginalExtension(); > >> $file->save($save_to); > > >> echo "<br>File saved: $save_to"; > >> echo "<br>File size: ".filesize($save_to); > > >> // get accounts > >> $accounts = AccountsPeer::doSelect(new > >> Criteria()); > >> $re_indexed_accounts = > >> $re_indexed_accounts_by_id = array(); > >> foreach ($accounts as $tmp_account) { > >> > >> $re_indexed_accounts[$tmp_account->getAffName().$tmp_account->getAffUser()] > >> = $tmp_account; > >> > >> $re_indexed_accounts_by_id[$tmp_account->getId()] = $tmp_account; > >> } > > >> // get parking programs > >> $parking_programs = > >> ParkingProgramPeer::doSelect(new Criteria()); > >> $re_indexed_parking_programs = array(); > >> foreach ($parking_programs as $pp) { > >> > >> $re_indexed_parking_programs[$pp->getId()] = $pp; > >> } > > >> // Process the uploaded file > >> $objReader = new PHPExcel_Reader_Excel2007; > >> $objPHPExcel = $objReader->load($save_to); > > >> // Define allowed columns > >> $allowed_columns = > >> array('domain','views','clicks','revenue','currency','date','park','account > >> '); > >> $re_indexed_data = array(); > >> $data = > >> $objPHPExcel->getActiveSheet()->toArray(); > > >> // Get db connections since we are using > >> two different db's here > >> $connection_cdb = Propel::getConnection(); > >> $connection_stats = > >> Propel::getConnection('cdb_stats'); > > >> // get domains > >> $domains_re_indexed = array(); > >> $query = 'SELECT * FROM domains'; > >> $statement = $connection_cdb->prepare($query); > >> $statement->execute(); > >> while ($dObj = > >> $statement->fetch(PDO::FETCH_OBJ)) > >> { > >> $domains_re_indexed[$dObj->dn] = > >> array('domain_id'=>$dObj->domain_ID, 'deal_id'=>$dObj->deal_id); > >> } > > >> try { > >> $validated_data = array(); > >> $sfLogger = > >> sfContext::getInstance()->getLogger(); > > >> foreach($data as $row_number => > >> $row) { > >> $batch_options = array(); > > >> // Check its not an empty > >> row > >> if (!strlen(implode("", > >> $row))) { > >> continue; > >> } > > >> foreach($row as > >> $col_number => $col) { > >> echo "<br>MEMORY: > >> ".memory_get_usage(); > >> if ($row_number == > >> 1) { > >> $col = > >> strtolower($col); > >> $key = > >> array_search($col, $allowed_columns); > >> if ($key > >> !== false) { > >> > >> $data_indexes[$col_number] = $allowed_columns[$key]; > >> } > >> } else { > >> if > >> (isset($data_indexes[$col_number]) && > >> strlen($data_indexes[$col_number]) && strlen($col)) { > >> if > >> ($data_indexes[$col_number] == "date") { > >> > >> $col = date("Y-m-d", strtotime(str_replace("/", "-", $col))); > >> } > >> > >> $batch_options[$data_indexes[$col_number]] = $col; > >> } > >> } > >> } > > >> if ($row_number > 1) { > > >> // Park info now > >> comes from the form > >> if > >> (!isset($values['parking_program']) || > >> !strlen($values['parking_program'])) { > >> throw new > >> Exception('Missing or invalid parking program'); > >> > > ... > > read more » -- If you want to report a vulnerability issue on symfony, please send it to security at symfony-project.com You received this message because you are subscribed to the Google Groups "symfony users" group. To post to this group, send email to symfony-users@googlegroups.com To unsubscribe from this group, send email to symfony-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/symfony-users?hl=en To unsubscribe, reply using "remove me" as the subject.
