That sounds ideal, but i think in my situation the user would most likely be waiting at the screen... i guess your design can still be applied in the same way with some ajax displaying to the user the processing as it proceeds...
On Mon, Apr 12, 2010 at 11:16 PM, Tony Piper <tpi...@tpiper.com> wrote: > Unless you absolutely require the upload to be processed with the user > present I would handle loading this data via a background task, so I > can split it up into as small chunks as I like/need without annoying > the user; What if they want to upload 50,000 records? What happens if > their browser dies on page 3? > > In my controller I'd move the uploaded file to a safe place and then > create an entry in a processing queue table with details of that file. > I'd then have a background task that runs every 5 minutes via cron and > processes any new files, notifying the user when their file has been > processed. If anything bad happens when processing the file the cron > job would notify the user and/or the web developer. > > cheers, > > Tony. > > On Apr 12, 1:32 pm, Richard U <richard....@gmail.com> wrote: >> Ok guys... thanks for your patience with me... >> Don't know if you guys had a chance to check out the code i pasted but >> if you didn't i am already doing most of the things stated in the >> replays... and i really think that this is the a php issue... not >> symfony/propel issue.. >> >> Having said that... i have started working on a solution where the >> file upload is done in a separate action... which then forwards the >> user to another action that processes the data in the uploaded file x >> amount of records at a time... refreshing the page and passing the >> pager's current status as a request parameter, so processing a file >> with 10000 1000 records at time would mean 10 page refreshes... >> >> Anyway if anyone gets a chance let me know what you think about this... >> >> Regards >> Richard >> >> Btw... to anyone who i offended earlier... my deepest apologies... >> sometimes frustration gets the better of me... >> >> >> >> On Mon, Apr 12, 2010 at 10:01 PM, Tony Piper <tpi...@tpiper.com> wrote: >> > Just to chip in with another possible cause: >> >> > I see you're doing all of this in a controller (and others have talked >> > about whether that's a good idea). Have you got the web debug toolbar >> > enabled? If so, it could be keeping a log of all DB queries executed >> > in your action. I had something similar with a batch job in 1.3/1.4. >> >> > There 's a trac ticket which gives some workarounds: >> >http://trac.symfony-project.org/ticket/3039 >> >> > Hope this helps. >> >> > cheers, >> >> > Tony. >> >> > On Apr 12, 8:53 am, Richard U <richard....@gmail.com> wrote: >> >> Dennis thats the function if you want to have a look at it... >> >> >> public function executeIndex(sfWebRequest $request) { >> >> >> ini_set('memory_limit','512M'); >> >> set_time_limit(60*5); >> >> sfConfig::set('sf_logging_enabled', false); >> >> sfConfig::set('sf_web_debug', false); >> >> >> $timer = new Timer; >> >> $timer->start_time(); >> >> >> $this->submited_bulk_stats = >> >> $request->getParameterHolder()->get('bulk_stats'); >> >> >> $this->form = new BulkAddStatsForm(); >> >> >> $this->count_updated_records = 0; >> >> $this->count_inserted_records = 0; >> >> >> if ($request->isMethod('post')) >> >> { >> >> >> >> $this->form->bind($request->getParameter('bulk_stats'), >> >> $request->getFiles('bulk_stats')); >> >> if ($this->form->isValid()) >> >> { >> >> $values = $this->form->getValues(); >> >> >> $file = $this->form->getValue('file'); >> >> $file_name = >> >> str_replace($file->getOriginalExtension(), '', >> >> $file->getOriginalName()); >> >> $save_to = >> >> sfConfig::get('sf_upload_dir').DIRECTORY_SEPARATOR.'uploaded_'.$file_name.' >> >> _'.date("dmYHis").$file->getOriginalExtension(); >> >> $file->save($save_to); >> >> >> echo "<br>File saved: $save_to"; >> >> echo "<br>File size: ".filesize($save_to); >> >> >> // get accounts >> >> $accounts = AccountsPeer::doSelect(new >> >> Criteria()); >> >> $re_indexed_accounts = >> >> $re_indexed_accounts_by_id = array(); >> >> foreach ($accounts as $tmp_account) { >> >> >> >> $re_indexed_accounts[$tmp_account->getAffName().$tmp_account->getAffUser()] >> >> = $tmp_account; >> >> >> >> $re_indexed_accounts_by_id[$tmp_account->getId()] = $tmp_account; >> >> } >> >> >> // get parking programs >> >> $parking_programs = >> >> ParkingProgramPeer::doSelect(new Criteria()); >> >> $re_indexed_parking_programs = array(); >> >> foreach ($parking_programs as $pp) { >> >> >> >> $re_indexed_parking_programs[$pp->getId()] = $pp; >> >> } >> >> >> // Process the uploaded file >> >> $objReader = new >> >> PHPExcel_Reader_Excel2007; >> >> $objPHPExcel = $objReader->load($save_to); >> >> >> // Define allowed columns >> >> $allowed_columns = >> >> array('domain','views','clicks','revenue','currency','date','park','account >> >> '); >> >> $re_indexed_data = array(); >> >> $data = >> >> $objPHPExcel->getActiveSheet()->toArray(); >> >> >> // Get db connections since we are using >> >> two different db's here >> >> $connection_cdb = Propel::getConnection(); >> >> $connection_stats = >> >> Propel::getConnection('cdb_stats'); >> >> >> // get domains >> >> $domains_re_indexed = array(); >> >> $query = 'SELECT * FROM domains'; >> >> $statement = $connection_cdb->prepare($query); >> >> $statement->execute(); >> >> while ($dObj = >> >> $statement->fetch(PDO::FETCH_OBJ)) >> >> { >> >> $domains_re_indexed[$dObj->dn] = >> >> array('domain_id'=>$dObj->domain_ID, 'deal_id'=>$dObj->deal_id); >> >> } >> >> >> try { >> >> $validated_data = array(); >> >> $sfLogger = >> >> sfContext::getInstance()->getLogger(); >> >> >> foreach($data as $row_number => >> >> $row) { >> >> $batch_options = array(); >> >> >> // Check its not an empty >> >> row >> >> if (!strlen(implode("", >> >> $row))) { >> >> continue; >> >> } >> >> >> foreach($row as >> >> $col_number => $col) { >> >> echo "<br>MEMORY: >> >> ".memory_get_usage(); >> >> if ($row_number >> >> == 1) { >> >> $col = >> >> strtolower($col); >> >> $key = >> >> array_search($col, $allowed_columns); >> >> if ($key >> >> !== false) { >> >> >> >> $data_indexes[$col_number] = $allowed_columns[$key]; >> >> } >> >> } else { >> >> if >> >> (isset($data_indexes[$col_number]) && >> >> strlen($data_indexes[$col_number]) && strlen($col)) { >> >> >> >> if ($data_indexes[$col_number] == "date") { >> >> >> >> $col = date("Y-m-d", strtotime(str_replace("/", "-", $col))); >> >> } >> >> >> >> $batch_options[$data_indexes[$col_number]] = $col; >> >> } >> >> } >> >> } >> >> >> if ($row_number > 1) { >> >> >> // Park info now >> >> comes from the form >> >> if >> >> (!isset($values['parking_program']) || >> >> !strlen($values['parking_program'])) { >> >> throw new >> >> Exception('Missing or invalid parking program'); >> >> >> >> ... >> >> read more » > > -- > If you want to report a vulnerability issue on symfony, please send it to > security at symfony-project.com > > You received this message because you are subscribed to the Google > Groups "symfony users" group. > To post to this group, send email to symfony-users@googlegroups.com > To unsubscribe from this group, send email to > symfony-users+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/symfony-users?hl=en > > To unsubscribe, reply using "remove me" as the subject. > -- If you want to report a vulnerability issue on symfony, please send it to security at symfony-project.com You received this message because you are subscribed to the Google Groups "symfony users" group. To post to this group, send email to symfony-users@googlegroups.com To unsubscribe from this group, send email to symfony-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/symfony-users?hl=en
