wow I feel like an @$$, sorry Kiril, didn't mean to call you kirk....
that's what I get for answering emails 5 minutes after waking up
james
On Jan 31, 2008, at 1:05 PM, James wrote:
> Hi Kirk,
>
> Before I do that, do you think it'll be able to handle the 1.4
> million records that it will eventually need to handle? I'd rather
> you not do the work if it's not going to do what we need to in
> production on the full data set.
>
> While you answer I'll work on getting that table statement.
> James
>
>
> On Jan 31, 2008, at 9:58 AM, Kiril Angov wrote:
>
>>
>> James, on first look your code does not seem to be very efficient and
>> what you are trying to accomplish can be done with Propel. If you
>> send
>> me the create sql statement for the tables involved I can rewrite you
>> script to show you what I mean.
>>
>> Kupo
>>
>> James wrote:
>>> Hey Guys,
>>>
>>> OK, so I am bringing this back up for another reason. Same data,
>>> same
>>> problem, different situation.
>>>
>>> I have a subset of the 1.4M records. 24,000 records roughly.
>>> One of
>>> these fields is a 9 digit code. Which is really 3 codes in one.
>>>
>>> 111222333 would be 3 codes 111, 222, 333 I need to create a map
>>> table between what these codes represent and the model for this
>>> data.
>>>
>>> The problem is when I get to 9750 inserts into the table the script
>>> errors out with a memory problem. I'm releasing all my memory in
>>> the
>>> script and would expect symfony/propel to be able to handle this.
>>>
>>> PHP Fatal error: Allowed memory size of 134217728 bytes exhausted
>>> (tried to allocate 7420 bytes) in /usr/share/pear/symfony/vendor/
>>> propel/util/BasePeer.php on line 444
>>>
>>> Fatal error: Allowed memory size of 134217728 bytes exhausted (tried
>>> to allocate 7420 bytes) in /usr/share/pear/symfony/vendor/propel/
>>> util/
>>> BasePeer.php on line 444
>>>
>>> Now I know I could get around this by using straight php, but I
>>> would
>>> like to use the tools the symfony provides with the data models to
>>> keep continuity straight. This isn't going to be the only data I
>>> need
>>> to manipulate, so I can see this being a major pain.
>>>
>>> I've included the code below. I am using Symfony 1.0.10
>>>
>>> Ideas?
>>>
>>> Thanks,
>>> James
>>>
>>>
>>> =
>>> =
>>> ====================================================================
>>>
>>> $countStart = 0;
>>> $countEnd = 23334;
>>>
>>> while($countStart != $countEnd)
>>> {
>>> query
>>> // batch process here
>>> $c = new Criteria();
>>> $c->setLimit(1);
>>> $c->setOffset($countStart++);
>>> $dataModel = DataModelPeer::doSelect($c);
>>>
>>> $actCodes = array();
>>> $actCodes[] = substr($dataMode[0]->getActCode(),0,3);
>>> $actCodes[] = substr($dataMode[0]->getActCode(),3,3);
>>> $actCodes[] = substr($dataMode[0]->getActCode(),6,3);
>>>
>>> foreach($actCodes as $actCode){
>>> $c2 = new Criteria();
>>> $c2->add(MelissaActcodePeer::CODE, $actCode);
>>> $ntee = MelissaActcodePeer::doSelect($c2);
>>>
>>> if(count($ntee)){
>>> $map = new ModelDataMelissaActcodeMap();
>>> $map->setModelDataId($dataMode[0]->getId());
>>> $map->setMelissaActcodeId($ntee[0]->getId());
>>> $map->save();
>>> $map=null;
>>> }
>>> }
>>> unset($c);
>>> unset($c2);
>>> unset($dataModel);
>>> unset($ntee);
>>> unset($map);
>>> unset($actCodes);
>>>
>>> }
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Dec 17, 2007, at 4:20 AM, James wrote:
>>>
>>>
>>>> Hi,
>>>>
>>>> It's me again. v1.0.8
>>>>
>>>> The whole reason I started playing with Symfony was for work. I've
>>>> made a personal app to get familiar with Symfony and tonight I
>>>> decided to take it to the work level. We have a database table
>>>> with
>>>> 1.4M records. We want to index these records in order to do a
>>>> search on them. I was following the askeet example modified for
>>>> our
>>>> data set I got it to work adding weighs on the words in the
>>>> company
>>>> names, but hit an error where I ran out of memory. This was not
>>>> very surprising. 1.4M records is a lot of data.
>>>>
>>>> I figured that the best possible way to handle this would be to go
>>>> through each record on at a time. It may take a little while to
>>>> run, but I knew that one record was within the memory limits.
>>>> Here's my code the rest is quite literally out of the askeet
>>>> example except it's not in an object it's simply functions in the
>>>> batch script.
>>>>
>>>> $countStart=0;
>>>> $countEnd=1400000;
>>>>
>>>> while($countStart != $countEnd){
>>>>
>>>> // batch process here
>>>> $c = new Criteria();
>>>> $c->setLimit(1);
>>>> $c->setOffset($countStart++);
>>>> $company = CompanyPeer::doSelect($c);
>>>>
>>>> $companyName = $company[0]->getCompany();
>>>> $companyId = $company[0]->getId();
>>>>
>>>> $companyNameText = str_repeat(" ".$companyName, 2);
>>>>
>>>> $stemmed = stemPhrase($companyNameText);
>>>>
>>>> $words = array_count_values($stemmed);
>>>>
>>>> updateSearchIndex($companyId,$words);
>>>>
>>>> // Point everything at null.
>>>> $c = null;
>>>> $company = null;
>>>> $companyName = null;
>>>> $companyId = null;
>>>> $componyNameText = null;
>>>> $stemmed = null;
>>>> $words = null;
>>>>
>>>> }
>>>>
>>>> When I run this code It does what it is suppose too up to about
>>>> 19000 lines. Then it dumps and says that it ran out of memory
>>>> again. My question is basically how can I work with so many
>>>> records
>>>> in Symfony without running out of memory, and without increasing
>>>> how
>>>> much memory php is allowed to use. A similar method worked for me
>>>> in php, so I am wondering what I may be missing.
>>>>
>>>> Thanks,
>>>> James
>>>>
>>>
>>>
>>>>
>>>
>>>
>>
>>
>> >>
>
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"symfony users" group.
To post to this group, send email to symfony-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/symfony-users?hl=en
-~----------~----~----~----~------~----~------~--~---
