Hi Kirk,
Before I do that, do you think it'll be able to handle the 1.4 million
records that it will eventually need to handle? I'd rather you not do
the work if it's not going to do what we need to in production on the
full data set.
While you answer I'll work on getting that table statement.
James
On Jan 31, 2008, at 9:58 AM, Kiril Angov wrote:
>
> James, on first look your code does not seem to be very efficient and
> what you are trying to accomplish can be done with Propel. If you send
> me the create sql statement for the tables involved I can rewrite you
> script to show you what I mean.
>
> Kupo
>
> James wrote:
>> Hey Guys,
>>
>> OK, so I am bringing this back up for another reason. Same data,
>> same
>> problem, different situation.
>>
>> I have a subset of the 1.4M records. 24,000 records roughly. One
>> of
>> these fields is a 9 digit code. Which is really 3 codes in one.
>>
>> 111222333 would be 3 codes 111, 222, 333 I need to create a map
>> table between what these codes represent and the model for this data.
>>
>> The problem is when I get to 9750 inserts into the table the script
>> errors out with a memory problem. I'm releasing all my memory in the
>> script and would expect symfony/propel to be able to handle this.
>>
>> PHP Fatal error: Allowed memory size of 134217728 bytes exhausted
>> (tried to allocate 7420 bytes) in /usr/share/pear/symfony/vendor/
>> propel/util/BasePeer.php on line 444
>>
>> Fatal error: Allowed memory size of 134217728 bytes exhausted (tried
>> to allocate 7420 bytes) in /usr/share/pear/symfony/vendor/propel/
>> util/
>> BasePeer.php on line 444
>>
>> Now I know I could get around this by using straight php, but I would
>> like to use the tools the symfony provides with the data models to
>> keep continuity straight. This isn't going to be the only data I
>> need
>> to manipulate, so I can see this being a major pain.
>>
>> I've included the code below. I am using Symfony 1.0.10
>>
>> Ideas?
>>
>> Thanks,
>> James
>>
>>
>> =
>> =====================================================================
>>
>> $countStart = 0;
>> $countEnd = 23334;
>>
>> while($countStart != $countEnd)
>> {
>> query
>> // batch process here
>> $c = new Criteria();
>> $c->setLimit(1);
>> $c->setOffset($countStart++);
>> $dataModel = DataModelPeer::doSelect($c);
>>
>> $actCodes = array();
>> $actCodes[] = substr($dataMode[0]->getActCode(),0,3);
>> $actCodes[] = substr($dataMode[0]->getActCode(),3,3);
>> $actCodes[] = substr($dataMode[0]->getActCode(),6,3);
>>
>> foreach($actCodes as $actCode){
>> $c2 = new Criteria();
>> $c2->add(MelissaActcodePeer::CODE, $actCode);
>> $ntee = MelissaActcodePeer::doSelect($c2);
>>
>> if(count($ntee)){
>> $map = new ModelDataMelissaActcodeMap();
>> $map->setModelDataId($dataMode[0]->getId());
>> $map->setMelissaActcodeId($ntee[0]->getId());
>> $map->save();
>> $map=null;
>> }
>> }
>> unset($c);
>> unset($c2);
>> unset($dataModel);
>> unset($ntee);
>> unset($map);
>> unset($actCodes);
>>
>> }
>>
>>
>>
>>
>>
>>
>>
>>
>> On Dec 17, 2007, at 4:20 AM, James wrote:
>>
>>
>>> Hi,
>>>
>>> It's me again. v1.0.8
>>>
>>> The whole reason I started playing with Symfony was for work. I've
>>> made a personal app to get familiar with Symfony and tonight I
>>> decided to take it to the work level. We have a database table with
>>> 1.4M records. We want to index these records in order to do a
>>> search on them. I was following the askeet example modified for our
>>> data set I got it to work adding weighs on the words in the company
>>> names, but hit an error where I ran out of memory. This was not
>>> very surprising. 1.4M records is a lot of data.
>>>
>>> I figured that the best possible way to handle this would be to go
>>> through each record on at a time. It may take a little while to
>>> run, but I knew that one record was within the memory limits.
>>> Here's my code the rest is quite literally out of the askeet
>>> example except it's not in an object it's simply functions in the
>>> batch script.
>>>
>>> $countStart=0;
>>> $countEnd=1400000;
>>>
>>> while($countStart != $countEnd){
>>>
>>> // batch process here
>>> $c = new Criteria();
>>> $c->setLimit(1);
>>> $c->setOffset($countStart++);
>>> $company = CompanyPeer::doSelect($c);
>>>
>>> $companyName = $company[0]->getCompany();
>>> $companyId = $company[0]->getId();
>>>
>>> $companyNameText = str_repeat(" ".$companyName, 2);
>>>
>>> $stemmed = stemPhrase($companyNameText);
>>>
>>> $words = array_count_values($stemmed);
>>>
>>> updateSearchIndex($companyId,$words);
>>>
>>> // Point everything at null.
>>> $c = null;
>>> $company = null;
>>> $companyName = null;
>>> $companyId = null;
>>> $componyNameText = null;
>>> $stemmed = null;
>>> $words = null;
>>>
>>> }
>>>
>>> When I run this code It does what it is suppose too up to about
>>> 19000 lines. Then it dumps and says that it ran out of memory
>>> again. My question is basically how can I work with so many records
>>> in Symfony without running out of memory, and without increasing how
>>> much memory php is allowed to use. A similar method worked for me
>>> in php, so I am wondering what I may be missing.
>>>
>>> Thanks,
>>> James
>>>
>>
>>
>>>
>>
>>
>
>
> >
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"symfony users" group.
To post to this group, send email to symfony-users@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/symfony-users?hl=en
-~----------~----~----~----~------~----~------~--~---
