just add a simple colum as updated... and then if the record is process jst set the flag as updated. and select only records which are not flared in site A. and then limit the query for like 10000 and run it multiple times .. this is the simplest way without go in to more technical stuff.
this is in case u don't have more time to search and research around. On Sun, Jun 12, 2011 at 9:07 AM, AaronC <[email protected]> wrote: > Thanks guys. > > Looking good here now. I followed Stig's suggestions, and without > posting code here is what I did: > > NOTE: Yes, all tables are MySQL. No Framework for this. Straight PHP > as a one-off tool. > > 1. Get count of relevant records from Site A table > 2. Set a var to hold the limit of data blocks to process. (50k works > in my case) > 3. Run a quick while loop to build an array containing "LIMIT $max > OFFSET $current_start" strings. Like so: > // Build array of start/limit queries > $start = 0; > $block_max = 50000; > $block_limit_array = array(); > while($start <= $record_count) { > $limit = $start + $block_max - 1; > $block_limit_array[] = "LIMIT $block_max OFFSET $start"; > $start = $limit + 1; > } > 4. Run a foreach on the array created at 3, using the strings to make > a limiting query > 5. Process results > 6. Before the end of each loop, unset sql result variables > > I've just run this on the 350k table and it's perfect. Now to run on > the 1.3 million. > > Thanks for your help > Aaron > > -- > NZ PHP Users Group: http://groups.google.com/group/nzphpug > To post, send email to [email protected] > To unsubscribe, send email to > [email protected] > -- NZ PHP Users Group: http://groups.google.com/group/nzphpug To post, send email to [email protected] To unsubscribe, send email to [email protected]
