> *Handling (very) large files with PHP* > > Hello, I am planning a project in PHP, and I have few unsolved issues that > I'd like you to help me... > > The project will start by loading a file of about 50GB. > The file has a many objects with a pattern, for example, > > > Name: Joe > Joe likes to eat > ------------------- > Name: Daniel > Daniel likes to ask question on the PHP Mailing List > > > Anyway, so, I am going to convert it into a database, and I insist on using > PHP for this. > > So the questions are, > How would I open the file? will fopen fread($file, 1024) will work? if then, > how would I find the seperator, "------------------", without taking too > many resources? > I'll have a dedicated server for this project so I could use exec, so I am > wondering if I should use exec to split the file? > How many hours or days do you think it will take me to insert all of the > data, if I have about 8,000,000,000 (8 billion/milliard) entries (objects)? > > After I insert all the data, I'll have to start working with it as well - > for example, having a list of all people and what comes after the word > "likes" in their entry. > > What do you suggest? I am concerened I might not be able to fully acomplish > both high speed with working (example above) and both high speed when > watching the data and adding more "works" (as stated above) with PHP. What > do you think? > Since inserting to the database, after considering it, will probably be with > C. But if I wish to work with it - will PHP be good? > > What database should I use for so much info? > > > Thanks, Daniel >
hi Daniel, Interesting task you got there. The code below should solve your problem. $fh = fopen('data.txt', 'r'); $row = array(); while (($line = fgets($fh, 1024)) !== false) { if ($line === '-------------------') { // save $row into database $row = array(); continue; } if (strpos($line,'Name: ') !== false) { $row['name'] = str_replace('Name: ', '', $line); continue; } $row['likes'] = str_replace($row['name'] . ' likes ', '', $line); } fclose($fh); your database fields should include id --> autoincrement, primary name --> varchar(64) likes --> text now you should be able to do simple select query that pulls a list of names with likes not empty. please clock it for me. i would like to know how long it takes but the code should be fast. thanks and good luck. Virgil http://www.jampmark.com -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php