Re: [PHP] CSV file
Jonathan Tapicer wrote: > Hi, > > I don't say that reading the whole file into memory is the best > option, I just say that your approach goes to the disk twice, if you > have a 100MB file, you are reading 200MB from disk (unless there are > some pages cached in physical memory, but you never can know, in the > best case all the pages will be cached and you will only read 100MB > from disk); and mine, if you have 100MB or more physical memory > (available for the script) will only go once. Of course, if you have > low physical memory, in the worst case you will end up reading 200MB > from disk, because all the pages will go to the pagefile. > > So, both approaches have a worst case of reading approximately two > times the file size from disk and a best case of reading it > approximately only once from disk, and since you never know the actual > conditions of the memory, you can't say which approach is better > without additional information. > > In the general case, I think that your approach is better for large > files and mine for small files, with the size barrier depending on the > memory set up and current conditions at the time the script runs. > > This discussion went in a different direction considering the initial > question of Alain, but this could always be useful for someone. > > Jonathan > > On Thu, Jun 25, 2009 at 5:37 AM, Richard Heyes wrote: >> Hi, >> >>> Well, you are reading the whole file there (and throwing the data you >>> read not assigning the fgets result to anything), and then to store it >>> in the database you need to read it again, so you read the file twice. >>> It will probably better to store the data you read the first time in >>> an array and then store it in the database, that way you read it only >>> once. >> No, it's not. If the file is large then you could end up reading megs >> into memory. If physical memory is low then the pagefile will come >> into play and you'll get a lot of disk accesses. Reading 1 line at a >> time is far more efficient and with larger files will be faster. >> >> -- >> Richard Heyes >> HTML5 graphing: RGraph (www.rgraph.net - updated 20th June) >> PHP mail: RMail (www.phpguru.org/rmail) >> PHP datagrid: RGrid (www.phpguru.org/rgrid) >> PHP Template: RTemplate (www.phpguru.org/rtemplate) >> PHP SMTP: http://www.phpguru.org/smtp >> Stuart is still the winner. -- Thanks! -Shawn http://www.spidean.com -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] CSV file
Hi, I don't say that reading the whole file into memory is the best option, I just say that your approach goes to the disk twice, if you have a 100MB file, you are reading 200MB from disk (unless there are some pages cached in physical memory, but you never can know, in the best case all the pages will be cached and you will only read 100MB from disk); and mine, if you have 100MB or more physical memory (available for the script) will only go once. Of course, if you have low physical memory, in the worst case you will end up reading 200MB from disk, because all the pages will go to the pagefile. So, both approaches have a worst case of reading approximately two times the file size from disk and a best case of reading it approximately only once from disk, and since you never know the actual conditions of the memory, you can't say which approach is better without additional information. In the general case, I think that your approach is better for large files and mine for small files, with the size barrier depending on the memory set up and current conditions at the time the script runs. This discussion went in a different direction considering the initial question of Alain, but this could always be useful for someone. Jonathan On Thu, Jun 25, 2009 at 5:37 AM, Richard Heyes wrote: > Hi, > >> Well, you are reading the whole file there (and throwing the data you >> read not assigning the fgets result to anything), and then to store it >> in the database you need to read it again, so you read the file twice. >> It will probably better to store the data you read the first time in >> an array and then store it in the database, that way you read it only >> once. > > No, it's not. If the file is large then you could end up reading megs > into memory. If physical memory is low then the pagefile will come > into play and you'll get a lot of disk accesses. Reading 1 line at a > time is far more efficient and with larger files will be faster. > > -- > Richard Heyes > HTML5 graphing: RGraph (www.rgraph.net - updated 20th June) > PHP mail: RMail (www.phpguru.org/rmail) > PHP datagrid: RGrid (www.phpguru.org/rgrid) > PHP Template: RTemplate (www.phpguru.org/rtemplate) > PHP SMTP: http://www.phpguru.org/smtp > -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] CSV file
Hi, > Well, you are reading the whole file there (and throwing the data you > read not assigning the fgets result to anything), and then to store it > in the database you need to read it again, so you read the file twice. > It will probably better to store the data you read the first time in > an array and then store it in the database, that way you read it only > once. No, it's not. If the file is large then you could end up reading megs into memory. If physical memory is low then the pagefile will come into play and you'll get a lot of disk accesses. Reading 1 line at a time is far more efficient and with larger files will be faster. -- Richard Heyes HTML5 graphing: RGraph (www.rgraph.net - updated 20th June) PHP mail: RMail (www.phpguru.org/rmail) PHP datagrid: RGrid (www.phpguru.org/rgrid) PHP Template: RTemplate (www.phpguru.org/rtemplate) PHP SMTP: http://www.phpguru.org/smtp -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] CSV file
Well, you are reading the whole file there (and throwing the data you
read not assigning the fgets result to anything), and then to store it
in the database you need to read it again, so you read the file twice.
It will probably better to store the data you read the first time in
an array and then store it in the database, that way you read it only
once.
Anyway, doing it by file size is better.
On Wed, Jun 24, 2009 at 6:19 PM, Richard Heyes wrote:
> Hi,
>
>> To do the line count first, you have to read the whole file, how would
>> you do it?
>
> Something like this:
>
> $fp = fopen('/tmp/foo', 'r');
> $count = 0;
>
> while (!feof($fp)) {
> fgets($fp);
> ++$count;
> }
>
> --
> Richard Heyes
> HTML5 graphing: RGraph (www.rgraph.net - updated 20th June)
> PHP mail: RMail (www.phpguru.org/rmail)
> PHP datagrid: RGrid (www.phpguru.org/rgrid)
> PHP Template: RTemplate (www.phpguru.org/rtemplate)
> PHP SMTP: http://www.phpguru.org/smtp
>
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] CSV file
or $arr = file('foo.csv'); $count = count($arr);
On Wed, Jun 24, 2009 at 5:19 PM, Richard Heyes wrote:
> Hi,
>
>> To do the line count first, you have to read the whole file, how would
>> you do it?
>
> Something like this:
>
> $fp = fopen('/tmp/foo', 'r');
> $count = 0;
>
> while (!feof($fp)) {
> fgets($fp);
> ++$count;
> }
>
> --
> Richard Heyes
> HTML5 graphing: RGraph (www.rgraph.net - updated 20th June)
> PHP mail: RMail (www.phpguru.org/rmail)
> PHP datagrid: RGrid (www.phpguru.org/rgrid)
> PHP Template: RTemplate (www.phpguru.org/rtemplate)
> PHP SMTP: http://www.phpguru.org/smtp
>
> --
> PHP General Mailing List (http://www.php.net/)
> To unsubscribe, visit: http://www.php.net/unsub.php
>
>
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] CSV file
Hi,
> To do the line count first, you have to read the whole file, how would
> you do it?
Something like this:
$fp = fopen('/tmp/foo', 'r');
$count = 0;
while (!feof($fp)) {
fgets($fp);
++$count;
}
--
Richard Heyes
HTML5 graphing: RGraph (www.rgraph.net - updated 20th June)
PHP mail: RMail (www.phpguru.org/rmail)
PHP datagrid: RGrid (www.phpguru.org/rgrid)
PHP Template: RTemplate (www.phpguru.org/rtemplate)
PHP SMTP: http://www.phpguru.org/smtp
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] CSV file
To do the line count first, you have to read the whole file, how would you do it? On Wed, Jun 24, 2009 at 3:00 PM, Richard Heyes wrote: > Hi, > >> If you want to know how many lines there are *before* inserting to the >> database, you can't count "as you go", you have to either read the >> file twice or read it once, store it memory in a variable and then >> insert in the database. > > Sure you can, simply do the line count first, then the insert. If it's > a big file it will still be quicker than reading the whole thing into > memory. > > -- > Richard Heyes > HTML5 graphing: RGraph (www.rgraph.net - updated 20th June) > PHP mail: RMail (www.phpguru.org/rmail) > PHP datagrid: RGrid (www.phpguru.org/rgrid) > PHP Template: RTemplate (www.phpguru.org/rtemplate) > PHP SMTP: http://www.phpguru.org/smtp > -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] CSV file
Hi, > If you want to know how many lines there are *before* inserting to the > database, you can't count "as you go", you have to either read the > file twice or read it once, store it memory in a variable and then > insert in the database. Sure you can, simply do the line count first, then the insert. If it's a big file it will still be quicker than reading the whole thing into memory. -- Richard Heyes HTML5 graphing: RGraph (www.rgraph.net - updated 20th June) PHP mail: RMail (www.phpguru.org/rmail) PHP datagrid: RGrid (www.phpguru.org/rgrid) PHP Template: RTemplate (www.phpguru.org/rtemplate) PHP SMTP: http://www.phpguru.org/smtp -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] CSV file
Stuart wrote:
> 2009/6/24 Jonathan Tapicer :
>> If you want to know how many lines there are *before* inserting to the
>> database, you can't count "as you go", you have to either read the
>> file twice or read it once, store it memory in a variable and then
>> insert in the database.
>
> Do it in bytes rather than lines, then you don't waste time loading
> the file twice and you'll get the same end result.
>
> -Stuart
^Winner!
$file_name = '/path/to/file.csv';
$file_size = filesize($file_name);
$bytes_read = 0;
$new_percentage = 0;
$handle = fopen($file_name, 'r');
if ($handle) {
while (!feof($handle)) {
$line = fgets($handle);
$bytes_read = $bytes_read + strlen($line);
//insert line into DB
$new_percentage = round($bytes_read / $file_size * 100, 0);
if ($new_percentage > $percentage) {
$percentage = $new_percentage;
echo $percentage . "%\n";
flush();
}
}
fclose($handle);
}
--
Thanks!
-Shawn
http://www.spidean.com
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] CSV file
2009/6/24 Jonathan Tapicer : > If you want to know how many lines there are *before* inserting to the > database, you can't count "as you go", you have to either read the > file twice or read it once, store it memory in a variable and then > insert in the database. Do it in bytes rather than lines, then you don't waste time loading the file twice and you'll get the same end result. -Stuart -- http://stut.net/ > On Wed, Jun 24, 2009 at 11:12 AM, Richard Heyes wrote: >> Hi, >> >>> You can read the whole file (file_get_contents) and count the number >>> of "\n" in it, or read it line by line with fgets and store the lines >>> in an array, and then the number of lines is the count() of the array, >>> and you can use that array to store it in the database. >> >> If you have a billion line CSV then speed may suffer somewhat though. >> Best to still use fgets() or fgetcsv() and count as you go. >> >> -- >> Richard Heyes >> HTML5 graphing: RGraph (www.rgraph.net - updated 20th June) >> PHP mail: RMail (www.phpguru.org/rmail) >> PHP datagrid: RGrid (www.phpguru.org/rgrid) >> PHP Template: RTemplate (www.phpguru.org/rtemplate) >> PHP SMTP: http://www.phpguru.org/smtp >> > > -- > PHP General Mailing List (http://www.php.net/) > To unsubscribe, visit: http://www.php.net/unsub.php > > -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] CSV file
If you want to know how many lines there are *before* inserting to the database, you can't count "as you go", you have to either read the file twice or read it once, store it memory in a variable and then insert in the database. On Wed, Jun 24, 2009 at 11:12 AM, Richard Heyes wrote: > Hi, > >> You can read the whole file (file_get_contents) and count the number >> of "\n" in it, or read it line by line with fgets and store the lines >> in an array, and then the number of lines is the count() of the array, >> and you can use that array to store it in the database. > > If you have a billion line CSV then speed may suffer somewhat though. > Best to still use fgets() or fgetcsv() and count as you go. > > -- > Richard Heyes > HTML5 graphing: RGraph (www.rgraph.net - updated 20th June) > PHP mail: RMail (www.phpguru.org/rmail) > PHP datagrid: RGrid (www.phpguru.org/rgrid) > PHP Template: RTemplate (www.phpguru.org/rtemplate) > PHP SMTP: http://www.phpguru.org/smtp > -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] CSV file
Hi, > You can read the whole file (file_get_contents) and count the number > of "\n" in it, or read it line by line with fgets and store the lines > in an array, and then the number of lines is the count() of the array, > and you can use that array to store it in the database. If you have a billion line CSV then speed may suffer somewhat though. Best to still use fgets() or fgetcsv() and count as you go. -- Richard Heyes HTML5 graphing: RGraph (www.rgraph.net - updated 20th June) PHP mail: RMail (www.phpguru.org/rmail) PHP datagrid: RGrid (www.phpguru.org/rgrid) PHP Template: RTemplate (www.phpguru.org/rtemplate) PHP SMTP: http://www.phpguru.org/smtp -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] CSV file
You can read the whole file (file_get_contents) and count the number of "\n" in it, or read it line by line with fgets and store the lines in an array, and then the number of lines is the count() of the array, and you can use that array to store it in the database. Jonathan On Wed, Jun 24, 2009 at 9:05 AM, Alain Roger wrote: > Hi, > > i would like to import the content of a CSV file into my database. > but in order to show a progress bar to user (to let him know that process > is still working on) i would like to determine before to start, how many > records / lines are in the CSV file. > is there a way to do that ? > thanks a lot. > > -- > Alain > --- > Windows XP x64 SP2 / Fedora 10 KDE 4.2 > PostgreSQL 8.3.5 / MS SQL server 2005 > Apache 2.2.10 > PHP 5.2.6 > C# 2005-2008 > -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
[PHP] CSV file
Hi, i would like to import the content of a CSV file into my database. but in order to show a progress bar to user (to let him know that process is still working on) i would like to determine before to start, how many records / lines are in the CSV file. is there a way to do that ? thanks a lot. -- Alain --- Windows XP x64 SP2 / Fedora 10 KDE 4.2 PostgreSQL 8.3.5 / MS SQL server 2005 Apache 2.2.10 PHP 5.2.6 C# 2005-2008
Re: [PHP] CSV file with PHP
On Fri, 10 Oct 2003 18:09:14 -0400, Dan Anderson <[EMAIL PROTECTED]>
wrote:
while ($line = mysql_fetch_array($result))
{
if ($first)
{ foreach ($line as $key => $value)
{ if ($first)
{ fwrite($file_handle, $key); $first = FALSE; }
else
{ fwrite($file_handle, ", $key"); }
}
fwrite ($file_handle, "\n");
}
$number1 = TRUE;
foreach ($line as $value)
{
if ($number1) { fwrite ($file_hande, $value); $number1 = FALSE; } else {
fwiret ($file_hande, ", {$value}"; };
}
fwrite ($file_handle, "\n");
}
Here is a little more efficient code:
if ($row = mysql_fetch_assoc($result)) {
$line = '';
foreach ($row as $key => $value) {
$line .= "$key,";
}
fwrite($file_handle, substr($line, 0, -1)."\n");
do {
$line = '';
foreach ($row as $value) {
if (strpos($value, '"') ) {
$line .= sprintf('"%s",', str_replace('"', '\\"', $value));
} else {
$line .= "$value,";
}
}
fwrite($file_handle, substr($line, 0, -1)."\n");
} while ($line = mysql_fetch_assoc($result));
}
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] CSV file with PHP
You could do something like:
$result = mysql_query($query) or die(mysql_error()); // don't display
errors on production machines
$first = TRUE;
while ($line = mysql_fetch_array($result))
{
if ($first)
{
foreach ($line as $key => $value)
{
if ($first)
{ fwrite($file_handle, $key); $first = FALSE; }
else
{ fwrite($file_handle, ", $key"); }
}
fwrite ($file_handle, "\n");
}
$number1 = TRUE;
foreach ($line as $value)
{
if ($number1)
{ fwrite ($file_hande, $value); $number1 = FALSE; }
else
{ fwiret ($file_hande, ", {$value}"; };
}
fwrite ($file_handle, "\n");
}
That is pseudo code though. (i.e. I don't have time to test it and
debug it and make it nice and neat and readable and conform to your
specifications -- but it should give you a good idea of how to start).
-Dan
On Fri, 2003-10-10 at 17:38, Cesar Aracena wrote:
> Hi all,
>
> Does anybody knows how to make a CSV (comma separated values) file with
> PHP based on results fetched from MySQL? I need it to import it with
> Microsoft Outlook Express.
>
> Thanks in advanced,
>
> Cesar Aracena
> www.icaam.com.ar
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
[PHP] CSV file with PHP
Hi all, Does anybody knows how to make a CSV (comma separated values) file with PHP based on results fetched from MySQL? I need it to import it with Microsoft Outlook Express. Thanks in advanced, Cesar Aracena www.icaam.com.ar -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
