Hi Xiao,
Orchid Fairy (兰花仙子) wrote:
reach each line of every file (many files, each is gziped)
look for special info (like IP, request url, session_id, datetime etc).
count for them and write the result into a database
generate the daily report and monthly report
I'm afraid perl can't finish the daily job so I want to know the speed
difference between perl and C for this case.
You've seem to have received a lot of advice, so I'm not sure if this
will help. As others have said, if you are doing a regular expression,
then Perl will be easier to code than C.
Also, with both languages, you will have an IO bottleneck. If you read
and write information at the same time (i.e., you do not need to buffer
any information), then Perl should be fine. But, if you have to "count"
them, I presume you will need to store such information in a data
structure. If so, then with 1 TB of data, you may find that writing
your own data structure in C/C++ might be faster than Perl.
Three things you might want to consider:
1) If these are server logs, configure Apache so that it writes out the
logs in a friendly way. (i.e., tab-separated or something so that you
don't need a regular expression to get each field)
2) If #1 is not possible, then use Perl to make the data "friendly" for
a C program to sum up. Then you would use both Perl and C -- I've done
this before for, actually, server logs, and it was getting the best of
both worlds.
3) There are zlib libraries for both C and Perl; make sure you use them
so that you do in-memory decompression. If you expand to disk, then
you're going to add more I/O.
Good luck!
Ray
--
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
http://learn.perl.org/
