On 08/10/2013 21:46, Leena Gupta wrote:
Hello,
Looking for some inputs on Python's csv processing feature.
I need to process a large csv file every 5-10 minutes. The file could
contain 3mill to 10 mill rows and size could be 6MB to 10MB(+). As part
of the processing, I need to sum up a number value by grouping on
certain attributes and store the output in a datastore. I wanted to know
if Python is recommended and can it be used for processing data in csv
files of this size? Any issues that we need to be aware of? I believe
Python has a csv library as well.
Thanks!
1) As others have already stated Python can easily handle this.
2) I'd recommend using a csv.DictReader as it makes your code easier to
read, please see
http://docs.python.org/3/library/csv.html#csv.DictReader. Note the
fieldnames parameter is very useful if your files don't have header rows.
--
Roses are red,
Violets are blue,
Most poems rhyme,
But this one doesn't.
Mark Lawrence
_______________________________________________
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
