On Wed, Oct 09 2013,Leena Gupta wrote: > Hello, > > Looking for some inputs on Python's csv processing feature. > > I need to process a large csv file every 5-10 minutes. The file could > contain 3mill to 10 mill rows and size could be 6MB to 10MB(+). As part of > the processing, I need to sum up a number value by grouping on certain > attributes and store the output in a datastore. I wanted to know if Python > is recommended and can it be used for processing data in csv files of this > size? Any issues that we need to be aware of? I believe Python has a csv > library as well.
[snipped 6 lines] I've found pandas to be very useful for this. It provides good functions to read CSVs and higher order functions to sum the generated dataframes in pandas. sivaram -- _______________________________________________ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
