Re: [Tutor] Most efficient way to read large csv files with properly converted mixed data types.

[Alan Gauld via Tutor]

On 25/06/16 08:04, Ek Esawi wrote:

> genfromtxt or (2) looping through each line in the file and split, strip,
> and assign data type to each entry.
> 
> I am wondering if there is a better and more efficient alternative,
> especially to method 2 without using numpy or pandas. 

The csv module will be more reliable and probably faster than using
looping with split/strip. You will still need to do the data conversions
from strings to native data however. Depending how you currently do that
it may be possible to improve that process using
a mapping to function (similar to what genfromtext does).

> Alan Gauld mentioned namedtuples for another question. I read a little 
> about collections and in particular namedtuples
> but was not sure how to apply theme here, if they
> are applicable to begin with.

named tuples provide an alternative to dictionaries for read-only
data and are more readable than standard tuples, but whether they
have a role to play in this particular case is not clear, we'd need
to know a lot more about how you plan to access the data once its converted.

-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos


___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] For-else... Any other handy constructs hiding in Python?

[Michael Selik]

On Fri, Jun 24, 2016 at 11:58 AM Alex Hall  wrote:

> I know loops, comprehensions, ifs, and the like,
> but I never knew for-else was available. Are there other constructs that I
> may have missed?
>

Are you familiar with context managers?
https://www.python.org/dev/peps/pep-0343/
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] Most efficient way to read large csv files with properly converted mixed data types.

[Ek Esawi]

Hi All--



My work involves reading large csv files with mixed data types (integer,
float, string, time and date). I was able to accomplish the task using (1)
genfromtxt or (2) looping through each line in the file and split, strip,
and assign data type to each entry.



I am wondering if there is a better and more efficient alternative,
especially to method 2 without using numpy or pandas. Alan Gauld mentioned
namedtuples for another question. I read a little about collections and in
particular namedtuples but was not sure how to apply theme here, if they
are applicable to begin with.



Thanks in advance--EKE



An example of a file:



A B C D E

1  2.3   ‘aa’  10/01/2016  12:30

4  25.6 ‘bb’  02/02/2015  1:30
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor