On 3 Nov 2015, at 6:03 pm, Chris Barker - NOAA Federal <chris.bar...@noaa.gov>
wrote:
>
> I was more aiming to point out a situation where the NumPy's text file reader
> was significantly better than the Pandas version, so we would want to make
> sure that we properly benchmark any significant changes to NumPy's text
> reading code. Who knows where else NumPy beats Pandas?
> Indeed. For this example, I think a fixed-with reader really is a different
> animal, and it's probably a good idea to have a high performance one in
> Numpy. Among other things, you wouldn't want it to try to auto-determine data
> types or anything like that.
>
> I think what's on the table now is to bring in a new delimited reader -- I.e.
> CSV in its various flavors.
>
To add my own handful of change or at least another data point, I had been
looking into both
the pandas and the Astropy fast readers as a fast loadtxt/genfromtxt
replacement; at the time
I found the Astropy cparser source somewhat easier to dig into, although
looking now Pandas'
parser.pyx seems clear enough as well.
Some comparison of the two can be found at
http://astropy.readthedocs.org/en/stable/io/ascii/fast_ascii_io.html#speed-gains
Unfortunately the Astropy fast reader currently does not support fixed-width
format either, and
adding this functionality would require modifications to the tokenizer C code -
not sure how
extensive.
Cheers,
Derek
_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion
