[issue28642] csv reader losing rows with big files and tab delimiter

2016-11-09 Thread Marc Garcia
Marc Garcia added the comment: I agree that for my case, I was using the wrong quoting parameter, and if I specify that my file has no quotes, it works as expected. But I still think that in a different case, when a file do have quotes, but they are not paired, it'd be better to rai

[issue28642] csv reader losing rows with big files and tab delimiter

2016-11-09 Thread Marc Garcia
Marc Garcia added the comment: I could research a bit more on the problem. This is a minimal code that reproduces what happened: from io import StringIO import csv csv_file = StringIO('''1\t"A 2\tB''') reader = csv.reader(csv_file

[issue28642] csv reader losing rows with big files and tab delimiter

2016-11-08 Thread Marc Garcia
Marc Garcia added the comment: Sorry, my fault. It looks like having quotes in the file was the problem. As mentioned, adding the quoting parameter fixes the problem. I'd assume that if quotes are not paired, csv should raise an exception. And I don't think that all the different

[issue28642] csv reader loosing rows with big files and tab delimiter

2016-11-08 Thread Marc Garcia
New submission from Marc Garcia: I'm using the csv module from Python standard library, to read a 1.4Gb file with 11,157,064 of rows. The file is the Geonames dataset for all countries, which can be freely downloaded [1]. I'm using this code to read it: import csv