[issue1818] Add named tuple reader to CSV module

Change by Raymond Hettinger : -- stage: patch review -> resolved status: open -> closed ___ Python tracker ___ ___ Python-bugs-list

[issue1818] Add named tuple reader to CSV module

Skip Montanaro added the comment: FWIW, I relinquished my check-in privileges quite awhile ago. This should almost certainly no longer be assigned to me. S -- ___ Python tracker

[issue1818] Add named tuple reader to CSV module

Change by Raymond Hettinger : -- priority: low -> normal versions: +Python 3.8 -Python 3.5 ___ Python tracker ___

[issue1818] Add named tuple reader to CSV module

Changes by Cédric Krier cedric.kr...@b2ck.com: -- nosy: +ced ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1818 ___ ___ Python-bugs-list mailing

[issue1818] Add named tuple reader to CSV module

Skip Montanaro added the comment: I looked at this six years ago. I still haven't found a situation where I pined for a NamedTupleReader. That said, I have no objection to committing it if others, more well-versed in current Python code and NamedTuples than I gives it a pass. Note that I

[issue1818] Add named tuple reader to CSV module

Ilia Kurenkov added the comment: Friendly reminder that this exists. I know everyone's busy and this is marked as low-priority, but I'm gonna keep bumping this till we add a solution :) -- ___ Python tracker rep...@bugs.python.org

[issue1818] Add named tuple reader to CSV module

Raymond Hettinger added the comment: Skip or Barry, do you want to look at this? -- assignee: - skip.montanaro nosy: +skip.montanaro stage: needs patch - patch review versions: -Python 3.3 ___ Python tracker rep...@bugs.python.org

[issue1818] Add named tuple reader to CSV module

Ilia Kurenkov added the comment: As my contribution during the sprints at PyCon 2015, I've tweaked Jervis's patch a little and updated the tests/docs to work with Python 3.5. My only real change was placing the basic reader object inside a generator expression that filters out empty lines.

[issue1818] Add named tuple reader to CSV module

Daniel Lenski added the comment: Here's the class I have been using for reading namedtuples from CSV files: from collections import namedtuple from itertools import imap import csv class CsvNamedTupleReader(object): __slots__ = ('_r', 'row', 'fieldnames') def

[issue1818] Add named tuple reader to CSV module

Changes by Mark Lawrence breamore...@yahoo.co.uk: -- nosy: -BreamoreBoy ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1818 ___ ___

[issue1818] Add named tuple reader to CSV module

Changes by Andrew Svetlov andrew.svet...@gmail.com: -- nosy: +asvetlov ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1818 ___ ___ Python-bugs-list

[issue1818] Add named tuple reader to CSV module

Changes by Ahsan Nawroj ahsan.naw...@gmail.com: -- nosy: +ainur0160 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1818 ___ ___ Python-bugs-list

[issue1818] Add named tuple reader to CSV module

Changes by Ahsan Nawroj ahsan.naw...@gmail.com: -- nosy: -ainur0160 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1818 ___ ___ Python-bugs-list

[issue1818] Add named tuple reader to CSV module

Raymond Hettinger rhettin...@users.sourceforge.net added the comment: Unassigning, this needs fresh thought and a fresh patch from someone who can devote a little deep thinking on how to solve this problem cleanly. In the meantime, it is no problem to simply cast the CSV tuples into named

[issue1818] Add named tuple reader to CSV module

Mark Lawrence breamore...@yahoo.co.uk added the comment: Closing as no response to msg110598. -- status: pending - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1818 ___

[issue1818] Add named tuple reader to CSV module

Raymond Hettinger rhettin...@users.sourceforge.net added the comment: Re-opening because we ought to do something along these lines at some point. The DictReader and DictWriter are inadequate for preserving order and they are unnecessarily memory intensive (one dict per record). FWIW, the

[issue1818] Add named tuple reader to CSV module

Mark Lawrence breamore...@yahoo.co.uk added the comment: I suggest that this is closed unless anyone shows an active interest in it. -- nosy: +BreamoreBoy status: open - pending versions: +Python 3.2, Python 3.3 -Python 2.7, Python 3.1 ___ Python

[issue1818] Add named tuple reader to CSV module

Changes by Skip Montanaro s...@pobox.com: -- nosy: -skip.montanaro ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1818 ___ ___ Python-bugs-list

[issue1818] Add named tuple reader to CSV module

Éric Araujo mer...@netwok.org added the comment: See also this python-ideas thread: http://mail.python.org/pipermail/python-ideas/2010-April/006991.html -- nosy: +merwok ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1818

[issue1818] Add named tuple reader to CSV module

Skip Montanaro s...@pobox.com added the comment: Type conversion is a whole 'nuther kettle of fish. This particular thread is long and complex enough that it shouldn't be made more complex. -- ___ Python tracker rep...@bugs.python.org

[issue1818] Add named tuple reader to CSV module

Antoine Pitrou pit...@free.fr added the comment: I don't understand why NamedTupleReader requires the fieldnames array rather than the namedtuple class itself. If you could pass it the namedtuple class, users could choose whatever namedtuple subclass with whatever additional methods or behaviour

[issue1818] Add named tuple reader to CSV module

Skip Montanaro s...@pobox.com added the comment: I find this aspect of the proposal disturbing: If *fieldnames* is None the values in the first row of the *csvfile* will be used as the fieldnames I don't think this should be implicit. It makes the NamedTupleReader

[issue1818] Add named tuple reader to CSV module

Changes by Skip Montanaro s...@pobox.com: -- message_count: 26.0 - 25.0 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1818 ___ ___

[issue1818] Add named tuple reader to CSV module

Skip Montanaro s...@pobox.com added the comment: I retract my previous comment. I don't use the DictReader the way it operates (fieldnames==None = first row is a header) and forgot about that behavior. -- message_count: 25.0 - 26.0 ___ Python

[issue1818] Add named tuple reader to CSV module

Jervis Whitley jervi...@gmail.com added the comment: Jervis in csv.rst removed reference to reader.next() as a public method. Skip Because? I've not seen any discussion in this issue or in any Skip other forums Skip (most certainly not on the c...@python.org mailing list) which would Skip

[issue1818] Add named tuple reader to CSV module

Jervis Whitley jervi...@gmail.com added the comment: Antoine I don't understand why NamedTupleReader requires the Antoine fieldnames array Antoine rather than the namedtuple class itself. If you could pass it Antoine the namedtuple class, users could choose whatever namedtuple Antoine subclass

[issue1818] Add named tuple reader to CSV module

Antoine Pitrou pit...@free.fr added the comment: Ok, I got misled by the documentation (The contents of *fieldnames* are passed directly to be used as the namedtuple fieldnames), and your implementation is a bit difficult to follow. -- message_count: 28.0 - 29.0

[issue1818] Add named tuple reader to CSV module

Changes by Jervis Whitley jervi...@gmail.com: Removed file: http://bugs.python.org/file13268/ntreader5_py3_1.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1818 ___

[issue1818] Add named tuple reader to CSV module

Changes by Jervis Whitley jervi...@gmail.com: Added file: http://bugs.python.org/file13275/ntreader6_py27.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1818 ___

[issue1818] Add named tuple reader to CSV module

Jervis Whitley jervi...@gmail.com added the comment: Updated version of docs for 2.7 and 3k. -- message_count: 29.0 - 30.0 Added file: http://bugs.python.org/file13274/ntreader6_py3.diff ___ Python tracker rep...@bugs.python.org

[issue1818] Add named tuple reader to CSV module

Jervis Whitley jervi...@gmail.com added the comment: Added a patch against py3k branch. in csv.rst removed reference to reader.next() as a public method. Added file: http://bugs.python.org/file13263/ntreader4_py3_1.diff ___ Python tracker

[issue1818] Add named tuple reader to CSV module

Skip Montanaro s...@pobox.com added the comment: Jervis in csv.rst removed reference to reader.next() as a public method. Because? I've not seen any discussion in this issue or in any other forums (most certainly not on the c...@python.org mailing list) which would suggest that csv.reader's

[issue1818] Add named tuple reader to CSV module

Raymond Hettinger rhettin...@users.sourceforge.net added the comment: The two latest patches (ntreader4.diff and named_tuple_write_header.patch) seem like they are going in the right direction and are getting close. Barry or Skip, is this something you want in your module? -- stage: -

[issue1818] Add named tuple reader to CSV module

Skip Montanaro s...@pobox.com added the comment: Raymond Barry or Skip, is this something you want in your module? Sorry, I haven't really looked at this ticket other than to notice its presence. I wrote the DictReader/DictWriter functions way back when, so I'm pretty comfortable using them.

[issue1818] Add named tuple reader to CSV module

Barry A. Warsaw ba...@python.org added the comment: I think it would be useful to have. ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1818 ___ ___

[issue1818] Add named tuple reader to CSV module

Skip Montanaro s...@pobox.com added the comment: Hrm... I replied twice by email. Only one comment appears to have survived the long trip. Here's my second reply: Rob NamedTupleReader and NamedTupleWriter should be inverses. This Rob means that NamedTupleWriter needs to write

[issue1818] Add named tuple reader to CSV module

Skip Montanaro s...@pobox.com added the comment: Let me be more explicit. I don't know how it implements it, but I think you really need to give the user the option of specifying the field names and not reading/writing headers. It can't be implicit as I interpreted Rob's earlier comment:

[issue1818] Add named tuple reader to CSV module

Jervis Whitley jervi...@gmail.com added the comment: Skip Let me be more explicit. I don't know how it implements it, but I think Skip you really need to give the user the option of specifying the field Skip names and not reading/writing headers. It can't be implicit as I Skip interpreted

[issue1818] Add named tuple reader to CSV module

Rob Renaud rren...@google.com added the comment: I want to make sure I understand. Am I correct in believing that Skip thinks writing headers should be optional, while Jervis believes we should leave the burden to the NamedTupleWriter client? I agree that we should not unconditionally write

[issue1818] Add named tuple reader to CSV module

Skip Montanaro s...@pobox.com added the comment: More concretely, I don't think this is so onerous: names = [col1, col2, color] writer = csv.DictWriter(open(f.csv, wb), fieldnames=names, ...) writer.writerow(dict(zip(names, names))) ... or f = open(f.csv, rb) names =

[issue1818] Add named tuple reader to CSV module

Rob Renaud rren...@google.com added the comment: I did a search on Google code for the DictReader constructor. I analyzed the first 3 pages, the fieldnames parameter was used in 14 of 27 cases (discounting unittest code built into Python) and was not used in 13 of 27 cases. I suppose that

[issue1818] Add named tuple reader to CSV module

Raymond Hettinger rhettin...@users.sourceforge.net added the comment: I don't think you should write them by default. I've worked with lots of CSV files which have no headers. My experience has been the same as Skips. ___ Python tracker

[issue1818] Add named tuple reader to CSV module

Skip Montanaro s...@pobox.com added the comment: Rob I still don't like the lack of symmetry of supporting implicit Rob header reads, but not implicit header writes. A header is nothing more than a row in the CSV file with special interpretation applied by the user. There is nothing

[issue1818] Add named tuple reader to CSV module

Rob Renaud rren...@google.com added the comment: I am totally new to Python dev. I reinvented a NamedTupleReader tonight, only to find out that it was created a year ago. My primary motivation is that DictReader reads headers nicely, but DictWriter totally sucks at handling them. Consider

[issue1818] Add named tuple reader to CSV module

Rob Renaud rren...@google.com added the comment: My previous patch could write the header twice. But I am not sure about about how the writer should handle the fieldnames parameter on one hand, and the namedtuple._fields on the other. Added file:

[issue1818] Add named tuple reader to CSV module

Changes by Rob Renaud rren...@google.com: Removed file: http://bugs.python.org/file13187/named_tuple_write_header.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1818 ___

[issue1818] Add named tuple reader to CSV module

Jervis Whitley jervi...@gmail.com added the comment: Updated NamedTupleReader to give a rename=False keyword argument. rename is passed directly to the namedtuple factory function to enable automatic handling of invalid fieldnames. Two new tests for the rename keyword. Cheers, Added file:

[issue1818] Add named tuple reader to CSV module

Jervis Whitley jervi...@gmail.com added the comment: An implementation of a namedtuple reader and writer. Created a writer for the case where user would like to specify desired field names and default values on missing field names. e.g. mywriter = NamedTupleWriter(f, fieldnames=['f1', 'f2',

[issue1818] Add named tuple reader to CSV module

Raymond Hettinger rhettin...@users.sourceforge.net added the comment: Consider providing a hook to a function that converts non-conforming field names (ones with a leading underscore, leading digit, non-letter, keyword, or duplicate name). class NamedTupleReader: def __init__(self, f,

[issue1818] Add named tuple reader to CSV module

Raymond Hettinger rhettin...@users.sourceforge.net added the comment: In r69480, named tuples gained the ability to automatically rename invalid fieldnames. ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1818

[issue1818] Add named tuple reader to CSV module

Raymond Hettinger added the comment: Barry, any thoughts on this? __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1818 __ ___ Python-bugs-list mailing list Unsubscribe:

[issue1818] Add named tuple reader to CSV module

Skip Montanaro added the comment: I'd personally be kind of surprised if Barry had any thoughts on this. Is there any reason this couldn't be pushed down into the C code and replace the normal tuple output completely? In the absence of any fieldnames you could just dream some up, like field001,

[issue1818] Add named tuple reader to CSV module

New submission from Raymond Hettinger: Here's a proof-of-concept patch. If approved, will change from generator form to match the other readers and will add a test suite. The idea corresponds to what is currently done by the dict reader but returns a space and time efficient named tuple