[issue10515] csv sniffer does not recognize quotes at the end of line

[Irit Katriel]

Change by Irit Katriel :


--
resolution:  -> duplicate
stage: patch review -> resolved
status: open -> closed
superseder:  -> csv.Sniffer.sniff() regex error

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10515] csv sniffer does not recognize quotes at the end of line

[Mark Lawrence]

Change by Mark Lawrence :


--
nosy:  -BreamoreBoy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10515] csv sniffer does not recognize quotes at the end of line

[Mark Lawrence]

Mark Lawrence added the comment:

@David could you check Skip's patch out please.  It seems identical to your own 
other than changing the delimiter from the empty string to a comma.

--
nosy: +BreamoreBoy

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10515
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10515] csv sniffer does not recognize quotes at the end of line

[Skip Montanaro]

Changes by Skip Montanaro s...@pobox.com:


--
assignee: skip.montanaro - 
nosy:  -skip.montanaro

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10515
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10515] csv sniffer does not recognize quotes at the end of line

[R. David Murray]

R. David Murray rdmur...@bitdance.com added the comment:

I agree that the unquoted single column cases look weird. But changing it could 
affect other cases were there is more data.  I'll leave that problem to you :)  
But I do not think that should be backported.

The locals trick I stole from Barry. I think there ought to be a way to get at 
the class dict as the class is being built. But you are right that the docs 
imply that locals is not guaranteed to be it.  Perhaps a topic for python-dev.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10515
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10515] csv sniffer does not recognize quotes at the end of line

[Skip Montanaro]

Skip Montanaro s...@pobox.com added the comment:

Also, this comment in test_csv.py puzzles me:

# given that all three lines in sample3 are equal,
# I think that any character could have been 'guessed' as the
# delimiter, depending on dictionary order

As a human looking at sample3 it's obvious the '?' should
be the delimiter.  I can understand it picking '/' instead,
but it shouldn't punt.  (I stumbled upon this while trying
to make , be the default delimiter instead of None when
none of the regular expressions matched.)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10515
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10515] csv sniffer does not recognize quotes at the end of line

[Skip Montanaro]

Skip Montanaro s...@pobox.com added the comment:

Here's my candidate patch.  Instead of returning an empty string
as the delimiter it returns a comma.

--
Added file: http://bugs.python.org/file20111/csv_delimiter_tests.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10515
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10515] csv sniffer does not recognize quotes at the end of line

[Raymond Hettinger]

Changes by Raymond Hettinger rhettin...@users.sourceforge.net:


--
assignee:  - skip.montanaro

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10515
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10515] csv sniffer does not recognize quotes at the end of line

[Martin Budaj]

Martin Budaj m.bu...@gmail.com added the comment:

 What do you mean by there is a test for this case in csv.py?  

I meant test in regex on line 217 in python 2.7 and the following code (line 
258ff):

# there is *no* delimiter, it's a single column of quoted data
delim = ''
skipinitialspace = 0

However, it is intended to detect just lines starting and ending with quotes.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10515
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10515] csv sniffer does not recognize quotes at the end of line

[Skip Montanaro]

Skip Montanaro s...@pobox.com added the comment:

From the comment in the test_csv.py:

+# XXX: I don't know what the correct behavior should be for these.
+# Currently the first one raises an error that the delimiter can't
+# be determined while the second one returns '\r'.  The second
+# is obviously.
+('a,b,c,d\ne',  ''),
+('a,b,c,d\r\ne', ''),

Obviously what?  My guess would be wrong.  In the absence of any other
information \r\n has to be treated as a line separator.  It shouldn't be
considered as two separate characters, even in such a devoid-of-clues test
case.

Is the empty string a valid delimiter?  I've never tried it and it's been a
long time since I looked at any of the code.  I do use single-column CSV
files from time-to-time though.  I don't think a ',' would be a completely
unreasonable fallback scenario either.

Skip

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10515
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10515] csv sniffer does not recognize quotes at the end of line

[R. David Murray]

R. David Murray rdmur...@bitdance.com added the comment:

Yeah, obviously wrong.  I forgot to finish editing the comment.

I think a fallback of ',' makes more sense than ''.  What would a delimiter of 
nothing mean?  I don't think the unquoted case can be changed for backward 
compatibility reasons, so those tests I added should presumably be changed to 
confirm the existing behavior (with an appropriate comment).

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10515
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10515] csv sniffer does not recognize quotes at the end of line

[R. David Murray]

R. David Murray rdmur...@bitdance.com added the comment:

What do you mean by there is a test for this case in csv.py?  If I run 
sniffer against abcde\ndefgh\n I get a delimiter of 'e'.  If I run it against 
'a\nb\n', I get the could not determine delimiter error.

Attached is a reformulated test case that lets us see the errors individually 
and easily add more.  With this set of 10 tests, two pass without your patches 
(as you knew; those are your tests), and we have 4 failures and 4 errors.  With 
p1 applied we have 4 failures and 3 errors.  With p2 applied we have 5 failures.

As I said, I'm not at all sure what the results *should* be for various of 
these cases.  Certainly the long standing existing behavior seems to be to 
return one of the characters in an unquoted multicharacter sequence. 

That said, it seems like your patch p1 is good, but as you discovered not 
sufficient.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10515
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10515] csv sniffer does not recognize quotes at the end of line

[R. David Murray]

R. David Murray rdmur...@bitdance.com added the comment:

Forgot to attach the patch.

--
Added file: http://bugs.python.org/file20006/csv_delimiter_tests.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10515
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10515] csv sniffer does not recognize quotes at the end of line

[Martin Budaj]

Martin Budaj m.bu...@gmail.com added the comment:

I'm not sure about what the intended behavior for testAl should be, however I 
think that the file should be recognized as having one column of data and no 
delimiter (there is a test for this case in csv.py) and not raise an exception.

I attach patch for test_csv.py tests. It includes two more tests which 
currently fail (both contain \r\n as a newline)

--
Added file: http://bugs.python.org/file19944/test_csv.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10515
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10515] csv sniffer does not recognize quotes at the end of line

[R. David Murray]

R. David Murray rdmur...@bitdance.com added the comment:

Are you sure testA1 is correct?  It seems to me that in that case the sniffer 
can indeed not determine the delimiter, but I don't really understand the 
guessing algorithm.  The existing behavior on unquoted strings is...interesting 
:)

Also if you are willing to take the time it would be helpful to have the tests 
as a patch against test_csv.

--
nosy: +r.david.murray, skip.montanaro

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10515
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10515] csv sniffer does not recognize quotes at the end of line

[Martin Budaj]

Martin Budaj m.bu...@gmail.com added the comment:

Units test and two patches for 2.7 are included.

p1.patch fixes testEnd case reported yesterday

After running unittest it seems that also other case is broken (testAl -- if 
there is just one data item on the line, enclosed in quotes). Patch p2 fixes 
it, but should be checked for side-effects.

--
Added file: http://bugs.python.org/file19799/fix.tar.gz

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10515
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10515] csv sniffer does not recognize quotes at the end of line

[Ned Deily]

Changes by Ned Deily n...@acm.org:


Added file: http://bugs.python.org/file19800/test.py

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10515
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10515] csv sniffer does not recognize quotes at the end of line

[Ned Deily]

Changes by Ned Deily n...@acm.org:


--
keywords: +patch
Added file: http://bugs.python.org/file19802/p1.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10515
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10515] csv sniffer does not recognize quotes at the end of line

[Ned Deily]

Ned Deily n...@acm.org added the comment:

(Thanks. I've unpacked and uploaded the three files from your tar file.)

--
stage: unit test needed - patch review
Added file: http://bugs.python.org/file19803/p2.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10515
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10515] csv sniffer does not recognize quotes at the end of line

[Martin Budaj]

New submission from Martin Budaj m.bu...@gmail.com:

The method Sniffer._guess_quote_and_delimiter() in the module csv.py contains a 
bug in a regexp which checks for quotes around the last item of the line 
(example: a,b,c,d\n).

the pattern
'(?Pdelim[^\w\n\'])(?Pspace ?)(?Pquote[\']).*?(?P=quote)(?:$|\n)'
used in the regex should be changed to
'(?Pdelim[^\w\n\'])(?Pspace ?)(?Pquote[\']).*?(?P=quote)(?:$|\n)'

file csv.py: line 212 in python 2.6, line 216 in 2.7, line 207 in 3.1

--
components: Library (Lib)
messages: 17
nosy: Martin.Budaj
priority: normal
severity: normal
status: open
title: csv sniffer does not recognize quotes at the end of line
type: behavior
versions: Python 2.6, Python 2.7, Python 3.1

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10515
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10515] csv sniffer does not recognize quotes at the end of line

[Ned Deily]

Ned Deily n...@acm.org added the comment:

Thanks for the report. It would be helpful if you could supply a patch 
including a unit test for this against 3.2 and/or 2.7.  Note only security 
issues are accepted for 2.6.

--
nosy: +ned.deily
stage:  - unit test needed
versions: +Python 3.2 -Python 2.6

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10515
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10515] csv sniffer does not recognize quotes at the end of line

[Andrew McNamara]

Changes by Andrew McNamara andr...@object-craft.com.au:


--
nosy: +andrewmcnamara

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10515
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com