[issue24869] shlex lineno inaccurate with certain inputs

[Gareth Rees]

Gareth Rees added the comment:

I've made a pull request. (Not because I expect it to be merged as-is, but to 
provide a starting point for discussion.)

--
nosy: +petri.lehtinen, vinay.sajip

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24869] shlex lineno inaccurate with certain inputs

[Gareth Rees]

Changes by Gareth Rees :


--
pull_requests: +2849

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24869] shlex lineno inaccurate with certain inputs

[Gareth Rees]

Gareth Rees added the comment:

Here's a patch that implements my proposal (1) -- under this patch, tokens read 
from an input stream belong to a subtype of str with startline and endline 
attributes giving the line numbers of the first and last character of the 
token. This allows the accurate reporting of error messages relating to a 
token. I updated the documentation and added a test case.

--
keywords: +patch
Added file: http://bugs.python.org/file46479/issue24869.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24869] shlex lineno inaccurate with certain inputs

[Christopher Hoadley]

Changes by Christopher Hoadley :


--
type:  -> behavior
versions: +Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24869] shlex lineno inaccurate with certain inputs

[Gareth Rees]

Gareth Rees added the comment:

A third alternative:

3. Add a method whose effect is to consume comments and whitespace, but which 
does not yield a token. You could then call this method, and then look at 
shlex.lineno, which will be the line number of the first character of the next 
token (if there is a next token).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24869] shlex lineno inaccurate with certain inputs

[Gareth Rees]

Gareth Rees added the comment:

Just to restate the problem:

The use case is that when emitting an error message for a token, we want to 
include the number of the line containing the token (or the number of the line 
where the token started, if the token spans multiple lines, as it might if it's 
a string containing newlines).

But there is no way to satisfy this use case given the features of the shlex 
module. In particular, shlex.lineno (which looks as if it ought to help) is 
actually the line number of the first character that has not yet been consumed 
by the lexer, and in general this is not the same as the line number of the 
previous (or the next) token.

I can think of two alternatives that would satisfy the use case:

1. Instead of returning tokens as str objects, return them as instances of a 
subclass of str that has a property that gives the line number of the first 
character of the token. (Maybe it should also have properties for the column 
number of the first character, and the line and column number of the last 
character too? These properties would support better error messages.)

2. Add new methods that return tuples giving the token and its line number (and 
possibly column number etc. as in alternative 1).

My preference would be for alternative (1), but I suppose there is a very tiny 
risk of breaking some code that relied upon get_token returning an instance of 
str exactly rather than an instance of a subclass of str.

--
nosy: +Gareth.Rees

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24869] shlex lineno inaccurate with certain inputs

[Christopher Hoadley]

Christopher Hoadley added the comment:

This problem makes it impossible to use shlex to parse commands where a newline 
is intended to separate commands.
 
In the attached sample script, I created two input strings with the same tokens 
in the same order: the only difference is newlines and spaces. In the first 
string, each token is on its own line, and in the second string the first 2 
tokens are on the same line, and the third is on its own.
 
If you look at the lineno association with each token, it is identical between 
the two strings.  But, the two strings have completely different meanings! I 
have no way to distinguish the behavior.
 
If I want to use the feature of shlex where it will automatically include other 
command files, then I can't just sanitize the input before sending it on.
 
As it is, the only way that I can see that I can use shlex is if my command 
language uses some other symbol (i.e. ";") as a command line separator.  Since 
I am defining my own command language, I can do that, but it adds needless 
complication for the users.

--
nosy: +hoadlck
versions: +Python 3.5 -Python 3.4
Added file: http://bugs.python.org/file43361/ambigious_shlex.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24869] shlex lineno inaccurate with certain inputs

[Robert Escriva]

New submission from Robert Escriva:

The newlines calculated by the shlex module are inaccurate for certain inputs 
with comments inline.  I've attached a simple script that illustrates the 
problem.

My assumption here is that the lineno is supposed to match a line related to 
the current token.  I'm trying to use changes in the lineno to aggregate tokens 
into commands.  This may not be an intended use case.

--
components: Library (Lib)
files: badlex.py
messages: 248596
nosy: rescrv
priority: normal
severity: normal
status: open
title: shlex lineno inaccurate with certain inputs
versions: Python 3.4
Added file: http://bugs.python.org/file40180/badlex.py

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24869
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com