Re: [Tutor] shlex parsing

[Karim]

On 07/28/2011 02:27 AM, Steven D'Aprano wrote:

Karim wrote:


Hello All,

I would like to parse this TCL command line with shlex:

'-option1 [get_rule A1 B2] -option2 $VAR -option3 TAG'

And I want to get the splitted list:

['-option1', '[get_rule A1 B2]', '-option2',  '$VAR', '-option3',  
'TAG']


Then I will gather in tuple 2 by 2 the arguments.

I tried to the shlec properties attributes 'quotes', 'whitespace', 
etc...


I don't understand what you are doing here. Please show the code you use.

The shlex module doesn't support bracketed expressions. I recommend 
you write a post-processor. Start with doing this:


>>> import shlex
>>> text = '-option1 [get_rule A1 B2] -option2 $VAR -option3 TAG'
>>> shlex.split(text)
['-option1', '[get_rule', 'A1', 'B2]', '-option2', '$VAR', '-option3', 
'TAG']


then take that list and reassemble the pieces starting with '[' until 
']' Something like this, untested:



def reassemble(items):
result = []
bracketed = False
current = ''
for item in items:
if item.startswith('['):
bracketed = True
if bracketed:
current += item
if item.endswith(']'):
bracketed = False
result.append(current)
current = ''
else:
 result.append(item)
return result





Yes Steven this is the kind of code I wrote in a post earlier, but I 
forget to reinit as you did current equal to _token in my code, thanks 
for that for showing me to simply if/elif/elif/else levels:


Previous code was:

"I ended up with this and gave up with shlex:"

split = ['-option1', '[get_rule', 'A1', 'B2]', '-option2', '$VAR', 
'-option3', 'TAG']


procedure_found = False
result  = []

for token in split:
if not token.startswith('[') and not token.endswith(']') and not 
procedure_found:

result.append(token)
elif token.startswith('['):
procedure_found = True
_token = token
elif token.endswith(']'):
procedure_found = False
_token += ' ' + token
result.append(_token)
else:
_token += ' ' + token

print split
print result


But I make 'choux blanc'.


I don't know what that means.

This means 'white cabbage' in french = 'unsuccessful try'

Cheers
Karim
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] shlex parsing

[Steven D'Aprano]

Karim wrote:


Hello All,

I would like to parse this TCL command line with shlex:

'-option1 [get_rule A1 B2] -option2 $VAR -option3 TAG'

And I want to get the splitted list:

['-option1', '[get_rule A1 B2]', '-option2',  '$VAR', '-option3',  'TAG']

Then I will gather in tuple 2 by 2 the arguments.

I tried to the shlec properties attributes 'quotes', 'whitespace', etc...


I don't understand what you are doing here. Please show the code you use.

The shlex module doesn't support bracketed expressions. I recommend you 
write a post-processor. Start with doing this:


>>> import shlex
>>> text = '-option1 [get_rule A1 B2] -option2 $VAR -option3 TAG'
>>> shlex.split(text)
['-option1', '[get_rule', 'A1', 'B2]', '-option2', '$VAR', '-option3', 
'TAG']


then take that list and reassemble the pieces starting with '[' until 
']' Something like this, untested:



def reassemble(items):
result = []
bracketed = False
current = ''
for item in items:
if item.startswith('['):
bracketed = True
if bracketed:
current += item
if item.endswith(']'):
bracketed = False
result.append(current)
current = ''
else:
 result.append(item)
return result




But I make 'choux blanc'.


I don't know what that means.




--
Steven

___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] shlex parsing

[Karim]

On 07/28/2011 12:11 AM, Dan Stromberg wrote:


You could probably use a recursive descent parser with the standard 
library.


But if your management is OK with pyparsing, that might be easier, and 
a bit more clear as well.


Yes, I thought to use str method partition in a recursive way but using 
pyParsing still be easer.


Thanks
Cheers



On Wed, Jul 27, 2011 at 2:08 PM, Karim > wrote:



Thank you Dan for answering.

I ended with this and gave up with shlex:

split = ['-option1', '[get_rule', 'A1', 'B2]', '-option2', '$VAR',
'-option3', 'TAG']

procedure_found = False
result  = []

for token in split:
if not token.startswith('[') and not token.endswith(']') and
not procedure_found:
result.append(token)
elif token.startswith('['):
procedure_found = True
_token = token
elif token.endswith(']'):
procedure_found = False
_token += ' ' + token
result.append(_token)
else:
_token += ' ' + token

print split
print result

which gives the desired values:

['-option1', '[get_rule', 'A1', 'B2]', '-option2', '$VAR',
'-option3', 'TAG']
['-option1', '[get_rule A1 B2]', '-option2', '$VAR', '-option3',
'TAG']


Sure pyParsing seems to be pretty simple but my constraint is to use
standard lib (at maximum). To bad it is not part of python
standard libs.
On the other hand, I will have to regroup expression like
'-option1 $VAL == $CONSTRAINT'
in ['-option1', '$VAL == $CONSTRAINT'].

So it seems that I have no others choicse and have to use a parser
like pyParsing.

Regards
Karim


On 07/27/2011 10:44 PM, Dan Stromberg wrote:


I've not used the shlex module, but this feels more like an issue
to address with a parser than for a lexical analyzer - or perhaps
even both, since you're splitting on whitespace sometimes, and
matching square brackets sometimes.

I've used pyparsing for stuff a bit similar to this.

Or here's a list:
http://wiki.python.org/moin/LanguageParsing

On Wed, Jul 27, 2011 at 12:30 PM, Karim mailto:karim.liat...@free.fr>> wrote:


Hello All,

I would like to parse this TCL command line with shlex:

'-option1 [get_rule A1 B2] -option2 $VAR -option3 TAG'

And I want to get the splitted list:

['-option1', '[get_rule A1 B2]', '-option2',  '$VAR',
'-option3',  'TAG']

Then I will gather in tuple 2 by 2 the arguments.

I tried to the shlec properties attributes 'quotes',
'whitespace', etc...

But I make 'choux blanc'.

If somebody has complex experiences with  this module I am in.

Cheers
Karim
-- 
http://mail.python.org/mailman/listinfo/python-list








___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] shlex parsing

[Karim]

Thank you Dan for answering.

I ended with this and gave up with shlex:

split = ['-option1', '[get_rule', 'A1', 'B2]', '-option2', '$VAR', 
'-option3', 'TAG']


procedure_found = False
result  = []

for token in split:
if not token.startswith('[') and not token.endswith(']') and not 
procedure_found:

result.append(token)
elif token.startswith('['):
procedure_found = True
_token = token
elif token.endswith(']'):
procedure_found = False
_token += ' ' + token
result.append(_token)
else:
_token += ' ' + token

print split
print result

which gives the desired values:

['-option1', '[get_rule', 'A1', 'B2]', '-option2', '$VAR', '-option3', 
'TAG']

['-option1', '[get_rule A1 B2]', '-option2', '$VAR', '-option3', 'TAG']


Sure pyParsing seems to be pretty simple but my constraint is to use
standard lib (at maximum). To bad it is not part of python standard libs.
On the other hand, I will have to regroup expression like '-option1 $VAL 
== $CONSTRAINT'

in ['-option1', '$VAL == $CONSTRAINT'].

So it seems that I have no others choicse and have to use a parser like 
pyParsing.


Regards
Karim

On 07/27/2011 10:44 PM, Dan Stromberg wrote:


I've not used the shlex module, but this feels more like an issue to 
address with a parser than for a lexical analyzer - or perhaps even 
both, since you're splitting on whitespace sometimes, and matching 
square brackets sometimes.


I've used pyparsing for stuff a bit similar to this.

Or here's a list:
http://wiki.python.org/moin/LanguageParsing

On Wed, Jul 27, 2011 at 12:30 PM, Karim > wrote:



Hello All,

I would like to parse this TCL command line with shlex:

'-option1 [get_rule A1 B2] -option2 $VAR -option3 TAG'

And I want to get the splitted list:

['-option1', '[get_rule A1 B2]', '-option2',  '$VAR', '-option3',
 'TAG']

Then I will gather in tuple 2 by 2 the arguments.

I tried to the shlec properties attributes 'quotes', 'whitespace',
etc...

But I make 'choux blanc'.

If somebody has complex experiences with  this module I am in.

Cheers
Karim
-- 
http://mail.python.org/mailman/listinfo/python-list





___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

[Tutor] shlex parsing

[Karim]

Hello All,

I would like to parse this TCL command line with shlex:

'-option1 [get_rule A1 B2] -option2 $VAR -option3 TAG'

And I want to get the splitted list:

['-option1', '[get_rule A1 B2]', '-option2',  '$VAR', '-option3',  'TAG']

Then I will gather in tuple 2 by 2 the arguments.

I tried to the shlec properties attributes 'quotes', 'whitespace', etc...

But I make 'choux blanc'.

If somebody has complex experiences with  this module I am in.

Cheers
Karim
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor