Re: shlex parsing

[Karim]

On 07/29/2011 03:42 PM, Web Dreamer wrote:

whitespace_split = True

Thanks for the tip!

Cheers
Karim
--
http://mail.python.org/mailman/listinfo/python-list

Re: shlex parsing

[Karim]

On 07/29/2011 09:24 AM, Web Dreamer wrote:

Nobody a écrit ce jeudi 28 juillet 2011 18:37 dans
  :


On Thu, 28 Jul 2011 17:48:34 +0200, Web Dreamer wrote:


I would like to parse this TCL command line with shlex:

'-option1 [get_rule A1 B2] -option2 $VAR -option3 TAG'

s = s.replace('[','"[')
s = s.replace(']',']"')

Note that this approach won't work if you have nested brackets or braces.
That would require a real parser.

True,
I tried with the shlex class, but adding '[]' to a shlexobject.quotes
doesn't work.
Indeed, shlex expects an opening and closing quote to be the same, and [ is
different from ] so shlex therefore expects an opening [ to be closed with [
and not ]

My solution indeed only works as long as there are not any nested brackets.

Yeah I saw that shlex object behavior is slighty different from 
shlex.split()
fonction. the char minus by default is taken as a individual word and in 
split
it is not. get_token() method return '-' alone and you have to set 
object attributes

wordchars += '-' to have the same behavior as shlex.split().

Cheers
Karim

--
http://mail.python.org/mailman/listinfo/python-list

Re: shlex parsing

[Karim]

Just a little modification:

>>> tuple([(option, value) for option,value in 
zip(optionlist[0::2],optionlist[1::2])]) == 
tuple(zip(optionlist[0::2],optionlist[1::2]))

True

Indeed:

tuple(zip(optionlist[0::2],optionlist[1::2]))

shorter than:

tuple([(option, value) for option,value in 
zip(optionlist[0::2],optionlist[1::2])])



Karim

PS: I am from Grenoble, which place in France are from?

On 07/28/2011 06:37 PM, Nobody wrote:

On Thu, 28 Jul 2011 17:48:34 +0200, Web Dreamer wrote:


I would like to parse this TCL command line with shlex:

'-option1 [get_rule A1 B2] -option2 $VAR -option3 TAG'

s = s.replace('[','"[')
s = s.replace(']',']"')

Note that this approach won't work if you have nested brackets or braces.
That would require a real parser.



--
http://mail.python.org/mailman/listinfo/python-list

Re: shlex parsing

[Nobody]

On Thu, 28 Jul 2011 17:48:34 +0200, Web Dreamer wrote:

>> I would like to parse this TCL command line with shlex:
>> 
>> '-option1 [get_rule A1 B2] -option2 $VAR -option3 TAG'

 s = s.replace('[','"[')
 s = s.replace(']',']"')

Note that this approach won't work if you have nested brackets or braces.
That would require a real parser.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: shlex parsing

[Karim]

Hello You have you feet on earth Web Dreamer!

Very clever!
Beautiful hack!

Many Thanks

Karim

On 07/28/2011 05:48 PM, Web Dreamer wrote:

Karim a écrit ce mercredi 27 juillet 2011 21:30 
dans  :


Hello All,

I would like to parse this TCL command line with shlex:

'-option1 [get_rule A1 B2] -option2 $VAR -option3 TAG'

And I want to get the splitted list:

['-option1', '[get_rule A1 B2]', '-option2',  '$VAR', '-option3',  'TAG']

Then I will gather in tuple 2 by 2 the arguments.


Do this:


s = '-option1 [get_rule A1 B2] -option2 $VAR -option3 TAG'

Now if you don't enclose [get_rule A1 B2] in cotes, you will pull your hair off!
So:

s = s.replace('[','"[')
s = s.replace(']',']"')

Now:

s

'-option1 "[get_rule A1 B2]" -option2 $VAR -option3 TAG'

Lets continue:

import shlex
optionlist = shlex.split(s)
optionlist

['-option1', '[get_rule A1 B2]', '-option2', '$VAR', '-option3', 'TAG']

Now to get your tupple of two by two arguments:

argtuple = tuple([(option, value) for option,value in 
zip(optionlist[0::2],optionlist[1::2])])
argtuple

(('-option1', '[get_rule A1 B2]'), ('-option2', '$VAR'), ('-option3', 'TAG'))


whole code:

import shlex
s = '-option1 [get_rule A1 B2] -option2 $VAR -option3 TAG'
s = s.replace('[','"[')
s = s.replace(']',']"')
optionlist = shlex.split(s)
argtuple = tuple([(option, value) for option,value in 
zip(optionlist[0::2],optionlist[1::2]))


--
http://mail.python.org/mailman/listinfo/python-list

Re: shlex parsing

[Karim]

On 07/28/2011 12:11 AM, Dan Stromberg wrote:


You could probably use a recursive descent parser with the standard 
library.


But if your management is OK with pyparsing, that might be easier, and 
a bit more clear as well.


Yes, I thought to use str method partition in a recursive way but using 
pyParsing still be easer.


Thanks
Cheers



On Wed, Jul 27, 2011 at 2:08 PM, Karim > wrote:



Thank you Dan for answering.

I ended with this and gave up with shlex:

split = ['-option1', '[get_rule', 'A1', 'B2]', '-option2', '$VAR',
'-option3', 'TAG']

procedure_found = False
result  = []

for token in split:
if not token.startswith('[') and not token.endswith(']') and
not procedure_found:
result.append(token)
elif token.startswith('['):
procedure_found = True
_token = token
elif token.endswith(']'):
procedure_found = False
_token += ' ' + token
result.append(_token)
else:
_token += ' ' + token

print split
print result

which gives the desired values:

['-option1', '[get_rule', 'A1', 'B2]', '-option2', '$VAR',
'-option3', 'TAG']
['-option1', '[get_rule A1 B2]', '-option2', '$VAR', '-option3',
'TAG']


Sure pyParsing seems to be pretty simple but my constraint is to use
standard lib (at maximum). To bad it is not part of python
standard libs.
On the other hand, I will have to regroup expression like
'-option1 $VAL == $CONSTRAINT'
in ['-option1', '$VAL == $CONSTRAINT'].

So it seems that I have no others choicse and have to use a parser
like pyParsing.

Regards
Karim


On 07/27/2011 10:44 PM, Dan Stromberg wrote:


I've not used the shlex module, but this feels more like an issue
to address with a parser than for a lexical analyzer - or perhaps
even both, since you're splitting on whitespace sometimes, and
matching square brackets sometimes.

I've used pyparsing for stuff a bit similar to this.

Or here's a list:
http://wiki.python.org/moin/LanguageParsing

On Wed, Jul 27, 2011 at 12:30 PM, Karim mailto:karim.liat...@free.fr>> wrote:


Hello All,

I would like to parse this TCL command line with shlex:

'-option1 [get_rule A1 B2] -option2 $VAR -option3 TAG'

And I want to get the splitted list:

['-option1', '[get_rule A1 B2]', '-option2',  '$VAR',
'-option3',  'TAG']

Then I will gather in tuple 2 by 2 the arguments.

I tried to the shlec properties attributes 'quotes',
'whitespace', etc...

But I make 'choux blanc'.

If somebody has complex experiences with  this module I am in.

Cheers
Karim
-- 
http://mail.python.org/mailman/listinfo/python-list








-- 
http://mail.python.org/mailman/listinfo/python-list

Re: shlex parsing

[Dan Stromberg]

You could probably use a recursive descent parser with the standard library.

But if your management is OK with pyparsing, that might be easier, and a bit
more clear as well.

On Wed, Jul 27, 2011 at 2:08 PM, Karim  wrote:

> **
>
> Thank you Dan for answering.
>
> I ended with this and gave up with shlex:
>
> split = ['-option1', '[get_rule', 'A1', 'B2]', '-option2', '$VAR',
> '-option3', 'TAG']
>
> procedure_found = False
> result  = []
>
> for token in split:
> if not token.startswith('[') and not token.endswith(']') and not
> procedure_found:
> result.append(token)
> elif token.startswith('['):
> procedure_found = True
> _token = token
> elif token.endswith(']'):
> procedure_found = False
> _token += ' ' + token
> result.append(_token)
> else:
> _token += ' ' + token
>
> print split
> print result
>
> which gives the desired values:
>
> ['-option1', '[get_rule', 'A1', 'B2]', '-option2', '$VAR', '-option3',
> 'TAG']
> ['-option1', '[get_rule A1 B2]', '-option2', '$VAR', '-option3', 'TAG']
>
>
> Sure pyParsing seems to be pretty simple but my constraint is to use
> standard lib (at maximum). To bad it is not part of python standard libs.
> On the other hand, I will have to regroup expression like '-option1 $VAL ==
> $CONSTRAINT'
> in ['-option1', '$VAL == $CONSTRAINT'].
>
> So it seems that I have no others choicse and have to use a parser like
> pyParsing.
>
> Regards
> Karim
>
>
> On 07/27/2011 10:44 PM, Dan Stromberg wrote:
>
>
> I've not used the shlex module, but this feels more like an issue to
> address with a parser than for a lexical analyzer - or perhaps even both,
> since you're splitting on whitespace sometimes, and matching square brackets
> sometimes.
>
> I've used pyparsing for stuff a bit similar to this.
>
> Or here's a list:
> http://wiki.python.org/moin/LanguageParsing
>
>  On Wed, Jul 27, 2011 at 12:30 PM, Karim  wrote:
>
>>
>> Hello All,
>>
>> I would like to parse this TCL command line with shlex:
>>
>> '-option1 [get_rule A1 B2] -option2 $VAR -option3 TAG'
>>
>> And I want to get the splitted list:
>>
>> ['-option1', '[get_rule A1 B2]', '-option2',  '$VAR', '-option3',  'TAG']
>>
>> Then I will gather in tuple 2 by 2 the arguments.
>>
>> I tried to the shlec properties attributes 'quotes', 'whitespace', etc...
>>
>> But I make 'choux blanc'.
>>
>> If somebody has complex experiences with  this module I am in.
>>
>> Cheers
>> Karim
>>  --
>> http://mail.python.org/mailman/listinfo/python-list
>>
>
>
>
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: shlex parsing

[Karim]

Thank you Dan for answering.

I ended with this and gave up with shlex:

split = ['-option1', '[get_rule', 'A1', 'B2]', '-option2', '$VAR', 
'-option3', 'TAG']


procedure_found = False
result  = []

for token in split:
if not token.startswith('[') and not token.endswith(']') and not 
procedure_found:

result.append(token)
elif token.startswith('['):
procedure_found = True
_token = token
elif token.endswith(']'):
procedure_found = False
_token += ' ' + token
result.append(_token)
else:
_token += ' ' + token

print split
print result

which gives the desired values:

['-option1', '[get_rule', 'A1', 'B2]', '-option2', '$VAR', '-option3', 
'TAG']

['-option1', '[get_rule A1 B2]', '-option2', '$VAR', '-option3', 'TAG']


Sure pyParsing seems to be pretty simple but my constraint is to use
standard lib (at maximum). To bad it is not part of python standard libs.
On the other hand, I will have to regroup expression like '-option1 $VAL 
== $CONSTRAINT'

in ['-option1', '$VAL == $CONSTRAINT'].

So it seems that I have no others choicse and have to use a parser like 
pyParsing.


Regards
Karim

On 07/27/2011 10:44 PM, Dan Stromberg wrote:


I've not used the shlex module, but this feels more like an issue to 
address with a parser than for a lexical analyzer - or perhaps even 
both, since you're splitting on whitespace sometimes, and matching 
square brackets sometimes.


I've used pyparsing for stuff a bit similar to this.

Or here's a list:
http://wiki.python.org/moin/LanguageParsing

On Wed, Jul 27, 2011 at 12:30 PM, Karim > wrote:



Hello All,

I would like to parse this TCL command line with shlex:

'-option1 [get_rule A1 B2] -option2 $VAR -option3 TAG'

And I want to get the splitted list:

['-option1', '[get_rule A1 B2]', '-option2',  '$VAR', '-option3',
 'TAG']

Then I will gather in tuple 2 by 2 the arguments.

I tried to the shlec properties attributes 'quotes', 'whitespace',
etc...

But I make 'choux blanc'.

If somebody has complex experiences with  this module I am in.

Cheers
Karim
-- 
http://mail.python.org/mailman/listinfo/python-list





-- 
http://mail.python.org/mailman/listinfo/python-list

Re: shlex parsing

[Dan Stromberg]

I've not used the shlex module, but this feels more like an issue to address
with a parser than for a lexical analyzer - or perhaps even both, since
you're splitting on whitespace sometimes, and matching square brackets
sometimes.

I've used pyparsing for stuff a bit similar to this.

Or here's a list:
http://wiki.python.org/moin/LanguageParsing

On Wed, Jul 27, 2011 at 12:30 PM, Karim  wrote:

>
> Hello All,
>
> I would like to parse this TCL command line with shlex:
>
> '-option1 [get_rule A1 B2] -option2 $VAR -option3 TAG'
>
> And I want to get the splitted list:
>
> ['-option1', '[get_rule A1 B2]', '-option2',  '$VAR', '-option3',  'TAG']
>
> Then I will gather in tuple 2 by 2 the arguments.
>
> I tried to the shlec properties attributes 'quotes', 'whitespace', etc...
>
> But I make 'choux blanc'.
>
> If somebody has complex experiences with  this module I am in.
>
> Cheers
> Karim
> --
> http://mail.python.org/**mailman/listinfo/python-list
>
-- 
http://mail.python.org/mailman/listinfo/python-list