Re: [Python-Dev] Where are universal newlines handled in the parser/compiler?
On Sat, Aug 16, 2008 at 8:34 PM, Brett Cannon <[EMAIL PROTECTED]> wrote: > If you import a module that uses '\r\n' line endings, Python does the > right thing. But if you read in the bytes for the same file and then > pass it to compile() you get an unhelpful syntax error pointing at a > blank line. > > Normally I would say one should just open the source file as 'r' > instead of 'rb', but with source code that does not work well as their > can be a source encoding set. Lib/test/test_pep263.py is the perfect > example of this; Windows newlines with a koi8-r encoding. > > What I would like to do is get compile() to work properly with a bytes > stream just as if Python itself was handling the compilation through > import and from a file directly. But before I try to dig into the > parser to figure out where the translation of newlines occurs (or > where the translation option is set), I thought I would ask to see if > anyone just happened to know (I have already spent a few hours > figuring out why Latin-1 encodings were not working with compile() so > I don't want to go diving into the maze of function calls in the > parser again). Have a look at tok_nextc in Parser/tokenizer.c. > > -Brett > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/musiccomposition%40gmail.com > -- Cheers, Benjamin Peterson "There's no place like 127.0.0.1." ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Where are universal newlines handled in the parser/compiler?
If you import a module that uses '\r\n' line endings, Python does the right thing. But if you read in the bytes for the same file and then pass it to compile() you get an unhelpful syntax error pointing at a blank line. Normally I would say one should just open the source file as 'r' instead of 'rb', but with source code that does not work well as their can be a source encoding set. Lib/test/test_pep263.py is the perfect example of this; Windows newlines with a koi8-r encoding. What I would like to do is get compile() to work properly with a bytes stream just as if Python itself was handling the compilation through import and from a file directly. But before I try to dig into the parser to figure out where the translation of newlines occurs (or where the translation option is set), I thought I would ask to see if anyone just happened to know (I have already spent a few hours figuring out why Latin-1 encodings were not working with compile() so I don't want to go diving into the maze of function calls in the parser again). -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] eval() accepts compiled 'exec' statements
Patrick Maupin wrote:
This may or may not be relevant to 2.6 or 3.0 (because I don't have
those handy at the moment), but on 2.5 and earlier:
Python 2.5.2 (r252:60911, May 7 2008, 15:19:09)
[GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
print eval(compile('print "It works"\n', '', 'exec'))
It works
None
In 3.0b2,
print( eval(compile('print( "It works")\n', '', 'exec')))
does the same thing. So does
print( exec(compile('print( "It works")\n', '', 'exec')))
and I presume the exec statement would in 2.x.
Personally, I'm absolutely fine with this (because I have a use case,
naturally),
The only possible use I can think of would be is you are eval-ing
mixed-kind code objects:
def eval_and_go(cobj): # cobj = code object of whatever kind
x = eval(cobj)
if x is not None: # was not side-effect only
return g(x)
else:
return None
but eval() does work differently for compiled vs
non-compiled objects, and this behavior doesn't match what the doc at
http://docs.python.org/lib/built-in-funcs.html says, namely "This
function can also be used to execute arbitrary code objects (such as
those created by compile()). In this case pass a code object instead
of a string. The code object must have been compiled passing 'eval' as
the kind argument."
That doc statement is, itself, contradictory. How can it be an
"arbitrary code object" if it must have been compiled using 'eval'?
Perhaps all that is meant is "In order to return a result other than
None to eval's caller, the code object must have been compiled
passing 'eval' as the kind argument."
So, either the code or the doc should be fixed. I would vote +1 for
fixing the doc to match the code, but then I'm the sort of
dysfunctional programmer who sometimes has a use-case for expressions
with side-effects.
I filed a condensed version of this report, with my suggested added text, as
http://bugs.python.org/issue3569
tjr
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] parse_qs and parse_qsl functions
On Sat, Aug 16, 2008 at 7:04 AM, Facundo Batista <[EMAIL PROTECTED]> wrote: > Hi! > > The issue 600362 has two patches (one for 2.6 and the other for 3.0) > that are ready to commit (with a small change in the docs). This > patches relocates the parse_qs and parse_qsl functions into the > urlparse module (urllib.parse in 3k), bringing them from the cgi one. > > For backward compatibility, those functions are also accessible from > the cgi module, but in the documentation says that the function was > moved, and you should use it from the urlparse one. > > So, we don't have *any* change in the behaviour towards the final user. > > Two questions: > > - It's ok to do this now or we should wait for 3.1/2.7? > > - Should we add a deprecation warning in the cgi module for this functions? > > I propose to commit this now, but leave a deprecation warning for the > next release. > Obviously Barry's call, but I think it's fine to do what you are proposing. -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Memory tests in Unicode
Antoine Pitrou wrote: Facundo Batista gmail.com> writes: The test_raiseMemError() in test_unicode.py is complicating me the regression tests: tries to use all the memory in my system, which starts to swap as crazy, and almost freezes everything. When the test finishes (always pass ok), I have all the memory flushed so it take a few seconds to go back to the normal system. If the test does allocate the very large string, it means MemoryError isn't raised, which defeats the purpose of the test. I think that: u"a" * (sys.maxint // 2 - 100) should be replaced with: u"a" * (sys.maxsize - 100) The latter raises MemoryError reliably here, while the former doesn't. In 3.0, sys.maxint is gone, only sys.maxsize remains, so that some change must have been made for 3.0. Just checked: in 3.0b2 test_unicode.py has no test_raisexxx(). An assertRaises could also be used in place of the try/except, it would ensure the test does exercise what it is meant to exercise. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Memory tests in Unicode
2008/8/16 "Martin v. Löwis" <[EMAIL PROTECTED]>: > See bug http://bugs.python.org/issue3556 > > If no solution is forthcoming quickly, I recommend to remove/disable the > test. The Antoine patch works ok for me, and solves the whole issue. I'm on linux, if somebody can give it a try in another platform, we should commit it ASAP... Thanks! -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Memory tests in Unicode
> Are you having also this issue? Do you think that it should run not *always*? See bug http://bugs.python.org/issue3556 If no solution is forthcoming quickly, I recommend to remove/disable the test. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Memory tests in Unicode
Facundo Batista gmail.com> writes: > > I do *not* want to remove the test. You misunderstood my remark. If the test takes a lot of time and eats a lot of memory, it's precisely because the MemoryError isn't raised; and the test needs MemoryError to be raised in order to be meaningful. I was proposing a way of fixing the test so that the MemoryError is raised, which would also stop it from taking a lot of time. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Memory tests in Unicode
On Sat, 16 Aug 2008 13:01:33 -0300, Facundo Batista <[EMAIL PROTECTED]> wrote: 2008/8/16 Antoine Pitrou <[EMAIL PROTECTED]>: If the test does allocate the very large string, it means MemoryError isn't raised, which defeats the purpose of the test. I do *not* want to remove the test. Antoine wasn't suggesting removing it. He's suggesting that the test is not accomplishing its goal if the "except" suite isn't executed, and so the test should be changed to make this failure noticable. Jean-Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] eval() accepts compiled 'exec' statements
This may or may not be relevant to 2.6 or 3.0 (because I don't have
those handy at the moment), but on 2.5 and earlier:
Python 2.5.2 (r252:60911, May 7 2008, 15:19:09)
[GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> print eval(compile('print "It works"\n', '', 'exec'))
It works
None
>>>
Personally, I'm absolutely fine with this (because I have a use case,
naturally), but eval() does work differently for compiled vs
non-compiled objects, and this behavior doesn't match what the doc at
http://docs.python.org/lib/built-in-funcs.html says, namely "This
function can also be used to execute arbitrary code objects (such as
those created by compile()). In this case pass a code object instead
of a string. The code object must have been compiled passing 'eval' as
the kind argument."
That doc statement is, itself, contradictory. How can it be an
"arbitrary code object" if it must have been compiled using 'eval'?
Perhaps all that is meant is "In order to return a result other than
None to eval's caller, the code object must have been compiled
passing 'eval' as the kind argument."
So, either the code or the doc should be fixed. I would vote +1 for
fixing the doc to match the code, but then I'm the sort of
dysfunctional programmer who sometimes has a use-case for expressions
with side-effects.
Thanks,
Pat
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Memory tests in Unicode
2008/8/16 Antoine Pitrou <[EMAIL PROTECTED]>: > If the test does allocate the very large string, it means MemoryError isn't > raised, which defeats the purpose of the test. I do *not* want to remove the test. I just want to execute it *only* when I run "-u all" or "-u memory", not everytime I run the regression tests. Regards, -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Memory tests in Unicode
Facundo Batista gmail.com> writes: > > The test_raiseMemError() in test_unicode.py is complicating me the > regression tests: tries to use all the memory in my system, which > starts to swap as crazy, and almost freezes everything. When the test > finishes (always pass ok), I have all the memory flushed so it take a > few seconds to go back to the normal system. If the test does allocate the very large string, it means MemoryError isn't raised, which defeats the purpose of the test. I think that: u"a" * (sys.maxint // 2 - 100) should be replaced with: u"a" * (sys.maxsize - 100) The latter raises MemoryError reliably here, while the former doesn't. An assertRaises could also be used in place of the try/except, it would ensure the test does exercise what it is meant to exercise. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Memory tests in Unicode
Hi, Le Saturday 16 August 2008 15:43:28 Facundo Batista, vous avez écrit : > The test_raiseMemError() in test_unicode.py is complicating me the > regression tests: tries to use all the memory in my system, which > starts to swap as crazy, and almost freezes everything. On UNIX, it's possible to limit a process memory using RLIMIT_AS: from resource import getrlimit, setrlimit, RLIMIT_AS def limitMemory(soft): try: old_soft, hard = getrlimit(RLIMIT_AS) if old_soft != -1: soft = min(soft, old_soft) except ValueError: hard = -1 setrlimit(RLIMIT_AS, (soft, hard)) Values are in bytes. So it's possible to get MemoryError without using the whole system memory. Victor ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Memory tests in Unicode
Facundo Batista wrote: > The test_raiseMemError() in test_unicode.py is complicating me the > regression tests: tries to use all the memory in my system, which > starts to swap as crazy, and almost freezes everything. When the test > finishes (always pass ok), I have all the memory flushed so it take a > few seconds to go back to the normal system. > > I told myself that this test should run only in some special cases, > but didn't find any flag in regrtest.py (like "largefile", or > "network"), to put this test under. > > Are you having also this issue? Do you think that it should run not *always*? > > It's ok if I add a flag like "memory" that will hold "Tests that use > large amount of memory. Normally these tests take long to run because > use all your memory and make the system swap"? I asked a similar question the other day, so a definite +1 from me. Cheers, Nick. -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- http://www.boredomandlaziness.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] parse_qs and parse_qsl functions
Hi! The issue 600362 has two patches (one for 2.6 and the other for 3.0) that are ready to commit (with a small change in the docs). This patches relocates the parse_qs and parse_qsl functions into the urlparse module (urllib.parse in 3k), bringing them from the cgi one. For backward compatibility, those functions are also accessible from the cgi module, but in the documentation says that the function was moved, and you should use it from the urlparse one. So, we don't have *any* change in the behaviour towards the final user. Two questions: - It's ok to do this now or we should wait for 3.1/2.7? - Should we add a deprecation warning in the cgi module for this functions? I propose to commit this now, but leave a deprecation warning for the next release. Thanks!! -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Memory tests in Unicode
The test_raiseMemError() in test_unicode.py is complicating me the regression tests: tries to use all the memory in my system, which starts to swap as crazy, and almost freezes everything. When the test finishes (always pass ok), I have all the memory flushed so it take a few seconds to go back to the normal system. I told myself that this test should run only in some special cases, but didn't find any flag in regrtest.py (like "largefile", or "network"), to put this test under. Are you having also this issue? Do you think that it should run not *always*? It's ok if I add a flag like "memory" that will hold "Tests that use large amount of memory. Normally these tests take long to run because use all your memory and make the system swap"? Thank you!! -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Implementation of Camellia block cipher for Python Cryptography Toolkit.
> python-dev is not the right place - we deal with the core > Python distribution only. OTOH, amk.ca is probably a good I am sorry to my send email to wrong place, and thank you for letting me know the right place. I will send my email to the python-crypto mailing list, and will also post my patch to the tracker. Thank you, -- Yoshisato Yanagisawa <[EMAIL PROTECTED]> ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Implementation of Camellia block cipher for Python Cryptography Toolkit.
> Will you please review and test it? > I hope pycrypto will include the patch. Dear Yoshisato, python-dev is not the right place - we deal with the core Python distribution only. OTOH, amk.ca is probably a good place; you might also consider posting it to the Python-Crypto mailing list (http://listserv.surfnet.nl/archives/python-crypto.html) The best place for contributions is probably the pycrypto patches tracker, at http://sourceforge.net/projects/pycrypto http://sourceforge.net/tracker/?group_id=20937&atid=320937 Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Implementation of Camellia block cipher for Python Cryptography Toolkit.
Hi, I am afraid that here is the right place to say but I implemented a patch to add support for Camellia block cipher to Python Cryptography Toolkit (aka. pycrypto). Camellia is one of the approved encryption methods by European Union (one of the finalist of NESSIE) and has specified in several RFCs. It is also included in some open source softwares such as Linux, OpenSSL, Firefox 3, and so on. I put the patch at: http://www.csg.is.titech.ac.jp/~yanagisawa/Sites/text/camellia/pycrypto-2.0.1.patch in http://www.csg.is.titech.ac.jp/~yanagisawa/Sites/text/camellia-e.html Will you please review and test it? I hope pycrypto will include the patch. Thank you in advance, -- Yoshisato Yanagisawa <[EMAIL PROTECTED]> ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
