Re: [Python-Dev] A Subtle Bug in Class Initializations
This would be a good thing to fix. The only hard part is dealing with
thirdparty extensions.
Note we also have been working around this problem by putting PyType_Ready
calls in various generic code paths of the interpreter when an uninitialized
type passes through.
On Mon, Aug 6, 2018, at 11:02, Eddie Elizondo wrote:
> Solution:
> Here are my proposed solutions in order from less controversial to most
> controversial. Note that all of them I already tried in a local branch
> and are working:
>
> 1) Initialize all uninitialized types.
>
> Example:
> https://github.com/eduardo-elizondo/cpython/commit/bc53db3cf4e5a6923b0b1afa6181305553faf173
That fixes CPython but what about thirdparty extensions?
>
> 2) Move all PyVarObject_HEAD_INIT to NULL except PyType_Type,
> PyBaseObject_Type and the booleans.
>
> 3) Special case the initialization of PyType_Type and PyBaseObject_Type
> within PyType_Ready to now make all calls to PyVarObject_HEAD_INIT use
> NULL. To enable this a small change within PyType_Ready is needed to
> initialize PyType_Type PyBaseObject:
> if (base == NULL) {
> Py_TYPE(_Type) = _Type;
> Py_TYPE(type) = _Type;
> }
>
> Also, the booleans have to be fully initialized without calling
> PyVarObject_HEAD_INIT. I propose:
>
> struct _longobject _Py_FalseStruct = {
> PyObject_HEAD_INIT(_Type), 0, { 0 }
> };
>
> This will clean-up the entire codebase of this anti-pattern.
>
> Example:
> https://github.com/eduardo-elizondo/cpython/commit/542fd79e4279c64c077c127b175a8d856d3c5f0b
>
> 4) Modify PyVarObject_HEAD_INIT to ignore the input and initialize to
> NULL and 0.
> In order to prevent this antipattern within extension code as well, we
> should make PyVarObject_HEAD_INIT ignore the inputs and just set the
> value to NULL.
>
> #define PyVarObject_HEAD_INIT(type, size) \
> { PyObject_HEAD_INIT(NULL) 0 },
>
> This will prevent external code to have a semi-initialized type that is
> not initialized through PyType_Ready.
When does that fail, though? The first time someone tries to do anything with
the type?
>
> 5) Finally, I would go even further and suggest making
> PyVarObject_HEAD_INIT argumentless.
>
> #define PyVarObject_HEAD_INIT \
> { PyObject_HEAD_INIT(NULL) 0 },
>
> However, this breaks backward compatibility. That being said, this will
> make extension maintainers aware of this antipattern.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The curious case of 255 function arguments
Hi Andrea and Serhiy,
Thank you for your responses and clarifying that it is specifically the
CALL_FUNCTION. I tested this in my megascript and it will fail when trying
to call the functions directly and receive an error then (Py 2.x: fail at
call invocation, Py 3.y w/ y<7: fail at function definition).
@Serhiy I looked through the commits and had found
https://github.com/python/cpython/commit/5bb8b9134b0bb35a73c76657f41cafa3e4361fcd#diff-4d35cf8992b795c5e97e9c8b6167cb34
but the commit that removed the 255 checks also explains that this is
specifically about the call function (
https://github.com/python/cpython/commit/214678e44bf7773c0ed9c3684818354001d8f9ca#diff-4d35cf8992b795c5e97e9c8b6167cb34
), so indeed I should have been able to answer this myself.
The reason why I originally had encountered this was (as discussed in one
of the bug reports) from code that was generating a class hierarchy to
represent Doxygen's XML schema. The class constructors had >255 arguments,
but in executing the code it actually does still work in python 2.x. The
reason is because all of the arguments are defaulted to None, and during
execution of typical sample XML files, the explicit construction with all
>255 arguments virtually never happens.
f.write("def foo_2({0}):\n".format(",
".join(["a{0}=None".format(str(i)) for i in range(300)])))
f.write("print('foo_2 executed')\n\n")
# ... in generated __main__ ...
f.write("foo_2()\n\n")
foo_2() will succeed in python 2.x because the CALL_FUNCTION is not
explicitly getting more than 255 parameters. Very interesting!
Thank you both again for your responses, I am grateful to finally
understand the way in which success / failure works here :)
-Stephen
On Mon, Aug 6, 2018 at 2:17 AM, Serhiy Storchaka
wrote:
> 06.08.18 08:13, Stephen McDowell пише:
>
>> I've looked at the C code for a while and it is entirely non-obvious what
>> would lead to python *2* /allowing/ >255 arguments. Anybody happen to know
>> how / why the python *2* versions *succeed*?
>>
>
> The error message is misleading. It should be "more than 255 parameters".
> This limitation is due to the optimization used in Python 3 for call
> variables (see https://bugs.python.org/issue12399 for details).
>
> In all versions <3.7 there is a limitation on the number of explicit
> function arguments because of the limitation of the CALL_FUNCTION opcode.
>
> Thank you for reading, this is not a problem, just a burning desire for
>> closure (even if anecdotal) as to how this can be. I deeply love python,
>> and am not complaining! I stumbled across this and found it truly
>> confounding, and thought the gurus here may happen to recall what changed
>> in 3.x that lead the the error condition actually being asserted :)
>>
>
> Read the history of the code. Commit messages usually contain explanations
> or references to issues.
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/sjm324%
> 40cornell.edu
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Refactor __get_builtin_constructor on hasklib.py
Hi there,
Good to see you on python-dev. It's always good to see people getting
excited about helping with Python's development.
The changes you suggest are, unless I've missed something, purely cosmetic
- affecting readability only. This implies that you feel the code as it
stands isn't as readable as it could be. You must admit that "it looks
messy" is a matter of opinion, and alone isn't much of a justification for
making changes to a project that will reach its end of life in less than
eighteen months.
The code in the standard library is mostly fairly well-scrutinised, and as
PEP 8 reminds us, change made for purely stylistic reasons threatens to
introduce new bugs.
It's not obvious how much of the developer documentation you've seen, so it
might be worth mentioning https://devguide.python.org/ as a good starting
point for anyone wanting to contribute.
regards
Steve
Steve Holden
On Tue, Aug 7, 2018 at 8:00 AM, 蔡銘峯 wrote:
> Hello everybody,
> I am Park Tsai. I want to refactor __get_builtin_constructor on hasklib.py
> of python 2.7 (https://github.com/python/cpython/blob/2.7/Lib/hashlib.
> py#L72).
> This is the first time that I try to refactor code of CPython on GitHub,
> so I am very excited.
>
> This is __get_builtin_constructor code on hasklib.py ,as follows.
>
> def __get_builtin_constructor(name):
> try:
> if name in ('SHA1', 'sha1'):
> import _sha
> return _sha.new
> elif name in ('MD5', 'md5'):
> import _md5
> return _md5.new
> elif name in ('SHA256', 'sha256', 'SHA224', 'sha224'):
> import _sha256
> bs = name[3:]
> if bs == '256':
> return _sha256.sha256
> elif bs == '224':
> return _sha256.sha224
> elif name in ('SHA512', 'sha512', 'SHA384', 'sha384'):
> import _sha512
> bs = name[3:]
> if bs == '512':
> return _sha512.sha512
> elif bs == '384':
> return _sha512.sha384
> except ImportError:
> pass # no extension module, this hash is unsupported.
>
> raise ValueError('unsupported hash type ' + name)
>
>
> When I read this code, it looks messy, so I want to refactor it and make
> it become more clearly .
>
> Then, it will be like this
>
> def get_builtin_constructor(name):
> try:
> if name[:3] in ('SHA','sha'):
>if(name[3:]=='1'):
>import _sha
>return _sha.new
>
>elif (name[3:] == '224'):
>import _sha256
>return _sha256.sha224
>
>elif (name[3:] == '256'):
>import _sha256
>return _sha256.sha256
>
>elif (name[3:] == '384'):
>import _sha512
>return _sha512.sha384
>
>elif (name[3:] == '512'):
>import _sha512
>return _sha512.sha512
> elif name in ('MD5', 'md5'):
> import _md5
> return _md5.new
>
> except ImportError:
> pass # no extension module, this hash is unsupported.
>
> raise ValueError('unsupported hash type ' + name)
>
> I will be grateful for any help you can provide. I really appreciate any
> feedback you can offer!
>
> Best regards,
> Park Tsai !!
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> steve%40holdenweb.com
>
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Use of Cython
On Mon, Aug 6, 2018 at 11:49 AM Ronald Oussoren via Python-Dev wrote: > I have no strong opinion on using Cython for tests or in the stdlib, other > than that it is a fairly large dependency. I do think that adding a > “Cython-lite” tool the CPython distribution would be less ideal, creating and > maintaining that tool would be a lot of work without clear benefits over just > using Cython. Speaking of which, Dropbox is working on a new compiler they call "mypyc". mypyc will compile type-annotated Python code to an optimized C. The first goal is to compile mypy with it to make it faster, so I hope that the project will be completed. Essentially, mypyc will be similar to Cython, but mypyc is a *subset of Python*, not a superset. Interfacing with C libraries can be easily achieved with cffi. Being a strict subset of Python means that mypyc code will execute just fine in PyPy. They can even apply some optimizations to it eventually, as it has a strict and static type system. I'd be more willing to start using mypyc+cffi in CPython stdlib *eventually*, than Cython now. Cython is a relatively complex and still poorly documented language. I'm speaking from experience after writing thousands of lines of Cython in uvloop & asyncpg. In skillful hands Cython is amazing, but I'd be cautious to advertise and use it in CPython. I'm also -1 on using Cython to test C API. While writing C tests is annoying (I wrote a fair share myself), their very purpose is to make third-party tools/extensions more stable. Using a third-party tool to test C API to track regressions that break third-party tools feels wrong. Yury ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Refactor __get_builtin_constructor on hasklib.py
2.7 is for bug fixes only. Unless there is a bug to be fixed, I would leave
the code as is.
Mariatta
On Tue, Aug 7, 2018 at 8:14 AM 蔡銘峯 wrote:
> Hello everybody,
> I am Park Tsai. I want to refactor __get_builtin_constructor on hasklib.py
> of python 2.7 (
> https://github.com/python/cpython/blob/2.7/Lib/hashlib.py#L72).
> This is the first time that I try to refactor code of CPython on GitHub,
> so I am very excited.
>
> This is __get_builtin_constructor code on hasklib.py ,as follows.
>
> def __get_builtin_constructor(name):
> try:
> if name in ('SHA1', 'sha1'):
> import _sha
> return _sha.new
> elif name in ('MD5', 'md5'):
> import _md5
> return _md5.new
> elif name in ('SHA256', 'sha256', 'SHA224', 'sha224'):
> import _sha256
> bs = name[3:]
> if bs == '256':
> return _sha256.sha256
> elif bs == '224':
> return _sha256.sha224
> elif name in ('SHA512', 'sha512', 'SHA384', 'sha384'):
> import _sha512
> bs = name[3:]
> if bs == '512':
> return _sha512.sha512
> elif bs == '384':
> return _sha512.sha384
> except ImportError:
> pass # no extension module, this hash is unsupported.
>
> raise ValueError('unsupported hash type ' + name)
>
>
> When I read this code, it looks messy, so I want to refactor it and make
> it become more clearly .
>
> Then, it will be like this
>
> def get_builtin_constructor(name):
> try:
> if name[:3] in ('SHA','sha'):
>if(name[3:]=='1'):
>import _sha
>return _sha.new
>
>elif (name[3:] == '224'):
>import _sha256
>return _sha256.sha224
>
>elif (name[3:] == '256'):
>import _sha256
>return _sha256.sha256
>
>elif (name[3:] == '384'):
>import _sha512
>return _sha512.sha384
>
>elif (name[3:] == '512'):
>import _sha512
>return _sha512.sha512
> elif name in ('MD5', 'md5'):
> import _md5
> return _md5.new
>
> except ImportError:
> pass # no extension module, this hash is unsupported.
>
> raise ValueError('unsupported hash type ' + name)
>
> I will be grateful for any help you can provide. I really appreciate any
> feedback you can offer!
>
> Best regards,
> Park Tsai !!
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/mariatta.wijaya%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Refactor __get_builtin_constructor on hasklib.py
Hello,
Welcome to the mailing list Park!
On Tue, Aug 7, 2018 at 12:30 PM, 蔡銘峯 wrote:
> Hello everybody,
> I am Park Tsai. I want to refactor __get_builtin_constructor on hasklib.py
> of python 2.7
> (https://github.com/python/cpython/blob/2.7/Lib/hashlib.py#L72).
> This is the first time that I try to refactor code of CPython on GitHub, so
> I am very excited.
>
> This is __get_builtin_constructor code on hasklib.py ,as follows.
>
> def __get_builtin_constructor(name):
> try:
> if name in ('SHA1', 'sha1'):
> import _sha
> return _sha.new
> elif name in ('MD5', 'md5'):
> import _md5
> return _md5.new
> elif name in ('SHA256', 'sha256', 'SHA224', 'sha224'):
> import _sha256
> bs = name[3:]
> if bs == '256':
> return _sha256.sha256
> elif bs == '224':
> return _sha256.sha224
> elif name in ('SHA512', 'sha512', 'SHA384', 'sha384'):
> import _sha512
> bs = name[3:]
> if bs == '512':
> return _sha512.sha512
> elif bs == '384':
> return _sha512.sha384
> except ImportError:
> pass # no extension module, this hash is unsupported.
>
> raise ValueError('unsupported hash type ' + name)
>
>
> When I read this code, it looks messy, so I want to refactor it and make it
> become more clearly .
>
> Then, it will be like this
>
> def get_builtin_constructor(name):
> try:
> if name[:3] in ('SHA','sha'):
>if(name[3:]=='1'):
>import _sha
>return _sha.new
>
>elif (name[3:] == '224'):
>import _sha256
>return _sha256.sha224
>
>elif (name[3:] == '256'):
>import _sha256
>return _sha256.sha256
>
>elif (name[3:] == '384'):
>import _sha512
>return _sha512.sha384
>
>elif (name[3:] == '512'):
>import _sha512
>return _sha512.sha512
IMHO, this decreases the readability of the code. Also, we're doing
string slicing at every conditional statement which doesn't make much
sense.
What do you find not interesting with the current code? What is the
motivation behind this change?
Best,
Sanyam
--
Mozilla Rep
http://www.SanyamKhurana.com
Github: CuriousLearner
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Refactor __get_builtin_constructor on hasklib.py
Hello everybody,
I am Park Tsai. I want to refactor __get_builtin_constructor on hasklib.py
of python 2.7 (https://github.com/python/cpython/blob/2.7/Lib/hashlib.py#L72).
This is the first time that I try to refactor code of CPython on GitHub, so
I am very excited.
This is __get_builtin_constructor code on hasklib.py ,as follows.
def __get_builtin_constructor(name):
try:
if name in ('SHA1', 'sha1'):
import _sha
return _sha.new
elif name in ('MD5', 'md5'):
import _md5
return _md5.new
elif name in ('SHA256', 'sha256', 'SHA224', 'sha224'):
import _sha256
bs = name[3:]
if bs == '256':
return _sha256.sha256
elif bs == '224':
return _sha256.sha224
elif name in ('SHA512', 'sha512', 'SHA384', 'sha384'):
import _sha512
bs = name[3:]
if bs == '512':
return _sha512.sha512
elif bs == '384':
return _sha512.sha384
except ImportError:
pass # no extension module, this hash is unsupported.
raise ValueError('unsupported hash type ' + name)
When I read this code, it looks messy, so I want to refactor it and make it
become more clearly .
Then, it will be like this
def get_builtin_constructor(name):
try:
if name[:3] in ('SHA','sha'):
if(name[3:]=='1'):
import _sha
return _sha.new
elif (name[3:] == '224'):
import _sha256
return _sha256.sha224
elif (name[3:] == '256'):
import _sha256
return _sha256.sha256
elif (name[3:] == '384'):
import _sha512
return _sha512.sha384
elif (name[3:] == '512'):
import _sha512
return _sha512.sha512
elif name in ('MD5', 'md5'):
import _md5
return _md5.new
except ImportError:
pass # no extension module, this hash is unsupported.
raise ValueError('unsupported hash type ' + name)
I will be grateful for any help you can provide. I really appreciate any
feedback you can offer!
Best regards,
Park Tsai !!
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [python-committers] [RELEASED] Python 3.4.9 and Python 3.5.6 are now available
On 8/6/2018 11:38 AM, Charalampos Stratakis wrote: > A side note on your side note. Different distro's have different > standards, use/customer cases to address etc. In enterprise > distributions the usual scheme is that the version that you see is the > minimum one and many fixes coming from upstream or the redistributor > are incorporated on top of that version. Just check the package > changelogs. :) CVE's do get fixed and there is actually cooperation > with upstream on different levels in regards to those. And speaking > here as one of the people doing that for one of the enterprise > distros. > a) good to hear b) On AIX they stayed with ssh at version 6.0 for so long, that even with all the CVE et al included it was still extremely weak compared to 6.7 and later when they tightened the default ciphers. And yes, I fell over the change - but was glad, in the end, to rid of weak ssh clients. c) read package changelogs. The :) is because they are hard to read or non-existent. I do not mean to criticize any "enterprise" methods. My "enterprise" of choice is AIX and when it comes to OSS I dare say everyone else does a better job (which is why I got started with packaging in the first place - but only what I need and/or someone requests). However, I do find it very very hard to know what python 2.7.5 has or has not, that 2.7.15 now has. There are, iirc, quite a few important changes. The "hard" freeze seems to have come at roughly 2.7.8 or 2.7.9 (just a guess). Also, as I am trying to test on other platforms it gets a bit frustrating when the latest python3 I can find is a v3.4.X. Might be good project developers (in general, not meant as specific to python) to understand that version number changes are not followed - blindly - by enterprise patch management and being too quick with version number changes will make it more difficult for users to know what they have. p.s. I do not do this (packaging/patch management) for any "distro". In that sense I am "just a consumer" who "rolls his own" when/if needed. pEpkey.asc Description: application/pgp-keys ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
