[Python-Dev] Re: PEP-376 and PEP-427 interpretation
I meant to exclude md5 and sha1, e.g. hash functions with known problems. SHA224 would be a weird choice but it wouldn't personally offend me otherwise. It would be fun to see how many wheel handlers support non-sha256 hash functions. On Mon, Mar 29, 2021 at 9:56 PM Theallredman via Python-Dev < python-dev@python.org> wrote: > Thank you. I can't think of a compelling reason someone would want to > choose SHA224 over SHA256 in the context of wheel generation. It just that > the PEPs are usually pretty explicit and SHA224 seemed to be implicitly > excluded from RECORD files. And I'm considering the details of making a > pretty pedantic wheel generation PEP517 backend. > > Eldon > > ‐‐‐ Original Message ‐‐‐ > On Monday, March 29, 2021 2:16 PM, Paul Moore wrote: > > > On Mon, 29 Mar 2021 at 17:40, Theallredman via Python-Dev > > python-dev@python.org wrote: > > > > > So going back to my actual question SHA224 is disallowed in record > files because it's bit length is less then 256? > > > > It doesn't look like it's ever been excluded. The only explicit > > exclusions are MD5 and SHA1 as you point out. Do you have a particular > > reason to want to use SHA224? Pretty much everyone is using SHA256, as > > far as I know. > > > > Paul > > > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/NB6MLDDDJHRTRMOEWDFG5IYGZCP65K6V/ > Code of Conduct: http://python.org/psf/codeofconduct/ > ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/P4575PDTQCUYMYZVOSJPYGD3J5YKNCFY/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP: Modify the C API to hide implementation details
Sorry that this is a bit off-topic. cffi would be a user of any new C API. I've tried to make sure ABI3 is supported in setuptools and wheel, with varying success. Apparently virtualenvs and Windows have problems. I'm excited about the possibility of a better C API and possibly ABI. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/XFHKIXSEEZMCC7HZ3N4GP6SBUBYACI6K/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP: Modify the C API to hide implementation details
It can be done exactly as passing a void* when registering a C callback, and getting it passed back to your callback function. https://cffi.readthedocs.io/en/latest/ref.html#ffi-new-handle-ffi-from-handle https://bitbucket.org/dholth/kivyjoy/src/aaeab79b2891782209a1219cd65a4d9716cea669/kivyjoy/controller.py#lines-49 https://bitbucket.org/dholth/kivyjoy/src/aaeab79b2891782209a1219cd65a4d9716cea669/kivyjoy/__init__.py#lines-15 ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/QDHQANZJAKPTTOJHXLGFSJWKLE5NA6ZU/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP: Modify the C API to hide implementation details
Was it regular cffi or cffi's embedding API, which is used a bit
differently than regular cffi, that "seems to only solve a fraction of the
problem"? Was just playing around with the embedding API and was impressed.
In Python:
@ffi.def_extern()
def uwsgi_pyexample_init():
print("init called")
return 0
In C (embedded in the same plugin):
CFFI_DLLEXPORT struct uwsgi_plugin pyexample_plugin = {
.init = uwsgi_pyexample_init
};
Seems to be happily importing and exporting APIs. Interpreter starts the
first time a @ffi.def_extern() function is called.
https://cffi.readthedocs.io/en/latest/embedding.html
https://github.com/unbit/uwsgi/blob/f6ad0c6dfe431d91ffe365bed3105ed052bef6e4/plugins/pyexample/pyexample_plugin.py
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/S5P524LXXNIXIB2MUATZNHYQE57MXCQ5/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: python3 -bb and hash collisions
On Tue, Sep 10, 2019 at 8:38 PM Matt Billenstein wrote:
> On Tue, Sep 10, 2019 at 10:42:52AM -0400, Daniel Holth wrote:
> > I stopped using Python 3 after learning about str(bytes) by finding it
> in my
> > corrupted database. Ever since then I've been anxious about changing to
> the new
> > language, since it makes it so easy to convert from bytes to unicode by
> > accident without specifying a valid encoding. So I would like to see a
> future
> > where str(bytes) is effectively removed. I started working on a pull
> request
> > that adds an API to toggle str(bytes) at runtime with a thread local
> (instead
> > of requiring a command line argument), so you could do with
> no_str_bytes(): if
> > you were worried about the feature, but got a bit stuck in the internals.
>
> How is this different than all the str -> unicode bugs we had in python2?
> If
> you have special needs, you can always monkey-patch it in plain python
> code by
> overriding __builtins__.str with something that asserts the given arg is
> not
> bytes.
>
> m
>
> --
> Matt Billenstein
> m...@vazor.com
> http://www.vazor.com/
>
It's different. One hint is that there's already an option to disable the
feature. The old style of error will occasionally reveal itself with decode
errors but the new style error happens silently, you discover it somehow,
then enable the -bb option, track down the source of the error, and deal
with the fallout. The proposed change would allow `print(bytes)` for
(de)bugging by letting you toggle `python3 -bb` behavior at runtime instead
of only at the command line. Or you could debug more explicitly by
`print(bytes.decode('ebcdic'))` or `print(repr(bytes))`
I didn't realize you could override __builtins__.str. That's interesting.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/I5RQRUGLAKLZ6TLPSO2S5VRL5D4CQNGT/
[Python-Dev] Re: python3 -bb and hash collisions
On Sat, Jun 22, 2019 at 2:48 AM Serhiy Storchaka wrote: > 22.06.19 01:08, Daniel Holth пише: > > Thanks. I think I might like an option to disable str(bytes) without > > disabling str != bytes. Unless the second operation would also corrupt > > output. > > > > Came across this kind of set in the hyper http library which uses a set > > to accept certain headers with either str or bytes keys. > > Does that library support Python 2? If it is true than you have a > problem, because u'abc' == b'abc' in Python 2 and u'abc' != b'abc' in > Python 3. > > If it is Python 3 only, you can just ignore BytesWarning. It was added > purely to help to catch subtle bugs in transition to Python 3. In > future, after Python 2 be out of use, BytesWarning will become deprecated. > I stopped using Python 3 after learning about str(bytes) by finding it in my corrupted database. Ever since then I've been anxious about changing to the new language, since it makes it so easy to convert from bytes to unicode by accident without specifying a valid encoding. So I would like to see a future where str(bytes) is effectively removed. I started working on a pull request that adds an API to toggle str(bytes) at runtime with a thread local (instead of requiring a command line argument), so you could do with no_str_bytes(): if you were worried about the feature, but got a bit stuck in the internals. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/5HIHFIJODYSLFCEASM37K3V7XPAF56PK/
[Python-Dev] Re: python3 -bb and hash collisions
The answer bytes == str is just False. That doesn't put b'' in your database by accident. It could be useful to separate the two kinds of warnings. On Fri, Jun 21, 2019, 18:57 Ivan Pozdeev via Python-Dev < python-dev@python.org> wrote: > On 22.06.2019 1:08, Daniel Holth wrote: > > Thanks. I think I might like an option to disable str(bytes) without > disabling str != bytes. Unless the second operation would also corrupt > output. > > You can't compare str to bytes without knowing the encoding the bytes are > supposed to be in (see > https://stackoverflow.com/questions/49991870/python-default-string-encoding > for details). > > And if you do know the encoding, you can as well compare > `str.encode(encoding) != bytes` / `str != bytes.decode(encoding)`. > > > Came across this kind of set in the hyper http library which uses a set to > accept certain headers with either str or bytes keys. > > On Tue, Jun 18, 2019, 13:05 Christian Heimes wrote: > >> On 18/06/2019 18.32, Daniel Holth wrote: >> > set([u"foo", b"foo]) will error because the two kinds of string have the >> > same hash, and this causes a comparison. Is that correct? >> >> Yes, it will fail with -bb, because it turns comparison between str and >> bytes into an error. This can also happen with other strings when >> hash(u'somestring') & mask == hash(b'otherbytes') & mask. The mask of a >> set starts with PySet_MINSIZE - 1 == 8 and increases over team. >> >> Christian >> >> >> ___ >> Python-Dev mailing list -- python-dev@python.org >> To unsubscribe send an email to python-dev-le...@python.org >> https://mail.python.org/mailman3/lists/python-dev.python.org/ >> Message archived at >> https://mail.python.org/archives/list/python-dev@python.org/message/ZIF2MRBWSMSCFP6E7PZOBI5KYP46QZPK/ >> > > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to > python-dev-leave@python.orghttps://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/R6E7FAR36UO6XHQSIAVF4DIM7G23ADJP/ > > -- > Regards, > Ivan > > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/XAN44UH5X5PYNSHY5ONULXIJF4DLBXF6/ > ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/36DKFLVTBABEZPDX7MYHP7H2TVDZTOHG/
[Python-Dev] Re: python3 -bb and hash collisions
Thanks. I think I might like an option to disable str(bytes) without disabling str != bytes. Unless the second operation would also corrupt output. Came across this kind of set in the hyper http library which uses a set to accept certain headers with either str or bytes keys. On Tue, Jun 18, 2019, 13:05 Christian Heimes wrote: > On 18/06/2019 18.32, Daniel Holth wrote: > > set([u"foo", b"foo]) will error because the two kinds of string have the > > same hash, and this causes a comparison. Is that correct? > > Yes, it will fail with -bb, because it turns comparison between str and > bytes into an error. This can also happen with other strings when > hash(u'somestring') & mask == hash(b'otherbytes') & mask. The mask of a > set starts with PySet_MINSIZE - 1 == 8 and increases over team. > > Christian > > > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/ZIF2MRBWSMSCFP6E7PZOBI5KYP46QZPK/ > ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/R6E7FAR36UO6XHQSIAVF4DIM7G23ADJP/
[Python-Dev] python3 -bb and hash collisions
set([u"foo", b"foo]) will error because the two kinds of string have the same hash, and this causes a comparison. Is that correct? ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/MRKTBTY7ZTU3I6SCWRHB533O2QMYA4VR/
Re: [Python-Dev] Feature request: Change a Dependency Package Version During Package Initiation
This sounds exactly like what people used to do with eggs. You could have multiple versions of a package on the path as eggs and then require a version at runtime. The approach has problems. Ruby also abandoned a strategy where random app code depends on package management code at runtime. One better strategy is to set up a python path in a wrapper script. On Fri, May 17, 2019, 11:27 Brett Cannon wrote: > Thanks for the idea but there are currently no plans to support such a > feature. If you would like to see it then you will need to write a PEP with > a proof-of-concept to demonstrate how you would expect such a feature to > work. > > On Fri., May 17, 2019, 07:55 Q via Python-Dev, > wrote: > >> A lot of the Python code we use in production are used directly as >> imports in other python >> distributions (such as the python comes with the finite element software >> Abaqus and MSC Marc), many >> packages (such as matplotlib, numpy) that may have varying versioned >> dependencies. >> >> I was wondering if this could be expanded to allow a version to be set >> within a package and have >> that propagate to all modules in that package. For example in the root >> init.py if I set >> multiversion(tornado, 2.2.1) then all modules in that package will use >> tornado 2.2.1 when I import >> tornado. >> >> See a relevant issue on github: >> https://github.com/mitsuhiko/multiversion/issues/1 >> >> Thank you! >> Qiang >> >> ___ >> Python-Dev mailing list >> Python-Dev@python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/brett%40python.org >> > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Imports with underscores
Easily solved with the totally evil ninja mode pattern of module initialization. It has yet to catch on. def ninja(): global exported import os def exported(): # do something ninja() del ninja On Mon, Jan 9, 2017 at 1:13 PM Sven R. Kunze wrote: > Interesting to see that others have the same problem. > > We also had this kind of "over-protective" behavior. As far as I know, our > devs stopped doing it as it feels cumbersome. > > > Another argument for this is: when using PyCharm, this IDE will suggest > imports from those modules which aren't the original ones. So, you might > import from a third-party module. Over time, however, people learn to pick > the "right" module to import from. > > Cheers, > Sven > > > On 09.01.2017 12:42, Steve Holden wrote: > > One of my developers recently submitted a pull request incuding a number > of lines like > > import os as _os > > When I asked him why he suggested a) this would improve encapsulation, and > b) the practice was supported in the stdlib. Further investigation reveals > that some modules (e.g. argparse, crypt, difflib, random) do use this > technique, but it is far from universal. > > So I thought it would be useful to get input from current devs about the > value of this practice, since to me it seems somewhat anti-pythonic. What > advantages does it confer? > > regards > Steve Holden > > > ___ > Python-Dev mailing > listPython-Dev@python.orghttps://mail.python.org/mailman/listinfo/python-dev > > > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/srkunze%40mail.de > > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] File system path encoding on Windows
A nice recent blog post about paths encoding in a media player. http://beets.io/blog/paths.html . It's not merely the porting that makes it hard. On Mon, Aug 29, 2016, 19:32 Victor Stinner wrote: > 2016-08-24 17:44 GMT+02:00 Steve Dower : > > I know Nick and Victor like the idea of a -X flag (or a direct -utf8 > flag), > > but I prefer more specific environment variables: > > > > - PYTHONWINDOWSLEGACYSTDIO (for the console changes) > > - PYTHONWINDOWSLEGACYPATHENCODING (assuming getfilesystemencoding() is > utf8) > > - PYTHONWINDOWSLEGACYLOCALEENCODING (assuming getpreferredencoding() is > > utf8) > > Hum, you propose the opposite behaviour of my proposed "-X utf8" > switch :-) You propose to force UTF-8 by default, but add a new flag > to revert to the old behaviour. Right? > > About (...)LEGACYSTDIO: I don't think that we need such option. Using > Unicode is the obvious choice if stdout/stderr is the Windows console > no? I'm not talking about the case where stdout/stderr is redirected > (into a file or a pipe). Are you aware of any case where the "legacy > stdio" is better than the "new" (win_unicode_console-like) stdio? > > (...)PATHENCODING and (...)LOCALEENCODING: please don't do that. I > tried once in Python 3.2 on Linux, and it failed badly: > https://mail.python.org/pipermail/python-dev/2010-October/104517.html > > I don't think that it's a good idea to add an option to use two > different encodings inside the same process. It's too common to > exchange data between the "path encoding world" and "locale encoding > world". > > Hum, currently, is it possible to get two different encodings for > "path encoding" and "locale encoding"? Or both encodings are the ANSI > code page? I only recall that the console uses a different encoding, > the OEM code page. > > Victor > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] File system path encoding on Windows
#1 sounds like a great idea. I suppose surrogatepass solves approximately the same problem of Rust's WTF-8, which is a way to round-trip bad UCS-2? https://simonsapin.github.io/wtf-8/ #2 sounds like it would leave several problems, since mbcs is not the same as a normal text encoding, IIUC it depends on the active code page. So if your active code page is Russian you might not be able to encode Japanese characters into MBCS. Solution #2a Modify Windows so utf-8 is a valid value for the current MBCS code page. On Fri, Aug 19, 2016 at 3:01 PM Steve Dower wrote: > Hi python-dev > > About a week ago I proposed on python-ideas making some changes to how > Python deals with encodings on Windows, specifically in relation to how > Python interacts with the operating system. > > Changes to the console were uncontroversial, and I have posted patches > at http://bugs.python.org/issue1602 and > http://bugs.python.org/issue17620 to enable the full range of Unicode > input to be used at interactive stdin/stdout. > > However, changes to sys.getfilesystemencoding(), which determines how > the os module (and most filesystem functions in general) interpret bytes > parameters, were more heatedly discussed. I've summarised the discussion > in this email > > I'll declare up front that my preferred change is to treat bytes as > utf-8 in Python 3.6, and I've posted a patch to do that at > http://bugs.python.org/issue27781. Hopefully I haven't been too biased > in my presentation of the alternatives, but this is so you at least know > which way I'm biased. > > I'm looking for some agreement on the answers to the questions I pose in > the summary. > > There is much more detail about them presented after that, as there are > a number of non-obvious issues at play here. I suspect this will > eventually become a PEP, but it's presented here as a summary of a > discussion and not a PEP. > > Cheers, > Steve > > Summary > === > > Representing file system paths on Windows as bytes may result in data > loss due to the way Windows encodes/decodes strings via its bytes API. > > We can mitigate this by only using Window's Unicode API and doing our > own encoding and decoding (i.e. within posixmodule.c's path converter). > Invalid characters could cause encoding exceptions rather than data loss. > > We can go further to fix this by declaring the encoding of bytes paths > on Windows must be utf-8, which would also prevent encoding exceptions, > as utf-8 can fully represent all paths on Windows (natively utf-16-le). > > Even though using bytes for paths on Windows has been deprecated for > three releases, this is not widely known and it may be too soon to > change the behaviour. > > Questions: > * should we always use Window's Unicode APIs instead of switching > between bytes/Unicode based on parameter type? > * should we allow users to pass bytes and interpret them as utf-8 rather > than letting Windows do the decoding? > * should we do it in 3.6, 3.7 or 3.8? > > Background > == > > File system paths are almost universally represented as text in some > encoding determined by the file system. In Python, we expose these paths > via a number of interfaces, such as the os and io modules. Paths may be > passed either direction across these interfaces, that is, from the > filesystem to the application (for example, os.listdir()), or from the > application to the filesystem (for example, os.unlink()). > > When paths are passed between the filesystem and the application, they > are either passed through as a bytes blob or converted to/from str using > sys.getfilesystemencoding(). The result of encoding a string with > sys.getfilesystemencoding() is a blob of bytes in the native format for > the default file system. > > On Windows, the native format for the filesystem is utf-16-le. The > recommended platform APIs for accessing the filesystem all accept and > return text encoded in this format. However, prior to Windows NT (and > possibly further back), the native format was a configurable machine > option and a separate set of APIs existed to accept this format. The > option (the "active code page") and these APIs (the "*A functions") > still exist in recent versions of Windows for backwards compatibility, > though new functionality often only has a utf-16-le API (the "*W > functions"). > > In Python, we recommend using str as the default format because (with > the surrogateescape handling on POSIX), it can correctly round-trip all > characters used in paths. On Windows this is strongly recommended > because the legacy OS support for bytes cannot round-trip all characters > used in paths. Our support for bytes explicitly uses the *A functions > and hence the encoding for the bytes is "whatever the active code page > is". Since the active code page cannot represent all Unicode characters, > the conversion of a path into bytes can lose information without warning > (and we can't get a warning from the OS here - more on this later). > > A
Re: [Python-Dev] PyPI index deprecation
You may know that there are approximately 3 pypi maintainers, all overworked and one paid. It is amazing that it works at all. I don't know anything about that particular decision though. On Wed, Jul 13, 2016 at 1:21 PM Dmitry Trofimov < dmitry.trofi...@jetbrains.com> wrote: > Hi, > > as you probably already know, today the PyPI index page ( > https://pypi.python.org/pypi?%3Aaction=index) was deprecated and ceased > to be. > > Among other things it affected PyCharm IDE that relied on that page to > enable packaging related features from the IDE. As a result users of > PyCharm can no longer install/update PyPI packages from PyCharm. > > Here is an issue about that in our tracker: > https://youtrack.jetbrains.com/issue/PY-20081 > > Given that there are several hundred thouthands of PyCharm users in the > world -- all 3 editions: Professional, Community, and Educational are > affected -- this can lead to a storm of a negative feedback, when people > will start to face the denial of the service. > > The deprecation of the index was totally unexpected for us and we weren't > prepared for that. Maybe we missed some announcement. > > We will be very happy if the functionality of the index is restored at > least for some short > period of time: please, give as a couple of weeks. That will allow us to > implement a workaround and provide the fix for the several latest major > versions of PyChram. > > Does anybody know who is responsible for that decision and whom to > connect about it? Please help. > > Best regards, > > Dmitry Trofimov > PyCharm Team Lead > JetBrainshttp://www.jetbrains.com > The Drive To Develop > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Should PY_SSIZE_T_CLEAN break Py_LIMITED_API?
I was using Py_LIMITED_API under 3.5 and PY_SSIZE_T_CLEAN was set, this causes some functions not in the limited api to be used and the resulting extension segfaults in Linux. Is that right? Thanks, Daniel ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Making stdlib modules optional for distributions (Was: Breaking up the stdlib (Was: release cadence))
Yes, not too long ago I installed "every" Python package on Ubuntu, and Python basically would not start. Perhaps some plugin system was trying to import everything and caused a segfault in GTK. The "short sys.path" model of everything installed is importable has its limits. On Thu, Jul 7, 2016 at 9:24 AM Barry Warsaw wrote: > On Jul 07, 2016, at 08:12 AM, Eric V. Smith wrote: > > >One thing to keep in mind if we do this is how it interacts with the -S > >command line option to not include site-packages in sys.path. I currently > use > >-S to basically mean "give my python as it was distributed, and don't > include > >anything that was subsequently added by adding other RPM's (or package > >manager of your choice)". I realize that's a rough description, and > possibly > >an abuse of -S. If using -S were to start excluding parts of the stdlib, > that > >would be a problem for me. > > It's an important consideration, and leads to another discussion that's > recurred over the years. Operating systems often want an "isolated" > Python, > similar to what's given by -I, which cannot be altered by subsequent > installs. It's one of the things that lead to the Debian ecosystem using > dist-packages for PyPI installed packages. Without isolation, it's just > too > easy for some random PyPI thing to break your system, and yes, that has > really > happened in the past. > > So if we go down the path of moving more of the stdlib to site-packages, we > also need to get serious about a system-specific isolated Python. > > Cheers, > -Barry > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Why does base64 return bytes?
It would be a codec. base64_text in the codecs module. Probably 1 line different than the existing codec. Very easy to use and maintain. Less surprising and less error prone for everyone who thinks base64 should convert between bytes to text. Sounds like an obvious win to me. On Wed, Jun 15, 2016 at 11:08 AM Isaac Morland wrote: > On Wed, 15 Jun 2016, Greg Ewing wrote: > > > Simon Cross wrote: > >> If we only support one, I would prefer it to be bytes since (bytes -> > >> bytes -> unicode) seems like less overhead and slightly conceptually > >> clearer than (bytes -> unicode -> bytes), > > > > Whereas bytes -> unicode, followed if needed by unicode -> bytes, > > seems conceptually clearer to me. IOW, base64 is conceptually a > > bytes-to-text transformation, and the usual way to represent > > text in Python 3 is unicode. > > And in CPython, do I understand correctly that the output text would be > represented using one byte per character? If so, would there be a way of > encoding that into UTF-8 that re-used the raw memory that backs the > Unicode object? And, therefore, avoids almost all the inefficiency of > going via Unicode? If so, this would be a win - proper use of Unicode to > represent a text string, combined with instantaneous conversion into a > bytes object for the purpose of writing to the OS. > > Isaac Morland CSCF Web Guru > DC 2619, x36650 WWW Software Specialist > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Why does base64 return bytes?
In that case could we just add a base64_text() method somewhere? Who would like to measure whether it would be a win? On Wed, Jun 15, 2016 at 8:34 AM Steven D'Aprano wrote: > On Tue, Jun 14, 2016 at 09:40:51PM -0700, Guido van Rossum wrote: > > I'm officially on vacation, but I was surprised that people now assume > > RFCs, which specify internet protocols, would have a bearing on > programming > > languages. (With perhaps an exception for RFCs that specifically specify > > how programming languages or their libraries should treat certain > specific > > issues -- but I found no evidence that this RFC is doing that.) > > Sorry to disturb your vacation! > > I hoped that there might have been a nice simple answer, like "the > main use-case for Base64 is the email module, which needs bytes, and > thus it was decided". Or even "because backwards compatibility". > > Thanks to everyone for their constructive comments, and expecially Mark > for digging up the original discussion on the Python-3000 list. I'm > satisfied that the choice made by Python is the right choice, and that > it meets the spirit (if, arguably, not the letter) of the RFC. > > > -- > Steve > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Why does base64 return bytes?
IMO this is more a philosophical problem than a programming problem. base64 has a dual-nature. It is both text and bytes. At least it should fit in a 1-byte-per-character efficient Python 3 unicode string also. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New hash algorithms: SHA3, SHAKE, BLAKE2, truncated SHA512
OpenSSL sucks. Python would only have to bundle a reference implementation of the new hash algorithm(s), and unlike TLS suites they tend to just work. BLAKE2 is important, since it removes the last objection to replacing MD5 - speed - that has made it hard for cryptography fans to convince MD5 users to upgrade. On Fri, May 27, 2016 at 7:13 AM M.-A. Lemburg wrote: > On 27.05.2016 13:03, Donald Stufft wrote: > > > >> On May 27, 2016, at 6:54 AM, M.-A. Lemburg wrote: > >> > >> IMO, relying on OpenSSL is a better strategy than providing > >> (and maintaining) our own compatibility versions. Until OpenSSL > >> has them, people can use Björn's package: > > > > Even now, hashlib doesn’t rely on OpenSSL if I recall, I mean it will > > use it if OpenSSL is available but otherwise it has internal > implementations > > too. > > I know, but still don't think that's a good idea. It makes > sense in case you don't want to carry around OpenSSL all the > time, but how often does that happen nowadays ? > > BTW: If I recall correctly, those hash implementations predate > the deeper support for OpenSSL we now have in Python. > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Experts (#1, May 27 2016) > >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ > >>> Python Database Interfaces ... http://products.egenix.com/ > >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ > > > ::: We implement business ideas - efficiently in both time and costs ::: > >eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg >Registered at Amtsgericht Duesseldorf: HRB 46611 >http://www.egenix.com/company/contact/ > http://www.malemburg.com/ > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adding a threadlocal to the Python interpreter
Thank you. I started my patch http://bugs.python.org/issue27134 to allow Python code to set a flag causing the evil str(b'bytes') to raise an exception. I wasn't sure exactly which module to put it in, so it's in _string. Please let me know the best place to put the feature and what I should polish to get it in. Thanks, Daniel Holth On Thu, May 19, 2016 at 6:34 AM Christian Heimes wrote: > On 2016-05-19 04:30, Nick Coghlan wrote: > > On 18 May 2016 at 23:20, Daniel Holth wrote: > >> I would like to take another stab at adding a threadlocal "str(bytes) > raises > >> an exception" to the Python interpreter, but I had a very hard time > >> understanding both how to add a threadlocal value to either the > interpreter > >> state or the threadlocal dict that is part of that state, and then how > to > >> access the same value from both Python and CPython code. The structs > were > >> there but it was just hard to understand. Can someone explain it to me? > > > > Christian covered the C aspects of the API, while the general purpose > > Python aspects live in the threading module. > > > > However, the Python level thread-local API doesn't provide direct > > access to the thread state dict. Instead, it provides access to > > subdicts stored under per-object keys in that dict, keyed as > > "thread.local.": > > In case you wonder about subdicts, they are required to provide multiple > thread local objects. Each thread local instance has its own key in each > thread state dict. The approach enables thread local to have independent > storage objects. > > Christian > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Adding a threadlocal to the Python interpreter
I would like to take another stab at adding a threadlocal "str(bytes) raises an exception" to the Python interpreter, but I had a very hard time understanding both how to add a threadlocal value to either the interpreter state or the threadlocal dict that is part of that state, and then how to access the same value from both Python and CPython code. The structs were there but it was just hard to understand. Can someone explain it to me? Thanks, Daniel Holth ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Type hints -- a mediocre programmer's reaction
On Thu, Apr 23, 2015 at 9:55 AM, Paul Sokolovsky wrote: > Hello, > > On Thu, 23 Apr 2015 09:15:44 -0400 > Daniel Holth wrote: > > [] > >> >> Also ask why no one used type specifier, they are possible since >> >> Python 3.0 ? >> >> Because it is the wrong way for Python. >> > >> > That's an example of how perceptions differ. In my list, everyone(*) >> > uses them - MyPy, MicroPython, etc. Even more should use them (any >> > JIT module, which are many), but sit in the bushes, waiting for a >> > kick, like PEP484 provides. >> >> It's OK that type hints are only to assist the programmer. > > Yes, it's OK to have a situation where type hints assist only a > programmer. It's not OK to think that type hints may be useful only for > programmer, instead of bunch more purposes, several of which > were already shown in the long previous discussion. > >> PyPy's FAQ >> has an explanation of why type hints are not for performance. >> http://pypy.readthedocs.org/en/latest/faq.html#would-type-annotations-help-pypy-s-performance > > You probably intended to write "why type hints are not for *PyPy's* > performance". There're many other language implementations and modules > for which it may be useful, please don't limit your imagination by a > single case. > > And speaking of PyPy, it really should think how to improve its > performance - not of generated programs, but of generation itself. If > compilation of a trivial program on a pumpy hardware takes 5 minutes > and gigabytes of RAM and diskspace, few people will use it for other > purposes beyond curiosity. There's something very un-Pythonic in > waiting 5 mins just to run 10-line script. Type hints can help here > too ;-) (by not wasting resources propagating types thru the same old > standard library for example). Naturally, PyPy is very controversial. Type annotations can help to compile Python into a subset of Python. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Type hints -- a mediocre programmer's reaction
On Thu, Apr 23, 2015 at 5:59 AM, Paul Sokolovsky wrote: > Hello, > > On Thu, 23 Apr 2015 10:43:52 +0200 > Wolfgang Langner wrote: > > [] > >> Also ask why no one used type specifier, they are possible since >> Python 3.0 ? >> Because it is the wrong way for Python. > > That's an example of how perceptions differ. In my list, everyone(*) > uses them - MyPy, MicroPython, etc. Even more should use them (any JIT > module, which are many), but sit in the bushes, waiting for a kick, like > PEP484 provides. It's OK that type hints are only to assist the programmer. PyPy's FAQ has an explanation of why type hints are not for performance. http://pypy.readthedocs.org/en/latest/faq.html#would-type-annotations-help-pypy-s-performance ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Type hints -- a mediocre programmer's reaction
On Tue, Apr 21, 2015 at 1:10 PM, Guido van Rossum wrote: > On Tue, Apr 21, 2015 at 9:17 AM, R. David Murray > wrote: >> >> Please be respectful rather than inflammatory. If you read what I >> wrote, I did not say that I was going to stop contributing, I >> specifically talked about that gut reaction being both emotional and >> illogical. That doesn't make the reaction any less real, and the fact >> that such reactions exist is a data point you should consider in >> conducting your PR campaign for this issue. (I don't mean that last as >> a negative: this issue *requires* an honest PR campaign.) > > > Well, my own reactions at this point in the flame war are also quite > emotional. :-( > > I have done my best in being honest in my PR campaign. But I feel like the > opposition (not you, but definitely some others -- have you seen Twitter?) > are spreading FUD based on an irrational conviction that this will destroy > Python. It will not. It may not prove the solution to all Python's problems > -- there's always 3.6. (Oh wait, Python 2.7 is perfect. I've heard that > before -- Paul Everitt famously said the same of Python 1.5.2. Aren't you > glad I didn't take him literally? :-P ) > > -- > --Guido van Rossum (python.org/~guido) I am looking forward to using type annotations. I am not very good at remembering the operations that are available on the objects passed around in my code, but I am very good at typing CTRL-space. To get that I am happy to modify my code with weird docstrings or any other notation. Good support for completion, aided by standard annotations, eliminates a huge amount of cross-referencing while coding. I'm also hopeful that static type checking, aided with annotations, will help with unicode porting. Duck typing does not work very well when you are trying to differentiate between bytes and str. Also, Python 1.5.2 was pretty good :-) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] easy_install ?
The other option might be to use http://bitbucket.org/dholth/setup-requires It uses pip to install requirements into an isolated directory before setup.py runs, with pip, doing exactly what you requested. On Feb 24, 2015 5:44 PM, "Nick Coghlan" wrote: > > On 25 Feb 2015 07:23, "Alexander Belopolsky" < > alexander.belopol...@gmail.com> wrote: > > > > > > On Tue, Feb 24, 2015 at 2:03 PM, Daniel Holth wrote: > > > > > > > Is there a recommended way to invoke pip from setup.py? When I > specify > > > > "tests_require=" and run "python setup.py test", the requirements get > > > > installed using setuptools' easy_install function. > > > > > > The solution is to not do that. A substitute is to specify your test > > > requirements in a [test] extra and install them with pip or to run > > > tests with tox. This gives control of the installer back to the user > > > instead of the setup.py author. > > > > > > Isn't this a chicken and egg problem? I currently have > > > > tests_require=['tox'], > > > > and this is exactly what tox recommends: > > > > > https://testrun.org/tox/latest/example/basic.html#integration-with-setuptools-distribute-test-commands > > > > > > Note that my CI box is a CentOS 6.5 with Python 2.6.6, setuptools 0.6. > This is still a very common server configuration. What is the recommended > way to bootstrap tox in such environment? > > If running in the system Python isn't absolutely essential, then the > Python 2.7 collection from softwarecollections.org is the preferred way > to get a newer Python 2 (including pip et al) on CentOS. You can also get > access to Python 3 that way. > > Failing that, pip & virtualenv are also available from the EPEL 6 repos. > > Both of those approaches rely on the system package manager to do the > bootstrapping of the Python specific tooling. > > If both softwarecollections.org and EPEL are considered unacceptable > dependencies, then you're going to have to do your own bootstrapping for > PyPI access on CentOS (which may include relying on easy_install to > bootstrap pip and/or virtualenv) > > Regards, > Nick. > > > > > > > ___ > > Python-Dev mailing list > > Python-Dev@python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > > > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] easy_install ?
That might mostly do what you want, since tox could install any additional test requirements based on its configuration. On Tue, Feb 24, 2015 at 4:21 PM, Alexander Belopolsky wrote: > > On Tue, Feb 24, 2015 at 2:03 PM, Daniel Holth wrote: >> >> > Is there a recommended way to invoke pip from setup.py? When I specify >> > "tests_require=" and run "python setup.py test", the requirements get >> > installed using setuptools' easy_install function. >> >> The solution is to not do that. A substitute is to specify your test >> requirements in a [test] extra and install them with pip or to run >> tests with tox. This gives control of the installer back to the user >> instead of the setup.py author. > > > Isn't this a chicken and egg problem? I currently have > > tests_require=['tox'], > > and this is exactly what tox recommends: > > https://testrun.org/tox/latest/example/basic.html#integration-with-setuptools-distribute-test-commands > > > Note that my CI box is a CentOS 6.5 with Python 2.6.6, setuptools 0.6. This > is still a very common server configuration. What is the recommended way to > bootstrap tox in such environment? > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] easy_install ?
On Tue, Feb 24, 2015 at 1:31 PM, Alexander Belopolsky wrote: > > On Tue, Feb 24, 2015 at 11:44 AM, Paul Moore wrote: >> >> And if pip won't work, it would be good to >> know why. > > > Is there a recommended way to invoke pip from setup.py? When I specify > "tests_require=" and run "python setup.py test", the requirements get > installed using setuptools' easy_install function. The solution is to not do that. A substitute is to specify your test requirements in a [test] extra and install them with pip or to run tests with tox. This gives control of the installer back to the user instead of the setup.py author. https://pythonhosted.org/setuptools/setuptools.html#declaring-extras-optional-features-with-their- own-dependencies ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On Sun, Feb 15, 2015 at 1:45 PM, Serhiy Storchaka wrote: > On 15.02.15 18:21, Thomas Wouters wrote: >> >> which requires that extension modules are stored uncompressed (simple) >> and page-aligned (harder, as the zipfile.ZipFile class doesn't directly >> support page-aligning anything > > > It is possible to add this feature to ZipFile. It can be useful because will > allow to mmap uncompressed files in ZIP file. One of the cool things about ZIP files is that they ignore any data that is not referenced by the index (which is at the end of the file). That is why we can put a #! at the beginning of the file, prefix .exe self-extractors, or put extra data between files in the archive. You could probably even overlap archive members if you were motivated. But it would be best to let the uncompressed extensions be standard ZIP members. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
Go ahead, make my pep. I will appreciate seeing it happen. On Feb 15, 2015 8:47 AM, "Nick Coghlan" wrote: > On 15 February 2015 at 23:21, Paul Moore wrote: > > On 15 February 2015 at 08:59, Nick Coghlan wrote: > >> The other option would to cut PEP 441 way back to *just* be about > >> standardising and registering the file associations, and recommending > >> the use of pip to obtain the build machinery with (whether pyzaa, > >> pyzzer or Twitter's more comprehensive pex). It would be a short PEP, > >> but potentially still worth it for the improved visibility of the > >> decision when folks are trying to figure out what "pyz" and "pyzw" > >> files are later. > > > > Ok, thinking about this a little more. > > > > Getting the extension support is the key thing on Windows - at the > > moment, people are faced with adding their own file associations or > > putting binary data in a .py file, neither of which is a nice choice. > > Tooling is important, though - sure, you can zip the data up and put a > > header on, but it's fiddly. > > > > Which brings us full circle. A simple module, executable as "python -m > > zipapp" (see below re name) which exports a single function, pack() > > that creates the archive. If we want to provide a script to wrap the > > module, like pyvenv.py does for venv, I've no objection to that - > > presumably it would go in Tools/Scripts? If people (like me) want to > > experiment with a more programmatic API for building pyz files, they > > can do so on PyPI, and if such a thing becomes sufficiently mature we > > can look then at proposing it for inclusion in the stdlib, as an > > extension to the zipapp module. > > > > Regarding naming, I'm happy to go with zipapp if it's your preference. > > Presumably the wrapper in Tools/Scripts would be pyzipapp.py? > > Or we just skip the wrapper and make "python -m zipapp" the > recommended invocation style. Adding a wrapper later is fairly easy, > but removing it would be difficult. > > > > > So the usage would be something like > > > > python -m zipapp [options] dir_to_zip > > > > Options: > > -p The interpreter to use on the shebang line > > (defaulting to /usr/bin/env python) > > -o archive_name The name of the output file (defaulting to the > > source directory name with a .pyz extension) > > If the argument has no extension, add '.pyz' > > -m module:function The entry point to call (written to __main__.py) > > Using this is an error if there is a > > __main__.py, and mandatory if there isn't > > > > If you want anything more complex, it's easy enough to write your own > > script based on zipfile, or use one of the modules on PyPI. > > > > Does this sound reasonable? If it's OK, I'll go ahead and prepare an > > update to the PEP and an implementation. (Steve, looks like I may be > > learning how to maintain the wix files after all - wish me luck :-)) > > If I hear no objections in the next couple of days, I'll assume > > everyone's OK with it and I'll prepare a PEP update and a patch. > > Sounds good to me. > > Daniel, do you mind if Paul becomes a co-author on PEP 441 and updates > it as described? It seems a bit tidier than allocating a new PEP > number and rejecting PEP 441, when the revised proposal is essentially > just a simplified and renamed version of your original idea. > > Cheers, > Nick. > > -- > Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Sysadmin tasks
On Wed, Oct 1, 2014 at 1:59 PM, Mark Shannon wrote: > Hi, > > http://speed.python.org/ > could do with some love. +1 ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Internal representation of strings and Micropython
On Thu, Jun 5, 2014 at 11:59 AM, Paul Moore wrote: > On 5 June 2014 14:15, Nick Coghlan wrote: >> As I've said before in other contexts, find me Windows, Mac OS X and >> JVM developers, or educators and scientists that are as concerned by >> the text model changes as folks that are primarily focused on Linux >> system (including network) programming, and I'll be more willing to >> concede the point. > > There is once again a strong selection bias in this discussion, by its > very nature. People who like the new model don't have anything to > complain about, and so are not heard. > > Just to support Nick's point, I for one find the Python 3 text model a > huge benefit, both in practical terms of making my programs more > robust, and educationally, as I have a far better understanding of > encodings and their issues than I ever did under Python 2. Whenever a > discussion like this occurs, I find it hard not to resent the people > arguing that the new model should be taken away from me and replaced > with a form of the old error-prone (for me) approach - as if it was in > my best interests. > > Internal details don't bother me - using UTF8 and having indexing be > potentially O(N) is of little relevance. But make me work with a > string type that *doesn't* abstract a string as a sequence of Unicode > code points and I'll get very upset. Once you get past whether str + bytes throws an exception which seems to be the divide most people focus on, you can discover new things like dance-encoded strings, bytes decoded using an incorrect encoding intended to be transcoded into the correct encoding later, surrogates that work perfectly until .encode(), str(bytes), APIs that disagree with you about whether the result should be str or bytes, APIs that return either string or bytes depending on their initializers and so on. Unicode can still be complicated in Python 3 independent of any judgement about whether it is worse, better, or different than Python 2. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Internal representation of strings and Micropython
On Wed, Jun 4, 2014 at 10:12 AM, Steven D'Aprano wrote: > On Wed, Jun 04, 2014 at 01:14:04PM +, Steve Dower wrote: >> I'm agree with Daniel. Directly indexing into text suggests an >> attempted optimization that is likely to be incorrect for a set of >> strings. > > I'm afraid I don't understand this argument. The language semantics says > that a string is an array of code points. Every index relates to a > single code point, no code point extends over two or more indexes. > There's a 1:1 relationship between code points and indexes. How is > direct indexing "likely to be incorrect"? "Useful" is probably a better word. When you get into the complicated languages and you want to know how wide something is, and you might have y with two dots on it as one code point or two and left-to-right and right-to-left indicators and who knows what else... then looking at individual code points only works sometimes. I get the slicing idea. I like the idea that encoding to utf-8 would be the fastest thing you can do with a string. You could consider doing regexps in that domain, and other implementation specific optimizations in exactly the same way that any Python implementation has them. None of this would make it harder to move a servo. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Internal representation of strings and Micropython
MicroPython is going to be significantly incompatible with Python anyway. But you should be able to run your mp code on regular Python. On Wed, Jun 4, 2014 at 9:39 AM, Serhiy Storchaka wrote: > 04.06.14 04:17, Steven D'Aprano написав(ла): > >> Would either of these trade-offs be acceptable while still claiming >> "Python 3.4 compatibility"? >> >> My own feeling is that O(1) string indexing operations are a quality of >> implementation issue, not a deal breaker to call it a Python. I can't >> see any requirement in the docs that str[n] must take O(1) time, but >> perhaps I have missed something. > > > I think than breaking O(1) expectation for indexing makes the implementation > significant incompatible with Python. Virtually all string operations in > Python operates with indices. > > O(1) indexing operations can be kept with minimal memory requirements if > implement Unicode internally as modified UTF-8 plus optional array of > offsets for every, say, 32th character (which even can be compressed to an > array of 16-bit or 32-bit integers). > > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Internal representation of strings and Micropython
If we're voting I think representing Unicode internally in micropython as utf-8 with O(N) indexing is a great idea, partly because I'm not sure indexing into strings is a good idea - lots of Unicode code points don't make sense by themselves; see also grapheme clusters. It would probably work great. On Wed, Jun 4, 2014 at 7:49 AM, Paul Sokolovsky wrote: > Hello, > > On Wed, 4 Jun 2014 20:53:46 +1000 > Chris Angelico wrote: > >> On Wed, Jun 4, 2014 at 8:38 PM, Paul Sokolovsky >> wrote: >> > And I'm saying that not to discourage Unicode addition to >> > MicroPython, but to hint that "force-force" approach implemented by >> > CPython3 and causing rage and split in the community is not >> > appreciated. >> >> FWIW, it's Python 3 (the language) and not CPython 3.x (the >> implementation) that specifies Unicode strings in this way. > > Yeah, but it's CPython what dictates how language evolves (some people > even think that it dictates how language should be implemented!), so all > good parts belong to Python3, and all bad parts - to CPython3, > right? ;-) > >> I don't >> know why it has to cause a split in the community; this is the one way >> to make sure *everyone's* strings work perfectly, rather than having >> ASCII strings work fine and others start tripping over problems in >> various APIs. > > It did cause split in the community, that's the fact, that's why > Python2 and Python3 are at the respective positions. Anyway, I'm not > interested in participating in that split, I did not yet uttered my > opinion on that publicly enough, so I seized a chance to drop some > witty remarks, but I don't want to start yet another Unicode flame. > > > > So, let's please be back to Unicode storage representation in > MicroPython. So, https://github.com/micropython/micropython/issues/657 > discussed technical aspects, in a recent mail on this list I expressed > my opinion why following CPython way is not productive (for development > satisfaction and evolution of Python community, to be explicit). > > Final argument I would have is that you certainly can implement Unicode > support the PEP393 way - it would be enormous help and would be gladly > accepted. The question, how useful it will be for MicroPython. It > certainly will be useful to report passing of testsuites. But will it > be *really* used? > > For microcontroller board, it might be too heavy (put simple, with it, > people will be able to do less (== heap running out sooner)), than > without it, so one may expect it to be disabled by default. Then POSIX > port is there surely not to let people replace "python" command > with "micropython" and run Django, but to let people develop and debug > their apps with more comfort than on embedded board. So, it should > behave close to MCU version, and would follow with MCU choice > re: Unicode. > > That's actually the reason why I keep up this discussion - not for the > sake of argument or to bash Python3's Unicode choices. With recent > MicroPython announcement, we surely looked for more people to > contribute to its development. But then we (or at least I can speak for > myself), would like to make sure that these contribution are actually > the most useful ones (for both MicroPython, and Python community in > general, which gets more choices, rather than just getting N% smaller > CPython rewrite). > > So, you're not sure how O(N) string indexing will work? But MicroPython > offers a great opportunity to try! And it's something new and exciting, > which surely will be useful (== will save people memory), not just > something old and boring ;-). > > >> >> ChrisA > > > -- > Best regards, > Paul mailto:pmis...@gmail.com > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Internal representation of strings and Micropython
Can of worms, opened. On Jun 4, 2014 7:20 AM, "Chris Angelico" wrote: > On Wed, Jun 4, 2014 at 9:12 PM, Paul Sokolovsky wrote: > > An alternative view is that the discussion on the tracker showed Python > > developers' mind-fixation on implementing something the way CPython does > > it. And I didn't yet go to that argument, but in the end, MicroPython > > does not try to rewrite CPython or compete with it. So, having few > > choices with pros and cons leading approximately to the tie among them, > > it's the least productive to make the same choice as CPython did. > > I'm not a CPython dev, nor a Python dev, and I don't think any of the > big names of CPython or Python has showed up on that tracker as yet. > But why is "be different from CPython" such a valuable choice? CPython > works. It's had many hours of dev time put into it. Problems have been > identified and avoided. Throwing that out means throwing away a > freely-given shoulder to stand on, in an Isaac Newton way. > > http://www.joelonsoftware.com/articles/fog69.html > > ChrisA > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Some notes about MicroPython from an observer
- micropython is designed to run on a machine with 192 kilobytes of RAM and perhaps a megabyte of FLASH. The controller can execute read-only code directly from FLASH. There is no dynamic linker in this environment. (It also has a UNIX port). - However it does include a full Python parser and REPL, so the board can be programmed without a separate computer as opposed to, say, having to upload bytecode compiled on a regular computer. - It's definitely going to be a subset of Python. For example, func.__name__ is not supported - to make it more micro? - They have a C API. It is much different than the CPython C API. - It mas more than one code emitter. A certain decorator causes a function to be compiled to ARM Thumb code instead of bytecode. - It even has an inline assembler than translates Python-syntax ARM assembly (to re-use the same parser) into machine code. Most information from https://www.kickstarter.com/projects/214379695/micro-python-python-for-microcontrollers/posts and http://micropython.org/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pep8 reasoning
Fortunately, Unicode provides us with the COMBINING LOW LINE character, combining the horizontal space-savings of camelCase with the underscore-indicates-separation properties of _. And it's a valid Python identifier. convertx̲mlt̲oj̲son On Thu, Apr 24, 2014 at 12:25 PM, Chris Angelico wrote: > On Thu, Apr 24, 2014 at 11:59 PM, Barry Warsaw wrote: >> I will say this: the original preference for underscore_names in PEP 8 was >> spurred by user studies some of our early non-native English speaking users >> conducted many years ago. We learned that it was more difficult for many of >> them to parse mixedCase names than underscore_names. I'm afraid I probably >> no >> longer have references to those studies, but the difference was pronounced, >> IIRC, and I think it's easy to see why. Underscores can be scanned by the >> eye >> as spaces, while I'd hypothesize that the brain has to do more work to read >> mixedCase names. > > Underscores also play much more nicely with initialisms. How would you > spell a function named "Add HTTP Header"? > > add_HTTP_header > add_http_header > > addHTTPHeader > addHttpHeader > > Four options to choose from. The first two clearly separate the > initialism from the other words; take your pick whether you want it > uppercased or not, because it's separated either way. In mixedCase, > the first one merges the H of Header in with HTTP; with something less > well known, that can be a nasty readability problem. The second one is > probably more readable, but looks weird. Or, here's another one: > converting one thing into another, where both are named by their > initials: > > convert_XML_to_JSON > convert_xml_to_json > > convertXMLToJSON > convertXmlToJson > > Same four options. Which is the more readable? > > ChrisA > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Software integrators vs end users (was Re: Language Summit notes)
On Fri, Apr 18, 2014 at 12:55 PM, Paul Moore wrote: > On 18 April 2014 16:58, Nick Coghlan wrote: >> As part of thrashing out the respective distribution ecosystem roles >> of pip and conda (still a work in progress), we're at least converging >> on the notion that there are actually now *two* main ways of consuming >> Python: as a "software integrator" (the way most of us have >> traditionally consumed it, and the way that dominates most project >> documentation outside the scientific Python community) and as an "end >> user" (the way Linux system administrators have long consumed it, and >> the way scientists, financial analysts and folks just learning Python >> are likely best off consuming it). >> >> Making these different personas explicit is a process that has barely >> begun (this email is mostly based on some conversations I had in >> person at PyCon and via email during the sprints), but here's the gist >> (based on listing examples): > > Interesting perspective. However, I'm not convinced it's complete. > Specifically, there's one group of people who I encounter relatively > often, who don't seem to me to fit well into either category you're > proposing. That is, (Windows in my experience, but maybe Linux as > well) users who want to write "simple scripts" and for whom batch > files or similar are not sufficient. Such people typically don't have > the sort of "single application area" focus that your "end user" > category seems to imply, but on the other hand they don't really fit > the "software integrators" role in the sense of necessarily being > comfortable setting up their own development environment. > > I worry that your classification risks ignoring that group (maybe > because Unix users are well served with other alternatives than Python > for this type of task, or because on Unix "use the system Python" is > the right answer). > > Your list of "end user" targeted distributions seem to be limited to: > > - Linux distribution vendors > - Vendors focused on the essentially scientific community (in the > broadest sense) > - Embedded Python > > That's very far from being complete coverage of all the people *I'd* > like to be able to recommend Python to. Specifically, unless we're not > interested in "generic" Windows users, I think we need to offer *some* > form of equivalent of the OS-packaged Python on Linux for Windows > users. That's what the python.org builds, plus pip, wheels and PyPI, > give for Windows users now. Hmm, if we assume that supporting that > remains a priority, is what you're really saying that we *don't* try > to extend that to work for Linux/OSX, as doing so competes with the OS > vendors - but rather we see python.org binaries and binary > infrastructure like wheels as being focused on the Windows user > experience? > > (I wish I'd been at PyCon, this would have been a very interesting > discussion to have face to face. Email isn't ideal for this...) One more group that I find interesting is application users. These people should not need to notice that Python is in use at all, in contrast to the "build a virtualenv / install / pass through fire and death / use" workflow that is sometimes promoted. They are well served by good tools that make single-file zip distributions or py2exe etc. from a collection of wheels or sdists. Application users are using Python because a best-in-class program is written in Python and not because a Python program integrates better with other Python libraries. Conda is interesting because it is a system package manager, except it installs everything into what they sometimes describe as "C-level virtualenvs" rather than /. They've come up with a design that works well with Python programs but isn't particularly Python-specific at all. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mercurial sluggishness (was: this is what happens if you freeze all the modules required for startup)
On Tue, Apr 15, 2014 at 1:29 PM, Antoine Pitrou wrote: > Le 15/04/2014 19:09, Daniel Holth a écrit : > >> In case you were wondering, I'm using Ubuntu's "2.7.5+" and "3.3.2+". >> >> My feeling has long been that the speed of getting at the "--help" >> option or any initial user feedback from Mercurial or git is a big >> driver in perceived speed as opposed to how long the entire operation >> might take. But for me any initial speed improvements from git are >> fully offset by the feeling of irritation afterwards. /troll >> >> For me Python's startup time (warm) takes about 1/4 of the hg startup >> time in the worst case. I expect to both notice and appreciate any >> speedups and encourage all optimizers to optimize. > > > Well, if we optimize 11% out of that 1/4, I don't expect you to notice the > speedup at all ;-) > > > Regards > > Antoine. No one expects the Spanish Inquisition. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mercurial sluggishness (was: this is what happens if you freeze all the modules required for startup)
In case you were wondering, I'm using Ubuntu's "2.7.5+" and "3.3.2+". My feeling has long been that the speed of getting at the "--help" option or any initial user feedback from Mercurial or git is a big driver in perceived speed as opposed to how long the entire operation might take. But for me any initial speed improvements from git are fully offset by the feeling of irritation afterwards. /troll For me Python's startup time (warm) takes about 1/4 of the hg startup time in the worst case. I expect to both notice and appreciate any speedups and encourage all optimizers to optimize. On Tue, Apr 15, 2014 at 12:47 PM, Antoine Pitrou wrote: > Le 15/04/2014 17:42, Daniel Holth a écrit : > >> I find Python's startup time to be very sluggish. I wish it was less >> than 50 milliseconds (0.05 seconds) including running hg, which is the >> common threshold for "instant". On my machine 'python -c ""' takes >> about 0.05 seconds but 'python3 -c ""' takes 0.125 seconds. > > > Please quote exact versions. Different versions of Python 3 will have > different startup characteristics. > > Regards > > Antoine. > > > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mercurial sluggishness (was: this is what happens if you freeze all the modules required for startup)
I find Python's startup time to be very sluggish. I wish it was less than 50 milliseconds (0.05 seconds) including running hg, which is the common threshold for "instant". On my machine 'python -c ""' takes about 0.05 seconds but 'python3 -c ""' takes 0.125 seconds. I will be very happy to see any speedup. On Tue, Apr 15, 2014 at 11:34 AM, Skip Montanaro wrote: > Eric wrote: > >> Perhaps not so much "a very real advantage" as "less of a >> distraction". It's still significantly slower than 2.7. :) > > I'm still confused. I peeked in /usr/bin/hg. The only "system" modules > it imports directly are os and system (maybe I'm using an ancient > version?). After that, it imports its own lazy import module. This > suggests to me that Mercurial's import slowness is mostly in its own > code (I counted 104 Python modules and six shared objects in its > mercurial package, which isn't going to be affected (much?) by > freezing the Python standard modules. > > I'm not trying to be difficult here. I thought that way back in the > day a huge amount of work was done to remove needless filesystem > activity, and zip packaging has been around for quite awhile. > > As an experiment, I ran hg pull as > > /usr/bin/python -v /usr/bin/hg pull > > in my cpython repo then looked at the -v output. Summarizing the > output I saw the following: > > 30 imports (0 dlopens) > > Python banner printed > > 86 imports (18 dlopens) > > adding changesets message > > 5 imports (2 dlopens) > > adding manifests message > > 1 import (0 dlopens) > > adding file changes message > > 7 imports (3 dlopens) > > added ... changesets message > > 4 imports (0 dlopens) > > run 'hg update' message > > (I missed a "searching" message in there somewhere.) > > That's a total of 133 imports, 23 of which were satisfied by loading > an extension module. The imports break down as follows: > > 51 imports (4 dlopens) from the mercurial package > 5 imports from the hgext package > 7 imports from the encodings package > > Everything else is imported from the top level, and at a glance > appears to be all Python stdlib stuff. The key period of execution > looks to be between the printing of the Python banner and the printing > of the adding changesets message. I see 46 imports (2 dlopens) from > the mercurial or hgext packages. That leaves 40 stdlib imports, of > which 16 were satisfied by dlopen. > > As a final check, I saved all the stdlib import statements from the -v > output (77 statements) to a file and timed its execution: > > % time /usr/bin/python ~/tmp/pyimp.py > > real0m0.162s > user0m0.034s > sys 0m0.010s > > It doesn't take much time to import every stdlib module that Mercurial > needs. I don't know how much slower all this import machinery is in > 3.x (and can't test at work, as we don't have a copy laying about). It > would probably have to be 3x or more slower for it to have much > visible impact on even simple hg commands. I find it hard to believe > that freezing the stdlib is going to lower the barrier enough for the > Mercurial folks, if, in fact, import slowness is their main reason for > not moving to 3.x. > > Skip > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] this is what happens if you freeze all the modules required for startup
IIRC it is no longer the case that ZIP imports (involving only one file for a lot of modules) are much faster than regular FS imports? On Tue, Apr 15, 2014 at 10:34 AM, Eric Snow wrote: > On Tue, Apr 15, 2014 at 1:45 AM, Chris Angelico wrote: >> Specific use-case that I can see: Mercurial. In a git vs hg shoot-out, >> git will usually win on performance, and hg is using Py2; migrating hg >> to Py3 will (if I understand the above figures correctly) widen that >> gap, so any improvement done to startup performance will give a very >> real advantage. > > Perhaps not so much "a very real advantage" as "less of a > distraction". It's still significantly slower than 2.7. :) > > -eric > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 465: A dedicated infix operator for matrix multiplication
On Tue, Apr 8, 2014 at 12:08 AM, Nick Coghlan wrote: > > On 7 Apr 2014 21:58, "MRAB" wrote: >> >> On 2014-04-08 02:45, Guido van Rossum wrote: >>> >>> So what? Aren't we allowed to have fun? :-) >>> >> Next thing you know, he'll be threatening people with The Comfy Chair! > > You may want to take a look at the packaging metadata 2.0 spec ;) > > I was also going to add a +1 for the actual topic of this thread, but > Guido's acceptance of the PEP rendered that point rather moot :) @ [1] ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3
On Thu, Mar 27, 2014 at 2:53 PM, Guido van Rossum wrote:
> So what's the use case for Python 2/3 compatible code? IMO the main use case
> for the PEP is simply to be able to construct bytes from a combination of a
> template and some input that may include further bytes and numbers. E.g. in
> asyncio when you write an HTTP client or server you have to construct bytes
> to write to the socket, and I'd be happy if I could write b'HTTP/1.0 %d
> %b\r\n' % (status, message) rather than having to use
> str(status).encode('ascii') and concatenation or join().
It seems to be notoriously difficult to understand or explain why
Unicode can still be very hard in Python 3 or in code that is in the
middle of being ported or has to run in both interpreters. As far as I
can tell part of it is when a symbol has type(str or bytes) depending
(declared as if we had a static type system with union types); some of
it is because incorrect mixing can happen without an exception, only
to be discovered later and far away in space and time from the error
(worse of all in a serialized file), and part of it is all of the not
easily checkable "types" a particular Unicode object has depending on
whether it contains surrogates or codes > n. Sometimes you might
simply disagree about whether an API should be returning bytes or
Unicode in mildly ambiguous cases like base64 encoding. Sometimes
Unicode is just intrinsically complicated.
For me this PEP holds the promise of being able to do work in the
bytes domain, with no accidental mixing ever, when I *really* want
bytes. For 2+3 I would get exceptions sometimes in Python 2 and
exceptions all the time in Python 3 for mistakes. I hope this is less
error prone in strict domains than for example u"string
processing".encode('latin1'). And I hope that there is very little
type(str or int) in HTTP for example or other "legitimate" bytes
domains but I don't know; I suspect that if you have a lot of problems
with bytes' %s then it's a clue you should use (u"%s" %
(argument)).encode() instead.
sprintf()'s version of %s just takes a char* and puts it in without
doing any type conversion of course. IANACL (I am not a C lawyer).
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3
I feel not including %s is nuts. Should I write .replace('%b', '%s')?
All I desperately need are APIs that provide enough unicode / str type
safety that I get an exception when mixing them accidentally... in my
own code, dynamic typing is usually a bug. As has been endlessly
discussed, %s for bytes is a bit like exposing sprintf()...
On Thu, Mar 27, 2014 at 2:41 PM, Guido van Rossum wrote:
> Much better, but I'm still not happy with including %s at all. Otherwise
> it's accept-worthy. (How's that for pressure. :-)
>
>
> On Thu, Mar 27, 2014 at 11:04 AM, Ethan Furman wrote:
>>
>> On 03/27/2014 10:55 AM, Ethan Furman wrote:
>>>
>>> On 03/27/2014 10:29 AM, Guido van Rossum wrote:
I also don't understand why we can't use %b instead of %s. AFAIK %b
currently doesn't mean anything and I somehow don't
expect we're likely to add it for other reasons (unless there's a
proposal I'm missing?). Just like we use %a instead of
%r to remind people that it's not quite the same (since it applies
.encode('ascii', 'backslashreplace')), shouldn't we
use anything *but* %s to remind people that that is also not the same
(not at all, in fact)? The PEP's argument against
%b ("rejected as not adding any value either in clarity or simplicity")
is hardly a good reason.
>>>
>>>
>>> The biggest reason to use %s is to support a common code base for 2/3
>>> endeavors. The biggest reason to not include %b
>>> is that it means binary number in format(); given that each type can
>>> invent it's own mini-language, this probably isn't
>>> a very strong argument against it.
>>>
>>> I have moderate feelings for keeping %s as a synonym for %b for backwards
>>> compatibility with Py2 code (when it's
>>> appropriate).
>>
>>
>> Changed to:
>>
>> --
>> ``%b`` will insert a series of bytes. These bytes are collected in one of
>> two
>> ways:
>>
>> - input type supports ``Py_buffer`` [4]_?
>>
>> use it to collect the necessary bytes
>>
>> - input type is something else?
>> use its ``__bytes__`` method [5]_ ; if there isn't one, raise a
>> ``TypeError``
>>
>> In particular, ``%b`` will not accept numbers nor ``str``. ``str`` is
>> rejected
>> as the string to bytes conversion requires an encoding, and we are
>> refusing to
>> guess; numbers are rejected because:
>>
>> - what makes a number is fuzzy (float? Decimal? Fraction? some user
>> type?)
>>
>> - allowing numbers would lead to ambiguity between numbers and textual
>> representations of numbers (3.14 vs '3.14')
>>
>> - given the nature of wire formats, explicit is definitely better than
>> implicit
>>
>> ``%s`` is included as a synonym for ``%b`` for the sole purpose of making
>> 2/3 code
>> bases easier to maintain. Python 3 only code should use ``%b``.
>>
>> Examples::
>>
>> >>> b'%b' % b'abc'
>> b'abc'
>>
>> >>> b'%b' % 'some string'.encode('utf8')
>> b'some string'
>>
>> >>> b'%b' % 3.14
>>
>> Traceback (most recent call last):
>> ...
>> TypeError: b'%b' does not accept 'float'
>>
>> >>> b'%b' % 'hello world!'
>>
>> Traceback (most recent call last):
>> ...
>> TypeError: b'%b' does not accept 'str'
>>
>> --
>>
>>
>> --
>> ~Ethan~
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 3
I love it.
On Tue, Mar 25, 2014 at 6:37 PM, Ethan Furman wrote:
> Okay, I included that last round of comments (from late February).
>
> Barring typos, this should be the final version.
>
> Final comments?
>
> -
> PEP: 461
> Title: Adding % formatting to bytes and bytearray
> Version: $Revision$
> Last-Modified: $Date$
> Author: Ethan Furman
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 2014-01-13
> Python-Version: 3.5
> Post-History: 2014-01-14, 2014-01-15, 2014-01-17, 2014-02-22, 2014-03-25
> Resolution:
>
>
> Abstract
>
>
> This PEP proposes adding % formatting operations similar to Python 2's
> ``str``
> type to ``bytes`` and ``bytearray`` [1]_ [2]_.
>
>
> Rationale
> =
>
> While interpolation is usually thought of as a string operation, there are
> cases where interpolation on ``bytes`` or ``bytearrays`` make sense, and the
> work needed to make up for this missing functionality detracts from the
> overall
> readability of the code.
>
>
> Motivation
> ==
>
> With Python 3 and the split between ``str`` and ``bytes``, one small but
> important area of programming became slightly more difficult, and much more
> painful -- wire format protocols [3]_.
>
> This area of programming is characterized by a mixture of binary data and
> ASCII compatible segments of text (aka ASCII-encoded text). Bringing back a
> restricted %-interpolation for ``bytes`` and ``bytearray`` will aid both in
> writing new wire format code, and in porting Python 2 wire format code.
>
> Common use-cases include ``dbf`` and ``pdf`` file formats, ``email``
> formats, and ``FTP`` and ``HTTP`` communications, among many others.
>
>
> Proposed semantics for ``bytes`` and ``bytearray`` formatting
> =
>
> %-interpolation
> ---
>
> All the numeric formatting codes (such as ``%x``, ``%o``, ``%e``, ``%f``,
> ``%g``, etc.) will be supported, and will work as they do for str, including
> the padding, justification and other related modifiers. The only difference
> will be that the results from these codes will be ASCII-encoded text, not
> unicode. In other words, for any numeric formatting code `%x`::
>
>b"%x" % val
>
> is equivalent to
>
>("%x" % val).encode("ascii")
>
> Examples::
>
>>>> b'%4x' % 10
>b' a'
>
>>>> b'%#4x' % 10
>' 0xa'
>
>>>> b'%04X' % 10
>'000A'
>
> ``%c`` will insert a single byte, either from an ``int`` in range(256), or
> from
> a ``bytes`` argument of length 1, not from a ``str``.
>
> Examples::
>
> >>> b'%c' % 48
> b'0'
>
> >>> b'%c' % b'a'
> b'a'
>
> ``%s`` is included for two reasons: 1) `b` is already a format code for
> ``format`` numerics (binary), and 2) it will make 2/3 code easier as Python
> 2.x
> code uses ``%s``; however, it is restricted in what it will accept::
>
> - input type supports ``Py_buffer`` [6]_?
> use it to collect the necessary bytes
>
> - input type is something else?
> use its ``__bytes__`` method [7]_ ; if there isn't one, raise a
> ``TypeError``
>
> In particular, ``%s`` will not accept numbers (use a numeric format code for
> that), nor ``str`` (encode it to ``bytes``).
>
> Examples::
>
> >>> b'%s' % b'abc'
> b'abc'
>
> >>> b'%s' % 'some string'.encode('utf8')
> b'some string'
>
> >>> b'%s' % 3.14
> Traceback (most recent call last):
> ...
> TypeError: b'%s' does not accept numbers, use a numeric code instead
>
> >>> b'%s' % 'hello world!'
> Traceback (most recent call last):
> ...
> TypeError: b'%s' does not accept 'str', it must be encoded to `bytes`
>
>
> ``%a`` will call ``ascii()`` on the interpolated value. This is intended
> as a debugging aid, rather than something that should be used in production.
> Non-ASCII values will be encoded to either ``\xnn`` or ``\u``
> representation. Use cases include developing a new protocol and writing
> landmarks into the stream; debugging data going into an existing protocol
> to see if the problem is the protocol itself or bad data; a fall-back for a
> serialization format; or even a rudimentary serialization format when
> defining ``__bytes__`` would not be appropriate [8].
>
> .. note::
>
> If a ``str`` is passed into ``%a``, it will be surrounded by quotes.
>
>
> Unsupported codes
> -
>
> ``%r`` (which calls ``__repr__`` and returns a ``str``) is not supported.
>
>
> Proposed variations
> ===
>
> It was suggested to let ``%s`` accept numbers, but since numbers have their
> own
> format codes this idea was discarded.
>
> It has been suggested to use ``%b`` for bytes as well as ``%s``. This was
> rejected as not adding any value either in clarity or simplicity.
>
> It has been proposed to automatically use ``.encode('ascii','strict')`` for
> ``str`` arguments to ``%s``.
>
> - Rejected as thi
Re: [Python-Dev] Making proxy types easier to write and maintain
pypy's transparent proxy feature: http://pypy.readthedocs.org/en/latest/objspace-proxies.html#transparent-proxies On Thu, Mar 20, 2014 at 1:56 PM, Larry Hastings wrote: > On 03/20/2014 12:49 AM, Nick Coghlan wrote: > > So long as Graham's willing to go along with it, he doesn't have to to > be the one to write the PEP. > > > PEP? Why does it need a PEP? I didn't think it'd even be a new top-level > library module, it'd be an addition / additions to functools. > > > /arry > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 460 reboot
On Tue, Jan 14, 2014 at 1:52 PM, Guido van Rossum wrote:
> On Tue, Jan 14, 2014 at 9:45 AM, Chris Barker wrote:
>> On Tue, Jan 14, 2014 at 9:29 AM, Yury Selivanov
>> wrote:
>>>
>>> - Try str(), and do ".encode(‘ascii’, ‘stcict’)” on the result.
>>
>>
>> please no -- that's the source of a lot of pain in py2 now.
>>
>> having a failure as a result of the value, rather than the type, of an
>> object just makes hard-to-test for bugs. Everything will be hunky dory for
>> development and testing, then in deployment some idiot ( ;-) ) will pass in
>> some non-ascii compatible string and you get failure. And the person who
>> gets the failure doesn't understand why, or they wouldn't have passed in
>> non-ascii values in the first place...
>>
>> Ease of porting is nice, but let's not make it easy to port bug-prone code.
>
> Right. This is a big red flag to me as well.
>
> I think there is some inherent conflict between the extensible design
> of str.format() and the practical needs of people who are actually
> going to use formatting operations (either % or .format()) with bytes.
>
> The *practical* needs are mostly limited to supporting basic number
> formatting (decimal, hex, padding) and interpolation of anything that
> supports the buffer interface. It would also be nice if you didn't
> have to specify the type at all in the format string, i.e. {} should
> do the right thing for numbers and (all sorts of) bytes.
>
> But the way to arrive at this behavior without duplicating a whole lot
> of code seems to be to call the existing text-based __format__ API and
> convert the result to bytes -- for numbers this should be safe (their
> formatting produces just ASCII digits and a selected few other ASCII
> characters) but leads to an undesirable outcome for other types -- not
> just str but also e.g. lists or dicts containing str instances, since
> those call __repr__ on the contained items, and repr() may produce
> non-ASCII bytes.
>
> This is why my earlier proposal used ascii(), which is a "nerfed"(*)
> version of repr(). This does the right thing for numbers as well as
> for many other types (e.g. None, bool) and does something unpleasant
> for text strings that is perhaps better than the alternative.
>
> Which reminds me. Quite a few people have spoken out in favor of loud
> failures rather than silent "wrong" output. But I think that in the
> specific context of formatting output, there is a long and IMO good
> tradition of producing (slightly) wrong output in favor of more strict
> behavior. Consider for example what to do when a number doesn't fit in
> the given width. Would you rather raise an exception, truncate the
> value, or mess up the formatting? All languages newer than Fortran
> that I've used have chosen the latter, and I still agree it's a good
> idea. Similar with infinities, NaN, or None. (Yes, it's embarrassing
> to have a website displaying 'null'. But isn't a 500 even *more*
> embarrassing?)
>
> This doesn't mean I'm insensitive to the argument in favor of loud and
> early failure. It's just that I can see both sides of the coin, and
> I'm still deciding which argument is more important.
>
> (*) Gamer slang for a weapon made less dangerous. :-)
I think loud and early failure is important for porting while you
might still be trying to pound out the previously blurry encode/decode
boundaries. In this code str and bytes will be wrong everywhere. Some
APIs might return either str or bytes based on the input. Let it fail,
find the boundaries, and fix it until it does something useful without
failing. And it kindof depends on the context whether it is worse to
display weird ephemeral output or write the same weird output to long
term storage.
I'm not sure what to think about content-dependent failures on
protocols that are supposed to be ASCII-only-without-repr-noise.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 460 reboot
On Mon, Jan 13, 2014 at 4:59 PM, Guido van Rossum wrote:
> On Mon, Jan 13, 2014 at 1:29 PM, Glenn Linderman
> wrote:
>> On 1/13/2014 12:09 PM, Guido van Rossum wrote:
>>
>> Yeah, the %s behavior with a string argument was a messy attempt at
>> compromise. I was hoping to mimick a common use of %s in Python 2,
>> where it can be used with either an 8-bit string or a number as
>> argument, acting like %b in the former case and like %d in the latter
>> case. Not having %s at all in Python 3 means that porting requires
>> more thinking (== more opportunity for mistakes when you're converting
>> in bulk) and there's no easy way to write code that works in Python 2
>> and 3.
>>
>> If we have %b for strictly interpolating bytes, I'm fine with adding
>> %a for calling ascii() on the argument and then interpolating the
>> result after ASCII-encoding it.
>>
>> If somehow (unlikely though it seems) we end up keeping %s (e.g.
>> strictly to ease porting), we could also keep %r as an alias for %a.
>>
>>
>> %s for strictly interpolating bytes eases porting. Sad name, but good for
>> compatibility. When the blowup happens, due to having a str type passed, the
>> porter adds the appropriate .encode(...) to the parameter, so it doesn't
>> blow up on Py 3, and it'll be OK for Py 2 as well, will it not?
>
> Lots of code uses %s with numbers too, and probably the occasional
> None or list (relying on the Python 2 near-guarantee that most
> objects' str() is their repr() and that repr() nearly guarantees to
> return only ASCII).
>
> E.g. I'm sure you can find live code doing something like
>
> headers.append('Content-Length: %s\r\n' % len(body))
But if the alternative is spurious quotes then the choice is clear...
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 460 reboot
On Mon, Jan 13, 2014 at 3:11 PM, Yury Selivanov wrote: > On January 13, 2014 at 3:08:43 PM, Daniel Holth (dho...@gmail.com) wrote: >> >> I see it now. b"foo%sbar" % b'baz' should also expand to b"foob'foo'bar" >> >> Instead of "%b" could "%j" mean "I should have used + or join() >> here >> but was too lazy" and work on str too? > > Isn’t this just error prone? Since it’s a new format character, many, > probably, would write %s by mistake. And, besides, there was no %j > in python2. Merely a flesh wound. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 460 reboot
I see it now. b"foo%sbar" % b'baz' should also expand to b"foob'foo'bar"
Instead of "%b" could "%j" mean "I should have used + or join() here
but was too lazy" and work on str too?
On Mon, Jan 13, 2014 at 2:51 PM, Terry Reedy wrote:
> On 1/13/2014 1:40 PM, Brett Cannon wrote:
>
>> > So bytes formatting really needn't (and shouldn't, IMO) mirror str
>> > formatting.
>
>
> This was my presumption in writing byteformat().
>
>
>> I think one of the things about Guido's proposal that bugs me is that it
>> breaks the mental model of the .format() method from str in terms of how
>> the mini-language works. For str.format() you have the conversion and
>> the format spec (e.g. "{!r}" and "{:d}", respectively). You apply the
>> conversion by calling the appropriate built-in, e.g. 'r' calls repr().
>> The format spec semantically gets passed with the object to format()
>> which calls the object's __format__() method: ``format(number, 'd')``.
>>
>> Now Guido's suggestion has two parts that affect the mini-language for
>> .format(). One is that for bytes.format() the default conversion is
>> bytes() instead of str(), which is fine (probably want to add 'b' as a
>> conversion value as well to be consistent). But the other bit is that
>> the format spec goes from semantically meaning ``format(thing,
>> format_spec)`` to ``format(thing, format_spec).encode('ascii',
>> 'strict')`` for at least numbers. That implicitness bugs me as I have
>> always thought of format specs just leading to a call to format(). I
>> think I can live with it, though, as long as it is **consistently**
>> applied across the board for bytes.format(); every use of a format spec
>> leads to calling ``format(thing, format_spec).encode('ascii',
>> 'strict')`` no matter what type 'thing' would be and it is clearly
>> documented that this is done to ease porting and handle the common case
>> then I can live with it.
>
>
> This is how my byteformat function works, except that when no format_spec is
> given, byte and bytearrary objects are left unchanged rather than being
> decoded and encoded again.
>
>
>> This even gives people in-place ASCII encoding for strings by always
>> using '{:s}' with text which they can do when they port their code to
>> run under both Python 2 and 3. So you should be able to do
>> ``b'Content-Type: {:s}'.format('image/jpeg')`` and have it give ASCII.
>> If you want more explicit encoding to latin-1 then you need to do it
>> explicitly and not rely on the mini-language to do tricks for you.
>>
>> IOW I want to treat the format mini-language as a language and thus not
>> have any special-casing or massive shifts in meaning between
>> str.format() and bytes.format() so my mental model doesn't have to
>> contort based on whether it's str or bytes. My preference is not have
>> any, but if Guido is going say PBP here then I want absolute consistency
>> across the board in how bytes.format() tweaks things.
>>
>> As for %s for the % operator calling ascii(), I think that will be a
>> porting nightmare of finding out why your bytes suddenly stopped being
>> formatted properly and then having to crawl through all of your code for
>> that one use of %s which is getting bytes in. By raising a TypeError you
>> will very easily detect where your screw-up occurred thanks to the
>> traceback; do so otherwise feels too much like implicit type conversion
>> and ask any JavaScript developer how that can be a bad thing.
>
>
> I personally would not add 'bytes % whatever'.
>
> --
> Terry Jan Reedy
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 460 reboot
On Mon, Jan 13, 2014 at 12:42 PM, R. David Murray wrote: > On Mon, 13 Jan 2014 12:41:18 +0100, Antoine Pitrou > wrote: >> On Sun, 12 Jan 2014 18:11:47 -0800 >> Guido van Rossum wrote: >> > On Sun, Jan 12, 2014 at 5:27 PM, Ethan Furman wrote: >> > > On 01/12/2014 04:47 PM, Guido van Rossum wrote: >> > >> %s seems the trickiest: I think with a bytes argument it should just >> > >> insert those bytes (and the padding modifiers should work too), and >> > >> for other types it should probably work like %a, so that it works as >> > >> expected for numeric values, and with a string argument it will return >> > >> the ascii()-variant of its repr(). Examples: >> > >> >> > >> b'%s' % 42 == b'42' >> > >> b'%s' % 'x' == b"'x'" (i.e. the three-byte string containing an 'x' >> > >> enclosed in single quotes) >> > > >> > > I'm not sure about the quotes. Would anyone ever actually want those in >> > > the >> > > byte stream? >> > >> > Perhaps not, but it's a hint that you should probably think about an >> > encoding. It's symmetric with how '%s' % b'x' returns "b'x'". Think of >> > it as payback time. :-) >> >> What is the use case for embedding a quoted ASCII-encoded representation >> in a byte stream? > > There is no use case in the sense you are asking, just like there is no > real use case for '%s' % b'x' producing "b'x'". But the real use case > is exactly the same: to let you know your code is screwed up without > actually blowing up with a encoding Exception. > > For the record, I like Guido's logic and proposal. I don't understand > Nick's objection, since I don't see the difference between the situation > here where a string gets interpolated into bytes as 'xxx' and the > corresponding situation where bytes gets interpolated into a string > as b'xxx'. Why struggle to keep bytes interpolation "pure" if string > interpolation isn't? > > Guido's proposal makes the language more symmetric, and thus more > consistent and less surprising. Exactly the hallmarks of Python's design > sense, IMO. (Big surprise, right? :) > > Of course, this point of view *is* based on the idea that when you are > doing interpolation using %/.format, you are in fact primarily concerned > with ASCII compatible byte streams. This is a Practicality sort of > argument. It is, after all, by far the most common use case when > doing interpolation[*]. > > If you wanted to do a purist version of this symmetry, you'd have bytes(x) > calling __bytes__ if it was defined and falling back to calling a > __brepr__ otherwise. > > But what would __brepr__ implement? The variety of format codes in > the struct module argues that there is no "one obvious" binary > repr for most types. (Those that have one would implement __bytes__). > And what would be the __brepr__ of an arbitrary 'object'? > > Faced with the impracticality of defining __brepr__ usefully in any "pure > bytes" form, it seems sensible to admit that the most useful __brepr__ > is the ascii() encoding of the __repr__. Which naturally produces 'xxx' > as the __brepr__ of a string. > > This does cause things to get a little un-pretty when you are operating > at the python prompt: > > >>> b'%s' % object > b'""' > > But then again that is most likely really not what you mean to do, so > it becomes a big red flag...just like b'xxx' is a small red flag when > you accidentally interpolate unencoded bytes into a string. > > --David > > PS: When I first read Guido's remark that the result of interpolating a > string should be 'xxx', I went Wah? I had to reason my way through to > it as above, but to him it was just the natural answer. Guido isn't > always right, but this kind of automatic language design consistency > is one reason he's the BDFL. > > [*] I still think that you mostly want to design your library so that > you are handling the text parts as text and the bytes parts as bytes, > and encoding/gluing them as appropriate at the IO boundary. But if Guido > says his real code would benefit by being able to interpolate ASCII into > bytes at certain points, I'll believe him. If you think corrupted data is easier or more pleasant to track down than encoding exceptions then I think you are strange. It makes porting really difficult while you are still trying to figure out where the bytes/str boundaries are. I am now deeply suspicious of all % formatting. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 460 reboot
On Sun, Jan 12, 2014 at 9:18 PM, Guido van Rossum wrote: > On Sun, Jan 12, 2014 at 6:07 PM, Daniel Holth wrote: >> Is there a formatting character that means "anything except a unicode >> string" to prevent accidentally interpolating a Unicode string into a >> bytes string without [a sane] encoding? > > No, and we shouldn't introduce one. An operation should either work > for no type, one type, a few specific types, or all types. Something > that works for all but one type will *appear* to work for all types to > a casually experimenting user and may pass extensive unittests, > leaving a bomb that can detonate when you least expect it. That pretty much describes how I feel about str(bytes). I would accept "only a bytes" or "only a string" as consolation formatting characters :-) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 460 reboot
On Sun, Jan 12, 2014 at 8:27 PM, Ethan Furman wrote: > On 01/12/2014 04:47 PM, Guido van Rossum wrote: >> >> >> %s seems the trickiest: I think with a bytes argument it should just >> insert those bytes (and the padding modifiers should work too), and >> for other types it should probably work like %a, so that it works as >> expected for numeric values, and with a string argument it will return >> the ascii()-variant of its repr(). Examples: >> >> b'%s' % 42 == b'42' >> b'%s' % 'x' == b"'x'" (i.e. the three-byte string containing an 'x' >> enclosed in single quotes) > > > I'm not sure about the quotes. Would anyone ever actually want those in the > byte stream? > > -- > ~Ethan~ Is there a formatting character that means "anything except a unicode string" to prevent accidentally interpolating a Unicode string into a bytes string without [a sane] encoding? ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 460: Add bytes % args and bytes.format(args) to Python 3.5
On Sat, Jan 11, 2014 at 4:28 PM, Terry Reedy wrote: > On 1/11/2014 1:44 PM, Stephen J. Turnbull wrote: > >> We already *have* a type in Python 3.3 that provides text >> manipulations on arrays of 8-bit objects: str (per PEP 393). >> >> > BTW: I don't know why so many people keep asking for use cases. >> > Isn't it obvious that text data without known (but ASCII compatible) >> > encoding or multiple different encodings in a single data chunk >> > is part of life ? >> >> Isn't it equally obvious that if you create or read all such ASCII- >> compatible chunks as (encoding='ascii', errors='surrogateescape') that >> you *don't need* string APIs for bytes? >> >> Why do these "text chunks" need to be bytes in the first place? >> That's why we ask for use cases. AFAICS, reading and writing ASCII- >> compatible text data as 'latin1' is just as fast as bytes I/O. So >> it's not I/O efficiency, and (since in this model we don't do any >> en/decoding on bytes/str), it's not redundant en/decoding of bytes to >> str and back. > > > The problem with some criticisms of using 'unicode in Python 3' is that > there really is no such thing. Unicode in 3.0 to 3.2 used the old internal > model inherited from 2.x. Unicode in 3.3+ uses a different internal model > that is a game changer with respect to certain issues of space and time > efficiency (and cross-platform correctness and portability). So at least > some the valid criticisms based on the old model are out of date and no > longer valid. -1 on adding more surrogateesapes by default. It's a pain to track down where the encoding errors came from. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python3 "complexity" (was RFC: PEP 460: Add bytes...)
So the customer you're looking for is the person who cares a lot about encodings, knows how to do Unicode correctly, and has noticed that certain valid cases not limited to imperialist simpletons (dealing with specific common things invented before 1996, dealing with mixed encodings, doing what Nick describes as "ASCII compatible binary protocols") are *more complicated to do correctly* in Python 3 because Python 3 undeniably has more complicated though probably better *Unicode* support. N.b. WSGI, email, url parsing etc. The same person loves Python, all the other Python 3 features, and probably you personally, but mostly does not write programs in the domains that Python 3 makes easier. They emphatically do not want the Python 2 model especially not implicit coercion. They only want additional tools for text or string processing in Python 3. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 460: Add bytes % args and bytes.format(args) to Python 3.5
On Wed, Jan 8, 2014 at 2:17 PM, Stefan Behnel wrote:
> Victor Stinner, 06.01.2014 14:24:
>> Abstract
>>
>> Add ``bytes % args`` operator and ``bytes.format(args)`` method to
>> Python 3.5.
>
> Here is a counterproposal. Let someone who needs this feature write a
> library that does byte string formatting. That properly handles it, a full
> featured tool set. Write it in Cython if you need raw speed, that will also
> help in making it run in both Python 2 and Python 3, or in providing easy
> integration with buffers like the array module, various byte containers,
> NumPy, etc.
> I'm confident that this will show that the current Py2 code that
> (legitimately) does byte string formatting can actually be improved,
> simplified or sped up, at least in some corners. I'm sure Py2 byte string
> formatting wasn't perfect for this use case either, it just happened to be
> there, so everyone used it and worked around its particular quirks for the
> particular use case at hand. (Think of "%s" % some_unicode_value, for
> example.)
>
> Instead of waiting for 3.5, a third party library allows users to get
> started porting their code earlier, and to make it work unchanged with
> Python versions before 3.5.
Maybe we can enumerate some of the stated drawbacks of b''.format()
Convenient string processing tools for bytes will make people ignore
Unicode or fail to notice it or do it wrong? (As opposed to the
alternative causing them to learn how to process and produce Unicode
correctly?)
Similar APIs on bytes and str will prevent implicit "assert
isinstance(x, str)" checks?
More-prevalent bytes will propagate across the program causing bugs?
A-la open(b'filename').name vs open('filename').name ?
It will take a long time.
Hopeful benefits may include easier porting and greater Py3 adoption,
less encoding dances and/or decoding non-Unicode into Unicode just to
make things work, hopefully fewer surrogate-encoded bytes and
therefore fewer encoding-bugs-distant-from-source-of-invalid-text, ...
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 460: Add bytes % args and bytes.format(args) to Python 3.5
On Tue, Jan 7, 2014 at 10:36 AM, Stephen J. Turnbull wrote: > Daniel Holth writes: > > > Isn't it true that if you have bytes > 127 or surrogate escapes then > > encoding to latin1 is no longer as fast as memcpy? > > Be careful. As phrased, the question makes no sense. You don't "have > bytes" when you are encoding, you have characters. > > If you mean "what happens when my str contains characters in the range > 128-255?", the answer is encoding a str in 8-bit representation to > latin1 is effectively memcpy. If you read in latin1, it's memcpy all > the way (unless you combine it with a non-latin1 string, in which case > you're in the cases below). > > If you mean "what happens when my str contains characters in the range >> 255", you have to truncate 16-bit units to 8 bit units; no memcpy. > > Surrogates require >= 16 bits; no memcpy. That is neat. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 460: Add bytes % args and bytes.format(args) to Python 3.5
+1 I have always been delighted that it is possible to manipulate binary data in Python using string operations. It's not just immoral non-Unicode text processing. A poor man's ASN.1 generator is an example of a very non-text thing that might be convenient to write with a few %s fill-in-the-blanks. Isn't it true that if you have bytes > 127 or surrogate escapes then encoding to latin1 is no longer as fast as memcpy? On Tue, Jan 7, 2014 at 8:22 AM, Serhiy Storchaka wrote: > Most popular formatting codes in Mercurial sources: > >2519 %s > 493 %d > 102 %r > 48 %Y > 47 %M > 41 %H > 39 %S > 38 %m > 33 %i > 29 %b > 23 %ld > 19 %ln > 12 %.3f > 10 %a > 10 %.1f > 9 %(val)r > 9 %p > 9 %.2f > 8 %I > 6 %n > 5 %(val)s > 5 %.0f > 5 %02x > 4 %f > 4 %c > 4 %12s > 3 %(user)s > 3 %(id)s > 3 %h > 3 %(bzdir)s > 3 %0.2f > 3 %02d > > > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backward-incompatible change to random.randrange in 2.7.6
On Dec 18, 2013 11:54 AM, "Tim Peters" wrote: > [Daniel Holth] > > But who could forget njzrs' wasp UAV software line 107, using > > int=float? > https://github.com/nzjrs/wasp/blob/master/sw/groundstation/wasp/__init__.py#L107 > > I could forget it ;-) The remarkable thing about the two instances of: > > random.randrange(0.0,1.0, int=float) > > in that file is that they're obscure and inefficient ways to spell: > > random.random() > You can keep your "premature optimizations" thank you :-) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backward-incompatible change to random.randrange in 2.7.6
But who could forget njzrs' wasp UAV software line 107, using int=float? https://github.com/nzjrs/wasp/blob/master/sw/groundstation/wasp/__init__.py#L107 On Wed, Dec 18, 2013 at 10:49 AM, Guido van Rossum wrote: > BTW, I bet a lavish dinner at PyCon that it is *only* Zope/ZODB that > does this. In the time we added this bogus dependency on undocumented > parameters, the PythonLabs team was at Zope and we didn't always get > our boundaries straight. > > On Wed, Dec 18, 2013 at 2:51 AM, Nick Coghlan wrote: >> On 18 December 2013 20:17, Chris Angelico wrote: >>> On Wed, Dec 18, 2013 at 8:43 PM, Serhiy Storchaka >>> wrote: 18.12.13 04:40, Benjamin Peterson написав(ла): > Mostly yes, but at least you could tell people to upgrade straight to > 2.7.7 and skip 2.7.6. It'll make the people to postpone the upgrade to 2.7.6 (which fixes many security bugs) until 2.7.7 release, instead of correcting their morally-broken programs. >>> >>> If this is considered enough breakage to be a problem, would it be >>> possible to issue a 2.7.6.1 or 2.7.6+fixed release that's identical to >>> 2.7.6 but with this change reverted? It'd be a minor mess, but then >>> people would still get those security fixes, and it means not breaking >>> stuff in a point release. >> >> If we revert it, it means we do 2.7.7 ASAP so that *everyone* can just >> skip straight to 2.7.7. >> >> That kind of user visible change shouldn't have been made in a point >> release, regardless of what the docs said. It just isn't worth the >> risk of breaking the code of people that are relying on what's >> possible rather than what the docs say. >> >> Cheers, >> Nick. >> >> >> -- >> Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia >> ___ >> Python-Dev mailing list >> Python-Dev@python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > -- > --Guido van Rossum (python.org/~guido) > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 0404 and VS 2010
On Thu, Nov 21, 2013 at 4:12 PM, Paul Moore wrote: > On 21 November 2013 21:02, Greg Ewing wrote: >> Is that much different from package authors having to >> release binaries for different versions of Python, >> if they want to support older versions? >> >> Having multiple binaries for the same x.y version >> is different from what's been done before, but it >> seems to me an unavoidable consequence of supporting >> one x.y version for longer than usual. > > None of the currently available binary distribution formats > distinguish Windows binaries by anything other than minor version. For > wheels (and I think eggs), this is a showstopper as the name is > essential metadata (compatibility tags) for the other formats (wininst > and msi) the name is merely informational - packagers could rename, > but (a) they will forget, and (b) the users won't know if they have or > not. > > Before we can cleanly support multiple ABIs for a single minor version > on Windows, we need to have a resolution of this dilemma (which may be > nothing more than "only binaries for the python.org builds are allowed > on PyPI"...) > > Paul As for wheel it actually does support an ABI tag that is separate from the Python version and the architecture. It's the second one pyversion-abi-architecture as in py27-none-any or py27-cp27-linux_i386 (spelling?). The build tool and installer would have to be modified to be aware of any newly defined ABI tags. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] flaky tests caused by repr() sort order
+1 on unsorted repr(). It makes it obvious that the collection is not sorted. On Thu, Nov 21, 2013 at 1:10 PM, Christian Heimes wrote: > Am 21.11.2013 18:57, schrieb Tim Peters: >> Best to change the failing tests. For example, _they_ can sort the >> dict keys if they rely on a fixed order. Sorting in general is a >> dubious idea because it can be a major expense with no real benefit >> for most uses. > > I don't consider repr() as a performance critical function. It's mostly > used for debugging. > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] non-US zip archives support in zipfile.py
On Tue, Oct 15, 2013 at 8:02 AM, Steven D'Aprano wrote: > On Tue, Oct 15, 2013 at 10:20:03AM +0200, Victor Stinner wrote: >> Both issues are related and should be fixed in the *same* commit. >> Python should provide an option to encode explicitly filenames to >> UTF-8. Or maybe better: encode to UTF-8 by default, with an option to >> encode to cp437, and another to encode to an arbitrary encoding. > > Why single out cp437 for special treatment? Surely it's enough to use > UTF-8 by default, or specify an arbitrary encoding. It is part of the ZIP specification. CP437 or UTF-8 are the two official choices, but other encodings happen on Russian, Japanese systems. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] non-US zip archives support in zipfile.py
FYI zipfile does do UTF-8 (http://hg.python.org/cpython/file/d7ebe03fa752/Lib/zipfile.py#l368). Only the non-Unicode encodings might need some help. I like that the patch is only concerned with decoding. Is it necessary to support writing non-UTF8 encodings? On Tue, Oct 15, 2013 at 10:20 AM, Victor Stinner wrote: > Both issues are related and should be fixed in the *same* commit. > Python should provide an option to encode explicitly filenames to > UTF-8. Or maybe better: encode to UTF-8 by default, with an option to > encode to cp437, and another to encode to an arbitrary encoding. > > Victor > > 2013/10/15 Alexander Belopolsky : >> On Mon, Oct 14, 2013 at 6:13 PM, Victor Stinner >> wrote: >>> I opened an issue proposing exactly the same change, but I didn't provide a >>> patch. >> >> I found this: >> >> http://bugs.python.org/issue10614 >> >> but it has (seemingly incorrect) patch. >> >> Also related: >> >> http://bugs.python.org/issue10972 >> >> Victor - is any of these what you had in mind? > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] On the dangers of giving developers the best resources
Sounds like you are suggesting we get a raspberry pi. All sorts of dumb waste shows up when you run code on those. El oct 8, 2013 4:46 p.m., "Guido van Rossum" escribió: > Let's agree to disagree then. I see your methodology used all around me > with often problematic results. Maybe devs should have two machines -- one > monster that is *only* usable to develop fast, one small that where they > run their own apps (email, web browser etc.). > > > On Tue, Oct 8, 2013 at 1:30 PM, Tim Delaney > wrote: > >> On 9 October 2013 03:35, Guido van Rossum wrote: >> >>> On Tue, Oct 8, 2013 at 8:33 AM, R. David Murray >>> wrote: >>> PS: I have always thought it sad that the ready availability of memory, CPU speed, and disk space tends to result in lazy programs. I understand there is an effort/value tradeoff, and I make those tradeoffs myself all the time...but it still makes me sad. Then, again, in my early programming days I spent a fair amount of time writing and using Forth, and that probably colors my worldview. :) >>> >>> I never used or cared for Forth, but I have the same worldview. I >>> remember getting it from David Rosenthal, an early Sun reviewer. He stated >>> that engineers should be given the smallest desktop computer available, not >>> the largest, so they would feel their users' pain and optimize >>> appropriately. Sadly software vendors who are also hardware vendors have >>> incentives going in the opposite direction -- they want users to feel the >>> pain so they'll buy a new device. >>> >> >> I look at it a different way. Developers should be given powerful >> machines to speed up the development cycle (especially important when >> prototyping and in the code/run unit test cycle), but everything should be >> tested on the smallest machine available. >> >> It's also a good idea for each developer to have a resource-constrained >> machine for developer testing/profiling/etc. Virtual machines work quite >> well for this - you can modify the resource constraints (CPU, memory, etc) >> to simulate different scenarios. >> >> I find that this tends to better promote the methodology of "make it >> right, then make it fast (small, etc)", which I subscribe to. Optimising >> too early leads to all your code being complicated, rather than just the >> bits that need it. >> >> Tim Delaney >> > > > > -- > --Guido van Rossum (python.org/~guido) > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com > > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 453 (pip bootstrapping) ready for pronouncement?
On Fri, Sep 27, 2013 at 12:15 PM, R. David Murray wrote: > On Fri, 27 Sep 2013 15:26:41 +0100, Paul Moore wrote: >> On 27 September 2013 15:08, Stephen J. Turnbull wrote: >> >> New users on Windows and Mac OS X. I've heard many more complaints >> > > from folks running tutorials about the pip bootstrapping process than >> > > I ever have from the community at large about the GIL :P >> > >> > I bet those users are *not* running third-party distros, but rather >> > are sitting in front of pretty close to plain vanilla factory installs >> > of the OS, no? And "new users" on Mac OS X already have "old installs" >> > of Python, no? >> >> Windows users who don't use a third-party distro like Enthought, >> generally download the python.org installer. At the moment, that >> doesn't give them a "pip" command. So if they want to install any >> third party package, they have to start by installing pip. The >> instructions for that are reasonably clear, but non-trivial, largely >> because tools like curl are not commonly available on Windows, and by >> default running a Python script may not do what you expect. >> >> Rather than try to fix these problems (which are *hard*) the intent is >> to have the pip command installed by the python.org installer. > [...] >> I can't speak for Linux distros or OSX users, but for Windows I do >> believe that this is a significant improvement, and worth the (IMO, >> negligible) risk involved in adding this to a maintenance release. > > I'm not an OS X user, and probably most people on this list use macports > or something similar, which essentially puts them in the same boat as > the linux users...and there's a section in the PEP about that (that's > where the message about installing pip if you run pip and the distro > didn't include it with python is supposed to come from). > > For OS X users *not* using something like macports, I'm pretty sure they > are going to be in a similar boat to the Windows users, with just a touch > of added confusion coming from the fact that an older version of Python > is already installed. But the instructions they will find on the web > for installing package X (once this change hits the field) will be to > install the newest version of 2.7 (or 3) using the python.org installer, > and then they will have the pip command and can go from there. > > --David OS X and Linux change Python in ways that can be confusing to new and experienced users, like not installing the stdlib source code by default, or not installing the profiler, or simply by being out of date. On these platforms attempting to use the system Python for development can be a costly mistake instead of a convenience. Users get a more consistent experience by starting with the installers from python.org. Hopefully the tutorials will reflect the consistency added by this PEP. This PEP only gets pip, an operation which is a recurring inconvenience for me even though I use Linux. It does not solve any of the problems you may have after pip has been installed. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tweaking PEP 8 guidelines for use of leading underscores
https://pypi.python.org/pypi/apipkg provides a much more effective way to denote API than an _ On Wed, Jul 17, 2013 at 12:20 PM, R. David Murray wrote: > On Tue, 16 Jul 2013 17:46:34 -0400, Terry Reedy wrote: >> On 7/16/2013 9:39 AM, R. David Murray wrote: >> > On Tue, 16 Jul 2013 23:19:21 +1000, Steven D'Aprano >> > wrote: >> >> >> For example, pkgutil includes classes with single-underscore methods, >> >> which I take as private. It also has a function simplegeneric, which is >> >> undocumented and not listed in __all__. In in the absence of even a >> >> comment saying "Don't use this", I take it as an oversight, not policy >> >> that simplegeneric is private. >> > >> > I think you'd be wrong about that, though. simplegeneric should really be >> > treated as private. I'm speaking here not about the general principle of >> > the thing, but about my understanding of simplegeneric's specific history. >> >> I think Steven (valid) point is "Why not, then, say it is internal >> either in docs or name?"-- which in this case would be in the docs. > > I don't think that's what he was saying; but yes, we should do that :) > > --David > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] stat module in C -- what to do with stat.py?
cffi makes this kind of constant-grabbing very easy. However this is a tiny module so no problem with having a C version. On Thu, Jun 20, 2013 at 11:04 AM, Thomas Wouters wrote: > > > > On Thu, Jun 20, 2013 at 8:01 AM, Brett Cannon wrote: >> >> >> >> >> On Thu, Jun 20, 2013 at 10:14 AM, Eric V. Smith >> wrote: >>> >>> On 6/20/2013 9:40 AM, Nick Coghlan wrote: >>> > On 20 June 2013 23:29, Christian Heimes wrote: >>> >> Am 20.06.2013 15:21, schrieb Florent: >>> >>> we already have "_pyio.py", we could have "_pystat.py". >>> >> >>> >> Works for me. >>> > >>> > I suggest following the guidelines in PEP 399 for cross implementation >>> > compatibility of the standard library: >>> > http://www.python.org/dev/peps/pep-0399/#details >>> > >>> > 1. Keep stat.py >>> > 2. Make the C version "_stat" >>> > 3. Add the following to the end of stat.py: >>> > >>> > try: >>> > from _stat import * >>> > except ImportError: >>> > pass >>> > >>> > Run the tests with and without the C module in the test suite (again, >>> > as per PEP 399). >>> >>> Agreed with the above. >>> >>> But isn't the real problem with this module in Python the fact that the >>> constants might be wrong? I'm not sure what, if anything, we can do >>> about that. >> >> >> There isn't anything we can do beyond at least trying to provide >> reasonable defaults when something better isn't available (which is what the >> stats module has done all this time). It might make testing difficult for >> the Python code when the C code has the right values, but I don't think it's >> necessarily worth tossing out the Python code as backup. > > > If the .py file is going to be wrong or incomplete, why would we want to > keep it -- or use it as fallback -- at all? If we're dead set on having a > .py file instead of requiring it to be part of the interpreter (whichever > that is, however it was built), it should be generated as part of the build > process. Personally, I don't see the value in it; other implementations will > need to do *something* special to use it anyway. > > -- > Thomas Wouters > > Hi! I'm an email virus! Think twice before sending your email to help me > spread! > > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bilingual scripts
On Tue, May 28, 2013 at 2:04 PM, Barry Warsaw wrote: > On May 28, 2013, at 01:57 PM, Daniel Holth wrote: > >>Wheel has no mechanism for renaming scripts (or any file) based on the >>Python version used to install. Instead you would have to build >>python-version-specific packages for each desired script name. > > Note that I'm not trying to borrow any implementation details from wheels, > just the file naming conventions (compatibility tags) described in PEP 425. > It would still be up to upstream package or distro tools to fiddle the > installed file names. I'm just saying that I prefer a setup.py without too many Python-version-specific differences, since it would look pretty silly to install a wheel of nose generated on Python 3.2 on Python 3.3 and have the wrong version suffix on the scripts. I like the plainly named scripts without version suffixes. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bilingual scripts
On Tue, May 28, 2013 at 1:30 PM, Barry Warsaw wrote: > On May 25, 2013, at 03:12 AM, Chris McDonough wrote: > >>You probably already know this, but I'll mention it anyway. This >>probably matters a lot for nose and pyflakes, but I'd say that for tox >>it should not, it basically just scripts execution of shell commands. >>I'd think maybe in cases like tox (and others that are compatible with >>both Python 2 and 3) the hashbang should just be set to >>"#!/usr/bin/python" unconditionally. > > Unfortunately, not entirely so: > > https://bitbucket.org/hpk42/tox/issue/96/cant-have-a-python-3-setuppy > >>Maybe we could also think about modifying pyflakes so that it can >>validate both 2 and 3 code (choosing one or the other based on a header >>line in the validated files and defaulting to the version of Python >>being run). This is kind of the right thing anyway. > > Agreed. Auto-detection may need to be accompanied by a command line option to > override in some cases. But I agree, that in general, it would be very nice > if the script itself were actually bilingual. (But then, see my previous > comment about cross-interpreter dependencies.) > >>Nose is a bit of a special case. I personally never run nosetests >>directly, I always use setup.py nosetests, which makes it not matter. > > Which is morally equivalent to `$python -m nose`. > >>In general, I'd like to think that scripts that get installed to global >>bindirs will execute utilities that are useful independent of the >>version of Python being used to execute them. > > Agreed. I'm trying to tease out some conventions we can recommend for when > this can't be the case for whatever reason. > > -Barry Wheel has no mechanism for renaming scripts (or any file) based on the Python version used to install. Instead you would have to build python-version-specific packages for each desired script name. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 4XX: pyzaa "Improving Python ZIP Application Support"
Everyone seems to like the first half of this simple PEP adding the extensions. The 3-letter extension for windowed apps can be "pzw" while the "pyz" extension for console apps stays the same. The second half, the tool https://bitbucket.org/dholth/pyzaa/src/tip/pyzaa.py?at=default is less mature, but there's not a whole lot to do in a simple tool that may serve more as an example: you can open any file with ZipFile in append mode, even one that is not a zip file and just contains the #!python shebang line. Thanks, Daniel ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 4XX: pyzaa "Improving Python ZIP Application Support"
As the PEP author I declare we can have 3-letter extensions. It is not a big deal. Daniel Holth On Mon, May 6, 2013 at 4:30 PM, Richard Oudkerk wrote: > So the bug would just cause .pyzw files to be opened with py instead of pyw? > Won't this be harmless? > > I think the worst that would happen would be that you get a redundant > console window if you are not already running powershell inside a console. > > -- > Richard > > > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Why can't I encode/decode base64 without importing a module?
On Thu, Apr 25, 2013 at 10:07 AM, Barry Warsaw wrote:
> On Apr 25, 2013, at 03:34 PM, Lennart Regebro wrote:
>
>>In the case of JSON objects, they are intended for data exchange, and
>>hence in the end need to be byte strings.
>
> Except that they're not.
>
> http://bugs.python.org/issue10976
>
> -Barry
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com
What am I doing wrong in this JSON crypto signature verification
snippet that features many conversions between binary and text?
recipients = jwsjs["recipients"]
encoded_payload = binary(jwsjs["payload"])
headers = []
for recipient in recipients:
h = binary(recipient["header"])
s = binary(recipient["signature"])
header = json.loads(native(urlsafe_b64decode(h)))
vk = urlsafe_b64decode(binary(header["jwk"]["vk"]))
secured_input = b".".join((h, encoded_payload))
sig = urlsafe_b64decode(s)
sig_msg = sig+secured_input
verified_input = native(ed25519ll.crypto_sign_open(sig_msg, vk))
verified_header, verified_payload = verified_input.split('.')
verified_header = binary(verified_header)
decoded_header = native(urlsafe_b64decode(verified_header))
headers.append(json.loads(decoded_header))
verified_payload = binary(verified_payload)
# only return header, payload that have passed through the crypto library.
payload = json.loads(native(urlsafe_b64decode(verified_payload)))
return headers, payload
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 4XX: pyzaa "Improving Python ZIP Application Support"
On Wed, Apr 3, 2013 at 1:26 AM, Stefan Behnel wrote: > Brett Cannon, 02.04.2013 19:28: >> On Tue, Apr 2, 2013 at 1:20 PM, Steve Dower wrote: >> >>>> python -m pyzaa pack [-o path/name] [-m module.submodule:callable] [-c] >>> [-w] [-p interpreter] directory: >>>> >>>>ZIP the contents of directory as directory.pyz or [-w] >>> directory.pyzw. Adds the executable flag to the archive. >>>> >>>> ... >>>> >>>>-p interpreter include #!interpreter as the first line of the archive >>> >>> What happens when -p is omitted? I'd hope it would add the interpreter >>> used to create the zip (or at least the major version), but that may not be >>> ideal for some reason that I haven't thought of yet. >> >> Question is whether ``/usr/bin/python3.3`` is better or ``/usr/bin/env >> python3.3``. I vote for the latter since it gets you the right thing >> without having to care about whether the interpreter moved or is being >> hidden by a user-installed interpreter. > > It can't work properly from within a virtualenv when you write > "/usr/bin/python", so using "/usr/bin/env" instead is actually required. > > Stefan Pushed as Draft PEP 441, tooling prototyped (with less than awesome CLI) at https://bitbucket.org/dholth/pyzaa or https://crate.io/packages/pyzaa Thanks, Daniel Holth ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 4XX: pyzaa "Improving Python ZIP Application Support"
https://docs.google.com/document/d/1MKXgPzhWD5wIUpoSQX7dxmqgTZVO6l9iZZis8dnri78/edit?usp=sharing PEP: 4XX Title: Improving Python ZIP Application Support Author: Daniel Holth Status: Draft Type: Standards Track Python-Version: 3.4 Created: 30 March 2013 Post-History: 30 March 2013, 1 April 2013 Improving Python ZIP Application Support Python has had the ability to execute directories or ZIP-format archives as scripts since version 2.6. When invoked with a zip file or directory as its first argument the interpreter adds that directory to sys.path and executes the __main__ module. These archives provide a great way to publish software that needs to be distributed as a single file script but is complex enough to need to be written as a collection of modules. This feature is not as popular as it should be, mainly because no one’s heard of it because it wasn’t promoted as part of Python 2.6, but also because Windows users don’t have a file extension (other than .py) to associate with the launcher. This PEP proposes to fix these problems by re-publicising the feature, defining the .pyz and .pyzw extensions as “Python ZIP Applications” and “Windowed Python ZIP Applications”, and providing some simple tooling to manage the format. A New Python ZIP Application Extension The Python 3.4 installer will associate .pyz and .pyzw “Python ZIP Applications” with the platform launcher so they can be executed. A .pyz archive is a console application and a .pyzw archive is a windowed application, indicating whether the console should appear when running the app. Why not use .zip or .py? Users expect a .zip file would be opened with an archive tool, and users expect .py to be opened with a text editor. Both would be confusing for this use case. For UNIX users, .pyz applications should be prefixed with a #! line pointing to the correct Python interpreter and an optional explanation. #!/usr/bin/env python3 # This is a Python application stored in a ZIP archive. (binary contents of archive) As background, ZIP archives are defined with a footer containing relative offsets from the end of the file. They remain valid when concatenated to the end of any other file. This feature is completely standard and is how self-extracting ZIP archives and the bdist_wininst installer format work. Minimal Tooling: The pyzaa Module This PEP also proposes including a simple application for working with these archives: The Python Zip Application Archiver “pyzaa” (rhymes with “huzzah” or “pizza”). “pyzaa” can archive or extract these files, compile bytecode, and can write the __main__ module if it is not present. Usage python -m pyzaa (pack | compile) python -m pyzaa pack [-o path/name] [-m module.submodule:callable] [-c] [-w] [-p interpreter] directory: ZIP the contents of directory as directory.pyz or [-w] directory.pyzw. Adds the executable flag to the archive. -c compile .pyc files and add them to the archive -p interpreter include #!interpreter as the first line of the archive -o path/name archive is written to path/name.pyz[w] instead of dirname. The extension is added if not specified. -m module.submodule:callable __main__.py is written as “import module.submodule; module.submodule.callable()” pyzaa pack will warn if the directory contains C extensions or if it doesn’t contain __main__.py. python -m pyzaa compile arcname.pyz[w] The Python files in arcname.pyz[w] are compiled and appended to the ZIP file. A standard ZIP utility or Python’s zipfile module can unpack the archives. FAQ Q. Isn’t pyzaa just a very thin wrapper over zipfile and compileall? A. Yes. Q. How does this compete with existing sdist/bdist formats? A. There is some overlap, but .pyz files are especially interesting as a way to distribute an installer. They may also prove useful as a way to deliver applications when users shouldn’t be asked to perform virtualenv + “pip install”. References [1] http://bugs.python.org/issue1739468 “Allow interpreter to execute a zip file” [2] http://bugs.python.org/issue17359 “Feature is not documented” Copyright This document has been placed into the public domain. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] What code in Python creates .exe launchers for 'entry_points' in 'setup.py'?
The new "distlib" also provides that feature On Apr 1, 2013 4:58 PM, "Éric Araujo" wrote: > Hello, > > Le 01/04/2013 16:51, Grigory Petrov a écrit : > > But what python code is responsible for creation of this bootstrapper > > executables? I have searched python 2.7.3 source code for some time, but > > can't find a place. > > Setuptools is not in the standard library, so you would need to search > the setuptools or distribute codebases. > > Regards > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Accepting PEP 434, Idle Enhancement Exception
Yes, it would probably make more sense to split the editor and shell processes as many Python IDEs do, with IDLE running in CPython and the user's computation running in the chosen interpreter. On Sat, Mar 30, 2013 at 8:35 AM, Maciej Fijalkowski wrote: > On Sat, Mar 30, 2013 at 5:26 AM, Antoine Pitrou wrote: >> On Sat, 30 Mar 2013 08:33:38 +0200 >> Simon Cross wrote: >> >>> Having a standalone version of IDLE might be really useful to >>> alternative Python implementations. >> >> Why? > > I don't think it's worth discussing - tkinter does not work on any > other implementation than CPython and it seems it won't work. It's a > bit pity, but I guess if I felt really bad about it, I should just > make it work. > > PS. are there idle projects in SoC? Maybe we should put a more > pypy-friendly one there too? > > Cheers, > fijal > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Safely importing zip files with C extensions
Jim Fulton is right that weird failures are a characteristic of zipped
eggs, so one of the #1 requests for setuptools is how to prohibit
zipping from ever happening. This is an important reason why wheel is
billed as an installation format -- fewer users with pitchforks. It's
very cool that it works though. Debugging is slightly easier than it
was in the old days because pdb can now read the source code from the
zip.
An unzipped wheel as a directory with the same name as the wheel would
be a more reliable solution that might be interesting to work with. It
would work the same as egg unless you had important files in the
.data/ (currently mostly used for console scripts and include files)
directory. However it was always confusing to have more than one kind
(zipped, unzipped) of egg.
On Wed, Mar 27, 2013 at 4:41 PM, Vinay Sajip wrote:
> Amaury Forgeot d'Arc gmail.com> writes:
>
>
>> Better: just put the wheel path to sys.path
> sys.path.append('/tmp/simplejson-3.1.2-cp27-none-linux_x86_64.whl')
>> and let a sys.path_hook entry do the job.
>
> That's what the mount() actually does - adds the wheel to a registry that an
> import hook uses. You also need a place to check that the wheel being mounted
> is compatible with the Python doing the mounting - I'm not sure whether what
> the import hook should do if e.g. there is a compatibility problem with the
> wheel (e.g. is it clear that it should always raise an ImportError? Or ignore
> the wheel - seems wrong? Or do something else?)
>
> Regards,
>
> Vinay Sajip
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] IDLE in the stdlib
I showed IDLE to my 6-year-old on the Raspberry Pi and I'm convinced it is cool. Gave up on trying to (slowly) install bpython. We were multiplying large numbers and counting to 325,000 in no time. It might not be for *me* but I'm not going to teach my daughter a large IDE any time soon. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] IDLE in the stdlib
On Wed, Mar 20, 2013 at 3:54 PM, Barry Warsaw wrote: > On Mar 20, 2013, at 12:40 PM, Guido van Rossum wrote: > >>I didn't hear any at the sprint here. > > JFDI! :) > > -Barry +1 why are we still talking show me the patches ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] built-in Python test runner (was: Python Language Summit at PyCon: Agenda)
On Tue, Mar 5, 2013 at 2:49 PM, Barry Warsaw wrote: > On Mar 05, 2013, at 02:11 AM, Donald Stufft wrote: > >>Doesn't setuptools/distribute already have a setup.py test command? That >>seems like the easiest way forward? > > Yes, and in theory it can make `python setup.py test` work well. But there > are lots of little details (such as API differences for ensuring that doctests > run, "additional tests" discovery, etc.) that make this often not work so well > in practice. Some of that is social and some of it is technical. I still > claim that including test suite information in a package's metadata would be a > win, but maybe that's just too much to hope for right now. It would be a win, but "parsing the metadata" is just not what happens right now, let alone writing anything about which and where the modules are defined in the sdist. We can barely install packages by using the dependency metadata from PKG-INFO; pip always re-generates it from "setup.py egg_info". Your testing metadata prototype would only have to write two lines to the metadata instead of one a-la: Extension: flufl; flufl/test_suite: nose.collector; document the extension; write some tool to actually parse the metadata and invoke the tests; it may become a core feature in the next version, or having a monolithic specification may become less important. Thanks, Daniel Holth ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] running tests; mebs
>> As a community, we know how important tests are, so I think our tools should >> reflect that and make it easy for those tests to be expressed. As a selfish >> side-effect, I want to reduce the amount of guesswork I need to perform in >> order to know how to run a package's test when I `$vcs clone` their >> repository. ;) >> > > > Distutils2 had a way of specifying this in the metadata. It looks like this > hasn't made it into the reboot: > > http://alexis.notmyidea.org/distutils2/distutils/newcommands.html > > Michael > >> Cheers, >> -Barry I'm not aware of a reboot of the setup.py replacement / improvement effort. The work that has been done has proceeded backwards from the installer end of things. I had a proposal called "mebs, not an actual project". A completely plugin-based system would recognize any sdist format and provide a minimal, consistent interface. Add tests to the below text from October. ... A very simple meta-build system "mebs" is used to recognize sdists and build binary packages. Build systems provide plugins having three methods, .recognize() .metadata() .build() An installer downloads an sdist. For each installed build plugin, .recognize(dir) is called. The first plugin to return True is used. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python Language Summit at PyCon: Agenda
On Mon, Mar 4, 2013 at 3:14 PM, Barry Warsaw wrote: > On Mar 04, 2013, at 03:02 PM, Daniel Holth wrote: > >>setup.py's setup(test_suite="x")... not sure if this is a distutils or >>setuptools feature. PEP 426 has an extension mechanism that could do >>the job. > > Shouldn't "testing" be a first order feature? Unfortunately there are so many potential first-order features that we've had to leave some out in order to save time. "How to run the tests" is not something that you need to know when searching PyPI for a distribution and its dependencies. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python Language Summit at PyCon: Agenda
On Mon, Mar 4, 2013 at 2:26 PM, Berker Peksağ wrote: > On Mon, Mar 4, 2013 at 9:14 PM, Barry Warsaw wrote: >> On Mar 04, 2013, at 07:41 PM, Antoine Pitrou wrote: >> $ python -m unittest discover $ python setup.py test $ python setup.py nosetests $ python -m nose test $ nosetests-X.Y Besides having a multitude of choices, there's almost no way to automatically discover (e.g. by metadata inspection or some such) how to invoke the tests. You're often lucky if there's a README.test and it's still accurate. >>> >>>I hope we can have a "pytest" utility that does the right thing in 3.4 :-) >>>Typing "python -m unittest discover" is too cumbersome. >> >> Where is this work being done (e.g. is there a PEP)? > > There is an open issue on the tracker: http://bugs.python.org/issue14266 > > --Berker setup.py's setup(test_suite="x")... not sure if this is a distutils or setuptools feature. PEP 426 has an extension mechanism that could do the job. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cffi in stdlib
On Wed, Feb 27, 2013 at 4:18 AM, Ronald Oussoren wrote: > > On 27 Feb, 2013, at 10:06, Maciej Fijalkowski wrote: > >> On Wed, Feb 27, 2013 at 9:29 AM, Ronald Oussoren >> wrote: >>> >>> On 26 Feb, 2013, at 16:13, Maciej Fijalkowski wrote: >>> Hello. I would like to discuss on the language summit a potential inclusion of cffi[1] into stdlib. >>> >>> The API in general looks nice, but I do have some concens w.r.t. including >>> cffi in the stdlib. >>> >>> 1. Why is cffi completely separate from ctypes, instead of layered on top >>> of it? That is, add a utility module to ctypes that can parse C >>> declarations and generate the right ctypes definitions. >> >> Because ctypes API is a mess and magic. We needed a cleaner (and much >> smaller) model. > > The major advantages of starting over is probably that you can hide the > complexity and that opens opportunities for optimizations. That said, I'm not > convinced that ctypes is unnecessarily complex. cffi actually does have a ctypes backend in addition to the ffi and "compile a CPython extension" backends. But the ctypes backend is guaranteed to be slow and messy because it is ctypes. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cffi in stdlib
+1 On Tue, Feb 26, 2013 at 12:14 PM, Maciej Fijalkowski wrote: > On Tue, Feb 26, 2013 at 7:07 PM, Eli Bendersky wrote: > > > > > > > > On Tue, Feb 26, 2013 at 8:39 AM, Maciej Fijalkowski > > wrote: > >> > >> On Tue, Feb 26, 2013 at 6:34 PM, Eli Bendersky > wrote: > >> > > >> > On Tue, Feb 26, 2013 at 7:42 AM, Nick Coghlan > >> > wrote: > >> >> > >> >> On Wed, Feb 27, 2013 at 1:13 AM, Maciej Fijalkowski < > fij...@gmail.com> > >> >> wrote: > >> >> > Hello. > >> >> > > >> >> > I would like to discuss on the language summit a potential > inclusion > >> >> > of cffi[1] into stdlib. > >> >> > >> >> I think cffi is well worth considering as a possible inclusion for > >> >> Python 3.4. (In particular, I'm a fan of the fact it just uses C > >> >> syntax to declare what you're trying to talk to) > >> > > >> > > >> > I'm cautiously +0.5 because I'd really like to see a strong comparison > >> > case > >> > being made vs. ctypes. I've used ctypes many times and it was easy and > >> > effortless (well, except the segfaults when wrong argument types are > >> > declared :-). I'll be really interesting in seeing concrete examples > >> > that > >> > demonstrate how CFFI is superior. > >> > >> My main issue with ctypes, other than confusing API, which is up to > >> taste, is that you just cannot wrap some libraries, like OpenSSL > >> because of API vs ABI. OpenSSL uses macros extensively. Another point > >> is that even C POSIX stdlib gives you incomplete structs and you have > >> to guess where to put what fields. > > > > > > Yep, I can think of some reasons too. But just mentioning it so you > remember > > explicitly listing the advantages when it comes to writing a PEP or some > > other sort of formal proposal. An FWIW, I think there's already enough > > positive feedback to at least start drafting a PEP. It can always be > > rejected later ;-) > > > > Eli > > > > It's copy-pasted in the original mail and it's in the docs. Anyway, > this is just to familizarize people with the library. We're going to > discuss it on the language summit at pycon. > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
On Tue, Feb 19, 2013 at 5:10 PM, M.-A. Lemburg wrote: > On 19.02.2013 23:01, Daniel Holth wrote: > > On Tue, Feb 19, 2013 at 4:34 PM, M.-A. Lemburg wrote: > > > >> On 19.02.2013 14:40, Nick Coghlan wrote: > >>> On Tue, Feb 19, 2013 at 11:23 PM, M.-A. Lemburg > wrote: > >>>> * PEP 426 doesn't include any mention of the egg distribution format, > >>>> even though it's the most popular distribution format at the moment. > >>>> It should at least include the location of the metadata file > >>>> in eggs (EGG-INFO/PKG-INFO) and egg installations > >>>> (/EGG-INFO/PKG-INFO). > >>> > >>> "Other tools involved in Python distribution may also use this format." > >>> > >>> The egg format has never been, and never will be, officially endorsed > >>> by python-dev. The wheel format is the standard format for binary > >>> distribution, and PEP 376 defines the standard location for metadata > >>> on installed distributions. > >> > >> Oh, come on, Nick, that's just silly. setuptools was included in stdlib > >> for a short while, so the above is simply wrong. Eggs are the most > >> widely used binary distribution format for Python package on PyPI: > >> > >> # wc *files.csv > >> 25585 25598 1431013 2013-02-19-egg-files.csv > >>46194640 236694 2013-02-19-exe-files.csv > >> 254 255 13402 2013-02-19-msi-files.csv > >> 104691 104853 5251962 2013-02-19-tar-gz-files.csv > >> 24 241221 2013-02-19-whl-files.csv > >> 17937 18022 905913 2013-02-19-zip-files.csv > >> 153110 153392 7840205 total > >> > >> (based on todays PyPI stats) > >> > >> It doesn't really help ignoring realities... and I'm saying > >> that as one of the core devs who got setuptools kicked out of > >> the stdlib again. > >> > >> -- > >> Marc-Andre Lemburg > >> eGenix.com > >> > > > > The wheel philosophy is that it should be supported by both python-dev > and > > setuptools and that you should feel happy about using setuptools if you > > like it whether or not python-dev (currently) endorses that. If you are > > using setuptools (distribute's pkg_resources) then you can use both at > the > > same time. > > > > Distribute, distutils and setuptools' problems have not been well > > understood which I think is why there has been a need to discredit > > setuptools by calling it non-standard. It is the defacto standard. If > your > > packages have dependencies there is no other choice. Wheel tries to solve > > the real problem by allowing you to build a package with setuptools while > > giving the end-user the choice of installing setuptools or not. > > > > Of course eggs are the most popular right now. The wheel format is very > > egg-like while avoiding some of egg's problems. See the comparison in the > > PEP or read the story on wheel's rtfd. The wheel project includes tools > to > > losslessly convert eggs or bdist_wininst to wheel. > > That's all fine, but it doesn't explain the refusal to add the > documentation of the location of the PKG-INFO file in eggs ? It would just be a sentence, I wouldn't have a problem with it but I also don't see why it would be necessary. Even setuptools doesn't touch the file usually. Right now distribute's pkg_resources currently only understands Requires-Dist if it is inside a .dist-info directory. Sorry, Chris must have meant http://hg.python.org/distlib/ . I was struggling to imagine a world where that is more visible than something on bitbucket. Half the comments have been about putting something in stdlib right away, something that I do not understand at all as a voracious PyPI or https://crate.io/ user. You should secure both the metadata and the packages but it doesn't have anything to do with the serialization format. Wheel is designed so that the metadata is at the end of the .zip and can be retrieved with an HTTP partial request. The feature may or may not be useful. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
On Tue, Feb 19, 2013 at 4:34 PM, M.-A. Lemburg wrote: > On 19.02.2013 14:40, Nick Coghlan wrote: > > On Tue, Feb 19, 2013 at 11:23 PM, M.-A. Lemburg wrote: > >> * PEP 426 doesn't include any mention of the egg distribution format, > >> even though it's the most popular distribution format at the moment. > >> It should at least include the location of the metadata file > >> in eggs (EGG-INFO/PKG-INFO) and egg installations > >> (/EGG-INFO/PKG-INFO). > > > > "Other tools involved in Python distribution may also use this format." > > > > The egg format has never been, and never will be, officially endorsed > > by python-dev. The wheel format is the standard format for binary > > distribution, and PEP 376 defines the standard location for metadata > > on installed distributions. > > Oh, come on, Nick, that's just silly. setuptools was included in stdlib > for a short while, so the above is simply wrong. Eggs are the most > widely used binary distribution format for Python package on PyPI: > > # wc *files.csv > 25585 25598 1431013 2013-02-19-egg-files.csv >46194640 236694 2013-02-19-exe-files.csv > 254 255 13402 2013-02-19-msi-files.csv > 104691 104853 5251962 2013-02-19-tar-gz-files.csv > 24 241221 2013-02-19-whl-files.csv > 17937 18022 905913 2013-02-19-zip-files.csv > 153110 153392 7840205 total > > (based on todays PyPI stats) > > It doesn't really help ignoring realities... and I'm saying > that as one of the core devs who got setuptools kicked out of > the stdlib again. > > -- > Marc-Andre Lemburg > eGenix.com > The wheel philosophy is that it should be supported by both python-dev and setuptools and that you should feel happy about using setuptools if you like it whether or not python-dev (currently) endorses that. If you are using setuptools (distribute's pkg_resources) then you can use both at the same time. Distribute, distutils and setuptools' problems have not been well understood which I think is why there has been a need to discredit setuptools by calling it non-standard. It is the defacto standard. If your packages have dependencies there is no other choice. Wheel tries to solve the real problem by allowing you to build a package with setuptools while giving the end-user the choice of installing setuptools or not. Of course eggs are the most popular right now. The wheel format is very egg-like while avoiding some of egg's problems. See the comparison in the PEP or read the story on wheel's rtfd. The wheel project includes tools to losslessly convert eggs or bdist_wininst to wheel. I am confident distlib can thrive outside of the standard library! Why the rush to kill it before its prime? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec for distribution metadata 2.0
On Tue, Feb 19, 2013 at 3:25 PM, Paul Moore wrote: > On 19 February 2013 13:40, Nick Coghlan wrote: > >> If a tools wants to support metadata 2.0, it has to support all > >> the complicated stuff as well, i.e. handle the requires fields, > >> the environment markers and version comparisons/sorting. > > > > Which is what distutils2 can be used for now, and what distlib will > > provide without the unwanted build system infrastructure in > > distutils2. > > One particular item in Metadata 2.0 which needs to be addressed is > requirements specifiers, as used in (for example) Requires-Dist. These > take forms like "DistName (>= ver)". There are a number of potential > issues here: > > 1. This differs from the pkg_resources format (which omits the > parentheses). Having 2 subtly different formats is not a good idea in > the long term. At the moment, pkg_resources format is used in pip > requirements (on the command line and in requirement files) as well as > in setuptools/distribute and possibly elsewhere. > 2. There is currently no code that I am aware of that implements this > spec. I don't believe distlib does (yet - give Vinay 5 minutes and who > knows? :-)), pkg_resources as I said implements a different format, > and distutils2, apart from being a big dependency to cover just this > point, only matches the version (not the project name) and presumably > does so using the distutils2 version ordering (which is incompatible > with Metadata 2.0). > 3. The format is fiddly to parse and process - nobody is likely to > implement it for themselves without a library that does it (especially > not when using pkg_resources is so easy). > > The PEP needs a rationale as to why the pkg_resources format wasn't > used, and should suggest a migration path for projects which currently > use the pkg_resources style (such as pip's requirements files). > > Paul We did add full support for Metadata 2.0 Requires-Dist in pkg_resources. The fiddling happens here: https://bitbucket.org/tarek/distribute/src/tip/pkg_resources.py?at=default#cl-2469 The code removes the parenthesis and adds == when no operator was specified to convert us back to the pkg_resources format. The rest of pkg_resources carries on as before. Wheel installs would not pull in dependencies otherwise. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec fordistribution metadata 2.0
On Tue, Feb 19, 2013 at 11:26 AM, Paul Moore wrote: > On 19 February 2013 13:59, Nick Coghlan wrote: > > It's OK if people don't want to read the detailed rationale provided > > for each of the major changes as part of the PEP, or if they want to > > dispute a particular piece of that rationale. But merely going "it's > > too complicated!" or "metadata 1.2 failed, so 2.0 will fail as well!" > > is not a reasonable response. Software distribution is complicated - > > trying to oversimplify it is one of the reasons setuptools became > > necessary. > > Nevertheless, the landscape is confusing. PEPs 241, 314, 345, 426, > 390, 376 and 386 are all relevant to one extent or another, and only > PEPs 426 (Metadata 2.0) and 390 (Static Metadata for Distutils - > setup.cfg) are still in Draft format. The others are all Accepted or > Final. And yet PEPs 345, 390 and 386 are unused and likely to remain > so. > > I believe that the only ones that are *actually* of use are 241, 314 > and 426 (Metadata 1.0, 1.1 and 2.0, but not 1.2) and 376 (Database of > Installed Python Distributions) although 376 still has a number of > flaws. I'd suggest that these should be marked as Final, and the > others as Rejected, so that we have a clear statement of what is > actually supported. > > Making sure that users have the means to write code that *uses* these > standards using functionality available in the stdlib is then the next > step as you say. It is critical that this is done, because packaging > tools are unique in that the barrier to using external dependencies is > particularly high for them - for example, pip's reliance on > distribute/setuptools is necessary, but has been problematic at times. > > Paul > > PS Apologies for using a load of PEP numbers without explanation. > Here's a glossary: > > Metadata 1.0 - PEP 241 > Metadata 1.1 - PEP 314 > Metadata 1.2 - PEP 345 > Metadata 1.3 - PEP 426 > Static Metadata for Distutils - PEP 390 > Database of Installed Python Distributions - PEP 376 > Changing the version comparison module in Distutils - PEP 386 > The documentation is bad. The standard library will almost certainly grow code that can interpret these PEPs. Packaging tool authors can handle these specs but no sane person reads PEPs to figure out how to submit a simple library to pypi. No end user cares about metadata versions either or even notices that sdists contain PKG-INFO at their root. You get what setup() produces and that's it. Wheel installers can work without being installed into the target environment at all. This is very hard to understand if you have been using the distutils model (the installed package imports the installer and packages are always rebuilt from source) for more than a decade. This feature does something to remedy the setuptools chicken/egg problem. We have eliminated the egg ;-) Once the installer doesn't need to be installed, repeatedly, in every virtualenv, hundreds of times a day, users could just have a single up-to-date standalone copy of pip invoked as "python pip.zip install x". Additional packages needed only for builds (even setuptools) may be installed automatically by a build system (like pip) when building rather than using packages. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: PEP 426 is now the draft spec fordistribution metadata 2.0
On Feb 19, 2013 6:57 AM, wrote: > > I've never seen environment markers being used or supported > > in the wild. > > > > I'm not against modernizing the format, but given that version 1.2 > > has been out for around 8 years now, without much following, > > I think we need to make the implementation bit a requirement > > before accepting the PEP. > > Elephant in the room? Somethin's been around for so long and hasn't got > any sort of > mainstream support is *not* a good candidate for a requirement but maybe > it's me > being naive. > I think 1.2 was started and then amended over a long period of time to become what it is today. I wanted to edit it again just to add Provides-Extra. It turned out to be more practical to make a new PEP. Nick explains that for complicated reasons the implementation of Metadata 1.2 was not successful. For Metadata 2.0 we had a different strategy: put it into distribute first. You will find that this aids adoption. distlib will provide another implementation that is in many ways interchangeable. We also have three implementations of an installer for the related wheel format, three implementations of the wheel builder, and two surprisingly short implementations of environment markers. Yes, the wheel project has the same name as the wheel format. Metadata 2.0's Provides-Extra field is there to represent the setuptools "extras" feature which is necessary for a very large number of distributions on pypi. For a while the most common environment markers will look like just "extra == 'foo'". So it should not be surprising that distribute is the first tool to recognize Provides-Extra. Previously distribute did not use METADATA / PKG-INFO for dependencies, instead reading from requires.txt If you can get around the fact that we've incorporated most of another PEP 386 into v2.0, rewritten and explained everything to death, I would argue that 2.0 is actually simpler than 1.2. Only four headers are actually required in a valid Metadata 2.0 file and a number of the fields are informational and do not have to be understood by the installer. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 426 is now the draft spec for distribution metadata 2.0
On Sun, 17 Feb 2013 20:11:23 +1000 Nick Coghlan wrote: > > Major changes since the last draft: > > 1. Metadata-Version is 2.0 rather than 1.3, and the field now has the > same major.minor semantics as are defined for wheel versions in PEP > 427 (i.e. if a tool sees a major version number it doesn't recognise, > it should give up rather than trying to guess what to do with it, > while it's OK to process a higher minor version) So is it possible to bundle several versioned metadata files in order to maximize compatibility? Or does a distribution with 2.0 metadata have to abandon all hope of being compatible with all tools? Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com Not likely to matter for a while as the current md v1 tools don't understand this new obsolescence rule :-) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Submitting PEP 425 (compatibility tags)
This is the improved compatibility tags PEP 425, specifying how part of the
Wheel PEP 427 filenames work. Last time we discussed whether replacing .
with _ was ugly but I concluded it was harmless.
Submitted for your consideration,
PEP: 425
Title: Compatibility Tags for Built Distributions
Version: $Revision$
Last-Modified: 07-Aug-2012
Author: Daniel Holth
BDFL-Delegate: Nick Coghlan
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 27-Jul-2012
Python-Version: 3.4
Post-History: 8-Aug-2012, 18-Oct-2012, 15-Feb-2013
Abstract
This PEP specifies a tagging system to indicate with which versions of
Python a built or binary distribution is compatible. A set of three
tags indicate which Python implementation and language version, ABI,
and platform a built distribution requires. The tags are terse because
they will be included in filenames.
PEP Editor's Note
=
While the naming scheme described in this PEP will not be supported directly
in the standard library until Python 3.4 at the earliest, draft
implementations may be made available in third party projects.
Rationale
=
Today "python setup.py bdist" generates the same filename on PyPy
and CPython, but an incompatible archive, making it inconvenient to
share built distributions in the same folder or index. Instead, built
distributions should have a file naming convention that includes enough
information to decide whether or not a particular archive is compatible
with a particular implementation.
Previous efforts come from a time where CPython was the only important
implementation and the ABI was the same as the Python language release.
This specification improves upon the older schemes by including the Python
implementation, language version, ABI, and platform as a set of tags.
By comparing the tags it supports with the tags listed by the
distribution, an installer can make an educated decision about whether
to download a particular built distribution without having to read its
full metadata.
Overview
The tag format is {python tag}-{abi tag}-{platform tag}
python tag
‘py27’, ‘cp33’
abi tag
‘cp32dmu’, ‘none’
platform tag
‘linux_x86_64’, ‘any’
For example, the tag py27-none-any indicates compatible with Python 2.7
(any Python 2.7 implementation) with no abi requirement, on any platform.
Use
===
The `wheel` built package format includes these tags in its filenames,
of the form ``{distribution}-{version}(-{build tag})?-{python tag}-{abi
tag}-{platform tag}.whl``. Other package formats may have their own
conventions.
Details
===
Python Tag
--
The Python tag indicates the implementation and version required by
a distribution. Major implementations have abbreviated codes, initially:
* py: Generic Python (does not require implementation-specific features)
* cp: CPython
* ip: IronPython
* pp: PyPy
* jy: Jython
Other Python implementations should use `sys.implementation.name`.
The version is `py_version_nodot`. CPython gets away with no dot,
but if one is needed the underscore `_` is used instead. PyPy should
probably use its own versions here `pp18`, `pp19`.
The version can be just the major version `2` or `3` `py2`, `py3` for
many pure-Python distributions.
Importantly, major-version-only tags like `py2` and `py3` are not
shorthand for `py20` and `py30`. Instead, these tags mean the packager
intentionally released a cross-version-compatible distribution.
A single-source Python 2/3 compatible distribution can use the compound
tag `py2.py3`. See `Compressed Tag Sets`, below.
ABI Tag
---
The ABI tag indicates which Python ABI is required by any included
extension modules. For implementation-specific ABIs, the implementation
is abbreviated in the same way as the Python Tag, e.g. `cp33d` would be
the CPython 3.3 ABI with debugging.
The CPython stable ABI is `abi3` as in the shared library suffix.
Implementations with a very unstable ABI may use the first 6 bytes (as
8 base64-encoded characters) of the SHA-256 hash of ther source code
revision and compiler flags, etc, but will probably not have a great need
to distribute binary distributions. Each implementation's community may
decide how to best use the ABI tag.
Platform Tag
The platform tag is simply `distutils.util.get_platform()` with all
hyphens `-` and periods `.` replaced with underscore `_`.
* win32
* linux_i386
* linux_x86_64
Use
===
The tags are used by installers to decide which built distribution
(if any) to download from a list of potential built distributions.
The installer maintains a list of (pyver, abi, arch) tuples that it
will support. If the built distribution's tag is `in` the list, then
it can be installed.
For example, an installer running under CPython 3.3 on a linux_x86_64
system might support::
1. cp33-cp33m-linux_x86_64
2. cp33-abi3-linux_x86_64
3. cp33-none-linux_x86_64
4. cp33-none-any
5. cp3-none-any
6. cp32-none
Re: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3)
On Fri, Feb 15, 2013 at 12:27 PM, Erik Bray wrote: > On Sun, Feb 3, 2013 at 5:24 PM, Vinay Sajip > wrote: > > Éric Araujo netwok.org> writes: > > > >> Looks like we agree that a basic tool able to bootstrap the packaging > >> story is needed :) > > > > Agreed. Just because distutils can't easily/reliably build things that > are > > better built with SCons/WAF/tup/whatever, doesn't mean that we shouldn't > have > > the ability to build pure-Python distributions and distributions > including C > > libs and extensions, with the ability to extend easily by third-party > tools. It > > just needs to be done in a way which is easy to build on, so the included > > battery stays small and simple. Easier said than done, I know :-) > > > > Regards, > > > > Vinay Sajip > > Sorry to revive an old-ish discussion--I'm just catching up on things. > But I just wanted to add that distutils is still pretty okay for > building reasonably complex projects. Although it does not rise to > the level of complexity of Numpy or SciPy, the Astropy project > (https://github.com/astropy/astropy) has managed to put together a > pretty nice build system on top of mostly-plain distutils (it does use > distribute but primarily just for 2to3 support). > > > This has necessitated a number of hacks to overcome shortcomings and > bugs in distutils, but most of those shortcomings could probably be > fixed in distutils within the framework of a slightly lifted freeze. > But in any case I haven't found it worthwhile to switch to something > like bento when the batteries included in the stdlib have been mostly > Good Enough. Having fewer installation dependencies has also made it > significantly easier for non-advanced users to install. Even the > distribute requirement doesn't add too much overhead, as most users > have it on their systems by default now, and for those who don't > distribute_setup.py works okay. > > TL;DR, strong -1 on the stdlib "getting out of the build business". > Also as I think Nick already mentioned one of the wins of > Setup-Requires-Dist is to have a standard way to bring in extra build > requirements (such as bento) so if we have better support for a > feature like that it's not necessary to "bless" any preferred tool. > Distutils is not really going away. We need it to build the existing 28,000 packages. However empirically it seems if you try to write a significant extension to or improvement of distutils then you are likely to get burnt out and switch careers. Instead of literally killing distutils we hope to make it very easy to use other build tools when you need them and not use any build tools at all when you don't. As a thought experiment: what if one of those third party build tools hosted on pypi was distutils itself? What would you need to do to make that happen? The packaging peps PEP-376 and so on are brilliant because they are simple enough to be implemented twice. If we had better ways to separate interface from implementation in Python I'd like to see two of whatever else we come up with for packaging. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Usage of += on strings in loops in stdlib
On Wed, Feb 13, 2013 at 7:10 AM, Serhiy Storchaka wrote:
> On 13.02.13 10:52, Larry Hastings wrote:
>
>> I've always hated the "".join(array) idiom for "fast" string
>> concatenation--it's ugly and it flies in the face of TOOWTDI. I think
>> everyone should use "x = a + b + c + d" for string concatenation, and we
>> should just make that fast.
>>
>
> I prefer "x = '%s%s%s%s' % (a, b, c, d)" when string's number is more than
> 3 and some of them are literal strings.
>
Fixed: x = ('%s' * len(abcd)) % abcd
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3)
He is being self deprecating. Its also true that python dev can't recommend bento wholesale. That is fine with me. On Feb 3, 2013 5:36 PM, "Vinay Sajip" wrote: > Simon Cross gmail.com> writes: > > > For the record, all the reasons listed at [1] appear trivial. > > In Bento's author's own words - "Weak documentation", "Mediocre code > quality", > "at a lower level, a lot of code leaves to be desired" may be trivial if > David > is just being self-deprecating, but what if he isn't? Or perhaps that part > of > the page is out of date, and needs updating? I can certainly agree with the > "Weak documentation" part of the assessment, but this makes it hard to > assess > the package as a whole. Note that I'm not sniping - writing good > documentation > is hard work. > > Regards, > > Vinay Sajip > > > > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3)
They can be signed with pypi detached signatures already. It works now exactly as for sdist. The innovation was supposed to be in convenience for the signer, in allowing keys to be trusted per package and for a list of dependencies and the expected signing keys to be shared easily. Does anyone have a concise pure py3 s/mime implementation? On Feb 3, 2013 4:52 PM, "Vinay Sajip" wrote: > Paul Moore gmail.com> writes: > > > So it's perfectly possible to use wheels right now, without the pip > > integration. But the pip developers don't want to integrate the wheel > > format just because it exists - they want the assurance that it's an > > accepted format supported by PEPs, hence the interest in getting the 3 > > wheel PEPs (of which the metadata PEP is the first) accepted. > > Likewise, I will look at the possibility of providing wheel support in > distlib, > once it has been accepted as a standard and the open issues (such as > signature > scheme) have been resolved. > > Regards, > > Vinay Sajip > > > > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] BDFL delegation for PEP 426 + distutils freeze
I did think that updating distutils to have basic support for the packaging
PEPs was a decent idea, but then it wound up being more or less rewritten
entirely like I've been cajoled into doing with PEP 426 (Metadata). I don't
know whether distutils(1) can survive the minimum changes required for
wheel (.dist-info folders and a bdist_wheel command) without breaking a
significant number of its clients or not. History suggests no. It may still
be worth a shot.
bdist_wheel started out as a distutils2 plugin but it was inconvenient to
develop it inside the CPython repository and it was not practical to
convert projects over from setuptools to distutils2.
The pypi metadata has had a standardlib generator for a long time without
having anything in the standard library to make use of the metadata. Wheels
are just .zip files of one package's share of the site-packages directory.
Why not support them asymmetrically. I value the PEP and consensus for the
format more than the implementation.
One idea that's sure to be controversial would be to include a bootstrap
installer as a script without committing to a Python API (just a
command-line API). Since Python 2.6, wheel's own wheel archive can be used
to install wheels:
$ python wheel-0.14.0-py2.py3-none-any.whl/wheel -h
usage: wheel [-h] {keygen,sign,verify,unpack,install,convert,help} ...
On Sun, Feb 3, 2013 at 2:40 PM, Chris Jerdonek wrote:
> On Sun, Feb 3, 2013 at 10:33 AM, Éric Araujo wrote:
> > Le 03/02/2013 07:48, Antoine Pitrou a écrit :
> >> I vote for removing the "distutils is frozen" principle.
> > I’ve also been thinking about that. There have been two exceptions to
> > the freeze, for ABI flags in extension module names and for pycache
> > directories. When the stable ABI was added and MvL wanted to change
> > distutils (I don’t know to do what exactly), Tarek stood firm on the
> > freeze and asked for any improvement to go into distutils2, and after
> > MvL said that he would not contibute to an outside project, we merged d2
> > into the stdlib. Namespace packages did not impact distutils either.
> > Now that we’ve removed packaging from the stdlib, we have two Python
> > features that are not supported in the standard packaging system, and I
> > agree that it is a bad thing for our users.
> >
> > I’d like to propose a reformulation of the freeze:
>
> This could be common knowledge, but is the current formulation of the
> freeze spelled out somewhere?
>
> --Chris
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
