Re: [Python-Dev] OpenIndiana and Solaris support
On Mon, 26 Sep 2016 at 15:38 Guido van Rossumwrote: > Thanks for the reality check Trent! I think if enough people with core > committer bits want to keep supporting Solaris / Illumos / OpenIndiana > / other variants that's fine, but I don't think that just having some > VMs to test on is enough -- we also need people who can fix problems > if those buildbots start failing, and that requires pretty specialized > knowledge. Plus of course we won't know if fixing it for OpenIndiana > will also fix it for Solaris 11 Express or for other Illumos forks. > (For Linux it's easier to assess these things because so many people > in open source use Linux and its many forks.) > The official requirement to support a platform is a stable buildbot and a core dev to keep the support up: https://www.python.org/dev/peps/pep-0011/#supporting-platforms. Victor has asked that the OpenIndiana buildbot be removed from the stable pool as it consistently throws MemoryError which means its support is not improving. If Trent is willing to maintain a buildbot in a Joyent VM that at least takes care of that part, but it still requires Jesus to volunteer to keep the support up if it's going to be supported for free. Otherwise Joyent could consider contracting with one of the various core devs who happen to be consultants to help maintain the support. At minimum, though, a new buildbot could go into the unstable pool so illumos devs can keep an eye on when things break to try and get platform-independent changes upstreamed that happen to help illumos (e.g. no #ifdef changes specific to illumos, but if something just needed to be made more robust and it happens to help illumos that's typically fine). > > On Mon, Sep 26, 2016 at 3:32 PM, Trent Mick wrote: > > I work for Joyent (joyent.com) now, which employs a number of devs that > work > > on illumos (illumos.org). We also provide cloud infrastructure. Would it > > help if we offered one or more instances (VMs) on which to run buildbot > > slaves (and on which volunteers for bug fixing could hack)? I know a > lot of > > people in the illumos community would be quite sad to have it dropped as > a > > core Python plat. > > > > Guido, > > Yes you are correct that Oracle owns the Solaris brand. > > > > tl;dr history if you care: > > - sunos -> Solaris > > - Sun open sources Solaris, called OpenSolaris (2005) > > - Oracle acquires Sun and closes Solaris (Aug 2010). Shortly after, the > > community forks OpenSolaris and calls it illumos (Sep 2010) > > - OpenIndiana is a distro of illumos (somewhat similar to how Ubuntu is a > > distro of Linux). Other distros are SmartOS (the one Joyent works on), > and > > OmniOS. > > - Oracle continues work on Solaris, releasing "Solaris 11 Express". > > > > I've no real numbers of usage of illumos vs Solaris 11 vs others. > > > > Cheers, > > Trent > > > > p.s. I hear that Jesus is also in contact with some of the illumos-devs > on > > IRC (and perhaps email). I hope we can help there. > > > > -- > --Guido van Rossum (python.org/~guido) > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] TextIO seek and tell cookies
Yeah, that should work. The implementation is something like a byte offset to the start of a line plus a character count, plus some misc flags. I found this implementation in the 2.6 code (the last version where it was pure Python code): def _pack_cookie(self, position, dec_flags=0, bytes_to_feed=0, need_eof=0, chars_to_skip=0): # The meaning of a tell() cookie is: seek to position, set the # decoder flags to dec_flags, read bytes_to_feed bytes, feed them # into the decoder with need_eof as the EOF flag, then skip # chars_to_skip characters of the decoded result. For most simple # decoders, tell() will often just give a byte offset in the file. return (position | (dec_flags<<64) | (bytes_to_feed<<128) | (chars_to_skip<<192) | bool(need_eof)<<256) def _unpack_cookie(self, bigint): rest, position = divmod(bigint, 1<<64) rest, dec_flags = divmod(rest, 1<<64) rest, bytes_to_feed = divmod(rest, 1<<64) need_eof, chars_to_skip = divmod(rest, 1<<64) return position, dec_flags, bytes_to_feed, need_eof, chars_to_skip On Mon, Sep 26, 2016 at 3:43 PM, Greg Ewingwrote: > Ben Leslie wrote: >> >> But the idea of transmitting these offsets outside of a running >> process is not something that I had anticipated. It got me thinking: >> is there a guarantee that these opaque values returned from tell() is >> stable across different versions of Python? > > > Are they even guaranteed to work on a different file > object in the same process? I.e. if you read some stuff > from a file, do tell() on it, then close it, open it > again and seek() with that token, are you guaranteed to > end up at the same place in the file? > > -- > Greg > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] TextIO seek and tell cookies
Ben Leslie wrote: But the idea of transmitting these offsets outside of a running process is not something that I had anticipated. It got me thinking: is there a guarantee that these opaque values returned from tell() is stable across different versions of Python? Are they even guaranteed to work on a different file object in the same process? I.e. if you read some stuff from a file, do tell() on it, then close it, open it again and seek() with that token, are you guaranteed to end up at the same place in the file? -- Greg ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] OpenIndiana and Solaris support
Thanks for the reality check Trent! I think if enough people with core committer bits want to keep supporting Solaris / Illumos / OpenIndiana / other variants that's fine, but I don't think that just having some VMs to test on is enough -- we also need people who can fix problems if those buildbots start failing, and that requires pretty specialized knowledge. Plus of course we won't know if fixing it for OpenIndiana will also fix it for Solaris 11 Express or for other Illumos forks. (For Linux it's easier to assess these things because so many people in open source use Linux and its many forks.) On Mon, Sep 26, 2016 at 3:32 PM, Trent Mickwrote: > I work for Joyent (joyent.com) now, which employs a number of devs that work > on illumos (illumos.org). We also provide cloud infrastructure. Would it > help if we offered one or more instances (VMs) on which to run buildbot > slaves (and on which volunteers for bug fixing could hack)? I know a lot of > people in the illumos community would be quite sad to have it dropped as a > core Python plat. > > Guido, > Yes you are correct that Oracle owns the Solaris brand. > > tl;dr history if you care: > - sunos -> Solaris > - Sun open sources Solaris, called OpenSolaris (2005) > - Oracle acquires Sun and closes Solaris (Aug 2010). Shortly after, the > community forks OpenSolaris and calls it illumos (Sep 2010) > - OpenIndiana is a distro of illumos (somewhat similar to how Ubuntu is a > distro of Linux). Other distros are SmartOS (the one Joyent works on), and > OmniOS. > - Oracle continues work on Solaris, releasing "Solaris 11 Express". > > I've no real numbers of usage of illumos vs Solaris 11 vs others. > > Cheers, > Trent > > p.s. I hear that Jesus is also in contact with some of the illumos-devs on > IRC (and perhaps email). I hope we can help there. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] OpenIndiana and Solaris support
I work for Joyent (joyent.com) now, which employs a number of devs that work on illumos (illumos.org). We also provide cloud infrastructure. Would it help if we offered one or more instances (VMs) on which to run buildbot slaves (and on which volunteers for bug fixing could hack)? I know a lot of people in the illumos community would be quite sad to have it dropped as a core Python plat. Guido, Yes you are correct that Oracle owns the Solaris brand. tl;dr history if you care: - sunos -> Solaris - Sun open sources Solaris, called OpenSolaris (2005) - Oracle acquires Sun and closes Solaris (Aug 2010). Shortly after, the community forks OpenSolaris and calls it illumos (Sep 2010) - OpenIndiana is a distro of illumos (somewhat similar to how Ubuntu is a distro of Linux). Other distros are SmartOS (the one Joyent works on), and OmniOS. - Oracle continues work on Solaris, releasing "Solaris 11 Express". I've no real numbers of usage of illumos vs Solaris 11 vs others. Cheers, Trent p.s. I hear that Jesus is also in contact with some of the illumos-devs on IRC (and perhaps email). I hope we can help there. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 3.6.0 Beta Phase Development
The issue tracker is your friend! On Mon, Sep 26, 2016 at 5:25 AM, Michael Feltwrote: > > On 13/09/2016 02:15, Ned Deily wrote: >> >> the challenge is to put the finishing touches on the features and >> documentation, squash bugs, and test test test. The next preview release >> will be 3.6.0b2 > > > Found one typo in Modules/_io/_iomodule.h on line 156 - #endif^L rather than > #endif (posted as an issue, but I suppose just a note here would have been > enough) > > I have a longish list of messages to stderr from the compiler (IBM xlc) on > AIX. Rather than spam everyone with those - would opening an issue be the > way forward, or just sending the file to a person - rather than the list. > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 3.6.0 Beta Phase Development
On 13/09/2016 02:15, Ned Deily wrote: the challenge is to put the finishing touches on the features and documentation, squash bugs, and test test test. The next preview release will be 3.6.0b2 Found one typo in Modules/_io/_iomodule.h on line 156 - #endif^L rather than #endif (posted as an issue, but I suppose just a note here would have been enough) I have a longish list of messages to stderr from the compiler (IBM xlc) on AIX. Rather than spam everyone with those - would opening an issue be the way forward, or just sending the file to a person - rather than the list. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] TextIO seek and tell cookies
On Mon, Sep 26, 2016, at 05:30, Ben Leslie wrote: > I think the case of JSON or SQL database is even more important though. > > tell/seek can return 129-bit integers (maybe even more? my maths might > be off here). > > The very large integers that can be returned by tell() will break > serialization to JSON, and storing in a SQL database (at least for > most database types). > > What is the value of comparing these to plain integers? Unless you > happen to know the magic encoding it isn't going to be very useful I > think? I assume the value is that in the circumstances in which all of the flags and other bits are zero, they can be used as offsets in precisely the way that you used them. It may also be possible that in some cases where they are not zero, doing arithmetic with them is still "safe" since the real offset is still in the low-order bits. I don't know if those circumstances are predictable enough for it to be worthwhile. Changing it would obviously break code that does this (unless, perhaps, it were changed to be a class with arithmetic operators), the question is whether such code "deserves" to be broken. In my own tests, even a UTF-8-sig file with DOS line endings "worked". Does anyone have information about what circumstances can reliably cause tell() to return values that are *not* simple integers? Maybe it has something to do with working with stateful encodings like iso-2022 or UTF-7? What was the situation that caused your problem? ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] TextIO seek and tell cookies
It was pointed out in private email that technically JSON can represent very large integers even if ECMAScript itself can't. But the idea of transmitting these offsets outside of a running process is not something that I had anticipated. It got me thinking: is there a guarantee that these opaque values returned from tell() is stable across different versions of Python? My reading of opaque is that it could be subject to change, but that possibly isn't the intent. It seems that since the sizeof(int) and sizeof(Py_off_t) could be different in different builds of Python even off the same version, then the opaque value returned is necessarily going to be different between builds of even the same version of Python. It seems like it would be prudent to discourage the sharing of these opaque cookies (such as via a database or interchange formats) as you'd have to be very sure that they would be interpreted correctly in any receiving instance. Cheers, Ben On 26 September 2016 at 02:30, Ben Lesliewrote: > I think the case of JSON or SQL database is even more important though. > > tell/seek can return 129-bit integers (maybe even more? my maths might > be off here). > > The very large integers that can be returned by tell() will break > serialization to JSON, and storing in a SQL database (at least for > most database types). > > What is the value of comparing these to plain integers? Unless you > happen to know the magic encoding it isn't going to be very useful I > think? > > Cheers, > > Ben > > On 25 September 2016 at 21:18, Guido van Rossum wrote: >> Be careful though, comparing these to plain integers should probably >> be allowed, and we also should make sure that things like >> serialization via JSON or storing in an SQL database don't break. I >> personally think it's one of those "learn not to touch the stove" >> cases and there's limited value in making this API idiot proof. >> >> On Sun, Sep 25, 2016 at 9:05 PM, Nick Coghlan wrote: >>> On 26 September 2016 at 10:21, MRAB wrote: On 2016-09-26 00:21, Ben Leslie wrote: > Are there any downsides to this? I've made some progress developing a > patch to change this functionality. Is it worth polishing and > submitting? > An alternative might be a subclass of int. >>> >>> It could make sense to use a subclass of int that emitted deprecation >>> warnings for integer arithmetic, and then eventually disallowed it >>> entirely. >>> >>> Cheers, >>> Nick. >>> >>> -- >>> Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia >>> ___ >>> Python-Dev mailing list >>> Python-Dev@python.org >>> https://mail.python.org/mailman/listinfo/python-dev >>> Unsubscribe: >>> https://mail.python.org/mailman/options/python-dev/guido%40python.org >> >> >> >> -- >> --Guido van Rossum (python.org/~guido) >> ___ >> Python-Dev mailing list >> Python-Dev@python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/benno%40benno.id.au ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] TextIO seek and tell cookies
I think the case of JSON or SQL database is even more important though. tell/seek can return 129-bit integers (maybe even more? my maths might be off here). The very large integers that can be returned by tell() will break serialization to JSON, and storing in a SQL database (at least for most database types). What is the value of comparing these to plain integers? Unless you happen to know the magic encoding it isn't going to be very useful I think? Cheers, Ben On 25 September 2016 at 21:18, Guido van Rossumwrote: > Be careful though, comparing these to plain integers should probably > be allowed, and we also should make sure that things like > serialization via JSON or storing in an SQL database don't break. I > personally think it's one of those "learn not to touch the stove" > cases and there's limited value in making this API idiot proof. > > On Sun, Sep 25, 2016 at 9:05 PM, Nick Coghlan wrote: >> On 26 September 2016 at 10:21, MRAB wrote: >>> On 2016-09-26 00:21, Ben Leslie wrote: Are there any downsides to this? I've made some progress developing a patch to change this functionality. Is it worth polishing and submitting? >>> An alternative might be a subclass of int. >> >> It could make sense to use a subclass of int that emitted deprecation >> warnings for integer arithmetic, and then eventually disallowed it >> entirely. >> >> Cheers, >> Nick. >> >> -- >> Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia >> ___ >> Python-Dev mailing list >> Python-Dev@python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > -- > --Guido van Rossum (python.org/~guido) > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/benno%40benno.id.au ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] TextIO seek and tell cookies
On 25 September 2016 at 17:21, MRABwrote: > On 2016-09-26 00:21, Ben Leslie wrote: >> >> Hi all, >> >> I recently shot myself in the foot by assuming that TextIO.tell >> returned integers rather than opaque cookies. Specifically I was >> adding an offset to the value returned by TextIO.tell. In retrospect >> this doesn't make sense/ >> >> Now, I don't want to drive change simply because I failed to read the >> documentation carefully, but I think the current API is very easy to >> misuse. Most of the time TextIO.tell returns a cookie that is actually >> an integer and adding an offset to it and seek-ing works fine. >> >> The only indication you get that you are mis-using the API is that >> sometimes tell returns a cookie that when you add an integer offset to >> it will cause seek() to fail with an OverflowError. >> >> Would it be possible to change the API to return something more >> opaque? E.g.: rather than converting the C cookie structure to a long, >> could it instead be converted to a bytes() object. >> >> (I.e.: Change textiowrapper_build_cookie to use >> PyBytes_FromStringAndSize rather than _PyLong_FromByteArray and >> equivalent for textiowrapper_parse_cookie). >> >> This would ensure the return value is never mis-used and is probably >> also faster using bytes objects than converting to/from an integer. >> > why would it be faster? It's an integer internally. It isn't an integer internally though, it is a cookie: typedef struct { Py_off_t start_pos; int dec_flags; int bytes_to_feed; int chars_to_skip; char need_eof; } cookie_type; The memory view of this structure is then converted to a long. Surely converting to a PyLong is more work than converting to bytes? In any case, performance really isn't the motivation here. Cheers, Ben ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com