[issue3132] implement PEP 3118 struct changes

Changes by Stefan Krah : -- versions: +Python 3.6 -Python 3.3 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: ht

[issue3132] implement PEP 3118 struct changes

Stefan Krah added the comment: Here's a grammar that roughly describes the subset that NumPy supports. As for implementing this in the struct module: There is a new data description language on the horizon: http://datashape.readthedocs.org/en/latest/ It does not have all the low-level capab

[issue3132] implement PEP 3118 struct changes

Changes by Stefan Krah : -- nosy: -skrah ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.or

[issue3132] implement PEP 3118 struct changes

Paul Hoffman added the comment: Whoops, never mind. Thanks for the pointer to 11734. -- ___ Python tracker ___ ___ Python-bugs-list mai

[issue3132] implement PEP 3118 struct changes

Mark Dickinson added the comment: Paul: there's already an open issue for adding float16 to the struct module: see issue 11734. -- ___ Python tracker ___

[issue3132] implement PEP 3118 struct changes

Paul Hoffman added the comment: Following up on http://mail.python.org/pipermail/python-ideas/2011-March/009656.html, I would like to request that struct also handle half-precision floats directly. It's a short change, and half-precision floats are becoming much more popular in applications.

[issue3132] implement PEP 3118 struct changes

Changes by Martin Panter : -- nosy: +vadmium ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.

[issue3132] implement PEP 3118 struct changes

Changes by Alexander Belopolsky : -- nosy: +belopolsky -Alexander.Belopolsky ___ Python tracker ___ ___ Python-bugs-list mailing list U

[issue3132] implement PEP 3118 struct changes

Nick Coghlan added the comment: Following up here after rejecting #15622 as invalid The "unicode" codes in PEP 3118 need to be seriously rethought before any related changes are made in the struct module. 1. The 'c' and 's' codes are currently used for raw bytes data (represented as bytes obj

[issue3132] implement PEP 3118 struct changes

Changes by Arfrever Frehtes Taifersar Arahesis : -- nosy: +Arfrever ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscrib

[issue3132] implement PEP 3118 struct changes

Stefan Krah added the comment: Yes, definitely. I'm going to push a new memoryview implementation (complete for all 1D/native format cases) in a couple of days. Once that is done, perhaps we could create a memoryview-struct branch on top of that. -- nosy: +skrah __

[issue3132] implement PEP 3118 struct changes

Meador Inge added the comment: Is this work something that might be suitable for the features/pep-3118 repo (http://hg.python.org/features/pep-3118/) ? -- ___ Python tracker ___

[issue3132] implement PEP 3118 struct changes

Mark Dickinson added the comment: I'm going to unassign for now; I still hope to look at this at some point, but can't see a time in the near future when it's going to happen. -- assignee: mark.dickinson -> ___ Python tracker

[issue3132] implement PEP 3118 struct changes

Mark Dickinson added the comment: Yes, there's interest (at least here). I've just been really short on Python-time recently, so haven't found time to review your patch. -- ___ Python tracker

[issue3132] implement PEP 3118 struct changes

Meador Inge added the comment: Is there still any interest in this work? -- ___ Python tracker ___ ___ Python-bugs-list mailing list U

[issue3132] implement PEP 3118 struct changes

Changes by Antoine Pitrou : -- assignee: meador.inge -> mark.dickinson stage: needs patch -> patch review versions: +Python 3.3 -Python 3.2 ___ Python tracker ___

[issue3132] implement PEP 3118 struct changes

Meador Inge added the comment: Attached is the latest version of the struct string patch. I tested on OS X 10.6.5 (64-bit) and Ubuntu 10.04 (32-bit). I also scanned for memory problems with Valgrind. There is one test failing on 32-bit systems ('test_crasher'). This is due to the fact tha

[issue3132] implement PEP 3118 struct changes

Mark Dickinson added the comment: > Another possibility is to implement the 'O' format unsafely [...] Hmm. I don't much like that idea. Historically, it's supposed to be very difficult to segfault the Python interpreter with pure Python code (well except if you're using ctypes, I guess). -

[issue3132] implement PEP 3118 struct changes

Pauli Virtanen added the comment: > I still like the idea of scoped endianness markers in the substructs, > but if we have to abandon that for compatibility with NumPy that's > okay. That, or change the Numpy implementation. I don't believe there's yet much code in the wild that changes the a

[issue3132] implement PEP 3118 struct changes

Mark Dickinson added the comment: > For reference, Numpy's PEP 3118 implementation is here BTW, does this already exist in a released version of NumPy? If not, when is it likely to appear in the wild? -- ___ Python tracker

[issue3132] implement PEP 3118 struct changes

Mark Dickinson added the comment: > For reference, Numpy's PEP 3118 implementation is here: Thanks for that, and the other information you give; that's helpful. It sounds like we're on the same page with respect to alignment of substructs. (Bar the mostly academic question of max versus lcm

[issue3132] implement PEP 3118 struct changes

Pauli Virtanen added the comment: For reference, Numpy's PEP 3118 implementation is here: http://github.com/numpy/numpy/blob/master/numpy/core/_internal.py#L357 http://github.com/numpy/numpy/blob/master/numpy/core/src/multiarray/buffer.c#L76 It would be a good idea to ensure that the numpy an

[issue3132] implement PEP 3118 struct changes

Changes by Meador Inge : -- assignee: teoliphant -> meador.inge priority: critical -> high stage: unit test needed -> needs patch ___ Python tracker ___ __

[issue3132] implement PEP 3118 struct changes

Travis Oliphant added the comment: On May 19, 2010, at 2:38 PM, Mark Dickinson wrote: > > Mark Dickinson added the comment: > > Travis, this issue is still assigned to you. Do you plan to work on this at > some stage, or may I unassign you? > You may unassign it from me. Unfortunately,

[issue3132] implement PEP 3118 struct changes

Mark Dickinson added the comment: One more reference: http://msdn.microsoft.com/en-us/library/9dbwhz68(v=VS.80).aspx gives essentially the same rules for MSVC. "The alignment of the beginning of a structure or a union is the maximum alignment of any individual member. Each member within the

[issue3132] implement PEP 3118 struct changes

Mark Dickinson added the comment: Another snippet, from the latest public draft of the System V x86-64 ABI [1]: """Structures and unions assume the alignment of their most strictly aligned compo- nent. Each member is assigned to the lowest available offset with the appropriate alignment. The

[issue3132] implement PEP 3118 struct changes

Mark Dickinson added the comment: > The C99 standard does not specify what the behavior should be. Right; it's down to the platform ABI. I think the least common multiple of the alignment requirements of the struct members is the way to go, though. It's difficult to imagine an ABI for which

[issue3132] implement PEP 3118 struct changes

Meador Inge added the comment: > Granted, yes. But I wouldn't expect the same padding for 'BT{BI}' and > 'BBI'. 'BT{BI}' should match a C struct which itself has an embedded > struct. For C, I get the following results on my machine: I wasn't sure. The C99 standard does not specify what t

[issue3132] implement PEP 3118 struct changes

Mark Dickinson added the comment: (whoops -- used unsigned struct codes to correspond to signed C types there; but it shouldn't make any difference to the sizes and padding). > I am tempted to suggest that we just go back to having one specifier at the > beginning of the string :). Things s

[issue3132] implement PEP 3118 struct changes

Mark Dickinson added the comment: > For example, I would expect the same padding for 'BT{I}' and 'BI'. Granted, yes. But I wouldn't expect the same padding for 'BT{BI}' and 'BBI'. 'BT{BI}' should match a C struct which itself has an embedded struct. For C, I get the following results on my

[issue3132] implement PEP 3118 struct changes

Meador Inge added the comment: > As a separate issue, I notice that the new 'T{}' code doesn't respect > multiplicities, e.g., as in 'H3T{HHL}'. Is that > intentional/desirable? That could have been an oversight on my part. I don't see any immediate reason why we wouldn't allow it. > But

[issue3132] implement PEP 3118 struct changes

Mark Dickinson added the comment: After a bit more thought (and after soliciting a couple of opinions on #python-dev), I'm convinced that endianness changes within a substruct should be local to that substruct: - it makes the meaning of '>2T{HT{HH __

[issue3132] implement PEP 3118 struct changes

Mark Dickinson added the comment: Thanks for the new patch. > "... If un-aligned, native data-types are requested, then the > endian specification is '^'." > > However, I am not quite sure how to interpret the last sentence. Hmm. Seems like the PEP authors are proposing a new byteorder/alignm

[issue3132] implement PEP 3118 struct changes

Antoine Pitrou added the comment: > > The possibility of mixing native size/alignment with standard > > size/alignment in a single format string makes me a bit uneasy > > I agree. It is hard for me to see how this might be used. Without having anything more constructive to add, I also agree

[issue3132] implement PEP 3118 struct changes

Meador Inge added the comment: > is that correct, or should the production list be something like: Yup, you are right. I will change the grammar. > Whether these cases are valid or not (personally, I think they should > be), we should add some tests for them. '<' *is* currently valid, I >

[issue3132] implement PEP 3118 struct changes

Mark Dickinson added the comment: One more design question. For something like: 'H}H', what endianness should be used when packing/unpacking the last 'H'? Should the switch to '>' within the embedded struct be regarded as local to the struct? With your patch, I get: >>> pack('H}H', 1, (2,)

[issue3132] implement PEP 3118 struct changes

Mark Dickinson added the comment: Ah, it should have been: assert(soself->s_tree != NULL); Got it now. All tests pass. :) -- ___ Python tracker ___ ___

[issue3132] implement PEP 3118 struct changes

Mark Dickinson added the comment: Hmm. Something's not quite right: the _struct module fails to compile for me with this patch. I get: /Users/dickinsm/python/svn/py3k/Modules/_struct.c: In function ‘s_unpack’: /Users/dickinsm/python/svn/py3k/Modules/_struct.c:1730: error: ‘PyStructObject’ h

[issue3132] implement PEP 3118 struct changes

Mark Dickinson added the comment: Travis, this issue is still assigned to you. Do you plan to work on this at some stage, or may I unassign you? -- ___ Python tracker ___ _

[issue3132] implement PEP 3118 struct changes

Mark Dickinson added the comment: Thanks for the Rietveld upload. I haven't had a chance to review this properly yet, but hope to do so within the next few days. One question: the production list you added to the docs says: format_string: (`byte_order_specifier`? `type_string`)* This sugg

[issue3132] implement PEP 3118 struct changes

Meador Inge added the comment: Sure - http://codereview.appspot.com/1258041 -- ___ Python tracker ___ ___ Python-bugs-list mailing lis

[issue3132] implement PEP 3118 struct changes

Mark Dickinson added the comment: Thanks for this. Any chance you could upload the patch to Rietveld (http://codereview.appspot.com/) for ease of review? -- ___ Python tracker

[issue3132] implement PEP 3118 struct changes

Meador Inge added the comment: Attached is a patch that implements part of the additions. More specifically, the 'T{}' syntax and the ability to place byte-order specifiers ('<', '>', '@', '^', '!", '=') anywhere in the struct string. The changes dictated by the PEP are so big that it is b

[issue3132] implement PEP 3118 struct changes

Changes by Noufal : -- nosy: +noufal ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mail

[issue3132] implement PEP 3118 struct changes

Mark Dickinson added the comment: > One benefit of having a type code for 'long double' (assuming you are > mapping the value to the platform's 'long double') is that the you > don't have to know how many bytes are in the underlying representation. Agreed: it's nice to have struct.pack already

[issue3132] implement PEP 3118 struct changes

Meador Inge added the comment: > The main thing that I realized from this is that unpacking as a ctypes long > double isn't all that useful for someone who wants to be able to do arithmetic > on the unpacked result. I agree. Especially since ctypes 'long double' maps to a Python float and '

[issue3132] implement PEP 3118 struct changes

Mark Dickinson added the comment: About long doubles again: I just encountered someone on the #python IRC channel who wanted to know whether struct.pack and struct.unpack supported reading and writing of x87 80-bit long doubles (padded to 12 bytes each in the input). A few quotes from him/h

[issue3132] implement PEP 3118 struct changes

Mark Dickinson added the comment: [Meador Inge] > So the next step is to kick off a thread on python-dev summarizing the > questions\problems we have come up with? I can get that started. Sounds good. I'd really like to see some examples of how these struct-module additions would be used in

[issue3132] implement PEP 3118 struct changes

Meador Inge added the comment: Mark, > I think a lot of this discussion needs to go back to python-dev; with > luck, we can get some advice and clarifications from the PEP authors > there. So the next step is to kick off a thread on python-dev summarizing the questions\problems we have com

[issue3132] implement PEP 3118 struct changes

Mark Dickinson added the comment: I'm looking for previous discussions of this PEP. There's a python-dev thread in April 2007: http://mail.python.org/pipermail/python-dev/2007-April/072537.html Are there other discussions that I'm missing? -- ___

[issue3132] implement PEP 3118 struct changes

Mark Dickinson added the comment: > And under what conditions would a ctype long double be used vs. a > Decimal object. Well, I'm guessing that this was really just an open question for the PEP, and that the PEP authors hadn't decided which of these two options was more appropriate. If all l

[issue3132] implement PEP 3118 struct changes

Meador Inge added the comment: Hi All, On Sat, Feb 13, 2010 at 5:07 AM, Mark Dickinson wrote: > > Mark Dickinson added the comment: > > Some of the proposed struct module additions look far from straightforward; > I find that section of the PEP significantly lacking in details and > motivati

[issue3132] implement PEP 3118 struct changes

Mark Dickinson added the comment: Whoops. ctypes does have long double, of course. Apologies. -- ___ Python tracker ___ ___ Python-b

[issue3132] implement PEP 3118 struct changes

Mark Dickinson added the comment: Some of the proposed struct module additions look far from straightforward; I find that section of the PEP significantly lacking in details and motivation. "Unpacking a long-double will return a decimal object or a ctypes long-double." Returning a Decimal ob

[issue3132] implement PEP 3118 struct changes

Travis Oliphant added the comment: On Feb 12, 2010, at 7:29 PM, Meador Inge wrote: > > Meador Inge added the comment: > > Is anyone working on implementing these new struct modifiers? If > not, then I would love to take a shot at it. That would be great. -Travis -- _

[issue3132] implement PEP 3118 struct changes

Benjamin Peterson added the comment: 2010/2/12 Meador Inge : > > Meador Inge added the comment: > > Is anyone working on implementing these new struct modifiers?  If not, then I > would love to take a shot at it. Not to my knowledge. -- ___ Python

[issue3132] implement PEP 3118 struct changes

Meador Inge added the comment: Is anyone working on implementing these new struct modifiers? If not, then I would love to take a shot at it. -- nosy: +minge ___ Python tracker

[issue3132] implement PEP 3118 struct changes

Changes by Mark Dickinson : -- nosy: +mark.dickinson ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail

[issue3132] implement PEP 3118 struct changes

Daniel Diniz added the comment: Travis, Do you think you can contribute for this to actually land in 3.2? Having a critical issue slipping from 3.0 to 3.3 would be bad... Does this supersede issue 2395 or is this a subset of that one.? -- nosy: +ajaksu2 stage: -> test needed versions:

[issue3132] implement PEP 3118 struct changes

Travis Oliphant <[EMAIL PROTECTED]> added the comment: This can be re-targeted to 3.1 as described. ___ Python tracker <[EMAIL PROTECTED]> ___ __

[issue3132] implement PEP 3118 struct changes

Antoine Pitrou <[EMAIL PROTECTED]> added the comment: > Actually, this may be a requirement of #2394; PEP 3118 states that > memoryview.tolist would use the struct module to do the unpacking. :-( However, we don't have any examples of the buffer API / memoryview object working with something els

[issue3132] implement PEP 3118 struct changes

Benjamin Peterson <[EMAIL PROTECTED]> added the comment: Actually, this may be a requirement of #2394; PEP 3118 states that memoryview.tolist would use the struct module to do the unpacking. ___ Python tracker <[EMAIL PROTECTED]>

[issue3132] implement PEP 3118 struct changes

Antoine Pitrou <[EMAIL PROTECTED]> added the comment: Let's retarget it to 3.1 then. It's a new feature, not a behaviour change or a deprecation, so adding it to 3.0 isn't a necessity. -- components: +Library (Lib) nosy: +pitrou priority: release blocker -> critical versions: +Python 3.1

[issue3132] implement PEP 3118 struct changes

Barry A. Warsaw <[EMAIL PROTECTED]> added the comment: It's looking pessimistic that this is going to make it by beta 3. If they can't get in by then, it's too late. -- nosy: +barry ___ Python tracker <[EMAIL PROTECTED]>

[issue3132] implement PEP 3118 struct changes

Changes by Benjamin Peterson <[EMAIL PROTECTED]>: -- priority: critical -> release blocker ___ Python tracker <[EMAIL PROTECTED]> ___ __

[issue3132] implement PEP 3118 struct changes

Jean Brouwers <[EMAIL PROTECTED]> added the comment: If the struct changes are made, add also 2 formats for C types ssize_t and size_t, perhaps 'z' resp. 'Z'. In particular since on platforms sizeof(size_t) != sizeof(long). -- nosy: +MrJean1 ___ Py

[issue3132] implement PEP 3118 struct changes

New submission from Benjamin Peterson <[EMAIL PROTECTED]>: It seems the new modifiers to the struct.unpack/pack module that were proposed in PEP 3118 haven't been implemented yet. -- assignee: teoliphant messages: 68347 nosy: benjamin.peterson, teoliphant priority: critical severity: nor