Meador Inge <mead...@gmail.com> added the comment:
Hi All,
On Sat, Feb 13, 2010 at 5:07 AM, Mark Dickinson <rep...@bugs.python.org>wrote:
>
> Mark Dickinson <dicki...@gmail.com> added the comment:
>
> Some of the proposed struct module additions look far from straightforward;
> I find that section of the PEP significantly lacking in details and
> motivation.
>
I agree.
> "Unpacking a long-double will return a decimal object or a ctypes
> long-double."
>
> Returning a Decimal object here doesn't make a lot of sense, since Decimal
> objects aren't generally compatible with floats. And ctypes long double
> objects don't seem to exist, as far as I can tell. It might be better not
> to add this code.
And under what conditions would a ctype long double be used vs. a Decimal
object.
>
Another bit that's not clear to me: how is unpacking an object pointer
> expected to work, and how would it typically be used? What if the unpacked
> pointer no longer points to a valid Python object? How would this work in
> other Python implementations?
>
I guess if an object associated with the packed address does not exist, then
you would unpack None (?). This is especially a problem if the struct-sting
is being sent over the wire to another machine.
> For the 'X{}' format (pointer to a function), is this supposed to mean a
> Python function or a C function?
>
I read that as a Python function. However, I am not completely sure how the
prototype would be enforced when unpacking. I am also wondering, though,
how the signatures on pointers-to-functions are specified? Are
the arguments and return type full struct strings as well?
> What's a 'specific pointer'?
I think this means a pointer to a specific type, e.g. '&d' is a pointer to a
double. If this is the case, though, the use cases are not completely clear
to me.
I also have the following questions:
* Can pointers be nested, '&&d' ?
* What nesting level can structures have? Arbitrary?
* The new array syntax claims "multi-dimensional array of whatever follows".
Truly whatever? Arrays of structures? Arrays of pointers?
* "complex (whatever the next specifier is)". Not really 'whatever'. You
can not have a 'complex bool' or 'complex int'. What other types of
complex are there besides complex double?
* How do array specifiers and pointer specifiers mix? For example, would
'(2, 2)&d' be a two-by-two array of pointers to doubles? What about
'&(2, 2)d'? Is this a pointer to an two-by-two array of doubles?
The new features of the struct-string syntax are so different that I think
we
need to specify a grammar. I think it will clarify some of the open
questions.
In addition, I was thinking that a reasonable implemenation strategy would
be to keep the current struct-string syntax mostly in place within the C
module
implementation. The C implementation would just provide an interface to
pack\unpack sequences of primitive data elements. Then we could write a
layer in the Python 'struct' module that took care of the higher-order
concepts like nested structures, arrays, named values, and pointers to
functions. The higher-order concepts would be mapped to the appropriate
primitive sequence strings.
I think this will simplify the implementation and will provide a way to
phase
it. We can implement the primitive type extensions in C first followed by
the higher-level Python stuff. The result of each phase is immediately
usuable.
I have attached a patch against the PEP containing my current thoughts on
fleshing out the grammar and some of the current open questions. This still
needs work, but I wanted to share to see if I am on the right track.
Please advise on how to proceed.
----------
keywords: +patch
Added file: http://bugs.python.org/file16241/unnamed
Added file: http://bugs.python.org/file16242/pep-3118.patch
_______________________________________
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue3132>
_______________________________________<div>Hi All,</div><div><br></div>On Sat, Feb 13, 2010 at 5:07 AM, Mark
Dickinson <span dir="ltr"><<a
href="mailto:rep...@bugs.python.org";>rep...@bugs.python.org</a>></span>
wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<br>
Mark Dickinson <<a
href="mailto:dicki...@gmail.com";>dicki...@gmail.com</a>> added the
comment:<br>
<br>
Some of the proposed struct module additions look far from straightforward; Â I
find that section of the PEP significantly lacking in details and
motivation.<br></blockquote><div>Â </div><div>I
agree.</div><div>Â </div><blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex;">
"Unpacking a long-double will return a decimal object or a ctypes
long-double."<br>
<br>
Returning a Decimal object here doesn't make a lot of sense, since Decimal
objects aren't generally compatible with floats. Â And ctypes long double
objects don't seem to exist, as far as I can tell. Â It might be better not
to add this code.</blockquote>
<div><br></div><div>And under what conditions would a ctype long double be used
vs. a Decimal object.</div><div>Â </div><blockquote class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
 </blockquote><blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex;">
Another bit that's not clear to me: Â how is unpacking an object pointer
expected to work, and how would it typically be used? Â What if the unpacked
pointer no longer points to a valid Python object? Â How would this work in
other Python implementations?<br>
</blockquote><div><br></div><div>I guess if an object associated with the
packed address does not exist, then you would unpack None (?). Â This is
especially a problem if the struct-sting is being sent over the wire to another
machine.</div>
<div>Â </div><blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex;">
For the 'X{}' format (pointer to a function), is this supposed to mean
a Python function or a C function?<br></blockquote><div><br></div><div>I read
that as a Python function. Â However, I am not completely sure how the
prototype would be enforced when unpacking. Â I am also wondering, though, how
the signatures on pointers-to-functions are specified?  Are the arguments and
return type full struct strings as well?</div>
<div>Â </div><blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex;">
What's a 'specific pointer'?</blockquote><div><br></div><div>I
think this means a pointer to a specific type, e.g. '&d' is a
pointer to a double. If this is the case, though, the use cases are not
completely clear to me.</div>
</div><br>I also have the following questions:
<div><br></div><div><div>* Can pointers be nested, '&&d'
?</div><div>* What nesting level can structures have? Arbitrary?</div><div>*
The new array syntax claims "multi-dimensional array of whatever
follows". Â </div>
<div>Â Â Truly whatever? Arrays of structures? Arrays of pointers?</div><div>*
"complex (whatever the next specifier is)". Â Not really
'whatever'. Â You</div><div>Â Â can not have a 'complex bool'
or 'complex int'. Â What other types of</div>
<div>Â Â complex are there besides complex double?</div><div>* How do array
specifiers and pointer specifiers mix? Â For example,
would</div><div>Â Â '(2, 2)&d' be a two-by-two array of pointers to
doubles? Â What about</div>
<div>Â Â '&(2, 2)d'? Â Is this a pointer to an two-by-two array of
doubles?</div><div><br></div><div>The new features of the struct-string syntax
are so different that I think we </div><div>need to specify a grammar.  I
think it will clarify some of the open</div>
<div>questions. Â </div><div><br></div><div>In addition, I was thinking that a
reasonable implemenation strategy would</div><div>be to keep the current
struct-string syntax mostly in place within the C
module </div><div>implementation.  The C implementation would just provide an
interface to </div>
<div>pack\unpack sequences of primitive data elements. Â Then we could write
a </div><div>layer in the Python 'struct' module that took care of the
higher-order </div><div>concepts like nested structures, arrays, named values,
and pointers to</div>
<div>functions. Â The higher-order concepts would be mapped to the
appropriate</div><div>primitive sequence strings.</div><div><br></div><div>I
think this will simplify the implementation and will provide a way to
phase</div>
<div>it. Â We can implement the primitive type extensions in C first followed
by</div><div>the higher-level Python stuff. Â The result of each phase is
immediately usuable.</div><div><br></div><div>I have attached a patch against
the PEP containing my current thoughts on</div>
<div>fleshing out the grammar and some of the current open questions. Â This
still needs work, but I wanted to share to see if I am on the right track.
 Please advise on how to proceed.</div></div>
Index: pep-3118.txt
===================================================================
--- pep-3118.txt (revision 78193)
+++ pep-3118.txt (working copy)
@@ -637,9 +637,54 @@
ctypes and NumPy for example). The Python 2.5 specification is
at http://docs.python.org/library/struct.html.
-Here are the proposed additions:
+The grammar of the original struct string-syntax is very simple. It consist of
+a byte-order specifier followed by a character code which may optionally be
+preceded by a numeric count::
+ struct-string ::= <byte-order-specifier> (<count> <code>)*
+ byte-order-specifier ::= '!' | '@' | '=' | '>' | '<' | '^'
+ count ::= <integer>
+ code ::= 'x' | 'c' | 'b' | 'B' | '?' | 'h' | 'H' | 'i' | 'I' | 'l' | 'L'
+ | 'q' | 'Q' | 'f' | 'd' | 's' | 'p' | 'P'
+The proposed additions will extend this syntax by adding support for new
+primitive types, name specifiers, detailed structure layout, and
+function-pointers. The high-level structure of the new syntax is::
+
+ struct-string ::= (<type-string> <name-specifier>?)*
+ type-string ::= <primitive>
+ | <structure>
+ | <pointer-to-function>
+
+The struct module will be changed to understand this new syntax and
+return appropriate Python objects on unpacking. White-space in the
+struct-string syntax will be ignored, if it isn't already. The details
+of the grammar elements *primitive*, *name-specifier*, *structure*,
+and *pointer-to-function* are described in the following sub-sections.
+
+In addition to implementing this new syntax, functions should be added to
+ctypes to create a ctypes object from a struct description, and add
+long-double, and ucs-2 to ctypes.
+
+Primitive Extensions
+---------------------
+
+The extenstions to the primitive data types will provide new type codes,
+the ability to change byte-order mid-stream, and a way to specify pointers
+to data::
+
+ primitive ::= <byte-order-specifier> <count>? <array-specifier>?
<pointer-specifier>+ <code>
+ byte-order-specifier ::= '!' | '@' | '=' | '>' | '<' | '^'
+ count ::= <integer>
+ pointer-specifier ::= '&'
+ array-specifier ::= '(' <integer> (',' <integer>)* ')'
+ code ::= 'x' | 'c' | 'b' | 'B' | '?' | 'h' | 'H' | 'i' | 'I' | 'l' | 'L'
+ | 'q' | 'Q' | 'f' | 'd' | 's' | 'p' | 'P' | 't' | 'g' | 'u' | 'w'
+ | 'O' | 'Z'
+
+The new *code* types and modifications to old *code* types are described as
+follows:
+
================ ===========
Character Description
================ ===========
@@ -650,45 +695,75 @@
'u' ucs-2
'w' ucs-4
'O' pointer to Python Object
-'Z' complex (whatever the next specifier is)
-'&' specific pointer (prefix before another character)
-'T{}' structure (detailed layout inside {})
-'(k1,k2,...,kn)' multi-dimensional array of whatever follows
-':name:' optional name of the preceeding element
-'X{}' pointer to a function (optional function
- signature inside {} with any return value
- preceeded by -> and placed at the end)
+'Z' complex
================ ===========
-The struct module will be changed to understand these as well and
-return appropriate Python objects on unpacking. Unpacking a
-long-double will return a decimal object or a ctypes long-double.
-Unpacking 'u' or 'w' will return Python unicode. Unpacking a
-multi-dimensional array will return a list (of lists if >1d).
-Unpacking a pointer will return a ctypes pointer object. Unpacking a
-function pointer will return a ctypes call-object (perhaps). Unpacking
-a bit will return a Python Bool. White-space in the struct-string
-syntax will be ignored if it isn't already. Unpacking a named-object
-will return some kind of named-tuple-like object that acts like a
-tuple but whose entries can also be accessed by name. Unpacking a
-nested structure will return a nested tuple.
+The *pointer-specifier* is used to denote a pointer to a specified type.
+For example, '&g' represents a pointer to a long-double and '&&I' respresents
+a pointer to a pointer to an unsigned integer.
-Endian-specification ('!', '@','=','>','<', '^') is also allowed
-inside the string so that it can change if needed. The
-previously-specified endian string is in force until changed. The
-default endian is '@' which means native data-types and alignment. If
-un-aligned, native data-types are requested, then the endian
+The *array-specifier* is used to denote homogenous multi-dimensional areas
+of memory. The type-code immediately following the array-specifier denotes
+the type of the elements in that memory area. This extension also allows
+specifying if the data is supposed to be viewed as a (C-style contiguous,
+last-dimension varies the fastest) multi-dimensional array of a particular
+format.
+
+The *byte-order-specifier* is now allowed inside the string so that it can
+change if needed. The previously specified byte-order specifier is in force
+until changed. The default byte-order is '@' which means native data-types
and
+alignment. If un-aligned, native data-types are requested, then the byte-order
specification is '^'.
-According to the struct-module, a number can preceed a character
-code to specify how many of that type there are. The
-``(k1,k2,...,kn)`` extension also allows specifying if the data is
-supposed to be viewed as a (C-style contiguous, last-dimension
-varies the fastest) multi-dimensional array of a particular format.
+The following unpacking requirements must be met:
-Functions should be added to ctypes to create a ctypes object from
-a struct description, and add long-double, and ucs-2 to ctypes.
+* a long-double will return a decimal object or a ctypes long-double.
+* 'u' or 'w' will return Python unicode.
+* a multi-dimensional array will return a list (of lists if >1d).
+* a pointer will return a ctypes pointer object.
+* a bit will return a Python Bool.
+* a pointer to a Python Object will return the Python Object associated with
+ the address encoded in the struct-string if a Python Object of that address
+ exist. Otherwise, None is returned.
+Named Specifier
+---------------
+The name specifier is used to associate names with values in the
struct-string::
+
+ name-specifier := ':' <string> ':'
+
+The following unpacking requirements must be met:
+
+* a named-object will return some kind of named-tuple-like object
+ that acts like a tuple but whose entries can also be accessed by name.
+
+
+Structure Layout
+----------------
+The new structure layout syntax provides a way to specify a detailed
+structure layout::
+
+ structure := 'T' '{' <struct-string> '}'
+
+Structures may be arbitrarily nested.
+
+The following unpacking requirements must be met:
+
+* Unpacking a nested structure will return a nested tuple.
+
+
+Pointer to Functions
+--------------------
+The pointer to a function syntax provides a way to specify ...::
+
+ pointer-to-function := 'X' '{' <signature>? '}
+ signature := <struct-string> '->' <string-string>
+
+The following unpacking requirements must be met:
+
+* a function pointer will return a ctypes call-object (perhaps).
+
+
Examples of Data-Format Descriptions
====================================
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
