[issue12528] Implement configurable bitfield allocation strategy

[Charles Machalow]

Charles Machalow added the comment:

Was browsing and found this.

This option would be very useful as it could help avoid a current bug I've had 
to deal with : https://bugs.python.org/issue29753. My use case works with 
data/structs from another device all-together, so I can't control it's packing. 
However since GCC/Linux builds currently have the specified bug, I can't get 
around it. This option would have allowed me to just specify the MSVC strategy, 
which in general is what I want to not have to have OS-specific behavior. With 
the bug in mind, this could have almost be a bug-workaround for me.

--
nosy: +Charles Machalow

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12528] Implement configurable bitfield allocation strategy

[Meador Inge]

Meador Inge mead...@gmail.com added the comment:

As stated, how a particular compiler allocates bitfields is *extremely* 
implementation specific.  There can be differences in implementations between 
different compilers, different *versions* of the same compiler, and different 
invocations of the same compiler where the options are varied.

I am wondering whether adding this feature will open up a can of worms that we 
don't want to deal with.  There are other options beyond MSVC and GCC that seem 
reasonable.  For example, GCC packs bitfields together on structures defined 
with '__attribute__((packed))'.  Do we need a GCCPACKED option now?

Also, GCC 4.4 fixed a bug that can lead to differences in structure layout from 
previous versions.  See -Wpacked-bitfield-compat option [1].
Do we need a GCC44 option now?

Finally, structure layout is architecture specific.  GCC for x86, for example, 
has the 'ms_struct' attribute extensions for x86 [2].  Does this mean that for 
a GCC compiled Python that the MSVC option will only work for an x86 host?

My point is that there are many, many variations on how a *single* compiler can 
allocate bitfields.  So just saying GCC allocation
strategy is not adequate.

So, lets take a step back.  What exact problem is this feature trying
to solve?  Is one of the use cases that 'ctypes' for a Windows hosted Python 
built with MSVC++ can interop with C code compiled with a Windows  hosted GCC 
(e.g. cygwin)?  If so, then is that realistic?  ISTM, that there are other ABI 
differences that would prevent that from working.

It seem perfectly reasonable to me that ctypes will only interact with bits 
that were constructed with the exact same compiler (and options) as the 
interpreter itself.  If it is not already, then we should document this.

[1] http://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html
[2] http://gcc.gnu.org/onlinedocs/gcc/Variable-Attributes.html

--
nosy: +meadori

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12528
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12528] Implement configurable bitfield allocation strategy

[Vlad Riscutia]

Vlad Riscutia riscutiav...@gmail.com added the comment:

Well currently we pack bitfields with an algorithm that uses #ifdefs for GCC 
and MSVC builds. This feature tries to remove the hardcoded behavior and 
implement it as a runtime option. This should improve interop with other 
compilers. Currently I provided these for MSVC-style and GCC-style bitfield 
allocations. These, of course, can be extended with other strategies.

I am not sure that the fact that GCC has different types of bitfield 
allocations in different versions is a point against this feature. Consider 
that in our current code we don't use compiler bitfield allocation, we create 
the structure layout using our own algorithm, interop might be broken even if 
Python gets built with same version of GCC as the binary we want to interop 
with as long as algorithm is out of date. This patch should provide some 
flexibility in this matter.

Wouldn't a GCC44 constant provided at API level be better than saying you 
can't interop with anything build with GCC 4.4 and up? Or vice-versa, anything 
built with GCC  4.4.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12528
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12528] Implement configurable bitfield allocation strategy

[Terry J. Reedy]

Terry J. Reedy tjre...@udel.edu added the comment:

As a (Windows) user, I would like to be able to download any working 
pre-compiled shared library (.dll) and access it via ctypes. The particular 
compiler used to compile cpythonx.y.z should not determine whether a Pythonx.y 
program works. The use of VSC2008 is not part of the Python3.2 definition. So I 
am in favor of features than makes ctypes more likely to work.

I understand that this should be easy if the datatypes sent and received are 
standard ints, floats, and arrays thereof, since the bit patterns are then 
knowable. I gather that the problem with custom bitfields is that the resulting 
bit pattern format is not only not documented in the .dll, but is also not 
determined by the external documentation (the .h files).

Does anyone know how Cython, for instance, handles this problem? Stephen Behnel 
recommends it as an alternative to ctypes. Does it even try to deal with 
bitfields?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12528
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12528] Implement configurable bitfield allocation strategy

[Meador Inge]

Meador Inge mead...@gmail.com added the comment:

Hi Vlad,

On Thu, Sep 1, 2011 at 1:30 PM, Vlad Riscutia rep...@bugs.python.org wrote:

 Vlad Riscutia riscutiav...@gmail.com added the comment:

 Well currently we pack bitfields with an algorithm that uses #ifdefs for GCC 
 and MSVC builds. This feature tries to remove the
 hardcoded behavior and implement it as a runtime option. This should improve 
 interop with other compilers. Currently I provided these
 for MSVC-style and GCC-style bitfield allocations. These, of course, can be 
 extended with other strategies.

Yup, I understand what the feature is doing.  I just wanted to
reiterate (as I am sure you already know) that typically the rules for
implementing bit fields in a particular compiler are pretty complex.
I just wanted to make sure that the use cases are out there to justify
the complexity.  issue11920 really does seem like a use case where
someone would otherwise be stuck without a runtime configurable
allocation strategy.

BTW, out of curiosity I explored the packed case that I mentioned
earlier with GCC more (reproduction case attached):

[meadori@motherbrain ctypes]$ make clean; make
rm -f foo.o libfoo.so.1.0
gcc -Wall -fPIC  -c foo.c
gcc -shared -Wl,-soname,libfoo.so.1 -o libfoo.so.1.0 foo.o
python repro.py
In Python: (85, 85)
From C: (85, 85)
From Python: (85, 85)
[meadori@motherbrain ctypes]$ make clean; CFLAGS=-DPACK=1 make
rm -f foo.o libfoo.so.1.0
gcc -Wall -fPIC -DPACK=1 -c foo.c
gcc -shared -Wl,-soname,libfoo.so.1 -o libfoo.so.1.0 foo.o
python repro.py
In Python: (85, 85)
From C: (85, 85)
From Python: (85, 170)

This shows that there are already cases that can't be handled with
packed bit fields and GCC.  This runtime configuration feature, could
fix this case as well.  However, it is probably better to wait for a
real world use case before implementing some pathological case that I
cooked up ;-)

 Wouldn't a GCC44 constant provided at API level be better than saying you 
 can't interop with anything build with GCC 4.4 and up? Or  vice-versa, 
 anything built with GCC  4.4.

Yeah, probably so.  I think the compiler constraint I stated before is
probably a bit too strong.  I need to think about it more.  But,
again, my point is that there are a lot of different variations when
it comes to bit field allocations.  I don't think we want to end up
implementing all of them.

--
Added file: http://bugs.python.org/file23084/ctypes-packed-bitfields.tar.bz2

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12528
___

ctypes-packed-bitfields.tar.bz2
Description: BZip2 compressed data
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12528] Implement configurable bitfield allocation strategy

[Vlad Riscutia]

Vlad Riscutia riscutiav...@gmail.com added the comment:

Updated patch to reflect review feedback. Allocation strategy is now specified 
as string in Python code.

I kept asserts in CanContinueField/CanExpandField because, as I said, default 
case should never be hit. Input is validated in stgdict and this should make it 
clear to whoever reads the code that something is very wrong if execution gets 
to default case.

--
Added file: 
http://bugs.python.org/file22728/configurable_bitfield_allocation_v2.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12528
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12528] Implement configurable bitfield allocation strategy

[Amaury Forgeot d'Arc]

Amaury Forgeot d'Arc amaur...@gmail.com added the comment:

My review of the patch: http://bugs.python.org/review/12528/show

--
nosy: +amaury.forgeotdarc
stage:  - patch review

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12528
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12528] Implement configurable bitfield allocation strategy

[Vlad Riscutia]

New submission from Vlad Riscutia riscutiav...@gmail.com:

Opened this issue to track configurable bitfield allocation strategy. This will 
address issues like http://bugs.python.org/issue6069, 
http://bugs.python.org/issue11920.

Summary: the way bitfields are allocated is up to the compiler not defined by 
standard. MSVC and GCC have different strategies to perform the allocation so 
the size of bitfield structures can be different depending on compiler. 
Currently we hardcode allocation strategy to be GCC-way on non-Windows and 
MSVC-way on Windows which raises issues when trying to interop on Windows with 
GCC binaries.

Short term this solution will enable interop between MSVC compiled Python with 
GCC compiled binaries under Windows. It will also enable addressing other 
possible compiler interop issues in the future, for compilers that don't use 
GCC strategy.

Following is copied from thread discussing this:


On 6/25/2011 12:33 PM, Vlad Riscutia wrote:
I recently started looking at some ctypes issues. I dug a bit into
http://bugs.python.org/issue6069 and then I found
http://bugs.python.org/issue11920. They both boil down to the fact that
bitfield allocation is up to the compiler, which is different in GCC and
MSVC. Currently we have hard-coded allocation strategy based on paltform
in cfields.c:

 if (bitsize /* this is a bitfield request */

 *pfield_size /* we have a bitfield open */
 #ifdef MS_WIN32
   /* MSVC, GCC with -mms-bitfields */
 dict-size * 8 == *pfield_size
 #else
   /* GCC */
 dict-size * 8= *pfield_size
 #endif
 (*pbitofs + bitsize)= *pfield_size) {
   /* continue bit field */
   fieldtype = CONT_BITFIELD;
 #ifndef MS_WIN32
   } else if (bitsize /* this is a bitfield request */
 *pfield_size /* we have a bitfield open */
 dict-size * 8= *pfield_size
 (*pbitofs + bitsize)= dict-size * 8) {
   /* expand bit field */
   fieldtype = EXPAND_BITFIELD;
 #endif

So when creating a bitfield structure, it's size can be different on
Linux vs Windows.

class MyStructure(ctypes.BigEndianStructure):
_pack_  = 1# aligned to 8 bits, not ctypes default of 32
_fields_= [
   (Data0,   ctypes.c_uint32, 32),
   (Data1,   ctypes.c_uint8, 3),
   (Data2,   ctypes.c_uint16, 12),
  ]

sizeof for above structure is 6 on GCC build and 7 on MSVC build. This
leads to some confusion and issues, because we can't always interop
correctly with code compiled with different compiler than the one Python
is compiled with on the platform.

Just curious, are you saying that this is the 'cause' of the two bug reports, 
or 'just' something you discovered while investigating them?


 Short term solution is to add a warning in the documentation about this.

For 2.7/3.2, yes.


 Longer term though, I think it
would be better to add a property on the Structure class for
configurable allocation strategy, for example Native (default), GCC,
MSVC and when allocating the bitfield, use given strategy. Native would
behave similar to what happens now, but we would also allow GCC-style
allocation on Windows for example and the ability to extend this if we
ever run into similar issues with other compilers. This shouldn't be too
difficult to implement, will be backwards compatible and it would
improve interop. I would like to hear some opinions on this.

If this would allow the MSVC-compilied Python to better access dlls compiled 
with gcc (cygwin) on Windows, definitely -- in 3.3.
If the new feature is (currently) only useful on Windows, doc should say so.

-- 
Terry Jan Reedy

/copy

Attached is patch with initial refactoring of cfield.c to enable configurable 
allocation. Next step is to provide a way to configure this through Python 
library. I will also look at updating documentation to point out the known 
issue.

--
components: ctypes
files: cfield_bitfield_refactoring.diff
keywords: patch
messages: 140088
nosy: terry.reedy, vladris
priority: normal
severity: normal
status: open
title: Implement configurable bitfield allocation strategy
type: feature request
versions: Python 3.3
Added file: http://bugs.python.org/file22617/cfield_bitfield_refactoring.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12528
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12528] Implement configurable bitfield allocation strategy

[Santoso Wijaya]

Changes by Santoso Wijaya santoso.wij...@gmail.com:


--
nosy: +santa4nt

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12528
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12528] Implement configurable bitfield allocation strategy

[Vlad Riscutia]

Vlad Riscutia riscutiav...@gmail.com added the comment:

Removed previously attached partial patch, this is complete patch.

Summary:
Added following 3 constants in ctypes:
ctypes.BITFIELD_ALLOCATION_NATIVE
ctypes.BITFIELD_ALLOCATION_GCC
ctypes.BITFIELD_ALLOCATION_MSVC

Setting _bitfield_allocation_ attribute to one of these on a class declaration 
inheriting from Structure will force specified allocation of the bitfield.

NATIVE is equivalent to not specifying anything. 
GCC will do GCC-style allocation (what Python does now on non-Windows)
MSVC will do MSVC-style allocation (what Python does now on Windows)

I added unittests to cover these and ran full suit on both Windows and Linux.

Still have to update documentation to mention this. Will submit diff for that 
after this gets reviewed.

--
Added file: 
http://bugs.python.org/file22618/configurable_bitfield_allocation.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12528
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12528] Implement configurable bitfield allocation strategy

[Vlad Riscutia]

Changes by Vlad Riscutia riscutiav...@gmail.com:


Removed file: http://bugs.python.org/file22617/cfield_bitfield_refactoring.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12528
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com