[issue29753] Ctypes Packing Bitfields Incorrectly - Linux

[miss-islington]

miss-islington  added the comment:


New changeset 0d7ad9fb38c041c46094087b0cf2c8ce44916b11 by Filipe Laíns in 
branch 'master':
bpo-29753: fix merging packed bitfields in ctypes struct/union (GH-19850)
https://github.com/python/cpython/commit/0d7ad9fb38c041c46094087b0cf2c8ce44916b11


--
nosy: +miss-islington

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29753] Ctypes Packing Bitfields Incorrectly - Linux

[Antoine Pitrou]

Change by Antoine Pitrou :


--
versions: +Python 3.8, Python 3.9 -Python 2.7, Python 3.4, Python 3.5, Python 
3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29753] Ctypes Packing Bitfields Incorrectly - Linux

[Filipe Laíns]

Filipe Laíns  added the comment:

My patches should resolve this fully. Please test it to make sure I did not 
miss any corner case.

https://github.com/python/cpython/pull/19850

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29753] Ctypes Packing Bitfields Incorrectly - Linux

[Filipe Laíns]

Change by Filipe Laíns :


--
keywords: +patch
pull_requests: +19167
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/19850

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29753] Ctypes Packing Bitfields Incorrectly - Linux

[Filipe Laíns]

Change by Filipe Laíns :


--
nosy: +FFY00

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29753] Ctypes Packing Bitfields Incorrectly - Linux

[Sam Price]

Sam Price  added the comment:

I ran into this bug on mac and submitted a duplicate issue of 39858


If you add the check *pfield_size != *pbitofs it fixes this bug.

#ifndef MS_WIN32
} else if (bitsize /* this is a bitfield request */
&& *pfield_size /* we have a bitfield open */
&& *pfield_size != *pbitofs /* Current field has been filled, start new 
one */
&& dict->size * 8 >= *pfield_size
&& (*pbitofs + bitsize) <= dict->size * 8) {
/* expand bit field */
fieldtype = EXPAND_BITFIELD;
#endif


However this would not fix the results where you expand a bitfield around 3 
bytes.
clang produces the 3 bytes if you try and expand around.
#include 
#include 
typedef struct testA{
uint8_t a0 : 7;
uint8_t a1 : 7;
uint8_t a2 : 7;
uint8_t a3 : 3;
}  __attribute__((packed)) testA;

int main(){
printf("%d\n", sizeof(testA)); /* Prints out 3 */
return 0;
}
python prints out a size of 4.

--
nosy: +thesamprice

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29753] Ctypes Packing Bitfields Incorrectly - Linux

[hai shi]

Change by hai shi :


--
nosy: +shihai1991

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29753] Ctypes Packing Bitfields Incorrectly - Linux

[Karl Ding]

Karl Ding  added the comment:

I believe the example can be simplified to the following:

class MyStructure(ctypes.Structure):
_pack_ = 1
_fields_ = [('A', ctypes.c_uint32, 8)]

assert ctypes.sizeof(MyStructure) == 1

Here, ctypes.sizeof returns 4 on my machine (64-bit Ubuntu 18.04 LTS), while I 
would expect it to return 1 like GCC. This is using a tree checked out at 
33ce3f012f249782507df896824b045b34f765aa, if it makes a difference.

If we compile the equivalent C code (on 64-bit Ubuntu 18.04 LTS):

#include 
#include 
#include 

struct my_structure_uint32 {
  uint32_t a : 8;
} __attribute__((packed));

int main(int argc, char *argv[]) {
  printf("sizeof(struct my_structure_uint32): %d\n", sizeof(struct 
my_structure_uint32));
  printf("alignof(struct my_structure_uint32): %d\n", alignof(struct 
my_structure_uint32));
  return 0;
}

Using the following GCC invocation:

gcc temp.c -o test && ./test

We get the following result:

sizeof(struct my_structure_uint32): 1
alignof(struct my_structure_uint32): 1

However, if I compile with MSVC bitfields:

gcc -mms-bitfields test.c -o test && ./test

We get the following result:

sizeof(struct my_structure_uint32): 4
alignof(struct my_structure_uint32): 1

Similarly, adding a second and third 8-bit wide bitfield member fails and 
returns 4, instead of the expected 2 and 3.

This is not just c_uint32, c_uint16 and c_int also exhibit the same behaviour:

class MyStructure(ctypes.Structure):
_pack_ = 1
_fields_ = [('A', ctypes.c_uint16, 8)]

assert ctypes.sizeof(MyStructure) == 1

--
nosy: +karlding

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29753] Ctypes Packing Bitfields Incorrectly - Linux

[Marc Le Roy]

Marc Le Roy  added the comment:

No solution found to solve this issue ?
The anomaly is not a cross platform inconsistency, it is an inconsistency 
between the behaviours of GCC and ctypes, both under Linux or Cygwin, when 
defining packed structures :

[Marc@I7-860 ~/dev/python/ctypes-bitfields-bug] make test
./bitfield_test1
sizeof(BF32) = 12 ; Memory dump of BF32 = 0x
sizeof(BF64) = 12 ; Memory dump of BF64 = 0x
python3 bitfield_test1.py
sizeof(BF32) = 16 ; Memory dump of BF32 = 0xff00ff00ff00ff00
sizeof(BF64) = 16 ; Memory dump of BF64 = 0x

--
Added file: https://bugs.python.org/file47511/ctypes-bitfields-bug.tar.gz

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29753] Ctypes Packing Bitfields Incorrectly - Linux

[Charles Machalow]

Charles Machalow  added the comment:

All of Python is implicitly cross platform. If something isn't actually cross 
platform, it should be mentioned explicitly in the documentation. For example 
see the mmap documentation, it explicitly say on Unix it does X, on Windows it 
does Y. We should be as explicit as possible for ctypes to say on Windows we 
expect packing like X (GCC) and on Linux we expect packing like Y (VC++).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29753] Ctypes Packing Bitfields Incorrectly - Linux

[Antti Haapala]

Antti Haapala  added the comment:

"Antti, is there a place in the ctypes documentation that explicitly says 
ctypes is not meant to be used cross-platform? If not, shouldn't that be 
mentioned?"

I don't know about that, but the thing is nowhere does it say that it is meant 
to be used cross-platform. It just says it allows defining C types. It is 
somewhat implied that C types are not cross-platform at binary level, at all.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29753] Ctypes Packing Bitfields Incorrectly - Linux

[Charles Machalow]

Charles Machalow  added the comment:

Antti, is there a place in the ctypes documentation that explicitly says ctypes 
is not meant to be used cross-platform? If not, shouldn't that be mentioned?

I think ultimately ctypes should default to standard OS/compiler behavior, but 
should allow the flexibility for one to override for cross-platform 
interchangeability.

In the C++ code here, the code is explicitly checking if Windows (or GCC) to 
choose behavior. In theory, that could be changed to allow runtime-choice for 
packing behavior. 

Of course, that is probably best for a feature issue. For this one, I'd just 
like to have the behavior on Linux match GCC, as that is the definitive bug 
here as our example has shown.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29753] Ctypes Packing Bitfields Incorrectly - Linux

[Marc Le Roy]

Change by Marc Le Roy :


--
nosy: +mleroy003

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29753] Ctypes Packing Bitfields Incorrectly - Linux

[Antti Haapala]

Antti Haapala added the comment:

To Charles first: "Gives back a sizeof of 8 on Windows and 10 on Linux. The 
inconsistency makes it difficult to have code work cross-platform." 

The bitfields in particular and ctypes in general have *never* been meant to be 
cross-platform - instead they just must need to match the particular C compiler 
behaviour of the platform, thus the use of these for cross platform work is 
ill-advised - perhaps you should just use the struct module instead.

However, that said, on Linux, sizeof these structures - packed or not - do not 
match the output from GCC; unpacked one has sizeof 12 and packed 10 on my 
Python 3.5, but they're both 8 bytes on GCC. This is a real bug.

GCC says that the bitfield behaviour is: 
https://gcc.gnu.org/onlinedocs/gcc-4.9.1/gcc/Structures-unions-enumerations-and-bit-fields-implementation.html

Whether a bit-field can straddle a storage-unit boundary (C90 6.5.2.1, C99 and 
C11 6.7.2.1).

Determined by ABI.
The order of allocation of bit-fields within a unit (C90 6.5.2.1, C99 and C11 
6.7.2.1).

Determined by ABI.
The alignment of non-bit-field members of structures (C90 6.5.2.1, C99 and C11 
6.7.2.1).

Determined by ABI. 

Thus, the actual behaviour need to be checked from the API documentation of the 
relevant platform. However - at least for unpacked structs - the x86-64 
behaviour is that a bitfield may not cross an addressable unit.

--
nosy: +ztane
title: Ctypes Packing Incorrectly - Linux -> Ctypes Packing Bitfields 
Incorrectly - Linux

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com