[issue22273] abort when passing certain structs by value using ctypes

STINNER Victor added the comment: New changeset bfe1f74e39d0049a829962050e86a6a2d2a2781e by Victor Stinner in branch '3.8': [3.8] bpo-3832: Fix compiler warnings (GH-16518) https://github.com/python/cpython/commit/bfe1f74e39d0049a829962050e86a6a2d2a2781e --

[issue22273] abort when passing certain structs by value using ctypes

Vinay Sajip added the comment: New changeset c9a413ede47171a224c72dd34122005170caaad4 by Vinay Sajip (Victor Stinner) in branch 'master': bpo-38321: Fix PyCStructUnionType_update_stgdict() warning (GH-16492) https://github.com/python/cpython/commit/c9a413ede47171a224c72dd34122005170caaad4

[issue22273] abort when passing certain structs by value using ctypes

Change by STINNER Victor : -- pull_requests: +16079 pull_request: https://github.com/python/cpython/pull/16492 ___ Python tracker ___

[issue22273] abort when passing certain structs by value using ctypes

Change by Vinay Sajip : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker ___ ___

[issue22273] abort when passing certain structs by value using ctypes

Vinay Sajip added the comment: New changeset b92b8c53f63d2e805288ac2c665bf5887d1b7ca6 by Vinay Sajip in branch '3.8': [3.8] bpo-22273: Changed conditions for ctypes array-in-struct handling. (GH-16381) (GH-16401)

[issue22273] abort when passing certain structs by value using ctypes

Change by Vinay Sajip : -- pull_requests: +15983 pull_request: https://github.com/python/cpython/pull/16401 ___ Python tracker ___

[issue22273] abort when passing certain structs by value using ctypes

Vinay Sajip added the comment: New changeset d015714f89af5c57b7071a74eeca618577b3dfe4 by Vinay Sajip in branch '3.7': [3.7] bpo-22273: Changed conditions for ctypes array-in-struct handling. (GH-16381) (GH-16400)

[issue22273] abort when passing certain structs by value using ctypes

Change by Vinay Sajip : -- pull_requests: +15982 pull_request: https://github.com/python/cpython/pull/16400 ___ Python tracker ___

[issue22273] abort when passing certain structs by value using ctypes

Vinay Sajip added the comment: New changeset cc28ed2421bf3953efc0fbde387f28722f3801e2 by Vinay Sajip in branch 'master': bpo-22273: Removed temporary test skipping on PPC platforms. (GH-16399) https://github.com/python/cpython/commit/cc28ed2421bf3953efc0fbde387f28722f3801e2 --

[issue22273] abort when passing certain structs by value using ctypes

Vinay Sajip added the comment: > Please check if bpo-38272 regression is caused by this issue. Yes, it is. Being worked on right now. Sorry for the noise. -- ___ Python tracker

[issue22273] abort when passing certain structs by value using ctypes

Change by Vinay Sajip : -- pull_requests: +15981 pull_request: https://github.com/python/cpython/pull/16399 ___ Python tracker ___

[issue22273] abort when passing certain structs by value using ctypes

Vinay Sajip added the comment: New changeset 417089e88bd4ea146b9497e06e8edeb58a59cd65 by Vinay Sajip in branch 'master': bpo-22273: Re-enabled ctypes test on ARM machines. (GH-16388) https://github.com/python/cpython/commit/417089e88bd4ea146b9497e06e8edeb58a59cd65 --

[issue22273] abort when passing certain structs by value using ctypes

Change by Vinay Sajip : -- pull_requests: +15970 pull_request: https://github.com/python/cpython/pull/16388 ___ Python tracker ___

[issue22273] abort when passing certain structs by value using ctypes

STINNER Victor added the comment: Please check if bpo-38272 regression is caused by this issue. "FAIL: test_array_in_struct (ctypes.test.test_structures.StructureTestCase)" on ARMv7. -- nosy: +vstinner ___ Python tracker

[issue22273] abort when passing certain structs by value using ctypes

Change by Vinay Sajip : -- pull_requests: +15961 pull_request: https://github.com/python/cpython/pull/16381 ___ Python tracker ___

[issue22273] abort when passing certain structs by value using ctypes

Vinay Sajip added the comment: New changeset 57dc7d5ae8ebfb6da1ea2b25e61260ecb9c79faf by Vinay Sajip in branch 'master': bpo-22273: Disabled tests while investigating buildbot failures on ARM7L/PPC64. (GH-16377)

[issue22273] abort when passing certain structs by value using ctypes

Change by Vinay Sajip : -- pull_requests: +15956 pull_request: https://github.com/python/cpython/pull/16377 ___ Python tracker ___

[issue22273] abort when passing certain structs by value using ctypes

Vinay Sajip added the comment: New changeset 16c0f6df62a39f9f7712b1c0577de4eefcb4c1bf by Vinay Sajip (Miss Islington (bot)) in branch '3.7': bpo-22273: Update ctypes to correctly handle arrays in small structur… (GH-15839) (GH-16369)

[issue22273] abort when passing certain structs by value using ctypes

Vinay Sajip added the comment: New changeset ce62dcc460cf88f663c34c4a0948c1ee1dc53f4d by Vinay Sajip (Miss Islington (bot)) in branch '3.8': bpo-22273: Update ctypes to correctly handle arrays in small structur… (GH-15839) (GH-16370)

[issue22273] abort when passing certain structs by value using ctypes

Change by miss-islington : -- pull_requests: +15950 pull_request: https://github.com/python/cpython/pull/16370 ___ Python tracker ___

[issue22273] abort when passing certain structs by value using ctypes

Change by miss-islington : -- pull_requests: +15949 pull_request: https://github.com/python/cpython/pull/16369 ___ Python tracker ___

[issue22273] abort when passing certain structs by value using ctypes

Vinay Sajip added the comment: New changeset 12f209eccb1587e8c98057d9c5f865c21f4a16c1 by Vinay Sajip in branch 'master': bpo-22273: Update ctypes to correctly handle arrays in small structur… (GH-15839) https://github.com/python/cpython/commit/12f209eccb1587e8c98057d9c5f865c21f4a16c1

[issue22273] abort when passing certain structs by value using ctypes

Change by Vinay Sajip : -- pull_requests: +15484 pull_request: https://github.com/python/cpython/pull/15839 ___ Python tracker ___

[issue22273] abort when passing certain structs by value using ctypes

Change by Cheryl Sabella : -- versions: +Python 3.8 -Python 2.7, Python 3.4, Python 3.5, Python 3.6 ___ Python tracker ___ ___

[issue22273] abort when passing certain structs by value using ctypes

Vinay Sajip added the comment: Yes, the patch needs improving as per the suggestion in msg288493 (not had the time since to do any work on it), followed by a review of the changes. -- ___ Python tracker

[issue22273] abort when passing certain structs by value using ctypes

Ilya Kulakov added the comment: Is there anything to be done for this patch to get merged? -- ___ Python tracker ___

[issue22273] abort when passing certain structs by value using ctypes

Vinay Sajip added the comment: > Perhaps it should instead use two specific flags, TYPEFLAG_HASBITFIELD and > TYPEFLAG_HASUNION This seems better at first sight. It's not making any suitability decisions (apart from doing the unrolling), and the meaning of these flags will be less volatile

[issue22273] abort when passing certain structs by value using ctypes

Eryk Sun added the comment: Perhaps it should instead use two specific flags, TYPEFLAG_HASBITFIELD and TYPEFLAG_HASUNION, which are propagated unconditionally from the base class and fields. As a base case, a union itself is flagged TYPEFLAG_HASUNION. Arrays are unrolled on X86_64 only if

[issue22273] abort when passing certain structs by value using ctypes

Vinay Sajip added the comment: > It occurs to me that in the 1st pass, it also needs to propagate the > non-argument flag from any field that has it set. So does that mean disallowing a structure which contains a union? What about if the final structure is large enough to require passing in

[issue22273] abort when passing certain structs by value using ctypes

Eryk Sun added the comment: > I'm not sure using this flag impacts on consistency with CFFI I meant consistency with respect to supported argument types. If someone contributed a workaround to CFFI, then I would rather port it, but it looks like they're also waiting for this to be addressed

[issue22273] abort when passing certain structs by value using ctypes

Vinay Sajip added the comment: > We need more feedback on this suggested flag, especially to stay consistent > with CFFI if possible. Undoubtedly, more feedback would be very helpful. I'm not sure using this flag impacts on consistency with CFFI particularly, since it's an internal

[issue22273] abort when passing certain structs by value using ctypes

Eryk Sun added the comment: I had suggested inheriting the TYPEFLAG_NONARGTYPE flag in StructUnionType_new. It requires a minor change to get basedict unconditionally, and then assign if (basedict) dict->flags |= basedict->flags & TYPEFLAG_NONARGTYPE; We need more feedback on

[issue22273] abort when passing certain structs by value using ctypes

Vinay Sajip added the comment: Just a thought - the TYPEFLAG_NONARGTYPE needs to be copied from the base class if set there, right? -- ___ Python tracker

[issue22273] abort when passing certain structs by value using ctypes

Eryk Sun added the comment: > Perhaps you mean 16 rather than 8? Sorry, that was a misfire. It should be 16. -- ___ Python tracker ___

[issue22273] abort when passing certain structs by value using ctypes

Vinay Sajip added the comment: Thanks for the comments. Using your suggestions simplifies things quite a bit. Still finding my way around :-) > Regarding structs with bitfields and unions, we could add an stgdict flag to > prevent passing them as arguments in the Unix X86_64 ABI Is this to

[issue22273] abort when passing certain structs by value using ctypes

Eryk Sun added the comment: Notes on fix-22273-02.diff: In the second pass over _fields_, you can (should) use dict->length and dict->proto for array types instead of the _length_ and _type_ attributes. When reassigning stgdict->ffi_type_pointer.elements, if use_broken_old_ctypes_semantics

[issue22273] abort when passing certain structs by value using ctypes

Changes by Vinay Sajip : -- stage: needs patch -> patch review ___ Python tracker ___

[issue22273] abort when passing certain structs by value using ctypes

Vinay Sajip added the comment: Patch attached, including tests. If it looks halfway sensible, I can work up a PR. -- Added file: http://bugs.python.org/file46660/fix-22273-02.diff ___ Python tracker

[issue22273] abort when passing certain structs by value using ctypes

Eryk Sun added the comment: The 24-byte struct gets passed on the stack, as it should be. In this case ffi_call doesn't abort() because examine_argument returns 0, which is due to the following code in classify_argument: if (words > 2) { /* When size > 16 bytes, if the first

[issue22273] abort when passing certain structs by value using ctypes

Vinay Sajip added the comment: I'm learning a bit about Linux calling conventions :-) But it also works when a 16-byte array is followed by 2 ints; if the two ints are removed, then it fails again. ctypes sets elements up in the first case to be a FFI_TYPE_POINTER slot followed by two slots

[issue22273] abort when passing certain structs by value using ctypes

Eryk Sun added the comment: I see now why you couldn't find ffi64.c. I've been using a 3.6 worktree. The libffi sources have been removed from master. For the union and bitfield problem, also see the crash reported in #26628. -- ___ Python tracker

[issue22273] abort when passing certain structs by value using ctypes

Eryk Sun added the comment: classify_argument is in Modules/_ctypes/libffi/src/x86/ffi64.c. This file is for the 64-bit Unix ABI. libffi doesn't use it for 64-bit Windows. Some (all?) Linux distros link ctypes to the system libffi. In my experience, building 3.6 on Linux defaults to the

[issue22273] abort when passing certain structs by value using ctypes

Vinay Sajip added the comment: Possibly also relevant: #16575 and https://github.com/libffi/libffi/issues/33 -- ___ Python tracker ___

[issue22273] abort when passing certain structs by value using ctypes

Vinay Sajip added the comment: Thanks for spelling it out for me, that's helpful. But I'm still confused about a couple of things: I can't find classify_argument in the Python source tree other than in Modules/_ctypes/libffi_osx/x86/x86-ffi64.c Is that the file you referred to as ffi64.c? I

[issue22273] abort when passing certain structs by value using ctypes

Eryk Sun added the comment: Structs that are larger than 32 bytes get copied to the stack (see classify_argument in ffi64.c), so we don't have to worry about classifying their elements for register passing. Thus if a new field is added for this in StgDictObject, then PyCArrayType_new should

[issue22273] abort when passing certain structs by value using ctypes

Vinay Sajip added the comment: I've not marked it "patch review" as the patch isn't complete. Just wanted to see if anyone can reproduce/explain the working on Windows/failing on Linux. -- ___ Python tracker

[issue22273] abort when passing certain structs by value using ctypes

Vinay Sajip added the comment: > I think we can special-case small arrays in PyCStructUnionType_update_stgdict Is that definitely the right place? And is doing it only for small arrays going to be enough? Currently, PyCStructUnionType_update_stgdict does dict = PyType_stgdict(desc); and then

[issue22273] abort when passing certain structs by value using ctypes

Changes by Vinay Sajip : -- nosy: +vinay.sajip ___ Python tracker ___ ___

[issue22273] abort when passing certain structs by value using ctypes

Eryk Sun added the comment: ctypes defines arrays as a pointer FFI type because they degenerate as pointers in C calls. But it's generally wrong to set a pointer FFI type for an array in the `elements` of a struct's FFI type. An integer array in a struct that's 16 bytes or less should be

[issue22273] abort when passing certain structs by value using ctypes

Changes by Eryk Sun : -- stage: -> needs patch ___ Python tracker ___ ___

[issue22273] abort when passing certain structs by value using ctypes

Changes by Eryk Sun : -- priority: normal -> high versions: +Python 3.5, Python 3.6, Python 3.7 ___ Python tracker ___

[issue22273] abort when passing certain structs by value using ctypes

Ilya Kulakov added the comment: The structure hack does not work on Windows 8, x64. -- nosy: +Ilya.Kulakov ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22273 ___

[issue22273] abort when passing certain structs by value using ctypes

Changes by Alexei Romanov drednout...@gmail.com: -- nosy: +alexei.romanov versions: +Python 2.7 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22273 ___

[issue22273] abort when passing certain structs by value using ctypes

Weeble added the comment: I had a closer look at the cif object in gdb. The ffi_type of the argument in question has size 16, alignment 1, type FFI_TYPE_STRUCT and elements contains a single nested ffi_type, of size 8, alignment 8, type FFI_TYPE_POINTER. I think this pointer type is wrong.

[issue22273] abort when passing certain structs by value using ctypes

Changes by Ned Deily n...@acm.org: -- nosy: +amaury.forgeotdarc, belopolsky, meador.inge ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22273 ___

[issue22273] abort when passing certain structs by value using ctypes

New submission from Weeble: I'm not 100% certain this is a bug yet, but I'm beginning to think it's likely. On 64-bit Linux, I can't pass a struct like this: struct S { uint8_t data[16]; }; ...to a function declared like this: void f(struct S); From experimentation with various