Hi all,
I actually wrote a reply but it seems it has never been sent or I just it to
Rogers address instead of the mailing list.
To answer your (Rogers) suggestion again: The value that gets partially
overwritten is 32Bit where the first 16Bit gets overwritten. Plus, this only
happens for the first array element. So I'm pretty sure that this is not
directly related to a 32/64 bit problem. However, I would not deny that there
is a possibility that another 64Bit issue that indirectly causes this symptom.
So, here's the thing: We've traced the problem a bit down.
1. The pointer casting is not necessary.
return add_results, errors
Is equivalent or at least, it results in the same behavior and problems, AFAIK.
2. Some cases we've tried:
- If we create the array, than return it, we see the memory corruption on the
client, but not on the server.
- If we create the array, but overwrite the elements with new structs in each
loop, we can see the first element gets corrupted as soon we assign the second
element to a new item. This corruption is already visible on the server side.
This is probably caused by creating items in the context of the loop and their
references being freed once the loop iterates to the next index.
- In both cases (especially in the second case), if we keep python references
to the structures by adding them to a python list in self (or the array itself
in case 1), the memory corruption is gone at least on the server side. The
client never receives the return values but a COMError. We tried to catch
exceptions on the server and add trace messages on comtypes but with no luck.
There seems to be no exception in the python code.
This leads me to the following assumptions:
1. Returning pointers is not sufficient. You need to keep the references to
that actual objects to prevent their memory being freed.
2. Keeping the references leads to a strange COMError that must be caused by
the COM internals like (un)mashalling for example. Unfortunately, I found no
way to get some insights. This a hughe black box where the server inserts some
data and the client receives a meaningless error.
3. Windows 7 / Server 2008 behaves differently regarding memory allocation,
freeing and.
But I still don't have a solution.
Does anybody has an idea on how to solve or at least on how to trace down what
causes the issue?
I'm pretty desperate and thankful for any advice.
// Jan
-----Ursprüngliche Nachricht-----
Von: python-win32 [mailto:python-win32-bounces+jan.wedel=ettex...@python.org]
Im Auftrag von Roger Upole
Gesendet: Freitag, 13. Dezember 2013 01:17
An: python-win32@python.org
Betreff: Re: [python-win32] Missing memory de/allocation in com server causes
APPCRASH?
I'm guessing you have a 32/64 bit problem, as most XP installs are 32-bit.
I'd take a closer look at hServer, since handles are pointer-sized (4 bytes on
32 bit and 8 bytes on x64).
Roger
"Wedel, Jan" <jan.we...@ettex.de> wrote in message
news:47ee7f193edc7e468e74bb0acfb17929011692c...@ex10mbox1b.hosting.inetserver.de...
Ok,
being pretty desperate, I tried a few ways of returning the data until I got it
partially working.
1. I returned null pointers:
return POINTER(OpcDa.tagOPCITEMRESULT)(), POINTER(HRESULT)()
Obviously, it did not return any sensible data, but it did not crash! The
client actually receives the null pointer which is better that crashing without
any information.
2. Using the above knowledge, I tried to refactor the existing code as follows:
- creating the arrays in advance:
add_results = (OpcDa.tagOPCITEMRESULT*count)()
errors = (HRESULT*count)()
- looping over all elements of the array and filling in the data:
(...)
add_results[index].hServer = server_item_handle
(...)
Errors[index] = HRESULT(S_OK)
- returning a new pointer to the arrays
return POINTER(OpcDa.tagOPCITEMRESULT)(add_results), POINTER(HRESULT)(errors)
- Do not store the arrays in python's "self"
Using this procedure, it does not crash the client and it does return data.
Now, I am wondering why I had to do this on Win Server
2008 and on Xp the old code worked.
However, this led me to next problem:
hServer is of type c_ulong which means 4 bytes and is the first element of the
tagOPCITEMRESULT structure. Reproducibly, the first two bytes of the first
element of the array gets corrupted *somewhere* in the same way, more
specifically it gets overwritten to 0x4A00. The value 0x4A doesn't ring any
bells that could reveal the cause for that. All following elements of the array
are not corrupted.
The question is, how could that happen?
What could possible overwrite memory?
Do I have to prevent freeing memory used by these arrays somehow before
returning them via COM?
Thanks,
//Jan
Von: python-win32 [mailto:python-win32-bounces+jan.wedel=ettex...@python.org]
Im Auftrag von Wedel, Jan
Gesendet: Mittwoch, 11. Dezember 2013 10:38
An: python-win32@python.org
Betreff: [python-win32] Missing memory de/allocation in com server causes
APPCRASH?
Hi,
I already posted a related question on the comtypes mailing list but
unfortunately Thomas Heller told me that he cannot actively provide support
anymore.
So I'm hoping to get some help here because it might also relate more to ctypes.
At first, sorry for the long mail but it's a rather complex issue and I want to
provide as much information and context as required.
Heres the problem:
We wrote a COM server using comtypes which worked well on Win XP. However, on
Windows 2008 and Win 7, the same code crashes after calling a specific method
that our server provides.
By "crashes" I mean the COM client (Tried it with a C++ and python client)
receives a COMError:
_ctypes.COMError: (-2147417851, 'Ausnahmefehler des Servers.', (None, None,
None, 0, None))
The Server does *not* throw any exceptions as far as I can see and keeps on
running. I added some printf debugging in comtypes but everything seems fine
and ends up calling some win32api functions after message dispatching is done.
Every time this happens, in the windows application event log, there is an
entry saying "python.exe" was the offending application, "ntdll.dll" was the
offending module. I also set up windows debugging modules with shows the
following stack trace (just first few lines at the top):
ntdll!RtlpLowFragHeapFree+0x31 (FPO: [Non-Fpo])
01 0021ed04 76f49c46 00390000 00000000 029e76e0 ntdll!RtlFreeHeap+0x101
02 0021ed18 7718afbd 00390000 00000000 029e76e8 kernel32!HeapFree+0x14
03 0021ed2c 7718afd9 7725f6f8 029e76e8 0021ed54 ole32!CRetailMalloc_Free+0x1c
(FPO: [Non-Fpo])
04 0021ed3c 75be6efc 029e76e8 01f685fe 0021edcc ole32!CoTaskMemFree+0x13
Since the problem arises reproducibly when calling one method the has a pointer
to an array as input and output, I believe this is some kind of memory issue
(which also the stack trace suggests) that was probably just not
recognized/caught by WinXP before but now shows up in more recent windows
versions.
My assumption is, that either I need to actively allocate or deallocate memory
for in/out parameters.
This is the method signature of the method that our server provides and that
crashes:
IOPCItemMgt::ValidateItems
HRESULT ValidateItems(
[in] DWORD dwCount,
[in, size_is(dwCount)] OPCITEMDEF * pItemArray, [in] BOOL bBlobUpdate, [out,
size_is(,dwCount)] OPCITEMRESULT ** ppValidationResults, [out,
size_is(,dwCount)] HRESULT ** ppErrors );
Assume that this is the signature in python comtypes server:
def ValidateItems(self, count, p_item_array, update_blob):
and this is the ctypes structure which should be returned in "
ppValidationResults ":
OpcDa.tagOPCITEMRESULT
How would I create these structures (especially ppValidationResults), its
contents and the pointers so that I would not create any memory issues, null
pointers or whatsoever?
My implementation creates a few items by calling
validation_result = OpcDa.tagOPCITEMRESULT() filling in the
contents with e.g.
validation_result.hServer = server_item_handle adding it to a
list
validation_results.append(validation_result)
then converting it to a pointer to an array with a helper function:
def p_list(items):
c_items = (type(items[0])*len(items))(*items)
p_items = cast(c_items, POINTER(type(items[0])))
return p_items
(...)
p_validation_results = p_list(validation_results)
doing the same with "errors" and then returning it:
return p_validation_results, p_errors
Is that correct?
Do I have to keep references to the list, the pointer or the item strcuture
objects to avoid them beeing garbage collected?
Thanks a lot!
--------------------------------------------------------------------------------
> _______________________________________________
> python-win32 mailing list
> python-win32@python.org
> https://mail.python.org/mailman/listinfo/python-win32
>
_______________________________________________
python-win32 mailing list
python-win32@python.org
https://mail.python.org/mailman/listinfo/python-win32
_______________________________________________
python-win32 mailing list
python-win32@python.org
https://mail.python.org/mailman/listinfo/python-win32
