I realise that this issue is very specialised, and I'm not sure which
mailing list would be best to host a thread based on it, so I'm posting
it here to start with: I'm happy to move it to another mysql list if
someone would like to suggest one.
Over the last year, I've been working on a multi-threaded application
(based on OpenLDAP) which queries MySQL via libiodbc and MyODBC (issuing
many searches and updates in parallel.) It had been running well and
stably, 24 hours a day, for over six months. Then, because of a security
vulnerability, I upgraded to the latest version of OpenLDAP, and
suddenly the application would not run for more than about three minutes
without segfaulting.
I've been working on this problem for about three weeks, and I believe
I've sorted it: if I'm right, there may be some general implications for
MyODBC. My main debugging tool has been mpatrol, a superb and
highly-recommended memory-debugging kit. Using this, (aided by coredump
examinations in gdb), I found that the segfaults were being caused by
attempts to access nonexistent memory: the sources were a number of
MyODBC functions (the exact one changed with each test run): the common
feature was that in each case, the final address in the mpatrol log
traceback pointed to a DBUG_xxx macro (usually DBUG_RETURN). At first, I
didn't believe this: then, in an attempt to get more information, I
incorporated a patch into the Linux kernel to generate a coredump from
the thread which had received the signal (rather than from the
application's master thread): the resulting coredump backtraces again
pointed to MyODBC source lines containing DBUG macros.
I rebuilt mysql without debugging support, then the ODBC libraries and
the application. Problem still as above. MyODBC doesn't have a
--without-debug option in its configure script (I assume this is because
debugging can be activated/deactivated using the debug option in the
odbc.ini file.) I therefore went through all the MyODBC code, commenting
out all references to the DBUG macros (replacing DBUG_RETURN and
DBUG_VOID_RETURN with normal return statements.) I then rebuilt MyODBC
and the application.
I believe that this gave a result. The application has now been running
under full load without segfaulting for over three hours. (That's more
than 2h 55m longer than at any time in the last two months. Obviously,
it will need to run for a lot longer before I fully trust it again, but
I would say that, even at this stage, I'm better than 95% convinced that
I've found and eliminated the problem.)
Occasionally, within MyODBC, I've seen DBUG_RETURN statements which call
MyODBC functions: my best guess at present is that the complex dynamic
memory operations going on within these functions is somehow conflicting
with DBUG's own dynamic memory management. (I noticed that at the end of
the original DBUG manual (mysql-x.xx.xx/dbug/user.r) there is a warning:
"Programs which use memory allocation functions other than malloc will
usually have problems using the standard dbug package. The most common
problem is multiply allocated memory.")
Has anyone else experienced similar problems with other multi-threaded
applications? I guess that two possible ways forward would be either (a)
introduce a configure option in MyODBC to totally exclude all the
debugging code (as in mysql) - I intend to try this myself; and/or (b)
rewrite DBUG's memory allocation to use the facilities in safemalloc.
Versions:
Linux: Redhat 6.2, kernel 2.2.16-3 (using Terje Malmedal's coredump
patch from http://www.movement.uklinux.net - there are a number of
patches addressing the threaded coredump issue, but this one was
specific for my kernel and it sure as hell did the business 8))
OpenLDAP: originally 2.0.7, then 2.0.11.
mysql: 3.23.38
MyODBC: 2.50.33
libiodbc: 3.0.5
mpatrol: 1.4.5 (http://www.cbmamiga.demon.co.uk/mpatrol)
---------------------------------------------------------------------
Before posting, please check:
http://www.mysql.com/manual.php (the manual)
http://lists.mysql.com/ (the list archive)
To request this thread, e-mail <[EMAIL PROTECTED]>
To unsubscribe, e-mail <[EMAIL PROTECTED]>
Trouble unsubscribing? Try: http://lists.mysql.com/php/unsubscribe.php
