I eventually figured out how to compile with some debug info and how to run X to get core dumps. I'm going to attach 3 things: XF86Config, XFree86.0.log, and a back trace from gdb of the core file. Of course, I'll provide anything else that can help.
I'm running XFree86 4.3.0 but I have seen exactly the same symptoms with 4.2.1 and 4.3.99.6. I have a Matrox G550 card in Dell Xeon workstation. One of the things in the log file is a note that unresolved symbols can cause server aborts and I do have some drm* unresolved symbols. The reason for this is that I need the 8 bit pseudocolor overlay and for some reason this doesn't work with DRI enabled so I commented out Load "glx" and Load "dri" in XF86Config. To ensure this wasn't causing the core dumps, I have tried the following combinations (and got server crashes each time except for the last):
1. comment out Load "glx" but leave Load "dri" - got glx* unresolved symbols instead
2. uncomment both Load "glx" and Load "dri" and make sure agpgart and mga modules are not loaded - no unresolved symbols but log indicates DRI is disabled, something tries to load agpgart module but my machine needs the try_unsupported flag so it fails and then mga fails to load.
3. load agpgart and mga modules before starting X and leave Load lines intact - 8 bit overlay doesn't work, only get 24 bit visuals, tool is painfully slow, i.e. crashes are better
Please let me know if I can provide any more information. I have other core files but this is the first one with some debug info in it.
Thanks, Malcolm
gdb.output
Description: Binary data
XF86Config
Description: Binary dataXFree86.0.log
Description: Binary data