---- Joe Lewis <j...@joe-lewis.com> wrote: > On 6/22/12 6:16 AM, oh...@cox.net wrote: > > ---- oh...@cox.net wrote: > >> ---- Rainer Jung<rainer.j...@kippdata.de> wrote: > >>> On 22.06.2012 06:10, Joe Lewis wrote: > >>>> On 6/21/12 10:02 PM, oh...@cox.net wrote: > >>>>> ---- Joe Lewis<j...@joe-lewis.com> wrote: > >>>>>> On 6/21/12 9:39 PM, oh...@cox.net wrote: > >>>>>>> ---- oh...@cox.net wrote: > >>>>>>>> ---- oh...@cox.net wrote: > >>>>>>>>> ---- Joe Lewis<j...@joe-lewis.com> wrote: > >>>>>>>>>> On 6/21/12 7:32 PM, oh...@cox.net wrote: > >>>>>>>>>>> ---- oh...@cox.net wrote: > >>>>>>>>>>>> ---- Joe Lewis<j...@joe-lewis.com> wrote: > >>>>>>>>>>>>> On 6/21/12 6:46 PM, oh...@cox.net wrote: > >>>>>>>>>>>>>> ---- Joe Lewis<j...@joe-lewis.com> wrote: > >>>>>>>>>>>>>>> On 6/21/12 5:49 PM, oh...@cox.net wrote: > >>>>>>>>>>>>>>>> ---- oh...@cox.net wrote: > >>>>>>>>>>>>>>>>> ---- Sorin Manolache<sor...@gmail.com> wrote: > >>>>>>>>>>>>>>>>>> And I forgot to say: run gdb in some sort of environment > >>>>>>>>>>>>>>>>>> where you see > >>>>>>>>>>>>>>>>>> your current source code line and a couple of surrounding > >>>>>>>>>>>>>>>>>> lines. You > >>>>>>>>>>>>>>>>>> could achieve this with the "list" command, but I prefer > >>>>>>>>>>>>>>>>>> running gdb in > >>>>>>>>>>>>>>>>>> emacs and let emacs do the nice listing of source code in > >>>>>>>>>>>>>>>>>> a different panel. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> S > >>>>>>>>>>>>>>>>> Here's the function from my source. It's the original > >>>>>>>>>>>>>>>>> from mod_headers.c, plus my printf: > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> static int header_post_config(apr_pool_t *pconf, > >>>>>>>>>>>>>>>>> apr_pool_t *plog, > >>>>>>>>>>>>>>>>> apr_pool_t *ptemp, > >>>>>>>>>>>>>>>>> server_rec *s) > >>>>>>>>>>>>>>>>> { > >>>>>>>>>>>>>>>>> printf("In header_post_config\n"); > >>>>>>>>>>>>>>>>> header_ssl_lookup = > >>>>>>>>>>>>>>>>> APR_RETRIEVE_OPTIONAL_FN(ssl_var_lookup); > >>>>>>>>>>>>>>>>> return OK; > >>>>>>>>>>>>>>>>> } > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Jim > >>>>>>>>>>>>>>>> Hi, > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> I was able to get the segfault to go away. Here's what I > >>>>>>>>>>>>>>>> had to do: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> - Created /etc/ld.so.conf.d/my.conf, and added the > >>>>>>>>>>>>>>>> directory where my libobaccess.so was > >>>>>>>>>>>>>>>> - Run 'ldconfig' to activate. > >>>>>>>>>>>>>>>> - In the apxs command, DON'T include the -L and -l arguments > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> After that, Apache appears to start ok, without segfault :)!! > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Thanks for all of the great help, esp. the suggestion about > >>>>>>>>>>>>>>>> checking "ldconfig -p". I still don't understand why, but > >>>>>>>>>>>>>>>> I'm just glad that I can get past this piece so now I can > >>>>>>>>>>>>>>>> debug my module :)... > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Later, > >>>>>>>>>>>>>>>> Jim > >>>>>>>>>>>>>>> I'm just glad this list is as good as it is! > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> FYI, the ldconfig is the dynamic linker control, and those > >>>>>>>>>>>>>>> /etc/ld.so.conf.d files provide additional search > >>>>>>>>>>>>>>> directories for the > >>>>>>>>>>>>>>> linker to check in when loading a library. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Joe > >>>>>>>>>>>>>>> -- > >>>>>>>>>>>>>>> http://www.silverhawk.net/ > >>>>>>>>>>>>>> Hi, > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Sorry to report, but my earlier report was a "false positive" > >>>>>>>>>>>>>> :)... > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> I forgot that the mod_headers.c that I was doing the earlier > >>>>>>>>>>>>>> testing with had all references to the libobaccess.so removed > >>>>>>>>>>>>>> :(!! > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> So, I'm still stuck with basically the same problem, now, > >>>>>>>>>>>>>> working with my "full" code, with the calls in it: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> - If I compile with -L and -l, Apache segfaults when it starts > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> - If I compile without -L and -l, then I get "undefined > >>>>>>>>>>>>>> symbol" errors when I try to start Apache, e.g.: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> [root@apachemodule build-mod_headers]# > >>>>>>>>>>>>>> /apps/httpd2222/bin/apachectl -k start -X > >>>>>>>>>>>>>> httpd: Syntax error on line 84 of > >>>>>>>>>>>>>> /apps/httpd2222/conf/httpd.conf: Cannot load > >>>>>>>>>>>>>> /apps/httpd2222/modules/mod_headers.so into server: > >>>>>>>>>>>>>> /apps/httpd2222/modules/mod_headers.so: undefined symbol: > >>>>>>>>>>>>>> ObResource_isProtected > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> That "ObResource_isProtected" should be a symbol in > >>>>>>>>>>>>>> libobaccess.so, and in fact, if I do "nm --dynamic", I get: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> [root@apachemodule build-mod_headers]# nm --dynamic > >>>>>>>>>>>>>> /apps/netpoint/lib64/libobaccess.so | grep > >>>>>>>>>>>>>> "ObResource_isProtected" > >>>>>>>>>>>>>> 00000000000a6d80 T ObResource_isProtected > >>>>>>>>>>>>>> [root@apachemodule build-mod_headers]# > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> I'm *assuming* that the reason for the "undefined symbol" > >>>>>>>>>>>>>> error is that libobaccess.so is actually not being loaded, > >>>>>>>>>>>>>> but then when I try to load libobaccess.so, either via -L and > >>>>>>>>>>>>>> -l in the apxs, or using LoadFile in httpd.conf, I get the > >>>>>>>>>>>>>> segfault (same gdb info, BTW). > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Catch-22? > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Sorry for the false alarm :(!! > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Jim > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> Not a catch-22. The -L and -l specify linker options when > >>>>>>>>>>>>> assembling > >>>>>>>>>>>>> the code. The ldconfig is a run-time thing. If you are > >>>>>>>>>>>>> getting the > >>>>>>>>>>>>> stderr messages, you are making it all the way into your > >>>>>>>>>>>>> library. I'd > >>>>>>>>>>>>> suggest commenting out the following line and see if you get > >>>>>>>>>>>>> farther : > >>>>>>>>>>>>> > >>>>>>>>>>>>> header_ssl_lookup = APR_RETRIEVE_OPTIONAL_FN(ssl_var_lookup); > >>>>>>>>>>>>> > >>>>>>>>>>>>> That should tell you if the problem is the ssl_var_lookup. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Joe > >>>>>>>>>>>>> -- > >>>>>>>>>>>>> http://www.silverhawk.net/ > >>>>>>>>>>>> Hi, > >>>>>>>>>>>> > >>>>>>>>>>>> Thanks for the suggestion. I just tried what you suggested, > >>>>>>>>>>>> and got a segfault when I started Apache with the modified > >>>>>>>>>>>> module. > >>>>>>>>>>>> > >>>>>>>>>>>> Jim > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> Hi, > >>>>>>>>>>> > >>>>>>>>>>> As a reminder, here's the gdb with the library loaded: > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> (gdb) b header_post_config > >>>>>>>>>>> Function "header_post_config" not defined. > >>>>>>>>>>> Make breakpoint pending on future shared library load? (y or [n]) > >>>>>>>>>>> y > >>>>>>>>>>> Breakpoint 1 (header_post_config) pending. > >>>>>>>>>>> (gdb) run -d /apps/httpd2222/ -f /apps/httpd2222/conf/httpd.conf > >>>>>>>>>>> Starting program: /apps/httpd2222/bin/httpd -d /apps/httpd2222/ > >>>>>>>>>>> -f /apps/httpd2222/conf/httpd.conf > >>>>>>>>>>> [Thread debugging using libthread_db enabled] > >>>>>>>>>>> [New Thread 182897610272 (LWP 11317)] > >>>>>>>>>>> Breakpoint 2 at 0x2a9751ea90: file mod_headers.c, line 1121. > >>>>>>>>>>> Pending breakpoint "header_post_config" resolved > >>>>>>>>>>> mod_headers-jl V0.09 - start calling OAM API > >>>>>>>>>>> In register_hooks > >>>>>>>>>>> In create_headers_dir_config > >>>>>>>>>>> In create_headers_dir_config > >>>>>>>>>>> In header_cmd > >>>>>>>>>>> In header_inout_cmd > >>>>>>>>>>> In parse_format_tag > >>>>>>>>>>> In parse_misc_string > >>>>>>>>>>> In create_headers_dir_config > >>>>>>>>>>> In header_cmd > >>>>>>>>>>> In header_inout_cmd > >>>>>>>>>>> In parse_format_tag > >>>>>>>>>>> In parse_misc_string > >>>>>>>>>>> [Switching to Thread 182897610272 (LWP 11317)] > >>>>>>>>>>> > >>>>>>>>>>> Breakpoint 2, header_post_config (pconf=0x573138, plog=0x5a52c8, > >>>>>>>>>>> ptemp=0x5a72d8, s=0x59d3a8) at mod_headers.c:1121 > >>>>>>>>>>> 1121 printf("In header_post_config\n"); > >>>>>>>>>>> (gdb) n > >>>>>>>>>>> 1120 { > >>>>>>>>>>> (gdb) n > >>>>>>>>>>> 1121 printf("In header_post_config\n"); > >>>>>>>>>>> (gdb) n > >>>>>>>>>>> In header_post_config > >>>>>>>>>>> 1122 header_ssl_lookup = > >>>>>>>>>>> APR_RETRIEVE_OPTIONAL_FN(ssl_var_lookup); > >>>>>>>>>>> (gdb) n > >>>>>>>>>>> 1124 } > >>>>>>>>>>> (gdb) n > >>>>>>>>>>> 0x00000000004360c7 in ap_run_post_config (pconf=0x573138, > >>>>>>>>>>> plog=0x5a52c8, ptemp=0x5a72d8, s=0x59d3a8) at config.c:91 > >>>>>>>>>>> 91 AP_IMPLEMENT_HOOK_RUN_ALL(int, post_config, > >>>>>>>>>>> (gdb) n > >>>>>>>>>>> > >>>>>>>>>>> Program received signal SIGSEGV, Segmentation fault. > >>>>>>>>>>> 0x0000003518d6c1e1 in BN_num_bits () from /lib64/libcrypto.so.4 > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> So, it's actually blowing up in "BN_num_bits()" in > >>>>>>>>>>> /lib64/libcrypto.so.4? > >>>>>>>>>>> > >>>>>>>>>>> Jim > >>>>>>>>>> When you see the : > >>>>>>>>>> > >>>>>>>>>> Program received signal SIGSEGV, Segmentation fault. > >>>>>>>>>> 0x0000003518d6c1e1 in BN_num_bits () from /lib64/libcrypto.so.4 > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> in gdb, type "bt" and hit enter. It should show the back trace > >>>>>>>>>> and how > >>>>>>>>>> you got to the BN_num_bits() function. > >>>>>>>>>> > >>>>>>>>>> Joe > >>>>>>>>>> -- > >>>>>>>>>> http://www.silverhawk.net/ > >>>>>>>>> Hi, > >>>>>>>>> > >>>>>>>>> Here's the bt full: > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> In header_inout_cmd > >>>>>>>>> In parse_format_tag > >>>>>>>>> In parse_misc_string > >>>>>>>>> [Switching to Thread 182897610272 (LWP 6676)] > >>>>>>>>> > >>>>>>>>> Breakpoint 2, header_post_config (pconf=0x573138, plog=0x5a52c8, > >>>>>>>>> ptemp=0x5a72d8, s=0x59d3a8) at mod_headers.c:1121 > >>>>>>>>> 1121 printf("In header_post_config\n"); > >>>>>>>>> (gdb) n > >>>>>>>>> 1120 { > >>>>>>>>> (gdb) n > >>>>>>>>> 1121 printf("In header_post_config\n"); > >>>>>>>>> (gdb) n > >>>>>>>>> In header_post_config > >>>>>>>>> 1122 header_ssl_lookup = > >>>>>>>>> APR_RETRIEVE_OPTIONAL_FN(ssl_var_lookup); > >>>>>>>>> (gdb) n > >>>>>>>>> 1124 } > >>>>>>>>> (gdb) n > >>>>>>>>> 0x00000000004360c7 in ap_run_post_config (pconf=0x573138, > >>>>>>>>> plog=0x5a52c8, ptemp=0x5a72d8, s=0x59d3a8) at config.c:91 > >>>>>>>>> 91 AP_IMPLEMENT_HOOK_RUN_ALL(int, post_config, > >>>>>>>>> (gdb) n > >>>>>>>>> > >>>>>>>>> Program received signal SIGSEGV, Segmentation fault. > >>>>>>>>> 0x0000003518d6c1e1 in BN_num_bits () from /lib64/libcrypto.so.4 > >>>>>>>>> (gdb) bt full > >>>>>>>>> #0 0x0000003518d6c1e1 in BN_num_bits () from /lib64/libcrypto.so.4 > >>>>>>>>> No symbol table info available. > >>>>>>>>> #1 0x0000003518da8f4e in X509_ATTRIBUTE_create () from > >>>>>>>>> /lib64/libcrypto.so.4 > >>>>>>>>> No symbol table info available. > >>>>>>>>> #2 0x0000003518dadea2 in asn1_ex_i2c () from /lib64/libcrypto.so.4 > >>>>>>>>> No symbol table info available. > >>>>>>>>> #3 0x0000003518dadf79 in asn1_ex_i2c () from /lib64/libcrypto.so.4 > >>>>>>>>> No symbol table info available. > >>>>>>>>> #4 0x0000003518dae0e1 in ASN1_item_ex_i2d () from > >>>>>>>>> /lib64/libcrypto.so.4 > >>>>>>>>> No symbol table info available. > >>>>>>>>> #5 0x0000003518dae5f2 in ASN1_template_i2d () from > >>>>>>>>> /lib64/libcrypto.so.4 > >>>>>>>>> No symbol table info available. > >>>>>>>>> #6 0x0000003518dae28e in ASN1_item_ex_i2d () from > >>>>>>>>> /lib64/libcrypto.so.4 > >>>>>>>>> No symbol table info available. > >>>>>>>>> #7 0x0000003518dae3c6 in ASN1_item_i2d () from > >>>>>>>>> /lib64/libcrypto.so.4 > >>>>>>>>> No symbol table info available. > >>>>>>>>> #8 0x0000002a987d9d3a in ssl_pphrase_Handle (s=0x59d3a8, > >>>>>>>>> p=0x5a72d8) at ssl_engine_pphrase.c:505 > >>>>>>>>> mc = (SSLModConfigRec *) 0x571738 > >>>>>>>>> sc = (SSLSrvConfigRec *) 0x668c38 > >>>>>>>>> pServ = (server_rec *) 0x65fa48 > >>>>>>>>> cpVHostID = 0x60add0 "www.example.com:443" > >>>>>>>>> szPath = > >>>>>>>>> "/apps/httpd2222/conf/certs/apache1.whatever.com.key\000\177\000\000\000Ü׿\177\000\000\000à׿\177", > >>>>>>>>> '\0'<rep eats 15 > >>>>>>>>> times>, > >>>>>>>>> "\001\000\000\000°å¿\177\000\000\000;\000\000\000+\000\000\000\020Û¿\177", > >>>>>>>>> '\0'<repeats 35 times>, "à\224k", > >>>>>>>>> ' ---Type<return> > >>>>>>>>> to continue, or q<return> to quit--- > >>>>>>>>> \0'<repeats 13 times>, > >>>>>>>>> "øé¿\177\000\000\000\020\000\000\000;\000\000\000\016\000\000\000\001\000\001\000\001", > >>>>>>>>> '\0'<repeats 15 times>, "n\000\000\000\000\004\000\000°å¿\177", > >>>>>>>>> '\0'<repeats 11 times>, "\020Û¿\177\000\000\000+\000\000\000;", > >>>>>>>>> '\0'<repeats 35 times>... > >>>>>>>>> pPrivateKey = (EVP_PKEY *) 0x6ba670 > >>>>>>>>> asn1 = Variable "asn1" is not available. > >>>>>>>>> (gdb) > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> Not really sure what to make of that though :(... > >>>>>>>>> > >>>>>>>>> Jim > >>>>>>>> Hi, > >>>>>>>> > >>>>>>>> Ok, something interesting. I noticed that the servername in the > >>>>>>>> gdb output was example.whatever.com, so I changed ServerName in > >>>>>>>> extras/httpd-ssl.conf to "apache1.whatever.com". Apache still > >>>>>>>> segfaulted after that. > >>>>>>>> > >>>>>>>> I noticed that igdb was showing my router IP address for > >>>>>>>> "mod_unique_id" (whatever that is), so I thought it was because I > >>>>>>>> didn't have the hostname in /etc/hosts. So, I added > >>>>>>>> "apache1.whatever.com" to /etc/hosts, and also turned Apache > >>>>>>>> LogLevel to debug, and now I get a different error: > >>>>>>>> > >>>>>>>> [root@apachemodule dev]# /apps/httpd2222/bin/apachectl -k start -X > >>>>>>>> mod_headers-jl V0.09 - start calling OAM API > >>>>>>>> In register_hooks > >>>>>>>> In create_headers_dir_config > >>>>>>>> In create_headers_dir_config > >>>>>>>> In header_cmd > >>>>>>>> In header_inout_cmd > >>>>>>>> In parse_format_tag > >>>>>>>> In parse_misc_string > >>>>>>>> In create_headers_dir_config > >>>>>>>> In header_cmd > >>>>>>>> In header_inout_cmd > >>>>>>>> In parse_format_tag > >>>>>>>> In parse_misc_string > >>>>>>>> In header_post_config > >>>>>>>> *** glibc detected *** corrupted double-linked list: > >>>>>>>> 0x00000000006b9710 *** > >>>>>>>> /apps/httpd2222/bin/apachectl: line 78: 7599 > >>>>>>>> Aborted $HTTPD $ARGV > >>>>>>>> [root@apachemodule dev]# > >>>>>>>> > >>>>>>>> > >>>>>>>> Here's the Apache error_log: > >>>>>>>> > >>>>>>>> [Thu Jun 21 23:24:11 2012] [info] mod_unique_id: using ip addr > >>>>>>>> 72.215.225.9 > >>>>>>>> [Thu Jun 21 23:24:12 2012] [info] Init: Seeding PRNG with 144 bytes > >>>>>>>> of entropy > >>>>>>>> [Thu Jun 21 23:24:12 2012] [info] Loading certificate& private > >>>>>>>> key of SSL-aware server > >>>>>>>> [Thu Jun 21 23:29:51 2012] [info] mod_unique_id: using ip addr > >>>>>>>> 72.215.225.9 > >>>>>>>> [Thu Jun 21 23:29:52 2012] [info] Init: Seeding PRNG with 144 bytes > >>>>>>>> of entropy > >>>>>>>> [Thu Jun 21 23:29:52 2012] [info] Loading certificate& private > >>>>>>>> key of SSL-aware server > >>>>>>>> [Thu Jun 21 23:29:56 2012] [info] mod_unique_id: using ip addr > >>>>>>>> 72.215.225.9 > >>>>>>>> [Thu Jun 21 23:29:57 2012] [info] Init: Seeding PRNG with 144 bytes > >>>>>>>> of entropy > >>>>>>>> [Thu Jun 21 23:29:57 2012] [info] Loading certificate& private > >>>>>>>> key of SSL-aware server > >>>>>>>> [root@apachemodule dev]# > >>>>>>>> > >>>>>>>> > >>>>>>>> and here's the gdb backtrace: > >>>>>>>> > >>>>>>>> (gdb) b header_post_config > >>>>>>>> Function "header_post_config" not defined. > >>>>>>>> Make breakpoint pending on future shared library load? (y or [n]) y > >>>>>>>> Breakpoint 1 (header_post_config) pending. > >>>>>>>> (gdb) run -d /apps/httpd2222/ -f /apps/httpd2222/conf/httpd.conf > >>>>>>>> Starting program: /apps/httpd2222/bin/httpd -d /apps/httpd2222/ -f > >>>>>>>> /apps/httpd2222/conf/httpd.conf > >>>>>>>> [Thread debugging using libthread_db enabled] > >>>>>>>> [New Thread 182897610272 (LWP 7644)] > >>>>>>>> Breakpoint 2 at 0x2a9751ea90: file mod_headers.c, line 1121. > >>>>>>>> Pending breakpoint "header_post_config" resolved > >>>>>>>> mod_headers-jl V0.09 - start calling OAM API > >>>>>>>> In register_hooks > >>>>>>>> In create_headers_dir_config > >>>>>>>> In create_headers_dir_config > >>>>>>>> In header_cmd > >>>>>>>> In header_inout_cmd > >>>>>>>> In parse_format_tag > >>>>>>>> In parse_misc_string > >>>>>>>> In create_headers_dir_config > >>>>>>>> In header_cmd > >>>>>>>> In header_inout_cmd > >>>>>>>> In parse_format_tag > >>>>>>>> In parse_misc_string > >>>>>>>> [Switching to Thread 182897610272 (LWP 7644)] > >>>>>>>> > >>>>>>>> Breakpoint 2, header_post_config (pconf=0x573138, plog=0x5a52c8, > >>>>>>>> ptemp=0x5a72d8, s=0x59d3a8) at mod_headers.c:1121 > >>>>>>>> 1121 printf("In header_post_config\n"); > >>>>>>>> (gdb) n > >>>>>>>> 1120 { > >>>>>>>> (gdb) n > >>>>>>>> 1121 printf("In header_post_config\n"); > >>>>>>>> (gdb) n > >>>>>>>> In header_post_config > >>>>>>>> 1122 header_ssl_lookup = > >>>>>>>> APR_RETRIEVE_OPTIONAL_FN(ssl_var_lookup); > >>>>>>>> (gdb) n > >>>>>>>> 1124 } > >>>>>>>> (gdb) n > >>>>>>>> 0x00000000004360c7 in ap_run_post_config (pconf=0x573138, > >>>>>>>> plog=0x5a52c8, ptemp=0x5a72d8, s=0x59d3a8) at config.c:91 > >>>>>>>> 91 AP_IMPLEMENT_HOOK_RUN_ALL(int, post_config, > >>>>>>>> (gdb) n > >>>>>>>> *** glibc detected *** corrupted double-linked list: > >>>>>>>> 0x00000000006b9710 *** > >>>>>>>> > >>>>>>>> Program received signal SIGABRT, Aborted. > >>>>>>>> 0x000000351432e26d in raise () from /lib64/tls/libc.so.6 > >>>>>>>> (gdb) bt full > >>>>>>>> #0 0x000000351432e26d in raise () from /lib64/tls/libc.so.6 > >>>>>>>> No symbol table info available. > >>>>>>>> #1 0x000000351432fa6e in abort () from /lib64/tls/libc.so.6 > >>>>>>>> No symbol table info available. > >>>>>>>> #2 0x0000003514363641 in __libc_message () from /lib64/tls/libc.so.6 > >>>>>>>> No symbol table info available. > >>>>>>>> #3 0x0000003514369512 in _int_free () from /lib64/tls/libc.so.6 > >>>>>>>> No symbol table info available. > >>>>>>>> #4 0x0000003514369846 in free () from /lib64/tls/libc.so.6 > >>>>>>>> No symbol table info available. > >>>>>>>> #5 0x0000002a9790b6ba in R_free () from > >>>>>>>> /apps/netpoint/lib64/libobaccess.so > >>>>>>>> No symbol table info available. > >>>>>>>> #6 0x0000002a9792bc41 in X509_CINF_free () from > >>>>>>>> /apps/netpoint/lib64/libobaccess.so > >>>>>>>> No symbol table info available. > >>>>>>>> #7 0x0000002a9790ed98 in X509_free () from > >>>>>>>> /apps/netpoint/lib64/libobaccess.so > >>>>>>>> No symbol table info available. > >>>>>>>> #8 0x0000002a987d97b6 in ssl_pphrase_Handle (s=0x59d3a8, > >>>>>>>> p=0x5a72d8) at ssl_engine_pphrase.c:243 > >>>>>>>> mc = (SSLModConfigRec *) 0x571738 > >>>>>>>> sc = (SSLSrvConfigRec *) 0x668c38 > >>>>>>>> pServ = (server_rec *) 0x65fa48 > >>>>>>>> cpVHostID = 0x60add0 "apache1.whatever.com:443" > >>>>>>>> szPath = > >>>>>>>> "/apps/httpd2222/conf/certs/apache1.whatever.com.crt\000\177\000\000\000Ü׿\177\000\000\000à׿\177", > >>>>>>>> '\0'<repeats 15 times>, > >>>>>>>> "\001\000\000\000°å¿\177\000\000\000;\000\000\000+\000\000\000\020Û¿\177", > >>>>>>>> '\0'<repeats 35 times>, "à\224k", '\0'<repeats 13 times>, > >>>>>>>> "øé¿\177\000\000\000\020\000\000\000;\000\000\000\016\000\000\000\001\000\001\000\001", > >>>>>>>> '\0'<repeats 15 times>, "n\000\000\000\000\004\000\000°å¿\177", > >>>>>>>> '\0'<repeats 11 times>, "\020Û¿\177\000\000\000+\000\000\000;", > >>>>>>>> '\0'<repeats 35 times>... > >>>>>>>> pPrivateKey = Variable "pPrivateKey" is not available. > >>>>>>>> (gdb) > >>>>>>>> > >>>>>>>> > >>>>>>>> Jim > >>>>>>>> > >>>>>>> I guess the thing that I'm puzzled about is why it goes from > >>>>>>> ssl_pphrase_Handle (which is part of Apache code?) to X509_free () > >>>>>>> in the libobaccess.so? Apache shouldn't even be aware of the > >>>>>>> libobaccess stuff yet (since I think that Apache is still trying to > >>>>>>> initialize SSL processing)? > >>>>>>> > >>>>>>> Jim > >>>>>> It shouldn't unless you've declared a shutdown hook or a clean up hook > >>>>>> of sorts. The looks of that stack trace look strikingly like an > >>>>>> exception handler firing off. What are all of the hooks you have > >>>>>> running? > >>>>>> > >>>>>> Joe > >>>>>> -- > >>>>>> http://www.silverhawk.net/ > >>>>>> > >>>>> Hi, > >>>>> > >>>>> I'm basically using the original mod_headers.c as a starter code, and > >>>>> stuck some stuff in there. Haven't done anything with the hooks etc. > >>>>> yet, so whatever was there is there: > >>>>> > >>>>> static void register_hooks(apr_pool_t *p) > >>>>> { > >>>>> printf("mod_headers-jl V0.09 - start calling OAM API\n"); > >>>>> printf("In register_hooks\n"); > >>>>> ap_register_output_filter("FIXUP_HEADERS_OUT", > >>>>> ap_headers_output_filter, > >>>>> NULL, AP_FTYPE_CONTENT_SET); > >>>>> ap_register_output_filter("FIXUP_HEADERS_ERR", > >>>>> ap_headers_error_filter, > >>>>> NULL, AP_FTYPE_CONTENT_SET); > >>>>> ap_hook_pre_config(header_pre_config,NULL,NULL,APR_HOOK_MIDDLE); > >>>>> ap_hook_post_config(header_post_config,NULL,NULL,APR_HOOK_MIDDLE); > >>>>> ap_hook_insert_filter(ap_headers_insert_output_filter, NULL, > >>>>> NULL, APR_HOOK_LAST); > >>>>> ap_hook_insert_error_filter(ap_headers_insert_error_filter, > >>>>> NULL, NULL, APR_HOOK_LAST); > >>>>> ap_hook_fixups(ap_headers_fixup, NULL, NULL, APR_HOOK_LAST); > >>>>> ap_hook_post_read_request(ap_headers_early, NULL, NULL, > >>>>> APR_HOOK_FIRST); > >>>>> } > >>>>> > >>>>> module AP_MODULE_DECLARE_DATA headers_module = > >>>>> { > >>>>> STANDARD20_MODULE_STUFF, > >>>>> create_headers_dir_config, /* dir config creater */ > >>>>> merge_headers_config, /* dir merger --- default is to > >>>>> override */ > >>>>> NULL, /* server config */ > >>>>> NULL, /* merge server configs */ > >>>>> headers_cmds, /* command apr_table_t */ > >>>>> register_hooks /* register hooks */ > >>>>> }; > >>>>> > >>>>> I guess that there is a: > >>>>> > >>>>> ap_hook_post_config(header_post_config,NULL,NULL,APR_HOOK_MIDDLE); > >>>>> > >>>>> in that register_hooks(), but it was there from the original code. > >>>>> > >>>>> BTW, I was checking, and it looks like there's an X509_free() function > >>>>> in both openssl and in libobaccess.so (used 'nm --dynamic<file>), so > >>>>> I'm wondering if things are confused, and maybe Apache code expects to > >>>>> call the openssl X509_free, but somehow ends up calling the > >>>>> X509_free() in libobaccess.so? > >>>>> > >>>>> What would cause something like that? If two .so files have the same > >>>>> function, how to avoid conflicts? > >>>>> > >>>>> Jim > >>>> Jim, you may have hit that on the head. I'm not sure of the load order > >>>> and looking up symbols, but it could very well change things. Try > >>>> re-ordering or adding a LoadFile before the SSL module and see if that > >>>> changes things. > >>> Look up order usually is load order. So if mod_ssl loads first and loads > >>> its dependency libcrypto, the symbol will always be found there even if > >>> the implementation in libobaccess.so is needed - and vice versa. > >>> > >>> On Solaris there is a -Bdirect linker flag which changes runtime search > >>> order so that each module would find a needed symbol first inits direct > >>> dependencies. Unfortunately AFAIK notihing similar exists for Linux. > >>> > >>> You would need to find ssl libs and libobaccess with compatible symbols. > >>> Best would be a libobaccess which is dynamically linked against a > >>> compatible version of libcrypto instead of - as it seems - being > >>> statically linked against it. > >>> > >>> Regards, > >>> > >>> Rainer > >>> > >> > >> Hi Rainer, > >> > >> Here's exactly what I found: > >> > >> [root@apachemodule ~]# ldd /apps/netpoint/lib64/libobaccess.so > >> libnsl.so.1 => /lib64/libnsl.so.1 (0x0000002a95ab8000) > >> libdl.so.2 => /lib64/libdl.so.2 (0x0000002a95bd0000) > >> libpthread.so.0 => /lib64/tls/libpthread.so.0 > >> (0x0000002a95cd3000) > >> libstdc++.so.6 => /lib64/libstdc++.so.6 (0x0000002a95de9000) > >> libm.so.6 => /lib64/tls/libm.so.6 (0x0000002a95fd9000) > >> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000002a9615f000) > >> libc.so.6 => /lib64/tls/libc.so.6 (0x0000002a9626d000) > >> /lib64/ld-linux-x86-64.so.2 (0x000000552aaaa000) > >> [root@apachemodule ~]# > >> [root@apachemodule ~]# nm --dynamic /apps/netpoint/lib64/libobaccess.so | > >> grep "X509_free" > >> 00000000002d8d60 T X509_free > >> > >> > >> [root@apachemodule ~]# ldd /lib64/libssl.so.0.9.7a > >> libgssapi_krb5.so.2 => /usr/lib64/libgssapi_krb5.so.2 > >> (0x0000003518700000) > >> libkrb5.so.3 => /usr/lib64/libkrb5.so.3 (0x0000003519000000) > >> libcom_err.so.2 => /lib64/libcom_err.so.2 (0x0000003518500000) > >> libk5crypto.so.3 => /usr/lib64/libk5crypto.so.3 > >> (0x0000003518900000) > >> libcrypto.so.4 => /lib64/libcrypto.so.4 (0x0000003518d00000) > >> libdl.so.2 => /lib64/libdl.so.2 (0x0000003514800000) > >> libz.so.1 => /usr/lib64/libz.so.1 (0x0000003514c00000) > >> libc.so.6 => /lib64/tls/libc.so.6 (0x0000003514300000) > >> libresolv.so.2 => /lib64/libresolv.so.2 (0x0000003518300000) > >> /lib64/ld-linux-x86-64.so.2 (0x000000552aaaa000) > >> [root@apachemodule ~]# > >> [root@apachemodule ~]# nm --dynamic /lib64/libssl.so.0.9.7a | grep > >> "X509_free" > >> U X509_free > >> > >> > >> [root@apachemodule ~]# ldd /lib64/libcrypto.so.0.9.7a > >> libgssapi_krb5.so.2 => /usr/lib64/libgssapi_krb5.so.2 > >> (0x0000003518700000) > >> libkrb5.so.3 => /usr/lib64/libkrb5.so.3 (0x0000003519000000) > >> libcom_err.so.2 => /lib64/libcom_err.so.2 (0x0000003518500000) > >> libk5crypto.so.3 => /usr/lib64/libk5crypto.so.3 > >> (0x0000003518900000) > >> libdl.so.2 => /lib64/libdl.so.2 (0x0000003514800000) > >> libz.so.1 => /usr/lib64/libz.so.1 (0x0000003514c00000) > >> libc.so.6 => /lib64/tls/libc.so.6 (0x0000003514300000) > >> libresolv.so.2 => /lib64/libresolv.so.2 (0x0000003518300000) > >> /lib64/ld-linux-x86-64.so.2 (0x000000552aaaa000) > >> [root@apachemodule ~]# > >> [root@apachemodule ~]# nm --dynamic /lib64/libcrypto.so.0.9.7a | grep > >> "X509_free" > >> 0000003518da9860 T X509_free > >> > >> > >> So: > >> > >> - liboaccess.so doesn't REFERENCE libcrypto, but it HAS an X509_free() > >> function in it. > >> - libssl.so.REFERENCES libcrypto, AND it HAS an X509_free function in it. > >> - libcrypto also has an X509_free function in it. > >> > >> From your msg above, are you saying that this situation can't be resolved > >> by trying to change load order? > >> > >> And the only way is what you said in your last paragraph (find another > >> libobccess.so)? > >> > >> Jim > >> > >> > > Hi, > > > > Also, BTW, the call to X509_free is apparently because in > > ssl_engine_pphrase.c, which it seems is part of mod_ssl, it has: > > > > /* > > * Free the X509 structure > > */ > > X509_free(pX509Cert); > > > > but that's calling the X509_free() in libobaccess.so, rather than the > > X509_free() in libcrypt.so? > > > > Jim > > The way I read it, the load order can help. Did you try changing > locations of the LoadFile for libobaccess.so to before or after the > LoadModule for the SSL engine? > > Joe > -- > http://www.silverhawk.net/
Hi, No, not yet. I was waiting for more feedback from here :). But I will do it now and post back. Right now the LoadFile for libobaccess.so is in front of (earlier in httpd.conf) the Loadmodule. Jim