Hello Perrin,
Thank you for your response.
PH> You could:
PH> - Make s $SIG{__DIE__} handler
PH> - Wrap your application in an eval {}
PH> The data you want is available from either the standard apache log
PH> formats or perl code in Apache::Status, so there's no need to worry
PH> about writing C here.
The trouble is, our code isn't getting invoked at all. Mod_perl (or
something lower-level like that) is dying complaining that our code
can't be found on the hits where this happens. As an aside, we already
have similar eval traps, including one over the main handler code in
an attempt to provide the best error handling for our clients as
possible.
A mod_perl handler is set up (in apache conf), such as a content
handler, like this:
PerlHandler XX::Clients::YY::ContentHandler
When that resource is invoked by via a web request (GET or POST,
usually), mod_perl will load the module
"XX::Clients::YY::ContentHandler" and then attempt to call a method
named "handler." It is this invocation of "handler" by mod_perl that
is randomly failing and we're trying to find more information about.
We added an "ErrorDocument 500" statement in the Apache config, and
set it to a CGI program, which dumps out the symbol table using
Devel::Symdump as well as dumping out %INC.
The symbol dump shows (only relevant lines copied, there were hundreds
of lines output)
arrays
[nothing related to this]
functions
XX::Clients::YY::ContentHandler::handler
hashes
[nothing related to this]
ios
[nothing related to this]
packages
XX::Clients::YY
XX::Clients::YY::ContentHandler
scalars
XX::Clients::YY::
XX::Clients::YY::ContentHandler::
XX::Clients::YY::ContentHandler::AUTOLOAD
XX::Clients::YY::ContentHandler::DESTROY
XX::Clients::YY::ContentHandler::handler
**NOTE** The last item listed under "functions" is is what mod_perl
complained was undefined in the error log; so it seems like it was
partially defined (since some things exist for it in the symbol
table).
%INC shows many, many modules, with the module's relative path as the
key and the full path as the value. However, NOTHING exists in it
regarding XX::Clients::YY, even though there are entries for other
working clients and has normal entries on normal hits.
---------------------------------------------------------------------
When everything is working normally, here is what we see in the Symbol
table related to our module for client YY (tons of stuff related to
this module, as before, the unrelated content was omitted from this
post):
arrays
XX::Clients::YY::ContentHandler::EXPORT
XX::Clients::YY::ContentHandler::EXPORT_FAIL
XX::Clients::YY::ContentHandler::ISA
XX::Clients::YY::ContentHandler::SUPER::ISA
functions
XX::Clients::YY::Custom::display_home
XX::Clients::YY::ContentHandler::_find_element
XX::Clients::YY::ContentHandler::_fix_menu
XX::Clients::YY::ContentHandler::_get_category_ids
XX::Clients::YY::ContentHandler::_num_spreads
XX::Clients::YY::ContentHandler::_record_audit_trail
XX::Clients::YY::ContentHandler::add_to_cart
XX::Clients::YY::ContentHandler::ceil
XX::Clients::YY::ContentHandler::check_children
XX::Clients::YY::ContentHandler::classes
XX::Clients::YY::ContentHandler::cleanup
XX::Clients::YY::ContentHandler::display_approval
XX::Clients::YY::ContentHandler::display_asset_management
XX::Clients::YY::ContentHandler::display_asset_search
XX::Clients::YY::ContentHandler::display_asset_search_cart
XX::Clients::YY::ContentHandler::display_asset_search_results
XX::Clients::YY::ContentHandler::display_asset_tree
XX::Clients::YY::ContentHandler::display_customized_preview
XX::Clients::YY::ContentHandler::display_dealer_info
XX::Clients::YY::ContentHandler::display_export
XX::Clients::YY::ContentHandler::display_forgot_password
XX::Clients::YY::ContentHandler::display_load
XX::Clients::YY::ContentHandler::display_photo
XX::Clients::YY::ContentHandler::display_product_results
XX::Clients::YY::ContentHandler::display_profile
XX::Clients::YY::ContentHandler::display_resize
XX::Clients::YY::ContentHandler::display_saved_documents
XX::Clients::YY::ContentHandler::display_step
XX::Clients::YY::ContentHandler::display_text
XX::Clients::YY::ContentHandler::display_tree
XX::Clients::YY::ContentHandler::display_welcome
XX::Clients::YY::ContentHandler::display_work_area
XX::Clients::YY::ContentHandler::do_nothing
XX::Clients::YY::ContentHandler::document_export
XX::Clients::YY::ContentHandler::download_zip
XX::Clients::YY::ContentHandler::fields
XX::Clients::YY::ContentHandler::funcs
XX::Clients::YY::ContentHandler::generate_preview
XX::Clients::YY::ContentHandler::handler
XX::Clients::YY::ContentHandler::new
XX::Clients::YY::ContentHandler::process_approval
XX::Clients::YY::ContentHandler::process_approval_request
XX::Clients::YY::ContentHandler::process_asset_category
XX::Clients::YY::ContentHandler::process_asset_search
XX::Clients::YY::ContentHandler::process_category
XX::Clients::YY::ContentHandler::process_category_select_pulldown
XX::Clients::YY::ContentHandler::process_creator
XX::Clients::YY::ContentHandler::process_dealer_info
XX::Clients::YY::ContentHandler::process_email_cart
XX::Clients::YY::ContentHandler::process_empty_cart
XX::Clients::YY::ContentHandler::process_export
XX::Clients::YY::ContentHandler::process_export_cart
XX::Clients::YY::ContentHandler::process_images_lookup
XX::Clients::YY::ContentHandler::process_images_search
XX::Clients::YY::ContentHandler::process_load
XX::Clients::YY::ContentHandler::process_photo
XX::Clients::YY::ContentHandler::process_profile
XX::Clients::YY::ContentHandler::process_save
XX::Clients::YY::ContentHandler::process_step
XX::Clients::YY::ContentHandler::process_text
XX::Clients::YY::ContentHandler::remove_from_cart
XX::Clients::YY::ContentHandler::start_adcreator
hashes
XX::Clients::YY::Custom::funcs
ios
XX::Clients::YY::ContentHandler::FILE
XX::Clients::YY::ContentHandler::PHOTO
packages
XX::Clients::YY
XX::Clients::YY::Config
XX::Clients::YY::Custom
XX::Clients::YY::ContentHandler
XX::Clients::YY::ContentHandler::::ISA::CACHE
XX::Clients::YY::ContentHandler::SUPER
scalars
XX::Clients::YY::
XX::Clients::YY::Config::
XX::Clients::YY::Config::BEGIN
XX::Clients::YY::Config::base_path
XX::Clients::YY::Config::base_url
XX::Clients::YY::Config::client_assets
XX::Clients::YY::Config::constrain
XX::Clients::YY::Config::data_path
XX::Clients::YY::Config::download_path
XX::Clients::YY::Config::download_url
XX::Clients::YY::Config::help
XX::Clients::YY::Config::logos_path
XX::Clients::YY::Config::ports
XX::Clients::YY::Config::preview_path
XX::Clients::YY::Config::preview_url
XX::Clients::YY::Config::remember_me_cookie
XX::Clients::YY::Config::saved
XX::Clients::YY::Config::template_path
XX::Clients::YY::Config::unprotected_path
XX::Clients::YY::Custom::
XX::Clients::YY::Custom::VERSION
XX::Clients::YY::Custom::account_type
XX::Clients::YY::Custom::adcreator_name
XX::Clients::YY::Custom::can
XX::Clients::YY::Custom::ccs_domain
XX::Clients::YY::Custom::check_welcome
XX::Clients::YY::Custom::creator_obj
XX::Clients::YY::Custom::display_approval
XX::Clients::YY::Custom::display_asset_category
XX::Clients::YY::Custom::display_asset_management
XX::Clients::YY::Custom::display_asset_search
XX::Clients::YY::Custom::display_asset_search_cart
XX::Clients::YY::Custom::display_asset_search_results
XX::Clients::YY::Custom::display_asset_tree
XX::Clients::YY::Custom::display_customized_preview
XX::Clients::YY::Custom::display_dealer_info
XX::Clients::YY::Custom::display_export
XX::Clients::YY::Custom::display_forgot_password
XX::Clients::YY::Custom::display_home
XX::Clients::YY::Custom::display_load
XX::Clients::YY::Custom::display_photo
XX::Clients::YY::Custom::display_product_results
XX::Clients::YY::Custom::display_profile
XX::Clients::YY::Custom::display_resize
XX::Clients::YY::Custom::display_saved_documents
XX::Clients::YY::Custom::display_step
XX::Clients::YY::Custom::display_text
XX::Clients::YY::Custom::display_tree
XX::Clients::YY::Custom::display_welcome
XX::Clients::YY::Custom::document_export
XX::Clients::YY::Custom::domain
XX::Clients::YY::Custom::funcs
XX::Clients::YY::Custom::generate_preview
XX::Clients::YY::Custom::global_session
XX::Clients::YY::Custom::handler_pre_method_call
XX::Clients::YY::Custom::highlight_elements_in_preview
XX::Clients::YY::Custom::hit_debug
XX::Clients::YY::Custom::import
XX::Clients::YY::Custom::load_account_package
XX::Clients::YY::Custom::load_account_params
XX::Clients::YY::Custom::preview_horizontal_size
XX::Clients::YY::Custom::preview_path
XX::Clients::YY::Custom::preview_quality
XX::Clients::YY::Custom::preview_resolution
XX::Clients::YY::Custom::preview_size_percent
XX::Clients::YY::Custom::preview_units
XX::Clients::YY::Custom::preview_vertical_size
XX::Clients::YY::Custom::process_approval
XX::Clients::YY::Custom::process_approval_request
XX::Clients::YY::Custom::process_asset_search
XX::Clients::YY::Custom::process_category
XX::Clients::YY::Custom::process_creator
XX::Clients::YY::Custom::process_dealer_info
XX::Clients::YY::Custom::process_email_cart
XX::Clients::YY::Custom::process_export
XX::Clients::YY::Custom::process_load
XX::Clients::YY::Custom::process_photo
XX::Clients::YY::Custom::process_profile
XX::Clients::YY::Custom::process_save
XX::Clients::YY::Custom::process_step
XX::Clients::YY::Custom::process_text
XX::Clients::YY::Custom::r
XX::Clients::YY::Custom::use_apache_filter
XX::Clients::YY::ContentHandler::
XX::Clients::YY::ContentHandler::::ISA::CACHE::
XX::Clients::YY::ContentHandler::AUTOLOAD
XX::Clients::YY::ContentHandler::BEGIN
XX::Clients::YY::ContentHandler::DESTROY
XX::Clients::YY::ContentHandler::EXPORT
XX::Clients::YY::ContentHandler::EXPORT_FAIL
XX::Clients::YY::ContentHandler::FILE
XX::Clients::YY::ContentHandler::ISA
XX::Clients::YY::ContentHandler::PHOTO
XX::Clients::YY::ContentHandler::SUPER::
XX::Clients::YY::ContentHandler::SUPER::ISA
XX::Clients::YY::ContentHandler::SUPER::ContentHandler
XX::Clients::YY::ContentHandler::SUPER::new
XX::Clients::YY::ContentHandler::_account_load
XX::Clients::YY::ContentHandler::_find_element
XX::Clients::YY::ContentHandler::_fix_menu
XX::Clients::YY::ContentHandler::_generate_preview
XX::Clients::YY::ContentHandler::_get_category_ids
XX::Clients::YY::ContentHandler::_init
XX::Clients::YY::ContentHandler::_num_spreads
XX::Clients::YY::ContentHandler::_record_audit_trail
XX::Clients::YY::ContentHandler::a
XX::Clients::YY::ContentHandler::account_type
XX::Clients::YY::ContentHandler::adcreator
XX::Clients::YY::ContentHandler::adcreator_name
XX::Clients::YY::ContentHandler::add_attribute
XX::Clients::YY::ContentHandler::add_to_cart
XX::Clients::YY::ContentHandler::b
XX::Clients::YY::ContentHandler::can
XX::Clients::YY::ContentHandler::ccs_domain
XX::Clients::YY::ContentHandler::ceil
XX::Clients::YY::ContentHandler::check_children
XX::Clients::YY::ContentHandler::classes
XX::Clients::YY::ContentHandler::cleanup
XX::Clients::YY::ContentHandler::creator_obj
XX::Clients::YY::ContentHandler::display_approval
XX::Clients::YY::ContentHandler::display_asset_management
XX::Clients::YY::ContentHandler::display_asset_search
XX::Clients::YY::ContentHandler::display_asset_search_cart
XX::Clients::YY::ContentHandler::display_asset_search_results
XX::Clients::YY::ContentHandler::display_asset_tree
XX::Clients::YY::ContentHandler::display_customized_preview
XX::Clients::YY::ContentHandler::display_dealer_info
XX::Clients::YY::ContentHandler::display_export
XX::Clients::YY::ContentHandler::display_forgot_password
XX::Clients::YY::ContentHandler::display_load
XX::Clients::YY::ContentHandler::display_photo
XX::Clients::YY::ContentHandler::display_product_results
XX::Clients::YY::ContentHandler::display_profile
XX::Clients::YY::ContentHandler::display_resize
XX::Clients::YY::ContentHandler::display_saved_documents
XX::Clients::YY::ContentHandler::display_step
XX::Clients::YY::ContentHandler::display_text
XX::Clients::YY::ContentHandler::display_tree
XX::Clients::YY::ContentHandler::display_welcome
XX::Clients::YY::ContentHandler::display_work_area
XX::Clients::YY::ContentHandler::do_nothing
XX::Clients::YY::ContentHandler::document_export
XX::Clients::YY::ContentHandler::domain
XX::Clients::YY::ContentHandler::download_zip
XX::Clients::YY::ContentHandler::fields
XX::Clients::YY::ContentHandler::funcs
XX::Clients::YY::ContentHandler::generate_preview
XX::Clients::YY::ContentHandler::global_session
XX::Clients::YY::ContentHandler::handler
XX::Clients::YY::ContentHandler::handler_pre_method_call
XX::Clients::YY::ContentHandler::highlight_elements_in_preview
XX::Clients::YY::ContentHandler::hit_debug
XX::Clients::YY::ContentHandler::import
XX::Clients::YY::ContentHandler::isa
XX::Clients::YY::ContentHandler::load_account_package
XX::Clients::YY::ContentHandler::load_account_params
XX::Clients::YY::ContentHandler::new
XX::Clients::YY::ContentHandler::preview_horizontal_size
XX::Clients::YY::ContentHandler::preview_path
XX::Clients::YY::ContentHandler::preview_quality
XX::Clients::YY::ContentHandler::preview_resolution
XX::Clients::YY::ContentHandler::preview_size_percent
XX::Clients::YY::ContentHandler::preview_units
XX::Clients::YY::ContentHandler::preview_vertical_size
XX::Clients::YY::ContentHandler::process_approval
XX::Clients::YY::ContentHandler::process_approval_request
XX::Clients::YY::ContentHandler::process_asset_category
XX::Clients::YY::ContentHandler::process_asset_search
XX::Clients::YY::ContentHandler::process_category
XX::Clients::YY::ContentHandler::process_category_select_pulldown
XX::Clients::YY::ContentHandler::process_creator
XX::Clients::YY::ContentHandler::process_dealer_info
XX::Clients::YY::ContentHandler::process_email_cart
XX::Clients::YY::ContentHandler::process_empty_cart
XX::Clients::YY::ContentHandler::process_export
XX::Clients::YY::ContentHandler::process_export_cart
XX::Clients::YY::ContentHandler::process_images_lookup
XX::Clients::YY::ContentHandler::process_images_search
XX::Clients::YY::ContentHandler::process_load
XX::Clients::YY::ContentHandler::process_photo
XX::Clients::YY::ContentHandler::process_profile
XX::Clients::YY::ContentHandler::process_save
XX::Clients::YY::ContentHandler::process_step
XX::Clients::YY::ContentHandler::process_text
XX::Clients::YY::ContentHandler::r
XX::Clients::YY::ContentHandler::remove_from_cart
XX::Clients::YY::ContentHandler::start_adcreator
XX::Clients::YY::ContentHandler::test_attribute
XX::Clients::YY::ContentHandler::use_apache_filter
main::_</usr/local/lib/perl5/site_perl/5.8.6/XX/Clients/YY/Config.pm
main::_</usr/local/lib/perl5/site_perl/5.8.6/XX/Clients/YY/Custom.pm
main::_</usr/local/lib/perl5/site_perl/5.8.6/XX/Clients/YY/ContentHandler.pm
and here is what we normally see in %INC:
'XX/Clients/YY/Config.pm' =>
'/usr/local/lib/perl5/site_perl/5.8.6/XX/Clients/YY/Config.pm',
'XX/Clients/YY/Custom.pm' =>
'/usr/local/lib/perl5/site_perl/5.8.6/XX/Clients/YY/Custom.pm',
'XX/Clients/YY/ContentHandler.pm' =>
'/usr/local/lib/perl5/site_perl/5.8.6/XX/Clients/YY/ContentHandler.pm',
============================================================================
So, to recap:
Normally, %INC and the perl symbol table has plenty of data about our
module XX::Clients::YY::ContentHandler and the "handler" method is
invoked just fine by mod_perl.
On some "random" (we have not been able to detect an occurrence pattern
or reliably reproduce the problem) hits, we get an error saying
"handler" is undefined. When this happens, %INC hash is missing the
module and the symbol table has some information but is lacking the
vast majority of it.
Mod_perl/perl/apache/something other than us, not us, controls the
loading and execution of our module, and we need help tracing down and
fixing this problem.
Does anyone have any ideas what would possibly cause items to be in
the symbol table for a module, but the module wouldn't be in %INC or
seemingly in memory at all (since we got an 'Undefined subroutine'
error when mod_perl tried to invoke the "handler" method)?
Please, if anyone has more ideas on things to try, we'd be delighted
to hear them.
--
James
"A computer once beat me at chess, but it was no match for me at
kick boxing." -- Emo Philips
"There are 10 types of people in the world. Those who understand
binary, and those who don't." -- Unknown
