Error in log - can't find ANONSUB's 'anon3' entry. - plus 500 response from server
Hi, [Apologies if this has hit the list before - I tried a few weeks ago and my message just seemed to vanish; not sure if my address isn't allowed to post or the list doesn't send the mail to the post author - we have been trying to implement a PSGI alternative version but have come to a brick wall on that] Our system admins have switched from using mpm_prefork (for over 10 years) to using mpm_event - this has improved memory usage significantly but, under periods of high load, we are getting : can't find ANONSUB's 'anon3' entry. can't find ANONSUB's 'anon4' entry. can't find ANONSUB's 'anon3' entry. I've Googled the error and found nothing of any use. I've checked out the mod_perl source code and found the error but have failed to make much further progress, We have reasonable competence in C and many years experience of Perl (although there are certainly gaps in our knowledge and bad habits). Our handler set-up is done as follows: PerlModule Zymonic::MP [Misc PerlSetVar commands] PerlPostConfigHandler Zymonic::MP::post_config SetHandler perl-script PerlResponseHandler Zymonic::MP I'm hoping we can make some progress on this issue with help from this mailing list... Key questions: 1) Is "can't find ANONSUB's 'anon3' entry" a known error with likely causes (I can only a handful of references on Google with none solved and none in the mailing list archives)? 2) The 'ANONSUB' error appears to be thrown by modperl_handler_anon_get which is looking for a sub in the array maintained by the modper_handler_* methods however I can't find how these methods are being called - the furthest back I can get from static analysis is modperl_handler_push_handlers calling modperl_handler_new_from_sv but I can't find any calls to modperl_handler_push_handlers. modperl_handler_new_from_sv is also called from the 'filters' code - but, as far as I know, we're not using any filters only a Response Handler. The question is what connects our "Perl*Handler" through to the modperl_handler_anon* functions? 3) There are plenty of MP_TRACE_h calls that appear to debug/log information - can this be enabled in configuration or at compile time and can it be made to log to file rather than having to use gdb with threads? I've not (yet) raised a bug since at the moment we can't make this easily reproducible - we just know that, on our existing systems with plenty of data, if we enable mpm_event instead of mpm_prefork we get these errors. We're trying to narrow down the circumstances and find a way of demonstrating it from a clean install with scripted steps, but as yet have not found a way of doing that. Thanks in advance, Alex Masidlover -- Technical Director Medoc Computers Ltd 0115 986 8786 Meadow House, Meadow Lane, Nottingham NG2 3HS www.medoc.com amasidlo...@medoc.co.uk Company registered in England and Wales with company number 1583691 VAT Registration number: GB 352 7406 60 *Any emails containing pictures and other inappropriate content are strictly prohibited. To comply with PCI-DSS we are unable to accept any logs files or attachments that may contain credit or debit card details. This e-mail contains confidential information and is for the exclusive use of the addressee/s. If you are not the addressee, then any distribution, copying or use of this e-mail is prohibited. If received in error, please advise the sender and delete it immediately. We accept no liability for any loss or damage suffered by any person arising from the use of this e-mail
Re: Bug - Strange issue with mod_perl 2.0.10 / Apache 2.4 corrupting nfreeze data
On Sun, 2016-04-03 at 17:03 +0200, Vincent Veyron wrote:
> On Sun, 03 Apr 2016 14:11:23 +0100
> Alex Masidlover wrote:
>
> >
> >
> > This has all worked perfectly up until I upgraded to Apache 2.4 /
> > mod_perl 2.0.10 -
> After upgrading to Apache 2.4 and mod_perl 2.0.9, I had to make those
> two changes to my application :
>
> In a PerlOutputFilterHandler, change '$content .= $buffer' to
> '$content .= decode_utf8($buffer)'
>
> And in response handlers, change '$args{$_} = $req->param($_)' to
> '$args{$_} = decode_utf8($req->param($_))'
>
> Not sure it applies to your case, but something changed in Apache 2.4
> concerning UTF-8 data.
>
> If I understood correctly, anything that goes through APR::Table is
> considered UTF-8, however the SvUTF8 flag is not set, so you get
> double encoding when processing your data.
>
Thanks for the suggestion - its taken a while but I've manage to free
up a machine to roll forwards and test on; unfortunately I've got no
experience with Apache and filters. I've come up with this:
package Zymonic::Decryptor::Filter;
use strict;
use base qw(Apache2::Filter);
use Encode qw(decode);
use constant BUFF_LEN => 1024;
use Apache2::Const -compile => 'OK';
sub utf8_filter : FilterRequestHandler {
my $f = shift;
my $content = '';
while($f->read(my $buffer, BUFF_LEN)) {
$content .= decode_utf8($buffer);
}
$f->print($content);
return Apache2::Const::OK;
}
but when I add it with:
PerlOutputFilterHandler Zymonic::Decryptor::Filter::utf8_filter
I get:
:Apache2 IO flush: (500) Unknown error 500 at -e line 0
in the Apache error log whenever I hit the server with a request.
I'm guessing I've done something very wrong with the filter, but I can
find very few examples of how to use Apache2::Filter.
Thanks,
Alex
--
Technical Director - Zednax Limited
W: http://www.zednax.com
T: +44 333 444 0160
F: +44 161 660 8010
Zednax Limited is registered in England and Wales, Company no. 05321754.
Registered address: Meadow House, Meadow Lane, Nottingham, NG2 3HS.
Zednax Limited is VAT registered, VAT registration no. GB 855 4468 92.
Bug - Strange issue with mod_perl 2.0.10 / Apache 2.4 corrupting nfreeze data
-8<-- Start Bug Report 8<-- 1. Problem Description: I'm running Perl 5.20.2 on a Gentoo virtual machine. We've had some code in place that uses a mod_perl handler to send complex hashref based structures between two servers using nfreeze / thaw; the code needed to be high performance therefore we used nfreeze/thaw rather than XML or similar. This has been working correctly on a variety of architectures since July 2014. The mod_perl server does: print nfreeze( $message_hash ) The client connects to the server use LWP::UserAgent and does: my $message = thaw( $response->decoded_content() ); The message from client to server is similarly encoded with nfreeze / thaw in a cgi parameter in multi-part-form-data. This has all worked perfectly up until I upgraded to Apache 2.4 / mod_perl 2.0.10 - I've even downgraded just those two packages to mod_perl 2.0.8 / Apache 2.2.31 and it starts working again. The message from the client gets to the server correctly and the server logs and processes the message correctly, but it then fails with an 'Out of memory!' error from the client when the response is received (due to this bug in Storable.pm) - however, regardless of that bug, something is corrupting the data in transit... I've picked apart what's going on and found there are an additional two bytes being inserted somehow. I've recorded the data in a /tmp/ file before the print statement in the mod_perl handler and I've used perl -d to inspect what's received: DB<3> say "0x$_" for unpack "(H2)*", $response->content() 0x05 0x0a 0x03 0x00 0x00 0x00 0x01 0x08 0xc2 0x81 0x00 0x00 0x00 0x02 0x6f 0x6b However, the server sent: 0x05 0x0a 0x03 0x00 0x00 0x00 0x01 0x08 0x00 0x00 0x00 0x02 0x6f 0x6b So somehow 2 bytes have appeared in the middle of the message 0x08 and 0xc2. I've also tried adding Content-Type => 'x-application/binary' and Content-Length headers to see if that fixes the issue. Thanks! 2. Used Components and their Configuration: *** mod_perl version 2.10 *** using /usr/lib64/perl5/vendor_perl/5.20.2/x86_64-linux-thread- multi/Apache2/BuildConfig.pm *** Makefile.PL options: MP_APR_CONFIG => /usr/bin/apr-1-config MP_APR_LIB => aprext MP_APXS=> /usr/sbin/apxs2 MP_COMPAT_1X => 1 MP_DEBUG => 0 MP_GENERATE_XS => 1 MP_LIBNAME => mod_perl MP_TRACE => 0 MP_USE_DSO => 1 *** The httpd binary was not found *** (apr|apu)-config linking info -laprutil-1 -lldap -llber -ldb-4.8 -lgdbm -lexpat -L/var/tmp/portage/dev-libs/apr-util-1.5.4/temp -lapr-1 -luuid -lrt -lcrypt -lpthread -ldl *** /usr/bin/perl -V Summary of my perl5 (revision 5 version 20 subversion 2) configuration: Platform: osname=linux, osvers=2.6.32-openvz-feoktistov.1, archname=x86_64- linux-thread-multi uname='linux alex.zednax.com 2.6.32-openvz-feoktistov.1 #2 smp wed apr 6 14:38:37 bst 2011 x86_64 intel(r) xeon(r) cpu l5630 @ 2.13ghz genuineintel gnulinux ' config_args='-des -Duseshrplib -Darchname=x86_64-linux-thread -Dcc=x86_64-pc-linux-gnu-gcc -Doptimize=-O -pipe -Dldflags=-Wl,-O1 -Wl,--as-needed -Dprefix=/usr -Dinstallprefix=/usr -Dsiteprefix=/usr/local -Dvendorprefix=/usr -Dscriptdir=/usr/bin -Dprivlib=/usr/lib64/perl5/5.20.2 -Darchlib=/usr/lib64/perl5/5.20.2/x86_64-linux-thread-multi -Dsitelib=/usr/local/lib64/perl5/5.20.2 -Dsitearch=/usr/local/lib64/perl5/5.20.2/x86_64-linux-thread-multi -Dvendorlib=/usr/lib64/perl5/vendor_perl/5.20.2 -Dvendorarch=/usr/lib64/perl5/vendor_perl/5.20.2/x86_64-linux-thread- multi -Dman1dir=/usr/share/man/man1 -Dman3dir=/usr/share/man/man3 -Dsiteman1dir=/usr/local/man/man1 -Dsiteman3dir=/usr/local/man/man3 -Dvendorman1dir=/usr/share/man/man1 -Dvendorman3dir=/usr/share/man/man3 -Dman1ext=1 -Dman3ext=3pm -Dlibperl=libperl.so.5.20.2 -Dlocincpth=/usr/include -Dglibpth=/lib64 /usr/lib64 -Duselargefiles -Dd_semctl_semun -Dcf_by=Gentoo -Dmyhostname=localhost -Dperladmin=root @localhost -Dinstallusrbinperl=n -Ud_csh -Uusenm -Di_ndbm -Di_gdbm -Di_db -Dusethreads -DDEBUGGING=none -Dinc_version_list=5.20.0/x86_64- linux-thread-multi 5.20.0 5.20.1/x86_64-linux-thread-multi 5.20.1 - Dlibpth=/usr/local/lib64 /lib64 /usr/lib64 -Dnoextensions=ODBM_File' hint=recommended, useposix=true, d_sigaction=define useithreads=define, usemultiplicity=define use64bitint=define, use64bitall=define, uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='x86_64-pc-linux-gnu-gcc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -fwrapv -fno-strict-aliasing -pipe -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64', optimize='-O -pipe', cppflags='-D_REENTRANT -D_GNU_SOURCE -fwrapv -fno-strict-aliasing -pipe' ccversion='', gccversion='4.7.3', gccosandvers='' intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16 ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8 a
