APR::Table sporadic error
Hi All, We have a modperl application that is an online homework system for math courses called webwork. My installation and a few others have seen a sporadic error from APR::Table that nobody quite knows how to diagnose. Here are two typical examples: Example 1: Can't locate object method "get" via package "APR::Table" at [WW]/lib/Apache/WeBWorK.pm line 74 Died within AnswerEvaluator::evaluate called at line 1 of (eval 1321) System: Ubuntu 8.04, Apache 2.2 Example 2: Failed to initialize course environment: Could not evaluate global environment file /opt/webwork/webwork2/conf/global.conf: Can't locate object method "get" via package "APR::Table" at /opt/webwork/webwork2/lib/Apache/WeBWorK.pm line 74. RHEL 5 with RedHat's build of Apache 2.2.3. 1 GB RAM, perl-status shows ARP::Table is loaded. APR::Table 0.009000 Fri Sep 19 04:30:28 2008 /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/APR/Table.pm I'm hoping you might be able to provide some suggestions of where to start looking for the source of this problem. The modperl handler is here: http://webwork.maa.org/viewvc/system/trunk/webwork2/lib/Apache/WeBWorK.pm?view=markup The error was not seen (and has not been seen) running mod_perl 1. Here is a report from one user: We've seen the APR::Table error sporadically, and have found it to be > chronic and page load dependent. We've found in every case that if we reload > the page the error will go away. Our hypothesis is that it's probably a > thread initialization problem, and that it shows up when the apache process > hands the WeBWorK job off to a new thread that doesn't have all of the > required Perl loaded before the it tries to execute the WeBWorK request. > That said, we don't know much about thread management in apache, so we don't > know if that's even credible. Any hints/tips/ideas would be very much appreciated. Thanks, Jason
Re: APR::Table sporadic error
>
> my $warnings = eval {$r->notes->get("warnings")};
> if( $@ ) {
>warn "APR::Table is '$INC{q{APR/Table.pm}}'";
>die $@;
> }
>
> you will see that it is not loaded.
>
>
Thanks Torsten - we'll try this out.
If you use a threaded Perl make sure every interpreter loads APR::Table.
> Do you by chance use PerlOptions +Parent in a VHost?
>
> No, the typical configuration is like this:
http://webwork.maa.org/viewvc/system/trunk/webwork2/conf/webwork.apache2-config.dist?revision=6531&view=markup
Thanks again,
Jason
Re: APR::Table sporadic error
Additional data:
One effected installation (Ubuntu 10.04.2, Apache 2.2.14, Perl 5.10.1,
libapache2-mod-perl2 2.0.4) reports that that eval
{$r->notes->get("warnings")} did not produce an error ($@ was not set) in
his case, but that this problem went away for him after reverting the ubuntu
perl updates from May 4 (http://www.ubuntu.com/usn/usn-1129-1/).
Those perl updates on Ubuntu seem to have triggered this error for the two
installations that reported it today, but the strange thing is that this is
not the first time we've seen this error - it was reported on a CentOS
installation and two RHEL 5 installations in January.
Perhaps RH et. al. applied the patches for the SVE's in January and Ubuntu
just now, and one of those changes is causing this. Still, we don't know,
so any other possible leads will be eagerly followed.
Jason
2011/5/6 Jason Aubrey
>
>
>> my $warnings = eval {$r->notes->get("warnings")};
>> if( $@ ) {
>>warn "APR::Table is '$INC{q{APR/Table.pm}}'";
>>die $@;
>> }
>>
>> you will see that it is not loaded.
>>
>>
> Thanks Torsten - we'll try this out.
>
> If you use a threaded Perl make sure every interpreter loads APR::Table.
>> Do you by chance use PerlOptions +Parent in a VHost?
>>
>> No, the typical configuration is like this:
>
>
> http://webwork.maa.org/viewvc/system/trunk/webwork2/conf/webwork.apache2-config.dist?revision=6531&view=markup
>
> Thanks again,
> Jason
>
>
perl 5.14 ExtUtils::MakeMaker bug affecting libapreq?
Hi All,
I recently used perlbrew to install perl 5.14.2, and then I compiled apache
2.2.22, mod_perl-2.0.5, and libapreq2-2.13. But then I think I'm getting a
binary incompatibility problem:
jason@plato:~$ perl -MAPR::Request -e 1
Not a CODE reference at
/home/jason/perl5/perlbrew/perls/perl-5.14.2/lib/5.14.2/i686-linux-thread-multi/DynaLoader.pm
line 213.
END failed--call queue aborted at
/home/jason/perl5/perlbrew/perls/perl-5.14.2/lib/site_perl/5.14.2/i686-linux-thread-multi/APR.pm
line 213.
Compilation failed in require at
/home/jason/perl5/perlbrew/perls/perl-5.14.2/lib/site_perl/5.14.2/i686-linux-thread-multi/APR/Pool.pm
line 23.
BEGIN failed--compilation aborted at
/home/jason/perl5/perlbrew/perls/perl-5.14.2/lib/site_perl/5.14.2/i686-linux-thread-multi/APR/Pool.pm
line 23.
Compilation failed in require at
/home/jason/perl5/perlbrew/perls/perl-5.14.2/lib/site_perl/5.14.2/i686-linux-thread-multi/APR/Request.pm
line 26.
BEGIN failed--compilation aborted at
/home/jason/perl5/perlbrew/perls/perl-5.14.2/lib/site_perl/5.14.2/i686-linux-thread-multi/APR/Request.pm
line 26.
Compilation failed in require.
BEGIN failed--compilation aborted.
I found this describing a possible bug in ExtUtils::MakeMaker that gives a
similar error for other modules:
http://lists.fedoraproject.org/pipermail/perl-devel/2011-June/037523.html
However, I could find no bug report about this on cpan (against libapreq2,
there are unattended bug reports about the general issue for
ExtUtils::MakeMaker). Also, I tried adding CCFLAGS =>
"$Config::Config{ccflags}", to the Makefile.PL in glue/perl, but maybe
that's not the right way to do it because I get the same error. Do you
think this is the cause of the error? If so, any suggestions for making it
work?
Thanks for any help...
Jason
reloading perl modules under heavy load
Hi All, We have modperl application running on 64bit (2 cores), freebsd 8.2, perl 1.14.1, modperl 2.0.5, apache 2.2.21, 24GB of ram and about 5.5k users. Additional system details are below. Under heavy load we are seeing errors of the form Failed to evaluate module Parser::Legacy: Attempt to reload > Parser/Legacy.pm aborted.\nCompilation failed in require at (eval 15635) > line 1 I vaguely suspect this is a concurrency issue. Requests that generate such errors often, but not always, call to GD.pm to build dynamically generated images. The errors often, but not always, refer to Parser::Legacy a module which is part of our application, but I think this is a red-herring since it only occurs under load, and I've seen such errors refer to other modules, for example DateTime. My server is the largest production freebsd machine running this application; most servers of this size run Red Hat or Ubuntu and as far as I can tell nobody has seen this error before. I honestly have no idea if this is a modperl issue, an apache issue, a freebsd issue, etc. But I thought I would start here since this is where the rubber (requests) meet the road (perl). Has anyone see such an error before? Or know of a similar case that might give me some additional clues? Thanks very much, Jason Aubrey OS: FreeBSD 8.2-RELEASE-p10 Perl: modperl 2.0.5 Platform: osname=freebsd, osvers=8.2-release-p2, archname=amd64-freebsd uname='freebsd ww64 8.2-release-p2 freebsd 8.2-release-p2 #0: sat sep 17 13:09:25 cdt 2011 root@ww64:usrobjusrsrcsysvmware amd64 ' config_args='-sde -Dprefix=/usr/local -Darchlib=/usr/local/lib/perl5/5.14.1/mach -Dprivlib=/usr/local/lib/perl5/5.14.1 -Dman3dir=/usr/local/lib/perl5/5.14.1/perl/man/man3 -Dman1dir=/usr/local/man/man1 -Dsitearch=/usr/local/lib/perl5/site_perl/5.14.1/mach -Dsitelib=/usr/local/lib/perl5/site_perl/5.14.1 -Dscriptdir=/usr/local/bin -Dsiteman3dir=/usr/local/lib/perl5/5.14.1/man/man3 -Dsiteman1dir=/usr/local/man/man1 -Ui_malloc -Ui_iconv -Uinstallusrbinperl -Dcc=cc -Duseshrplib -Dinc_version_list=none -Dccflags=-DAPPLLIB_EXP="/usr/local/lib/perl5/5.14.1/BSDPAN" -Doptimize=-O2 -fno-strict-aliasing -pipe -march=native -Ui_gdbm -Dusethreads=n -Dusemymalloc=y -Duse64bitint -Dusesitecustomize' hint=recommended, useposix=true, d_sigaction=define useithreads=undef, usemultiplicity=undef useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef use64bitint=define, use64bitall=define, uselongdouble=undef usemymalloc=y, bincompat5005=undef Compiler: cc='cc', ccflags ='-DAPPLLIB_EXP="/usr/local/lib/perl5/5.14.1/BSDPAN" -DHAS_FPSETMASK -DHAS_FLOATINGPOINT_H -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include', optimize='-O2 -fno-strict-aliasing -pipe -march=native', cppflags='-DAPPLLIB_EXP="/usr/local/lib/perl5/5.14.1/BSDPAN" -DHAS_FPSETMASK -DHAS_FLOATINGPOINT_H -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include' ccversion='', gccversion='4.2.1 20070719 [FreeBSD]', gccosandvers='' intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16 ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8 alignbytes=8, prototype=define Linker and Libraries: ld='cc', ldflags =' -Wl,-E -fstack-protector -L/usr/local/lib' libpth=/usr/lib /usr/local/lib libs=-lm -lcrypt -lutil perllibs=-lm -lcrypt -lutil libc=, so=so, useshrplib=true, libperl=libperl.so gnulibc_version='' Dynamic Linking: dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags=' -Wl,-R/usr/local/lib/perl5/5.14.1/mach/CORE' cccdlflags='-DPIC -fPIC', lddlflags='-shared -L/usr/local/lib -fstack-protector' Characteristics of this binary (from libperl): Compile-time options: MYMALLOC PERL_DONT_CREATE_GVSV PERL_MALLOC_WRAP PERL_PRESERVE_IVUV USE_64_BIT_ALL USE_64_BIT_INT USE_LARGE_FILES USE_PERLIO USE_PERL_ATOF USE_SITECUSTOMIZE Built under freebsd Compiled at Sep 17 2011 19:35:32 Apache: Server version: Apache/2.2.21 (FreeBSD) Server built: Sep 17 2011 20:41:36 Server's Module Magic Number: 20051115:30 Server loaded: APR 1.4.5, APR-Util 1.3.12 Compiled using: APR 1.4.5, APR-Util 1.3.12 Architecture: 64-bit Server MPM: Prefork threaded: no forked: yes (variable process count) Server compiled with -D APACHE_MPM_DIR="server/mpm/prefork" -D APR_HAS_SENDFILE -D APR_HAS_MMAP -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled) -D APR_USE_FLOCK_SERIALIZE -D APR_USE_PTHREAD_SERIALIZE -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT -D APR_HAS_OTHER_CHILD -D AP_HAVE_RE
Re: reloading perl modules under heavy load
Thanks all for your replies to my question. Because of the nature of our application, we can't really load everything at start up, but I did some digging and there are clearly some inefficiencies here and the situation would indeed be improved by cleaning these up. However, it does look like it may be the interaction between these inefficiencies and freebsd in particular that is causing the error. So, here's what I did: on a linux development box I wrote a script using Linux::Inotify2 to watch the directories containing the code for our application that we (think we) have to load at run time. A typical request generated an average of 225 IN_OPEN inotify events. (FWIW, it also generated around 850 IN_ACCESS events. I don't understand the difference, but IN_OPEN events seem more relevant to what I'm getting to.) Now, we have MaxClients set super high, but with just 150 concurrent requests that's 33,750 open files and that's not taking into account files opened for other processes on the machine, such as the kernel or the web server, etc. But, here's where freebsd comes in. The freebsd kernel has a bunch of tunables, one of which is kern.maxfiles. The freebsd manual says "This variable indicates the maximum number of file descriptors on your system... Each open file, socket, or fifo uses one file descriptor. A large-scale production server may easily require many thousands of file descriptors, depending on the kind and number of services running concurrently" Well, we had kern.maxfiles = 12328. Also possibly relevant is kern.ipc.somaxconn which "limits the size of the listen queue for accepting new TCP connections". We had that at the default of 128. I also don't understand the interaction between this setting and apache's MaxClients since when we had kern.ipc.somaxconn = 128 and MaxClients 150 we quickly maxed out max clients. So, we bumped up kern.maxfiles to 5 and kern.ipc.somaxconn to 8192 and things appeared to be better last night under heavy load. But, as I said, there are clearly some in efficiencies too in the way our application loads code at run time, and cleaning these up would certainly help. If you're interested, here is a discussion from a free-bsd mailing list yesterday about low kernel limits that a colleague forwarded to me: http://goo.gl/Dpzre Thanks again for all of your replies, Jason On Wed, Oct 3, 2012 at 12:48 PM, Perrin Harkins wrote: > On Wed, Oct 3, 2012 at 9:45 AM, Jason Aubrey wrote: > > Hi All, > > > > We have modperl application running on 64bit (2 cores), freebsd 8.2, > perl > > 1.14.1, modperl 2.0.5, apache 2.2.21, 24GB of ram and about 5.5k users. > > Additional system details are below. Under heavy load we are seeing > errors > > of the form > > > >> Failed to evaluate module Parser::Legacy: Attempt to reload > >> Parser/Legacy.pm aborted.\nCompilation failed in require at (eval 15635) > >> line 1 > > I remember seeing this kind of thing when I was trying to use > ActiveState's mod_perl-ish thing with IIS years ago. It would fail > to find things like strict.pm when it was under load. I guessed the > same thing: some sort of concurrency issue. I don't have a solution > to offer though, since I gave up on that one and used CGI. > > - Perrin >
Re: reloading perl modules under heavy load
On Thu, Oct 4, 2012 at 9:53 AM, demerphq wrote: > On 4 October 2012 16:36, Jason Aubrey wrote: > > Thanks all for your replies to my question. > > > > Because of the nature of our application, we can't really load > everything at > > start up, but I did some digging and there are clearly some > inefficiencies > > here and the situation would indeed be improved by cleaning these up. > > However, it does look like it may be the interaction between these > > inefficiencies and freebsd in particular that is causing the error. > > > > So, here's what I did: on a linux development box I wrote a script using > > Linux::Inotify2 to watch the directories containing the code for our > > application that we (think we) have to load at run time. A typical > request > > generated an average of 225 IN_OPEN inotify events. (FWIW, it also > > generated around 850 IN_ACCESS events. I don't understand the > difference, > > but IN_OPEN events seem more relevant to what I'm getting to.) > > > > Now, we have MaxClients set super high, but with just 150 concurrent > > requests that's 33,750 open files and that's not taking into account > files > > opened for other processes on the machine, such as the kernel or the web > > server, etc. > > > > But, here's where freebsd comes in. The freebsd kernel has a bunch of > > tunables, one of which is kern.maxfiles. The freebsd manual says "This > > variable indicates the maximum number of file descriptors on your > system... > > Each open file, socket, or fifo uses one file descriptor. A large-scale > > production server may easily require many thousands of file descriptors, > > depending on the kind and number of services running concurrently" > > > > Well, we had kern.maxfiles = 12328. Also possibly relevant is > > kern.ipc.somaxconn which "limits the size of the listen queue for > accepting > > new TCP connections". We had that at the default of 128. I also don't > > understand the interaction between this setting and apache's MaxClients > > since when we had kern.ipc.somaxconn = 128 and MaxClients 150 we quickly > > maxed out max clients. > > > > So, we bumped up kern.maxfiles to 5 and kern.ipc.somaxconn to 8192 > and > > things appeared to be better last night under heavy load. But, as I said, > > there are clearly some in efficiencies too in the way our application > loads > > code at run time, and cleaning these up would certainly help. > > You would also see reduced memory pressure and performance > improvements. Perl is actually pretty slow at compiling code (in > comparison to executing code), and code that is compiled prefork is > going to be shared, whereas code compiled post-fork is going to be > duplicated per process. > > By preloading you will both win performance improvements and reduce > memory pressure on your application. > > I strongly recommend you approach this subject aggressively. > > yves > Hi all, Here's another update which you will be glad to hear because it proves you are awesome (or at least that you were right). The freebsd tuning I mentioned below helped but didn't fix the problem. We still got a bunch of such errors after this tuning (fewer, but still a lot). Well, I added just one file to the list of code compiled in at start up and since doing so we have had exactly zero such errors. Admittedly, that file pulls in a large package, but this single 8-character change to one line of code seems to have had a tremendously positive impact on the performance of our application. Thanks again for all of of your help, Jason
Re: Alternatives to CGI perl module
Mathjax is the best for math display on the web. The mathematics professional societies use it, as do major projects like webwork ( github.com/openwebwork, webwork.maa.org) and Sage (sagemath.org). On Sunday, September 11, 2016, Igor Chudov wrote: > This mathml and asciimath is extremely interesting! Thank you! You are > awesome! > > I was a little unclear what browsers it works with, I would have to do > some testing. I hope that it is compatible with all modern browsers and > cell phones and such. If not it would not work, but if it is then I would > be delighted! I use something similar to asciimath internally, that I > developed 12 years ago, should not be hard to translate as I parse all > formulas deeply. > > > On Sun, Sep 11, 2016 at 9:06 PM, Michael Bochkaryov > wrote: > >> >> On Sun, Sep 11, 2016 at 4:51 PM, Ruben Safir > > wrote: >> >>> > GD::Graph is something that is not going to kill me (like if CGI.pm >>> became >>> > unavailable) but it is certainly a concern. >>> > >>> > Are there any alternatives to it? >>> > >>> imagemajic maybe >>> >> >> It also make sense to try something like this: >> https://www.mathjax.org/ >> >> Seems modern browsers may allow to move rendering to client side that >> also may decrease load on backend. >> >> >> Regards, >> Michael Bochkaryov >> >> >
