Re: LARGE PERL footprint
On Fri, 19 May 2000, David Larkin wrote: I require a large array of ints in a real application, just stripped problem down to bear bones for demo. Is your array sparse by any chance? If not your modperl daemon is going to get _much_ larger after you populate that array. If it's sparse, consider using a hash - that will give you a slight speed penalty at the benefit of not consuming quite so much ram. Other things to consider: Re-write this critical bit of code in XS (not as hard as it sounds). Use a database. Use the filesystem. -- Matt/ Fastnet Software Ltd. High Performance Web Specialists Providing mod_perl, XML, Sybase and Oracle solutions Email for training and consultancy availability. http://sergeant.org http://xml.sergeant.org
Re: RFC: Apache::Request::Forms (or something similar)
"Jeffrey W. Baker" wrote:
On Thu, 18 May 2000, brian moseley wrote:
On Thu, 18 May 2000, Autarch wrote:
pretty slow if you build a string using .= instead of using
smarter methods, like pushing strings onto an array and then
joining it.
You tried to sell me that when I was at CP, and I didn't buy it then
either. This is a benchmark of .= versus join. 50 20-byte strings
are joined:
Concat: 2 wallclock secs ( 1.34 usr + 0.27 sys = 1.61 CPU)
Join: 4 wallclock secs ( 3.63 usr + 0.19 sys = 3.82 CPU)
.= concatenation is way faster. Also, building a 50-element array in
Perl takes up vastly more memory than building a 1000-byte string.
The string and the Perl process together require an RSS of 11MB, while the
array and the Perl process eat an RSS of 51MB.
seems very odd. ".=" copies the string every time, and then
concatenates the new addition to the end. "join" may do something
internally like that, but i'd expect it to be optimized.
and so what?
that's all moot when it comes to generating html; you push strings
and then simply print:
@table = ();
loop {
@tr = ();
loop {
...
push(@row,"td",$data,"/td\n");
}
push(@table,"tr",@row,"/tr\n");
}
...
print(@table,$str,$val,$other);
(you could easily use more object-oriented CGI-like methods,
of course...)
no joining or concatenation needed.
-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
Their is five errers in this sentance.
Re: Q: DBMS update framework for use within Apache::DBI?
"Bruce W. Hoylman" wrote: "Gunther" == Gunther Birznieks [EMAIL PROTECTED] writes: Gunther This first criteria seems a tad odd to me. What business Gunther scenario is there for this? The framework is to support an intranet time tracking application. The business rules of the system dictate there are no deletions of previously submitted time ... only inserts/updates. Therefore I can say without question that an existing time record will *always* exist (a modify), albeit in a potentially 'dirty' state (rule #2). Furthermore, if a particular time period has not been accounted for, i.e. no time record has been previously submitted, a default time record will be presented for modification, and subsequent submission (an insert). This is seamless to the client requesting the period. If there is no default time record available, then a 0-hour time record is presented for modification/submission (also an insert). need custom record locking? how about adding a field such as 'lock' which is set when a record is requested, so only one user can modify it? i.e. your script sends a flag to others who've requested a 'lock'ed record. i'd also put a timeout on it so that if a user quits his browser or gets distracted, his page gets updated to another location after freeing up the lock... -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- Their is five errers in this sentance.
Apache::ASP #include virtual loses variables
Hello, I have stumbled upon an issue with Apache::ASP !--#include virtual-- directive. Included files do not seem to be able to access the same scope of variables. I am using the following test program: File 1.inc: !--#include virtual="2.inc"-- % $test .= '1'; % p$test = %=$test%/p File 2.inc: % $test = '2'; % One would expect the output to be "$test = 21", but it comes out as "$test = 1". I have tried the same thing with #include file instead of #include virtual and the result is correct. I am using Apache/1.3.9 on Red Hat Linux. My httpd.conf is setup for ASP as follows: Files ~ (\.inc) SetHandler perl-script PerlSetVar Global /tmp PerlSetVar Filter On PerlHandler Apache::ASP Apache::SSI /Files Does anyone know if this is a bug, or a feature, or did I perhaps setup ASP incorrectly? Is there a good workaround for this? (It is inconvenient for me to use #include file instead since I need to include files relevant to DOCUMENT_ROOT, as well as relevant to the location of the current file, but I could use that as a last resort.) Thanks, -Philip Mak ([EMAIL PROTECTED])
Re: Apache::ASP #include virtual loses variables
Use file includes. virtual includes are meant to execute anything and include its output, and is handles by Apache::SSI outside of Apache::ASP. File includes will be executed as perl asp subroutines in the same perl namespace as the including script. -- Joshua _ Joshua Chamas Chamas Enterprises Inc. NodeWorks free web link monitoring Huntington Beach, CA USA http://www.nodeworks.com1-714-625-4051 Philip Mak wrote: Hello, I have stumbled upon an issue with Apache::ASP !--#include virtual-- directive. Included files do not seem to be able to access the same scope of variables. I am using the following test program: File 1.inc: !--#include virtual="2.inc"-- % $test .= '1'; % p$test = %=$test%/p File 2.inc: % $test = '2'; % One would expect the output to be "$test = 21", but it comes out as "$test = 1". I have tried the same thing with #include file instead of #include virtual and the result is correct. I am using Apache/1.3.9 on Red Hat Linux. My httpd.conf is setup for ASP as follows: Files ~ (\.inc) SetHandler perl-script PerlSetVar Global /tmp PerlSetVar Filter On PerlHandler Apache::ASP Apache::SSI /Files Does anyone know if this is a bug, or a feature, or did I perhaps setup ASP incorrectly? Is there a good workaround for this? (It is inconvenient for me to use #include file instead since I need to include files relevant to DOCUMENT_ROOT, as well as relevant to the location of the current file, but I could use that as a last resort.) Thanks, -Philip Mak ([EMAIL PROTECTED])
SV: mod_perl 1.24, nmake test causes Apache Win32 to crash.
- Original Message - From: Randy Kobes [EMAIL PROTECTED] Sent: Saturday, May 20, 2000 05:50 Subject: Re: mod_perl 1.24, nmake test causes Apache Win32 to crash. | On Sat, 20 May 2000, Thomas wrote: | | hi, | I've run into some oddities.. | running nmake test causes to seriously crash | at "internal/table" while the same test with 1.22 passes fine. | Test "internal/api" FAILS for both 1.22 / 1.24 | | Both are compiled with identical setups using VC6 / WinNT | with the 1.3.12 / 5.00503 libs . | | suggestions, anyone ?? | | Hi, |With VC6/Win98, Apache_1.3.12 and Perl-5.6.0, | I get the internal/api.t tests passing, but internal/table.t | fails. This seems related to one of the changes in | Table.xs, as simply using Table.xs from mod_perl-1.23 | allows all tests to pass. |Running the tests with TEST_VERBOSE=1, which tests | fail for you in internal/api.t? | | best regards, | randy kobes | Hi Randy, ok, did what u suggested regarding Table.xs and internal/table.t is now a GO. :) Running in verbose mode, at internal/api.t throws a HTTP 500 Response and then skips ahead. From mod_perl_error_log; ( ! ) -- [Sat May 20 11:31:32 2000] [error] Can't locate object method "new" via package "Apache::Request" at N:/cpp/mod_perl-1.24/t/net/perl/api.pl line ... -- then, concluded there must be some sort of fubar with the module-tree. manually removed Apache.pm + everything Apache::*, re-ran nmake install and voila !. 100% pass! :)) best thomas loo.
Re: LARGE PERL footprint
Hi there, On Fri, 19 May 2000, David Larkin wrote: Can anyone help explain why PERL gives such a large memory footprint advise how to get around it. In addition to the other suggestions, you might want to try use integer; in the bits of your Perl code that manipulate integers. I guess I'm paying the price for PERL not being strongly typed, a feature I really like ( until now ;-) ) Hmmm. I require a large array of ints in a real application Er, real (as opposed to floating point:) ? 73, Ged.
Re: Re: RFC: Apache::Request::Forms (or something similar)
: : On Wed, 17 May 2000, Peter Haworth wrote: : : Drew Taylor and I are about to write a subclass of Apache::Request which : includes form element generation methods, a la CGI.pm. The current : favourite : name is Apache::Request::Forms, but we'd like to know if anyone has a : better : one. : : The module is currently planned to be fairly bare-boned, only adding : form : element generation methods for methods which will benefit from : CGI.pm-style : sticky values, and the parameters these methods take are likely to be a : lot : more restricted than CGI.pm's (not difficult, really). However, this : could : change in the unlikely event that we get deluged with feature requests. : : personally, i'd like to see Apache::HTML for generating html, written in : c. something simple along the lines of HTML::AsSubs, then another class : to glues it and Apache::Request together that provides CGI.pm features, : like 'sticky forms'. but, i haven't given that much thought. I would as well Doug. I find that I only use Apache::Request and Apache::Cookie and that I sometimes need the functionality of CGI qw(:forms), but reframe from using it due to the bloatish issues. i.e it would be great to have such a thing as it would complement Apache::Request and Apache::Cookie. ps How are you getting on with the eagerly awaited mod_perl-dev-1.99 ? Just my 2 pence worth of a feature request Greg Cope (who should get his head round learning C one day)
Re: LARGE PERL footprint
[EMAIL PROTECTED] (G.W. Haywood) wrote: Hi there, On Fri, 19 May 2000, David Larkin wrote: Can anyone help explain why PERL gives such a large memory footprint advise how to get around it. My general philosophy (well, at least in these matters) is that large chunks of reference data should be stored outside the application, in a database or other appropriate place. Let the code be the code, and the data be the data. If that's not possible, you can check out some of the CPAN modules that let you store data as bare C data. I've never had the need to use one, so I can't recommend anything. I require a large array of ints in a real application Er, real (as opposed to floating point:) ? As opposed to imaginary, probably. Or complex. =) ------ Ken Williams Last Bastion of Euclidity [EMAIL PROTECTED]The Math Forum
Re: LARGE PERL footprint
Matt Sergeant writes:
On Fri, 19 May 2000, David Larkin wrote:
I require a large array of ints in a real application, just stripped
problem down to bear bones for demo.
Is your array sparse by any chance? If not your modperl daemon is going to
get _much_ larger after you populate that array. If it's sparse, consider
using a hash - that will give you a slight speed penalty at the benefit of
not consuming quite so much ram. Other things to consider:
Re-write this critical bit of code in XS (not as hard as it sounds).
Use a database.
Use the filesystem.
Simply store them in native form in a big long string, just like C
would do. Pre-extend the string first, if you know how big you want it:
$intsize = 4; # alter if sizeof(int) != 4
$intarray = "\0" x (3 * $intsize);
Update index $i with
substr($intarray, $i * $intsize, $intsize) = pack("I", $newval);
which assumes an unsigned int, or use "i" for signed.
To retrieve index $i use
$val = unpack("I", substr($intarray, $i * $intsize, $intsize));
You can even retrive a bunch of them faster with
@ten_to_thirty = unpack("I20", substr(...));
or set a bunch with
substr(..., $intsize * 10) = pack("I20", @ten_to_thirty);
For that last bit of extra performance, do
sub INTSIZE () { 4 }; # alter if sizeof(int) != 4
--Malcolm
--
Malcolm Beattie [EMAIL PROTECTED]
Unix Systems Programmer
Oxford University Computing Services
Re: Apache::ASP #include virtual loses variables
On Sat, 20 May 2000, Joshua Chamas wrote:
Use file includes. virtual includes are meant to execute
anything and include its output, and is handles by Apache::SSI
outside of Apache::ASP. File includes will be executed as perl
asp subroutines in the same perl namespace as the
including script.
I see. There are two problems that I have with file includes though:
(1) I cannot specify a file's location relative to $ENV{'DOCUMENT_ROOT'}.
(2) I cannot specify a file's location relative to the directory the
current file is in.
For #1, I want to do something like this in all my pages:
!--#include virtual="/code/header.asp"--
!--#include virtual="/code/footer.asp"--
And for #2, I have an "index.inc" in all my directories. Each index.inc
has to include the one in its parent directory, e.g.:
!--#include file="../index.inc"--
so that directories can pass on properties to their subdirectories. If I
use include file for that, it will include files relative to the pathname
of the first ASP file, and not relative to the pathname of the ASP file
that actually has the include.
One reason I'm coding like this is because I want to give each directory a
title and append it to the page's title. e.g.:
http://www.girlsofanime.com/series/slayers/lina/ has title of
Girls of Anime::Anime Series::The Slayers::Lina Inverse
The way the title is constructed is:
In /index.inc: $title = 'Girls of Anime';
/series/index.inc: $title .= '::Anime Series';
/series/slayers/index.inc: $title .= '::The Slayers';
/series/slayers/lina/index.inc: $title .= '::Lina Inverse';
Is there a better way I can do this? Right now I'm thinking of either
trying to hack Apache::ASP to support #include virtual, or using absolute
pathnames or trying to put $ENV{'DOCUMENT_ROOT'} in the file path.
-Philip Mak ([EMAIL PROTECTED])
Re: Apache::ASP #include virtual loses variables
Philip Mak wrote:
I see. There are two problems that I have with file includes though:
(1) I cannot specify a file's location relative to $ENV{'DOCUMENT_ROOT'}.
(2) I cannot specify a file's location relative to the directory the
current file is in.
For #1, I want to do something like this in all my pages:
!--#include virtual="/code/header.asp"--
!--#include virtual="/code/footer.asp"--
For #1, know includes will be picked up from your Global directory,
so you can use that repository to share includes, instead of some
DOCUMENT_ROOT location. You can also use IncludesDir for this if
it is set.
And for #2, I have an "index.inc" in all my directories. Each index.inc
has to include the one in its parent directory, e.g.:
!--#include file="../index.inc"--
so that directories can pass on properties to their subdirectories. If I
use include file for that, it will include files relative to the pathname
of the first ASP file, and not relative to the pathname of the ASP file
that actually has the include.
...
http://www.girlsofanime.com/series/slayers/lina/ has title of
Girls of Anime::Anime Series::The Slayers::Lina Inverse
The way the title is constructed is:
In /index.inc: $title = 'Girls of Anime';
/series/index.inc: $title .= '::Anime Series';
/series/slayers/index.inc: $title .= '::The Slayers';
/series/slayers/lina/index.inc: $title .= '::Lina Inverse';
I would not do it this way, in fact the way I would do this
would not be with your methods at all, unless you want
to have each section to be arbitrarily different and
maintained by separate graphics designers. The way I would
do this thing is to lose the directory structure completely
and to have things be database driven with parameters from
?query_string like /index.asp?dir=, which you can build
the title for from the database because you know all the
parents for dir=.
The point here is that each ASP script is a whole program
by itself, and I would not recommend having hundreds or
thousands of them to have to compile for your site. If you
have meta data you want to display, you should really stick
as much of it as possible in a database like MySQL. In
the long run, your project will be much more maintainable
even if in the short run its easier to derive info from
unix directories flat files.
If you want to have a nicer /path_info scheme, we'll
probably have to add a patch for you to have Apache::ASP
not be bound to executing real files as it is currently.
This would be more similar to the way Mason does things.
--Joshua
Re: Apache::ASP #include virtual loses variables
On Sat, 20 May 2000, Joshua Chamas wrote: !--#include virtual="/code/header.asp"-- !--#include virtual="/code/footer.asp"-- For #1, know includes will be picked up from your Global directory, so you can use that repository to share includes, instead of some DOCUMENT_ROOT location. You can also use IncludesDir for this if it is set. Thanks! That pretty much lets me do exactly what I want to. And for #2, I have an "index.inc" in all my directories. Each index.inc has to include the one in its parent directory, e.g.: !--#include file="../index.inc"-- so that directories can pass on properties to their subdirectories. If I use include file for that, it will include files relative to the pathname of the first ASP file, and not relative to the pathname of the ASP file that actually has the include. ... http://www.girlsofanime.com/series/slayers/lina/ has title of Girls of Anime::Anime Series::The Slayers::Lina Inverse The way the title is constructed is: In /index.inc: $title = 'Girls of Anime'; /series/index.inc: $title .= '::Anime Series'; /series/slayers/index.inc: $title .= '::The Slayers'; /series/slayers/lina/index.inc: $title .= '::Lina Inverse'; I would not do it this way, in fact the way I would do this would not be with your methods at all, unless you want to have each section to be arbitrarily different and maintained by separate graphics designers. The way I would do this thing is to lose the directory structure completely and to have things be database driven with parameters from ?query_string like /index.asp?dir=, which you can build the title for from the database because you know all the parents for dir=. There are two reasons why I don't like doing it this way: (1) The URL is no longer human readable. My site will have a clear hierarchical structure, so I think it makes sense to mirror that in the directories. People who want to chop or type URLs (even though I have good navigation in my web design) can do so, and they can also look at the URL to get an idea of where they are. (2) Setting up a database seems to be overkill for this site. The only meta data that it has (or probably will ever have) is: - directory name label - What links are on the sidebar for this directory? - What advertising banner(s) is displayed on pages in this directory? My method of having an index.inc for each directory is fairly simple, yet it would seem to provide all the flexibility I need to implement this and the flatfiles are easy to maintain. I don't see a good reason to switch to database-based which seems to be significantly more complicated. The point here is that each ASP script is a whole program by itself, and I would not recommend having hundreds or thousands of them to have to compile for your site. If you have meta data you want to display, you should really stick as much of it as possible in a database like MySQL. In the long run, your project will be much more maintainable even if in the short run its easier to derive info from unix directories flat files. Each ASP script is compiled separately? I thought that !--#include file-- and !--#include virtual-- are supposed to work just like #include does in C, i.e. it pretends that the text of the included file was actually pasted directly into the program. Am I thinking about ASP include in the wrong way? If you want to have a nicer /path_info scheme, we'll probably have to add a patch for you to have Apache::ASP not be bound to executing real files as it is currently. This would be more similar to the way Mason does things. Well, I would like to suggest that you consider including !--#include virtual-- in the Apache::ASP distribution, so that included files use the same namespace. It doesn't make sense logically that include virtual behaves differently from include file (other than the way the filename/pathname is interpreted, of course). -Philip Mak ([EMAIL PROTECTED])
segfault: perl 5.6.0, apache 1.3.12, mod_perl 1.24 and XML::Parser
My apache dies about 30% of the time when handling any mod_perl request that requires XML::Parser. Any other page (even pages that use mod_perl) are 100% ok. Are there any known issues with this (besides the requirement for --disable-rule=expat)? This all worked fine with perl 5.005_03 + mod_perl 1.23. I'll get more info if nobody has seen this before. Mod_perl compiled with: perl Makefile.PL PERL_FIXUP=1 PERL_AUTHEN=1 PERL_CHILD_INIT=1 \ PERL_POST_READ_REQUEST=1 PERL_STACKED_HANDLERS=1 \ PERL_UTIL_API=1 DO_HTTPD=1 USE_APACI=1 PREP_HTTPD=1 Apache compiled with: export CFLAGS="-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64" ./configure \ --disable-rule=WANTHSREGEX \ --disable-rule=expat \ --prefix=/local/httpd0 \ --disable-module=env \ --disable-module=negotiation \ --disable-module=include \ --disable-module=asis \ --disable-module=imap \ --disable-module=userdir \ --disable-module=setenvif \ --disable-module=autoindex \ --disable-module=userdir \ --enable-module=status \ --activate-module=src/modules/perl/libperl.a % perl -V Summary of my perl5 (revision 5.0 version 6 subversion 0) configuration: Platform: osname=linux, osvers=2.2.15, archname=i686-linux uname='linux area51.ottawa.loran.com 2.2.15 #2 thu may 4 13:50:33 edt 2000 i686 unknown ' config_args='' hint=recommended, useposix=true, d_sigaction=define usethreads=undef use5005threads=undef useithreads=undef usemultiplicity=undef useperlio=undef d_sfio=undef uselargefiles=define use64bitint=undef use64bitall=undef uselongdouble=undef usesocks=undef Compiler: cc='gcc', optimize='-O2', gccversion=2.95 19990728 (release) cppflags='-fno-strict-aliasing -I/usr/local/include' ccflags ='-fno-strict-aliasing -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64' stdchar='char', d_stdstdio=define, usevfork=false intsize=4, longsize=4, ptrsize=4, doublesize=8 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12 ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8 alignbytes=4, usemymalloc=n, prototype=define Linker and Libraries: ld='gcc', ldflags =' -L/usr/local/lib' libpth=/usr/local/lib /lib /usr/lib libs=-lnsl -lndbm -lgdbm -ldb -ldl -lm -lc -lposix -lcrypt libc=/lib/libc-2.1.2.so, so=so, useshrplib=false, libperl=libperl.a Dynamic Linking: dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-rdynamic' cccdlflags='-fpic', lddlflags='-shared -L/usr/local/lib' Characteristics of this binary (from libperl): Compile-time options: USE_LARGE_FILES Built under linux Compiled at May 20 2000 12:16:02 @INC: /usr/lib/perl5/5.6.0/i686-linux /usr/lib/perl5/5.6.0 /usr/lib/perl5/site_perl/5.6.0/i686-linux /usr/lib/perl5/site_perl/5.6.0 /usr/lib/perl5/site_perl
Re: Apache::ASP #include virtual loses variables
| Well, I would like to suggest that you consider including !--#include | virtual-- in the Apache::ASP distribution, so that included files use the | same namespace. It doesn't make sense logically that include virtual | behaves differently from include file (other than the way the | filename/pathname is interpreted, of course). It does make sense to me, though. Consider one having very big (say 50k) include files being included from several other (say 100) scripts. Just sucking them in each script doing the include would cause *every* script growing by at least the size of the include. Now as Apache::ASP caches all compiled scripts, this would result in each httpd process growing by 50kB x 100 scripts = 5 MB, holding 98% redundant data. Giving each include a different namespace (or more accurate: compile it as a unique sub(), and thus having it's own lexical scope), will cause the include file being compiled and cached only once. Ime
frontend proxy really useful?
I was rereading http://perl.apache.org/guide/scenario.html#Buffering_Feature and was surprised to find: "Therefore if you don't use mod_proxy and mod_perl send its data directly to the client, and you have a big socket buffer, the mod_perl process will be released as soon as the last chunk of data will enter the buffer. Just like with mod_proxy, OS will worry to complete the data transfer." Is there any new information on whether this is the case? If it is, does it make the light frontend buffering proxy technique useless as long as your pages fit in the socket buffer size (256K on Solaris)? (assuming the proxy is just a dumb passthrough to one backend server) Thanks, Chris
