Re: global variable
On Wed, Feb 3, 2010 at 3:20 PM, André Warnier a...@ice-sa.com wrote: But if you give some more details about the platform, the Apache, and what you are trying to do, someone may be able to suggest an alternative. Something linke java servlet's global variable. Maybe set it via environment variable? Thanks.
Re: global variable
On Wed, Feb 3, 2010 at 4:18 PM, m...@normalperson.e4ward.com wrote: On Wed, Feb 3, 2010 at 3:20 PM, André Warnier a...@ice-sa.com wrote: But if you give some more details about the platform, the Apache, and what you are trying to do, someone may be able to suggest an alternative. Something linke java servlet's global variable. sorry for the typo, something like ...
Re: global variable
why not use memcached or FastMmap? Thanks On 2010-2-3 16:19, m...@normalperson.e4ward.com wrote: On Wed, Feb 3, 2010 at 4:18 PM,m...@normalperson.e4ward.com wrote: On Wed, Feb 3, 2010 at 3:20 PM, André Warniera...@ice-sa.com wrote: But if you give some more details about the platform, the Apache, and what you are trying to do, someone may be able to suggest an alternative. Something linke java servlet's global variable. sorry for the typo, something like ... -- Fayland Lam // http://www.fayland.org/
Re: global variable
On Wednesday 03 February 2010 06:45:06 m...@normalperson.e4ward.com wrote: Is there a method to setup a global variable for all modperl child processes? Also this variable will be updated sometime, when it get updated, all processes will know it. File::Map mmap()s a file and makes it visible as a perl string. You can modify it for example by using the lvalue form of substr(). MMapDB uses File::Map and provides some kind of a hash. (I have just uploaded version 0.07 that handles utf8 strings correctly (me thinks so)) There is a mod_slotmem.c floating around. It provides something like apache's scoreboard. That means each worker gets its own piece of shared memory to write to while all slots can be read by anyone. I think it can be interfaced to be used by Perl. Torsten
Re: [mp2] Apache2::SizeLimit should be using $s-rss, not $s-size for Linux::Smaps
On Tuesday 02 February 2010 22:58:13 Max Kanat-Alexander wrote:
All of my processes kept exiting with a report that they had a 300M
unshared size, which was clearly untrue, even from looking at top. After
some investigation, I discovered that Apache2::SizeLimit was calling
$s-size on the Linux::Smaps object, when instead it should be returning
$s-rss as the process size.
Well, I tend to disagree. (Fred, Adam please read on.)
The /proc/PID/statm based check returns the fields 0 and 2. According to the
following table from KERNEL/Documentation/filesystems/proc.txt field 0 is SIZE
and not RSS.
Table 1-3: Contents of the statm files (as of 2.6.8-rc3)
..
FieldContent
size total program size (pages)(same as VmSize in status)
resident size of memory portions (pages) (same as VmRSS in status)
shared number of pages that are shared (i.e. backed by a file)
trs number of pages that are 'code' (not including libs; broken,
includes data segment)
lrs number of pages of library(always 0 on 2.6)
drs number of pages of data/stack (including libs; broken,
includes library text)
dt number of dirty pages (always 0 on 2.6)
This is also consistent with (Smaps prints kb while statm shows pages, hence
the division by 4):
$ perl -MLinux::Smaps -le 'print Linux::Smaps-new(shift)-size/4' $$
3526
$ cat /proc/$$/statm
3525 881 396 144 0 495 0
So, either the far older statm technique is also wrong or the patch is wrong.
But on Solaris we do -s /proc/self/as. That is the size of the address space
of the process. Are we wrong there, as well?
Ok, on BSD it seems to be RSS:
# rss is in KB but ixrss is in BYTES.
# This is true on at least FreeBSD, OpenBSD, NetBSD
sub _bsd_size_check {
my @results = BSD::Resource::getrusage();
my $max_rss = $results[2];
my $max_ixrss = int ( $results[3] / 1024 );
return ($max_rss, $max_ixrss);
}
About the windows code I cannot say anything. What does
$peak_working_set_size mean? For me the wording seems a bit similar to
max(resident segment size).
So, we have at least 2 different meanings of the SIZE result. On BSD it is RSS
on Solaris and Linux SIZE. What is correct?
Let's see how it is used?
my ($size, $share, $unshared) = $class-_check_size();
return 1 if $MAX_PROCESS_SIZE $size $MAX_PROCESS_SIZE;
return 0 unless $share;
return 1 if $MIN_SHARE_SIZE $share $MIN_SHARE_SIZE;
return 1 if $MAX_UNSHARED_SIZE $unshared $MAX_UNSHARED_SIZE;
It is compared with $MAX_PROCESS_SIZE and there is no $MAX_PROCESS_RSS.
And what does the docs say?
=item * Apache2::SizeLimit-set_max_process_size($size)
This sets the maximum size of the process, including both shared and
unshared memory.
It talks about process size not RSS, again.
Now let's assume we would check RSS instead of SIZE.
When a new apache worker process is created its growth in SIZE depends on how
much memory it allocates additionally over time. But its RSS depends upon the
process' size and what part of it is swapped out. It seems to me that we want
to kill a worker if its size grows bigger than the initial worker size plus a
certain amount it is allowed to grow. If we would check RSS then a worker or
even all workers could be killed because some administrator does
swapoff /dev/...
and suddenly all pages that were swapped to this device are copied into RAM
and added to the worker's RSS. I think that would be wrong.
But you mentioned unshared size. How is it calculated?
Apache::SizeLimit::Core::_check_size() simply does $size-$share. On Linux this
is wrong. When you read /proc/PID/smaps the kernel walks through all pages
that belong to the process and does:
size+=pagesize;
if( page_is_in_RAM ) {
rss+=pagesize;
if( reference_count1 ) { /* how many processes map that page */
shared+=pagesize;
} else {
private+=pagesize;
}
}
When a page is not in RAM there is unfortunately no way to check the reference
count other than swap it in. And that is certainly too high a cost.
Currently Apache2::SizeLimit assumes that everything that is not in core is
not shared. This is certainly wrong.
Perhaps Apache2::SizeLimit::Core should read:
sub _check_size {
my $class = shift;
my ($size, $share, $unshared) = $class-_platform_check_size();
return ($size, $share, defined $unshared ? $unshared : $size - $share);
}
sub _linux_smaps_size_check {
my $class = shift;
return $class-_linux_size_check() unless $USE_SMAPS;
my $s = Linux::Smaps-new($$)-all;
return ($s-size,
$s-shared_clean + $s-shared_dirty,
$s-private_clean + $s-private_dirty);
}
That would be more what you want, I think.
Torsten
Re: global variable
On Wed, Feb 3, 2010 at 2:20 AM, André Warnier a...@ice-sa.com wrote: m...@normalperson.e4ward.com wrote: Hello, Is there a method to setup a global variable for all modperl child processes? Also this variable will be updated sometime, when it get updated, all processes will know it. As a general answer, no. That's right. Apache uses multiple processes and there's no built-in way to share perl variables between them. If you want to share data and don't want to use a database, look at tools like CHI on CPAN. - Perrin
Apache 1.3 end of life
FYI The Apache Software Foundation and the Apache HTTP Server Project are pleased to announce the release of version 1.3.42 of the Apache HTTP Server (Apache). This release is intended as the final release of version 1.3 of the Apache HTTP Server, which has reached end of life status. There will be no more full releases of Apache HTTP Server 1.3. However, critical security updates may be made available from the following website: http://www.apache.org/dist/httpd/patches/ see: http://www.apache.org/dist/httpd/Announcement1.3.html
Re: Best way to pass arguments to an Apache2::SubProcess?
Probably a stupid question but can't hurt to ask. In the docs they
consistently use @argv = qw(foo bar); Is there a reason for this, can
the args only be strings as semi-implied by somebody mentioning they
used JSON in another thread?
Or can I send $argv[0] = LARGE HASH ?
Thanks!
Tosh
Ihnen, David wrote:
Environment variables have size limits - you should pass it as the args
reference because of that, and the fact that you're sending a pointer rather
than the whole data helps too, though the system is probably copying it
somewhere in there you shouldn't have to worry about it.
The point they're inferring in the examples is that the second element is a
list reference - not whether it's a referenced to a named list or not is
irrelevant.
David
-Original Message-
From: Tosh Cooey [mailto:t...@1200group.com]
Sent: Tuesday, February 02, 2010 5:06 PM
To: modperl@perl.apache.org
Subject: Best way to pass arguments to an Apache2::SubProcess?
So my will has crumbled, mostly because I'm probably not very good at
the resident in memory part of mod_perl and I will be using
Apache2::SubProcess to fire off a sub-process.
There seems to be two easy ways to send data to my detached sub-process;
via an \...@args or via $r-subprocess_env-set.
I would like to pass large data structures which I will JSON encode and
I'm wondering which of the above would be best for that, if any, and
what the limitations are of either.
Also, I was wondering, in the docs for Apache2::SubProcess in the
section about properly detaching the sub-process it has in the example:
$r-spawn_proc_prog ('/path/to/detach_script.pl', $args);
Is there a reason $args is used rather than \...@args as outlined earlier
in the docs?
opt arg2: \...@argv ( ARRAY ref )
Or is it possible to create an $args-[0..10] and pass that?
Thanks!
Tosh
--
McIntosh Cooey - Twelve Hundred Group LLC - http://www.1200group.com/
Re: Apache 1.3 end of life
This is the end for Apache 1.X, so we'll need to find some time to upgrade to the Apache 2.X series for our Arcos backend server (the proxy already runs Apache 2). This means upgrading mod_perl and might involve some code changes (although hopefully not too much). Don't need to do this super soon, but we should have it on our radar for 7.70 or so. On 02/03/2010 10:13 AM, Adam Prime wrote: FYI The Apache Software Foundation and the Apache HTTP Server Project are pleased to announce the release of version 1.3.42 of the Apache HTTP Server (Apache). This release is intended as the final release of version 1.3 of the Apache HTTP Server, which has reached end of life status. There will be no more full releases of Apache HTTP Server 1.3. However, critical security updates may be made available from the following website: http://www.apache.org/dist/httpd/patches/ see: http://www.apache.org/dist/httpd/Announcement1.3.html -- Michael Peters Plus Three, LP
Re: Apache 1.3 end of life
Well, that was embarrassing :) Please ignore. -- Michael Peters Plus Three, LP
Re: global variable
I rewrote IPC::MMA from an earlier CPAN module so that I could use shared memory among Apache children. You can read about it at http://search.cpan.org/~mackenna/IPC-MMA-0.6/MMA.pod On Feb 2, 2010, at 9:45 PM, m...@normalperson.e4ward.com wrote: Hello, Is there a method to setup a global variable for all modperl child processes? Also this variable will be updated sometime, when it get updated, all processes will know it. Thanks.
Re: global variable
Looks like a great module. It says early on in the docs that it doesn't use references; does that mean we need to dereference in order to store that values? Thanks, Boysenberry Payne On Feb 3, 2010, at 10:17 AM, macke...@animalhead.com wrote: I rewrote IPC::MMA from an earlier CPAN module so that I could use shared memory among Apache children. You can read about it at http://search.cpan.org/~mackenna/IPC-MMA-0.6/MMA.pod On Feb 2, 2010, at 9:45 PM, m...@normalperson.e4ward.com wrote: Hello, Is there a method to setup a global variable for all modperl child processes? Also this variable will be updated sometime, when it get updated, all processes will know it. Thanks.
Re: global variable
Just thought to add: One of the most interesting uses of IPC::MMA is to create a shared memory in a PerlPostConfigHandler, and then use it to tie scalars or arrays or hashes in other modules into the shared memory. In write-seldom, read-mostly applications like the cache hash in Image::Size, this can be done without any change to the other module. cmac On Feb 3, 2010, at 9:20 AM, Boysenberry Payne wrote: Looks like a great module. It says early on in the docs that it doesn't use references; does that mean we need to dereference in order to store that values? Thanks, Boysenberry Payne On Feb 3, 2010, at 10:17 AM, macke...@animalhead.com wrote: I rewrote IPC::MMA from an earlier CPAN module so that I could use shared memory among Apache children. You can read about it at http://search.cpan.org/~mackenna/IPC-MMA-0.6/MMA.pod On Feb 2, 2010, at 9:45 PM, m...@normalperson.e4ward.com wrote: Hello, Is there a method to setup a global variable for all modperl child processes? Also this variable will be updated sometime, when it get updated, all processes will know it. Thanks.
Re: global variable
DirDB provides a very simple persistence, sharable data structure, using the file system.
Re: [mp2] Apache2::SizeLimit should be using $s-rss, not $s-size for Linux::Smaps
On 02/03/2010 04:57 AM, Torsten Förtsch wrote:
Well, I tend to disagree. (Fred, Adam please read on.)
Okay. Have you looked at the actual output of test.cgi?
Here's an example of these values just on my local machine, for my bash
interpreter:
[mka...@es-compy ~]$ cat /proc/self/statm
21045 125 107 13 0 65 0
[mka...@es-compy ~]$ cat /proc/self/status | grep Vm
VmPeak:84180 kB
VmSize:84180 kB
VmLck: 0 kB
VmHWM: 496 kB
VmRSS: 496 kB
VmData: 176 kB
VmStk:84 kB
VmExe:52 kB
VmLib: 1548 kB
VmPTE:48 kB
Would you consider that bash is taking up 84MB, or would you consider
it's taking up 496KB? I have 2GB in physical RAM and 2GB in swap. So,
theoretically, I should only be able to open 48 terminals before I OOM.
With 60 terminals open, here's the result of free:
total used free
Mem: 1990 1855134
-/+ buffers/cache: 1140850
Swap: 1999212 1787
If I were simply to add up the Virt column of the top 15 processes in
top on my machine, I'd be using somewhere in the vicinity of 20GB of RAM.
So, either the far older statm technique is also wrong or the patch is wrong.
I would suspect that if you're using column 0 as the size, then statm
is also indeed wrong.
=item * Apache2::SizeLimit-set_max_process_size($size)
This sets the maximum size of the process, including both shared and
unshared memory.
Ahh, but look at the output of both the shared and private
accessors, and they do not add up to VmSize. They add up to RSS.
Perhaps Apache2::SizeLimit::Core should read:
sub _check_size {
my $class = shift;
my ($size, $share, $unshared) = $class-_platform_check_size();
return ($size, $share, defined $unshared ? $unshared : $size - $share);
That would certainly help (because then you would get a more accurate
count of the current private RSS), although I'd argue that VmSize is
still meaningless. I can imagine few instances where you'd want to know
the total size of the address space of a process, even if most of it had
never been mapped to an actual memory page.
-Max
--
http://www.everythingsolved.com/
Competent, Friendly Bugzilla and Perl Services. Everything Else, too.
svn commit: r906273 - in /perl/Apache-DBI/trunk: Changes lib/Apache/DBI.pm
Author: ask
Date: Wed Feb 3 22:54:02 2010
New Revision: 906273
URL: http://svn.apache.org/viewvc?rev=906273view=rev
Log:
Fix bug to allow DBI in startup.pl etc again
Bug report from Adam Prime, patch from Lubomir Rintel
https://rt.cpan.org/Public/Bug/Display.html?id=36346
Modified:
perl/Apache-DBI/trunk/Changes
perl/Apache-DBI/trunk/lib/Apache/DBI.pm
Modified: perl/Apache-DBI/trunk/Changes
URL:
http://svn.apache.org/viewvc/perl/Apache-DBI/trunk/Changes?rev=906273r1=906272r2=906273view=diff
==
--- perl/Apache-DBI/trunk/Changes (original)
+++ perl/Apache-DBI/trunk/Changes Wed Feb 3 22:54:02 2010
@@ -1,6 +1,9 @@
Revision history for ApacheDBI.
1.08
+ - Fix bug to allow DBI in startup.pl etc again
+Bug report from Adam Prime, patch from Lubomir Rintel
+https://rt.cpan.org/Public/Bug/Display.html?id=36346
1.07 05/09/2008
- http://rt.cpan.org/Public/Bug/Display.html?id=31003
Modified: perl/Apache-DBI/trunk/lib/Apache/DBI.pm
URL:
http://svn.apache.org/viewvc/perl/Apache-DBI/trunk/lib/Apache/DBI.pm?rev=906273r1=906272r2=906273view=diff
==
--- perl/Apache-DBI/trunk/lib/Apache/DBI.pm (original)
+++ perl/Apache-DBI/trunk/lib/Apache/DBI.pm Wed Feb 3 22:54:02 2010
@@ -141,7 +141,9 @@
if (!$Rollback{$Idx}) {
my $r;
if (MP2) {
-$r = Apache2::RequestUtil-request;
+# We may not actually be in a request, but in Perl (or
+# equivalent such as startup.pl), in which case this would die.
+eval { $r = Apache2::RequestUtil-request };
}
elsif (Apache-can('push_handlers')) {
$r = 'Apache';
svn commit: r906279 - in /perl/Apache-DBI/trunk: Changes README TODO lib/Apache/AuthDBI.pm lib/Apache/DBI.pm
Author: ask Date: Wed Feb 3 23:05:42 2010 New Revision: 906279 URL: http://svn.apache.org/viewvc?rev=906279view=rev Log: Prepare v1.08 Modified: perl/Apache-DBI/trunk/Changes perl/Apache-DBI/trunk/README perl/Apache-DBI/trunk/TODO perl/Apache-DBI/trunk/lib/Apache/AuthDBI.pm perl/Apache-DBI/trunk/lib/Apache/DBI.pm Modified: perl/Apache-DBI/trunk/Changes URL: http://svn.apache.org/viewvc/perl/Apache-DBI/trunk/Changes?rev=906279r1=906278r2=906279view=diff == --- perl/Apache-DBI/trunk/Changes (original) +++ perl/Apache-DBI/trunk/Changes Wed Feb 3 23:05:42 2010 @@ -1,6 +1,6 @@ Revision history for ApacheDBI. -1.08 +1.08 February 3rd, 2010 - Fix bug to allow DBI in startup.pl etc again Bug report from Adam Prime, patch from Lubomir Rintel https://rt.cpan.org/Public/Bug/Display.html?id=36346 Modified: perl/Apache-DBI/trunk/README URL: http://svn.apache.org/viewvc/perl/Apache-DBI/trunk/README?rev=906279r1=906278r2=906279view=diff == --- perl/Apache-DBI/trunk/README (original) +++ perl/Apache-DBI/trunk/README Wed Feb 3 23:05:42 2010 @@ -1,7 +1,7 @@ DESCRIPTION: -This is version 1.08-dev of Apache::AuthDBI and Apache::DBI. +This is version 1.08 of Apache::AuthDBI and Apache::DBI. These modules are supposed to be used with the Apache server together with an embedded perl interpreter like mod_perl. They provide support for basic Modified: perl/Apache-DBI/trunk/TODO URL: http://svn.apache.org/viewvc/perl/Apache-DBI/trunk/TODO?rev=906279r1=906278r2=906279view=diff == --- perl/Apache-DBI/trunk/TODO (original) +++ perl/Apache-DBI/trunk/TODO Wed Feb 3 23:05:42 2010 @@ -1,10 +1,10 @@ -Showstoppers for 1.08-dev: --- -Migrate test suite to 1.08-dev +Showstoppers + +Migrate test suite [ ?? ] -Nice to have for 1.08-dev: +Nice to have + Add tests for other DBDs starting with PostgreSQL. Modified: perl/Apache-DBI/trunk/lib/Apache/AuthDBI.pm URL: http://svn.apache.org/viewvc/perl/Apache-DBI/trunk/lib/Apache/AuthDBI.pm?rev=906279r1=906278r2=906279view=diff == --- perl/Apache-DBI/trunk/lib/Apache/AuthDBI.pm (original) +++ perl/Apache-DBI/trunk/lib/Apache/AuthDBI.pm Wed Feb 3 23:05:42 2010 @@ -1,7 +1,7 @@ # $Id$ package Apache::AuthDBI; -$Apache::AuthDBI::VERSION = '1.08-dev'; +$Apache::AuthDBI::VERSION = '1.08'; # 1: report about cache miss # 2: full debug output Modified: perl/Apache-DBI/trunk/lib/Apache/DBI.pm URL: http://svn.apache.org/viewvc/perl/Apache-DBI/trunk/lib/Apache/DBI.pm?rev=906279r1=906278r2=906279view=diff == --- perl/Apache-DBI/trunk/lib/Apache/DBI.pm (original) +++ perl/Apache-DBI/trunk/lib/Apache/DBI.pm Wed Feb 3 23:05:42 2010 @@ -22,7 +22,7 @@ require_version DBI 1.00; -$Apache::DBI::VERSION = '1.08-dev'; +$Apache::DBI::VERSION = '1.08'; # 1: report about new connect # 2: full debug output
svn commit: r906280 - /perl/Apache-DBI/tags/rel_1_08/
Author: ask Date: Wed Feb 3 23:08:21 2010 New Revision: 906280 URL: http://svn.apache.org/viewvc?rev=906280view=rev Log: Tag v1.08 Added: perl/Apache-DBI/tags/rel_1_08/ - copied from r906279, perl/Apache-DBI/trunk/
