Re: Memory explodes loading CSV into hash
On Sun, Apr 28, 2002 at 05:18:24PM +0200, Ernest Lergon wrote:
Hi,
in a mod_perl package I load a CSV file on apache startup into a simple
hash as read-only class data to be shared by all childs.
A loading routine reads the file line by line and uses one numeric field
as hash entry (error checks etc. omitted):
package Data;
my $class_data = {};
ReadFile ( 'data.txt', $class_data, 4 );
sub ReadFile
{
my $filename = shift; # path string
my $data = shift; # ref to hash
my $ndx_field = shift; # field number
my ( record, $num_key );
local $_;
open ( INFILE, $filename );
while ( INFILE )
{
chomp;
record = split \t;
$num_key = $record[$ndx_field];
$data-{$num_key} = [ record ];
}
close ( INFILE );
}
sub new...
creates an object for searching the data, last result, errors etc.
sub find...
method with something like:
if exists $class_data-{$key} return...
etc.
Now I'm scared about the memory consumption:
The CSV file has 14.000 records with 18 fields and a size of 2 MB
(approx. 150 Bytes per record).
Omitting the loading, top shows, that each httpd instance has 10 MB (all
shared as it should be).
But reading in the file explodes the memory to 36 MB (ok, shared as
well)!
So, how comes, that 2 MB data need 26 MB of memory, if it is stored as a
hash?
Reading perldebguts.pod I did not expect such an increasing:
Description (avg.) CSV Perl
4 string fields (4 chars)16 bytes (32 bytes) 128 bytes
9 float fields (5 chars) 45 bytes (24 bytes) 216 bytes
5 string fields (rest) 89 bytes (32 bytes) 160 bytes
the integer key (20 bytes) 20 bytes
150 bytes 524 bytes
That will give 14.000 x 524 = approx. 7 MB, but not 26 MB !?
I tried this program in Perl (outside of modperl) and the memory
consumption is only 4.5MB:
#!/usr/bin/perl -w
$foo = {};
for ($i = 0; $i 14000; $i++) {
$foo-{sprintf('%020d', $i)} = 'A'x150;
}
;
1;
So I suggest something else might be going on causing your memory
problems.
-jwb
Re: ASP Cookieless Sessions (WAS Re: Apache::ASP)
Secure sessions are hard work. You need to sit down and evaluate whether or not you
actually need *secure* sessions. If you decide that enough is at stake to really
tighten the screws, then read on.
The problem of the session ID in HTTP_REFERER is easy to tackle. You just need to
rewrite every URL on your secure site to be a redirect from a page that doesn't
include the session ID. For example:
http://cnn.com/
becomes
http://myhost.com/strip_session?url=http://cnn.com/
and the HTTP_REFERER at cnn.com doesn't include the session ID. Clearly the path
strip_session needs to not require a session ID.
-jwb
On 12/07/99, Joshua Chamas wrote:
Serge Sozonoff wrote:
Hi Joshua,
I was wondering if you could give us a little update on the status of
Sessions w/o cookies for Apache::ASP?
Have you started coding it already or is it at the bottom
of a big list of things to do?
Many Thanks,
Serge
I have been working with Remi Fasol on this, and what we
have so far is a set up that uses the QueryString for
keeping track of session ids, in addition to the cookie
implementation that exists.
The QueryString method represents nice backup to the existing
cookie implementation for users that have cookies turned off,
but functionally is not as nice because someone will get a
new session id when reentering the site. So if a user starts
browsing off site, and instead of hitting the back button,
just reenters your URL in the location bar, they will start a
new session. Remi commented that Amazon's sessions seem to
have the same limitations.
These cookieless sessions are implemented with a config setting
SessionQueryStringID that if set, ASP will automatically parse
out of the QueryString the session-id, and use that if there
is no such session-id cookie. Then, whatever URL's you
want the system to keep track of, the web developer uses the
$Server-URL($url, { %params })
API extension to create a URL that will have a session id
built into it. This $Server-URL() method will be nice
for just building URL's in general with or without the
cookieless sessions, as it will joing the params hash
into a nice ?query_string automatically.
It reoccured to me just now (back from a sessions
methods discussion a long time ago) that these query string
cookies might show up in the referer logs of other sites if you
have offsite links on your session id pages. I tried a workaround
just now where a redirect program would handle offsite links, but
the HTTP_REFERER is sticky to the last page visited, and I see
no workaround to this security issue.
What options are there anyone, for real cookieless sessions,
without this security risk ??? We can't use IP authentication
because of proxies/NAT, maybe an SSL cert, but not everyone has
this, the UserAgent is not stratified enough to mean much,
so that what, when we are trying to get past cookies here.
-- Joshua
_
Joshua Chamas Chamas Enterprises Inc.
NodeWorks free web link monitoring Huntington Beach, CA USA
http://www.nodeworks.com1-714-625-4051
--
Jeffrey Baker * [EMAIL PROTECTED]
Re: installation problems
"Jesús Mª Lasso Sánchez" wrote: hi: I'm working with linux servers based on RedHat 6.0 with Apache 1.3.9. My problem is when compile the mod_perl module before to install Apache. More exactly when i'm doing: perl Makefile.PL it produces this error when is creating the makefile: + doing sanity check on compiler and options ** A test compilation with your Makefile configuration ** failed. This is most likely because your C compiler ** is not ANSI. Apache requires an ANSI C Compiler, such ** as gcc. The above error message from your compiler ** will also provide a clue. Aborting! the OS is RedHat 6.0 and the c compiler is the packet "egcs". The problem just happen when compile "mod_perl". I compiled Apache 1.3.9 without problems. Could you send the output of 'perl -V' and 'which `perl -V:cc`? Also what command line did you use for Makefile.PL? -jwb -- Jeffrey W. Baker * [EMAIL PROTECTED] Critical Path, Inc. * we handle the world's email * www.cp.net 415.808.8807
Where to start with mod_perl (was Re: Flaky software)
Glen Lee Edwards wrote: I'm brand new to the list, so I thought I'd better ask a question before making another post: What is the specific purpose/slant of this list? Is it a developer's list? A general list to help those running Mod Perl? Are only programming geniuses allowed to post, or can the average joe ask dumb questions? I've never run Mod Perl in a real life setting but like the concept. I own several domains on commercial servers that I plan to eventually move to my own server, so am trying to get acquainted the the best way to set it up and administer it. If this isn't a list for those just learning Mod Perl, please point me in the right direction. No this is just the place for beginners. First you should apply a minimum of effort by reading the guide at http://perl.apache.org/guide. Perhaps you might also want to buy the book at http://www.modperl.com/. After that you probably won't have any basic questions left :) But if you do, feel free to ask them here. -jwb -- Jeffrey W. Baker * [EMAIL PROTECTED] Critical Path, Inc. * we handle the world's email * www.cp.net 415.808.8807
Re: Perl Domination in CGI Programming
Stas Bekman wrote: [slashdot summary snipped] Well, if the trend continues we ought to learn Java some day :) Ha! Well they are going to have to make JServ operable at some point if they want to replace Perl. Seriously, could they make their configuration process more opaque? I went to the JServ morning session in Monterey, and I was extremely disappointed. The whole first hour was spent on building and configuring the servlets. They seem to have several non-orthogonal configuration files that live in different places and have very hard to explain syntax and on and on and on. Blah. Even as a developer I want something that has a simple and straight-forward installation procedure. For the operations department it is even worse. JServ has a long way to go before it doesn't suck in that regard. And don't even get me started on their session handling and load balancing [sic]. -jwb -- Jeffrey W. Baker * [EMAIL PROTECTED] Critical Path, Inc. * we handle the world's email * www.cp.net 415.808.8807
Re: Checking for valid dates
"Tubbs, Derric L" wrote: Is there a reasonably easy method to make sure that an entered date is valid, I.E. is not 30 Jan 99? I am using Time::Local to convert a date entered through an HTML form into the epoch offset (or whatever you call it) and would like to make sure that valid dates are entered. Thanks Just because I'm curious, please explain what isn't valid about 30 Jan 99. ObPerl: The Date::Calc module might be what you need. -- Jeffrey W. Baker * [EMAIL PROTECTED] Critical Path, Inc. * we handle the world's email * www.cp.net 415.808.8807
Re: mod_perl 2.21 segmentation fault
glibc 2.0.7 ^^^ Maybe you should consider upgrading to a non-experimental libc. Aside from that, I have no clue. I know dozens of people who use this same setup, and they all work out fine. Did you compile everything together yourself? What happense when you remove mod_perl or mod_ssl? Maybe you should wait for openSSL to actually be released. -jwb apache 1.3.9 mod_perl 2.21 mod_ssl 2.2.4 openssl 0.9.4 CGI.pm 2.56 Oracle 8.0.5 DBD::Oracle 1.03 DBI 1.13 XML::Parser 2.27 gcc 2.7.2.3 linux kernel 2.2.13 -- Radovan Semancik ([EMAIL PROTECTED]) System Engineer, Business Global Systems a.s. http://storm.alert.sk -- Jeffrey W. Baker * [EMAIL PROTECTED] Critical Path, Inc. * we handle the world's email * www.cp.net 415.808.8807
Re: Failing to reconnect after Oracle shutdown abort (Apache::DBI)
Greg Stark wrote:
*** From dbi-users - To unsubscribe, see the end of this message. ***
*** DBI Home Page - http://www.symbolstone.org/technology/perl/DBI/ ***
Tim Bunce [EMAIL PROTECTED] writes:
On Mon, Nov 01, 1999 at 09:01:48PM +, Tim Bunce wrote:
Has anyone experienced a situation where a process (httpd for example)
can't reconnect to Oracle after a "shutdown abort"?
Thanks for your replies.
The problem reported to me which prompted this email has actually
proven to be a user script error. The old database handle was being
reused, rather than the new one returned from a fresh call to
DBI-connect after the shutdown and restart.
With a web server you always want to reuse the database handle as long as
possible. That's true of any long-running application though.
How is the application supposed to know it has been disconnected?
Maybe the driver should (maybe optionally) automatically reconnect?
That's what the driver handle's ping method is for.
if (!$dbh-ping) { reconnect; }
-jwb
--
Jeffrey W. Baker * [EMAIL PROTECTED]
Critical Path, Inc. * we handle the world's email * www.cp.net
415.808.8807
Re: Apache::Session and File Upload (Was: Apache::Session hangs script)
Kip Cranford wrote:
Again, I'm using mod_perl 1.21, apache 1.3.9, Apache::Session 1.03, on a
RedHat 6 linux system with perl 5.005_03, and am using Netscape Comm.
4.51 as my browser.
The problem now seems to be Apache::Session and file uploads. My
handler is providing a simple file upload interface, and I'm using
Apache::Session to keep track of filenames, content types, sizes, etc.
Using a very simple script, in which I store only a single scalar
variable in my session, and using the "multipart/form-data" encoding
type on my form, I can get the script to hang every time. It _always_
hangs in the same place in the "op" function:
DB1 IPC::Semaphore::op(/usr/lib/perl5/5.00503/IPC/Semaphore.pm:90):
90: croak 'Bad arg count' if @_ % 3;
DB1 IPC::Semaphore::op(/usr/lib/perl5/5.00503/IPC/Semaphore.pm:91):
91: my $data = pack("s*",@_);
DB1 IPC::Semaphore::op(/usr/lib/perl5/5.00503/IPC/Semaphore.pm:92):
92: semop($$self,$data);
The problem is that you are leaking session handles. For
Apache::Session to work, there must be zero references to the session
hash at the end of the request.
-jwb
--
Jeffrey W. Baker * [EMAIL PROTECTED]
Critical Path, Inc. * we handle the world's email * www.cp.net
415.808.8807
Re: Runaway processes
Joshua Chamas wrote: Mike Dameron wrote: We have several scripts using Apache::ASP and DBI, which return information from an Oracle database. The problem is that if I run a report that taken a long time (not sure how long but over 30 minutes) and the connection either times out or I hit the stop button the oracle process which gets spawned does not die. I can see in the process list that the oracle process is still tied to a httpd process and continues to eat up cpu cycles. Why would you run a report from a web site that would take 30+ minutes to finish ? Typical web connection timeouts are a few minutes or less ? I think the real solution is to make your reports fast enough to run from a web site, like run the real report every night, and cache its results in the database for later that you can get them from the browser instantly. Another workable solution is to make the reports in real time, but deliver them by email. Use the web interface to initiate the reporting process and send a confirmation. Cheers, Jeffrey -- Jeffrey W. Baker * [EMAIL PROTECTED] Critical Path, Inc. * we handle the world's email * www.cp.net 415.808.8807
Re: please comment on new art for perl.apache.org
Alex Schmelkin wrote: 1. What browsers are different percentages of your user populations using? 2. What screen resolutions are different percentages of your user populations using? 3. What purpose will the site serve - simply providing information, or a marketing/evangelical need? How important are each of these considerations? 4. Who is the ultimate audience? - Who needs this information, and why? 5. What does the audience already know, and what do they need to know? 6. What is the feeling or mood to be? 7. What are some of the positive attributes of the current sites as you see them? What are some of the negative attributes? 1) and 2) are easy. All of them, and all of them. To design considering anything else is suicide. -jwb -- Jeffrey W. Baker * [EMAIL PROTECTED] Critical Path, Inc. * we handle the world's email * www.cp.net 415.808.8807
Re: Bug in libapreq makes form elements stick to an apache child
Cliff Rayman wrote: `perldoc -f defined` yields a couple of sentences: You may also use Cdefined() to check whether a subroutine exists, by saying Cdefined func without parentheses. On the other hand, use of Cdefined() upon aggregates (hashes and arrays) is not guaranteed to produce intuitive results, and should probably be avoided. I hate it when that happens. why not use: if(@foo_in) instead of: if (defined @foo_in) Yeah. I guess the reason I do the latter is b/c I want the code to reflect what I am actually trying to test. I don't really want to test the trueness of @foo, I want to test for it's existence. But in perl the operation is overloaded. Feh. I think I will test for defined $foo_in[0] instead. -- Jeffrey W. Baker * [EMAIL PROTECTED] Critical Path, Inc. * we handle the world's email * www.cp.net 415.808.8807
Re: mod_perl on Apache 2.0
Matt Sergeant wrote:
On Thu, 04 Nov 1999, Jeffrey Baker wrote:
I'm assuming that Perl itself is reentrant, since it has been embedded
in multithreaded environments before (IIS). Hopefully someone can
comment on that.
This work was based on PERL_OBJECT support, which is currently only
available on windows. It's a perl interpreter wrapped up in a C++ object so
that the whole thing is reentrant. For 5.6 I believe a similar
functionality but in C is being added (or is moving out of experimental
mode) called MULTIPLICITY.
I believe -DMULTIPLICITY is already in 5.005_03. Check "man perlembed".
Now suppose we have more than one interpreter instance
running at the same time. This is feasible, but only if
you used the -DMULTIPLICITY flag when building Perl. By
default, that sets PL_perl_destruct_level to 1.
Let's give it a try:
#include EXTERN.h
#include perl.h
/* we're going to embed two interpreters */
/* we're going to embed two interpreters */
#define SAY_HELLO "-e", "print qq(Hi, I'm $^X\n)"
int main(int argc, char **argv, char **env)
{
PerlInterpreter
*one_perl = perl_alloc(),
*two_perl = perl_alloc();
char *one_args[] = { "one_perl", SAY_HELLO };
char *two_args[] = { "two_perl", SAY_HELLO };
perl_construct(one_perl);
perl_construct(two_perl);
perl_parse(one_perl, NULL, 3, one_args, (char **)NULL);
perl_parse(two_perl, NULL, 3, two_args, (char **)NULL);
perl_run(one_perl);
perl_run(two_perl);
perl_destruct(one_perl);
perl_destruct(two_perl);
perl_free(one_perl);
perl_free(two_perl);
}
--
Jeffrey W. Baker * [EMAIL PROTECTED]
Critical Path, Inc. * we handle the world's email * www.cp.net
415.808.8807
Re: Installation Problem with Apache::Session and Mason
[EMAIL PROTECTED] wrote:
I'm having a problem setting up Apache::Session on
one machine and not another, and I'm hoping that
someone will recognize what I'm doing wrong.
I'm using Mason to make the connection to
the package when Apache starts up. On the problem
server, the following happens on server start-up:
starting httpd...
Ambiguous use of read = resolved to "read" = at
/usr/lib/perl5/site_perl/Apache/Session/SysVSemaphoreLocker.pm line
35.
Ambiguous use of write = resolved to "write" = at
/usr/lib/perl5/site_perl/Apache/Session/SysVSemaphoreLocker.pm line
35.
Ambiguous use of {read} resolved to {"read"} at
/usr/lib/perl5/site_perl/Apache/Session/SysVSemaphoreLocker.pm line
43.
Ambiguous use of {write} resolved to {"write"} at
/usr/lib/perl5/site_perl/Apache/Session/SysVSemaphoreLocker.pm line
44.
Ambiguous use of {read} resolved to {"read"} at
/usr/lib/perl5/site_perl/Apache/Session/SysVSemaphoreLocker.pm line
54.
Ambiguous use of {write} resolved to {"write"} at
/usr/lib/perl5/site_perl/Apache/Session/SysVSemaphoreLocker.pm line
61.
Ambiguous use of {read} resolved to {"read"} at
/usr/lib/perl5/site_perl/Apache/Session/SysVSemaphoreLocker.pm line
68.
Ambiguous use of {write} resolved to {"write"} at
/usr/lib/perl5/site_perl/Apache/Session/SysVSemaphoreLocker.pm line
74.
Ambiguous use of {read} resolved to {"read"} at
/usr/lib/perl5/site_perl/Apache/Session/SysVSemaphoreLocker.pm line
81.
Ambiguous use of {read} resolved to {"read"} at
/usr/lib/perl5/site_perl/Apache/Session/SysVSemaphoreLocker.pm line
85.
Ambiguous use of {write} resolved to {"write"} at
/usr/lib/perl5/site_perl/Apache/Session/SysVSemaphoreLocker.pm line
92.
Ambiguous use of {write} resolved to {"write"} at
/usr/lib/perl5/site_perl/Apache/Session/SysVSemaphoreLocker.pm line
96.
Ambiguous use of {read} resolved to {"read"} at
/usr/lib/perl5/site_perl/Apache/Session/SysVSemaphoreLocker.pm line
103.
Ambiguous use of {write} resolved to {"write"} at
/usr/lib/perl5/site_perl/Apache/Session/SysVSemaphoreLocker.pm line
106.
Ambiguous use of {read} resolved to {"read"} at
/usr/lib/perl5/site_perl/Apache/Session/SysVSemaphoreLocker.pm line
110.
Ambiguous use of {write} resolved to {"write"} at
/usr/lib/perl5/site_perl/Apache/Session/SysVSemaphoreLocker.pm line
111.
That is Perl being anal about the use of unquoted read and write as hash
keys. I should probably fix this but it isn't fatal.
The server does start, however. But when I try to access
a page that uses the session hash, the following appears
in the httpd error log:
[Thu Nov 4 11:17:21 1999] [error] Permission denied at
/usr/lib/perl5/site_perl/Apache/Session/SysVSemaphoreLocker.pm line
64.
This is a different problem. At some point someone other than the user
that your httpd runs as created the semaphore block that you are tyring
to use. First, consult the ipcs and ipcrm man pages. Then, as root,
remove the semaphore block that is causing the problem and allow the
httpd to recreate it as the nobody user.
-jwb
I've checked the permissions on the module itself, the
directory where the session files are stored, and the
permissions of the owner of the httpd process. I've
had no problem on another server, also running Linux
2.0.35. All the corresponding PERL modules seem synched,
as well as the Apache httpd binaries.
Here is the script that is called (part of Mason's
start up script called handler.pl):
# This block of code can be enabled to create a session-hash that
every
# component can access. This is useful for maintaining state
across
# multiple requests. The Apache::Session module is required.
#
my %session;
my $cookie = $r-header_in('Cookie');
$cookie =~ s/SESSION_ID=(\w*)/$1/;
tie %session, 'Apache::Session::File', $cookie, {'Directory' =
'/tmp/session'};
$r-header_out("Set-Cookie" =
"SESSION_ID=$session{_session_id};") if ( !$cookie );
# This creates a global called %session that is accessible in all
components.
# Feel free to rename this as needed.
#
local *HTML::Mason::Commands::session = \%session;
$ah-handle_request($r);
untie %HTML::Mason::Commands::session;
This is Apache-Session-1.03 and HTML-Mason-0.72 on
Apache 1.3.6.
Thanks for any help/hints you can provide.
Marc Kelly
[EMAIL PROTECTED]
--
Jeffrey W. Baker * [EMAIL PROTECTED]
Critical Path, Inc. * we handle the world's email * www.cp.net
415.808.8807
