Title: RE: Search Engine Theory
Check out the book "Managing Gigabytes"
Text indexing theory and algorithms. Source code too.
-Original Message-
From: Jamie Krasnoo [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, February 27, 2001 3:03 PM
To: Modperl
Subject: Search Engine Theory
C
Title: RE: Debugging mod_perl with gdb
Hey thanks. I'll try this. I tried the 'man gdb' command and it didn't help much I'm afraid...
-Original Message-
From: sterling [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, February 06, 2001 12:33 PM
To: Shane A
is doing when the memory corruption occurs. Obviously if I could narrow the offending line of code (if possible) I might be able to better understand where the real bug is.
Shane
-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, January 30, 2001 5:25
Modperlers...,
Hello, this is Shane Nay, I used to post quite a bit. (Hello Josh, Stas,
Peren, Doug, and many others of course) BTW Stas,
$contributors=~/Shane/Shane Nay/;, and big Thanks .
Well I'm posting now, (and resubscribed :) because I need someone to fill a
modperl develop
Went ahead and did that, same error. I have not added a
Cc: "Shane Reid" <[EMAIL PROTECTED]>; <[EMAIL PROTECTED]>
Sent: Tuesday, December 19, 2000 9:40 PM
Subject: Re: Apache::ASP
> Stas Bekman wrote:
> >
> > On Tue, 19 Dec 2000, Shane Reid wrote
/local/lib/perl5/site_perl/5.005/i386-freebsd/Digest/MD5.pm line 20.
BEGIN failed--compilation aborted at
/usr/local/lib/perl5/site_perl/5.005/Apache/ASP.pm line 13.
[Tue Dec 19 17:55:44 2000] [error] Undefined subroutine
&Apache::ASP::handler called.
Thanks
-Shane Reid
suexec wrapper and configured it,
can suexec be used, I was unclear as to if it could be used and if it was
being used if some files needed to be modified etc. Any help on correcting
this error would be appreciated...
Shane
suexec wrapper and configured it,
can suexec be used, I was unclear as to if it could be used and if it was
being used if some files needed to be modified etc. Any help on setting it
up suexec with modperl or just correcting this error would be
appreciated.
Shane Reid
Title: RE: :Parse segmentation fault
no
clue. I emailed your comment to a friend at work and see what he can make
of it. He's the fellow that found the cause of the segfault in the first
place.
Shane
-Original Message-From: Herrington, Jack
[mailto:[EMAIL PROTECTED]
: Shane Adams; '[EMAIL PROTECTED]'Subject: RE:
:Parse segmentation fault
I did a
little more digging around and found that you could also avoid the problem by
turning off EXPAT in apache with:
Rule
EXPAT=no
Which fix
is more preferable?
-Original Messag
Title: RE: :Parse segmentation fault
Yes. We found a problem in Expat.pm line 451 (in sub parse). The following chunk of code (latest version from cpan)
sub parse {
my $self = shift;
my $arg = shift;
croak "Parse already in progress (Expat)" if $self->{_State_};
$self->{_State_} =
ng with typeglobs under an eval ... I believe that * is a typeglob in perl yes?
-Original Message-
From: Shane Adams
Sent: Tuesday, September 26, 2000 4:16 PM
To: [EMAIL PROTECTED]
Subject: Core file (debugging info turned on/stack trace)
Apache 1.3.12, mod_perl 1.24, Perl 5.6.0, Redhat
Title: Core file (debugging info turned on/stack trace)
Apache 1.3.12, mod_perl 1.24, Perl 5.6.0, Redhat 6.1
...
Program terminated with signal 11, Segmentation fault.
...
0 0x814abd4 in Perl_sv_setsv (dstr=0x8d34514, sstr=0x84afba4) at sv.c:2774
#1 0x813b6b3 in Perl_pp_sassign () at pp_ho
Title: Core dumping
Sorry
I forgot to include the stack trace with debugging turned
on:
#0 0x81714f7 in Perl_dounwind (cxix=3) at
pp_ctl.c:1254#1 0x8171bd1 in Perl_die_where (
message=0x848eb68 "Can't upgrade that kind of scalar at
/home/shane/sparty/runtime/perl/lib/site_
seg fault is received on line 1254 in pp_ctl.c
(course you could probably tell that =)
To bad
I can't take a peek into the POPSUB macro itself... It's like 20 lines long and
there is no way to get gdb to display the macro. Least no way I
know.
Shane
How
the hell do people actua
x27;ve hit the latest releases of each piece it seems.
I don't know if this will shed any details as to what the problem is.
Any help is appreciated.
Shane
This causes a qmail process (there are like 20 different ones and I forget which is which - check the docs) to wake up and drain the queue.
If you want anymore speed then that, you have to either install ram disks or seriously write your own mta. We installed ram disks =)
Shane
-Original Messa
only allows "X amount" of memory,
and then it digs into its own swap rather than tearing down everyone elses
modperl.
Shane.
least
you _could_ have modperl and a machine of your own without paying the hefty
cost of a colocation service. (You can restrict memory for each VM, and each
VM has bios and all that jazz..., check it out vmware.com)
Thanks,
Shane.
(Credit where credit is due: This was originally Josh Chamas&#x
sion, just a timestamp. As simple as putting it as a
hidden field in the form.., i.e. popping a good old seconds since Jan 1 1970
aka unixdate.
> Does anyone have any experience with this?
In the thread sense were I have to make sure that multiple threads don't trash
eachothers data, yes. But in an interface sense, no..., but it's all the same
at some level.
Have a good one Keith,
Shane
second on a decent computer. (700Mhz or so) If persistent connections could be
worked out that number could be higher. The other problem is getting the right
data structure to handle this stuff..., working on that right now.)
Thanks,
Shane.
BTW: This is part of a larger project to abstract fa
put stuff in a session that will be used on 80% of
pages. Normally it turns out that I only insert/modify the session on
login, and try to keep subsequent updates to a minimum. (Cost to benefit
analysis, the session can save you a lot of sql queries, or it can add a lot)
I'm working on sites right now that have very specific needs in terms of
performance, but everyones needs are different.
Thanks,
Shane.
my sessions really small, and put
more permanent data elsewhere, does anyone else have a "best practices" sort of
thing on what should/should not be stored in a session? (This seems to come up
almost weekly I think)
Thanks,
Shane.
(Also you should read up on freeze thaw, because they can&#x
there a lower "overhead" to a function call in XS? (I know
it's all pretty trivial in terms of array creation of references, but I'm just
_curious_, not to mention it probably has to do a dlopen to the library, so it
might actually be higher..., but like I said I'm just sort of wondering)
Thanks,
Shane.
o posting,
but should have noted the copy when shifting off the args, implicit deep copy
for '='s is still a strange thing for a primarily c programmer...)
Thanks,
Shane.
and use it as such) it's 1) faster 2)
changes the value when it gets back to you, because it's not working from a
copy, it's working from the "real deal". So if you're thinking of passing huge
strings, then it is in your best interest to pass by reference..., and if
you're passing hashes and arrays directly... SHAME ON YOU! :-)
Happy perling!
Shane.
(I tried to use less obfuscated c code..., sorry if I failed :-( )
t down to
a dull roar. People that include mini commercials like Matt tend to keep it
pretty small and professional, like 4 lines. 15 is too much..., no yelling
either.
Shane.
(Totally offtopic, but I had to say something)
On Wed, 14 Jun 2000, you wrote:
> On Wed, Jun 14, 2000 at 11:29:23AM +0000, Shane Nay wrote:
>
> > I thought he was compiling everything statically. (httpd + modperl + embperl)
> > The way I suggested isn't right on any platform :-). (Didn't see the message
> >
sage
context under the "what happened log")
I can see from Jens-Uwe's reply that the DSO building process is a LOT
different on AIX than on linux with gnutools. (Hey what happened to gcc
--shared? :-)
Sorry,
Shane.
1.blah/src/ap/*.o)
If nm shows the symbols in place, I would say it's probably the AIX linkers
fault. I don't know much about AIX specifically but I have heard lots of
people complain about their linker :-(.
Thanks,
Shane.
e VMWare
to look at a prototype for a product I was helping out with I hadn't used
Windows in about 2 years. (BTW: Huge plug for VMWare, that's some
REALLY cool software, not just for runing windows on linux, but testing other
OSes... I can't say enough good stuff about vmware)
Thanks,
Shane.
handle a few
> hundred simultaneous users easily. If you want more, use a standard irc
> daemon as a hub, and connect several such perl servers to it; if you
> want to have a java applet client too, it can talk directly to the irc
> server.
Nothing could be more true. Single thread is the ONLY way to go..., anything
other than that is a massive waste of trying to implement.
Thanks,
Shane.
--
le total, then do
whatever you want :), but if you want it to be scalable, you need to
think long and hard about how you're using your sessions.
Shane.
>
> Option three (removing all page-specific state from the session hash)
> seems like the right thing to do. I'd like to avoid it
hierarchies in Perl, and you can have a
bunch of flat files that are impossible to make any mods to. In Java
everything MUST be a class. So it forces you to design things
right. But the underlying problem is what's right for one thing isn't
right for others. So Perl becomes sort of a "tool for any job", Java
becomes a tool only for a long term project. (Overhead to start)
Shane.
>
> Gerd
>
>
ied to use String tokens in java and run a benchmark
on it and compared it to a regex? PATHETIC. String handling is crap
in Java. The explanations I've heard are overhead for creating string
objects for each token seperator.
Oh well, those are my disorganized thoughts :-)
Shane.
On Sun,
e tends to lend itself to better performance in the long run anyway,
> because it's easier to optimise serious performance problems away.
Can't really disagree with that. Clean code is 100x easier to work on
later.
Shane.
rl/test.pl line 12.
> BEGIN failed--compilation aborted at /home/httpd/perl/test.pl line 12.
> Any hints, Thank you very much.
>
> Steven.
>
End of code:
1;
Every module has to return a true value at the end of it you could
probably do 1000; at the end, but 1 is convention.
Thanks,
Shane.
> ]- I'm not sure but I think phttpd is included in RH6.2
Wow..., you're right. What a trip. Okay, I'm going to grab the srpm
for RH's site, and extract it into a tarball. It'll be on my anonymous
ftp under the pub/matofali directory if anyone's intere
better as well, and can dish out to more
clients simultaneously. However, I think that apache will incorporate
a lot of the stuff used in Static accel's soon..., so the gap will be
small. But this is very OT..., we're dynamic remember! :-)
Thanks,
Shane.
On Mon, May 15, 2000 at 12:49:12
an apache does.
They only have ONE process per CPU :-). No matter how many clients
their dishing out to simultaneoulsy.
Thanks,
Shane.
>
ing a local copy
> soon. If not I'll mirror it, and let anyone know that's interested.
> So if you are, drop me a note.
>
> Thanks,
> Shane.
*Update* I just got an email reply to Zach (man that was like 5
minutes..., very cool), anyhow. He said that they pulled a bunch of
h
ainst
> > in kernel httpd serving outside of an embedded device. However, you
> > might want to look at phhttpd (people.redhat.com/zab/). It's faster
> > than khttpd, and it runs in userland.
> >
> > Thanks,
> > Shane.
>
> Shane,
>
> I noticed t
d it...
Well, I'm not sure specifically about khttpd..., I'm generally against
in kernel httpd serving outside of an embedded device. However, you
might want to look at phhttpd (people.redhat.com/zab/). It's faster
than khttpd, and it runs in userland.
Thanks,
Shane.
>
> Thanx
> =
> iVAN
> [EMAIL PROTECTED]
> =
W)
The last thing is maybe you should look into isolating a lot of this
stuff into modules. I'm sure your sub's are shared quite frequently
between many scripts..., well, there is no need to suck up all that
extra memory, just pull that into your own module set.
Thanks,
Shane.
The rule is
that is something
interesting to check up on. I think it's mostly that us mod_perl
programmers have been oftly busy :-). Lord knows I have.
Shane.
. In terms of what I do in the field. I make modules
that become "objects" of a certain report, or wrap up some more
complex logic in it. Then the application is just simply relagated to
"flow control" between the various objects. Now this is what I do in
perl... I tend to hand more over to the procedural language when
writing stuff in c/c++... mostly because mixing the two can be really
messy sometimes. (Not when writing an app from scratch, but
supporting a legacy c app. Including c++ in there makes things
nutty.)
Thanks Gunther,
Shane.
good
algorithms. It's supplemental to good algorithms, and indeed
sometimes crufty/beautiful Object interfaces can get in the way of good
algorithms. (OO interfaces sometimes make Nazi designers out of the
best of us..., we allow our sense of design to get in the way of our
sense of algorith
ly involved
with the caching)
Thanks,
Shane.
On Fri, May 05, 2000 at 03:24:40PM +0100, lists wrote:
> I am having a tricky problem with a mod_perl backend and a vanilla proxying
> front end
>
> I have set up:
>
> Apache 1.3.9 + Mod_perl 1.23 + Embperl 1.2.1 as a back-end ser
r part, that way you can analyze how much memory your
programs are using and whether you need to re-think your design
strategy or implement a cleaner. Any cleaner, a real aggresive one,
or a really kick back one. In any event, I just wanted to mention
that this was a really good idea of yours (the callbacks).
Thanks,
Shane.
ake a lot more people than one to implement. I'm thinking of
laying the foundation that others could build off of later, like
puting the mod_async thing together with void* pointers so it could
hold a pointer to anything and pass it back to who ever called it
later.
Anyhow..., that's later, I'm working on sending first... it's kind of
more universally applicable anyway.., the reading is a "specific
case".
Thanks,
Shane.
d only cost $100 (US) in the states,
and would greatly alleviate your problem. But if you're running
pre6..., downgrade, quick! :-)
Later,
Shane.
On Tue, May 02, 2000 at 09:34:45AM -0300, FEITO Nazareno wrote:
> Hi, wassup ppl...
> I have a little problem, when I start up my apache
module is in for lots of cool stuff. And I'm
speaking of lots of developers working on it. Four that I know of for
sure, two of which have already done a huge body of work on this
module.
Thanks,
Shane.
On Sat, Apr 29, 2000 at 12:35:39AM -0400, Greg Stark wrote:
>
> A few lessons
You need to take a sledge hammer to your config... get rid of almost
EVERYTHING. Here is a copy of mine..., oh, and you might want to look
back a few days, I posted a really good link about mod_proxy and
mod_rewrite. Or search apache's site for mod_proxy, or just proxy...
that's how I came up wi
tiny (500k, maybe smaller) processes feeding
these clients. I posted a URL that shows you how to config it with
modrewrite..., check it out, it was a couple days ago. Or just search
the mod_perl list for mod_proxy.
Later,
Shane.
On Thu, Apr 27, 2000 at 01:09:51PM -0300, FEITO Nazareno wrote
n the amount of free resources
for other programs.
Thanks,
Shane.
On Thu, Apr 27, 2000 at 04:35:26PM +0100, Matt Sergeant wrote:
> I'm behind a 64k leased line here (net access is *extremely* expensive
> here in the UK) and I was thinking, a proxy front end is probably really
> not necess
mmend you back it up with a real reason. Like
it's too slow, or it sucks up too much memory. Or developement time
is significantly impacted.
Maybe I'm just in a bad mood..., but this isn't the kind of thing that
should be on the list.
Thanks,
Shane.
ost to the list for it
it's likely that either I or Zach will answer really quickly.
It's at: people.redhat.com/zab/ , or they'll probably pull that down
at some point in the future since he doesn't work there anymore, drop
me an email since Zach no longer works for RH. Its just a lightning
speed HTTPD static accelerator.
Thanks,
Shane.
_proxy..., to me, there would be no reason to contemplate another
package. (Clearly HUGE personal bias here)
Thanks,
Shane.
BTW: As Mike Hall brought up, they wanted to kill it because there is
no maintainer for mod_proxy. There are a lot of people
working on it right now (Graham and Sam..., Gr
your
modperl process isn't really something to consider) But,
as Perrin noted, if you have some scripts that query TONS of data, and
others relatively small, the 2:1 ratio might make sense, but no matter
what 1:1 will always produce the most overall efficiency in the
"everythings local scenario".
Okay, these are my thoughts, what do you think?
Shane.
d up. So each of those mod_perl
threads must have a couple of mysql connections open, so you are
probably right..., he is partially CPU bound, but also io bound, so
his 8 processes was probably dead on. I didn't think about that for
some reason... I guess I have a one track CPU bound mind :-).
Thanks,
Shane.
nything about it because theres no apps. But
theres no apps because theres no OS support... :-)... I hate those
sorts of situations. Who knows when windows will get it? Maybe
Windows 3k.
(I should note, there have been lots of "tweaks" along the way, but
nothing essentially different in concept)
Thanks,
Shane.
>
> -Justin
he module, and give it some state
context, and the ability to read and write, it could be very very
usefull for nearly anything for apache. (Obviously sigqueue's aren't
implemented on every platform, so either poll(), or select would take
it's place in these scenarios... still more efficient than having a
"heavy" mod perl enabled server sending data to 28.8k client)
Anyhow... thanks Doug,
Shane.
(Man, you've got a good memory... that was over two months ago I
brought up the GC)
phaned program, and do some real work on it. It looks like it
was designed well to begin with, but suffered from "patch bloat" with
no one to look after it later.
Thanks,
Shane.
(BTW: in ./apache_blah/src/modules execute: cat
whateverthefile.patch|patch -p0 )
it
"works right", and theres no need to change it for the stuff we want
to add, so I'm a little reluctant to tear it apart just yet. Anyhow,
I'm getting too sleepy to finish this off, so tomorrow. (I have to
admit that this functionality your speaking of is probably the coolest
thing about the JServ engine..., with a little tweaking mod_proxy
could give us that..., but a lot of it's sort of patched up if you
know what I mean at the moment)
Thanks,
Shane.
>
> Les Mikesell
>[EMAIL PROTECTED]
then do what Matt said and
everything should work fine. I have a similar config over here.
Thanks,
Shane.
ook for this is "Advanced
Perl Programming"... published by Oreilly. (of course) There is a
chapter in there written by Malcolm Beatie (well pieces of the chapter)
that are pretty good..., but I'm afraid they might not go into enough
depth on these exact issues. Not only tha
moryhog. The total memory usage would be 20*40k=800k, and it
would never re allocate that variable down to a reasonable size?
That's the behaviour I thought that would happen, but I was thinking
the value would be retained through the stack (clearly my error).
(Okay, so sue me it would call somememory hog more than 20 times, I'm
just trying to clear up something :->)
Thanks,
Shane.
On Tue, Apr 18, 2000 at 02:07:24AM -0400, Jeff Stuart wrote:
> I understand that. :) And that was something that I had to learn myself.
> :) It's a BAD thing when suddenly your httpd process takes up 100 MB. :)
> It's just that it sounded like Shane was saying that his htt
You're right. I am mistaken :-(. Just tested it, and it was
something silly in an old script I had lying around that I thought was
a bug... my mistake. (Note to self: Test all examples before
posting... or you look like an idiot :-) )
Sorry,
Shane.
> I think you're mist
for arrays and hashes to do the following at
the bottom of each code block in registry scripts. (True good note)
@somearray = ();
%somehash =();
delete $somescalar;
(and don't forget untie :-> )
Basically it will clean up *most* of the memory taken up by these
variables.
Thanks,
Shane.
"Funnystuff is $funnystuff";
> $funnystuff="funny";
> }
>
> Try that, and you will truely find out how memory inefficient modperl
> is :-). I haven't tested that second one, but based on what I know
> about how perl works... it should prove... interesting
quot;;
}
Try that, and you will truely find out how memory inefficient modperl
is :-). I haven't tested that second one, but based on what I know
about how perl works... it should prove... interesting.
Thanks,
Shane.
>
> - Perrin
If anyone that knows the internals of mod_perl is seriously
interested in working on a garbage colector, drop me an email and
we'll talk about it. I'm pretty interested in doing this. You don't
have to know a lot of perl internals, but you would have to understand
the c code in mod
roblem with the design I had in mind. There are ways around this I
can see in my brain, but they are far from eloquent. If something
were blocking on a network read it would stop the WHOLE perl engine...
TERRIBLE, not usefull at all for anyone that's going to be doing
something like that.
I
believe threaded perl is an attempt to allow multiple simultaneous
requests going into a single perl engine that is "multi threaded".
There are problems with this... and it's difficult to accomplish, and
alltogether a slower approach than queing because of the context
switching type overhead. Not to mention the I/O issue of this...
yikes! makes my head spin.
Thanks,
Shane.
>
> Les Mikesell
>[EMAIL PROTECTED]
volved in that... actually that has a lot of appeal to
me. Hmm... I guess it's time to pick apache 2.0 stuff and do some
tinkering! :) As far as the present problem... I'm not all that
concerned about it. It actually falls outside of the area of my
responsibilities at our site..., I'm thinking for the other people in
the community mostly.
Thanks!
Shane
>
> - Perrin
>
logical that something
like this would have been developed, but possibly not. The seperation
of the two components seems like it should be done, but there must be
a reason why no one has done it yet... I'm afraid this reason would be
the apache module API doesn't lend itself to this.
Well, thanks to everyone in advance for their thoughts/comments...
Shane Nay.
als to a
hash that will denote whether a lexical has been modded in the course
of execution. That hash will be checked after running the program and
delete all the appropriate SV's, AV's, HV's, etc.
Just wanted to get your opinion since you've all been down this road,
and suggest something that may be cool, or have zero merit :).
Shane.
78 matches
Mail list logo
