Re: Why people not using mod_perl
On Thu, 2009-09-17 at 23:12 +0200, Torsten Foertsch wrote: On Thu 17 Sep 2009, Kiran Kumar wrote: There is also Padre (http://padre.perlide.org/) , You can write plugins and customize to your needs, there are already lots of plugins available http://search.cpan.org/search?query=padre%3A%3Apluginmode=all I have seen padre first time at the this year German perl workshop in February and tried it out a bit. What I miss is syntax highlighting and indentation for C, XS and Perl in one tool. Can padre handle this? Last time I looked it could not but that was half a year ago. I'm surprised that nobody has mentioned EPIC, the Perl plugin for Eclipse. It works really really well, at least as well as the Java version (although it can't do as much prediction as Java can because of the nature of static vs dynamic languages). Full subversion integration, bugzilla/trac/jira integration, regex debugger, good syntax highlighting, builtin Perl Tidy, Perl Critic etc etc and of course, you get support for other languages in the same application. There is a git plugin, but it is somewhat basic - it's the main reason I keep using subversion rather than git. I don't know how well it supports XS. If you've never tried it, I'd highly recommend it. Like all new environments, it takes a while to get used to, but it is worth making the effort. clint I am using Emacs for almost 20 years now but it lacks good XS support. Torsten
Re: Ways to scale a mod_perl site
On Wed, 16 Sep 2009, Igor Chudov wrote: On Wed, Sep 16, 2009 at 11:05 AM, Michael Peters mpet...@plusthree.comwrote: Reducing DB usage is more important than this. Also, before you go down that road you should look at adding a caching layer to your application (memcached is a popular choice). It is not going to be that helpful due to dynamic content. (which is my site's advantage). But this may be useful for other applications. That's a common misconception, I think. Even if a website is completely dynamic you can cache things. First misunderstanding: people think about caching a whole HTML page. In memcached you typically cache data structures. As an example I will take my portal software. It has a forum, blog, guestbook, it has a list of users who are online, the forum has a posts from the last 24 hours, and it has other things that are shown with every request (new private messages, notes for moderators about new forum threads, ...) Now, Should I fetch the online users from the database with every request? Should I fetch all the threads and authors of the last 24 hours whenever somebody requests that page, even if nothing has changed? First answer: the list of online users can be cached for, say 1 minute. Nobody will care or even notice. Only if someone logs in you will expire the entry in memcached. All other changes are not so important that you cannot cache them for one single minute. Make that 1 page view per second and you save 59 database requests per minute. Second: The list of the recent posts can be cached, let's say for 3 minutes. The entry in memcached is only expired explicitly when somebody posts a new thread/reply or a title is changed etc. I believe *every* application has things like that which you can cache. I know of a website that reduced its load dramatically when using memcached. It's quite a big webseite (300 million Page views per month locally (mostly requests from german speaking countries)). But some people are reluctant to use memcached. One person said to me what if storing the data in memcached is more work than fetching it from the database every time? I don't know what to say. Try it out. I know the example of the company who uses memcached. Did you know faceboook uses memcached very very extensively? If you're not sure: analyse your website usage. What kind of data is fetched how often. Make a testcase and use memcached for that and see what's faster. Another thing you could do is: seperate your database schema. Some tables do not connect to others. For example, my portal software is modular, so that you can activate/deactive certain modules. The easiest thing was to just create one (DBIx::Class) schema per module. Of course, what connects all these is the user schema, and because I cannot do joins to the user tables any more I might have a request more here and there. But I can sepereate all these schemas and put them all on their own database server. With every request, you only need a part of all the schemas. Typically the highest load is on the database so splitting the db to several servers like this might be an option. And last but not least: for searching the database, use a search engine. KinoSearch works quite well, and there are also other search engines for perl. regards, tina
Re: Ways to scale a mod_perl site
3) Being enabled by item 2, add more webservers and balancers 4) Create a separate database for cookie data (Apache::Session objects) ??? -- not sure if good idea -- I've never seen the need to do that. In fact, I would suggest you drop sessions altogether if you can. If you need any per-session information then put it in a cookie. If you need this information to be tamper-proof then you can create a hash of the cookie's data that you store as part of the cookie. If you can reduce the # of times that each request needs to actually hit the database you'll have big wins. Can I get you to explain this a little more? I don't see how this could be used for truly secure sites because I don't quite understand how storing a hash in a plain text cookie would be secure. The thing I hate most about my secure applications is the fact that I have to read the DB at the start of every request to ensure that the session cookie is valid and to extract information about the user from the session table using the session ID stored in the cookie. Hitting the DB is the quickest way to kill performance and scalability in my experience.I know a lot of true app servers (Websphere, etc..) store this data in cached memory, but I was unaware that there might be an option for doing this without using a DB with mod_perl .
Re: Ways to scale a mod_perl site
-Original Message- From: Brad Van Sickle bvansick...@gmail.com Sent: Sep 17, 2009 12:13 AM To: Michael Peters mpet...@plusthree.com Cc: Mod_Perl modperl@perl.apache.org Subject: Re: Ways to scale a mod_perl site but I was unaware that there might be an option for doing this without using a DB with mod_perl . As Tina said, how about using memcached for this case? Regards, Jeff Peng
Re: Ways to scale a mod_perl site
Michael, you inspired me to reimplement cookies this way. For my site, the
cookie table is the most frequently updated one (even though I do not grant
cookies to search engines). I will try to use this kind of implementation.
Even now, my users like the fact that they can stay signed on forever, but
now I can do it at no cost to myself.
A quick question, is there an existing perl module to do this sort of thing?
Igor
On Wed, Sep 16, 2009 at 12:11 PM, Michael Peters mpet...@plusthree.comwrote:
On 09/16/2009 12:13 PM, Brad Van Sickle wrote:
Can I get you to explain this a little more? I don't see how this could
be used for truly secure sites because I don't quite understand how
storing a hash in a plain text cookie would be secure.
If you need to store per-session data about a client that the client
shouldn't be able to see, then you just encrypt that data, base-64 encode it
and then put it into a cookie.
If you don't care if the user sees that information you just want to make
sure that they don't change it then add an extra secure hash of that
information to the cookie itself and then verify it when you receive it.
I like to use JSON for my cookie data because it's simple and fast, but any
serializer should work. Something like this:
use JSON qw(to_json from_json);
use Digest::MD5 qw(md5_hex);
use MIME::Base64::URLSafe qw(urlsafe_b64encode urlsafe_b64decode);
# to generate the cookie
my %data = ( foo = 1, bar = 2, baz = 'frob' );
$data{secure} = generate_data_hash(\%data);
my $cookie = urlsafe_b64encode(to_json(\%data));
print Cookie: $cookie\n;
# to process/validate the cookie
my $new_data = from_json(urlsafe_b64decode($cookie));
my $new_hash = delete $new_data-{secure};
if( $new_hash eq generate_data_hash($new_data) ) {
print Cookie is ok!\n;
} else {
print Cookie has been tampered with! Ignore.\n;
}
# very simple hash generation function
sub generate_data_hash {
my $data = shift;
my $secret = 'some configured secret';
return md5_hex($secret . join('|', map { $_ - $data-{$_} } keys
%$data));
}
Doing encryption and encoding on small bits of data (like cookies) in
memory will almost always be faster than having to hit the database
(especially if it's on another machine). But the biggest reason is that it
takes the load off the DB and puts it on the web machines which are much
easier to scale linearly.
I know a lot of true app servers (Websphere, etc..) store
this data in cached memory,
You could do the same with your session data, or even store it in a shared
resource like a BDB file. But unless it's available to all of your web
servers you're stuck with sticky sessions and that's a real killer for
performance/scalability.
--
Michael Peters
Plus Three, LP
Re: Ways to scale a mod_perl site
This?
http://search.cpan.org/~jkrasnoo/ApacheCookieEncrypted-0.03/Encrypted.pm
Catalyst has a plugin:
http://search.cpan.org/~lbrocard/Catalyst-Plugin-CookiedSession-0.35/lib/Catalyst/Plugin/CookiedSession.pm
Thanks.
On Fri, Sep 18, 2009 at 9:06 PM, Igor Chudov ichu...@gmail.com wrote:
Michael, you inspired me to reimplement cookies this way. For my site, the
cookie table is the most frequently updated one (even though I do not grant
cookies to search engines). I will try to use this kind of implementation.
Even now, my users like the fact that they can stay signed on forever, but
now I can do it at no cost to myself.
A quick question, is there an existing perl module to do this sort of thing?
Igor
On Wed, Sep 16, 2009 at 12:11 PM, Michael Peters mpet...@plusthree.com
wrote:
On 09/16/2009 12:13 PM, Brad Van Sickle wrote:
Can I get you to explain this a little more? I don't see how this could
be used for truly secure sites because I don't quite understand how
storing a hash in a plain text cookie would be secure.
If you need to store per-session data about a client that the client
shouldn't be able to see, then you just encrypt that data, base-64 encode it
and then put it into a cookie.
If you don't care if the user sees that information you just want to make
sure that they don't change it then add an extra secure hash of that
information to the cookie itself and then verify it when you receive it.
I like to use JSON for my cookie data because it's simple and fast, but
any serializer should work. Something like this:
use JSON qw(to_json from_json);
use Digest::MD5 qw(md5_hex);
use MIME::Base64::URLSafe qw(urlsafe_b64encode urlsafe_b64decode);
# to generate the cookie
my %data = ( foo = 1, bar = 2, baz = 'frob' );
$data{secure} = generate_data_hash(\%data);
my $cookie = urlsafe_b64encode(to_json(\%data));
print Cookie: $cookie\n;
# to process/validate the cookie
my $new_data = from_json(urlsafe_b64decode($cookie));
my $new_hash = delete $new_data-{secure};
if( $new_hash eq generate_data_hash($new_data) ) {
print Cookie is ok!\n;
} else {
print Cookie has been tampered with! Ignore.\n;
}
# very simple hash generation function
sub generate_data_hash {
my $data = shift;
my $secret = 'some configured secret';
return md5_hex($secret . join('|', map { $_ - $data-{$_} } keys
%$data));
}
Doing encryption and encoding on small bits of data (like cookies) in
memory will almost always be faster than having to hit the database
(especially if it's on another machine). But the biggest reason is that it
takes the load off the DB and puts it on the web machines which are much
easier to scale linearly.
I know a lot of true app servers (Websphere, etc..) store
this data in cached memory,
You could do the same with your session data, or even store it in a shared
resource like a BDB file. But unless it's available to all of your web
servers you're stuck with sticky sessions and that's a real killer for
performance/scalability.
--
Michael Peters
Plus Three, LP
--
Fayland Lam // http://www.fayland.org/
Re: Ways to scale a mod_perl site
On Fri, Sep 18, 2009 at 8:12 AM, Fayland Lam fayl...@gmail.com wrote:
This?
http://search.cpan.org/~jkrasnoo/ApacheCookieEncrypted-0.03/Encrypted.pmhttp://search.cpan.org/%7Ejkrasnoo/ApacheCookieEncrypted-0.03/Encrypted.pm
Catalyst has a plugin:
http://search.cpan.org/~lbrocard/Catalyst-Plugin-CookiedSession-0.35/lib/Catalyst/Plugin/CookiedSession.pmhttp://search.cpan.org/%7Elbrocard/Catalyst-Plugin-CookiedSession-0.35/lib/Catalyst/Plugin/CookiedSession.pm
This module seems to want libapreq.1-34, which I interpret as not being
compatible with mod_perl 2?
I tried installing it with CPAN on Ubuntu Jaunty and failed.
CPAN.pm: Going to build I/IS/ISAAC/libapreq-1.34.tar.gz
Please install mod_perl: 1.25 version 1.99
(Can't locate mod_perl.pm in @INC (@INC contains: /root/misc/life/modules
/root/lisleelectric.com /etc/perl /usr/local/lib/perl/5.10.0
/usr/local/share/perl/5.10.0 /usr/lib/perl5 /usr/share/perl5
/usr/lib/perl/5.10 /usr/share/perl/5.10 /usr/local/lib/site_perl .) at
Makefile.PL line 7.
) at Makefile.PL line 8.
BEGIN failed--compilation aborted at Makefile.PL line 18.
Warning: No success on command[/usr/bin/perl Makefile.PL INSTALLDIRS=site]
Warning (usually harmless): 'YAML' not installed, will not store persistent
state
ISAAC/libapreq-1.34.tar.gz
/usr/bin/perl Makefile.PL INSTALLDIRS=site -- NOT OK
Running make test
Igor
Thanks.
On Fri, Sep 18, 2009 at 9:06 PM, Igor Chudov ichu...@gmail.com wrote:
Michael, you inspired me to reimplement cookies this way. For my site,
the
cookie table is the most frequently updated one (even though I do not
grant
cookies to search engines). I will try to use this kind of
implementation.
Even now, my users like the fact that they can stay signed on forever,
but
now I can do it at no cost to myself.
A quick question, is there an existing perl module to do this sort of
thing?
Igor
On Wed, Sep 16, 2009 at 12:11 PM, Michael Peters mpet...@plusthree.com
wrote:
On 09/16/2009 12:13 PM, Brad Van Sickle wrote:
Can I get you to explain this a little more? I don't see how this could
be used for truly secure sites because I don't quite understand how
storing a hash in a plain text cookie would be secure.
If you need to store per-session data about a client that the client
shouldn't be able to see, then you just encrypt that data, base-64
encode it
and then put it into a cookie.
If you don't care if the user sees that information you just want to
make
sure that they don't change it then add an extra secure hash of that
information to the cookie itself and then verify it when you receive it.
I like to use JSON for my cookie data because it's simple and fast, but
any serializer should work. Something like this:
use JSON qw(to_json from_json);
use Digest::MD5 qw(md5_hex);
use MIME::Base64::URLSafe qw(urlsafe_b64encode urlsafe_b64decode);
# to generate the cookie
my %data = ( foo = 1, bar = 2, baz = 'frob' );
$data{secure} = generate_data_hash(\%data);
my $cookie = urlsafe_b64encode(to_json(\%data));
print Cookie: $cookie\n;
# to process/validate the cookie
my $new_data = from_json(urlsafe_b64decode($cookie));
my $new_hash = delete $new_data-{secure};
if( $new_hash eq generate_data_hash($new_data) ) {
print Cookie is ok!\n;
} else {
print Cookie has been tampered with! Ignore.\n;
}
# very simple hash generation function
sub generate_data_hash {
my $data = shift;
my $secret = 'some configured secret';
return md5_hex($secret . join('|', map { $_ - $data-{$_} } keys
%$data));
}
Doing encryption and encoding on small bits of data (like cookies) in
memory will almost always be faster than having to hit the database
(especially if it's on another machine). But the biggest reason is that
it
takes the load off the DB and puts it on the web machines which are much
easier to scale linearly.
I know a lot of true app servers (Websphere, etc..) store
this data in cached memory,
You could do the same with your session data, or even store it in a
shared
resource like a BDB file. But unless it's available to all of your web
servers you're stuck with sticky sessions and that's a real killer for
performance/scalability.
--
Michael Peters
Plus Three, LP
--
Fayland Lam // http://www.fayland.org/
Re: Updating cookies in header during request processing
On 09/18/2009 09:14 AM, Igor Chudov wrote: I realized that there is something I am missing. Sometimes I may need to put something into the session after I did $cgi-start_html. I can do it if the cookie is only a session ID, with session data stored in mysql. This might be a larger architectural problem them. IMO, you really shouldn't use CGI.pm for HTML generation. Most people use a template of some sort for that. And the next thing would be to reorganize your code so that you aren't tied to a specific order of doing things. For instance, I normally use CGI::Application which lets me do all kinds of things to the headers and content until I'm all ready to return at the end of my run mode. The order doesn't matter. But how can I change the cookie AFTER I called $cgi-start_html? Since cookies are part of the HTTP headers they need to go out before any HTML content. So instead of just printing the content as you go, why don't you collect your HTML into a variable and then print it all out at the end. -- Michael Peters Plus Three, LP
Re: Updating cookies in header during request processing
On Fri 18 Sep 2009, Igor Chudov wrote: But how can I change the cookie AFTER I called $cgi-start_html? As long as there is nothing sent out on the wire you can add headers. When the response header is gone you can't. start_html() only generates a string as far as know. So it depends upon when this string is sent. But with mod_perl you can circumvent even this limitation. Write a caching output filter. Torsten -- Need professional mod_perl support? Just hire me: torsten.foert...@gmx.net
Re: Ways to scale a mod_perl site
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 unsubscribe Jeff Peng wrote: -Original Message- From: Brad Van Sickle bvansick...@gmail.com Sent: Sep 17, 2009 12:13 AM To: Michael Peters mpet...@plusthree.com Cc: Mod_Perl modperl@perl.apache.org Subject: Re: Ways to scale a mod_perl site but I was unaware that there might be an option for doing this without using a DB with mod_perl . As Tina said, how about using memcached for this case? Regards, Jeff Peng - -- David Avery Front Gate Solutions 1711 South Congress Austin, TX 78704 Ph: 512-674-9364 -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkqzlKsACgkQ7SsBcHOnG7JsUwCfVTesb2CKmK2QtgBa5ZU9waTW XIQAoK0kbL1rlBBnXQ6rHl3bOHWf04yI =k0ZL -END PGP SIGNATURE-
Updating cookies in header during request processing
I was very excited by the suggestion to use cookies to store the entire session information, and to keep it safe by means of base64 encoding and MD5 hash with a secret salt, for storing session information securely on the client. I realized that there is something I am missing. Sometimes I may need to put something into the session after I did $cgi-start_html. I can do it if the cookie is only a session ID, with session data stored in mysql. But how can I change the cookie AFTER I called $cgi-start_html? If I can do that, I think that I am set to go. thanks Igor
Re: Updating cookies in header during request processing
Igor == Igor Chudov ichu...@gmail.com writes: Igor I was very excited by the suggestion to use cookies to store the entire Igor session information, and to keep it safe by means of base64 encoding and Igor MD5 hash with a secret salt, for storing session information securely on Igor the client. Ahh, phase 2 of cookie awareness. When you get to phase 3, you realize that cookies should really just be used to distinguish one browser from another, and hold everything server-side instead for far better security and flexibility. (Remember, server-side could be something as simple as DBM::Deep, which is a single-file zero-install module that gives you arbitrary persistent Perl data structures efficiently.) See my (slightly aged but still valid) write-up of this at: http://www.stonehenge.com/merlyn/WebTechniques/col61.html -- Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095 mer...@stonehenge.com URL:http://www.stonehenge.com/merlyn/ Smalltalk/Perl/Unix consulting, Technical writing, Comedy, etc. etc. See http://methodsandmessages.vox.com/ for Smalltalk and Seaside discussion
Re: Updating cookies in header during request processing
On 09/18/2009 10:33 AM, Randal L. Schwartz wrote: Ahh, phase 2 of cookie awareness. When you get to phase 3, you realize that cookies should really just be used to distinguish one browser from another, and hold everything server-side instead for far better security and flexibility. I disagree. Using cookies for session data has a lot of advantages: + No need to have a permanent data store (DBD::Deep is single server only and why waste the resources to go across the network to your DB or cache if you don't have to). Also no need to clean up this data store periodically. Having a single source for this data also kills scalability. + If it's commonly used data, putting it into a cookie will make it available to the client side Javascript. Why waste server resources to do what the client's machine can do. In fact, I find it's more flexible to have this information in the cookie since my front end folks can then use it to add functionality without having to trouble the back end folks. And securing a cookie is pretty easy. If the information is not sensitive then you just need to put a hash in it to make sure it's not tampered with. If it is sensitive, then encryption works for cookies too. I'm not saying there aren't uses for large server side sessions, but I think they are pretty few. -- Michael Peters Plus Three, LP
Re: Ways to scale a mod_perl site
On Wed, 16 Sep 2009, Michael Peters wrote: On 09/16/2009 12:13 PM, Brad Van Sickle wrote: Can I get you to explain this a little more? I don't see how this could be used for truly secure sites because I don't quite understand how storing a hash in a plain text cookie would be secure. If you need to store per-session data about a client that the client shouldn't be able to see, then you just encrypt that data, base-64 encode it and then put it into a cookie. How does the user invalidate that session? (in case the cookie leaked or something like that). Or how can the website owner log out a certain user? If I have a session cookie with data in the server database I can always invalidate that session by login out and thus removing the database entry. I personally prefer to have control over such things... Is one select per request that bad? if the website is completely dynamic you will probably have other requests as well? If you care about the number of selects you should IMHO better safe those with the help of caching.
Re: Updating cookies in header during request processing
Michael == Michael Peters mpet...@plusthree.com writes: Michael On 09/18/2009 10:33 AM, Randal L. Schwartz wrote: Ahh, phase 2 of cookie awareness. When you get to phase 3, you realize that cookies should really just be used to distinguish one browser from another, and hold everything server-side instead for far better security and flexibility. Michael I disagree. Using cookies for session data has a lot of advantages: [justifications snipped] Yes. Welcome to phase 2. Eventually, you'll enter phase 3. The smarter webdev people always do. I sounded exactly like you, once, and then grew out of it. The more you resist, the longer your delay. :) -- Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095 mer...@stonehenge.com URL:http://www.stonehenge.com/merlyn/ Smalltalk/Perl/Unix consulting, Technical writing, Comedy, etc. etc. See http://methodsandmessages.vox.com/ for Smalltalk and Seaside discussion
Re: Ways to scale a mod_perl site
On 09/18/2009 11:13 AM, Tina Mueller wrote: How does the user invalidate that session? (in case the cookie leaked or something like that). Or how can the website owner log out a certain user? When you generate the hash for the cookie, you can also include the timestamp and the IP address of the client. If the cookie leaks it can't be used (unless the person who steals it is also on the same NAT'd network and uses it quickly). But you'll have that same problem anyway. Is one select per request that bad? if the website is completely dynamic you will probably have other requests as well? One extra select on every request can add up. In most web architectures the DB is a scarce shared resource. If you care about the number of selects you should IMHO better safe those with the help of caching. Caching of sessions could help, but if you don't need to go down that road, why do it in the first place? -- Michael Peters Plus Three, LP
Re: Updating cookies in header during request processing
All due respect, but hat's a little condescending... I generally cringe when I hear anyone advocating that there is one right way to do things that should be used in every instance In addition to Michael's points (which are totally valid) I would add that your solution is great for small/medium sized sites but as soon as you scale into sites with very large amounts of traffic it quickly raises a lot of operational concerns about where to store that data in place that's retrievable in a short enough time frame to not degrade performance. Solving that problem is going to cost time and money and will sometimes result in your application caring about session affinity, which is another operational concern. I'm not saying that these problems aren't solvable, I'm simply saying that I don't think it's nearly as cut and dried as you seem to, especially when you look at the app from and operational perspective. I can see both sides of this argument and I can see situations in in which either solution might be advantageous over the other. A lot depends on budget, environmental layout, how often the session is updated, how much data you're storing, etc... Perhaps you could outline in a little more detail why you think storing everything server side is the only right way to do things? Randal L. Schwartz wrote: Michael == Michael Peters mpet...@plusthree.com writes: Michael On 09/18/2009 10:33 AM, Randal L. Schwartz wrote: Ahh, phase 2 of cookie awareness. When you get to phase 3, you realize that cookies should really just be used to distinguish one browser from another, and hold everything server-side instead for far better security and flexibility. Michael I disagree. Using cookies for session data has a lot of advantages: [justifications snipped] Yes. Welcome to phase 2. Eventually, you'll enter phase 3. The smarter webdev people always do. I sounded exactly like you, once, and then grew out of it. The more you resist, the longer your delay. :)
Re: Updating cookies in header during request processing
On 09/18/2009 11:15 AM, James Smith wrote: But cookies are in general not big enough to store the information that a user would store on a website! I'm not talking about eliminating a permanent data store for your users. I'm talking about replacing the session specific things. How much session specific data do you really need to store? If it's bigger than 4K per-user than yes you can't use a single cookie. But like I said before, the situations that you really need more than that for *session specific* data are pretty rare. and security is not just on your server (but also on the clients machine) so if the browser can read it - anyone that can compromise the browser can also read it - if it is on the server that is harder! It's almost as if people aren't reading my other emails :) If you need to prevent tampering, use a hash. If you need to completely secure the data, encrypt it. -- Michael Peters Plus Three, LP
Re: Updating cookies in header during request processing
On 09/18/2009 11:19 AM, Randal L. Schwartz wrote: Yes. Welcome to phase 2. Eventually, you'll enter phase 3. I used to be a phase 3 convert and then some really smart people convinced me otherwise :) -- Michael Peters Plus Three, LP
Re: Ways to scale a mod_perl site
Brad Van Sickle bvansick...@gmail.com writes: 3) Being enabled by item 2, add more webservers and balancers 4) Create a separate database for cookie data (Apache::Session objects) ??? -- not sure if good idea -- I've never seen the need to do that. In fact, I would suggest you drop sessions altogether if you can. If you need any per-session information then put it in a cookie. If you need this information to be tamper-proof then you can create a hash of the cookie's data that you store as part of the cookie. If you can reduce the # of times that each request needs to actually hit the database you'll have big wins. Can I get you to explain this a little more? I don't see how this could be used for truly secure sites because I don't quite understand how storing a hash in a plain text cookie would be secure. The general idea is that you store a cryptographic hash of the cookie information plus a secret only your app knows. Using | to show string contatenation, your cookie would be: YourCookieFields|HASH(YourCookieFields|YourSecret) An attacker can't create the right hash because they don't know your secret, and they can't change any fields in the cookie because the hash would become invalid. -Scott.
Re: Ways to scale a mod_perl site
Tina Mueller apa...@s05.tinita.de writes: On Wed, 16 Sep 2009, Michael Peters wrote: [...] If you need to store per-session data about a client that the client shouldn't be able to see, then you just encrypt that data, base-64 encode it and then put it into a cookie. How does the user invalidate that session? (in case the cookie leaked or something like that). Or how can the website owner log out a certain user? Right, that is the trade-off for improved performance and scalability. Different trade-offs will make sense for different sites. For most sites, the performance and scalability won't matter too much, but for some it will. Simple things like timestamping the cookie and expiring it after awhile can help some, but they will not get you the flexibility of keeping everything in a database. -Scott.
RE: Ways to scale a mod_perl site
It amounts to shared private key security.
Each web server, for instance, is configured with the key abcd1234
The session looks like
{ username = 'dog'
, group = 'canid'
, premium = 0
, login_time = 1253289574
}
I serialize that into a string with join '|', (map { $_, $session-{$_} } sort
keys %session;
$cookiebase = login_time|1253289574|group|canid|premium|0|username|dog
I apply md5_hex from the Digest::MD5 module
$signature = md5_hex($cookiebase . | . 'abcd1234');
Which yields
68b07c585c18282ea418937266b031d7
I then construct my cookie.
$cookie = join ':', %session, $signature;
So the cookie string looks like
premium:0:time:1253289574:username:dog:group:canid:68b07c585c18282ea418937266b031d7
When I receive the cookie on a request I just do the inverse.
my (%cookie, $signature) = split /:/, $cookie;
die 'BOGUS SESSION' unless ($signature eq md5_hex(join '|', (map { $_,
$session-{$_} } sort keys %cookie), 'abcd1234';
If you change the 'plaintext' string in any way - the md5_hex will change. If
you change or drop the signature, the md5_hex will change.
Its security through obscurity admittedly - security in that you can't see my
code, methodology, or shared secret configuration.
But most people consider that plenty secure for securing the session data.
David
-Original Message-
From: Brad Van Sickle [mailto:bvansick...@gmail.com]
Sent: Wednesday, September 16, 2009 9:13 AM
To: Michael Peters
Cc: Mod_Perl
Subject: Re: Ways to scale a mod_perl site
3) Being enabled by item 2, add more webservers and balancers
4) Create a separate database for cookie data (Apache::Session objects)
??? -- not sure if good idea --
I've never seen the need to do that. In fact, I would suggest you drop
sessions altogether if you can. If you need any per-session
information then put it in a cookie. If you need this information to
be tamper-proof then you can create a hash of the cookie's data that
you store as part of the cookie. If you can reduce the # of times that
each request needs to actually hit the database you'll have big wins.
Can I get you to explain this a little more? I don't see how this could
be used for truly secure sites because I don't quite understand how
storing a hash in a plain text cookie would be secure.
The thing I hate most about my secure applications is the fact that I
have to read the DB at the start of every request to ensure that the
session cookie is valid and to extract information about the user from
the session table using the session ID stored in the cookie. Hitting
the DB is the quickest way to kill performance and scalability in my
experience.I know a lot of true app servers (Websphere, etc..)
store this data in cached memory, but I was unaware that there might be
an option for doing this without using a DB with mod_perl .
Re: Odd error from subprogram in 'required'
On Sep 17, 2009, at 11:09 PM, Tuomo Salo wrote: Heck, every CGI perl script on our server uses the same code, which is defined in the 'appname-defaults' file for each application. I am going to hazard a guess that the other scripts run under mod_perl (as opposed to CGI) too. If not, then you might want to check http://perl.apache.org/docs/1.0/guide/porting.html for help in porting the scripts. When I looked more closely at the errors, they seemed to have been generated by a MSN searchbot, how I don't know. All of them were from IP addresses in the range 65.55.106.nnn which resolve to: msnbot-65-55-106-nnn.search.msn.com. It was complete happenstance that when I tested it, the msnbot hit it; I was focusing more on the mod_perl error than looking for the original error. We have tracked down most, if not all of the bugs we had when we went to the mod_perl system; I've got a well-thumbed copy of the porting guide on my desk at this moment :-) -- Bruce Johnson University of Arizona College of Pharmacy Information Technology Group Institutions do not have opinions, merely customs
Re: Updating cookies in header during request processing
Hi Randal, nice to see you. Your suggestion is where I am coming FROM: right now the cookie is only a key into the mysql table which holds session data. What I want is to stop using that table altogether and let the browser hold the information, in a manner that is straightforward, flexible and secure. Igor On Fri, Sep 18, 2009 at 9:33 AM, Randal L. Schwartz mer...@stonehenge.comwrote: Igor == Igor Chudov ichu...@gmail.com writes: Igor I was very excited by the suggestion to use cookies to store the entire Igor session information, and to keep it safe by means of base64 encoding and Igor MD5 hash with a secret salt, for storing session information securely on Igor the client. Ahh, phase 2 of cookie awareness. When you get to phase 3, you realize that cookies should really just be used to distinguish one browser from another, and hold everything server-side instead for far better security and flexibility. (Remember, server-side could be something as simple as DBM::Deep, which is a single-file zero-install module that gives you arbitrary persistent Perl data structures efficiently.) See my (slightly aged but still valid) write-up of this at: http://www.stonehenge.com/merlyn/WebTechniques/col61.html -- Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095 mer...@stonehenge.com URL:http://www.stonehenge.com/merlyn/ Smalltalk/Perl/Unix consulting, Technical writing, Comedy, etc. etc. See http://methodsandmessages.vox.com/ for Smalltalk and Seaside discussion
Re: Ways to scale a mod_perl site
On 09/18/2009 12:16 PM, Ihnen, David wrote: Its security through obscurity admittedly - security in that you can't see my code, methodology, or shared secret configuration. No it's not really through obscurity. Even if someone found out your method of serialization your data is still safe. It's only if they find out your secret key that you'll have problems. But that's the same for SSL, PGP and any other crypto. -- Michael Peters Plus Three, LP
Re: Updating cookies in header during request processing
On Fri, Sep 18, 2009 at 10:51 AM, Michael Peters mpet...@plusthree.comwrote: On 09/18/2009 11:15 AM, James Smith wrote: But cookies are in general not big enough to store the information that a user would store on a website! I'm not talking about eliminating a permanent data store for your users. I'm talking about replacing the session specific things. How much session specific data do you really need to store? If it's bigger than 4K per-user than yes you can't use a single cookie. But like I said before, the situations that you really need more than that for *session specific* data are pretty rare. In my case, in almost all instances, the only thing I would want to store is authenticated userid. But I cannot exclude other possibilities. Right now I use Apache::Session to store a hash in a mysql table sessions. I have a users table that holds their profile information such as their email, their picture, etc. If I can find a way to set that cookie after the request is processed, I would be very happy. I can _probably_ work around it and set cookie before doing start_html, but I would prefer not to. Regarding another comment on using templates, etc: I have basically a homegrown CMS on algebra.com. And I do support templates: the presentation is mostly decoupled from content, such as where ads are places, how navigation aids are presented etc, is all decoupled. I can work on the new template for a month and the users will not notice as they will see the old template. But I like what I have as it is my own. Igor
Re: Updating cookies in header during request processing
Igor == Igor Chudov ichu...@gmail.com writes: Igor In my case, in almost all instances, the only thing I would want to Igor store is authenticated userid. The problem with that is public web browsers. You *cannot* ensure the expiration of an auth cookie, so you'll have to have some sort of server-side data to say this user most recently authenticated at this time, so I still trust him. And once you've done that, why store *any* auth client side? Just brand the browser, as my article says. -- Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095 mer...@stonehenge.com URL:http://www.stonehenge.com/merlyn/ Smalltalk/Perl/Unix consulting, Technical writing, Comedy, etc. etc. See http://methodsandmessages.vox.com/ for Smalltalk and Seaside discussion
Re: Ways to scale a mod_perl site
On Fri, Sep 18, 2009 at 10:13 AM, Tina Mueller apa...@s05.tinita.de wrote: On Wed, 16 Sep 2009, Michael Peters wrote: On 09/16/2009 12:13 PM, Brad Van Sickle wrote: Can I get you to explain this a little more? I don't see how this could be used for truly secure sites because I don't quite understand how storing a hash in a plain text cookie would be secure. If you need to store per-session data about a client that the client shouldn't be able to see, then you just encrypt that data, base-64 encode it and then put it into a cookie. How does the user invalidate that session? (in case the cookie leaked or something like that). Or how can the website owner log out a certain user? Same way you do with a table: when the user logs out, you update their cookie to a new one, where userid is not set. If I have a session cookie with data in the server database I can always invalidate that session by login out and thus removing the database entry. I personally prefer to have control over such things... Is one select per request that bad? if the website is completely dynamic you will probably have other requests as well? Well, the cookie table is the one that gets hit a lot and grows out of control. It is hard to scale and replicate. Storing cookies on the browsers solves this completely. I can have a billion browsers connect to my site and no database growth will occur from that. If you care about the number of selects you should IMHO better safe those with the help of caching.
Re: Updating cookies in header during request processing
On 09/18/2009 12:57 PM, Randal L. Schwartz wrote: The problem with that is public web browsers. You *cannot* ensure the expiration of an auth cookie, so you'll have to have some sort of server-side data to say this user most recently authenticated at this time, so I still trust him. Why does this have to be server side? Why can't it be part of the cookie's (tamper proof) data itself? -- Michael Peters Plus Three, LP
Re: Ways to scale a mod_perl site
On Fri, Sep 18, 2009 at 12:11 PM, James Smith j...@sanger.ac.uk wrote: Igor Chudov wrote: On Fri, Sep 18, 2009 at 10:13 AM, Tina Mueller apa...@s05.tinita.dewrote: On Wed, 16 Sep 2009, Michael Peters wrote: On 09/16/2009 12:13 PM, Brad Van Sickle wrote: Can I get you to explain this a little more? I don't see how this could be used for truly secure sites because I don't quite understand how storing a hash in a plain text cookie would be secure. If you need to store per-session data about a client that the client shouldn't be able to see, then you just encrypt that data, base-64 encode it and then put it into a cookie. How does the user invalidate that session? (in case the cookie leaked or something like that). Or how can the website owner log out a certain user? Same way you do with a table: when the user logs out, you update their cookie to a new one, where userid is not set. You missed the point in the previous email - that is when the System logs the user out... User X does something naughty - you need to ban him from doing anything else so you make his cookie invalid - this information can only occur on the server side so you delete the reference in the database refering to this session.. You are now having to check this each time which is the same as getting the information out of the database... I also think that everybody putting lots of stuff in their cookies is not thinking about network latency bandwidth etc - remember unless you are very careful how you specify your cookies you end up sending them on every request - for images/js/css/etc this adds to both bandwidth and CPU power - so it's a case of swings and roundabouts... balance your considerations You will usually find a fast right through cache is the best solution for most information on the backend... and being careful to only really create sessions when you have to! Thanks. I think that I understand the issue a little better. When I delete someone's account, I blow away everything, their users table entry, and all their content. So when they have a cookie with an invalid userid, they cannot do that much -- but I gotta admit that I need to think this through a little better. i
mod_perl2 per directory
Hey there, i've a strange problem... According to the mod_perl 2.0 server configuration pages, the following should work: Location /__ias/jobserver SetHandler perl-script PerlInterpStart 1 PerlInterpMax 1 PerlInterpMaxRequests 1000 PerlResponseHandler MyModule /Location paradoxically starting my apache2 gives me the following error message: PerlInterpStart not allowed here What's wrong with my setup? Am i missing something? Regards, Philip
Re: Ways to scale a mod_perl site
While many great minds are here, I would like to focus on one point for a moment, which in my experience, has been the most critical: The database Before I were to ask any other of your questions (all of which were valid), I would ask myself: - What kind of database tables am I implementing? (innodb, berkley,etc) What effect do they have on the filesystejm, or the pagefile? - How have I defined connections, connection pooling, shared resources, partitions v logical drives, semaphores v shared memory handles to handles? - Have I analyzed which tables actually get used, and by which processes, and paid attention to which operations only require simple foreign to primary key relationships, and not complex JOINS? And secondarily: - Is it possible to set up simple READ-ONLY copies of frequently read but rarely changed data (such as login information) so that some work could be off-loaded in an intelligent manner without regards to load balancing? In short, the database's interaction with the main application is most commonly the issue, regardless of the underlying technologies. Start there, so says I. Matthew P gedanken -- Matthew Paluch 404.375.8898
Re: Odd error from subprogram in 'required'
Bruce Johnson wrote:
I'm getting this error:
[Thu Sep 17 10:52:13 2009] [error] Undefined subroutine
ModPerl::ROOT::ModPerl::PerlRunPrefork::home_oraweb_perl_avi_login_2epl::PrintHeader
called at /home/oraweb/perl/avi/login.pl line 43, FH1 line 71.\n
[Thu Sep 17 10:52:13 2009] [error] [client 65.55.106.207] malformed
header from script. Bad header=_OF_FUNC: login.pl
[Thu Sep 17 10:52:13 2009] [warn] /avi/login.pl did not send an HTTP header
The undefined subroutine IS defined; it's loaded via a line:
require avi-defaults;
And the subroutine is just:
sub PrintHeader {
return Content-type: text/html\n\n;
}
The line as called in login.pl is just:
print PrintHeader;
Now this subroutine is called *throughout* the application, as you might
guess, every time we write a html page for display, the app is comprised
of about 40-50 different scripts.
Heck, every CGI perl script on our server uses the same code, which is
defined in the 'appname-defaults' file for each application.
I am going to hazard a guess that the other scripts run under mod_perl
(as opposed to CGI) too. If not, then you might want to check
http://perl.apache.org/docs/1.0/guide/porting.html for help in porting
the scripts.
It might not be a bad idea even otherwise, since it has a very thorough
explanation of a very similar looking scenario (scenario 1):
http://perl.apache.org/docs/1.0/guide/porting.html#Name_collisions_with_Modules_and_libs
Hope this helps.
-Tuomo
ModPerl::Registry and custom error documents
Hi all, I searched and googled around for a while but couldn't find a solution for the following question. We're using ModPerl::Registry (PerlOptions +ParseHeaders) in combination with mod_perl to serve our pages in a CGI like environment. This has the advantage for us to have only a very very small layer of integration between the scripts an the special runtime environment. Now we want to serve custom made error documents and thought naively to get it work by just setting the status to the error code and serving a html-page as it works with a normal status 200 page. But this doesn't work. It seams that apache jumps right into the error handling stuff as soon as a status code = 400 is seen. How can we dynamically create own error documents without using the lower level mod_perl/apache api? Is there a simple way? Best regards Andreas Mock
Re: Ways to scale a mod_perl site
On Sep 16, 2009, at 9:13, Brad Van Sickle wrote: I've never seen the need to do that. In fact, I would suggest you drop sessions altogether if you can. If you need any per-session information then put it in a cookie. If you need this information to be tamper-proof then you can create a hash of the cookie's data that you store as part of the cookie. If you can reduce the # of times that each request needs to actually hit the database you'll have big wins. Can I get you to explain this a little more? I don't see how this could be used for truly secure sites because I don't quite understand how storing a hash in a plain text cookie would be secure. If you are just concerned about the cookie being changed; add a time stamp and a hash to the cookie data. There's an example on page 19 of http://develooper.com/talks/rww-mysql-2008.pdf ... If you are concerned about the cookie being readable at all, you can encrypt the whole thing. Either way it's tamper proof. - ask -- http://develooper.com/ - http://askask.com/
Re: mod_perl2 per directory
On page http://perl.apache.org/docs/2.0/user/config/ config.html#mod_perl_Directives_Argument_Types_and_Allowed_Location you can see the following line in the table that defines the scope of the various directives: PerlInterpStart TAKE1 SRV SRV means server scope which to me means that it cannot be applied to a Location. On Sep 18, 2009, at 4:14 AM, Philip Blatter wrote: Hey there, i've a strange problem... According to the mod_perl 2.0 server configuration pages, the following should work: Location /__ias/jobserver SetHandler perl-script PerlInterpStart 1 PerlInterpMax 1 PerlInterpMaxRequests 1000 PerlResponseHandler MyModule /Location paradoxically starting my apache2 gives me the following error message: PerlInterpStart not allowed here What's wrong with my setup? Am i missing something? Regards, Philip
Re: Why people not using mod_perl
On Fri, Sep 18, 2009 at 1:34 AM, Clinton Gormley cl...@traveljury.com wrote: I'm surprised that nobody has mentioned EPIC, the Perl plugin for Eclipse. It works really really well, at least as well as the Java version (although it can't do as much prediction as Java can because of the nature of static vs dynamic languages). The problem is necessarily dynamic vs static, but rather a language with a BNF compared to a language without. By BNF I actually mean a set of rules by which to parse (BNF actually being the rules in a particular format). Much of most modern IDE capabilities are hindered by not being able to parse the code. I am using Emacs for almost 20 years now but it lacks good XS support. Emacs has worked pretty well for me so far. I tried other IDEs, and I found most of the to be clunky and too window based. I find I'm most productive when I use Emacs. -wjt
