Re: [ANNOUNCE] Cache::Cache 1.0

2002-04-15 Thread DeWitt Clinton

On Sat, Apr 13, 2002 at 04:33:11PM +0200, Eric Cholet wrote:

 I'm seeing this problem when using Apache::SOAP, using Cache::Cache
 0.99 or 1.0. Downgrading to Digest::MD5 2.12 indeed solves it.

Hi -- I recently released version 1.01 of Cache::Cache to CPAN.  This
version uses Digest::SHA1 as a substitute for MD5 until the bugs in
that library are fixed.  If anyone has any issues with 1.0 and MD5,
please download 1.01 and let me know if anything goes wrong.

Cheers,

-DeWitt







[ANNOUNCE] Cache::Cache 1.0

2002-04-07 Thread DeWitt Clinton

Hi,

The following is an announcement of Cache::Cache 1.0.  Please read the
summary below for more details.

I want to ask a special favor of the modperl community (who tend to
get a lot of mileage out of this code) -- I'm concerned with the use
of Digest::MD5, particularly version 2.16, which is used by the code
to generate unique filenames in the file-based cache.  I've been
seeing some pretty ugly bugs with the the MD5 code, and this has been
confirmed on a few mailing lists and CPAN bug reports.  Please let me
know if you notice any weird behavior with your system after upgrading
to version 1.0.  (All unit test pass consistently, but under certain
conditions, such as when XML::DOM is loaded, I think that Digest::MD5
may always return the same hash, no matter what the key.)

Note that you can simply run Digest::MD5 2.12 and not 2.16 if you have
any issues in production.  So don't worry about upgrading to version
1.0 of Cache::Cache.

I'm thinking of switching to Digest::SHA1 in version 1.01 to avoid
this until a new version of Digest::MD5 is released.  Does anyone know
of a reason to avoid that?  (SHA1 is slightly slower, but very little
time in spent generating keys anyway, so it may not be a big deal.)

For more information on the MD5 issue, please see:

  http://rt.cpan.org/NoAuth/Bug.html?id=362.

Thanks to everyone for your help.  Cheers,

-DeWitt


Summary:

The Cache modules are designed to assist a developer in persisting
data for a specified period of time. Often these modules are used
in web applications to store data locally to save repeated and
redundant expensive calls to remote machines or databases. People
have also been known to use Cache::Cache for its straightforward
interface in sharing data between runs of an application or
invocations of a CGI-style script or simply as an easy to use
abstraction of the filesystem or shared memory.

Release Notes:

  Release of Cache::Cache version 1.0.  This version fixes outstanding
  issues on Win32 platforms and other minor bug fixes.
 
Project Homepage:

  http://perl-cache.sourceforge.net/

Tar/GZ:

  http://prdownloads.sourceforge.net/perl-cache/Cache-Cache-1.0.tar.gz
  http://www.cpan.org/authors/id/D/DC/DCLINTON/Cache-Cache-1.0.tar.gz
  
Changelog:

  http://sourceforge.net/project/shownotes.php?release_id=83292

CVS tree (viewcvs):

  http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/perl-cache/

The following is the Cache-Cache-1.0 README file:

Copyright (C) 2001 DeWitt Clinton  All Rights Reserved

   You may distribute under the terms of either the GNU General Public
   License or the Artistic License, as specified in the Perl README file.


NAME

  Cache::Cache


DESCRIPTION

  The Cache modules are designed to assist a developer in persisting
  data for a specified period of time.  Often these modules are used
  in web applications to store data locally to save repeated and
  redundant expensive calls to remote machines or databases.  People
  have also been known to use Cache::Cache for its straightforward
  interface in sharing data between runs of an application or
  invocations of a CGI-style script or simply as an easy to use
  abstraction of the filesystem or shared memory.

  The Cache package provides Cache::Cache, a generic interface
  for creating persistent data stores.  This interface is implemented
  by the Cache::MemoryCache, Cache::SharedMemoryCache, Cache::FileCache, 
  Cache::SizeAwareFileCache, Cache::SizeAwareMemoryCache, and 
  Cache::SizeAwareSharedMemoryCache classes. 

  This work aggregates and extends the obsolete File::Cache and
  IPC::Cache projects.


REQUIREMENTS

  Digest::MD5
  Error
  File::Spec
  File::Path
  IPC::ShareLite
  Storable


INSTALLATION

  perl Makefile.PL
  make
  make test
  make install


USAGE

  First, choose the best type of cache implementation for your needs.
  The simplest cache is the MemoryCache, which is suitable for
  applications that are serving multiple sequential requests, and
  wish to avoid making redundant expensive queries, such as an
  Apache/mod_perl application talking to a database.  If you wish to
  share that data between processes, then perhaps the
  SharedMemoryCache is appropriate, although its behavior is tightly
  bound to the underlying IPC mechanism, which varies from system to
  system, and is unsuitable for large objects or large numbers of
  objects.  When the SharedMemoryCache is not acceptable, then
  FileCache offers all of the same functionality with similar
  performance metrics, and it is not limited in terms of the number of
  objects or their size.  If you wish to maintain a strict limit on
  the size of a file system based cache, then the SizeAwareFileCache
  is the way to go.  Similarly, the SizeAwareMemoryCache and the
  SizeAwareSharedMemoryCache add size management functionality
  to the MemoryCache and SharedMemoryCache classes respectively.

  Using a cache is simple.  Here is some sample code for instantiating
  and using a file system

Re: [ANNOUNCE] Cache::Cache 1.0

2002-04-07 Thread DeWitt Clinton

On Sun, Apr 07, 2002 at 08:52:59PM +0300, Issac Goldstand wrote:

 So how exactly is the auto_purge meant to be used - the
 documentation is very hazy on that point (sure I can patch it - but
 you have to explain it to me first :)), and I just got lost when
 looknig at the module source.

The auto_purge functionality is designed to enable a cache to delete
expired cache entries periodically.  While user code could call the
purge( ) method every time they access the cache, it would be rather
inefficient and often completely unnecessary.  Thus the auto_purge
code automatically calls purge( ) after a user specified interval of
time has elapsed.

You set the amount of time between auto_purges by setting the
'auto_purge_interval' option when instantiating the cache.

After setting the 'auto_purge_interval' the purge method will be
called when a cache instance is created (via 'new') and the time
interval has elapsed.

Also, by setting 'auto_purge_on_set' to true, the cache will
automatically call purge when the 'set' method is called and the time
interval has elapsed.

Respectively, by setting 'auto_purge_on_get' to true, the cache will
automatically call purge when the 'get' method is called and the time
interval has elapsed.

If you tend to instantiate caches relatively frequently (such as
during the initialization of a modperl process), you can probably get
away with simply setting the 'auto_purge_interval'.  However, if you
processes run for a long time and you'd like to automatically call
purge( ) more frequently, you can could consider using the
'auto_purge_on_set' option.  I'd be surprised if people were using the
'auto_purge_on_get' functionality, as it would most likely be
overkill. 

Also, keep in mind that if you hit the same keys over and over again,
then it may be sufficient to rely on 'get' to delete expired entries
when they are requested.

Hope this helps clarify things!  Cheers,

-DeWitt



Re: transient object data

2001-12-22 Thread DeWitt Clinton

On Sat, Dec 22, 2001 at 06:11:33AM -0800, brian moseley wrote:

 doesn't it seem like there should be a way to denote object
 data as transient so that it doesn't get serialized by
 Storable, etc?

I'd love that as well.  For example, when persisting Cache::Object
instances I manually strip out transient (in this case, it is
re-creatable) data such as the name of the key and the size of the
object, in order to save space in storage.

Sounds like Perl 6 needs a generic notion of Serializable.  

-DeWitt





Re: [ANNOUNCE] Apache::Singleton 0.03

2001-12-22 Thread DeWitt Clinton

On Sat, Dec 22, 2001 at 10:39:03PM +0900, Tatsuhiko Miyagawa wrote:

 One instance for one server (across all httpd processes).
 Implemented using Cache::SharedMemoryCache (IPC).

Maybe you want to consider directly using Cache::SharedMemoryBackend
instead of the SharedMemoryCache class.  The full cache class has the
overhead of dealing with object expiration times, which isn't going to
be necessary for the Singleton.

Also, try experimenting with the FileBackend as your sharing mechanism
instead of shared memory.  If you have a large number of Singleton
objects (unlikely, but possible) then this may be a huge performance
win.

Cheers,

-DeWitt



Re: Comparison of different caching schemes

2001-12-12 Thread DeWitt Clinton

On Wed, Dec 12, 2001 at 03:05:33PM +1100, Rob Mueller (fastmail) wrote:

 I sat down the other day and wrote a test script to try out various
 caching implementations. The script is pretty basic at the moment, I
 just wanted to get an idea of the performance of different methods.

Rob, wow!  This is fantastic work!

I'm wondering if you could include a test on the Cache::FileBackend
class.  The Cache::FileCache class is not optimized for the particular
type of gets and sets that you were testing for.  In fact, each of
those requests will still need to go through the process of checking
for object expiration, which may add the overhead that you observe in
the benchmark.

Not that I'm expecting Cache::FileBackend to do significantly better
-- it still does name hashing, directory hashing, taint checking,
cloning of objects, etc.  But I am curious to see what difference it
makes.  You could try Cache::SharedMemoryBackend as well.

In general the Cache::* modules were designed for clarity and ease of
use in mind.  For example, the modules tend to require absolutely no
set-up work on the end user's part and try to be as fail-safe as
possible.  Thus there is run-time overhead involved.  That said, I'm
certainly not against performance.  :) These benchmarks are going to
be tremendously useful in identifying bottlenecks.  However, I won't
be able to optimize for these particular benchmarks, as Cache::Cache
is designed to do something different than straight gets and sets.

Again, thank you, Rob.  This is great,

-DeWitt




Re: [RFC] Apache::CacheContent - Caching PerlFixupHandler

2001-12-11 Thread DeWitt Clinton

On Tue, Dec 11, 2001 at 02:31:36AM -0800, Paul Lindner wrote:

 Right.  A more elaborate Apache::CacheContent would have a filename
 hash function, and a separate cache directory structure along the
 lines of Cache::FileCache.

Just curious -- any reason not to use Cache::Cache as the persistance
mechanism?  It was designed for exactly this scenario and could
provide a nice abstraction for the filesystem or shared memory, as
well as handle things like filename hashing and branching directories
(and namespaces, size limits, OS independance, taint checking, and
more).

-DeWitt




[ANNOUNCE] Cache::Cache 0.99

2001-12-09 Thread DeWitt Clinton

Summary:

  The Cache modules are designed to assist a developer in persisting
  data for a specified period of time.  Often these modules are used
  in web applications to store data locally to save repeated and
  redundant expensive calls to remote machines or databases.  People
  have also been known to use Cache::Cache for its straightforward
  interface in sharing data between runs of an application or
  invocations of a CGI-style script or simply as an easy to use
  abstraction of the filesystem or shared memory.


Release Notes:

  This release contains newly refactored code, updated documentation,
  and numerous clarifications and minor bug fixes.


Project Homepage:

  http://sourceforge.net/projects/perl-cache/


Tar/GZ:

  http://prdownloads.sourceforge.net/perl-cache/Cache-Cache-0.99.tar.gz


Changelog:

  http://sourceforge.net/project/shownotes.php?release_id=64913


CVS tree (viewcvs):

  http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/perl-cache/



The following is the Cache-Cache-0.99 README file:


Copyright (C) 2001 DeWitt Clinton  All Rights Reserved

   You may distribute under the terms of either the GNU General Public
   License or the Artistic License, as specified in the Perl README file.


NAME

  Cache::Cache


DESCRIPTION

  The Cache modules are designed to assist a developer in persisting
  data for a specified period of time.  Often these modules are used
  in web applications to store data locally to save repeated and
  redundant expensive calls to remote machines or databases.  People
  have also been known to use Cache::Cache for its straightforward
  interface in sharing data between runs of an application or
  invocations of a CGI-style script or simply as an easy to use
  abstraction of the filesystem or shared memory.

  The Cache package provides Cache::Cache, a generic interface
  for creating persistent data stores.  This interface is implemented
  by the Cache::MemoryCache, Cache::SharedMemoryCache, Cache::FileCache, 
  Cache::SizeAwareFileCache, Cache::SizeAwareMemoryCache, and 
  Cache::SizeAwareSharedMemoryCache classes. 

  This work aggregates and extends the obsolete File::Cache and
  IPC::Cache projects.


REQUIREMENTS

  Digest::MD5
  Error
  File::Spec
  File::Path
  IPC::ShareLite
  Storable


INSTALLATION

  perl Makefile.PL
  make
  make test
  make install


USAGE

  First, choose the best type of cache implementation for your needs.
  The simplest cache is the MemoryCache, which is suitable for
  applications that are serving multiple sequential requests, and
  which to avoid making redundant expensive queries, such as an
  Apache/mod_perl application talking to a database.  If you wish to
  share that data between processes, then perhaps the
  SharedMemoryCache is appropriate, although its behavior is tightly
  bound to the underlying IPC mechanism, which varies from system to
  system, and is unsuitable for large objects or large numbers of
  objects.  When the SharedMemoryCache is not acceptable, then
  FileCache offers all of the same functionality with similar
  performance metrics, and it is not limited in terms of the number of
  objects or their size.  If you wish to maintain a strict limit on
  the size of a file system based cache, then the SizeAwareFileCache
  is the way to go.  Similarly, the SizeAwareMemoryCache and the
  SizeAwareSharedMemoryCache add size management functionality
  to the MemoryCache and SharedMemoryCache classes respectively.

  Using a cache is simple.  Here is some sample code for instantiating
  and using a file system based cache.

use Cache::FileCache;

my $cache = new Cache::FileCache( );

my $customer = $cache-get( $name );

if ( not defined $customer )
{
  $customer = get_customer_from_db( $name );
  $cache-set( $name, $customer, 10 minutes );
}

return $customer;

  Please refer to the perldoc for Cache::Cache and the related
  implementations for complete documentation.


INCOMPATIBLE CHANGES

  Cache::Cache 0.99 contains the following incompatible changes:

* Error::Simple is thrown on exceptions

* the get_identifiers method has been deprecated in favor of the
  get_keys method

* the internal format of object in a FileCache has been modified,
  necessitating a clearing of the cache while upgrading (make test 
  does this for the standard cache root)


SEE ALSO

  The project homepage at http://perl-cache.sourceforge.net


AUTHOR

  Original author: DeWitt Clinton [EMAIL PROTECTED]

  Copyright (C) 2001 DeWitt Clinton




Re: Doing Authorization using mod_perl from a programmers perspective

2001-11-19 Thread DeWitt Clinton

On Mon, Nov 19, 2001 at 07:51:55AM -0800, Randal L. Schwartz wrote:

 But this is obvious.  I'm confused about why I'd have to explain it. :(

I posted this a year or two back:

[EMAIL PROTECTED]">http://mathforum.org/epigone/modperl/jytwortwor/[EMAIL PROTECTED]


Here is the relevant part of that post:


It makes sense to start with the requirements for what it means to 
implement those secure features.  My requirements have an obvious 
e-commerce bias, and should probably be heavily reviewed by anyone 
thinking of using this design for online banking or government work. 

First, I'd like to introduce the notion of a secure session.  Secure
sessions have some subtle differences from the traditional notion of
the session, and I'll point those out when they appear.

The secure session has the following properties:

*) The user is able to initiate a secure session by providing proper
credentials (i.e., a username and password pair) via a login process.

*) The user is able to terminate the secure session via a logout
process.

*) Secure sessions must be able to time out automatically.

*) Secure sessions must *never* transmit sensitive data (such as the
password) over insecure channels.

*) The secure session, while it requires the use of a secure protocol
(such as HTTPS), should not require the use of cookies.  Cookies,
however, can be employed to extend the functionality of the system.  

Additionally, I feel that one of the essential requirements to any
enterprise quality, highly scalable, and fault-tolerant system is the
ability to store session state on a tier other than the front-end.
This usually means storing state in a shared database, but there are
other options.

A very effective method of meeting the requirements above is to use a
two token architecture.  The first token is a user identifier, which
can be passed over insecure channels.  This user identification token
can be used to restore state that isn't particularly sensitive, such
as a shopping cart or user preferences.  The second token is a secure
identifier, which is never passed over insecure channels.  The secure
identifier is required to access sensitive data, such as credit card
data. (It is possible to create an architecture that uses *only* the
secure token, but there are significant benefits in terms of
flexibility afforded by using two tokens, so I won't go into the one
token model here.)

The fundamental goal of the secure token is to a) keep it safe from
prying eyes, and b) use it is a requirement for accessing secure data.

Tokens can be passed in one of two ways.  First, the token can be
passed as part of the URL.  Second, the token can be passed in a
cookie.  I've found it very useful to create an initialization
procedure inside applications that check both places for these tokens,
and abstract the particular mechanism used to pass them.  Note,
however, that is important to remember how the token was passed -- if
it came from a cookie, it is cosmetic to not pass it around in the
URL.  I also recommend creating a library used in your templates that
has a function to build URLs that automatically append the tokens if
necessary.  Since it is critical that secure tokens are never passed
in the clear, it really helps to have that function, because it can
prevent the secure token from being sent via an insecure page in a URL
(i.e., only setting secure tokens in URLs that contain https).

The user identification token is typically used to restore state from
the database.  It is created whenever a page is viewed and the client 
has either not provided one (via the URL or a cookie), or the token
they provide doesn't exist in the database.  This token should then be 
set in both the URL and in a cookie.  On the subsequent page request,
if the user identification token is found in a cookie, it is probably
safe to stop putting it in the URL.  However, if the token is found
only in the URL (probably meaning the client does not use cookies) the 
server should not try writing the cookie again.  

The advantage of the cookie is such that the user can insecurely
identify themselves across browser sessions.  This is perfectly
acceptable when used to restore some insensitive state data (such as a
name or shopping cart).  However, the system still functions without
the cookie.  The user can now login to associate their client with
particular stored user information.

The login process is as follows:

1) The client connects to a secure page that presents a form
requesting a username and password.  (There has been much discussion
about whether a HTTP form that POSTS to a secure page will encrypt the 
posted data.  There is too much ambiguity here -- I recommend using
HTTPS on the login form as well.)  The ACTION of the form must also
be a secure page.

2) On receiving the username and password, the server compares an
encrypted version of the password with the stored encrypted password
associated with that customer.  If the passwords do not match, 

Re: Cache::* and MD5 collisions [was: [OT] Data store options]

2001-11-08 Thread DeWitt Clinton

On Thu, Nov 08, 2001 at 01:11:21PM -0500, Barrie Slaymaker wrote:

 Even a bit more OT: one thing to watch out for, especially if you
 plan on caching a *lot* of data, is that the Cache::* modules did
 not do collision detection on MD5 collisions the last time I looked.
 Forgive me if that's changed recently.

I knew that a collision was technically possible, but the odds seemed
astronomically low.  But if security is a concern, then I agree that
MD5-ing the key isn't the best way to create a unique identifier.  The
thing that I liked most about MD5 (other than its convenience) is that
it tends to distribute very evenly.  For example, file system caches
fill their directories roughly equally when their paths are created
from MD5 hashed keys.  Doing something simple and unique like
URL-encoding the key to make a legal identifier (legal in the sense
that it is a valid filename) wouldn't distribute as evenly.

Barrie raised this question privately to me earlier this year, and
this is what I wrote at the time:

   Good question.  They would overwrite each other.  You may want to
   check out this RFC for more information about the odds of that:

http://www.faqs.org/rfcs/rfc1321.html 
   Specifically, they say that it is conjectured that the difficulty
   of coming up with two messages having the same message digest is on
   the order of 2^64 operations, and that the difficulty of coming up
   with any message having a given message digest is on the order
   of 2^128 operations.

   This means you would need 18,446,744,073,709,551,616 keys to make
   it 100% likely that there would be a collision.

   Assuming you were only willing to risk a 0.01% chance that there is
   a collision, you could still fit 1,844,674,407,370,955 keys.  Since
   each value stored will take at least 1k in memory, that number of
   keys would require 1,844,674 Terrabytes of storage.
   
   In other words, it is highly unlikely that you could get a collision
   in real world usage.


Note that I'm not disagreeing with Barrie at all -- I just took the
easy way out and refuted the likelihood of an infinite number of
monkeys hitting my code.  :)

-DeWitt




Re: Using DOM to build your output documents

2001-10-03 Thread DeWitt Clinton

On Wed, Oct 03, 2001 at 04:43:41PM -0700, Jeffrey W. Baker wrote:

 I'd also like to hear people's opinions on XML::DOM (I think it
 stinks and I've replaced it with a C DOM implementation).

You just proved that you already know everything you need to on the
subject of XML::DOM.  :)

Seriously though, what C DOM implementation did you choose and how did
you integrate it with your Perl code?

I've long since abandoned XML::DOM and use things XML::Parser::PerlSAX
instead, but that doesn't achieve what you are talking about vis-a-vis
building up documents element by element.  For output generation, I
still tend to push data into a template engine such as Embperl or
Template::Toolkit and use embedded control structures to manipulate
that data.  Interestingly, the company I used to work at (eZiba) just
ported their front-end over to Mason and ended up somewhere in-between
-- i.e., reusable page components that they build up at run-time.  

Check out the Avacet's (www.avacet.com) TemplateService for an example
of the push model in action.

Cheers,

-DeWitt




Re: MSIISProbes.pm v1.03

2001-09-28 Thread DeWitt Clinton

On Fri, Sep 28, 2001 at 08:49:22AM -0700, Nick Tonkin wrote:

 Cache::FileCache defaults to using /tmp for the location of its
 cache; does the system have /tmp (not sure what Cache::FileCache does if
 there's no /tmp, hafta look at the code).

You can manually override the temp directory by setting the
'cache_root' option when instantiating the cache.  If cache_root isn't
set, then File::Spec's tmpdir( ) routine will be called, which seems
to return a value on just about all the machines I've tested (judging
by the lack of temp directory bug reports).

Cheers,

-DeWitt



[ANNOUNCE] Cache::Cache 0.09

2001-09-19 Thread DeWitt Clinton


[Note:  A big thanks to everyone on modperl that has provided
invaluable insight and support for this release.]

Summary:

  The Perl Cache package provides Cache::Cache, a generic interface
  for creating persistent data stores.  This interface is implemented
  by the Cache::MemoryCache, Cache::SharedMemoryCache, Cache::FileCache, 
  Cache::SizeAwareFileCache, Cache::SizeAwareMemoryCache, and 
  Cache::SizeAwareSharedMemoryCache classes.  This work replaces 
  File::Cache and IPC::Cache.


Release Notes:

  This release fixes outstanding bugs in cache instantiation and race
  condition handling, and is the first release under the dual
  GPL/Artistic license.


Project Homepage:

  http://sourceforge.net/projects/perl-cache/


Tar/GZ:

  http://www.cpan.org/modules/by-module/Cache/Cache-Cache-0.09.tar.gz



Changelog:

  http://sourceforge.net/project/shownotes.php?release_id=52255

  - applied Axel Beckert patch to fix the expiration units
  - applied Ken Williams's directory creation patch to pass all tests
  - changed the license to be either the GPL or the Artistic license
  - added Jay Sachs' implementation of NullCache
  - modified the remove methods to avoid croaking if two cache
instances are both purging or limiting at the same time
  - migrated to a factory-like model with private contructors
to fix the auto_purge_interval functionality
  - updated the documentation to better reflect that size means
size in bytes


CVS tree (viewcvs):

  http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/perl-cache/



The following is the Cache-Cache-0.09 README file:



Copyright (C) 2001 DeWitt Clinton  All Rights Reserved

   You may distribute under the terms of either the GNU General Public
   License or the Artistic License, as specified in the Perl README file.


NAME

  Cache::Cache


DESCRIPTION

  The Perl Cache package provides Cache::Cache, a generic interface
  for creating persistent data stores.  This interface is implemented
  by the Cache::MemoryCache, Cache::SharedMemoryCache, Cache::FileCache, 
  Cache::SizeAwareFileCache, Cache::SizeAwareMemoryCache, and 
  Cache::SizeAwareSharedMemoryCache classes.  This work replaces 
  File::Cache and IPC::Cache.


REQUIREMENTS

  Digest::MD5
  File::Spec
  File::Path
  IPC::ShareLite
  Storable


INSTALLATION

  perl Makefile.PL
  make
  make test
  make install


USAGE

  First, choose the best type of cache implementation for your needs.
  The simplest cache is the MemoryCache, which is suitable for
  applications that are serving multiple sequential requests, and
  which to avoid making redundant expensive queries, such as an
  Apache/mod_perl application talking to a database.  If you wish to
  share that data between processes, then perhaps the
  SharedMemoryCache is appropriate, although its behavior is tightly
  bound to the underlying IPC mechanism, which varies from system to
  system, and is unsuitable for large objects or large numbers of
  objects.  When the SharedMemoryCache is not acceptable, then
  FileCache offers all of the same functionality with similar
  performance metrics, and it is not limited in terms of the number of
  objects or their size.  If you wish to maintain a strict limit on
  the size of a file system based cache, then the SizeAwareFileCache
  is the way to go.  Similarly, the SizeAwareMemoryCache and the
  SizeAwareSharedMemoryCache add size management functionality
  to the MemoryCache and SharedMemoryCache classes respectively.

 
  Using a cache is simple.  Here is some sample code for instantiating
  and using a MemoryCache:

use Cache::Cache qw( $EXPIRES_NEVER $EXPIRES_NOW );
use Cache::MemoryCache;

my $options_hash_ref = { 'default_expires_in' = '10 seconds' };

my $cache = new Cache::MemoryCache( $options_hash_ref );

my $expires_in = '10 minutes';

$cache-set( 'Key', 'Value', $expires_in );

# if the next line is called within 10 minutes, then this 
# will return the cache value

my $value = $cache-get( 'Key' );


  Please refer to the perldoc for Cache::Cache and the related
  implementations for complete documentation.


SEE ALSO

  File::Cache and IPC::Cache, and the project homepage at:

http://sourceforge.net/projects/perl-cache/


AUTHOR

  Original author: DeWitt Clinton [EMAIL PROTECTED]

  Last author: $Author: dclinton $

  Copyright (C) 2001 DeWitt Clinton





[PRE-ANNOUNCE] Cache::Cache 0.09

2001-09-10 Thread DeWitt Clinton

Hi,

If people have a moment, please check out Cache::Cache 0.09, which is
available here until I am ready to release it on CPAN:

  http://prdownloads.sourceforge.net/perl-cache/Cache-Cache-0.09.tar.gz 

Changes include:

   - applied Axel Beckert patch to fix the expiration units
   - applied Ken Williams's directory creation patch to pass all tests
   - changed the license to be either the GPL or the Artistic license
   - added Jay Sachs' implementation of NullCache
   - modified the remove methods to avoid croaking if two cache
 instances are both purging or limiting at the same time
   - migrated to a factory-like model with private contructors
 to fix the auto_purge_interval functionality
   - updated the documentation to better reflect that size means
 size in bytes


Cheers!

-DeWitt



Re: Shared memory caching revisited (was it's supposed to SHARE it, not make more!)

2001-09-04 Thread DeWitt Clinton

On Tue, Sep 04, 2001 at 12:14:52PM -0700, Rob Bloodgood wrote:

 ***OH WOW!***  So, DURING the course of composing this message, I've
 realized that the function expire_old_accounts() is now redundant!
 Cache::Cache takes care of that, both with expires_in and max_size.  I'm
 leaving it in for reference, just to show how it's improved. :-)

 [snip]
 
 use Apache::Session::Lock::Semaphore ();
 use Cache::SizeAwareSharedMemoryCache ();
 
 # this is used in %cache_options, as well as for locking
 use constant SIGNATURE = 'EXIT';
 use constant MAX_ACCOUNTS = 300;
 
 # use vars qw/%ACCOUNTS/;
 use vars qw/$ACCOUNTS $locker/;
 
 my %cache_options = ( namespace = SIGNATURE,
   default_expires_in =
 max_size = MAX_ACCOUNTS );


Very neat thought about how to use max_size to limit the the
accounts!  

Unfortunately, you demonstrated that I did a *terrible* job at
documenting what size means.

It means size in bytes, not items.

I will add max_items and limit_items to the TODO list.  In the
meantime, I will improve the documentation.

-DeWitt



Re: module to hit back at default.ida atack ?

2001-08-06 Thread DeWitt Clinton

On Tue, Aug 07, 2001 at 08:18:18PM +1000, Cees Hek wrote:

 So what your saying is that you have a list of potentially rooted machines
 that you are making publically available...  Doesn't sound like such a
 good idea to me...

So *that's* why Reuven has CodeRed.pm CC him on the warning emails.

And I thought he was just nuts.  ;)

-DeWitt




Re: odd behavior with Cache::Cache

2001-08-04 Thread DeWitt Clinton

On Sat, Aug 04, 2001 at 09:17:21AM -0700, brian moseley wrote:

 could this be because i'm only ever doing one set, at
 session creation time? since i'm using a memory cache, i
 didn't expect to have to call set every time i modify the
 cached data.

Yup.  That would be it.  Please see my response to my next post (the
one in which you mention solving the problem).


 also i didn't know that Storable was used underneath. why is
 this necessary?

To quote Jimi Williams, manager's decision.  I decided to optimize
for the more common scenario in which one is pulling objects out of
the cache and modifying them.  Rather than force the user to manually
clone the retrieved object, I figured that it was better to do the
clone up front and require the user to store the object again if they
modify it.  This is to preserve consistent semantics and behavior
across the various cache implementations.  I.e., what works for
MemoryCache should work for the SharedMemoryCache or the FileCache as
well.

That's not to say one couldn't subclass MemoryCache to have the
alternate behavior you propose...

-DeWitt






Re: odd behavior with Cache::Cache

2001-08-04 Thread DeWitt Clinton

On Sat, Aug 04, 2001 at 10:34:34AM -0700, brian moseley wrote:

 ps: i've modified my code to 1) only get once per request
 and 2) set at the end of each request. the net effect is
 that stuff works as expected. i'm reasonably happy with the
 current state of affairs, but...

Excellent, this is the right approach.  Sounds like I need to update
the documentation to say that objects retrieved from the cache are
not 'live,' they are clones.  If you want to save modifications,
remember to store them again in the cache.


 i don't like having to explicitly call set to force modifications to
 be written to the cache, and i'd prefer that get always return the
 same instance that i originally set. can these issues be considered
 for a future version of the interface? i see the need for the
 current behavior when using file-based caches, but perhaps there's a
 way to streamline operations for memory caches? perhaps Storable can
 be bypassed for memory caches?

Please see my last email on the subject, but for all intents and
purposes, I'd like caches to behave consistently, but we could
definitely create a special purpose live memory-based cache that
does what you want.



 also, has there been any thought given to locking cached
 items? when i'm using a shared cache with multiprocess
 apache, the opportunity exists for multiple requests to
 access a single session simultaneously, which can lead to
 races. which i'm happy to ignore for now but would be nice
 to eventually prevent.

A number of people have requested this, and I think it is a good idea.
For some back-end implementations, locking is conceptually easy to do,
such as the FileCache on a POSIX system.  And the SharedMemoryCache
already does some locking using IPC::ShareLite.  However, I wanted to
wait until I came up with the right level of granularity for the
locking API.  Plus, it would require a non-trivial bit of a rewrite to
separate the front-end of the cache (which would block on locks) from
the back-end (which would be aware of the locks and manage them).

However, the good news is that there isn't really a race on writes.
Basically, the last write wins.  It is tough to really figure out
(from the cache's perspective) what the appropriate behavior is in all
cases.  So the user should be responsible for locking if them want it.
Of course, that should be done through the cache API, which I
regretfully haven't added yet.

Cheers,

-DeWitt




Re: odd behavior with Cache::Cache

2001-08-04 Thread DeWitt Clinton

On Sat, Aug 04, 2001 at 03:06:37PM -0400, DeWitt Clinton wrote:

 To quote Jimi Williams, manager's decision.

Yes, I know it is Jimy Williams.  See what happens when a lifelong Red
Sox fan relocates down to NYC?  We start losing our minds...  :)

-DeWitt



[ANNOUNCE] Cache-Cache-0.08

2001-04-27 Thread DeWitt Clinton

Summary:

  The Perl Cache package provides Cache::Cache, a generic interface
  for creating persistent data stores.  This interface is implemented
  by the Cache::MemoryCache, Cache::SharedMemoryCache, Cache::FileCache, 
  Cache::SizeAwareFileCache, Cache::SizeAwareMemoryCache, and 
  Cache::SizeAwareSharedMemoryCache classes.  This work replaces 
  File::Cache and IPC::Cache.


Release Notes:

  This release adds the new auto purge feature, contains
  minor API enhancements, and includes improved documentation.


Project Homepage:

  http://sourceforge.net/projects/perl-cache/


Tar/GZ:

  http://prdownloads.sourceforge.net/perl-cache/Cache-Cache-0.08.tar.gz

  http://www.cpan.org/modules/by-module/Cache/Cache-Cache-0.08.tar.gz



Changelog:

  http://sourceforge.net/project/shownotes.php?release_id=32744



CVS tree (viewcvs):

  http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/perl-cache/


The following is the Cache-Cache-0.08 README file:


Copyright (C) 2001 DeWitt Clinton  All Rights Reserved
 
   This program is free software; you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
   the Free Software Foundation; either version 1, or (at your option)
   any later version.
  
   This program is distributed in the hope that it will be useful,
   but WITHOUT ANY WARRANTY; without even the implied warranty of
   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
   GNU General Public License for more details.
   
   You should have received a copy of the GNU General Public License
   along with this program; if not, write to the Free Software
   Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.


NAME

  Cache::Cache


DESCRIPTION

  The Perl Cache package provides Cache::Cache, a generic interface
  for creating persistent data stores.  This interface is implemented
  by the Cache::MemoryCache, Cache::SharedMemoryCache, Cache::FileCache, 
  Cache::SizeAwareFileCache, Cache::SizeAwareMemoryCache, and 
  Cache::SizeAwareSharedMemoryCache classes.  This work replaces 
  File::Cache and IPC::Cache.


REQUIREMENTS

  Digest::MD5
  File::Spec
  File::Path
  IPC::ShareLite
  Storable


INSTALLATION

  perl Makefile.PL
  make
  make test
  make install


USAGE

  First, choose the best type of cache implementation for your needs.
  The simplest cache is the MemoryCache, which is suitable for
  applications that are serving multiple sequential requests, and
  which to avoid making redundant expensive queries, such as an
  Apache/mod_perl application talking to a database.  If you wish to
  share that data between processes, then perhaps the
  SharedMemoryCache is appropriate, although its behavior is tightly
  bound to the underlying IPC mechanism, which varies from system to
  system, and is unsuitable for large objects or large numbers of
  objects.  When the SharedMemoryCache is not acceptable, then
  FileCache offers all of the same functionality with similar
  performance metrics, and it is not limited in terms of the number of
  objects or their size.  If you wish to maintain a strict limit on
  the size of a file system based cache, then the SizeAwareFileCache
  is the way to go.  Similarly, the SizeAwareMemoryCache and the
  SizeAwareSharedMemoryCache add size management functionality
  to the MemoryCache and SharedMemoryCache classes respectively.

 
  Using a cache is simple.  Here is some sample code for instantiating
  and using a MemoryCache:

use Cache::Cache qw( $EXPIRES_NEVER $EXPIRES_NOW );
use Cache::MemoryCache;

my $options_hash_ref = { 'default_expires_in' = '10 seconds' };

my $cache = new Cache::MemoryCache( $options_hash_ref );

my $expires_in = '10 minutes';

$cache-set( 'Key', 'Value', $expires_in );

# if the next line is called within 10 minutes, then this 
# will return the cache value

my $value = $cache-get( 'Key' );


  Please refer to the perldoc for Cache::Cache and the related
  implementations for complete documentation.


SEE ALSO

  File::Cache and IPC::Cache, and the project homepage at:

http://sourceforge.net/projects/perl-cache/


AUTHOR

  Original author: DeWitt Clinton [EMAIL PROTECTED]

  Last author: $Author: dclinton $

  Copyright (C) 2001 DeWitt Clinton




Re: Getting a Cache::SharedMemoryCache started

2001-03-27 Thread DeWitt Clinton

On Tue, Mar 27, 2001 at 09:50:33PM +0100, Matt wrote:

 I want to cache some values in my handlers so they don't keep having
 to look stuff up, and so I figured a Cache::SharedMemoryCache would
 be the way to go.

It all depends on what you want to cache and why.  If you wanted to
save a trip to the database on each request, a really simple mod_perl
handler could look something like this:

  package MyApplication;

  use strict;
  use Apache::Request;
  use Cache::SharedMemoryCache;

  sub handler
  {
my ( $request );

my $cache = new Cache::SharedMemoryCache( );

my $data = $cache-get( 'data' );

if ( not defined $data )
{
  $data = expensive_database_request( );

  $cache-set( 'data', $data );
}

$request-content_type( 'text/plain' );
$request-send_http_header( );
$request-print( "data: $data" );
  }
  

The first time this handler is run, the data will not be found in the
cache.  The second time, however, the data will be returned from the
cache, saving a trip to the database.

The other question is whether or not to share the cache instance
itself globally.  Technically, this is up to you.  Personally I
wouldn't bother, considering the overhead of instantiating the cache
is so low that I would rather keep it local to the handler (as I did
above).  I tend to only share data globally in mod_perl handlers very
judiciously, and even then I wrap the data in an object that provides
accessor methods.  For an example of an object that wraps shared data,
check out Cache::MemoryCache, which simply caches data inside the
process.

Hope this helps a bit.  Cheers,

-DeWitt





Re: Getting a Cache::SharedMemoryCache started

2001-03-27 Thread DeWitt Clinton

On Tue, Mar 27, 2001 at 05:19:15PM -0500, Pierre Phaneuf wrote:

 DeWitt Clinton wrote:
 
  The other question is whether or not to share the cache instance
  itself globally.  Technically, this is up to you.  Personally I
  wouldn't bother, considering the overhead of instantiating the
  cache is so low that I would rather keep it local to the handler
  (as I did above).  I tend to only share data globally in mod_perl
  handlers very judiciously, and even then I wrap the data in an
  object that provides accessor methods.  For an example of an
  object that wraps shared data, check out Cache::MemoryCache, which
  simply caches data inside the process.
 
 Wouldn't this cause the cache to be torn down and rebuilt for every
 request???

The cache instance itself would be, but the data behind it (a global
hash in the case of the MemoryCache, and a IPC::ShareLite store for
the SharedMemoryCache ) would stay around.  Since the instance isn't
that heavy, I feel better instantiating them fresh every request.  If
you need optimal performance, however, you may want to keep the
instance itself global.  

Which reminds me of something.  These cache objects are not currently
thread safe.  When should I start expecting multi-threaded
apache/mod_perl to become mainstream enough to warrant an overhaul of
the code?  I imagine that nearly most Perl libraries are not thread
safe, of course.  But code that will be used in mod_perl environments
needs to be, right?

-DeWitt




[ANNOUNCE] Cache-Cache-0.05

2001-03-20 Thread DeWitt Clinton

Summary:

  The Perl Cache package provides Cache::Cache, a generic interface
  for creating persistent data stores.  This interface is implemented
  by the Cache::MemoryCache, Cache::SharedMemoryCache, Cache::FileCache, 
  Cache::SizeAwareFileCache, Cache::SizeAwareMemoryCache, and 
  Cache::SizeAwareSharedMemoryCache classes.  This work replaces 
  File::Cache and IPC::Cache.


Release Notes:

  This release replaces IPC::Shareable with IPC::ShareLite as the
  backend for the SharedMemoryCache and updates external module
  dependencies to the latest versions.


Project Homepage:

  http://sourceforge.net/projects/perl-cache/


Tar/GZ:

  http://ftp1.sourceforge.net/perl-cache/Cache-Cache-0.05.tar.gz


Changelog:

  http://sourceforge.net/project/shownotes.php?release_id=27936


CVS tree (cvsweb):

  http://cvs.sourceforge.net/cgi-bin/cvsweb.cgi?cvsroot=perl-cache



The following is the Cache-Cache-0.05 README file:


Copyright (C) 2001 DeWitt Clinton  All Rights Reserved
 
   This program is free software; you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
   the Free Software Foundation; either version 1, or (at your option)
   any later version.
  
   This program is distributed in the hope that it will be useful,
   but WITHOUT ANY WARRANTY; without even the implied warranty of
   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
   GNU General Public License for more details.
   
   You should have received a copy of the GNU General Public License
   along with this program; if not, write to the Free Software
   Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.


NAME

  Cache::Cache


DESCRIPTION

  The Perl Cache package provides Cache::Cache, a generic interface
  for creating persistent data stores.  This interface is implemented
  by the Cache::MemoryCache, Cache::SharedMemoryCache, Cache::FileCache, 
  Cache::SizeAwareFileCache, Cache::SizeAwareMemoryCache, and 
  Cache::SizeAwareSharedMemoryCache classes.  This work replaces 
  File::Cache and IPC::Cache.


REQUIREMENTS

  Digest::MD5
  File::Spec
  File::Path
  IPC::ShareLite
  Storable

INSTALLATION

  perl Makefile.PL
  make
  make test
  make install


USAGE

  First, choose the best type of cache implementation for your needs.
  The simplest cache is the MemoryCache, which is suitable for
  applications that are serving multiple sequential requests, and
  which to avoid making redundant expensive queries, such as an
  Apache/mod_perl application talking to a database.  If you wish to
  share that data between processes, then perhaps the
  SharedMemoryCache is appropriate, although its behavior is tightly
  bound to the underlying IPC mechanism, which varies from system to
  system, and is unsuitable for large objects or large numbers of
  objects.  When the SharedMemoryCache is not acceptable, then
  FileCache offers all of the same functionality with similar
  performance metrics, and it is not limited in terms of the number of
  objects or their size.  If you wish to maintain a strict limit on
  the size of a file system based cache, then the SizeAwareFileCache
  is the way to go.  Similarly, the SizeAwareMemoryCache and the
  SizeAwareSharedMemoryCache add size management functionality
  to the MemoryCache and SharedMemoryCache classes respectively.
 
  Using a cache is simple.  Here is some sample code for instantiating
  and using a MemoryCache:

use Cache::Cache qw( $EXPIRES_NEVER $EXPIRES_NOW );
use Cache::MemoryCache;

my $options_hash_ref = { 'default_expires_in' = '10 seconds' };

my $cache = new Cache::MemoryCache( $options_hash_ref );

my $expires_in = '10 minutes';

$cache-set( 'Key', 'Value', $expires_in );

# if the next line is called within 10 minutes, then this 
# will return the cache value

my $value = $cache-get( 'Key' );

  Please refer to the perldoc for Cache::Cache and the related
  implementations for complete documentation.


SEE ALSO

  File::Cache and IPC::Cache


AUTHOR

  Original author: DeWitt Clinton [EMAIL PROTECTED]

  Last author: $Author: dclinton $

  Copyright (C) 2001 DeWitt Clinton




[ANNOUNCE] Cache-Cache-0.04

2001-03-17 Thread DeWitt Clinton

Summary:

  The Perl Cache package provides Cache::Cache, a generic interface
  for creating persistent data stores.  This interface is implemented
  by the Cache::MemoryCache, Cache::SharedMemoryCache, Cache::FileCache, 
  Cache::SizeAwareFileCache, Cache::SizeAwareMemoryCache, and 
  Cache::SizeAwareSharedMemoryCache classes.  This work replaces 
  File::Cache and IPC::Cache.

Release Notes:

  This release offers two new classes, the SizeAwareMemoryCache 
  and the SizeAwareSharedMemoryCache, replaces the Data::Dumper 
  persistence mechanism with Storable, and adds a new 
  SizeAwareCache interface, plus some small bug fixes. 

Project Homepage:

  http://sourceforge.net/projects/perl-cache/


Tar/GZ:

  http://ftp1.sourceforge.net/perl-cache/Cache-Cache-0.04.tar.gz


Changelog:

  http://sourceforge.net/project/shownotes.php?release_id=27594


CVS tree (cvsweb):

  http://cvs.sourceforge.net/cgi-bin/cvsweb.cgi?cvsroot=perl-cache



The following is the Cache-Cache-0.04 README file:


Copyright (C) 2001 DeWitt Clinton  All Rights Reserved
 
   This program is free software; you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
   the Free Software Foundation; either version 1, or (at your option)
   any later version.
  
   This program is distributed in the hope that it will be useful,
   but WITHOUT ANY WARRANTY; without even the implied warranty of
   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
   GNU General Public License for more details.
   
   You should have received a copy of the GNU General Public License
   along with this program; if not, write to the Free Software
   Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.


NAME

  Cache::Cache


DESCRIPTION

  The Perl Cache package provides Cache::Cache, a generic interface
  for creating persistent data stores.  This interface is implemented
  by the Cache::MemoryCache, Cache::SharedMemoryCache, Cache::FileCache, 
  Cache::SizeAwareFileCache, Cache::SizeAwareMemoryCache, and 
  Cache::SizeAwareSharedMemoryCache classes.  This work replaces 
  File::Cache and IPC::Cache.


REQUIREMENTS

  Digest::MD5
  File::Spec
  File::Path
  IPC::Shareable
  Storable

INSTALLATION

  perl Makefile.PL
  make
  make test
  make install


USAGE

  First, choose the best type of cache implementation for your needs.
  The simplest cache is the MemoryCache, which is suitable for
  applications that are serving multiple sequential requests, and
  which to avoid making redundant expensive queries, such as an
  Apache/mod_perl application talking to a database.  If you wish to
  share that data between processes, then perhaps the
  SharedMemoryCache is appropriate, although its behavior is tightly
  bound to the underlying IPC mechanism, which varies from system to
  system, and is unsuitable for large objects or large numbers of
  objects.  When the SharedMemoryCache is not acceptable, then
  FileCache offers all of the same functionality with similar
  performance metrics, and it is not limited in terms of the number of
  objects or their size.  If you wish to maintain a strict limit on
  the size of a file system based cache, then the SizeAwareFileCache
  is the way to go.  Similarly, the SizeAwareMemoryCache and the
  SizeAwareSharedMemoryCache add size management functionality
  to the MemoryCache and SharedMemoryCache classes respectively.
 
  Using a cache is simple.  Here is some sample code for instantiating
  and using a MemoryCache:

use Cache::Cache qw( $EXPIRES_NEVER $EXPIRES_NOW );
use Cache::MemoryCache;

my $options_hash_ref = { 'default_expires_in' = '10 seconds' };

my $cache = new Cache::MemoryCache( $options_hash_ref );

my $expires_in = '10 minutes';

$cache-set( 'Key', 'Value', $expires_in );

# if the next line is called within 10 minutes, then this 
# will return the cache value

my $value = $cache-get( 'Key' );

  Please refer to the perldoc for Cache::Cache and the related
  implementations for complete documentation.


SEE ALSO

  File::Cache and IPC::Cache


AUTHOR

  Original author: DeWitt Clinton [EMAIL PROTECTED]

  Last author: $Author: dclinton $

  Copyright (C) 2001 DeWitt Clinton





[OT] Re: mod_perl shared memory with MM

2001-03-11 Thread DeWitt Clinton

On Sun, Mar 11, 2001 at 03:33:12PM +0100, Christian Jaeger wrote:

 I've looked at Cache::FileCache now and think it's (currently) not 
 possible to use for IPC::FsSharevars:
 
 I really miss locking capabilities. Imagine a script that reads a 
 value at the beginning of a request and writes it back at the end of 
 the request. If it's not locked during this time, another instance 
 can read the same value and then write another change back which is 
 then overwritten by the first instance.


I'm very intrigued by your thinking on locking.  I had never
considered the transaction based approach to caching you are referring
to.  I'll take this up privately with you, because we've strayed far
off the mod_perl topic, although I find it fascinating.



 - why don't you use 'real' constants for $SUCCESS and the like? (use
 constant)

Two reasons, mostly historical, and not necessarily good ones.

One, I benchmarked some code once that required high performance, and
the use of constants was just slightly slower.

Two, I like the syntax $hash{$CONSTANT}.  If I remember correctly,
$hash{CONSTANT} didn't work.  This may have changed in newer versions
of Perl.

Obviously those are *very* small issues, and so it is mostly by habit
that I don't use constant.  I would consider changing, but it would
mean asking everyone using the code to change too, because they
currently import and use the constants as Exported scalars.

Do you know of a very important reason to break compatibility and
force the switch?  I'm not opposed to switching if I have to, but I'd
like to minimize the impact on the users.



 - you probably should either append the userid of the process to 
 /tmp/FileCache or make this folder globally writeable (and set the 
 sticky flag). Otherwise other users get a permission error.

As of version 0.03, the cache directories, but not the cache entries,
are globally writable by default.  Users can override this by changing
the 'directory_umask' option, or keep data private altogether by
changing the 'cache_root'.  What version did you test with?  There may
be a bug in there.



 - why don't you use Storable.pm? It should be much faster than Data::Dumper

The TODO contains "Replace Data::Dumper with Storable (maybe)".  :) 

The old File::Cache module used Storable, btw.

It will be trivial to port the new Cache::FileCache to use Storable.
I simply wanted to wait until I had the benchmarking code so I could
be sure that Storeable was faster.  Actually, I'm not 100% sure that I
expect Storeable to be faster than Data::Dumper.  If Data::Dumper
turns out to be about equally fast, then I'll stay with it, because it
is available on all Perl installations, I believe.

Do you know if Storeable is definitely faster?  If you have benchmarks
then I am more than happy to switch now.  Or, do you know of a reason,
feature wise, that I should switch?  Again, it is trivial to do so.



 I have some preliminary benchmark code -- only good for relative
 benchmarking, but it is a start.  I'd be happy to post the results
 here if people are interested.
 
 Could you send me the code?, then I'll look into benchmarking my
 module too.

I checked it in as Cache::CacheBenchmark.  It isn't good code, nor
does it necessarily work just yet.  I simply checked it in while I was
in the middle of working on it.  I'm turning it into a real
benchmarking class for the cache, and hopefully that will help you a
little bit.


Cheers,

-DeWitt





[ANNOUNCE] Cache-Cache-0.03

2001-03-10 Thread DeWitt Clinton

Summary:

  Perl Cache is the successor to the popular File::Cache and
  IPC::Cache perl libraries. This project unifies those modules under
  the generic Cache::Cache interface and implements Cache::FileCache,
  Cache::MemoryCache, Cache::SharedMemoryCache, and
  Cache::SizeAwareFileCache.


Release Notes:

  This release offers performance enhancements, expiration and size
  reduction bug fixes, an improved expiration string syntax, much
  better documentation, and a new cache API method.


Project Homepage:

  http://sourceforge.net/projects/perl-cache/


Tar/GZ:

  http://ftp1.sourceforge.net/perl-cache/Cache-Cache-0.03.tar.gz


Changelog:

  http://sourceforge.net/project/shownotes.php?release_id=26779


CVS tree (cvsweb):

  http://cvs.sourceforge.net/cgi-bin/cvsweb.cgi?cvsroot=perl-cache



The following is the Cache-Cache-0.03 README file:


Copyright (C) 2001 DeWitt Clinton  All Rights Reserved
 
   This program is free software; you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
   the Free Software Foundation; either version 1, or (at your option)
   any later version.
  
   This program is distributed in the hope that it will be useful,
   but WITHOUT ANY WARRANTY; without even the implied warranty of
   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
   GNU General Public License for more details.
   
   You should have received a copy of the GNU General Public License
   along with this program; if not, write to the Free Software
   Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.


NAME

  Cache::Cache


DESCRIPTION

  The Perl Cache package provides Cache::Cache, a generic interface
  for creating persistent data stores.  This interface is implemented
  by the Cache::FileCache, Cache::SizeAwareFileCache,
  Cache::MemoryCache, and Cache::SharedMemoryCache classes.  This work
  replaces File::Cache and IPC::Cache.


REQUIREMENTS

  Data::Dumper
  Digest::MD5
  File::Spec
  File::Path
  IPC::Shareable


INSTALLATION

  perl Makefile.PL
  make
  make test
  make install


USAGE

  First, choose the best type of cache implementation for your needs.
  The simplest cache is the MemoryCache, which is suitable for
  applications that are serving multiple sequential requests, and
  which to avoid making redundant expensive queries, such as an
  Apache/mod_perl application talking to a database.  If you wish to
  share that data between processes, then perhaps the
  SharedMemoryCache is appropriate, although its behavior is tightly
  bound to the underlying IPC mechanism, which varies from system to
  system, and is unsuitable for large objects or large numbers of
  objects.  When the SharedMemoryCache is not acceptable, then
  FileCache offers all of the same functionality with similar
  performance metrics, and it is not limited in terms of the number of
  objects or their size.  If you wish to maintain a strict limit on
  the size of a file system based cache, then the SizeAwareFileCache
  is the way to go.
 
  Using a cache is simple.  Here is some sample code for instantiating
  and using a MemoryCache:

use Cache::Cache qw( $EXPIRES_NEVER $EXPIRES_NOW );
use Cache::MemoryCache;

my $options_hash_ref = { 'default_expires_in' = '10 seconds' };

my $cache = new Cache::MemoryCache( $options_hash_ref );

my $expires_in = '10 minutes';

$cache-set( 'Key', 'Value', $expires_in );

# if the next line is called within 10 minutes, then this 
# will return the cache value

my $value = $cache-get( 'Key' );

  Please refer to the perldoc for Cache::Cache and the related
  implementations for complete documention.


SEE ALSO

  File::Cache and IPC::Cache


AUTHOR

  Original author: DeWitt Clinton [EMAIL PROTECTED]

  Last author: $Author: dclinton $

  Copyright (C) 2001 DeWitt Clinton















Re: [ANNOUNCE] Cache-Cache-0.03

2001-03-10 Thread DeWitt Clinton

On Sat, Mar 10, 2001 at 11:17:21AM -0800, Bill Moseley wrote:

 When you say successor to File::Cache does that means File::Cache
 will not be maintained as a separate module anymore?

There are no features in File::Cache that will not exist in
Cache::FileCache and Cache::SizeAwareFileCache.  Actually, as far as I
know, the only old feature missing at all from the new code is the
ability to tell the cache not to physically remove expired objects.
There is a backdoor around this in 0.03, however, so if you need that
functionality, it is already possible.

I'm hoping to retire File::Cache soon unless I get another maintainer.
But it seems silly to have redundant efforts.  The new code is much,
much, better.


 Have you though about making SharedMemoryCache flush to disk if it
 becomes full but before it's time to expire the data?

I've done a lot of thinking about a multi-layered cache
implementation.  The API would be the same, but it would be clever
about using MemoryCache - SharedMemoryCache - FileCache to make read
access to data super efficient, and only persist outward on writes.

I'll mark that down as a potential 0.04 feature.  Thanks, Bill!

-DeWitt




Re: [ANNOUNCE] Cache-Cache-0.03

2001-03-10 Thread DeWitt Clinton

On Sat, Mar 10, 2001 at 02:00:34PM -0600, Daniel Little (Metrex) wrote:

 Along the same lines, how about making SizeAwareMemoryCache as well
 so that you can specify just how much data you want stored in the
 cache.

Good idea.  I'll add that to the 0.04 feature list as well.

Thanks!

-DeWitt



Re: [ANNOUNCE] Cache-Cache-0.03

2001-03-10 Thread DeWitt Clinton

On Sat, Mar 10, 2001 at 12:10:35PM -0800, Perrin Harkins wrote:
 "Daniel Little (Metrex)" wrote:
  Along the same lines, how about making SizeAwareMemoryCache as well so that
  you can specify just how much data you want stored in the cache.
 
 Sounds like Joshua Chamas' Tie::Cache module.  It provides a
 size-limited LRU cache.

Note that Cache::SizeAwareFileCache does both a "next expiration" and
a LRU algorithm for limiting the size.  I'll use the same logic for
the SizeAwareMemoryCache.

Thanks,

-DeWitt



Re: mod_perl shared memory with MM

2001-03-10 Thread DeWitt Clinton

On Sat, Mar 10, 2001 at 04:35:02PM -0800, Perrin Harkins wrote:
 Christian Jaeger wrote:
  Yes, it uses a separate file for each variable. This way also locking
  is solved, each variable has it's own file lock.
 
 You should take a look at DeWitt Clinton's Cache::FileCache module,
 announced on this list.  It might make sense to merge your work into
 that module, which is the next generation of the popular File::Cache
 module.

Yes!  I'm actively looking for additional developers for the Perl
Cache project.  I'd love new implementations of the Cache interface.
Cache::BerkeleyDBCache would be wonderful.  Check out:
  
  http://sourceforge.net/projects/perl-cache/

For what it is worth, I don't explicitly lock.  I do atomic writes
instead, and have yet to hear anyone report a problem in the year the
code has been public.


  It's a bit difficult to write a realworld benchmark.
 
 It certainly is.  Benchmarking all of the options is something that I've
 always wanted to do and never find enough time for.

I have some preliminary benchmark code -- only good for relative
benchmarking, but it is a start.  I'd be happy to post the results
here if people are interested.

-DeWitt



[ANNOUNCE] Cache-Cache-0.02

2001-02-16 Thread DeWitt Clinton

Summary:

  Perl Cache is the successor to the popular File::Cache and
  IPC::Cache perl libraries. This project unifies those modules under
  the generic Cache::Cache interface and implements Cache::FileCache,
  Cache::MemoryCache, Cache::SharedMemoryCache, and
  Cache::SizeAwareFileCache.


Release Notes:

  This release implements Cache::SizeAwareFileCache, which  
  extends Cache::FileCache and adds dynamic cache sizing
  functionality


Project Homepage:

  http://sourceforge.net/projects/perl-cache/


Tar/GZ:

  http://download.sourceforge.net/perl-cache/Cache-Cache-0.02.tar.gz


Changelog:

  http://sourceforge.net/project/shownotes.php?release_id=24111


CVS tree (cvsweb):

  http://cvs.sourceforge.net/cgi-bin/cvsweb.cgi?cvsroot=perl-cache





Re: [OT] re:advocacy at perl.com

2001-02-16 Thread DeWitt Clinton

On Fri, Feb 16, 2001 at 03:14:37PM +, Matt Sergeant wrote:

 I wonder if I should write them an article about take23, and then
 use the funds to pay for articles for take23 :-)

Geez, Matt, you promote take23 every chance you get.  

That's worse than me mentioning Avacet (www.avacet.com), Perl Cache
(sourceforge.net/projects/perl-cache), and the fact that I'm looking
for a job (www.unto.net/resume.html).  ;)

-DeWitt



[ANNOUNCE] Cache::Cache 0.01

2001-02-13 Thread DeWitt Clinton

Hello:

  Perl Cache is the successor to the popular File::Cache and
  IPC::Cache perl libraries.  This project unifies those modules under
  the generic Cache::Cache interface and implements Cache::FileCache,
  Cache::MemoryCache, and Cache::SharedMemoryCache.


The project is being hosted on SourceForge at:

  http://sourceforge.net/projects/perl-cache/


I have released the first version of the unified Cache code as version
0.01, and it can be downloaded here:

  http://download.sourceforge.net/perl-cache/Cache-Cache-0.01.tar.gz


I have not implemented the full functionality of File::Cache (e.g.,
dynamic cache sizing) quite yet.  I am in the process of porting that
code and it will be included in the next release as a new module.

Comments and critique always welcome!

Cheers,

-DeWitt







Re: File::Cache problem

2001-02-07 Thread DeWitt Clinton

On Wed, Feb 07, 2001 at 04:02:39PM +0400, BeerBong wrote:

 Cache size after 24 hours of working

 via 'du -s' - 53M
 via 'perl -MFile::Cacje -e"print File::Cache::SIZE('PRTL')"' - 10M

That looks like a serious bug.  Can you double check on that?  Where
are you running the "du -s"?

The SIZE( ) method returns the total size of the objects in the cache,
not including the overhead of directories themselves.  As the
documentation explains, it isn't a perfect mechanism, but it should be
a lot closer than you experienced.



 What is the best way to limit cache ?
 May be variant when File::Cache::REDUCE_SIZE(10485760) in cron.dayly
 will be better solution ? I this case max_size doesnt need to specify
 in object initialization params...

I personally prefer using a cron job to periodically limit the size of
the cache.  I don't like using "max_size" because of the overhead
involved on every set.  I'd rather just fine tune the number of times
a cron script is executed (i.e., once an hour vs. once a day).



 Another kind of problem appears recently.
 
 In logs
 [Wed Feb  7 15:42:57 2001] [error] Couldn't rename 
/tmp/File::Cache/.description.tmp13984 to /tmp/File::Cache/.description at 
/usr/web/inc/WebCache.pm line 15
 [Wed Feb  7 15:44:37 2001] [error] Couldn't rename 
/tmp/File::Cache/apache/PRTL/b/5/a/b5a261aa13cf0ddfd41beb7d64960248.tmp13441 to 
/tmp/File::Cache/apache/PRTL/b/5/a/b5a261aa13cf0ddfd41beb7d64960248 at 
/usr/web/inc/WebCache.pm line 87
 ...propagated at /usr/web/inc/Apache/Portal.pm line 109.
 
 What I can do ?

Unfortunately the error message doesn't tell me enough.  If you can,
please modify lines 1578 and 1579 of Cache.pm to read as follows:

   rename ($temp_filename, $filename) or
 croak("Couldn't rename $temp_filename to $filename:  $!");


The next time it hangs, the error message will have more information.

I'll fix that in the next version of File::Cache.



 And another question (I'm use Linux Debian Potato)...
 Is there way to define params of the currently performing request
 (URI, script, etc) of hanging mod_perl process ? I browsed /proc/[PID of
 mod_perl]/, but I think this info is about first request. Am I wrong ?

Are you looking in /proc/[pid]/environ for the request parameters?
I'm surprised this information was available even for the first
request.  I don't believe environ is updated dynamically (I think it
is filled out once on the exec of the process).  It certainly won't
provide the information on subsequent requests.  In any case, the
format and behavior of /proc is so dependent on the particular version
of the Linux kernel that I wouldn't want to rely on it for anything
more than occasional debugging.

Two options for you would be to log the request every time via warn.
It will fill your logs very quickly, but hopefully it will hang
quickly and you can get the debugging information out.

Another solution is to compile apache and mod_perl with debugging
symbols and attach to it with "gdb".  I have tried this a long time,
though.  Please see the current thread about using gdb.


Best of luck!

-DeWitt



Re: pseudo-hashes? was: Data structure question

2001-01-23 Thread DeWitt Clinton

On Tue, Jan 23, 2001 at 10:06:13AM +, Matt Sergeant wrote:

 The only gain might be in a large DOM tree where there may be
 thousands of objects. But then you're really better off using an
 array based class instead (as I found out).

This is getting a bit off-topic, but I'm empirically found that the
DOM is not necessarily the best object model to use in a mod_perl
environment.  XML::DOM in particular has such a high overhead in terms
of memory (and memory leaks) and performance, that it is sometimes
inappropriate for a context that requires a small footprint, and
generally fast throughput (like mod_perl).

For example, in version 1 of the Avacet perl libraries, we were using
XML::DOM for both our XML-RPC mechanism and as the underlying data
structure for object manipulation.  In version 2, however, we created
an architecture that automatically converts between the language
agnostic XML and native blessed objects using a custom engine built on
the PerlSAX parser.  This reduced our memory footprint dramatically,
stopped up the memory leaks, and increased performance significantly.
Moreover, the object model now exposed is based on native perl objects
with an API geared toward property manipulation (i.e., get_foo,
set_foo) which is easier to program directly to than the DOM.

You can see this in action with the modules available in the
Avacet::Core::Rpc::Xml namespace at www.avacet.com.  

Best regards,

-DeWitt



Re: Caching search results

2001-01-08 Thread DeWitt Clinton

On Mon, Jan 08, 2001 at 10:10:25AM -0800, Perrin Harkins wrote:

 Always start with CPAN.  Try Tie::FileLRUCache or File::Cache for
 starters. A dbm would be fine too, but more trouble to purge old
 entries from.

If you find that File::Cache works for you, then you may also want to
check out the simplified and improved version in the Avacet code,
which additionally offers a unified service model for mod_perl
applications.  Services are available for templates (either Embperl or
Template Toolkit), XML-based configuratio, object caching, connecting
to the Avacet application engine, standardized error handling,
dynamically dispatching requests to modules, and many other things.

-DeWitt




[ANNOUNCE] Avacet Application Engine 2.0b2

2000-12-28 Thread DeWitt Clinton


Hi!  I've been quiet for a while on this list, but I still read it
religiously.  I wanted to take a second and share our announcement of
the release of the second beta of the Avacet Application Engine.

   Version 2.0b2 of the Avacet Application Engine is available for 
   download at http://www.avacet.com.

   The Avacet software suite consists of two main parts. The first is
   a language-neutral definition of data and interfaces for exchanging
   data, and the second is a reference implementation of an
   application engine that implements the Avacet framework and client
   libraries. It is suitable for the development of custom enterprise
   applications that require a high degree of scalability,
   reliability, and interoperability.

   The second beta release of the Avacet Application Engine adds a
   language-independent definition format for module interfaces,
   offers XML/RPC over HTTP, new HOWTO documentation geared toward new
   users, and many other enhancements.  This is the last beta before
   version 2.0 of the Avacet Application Engine


Perhaps more so than other application platforms, the Avacet engine is
targeted toward building enterprise quality applications in a Perl and
Apache/mod_perl environment.  The Avacet framework itself is a
language-neutral and open standard for describing data and protocols
for data exchange.  Because the definition is not represented in a
particular programming language, we are able to implement client
libraries in multiple languages (such as Perl) that can share common
middle-tier business logic.

I fundamentally believe that Perl, when written well, can be an
enterprise ready programming language, and that Apache/mod_perl is one
of the best environments (if not the best) for building web
applications.  To that end, we have built a feature rich set of
modules that ease the development of mod_perl applications.  The
Avacet Perl libraries offer a ModPerlApplication base class, as well
as a Service object model for dynamic templates (using HTML::Embperl),
XML based configuration, memory and file system based object caching,
an "exception" model for Perl, among many other features.

I'm announcing this on the mod_perl list because I hope that our code
offers value to engineers doing mod_perl development.  We are using
the Avacet engine and the Perl modules for our own e-commerce
application suite.  By taking advantage of the Avacet framework,
business logic can be shared between custom mod_perl apps, those we
develop at Avacet, and those developed at large by the community.

We have mailing lists set up at the Avacet site, and I invite people
to join us at the avacet-developers list.  I'd be happy to set up an
avacet-perl list if the mod_perl community finds these libraries to be
useful.

You are all invited to check out the second avacet beta at
http://www.avacet.com.  I look forward to hearing your feedback!

Best regards,

-DeWitt




[OT ANNOUNCE] Beans-0.01

2000-06-02 Thread DeWitt Clinton

Hi,

This may be one of the strangest programming things I've ever tried to
do.  Because I've recently been introduced (somewhat embarrassingly)
to the joys of Java Beans, I thought "hey, wouldn't this be great to
have in Perl, too?"

Well, since I'm masochistic like that, I've written a very partial
implementation of the Java Beans framework in Perl.

It only supports properties for the most part, so it is hardly ready
for serious projects.  But the framework is in place, and over time
this may actually turn out to have been a good use of time.

I'm including the README in this mail.  You can grab the 0.01 version
of Perl Beans at the following location:

   http://unto.net/archives/Beans-0.01.tar.gz

If anyone out there in the world thinks this idea has any merit
whatsoever, I'll probably create a SourceForge project for it and
submit it to CPAN.  As always, feedback, advice, and criticism are
always welcome.

-DeWitt


README:


Copyright (C) 2000 DeWitt Clinton [EMAIL PROTECTED], Avacet, Inc.

   This program is free software; you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
   the Free Software Foundation; either version 1, or (at your option)
   any later version.
  
   This program is distributed in the hope that it will be useful,
   but WITHOUT ANY WARRANTY; without even the implied warranty of
   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
   GNU General Public License for more details.
   
   You should have received a copy of the GNU General Public License
   along with this program; if not, write to the Free Software
   Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.

NAME

  Beans::*

DESCRIPTION
 
  Beans is a (very) partial implementation of the Java Beans framework

INSTALLATION

  perl Makefile.PL
  make
  make test
  make install

NOTES

  The implementation is not even close to being complete.  While a
  number of class stubs have been provided, most of the interesting
  parts are still not functional.

  In particular, the entire event model is absent, along with the
  persistence mechanisms.  Additionally, since this is at proof of
  concept stage, not effort has been put into making these classes
  performant.  Numerous caching techniques could be put in place to
  speed things up.

  However, over time there is a reasonable chance that the missing
  pieces are filled in.  Feel free to contribute!


USAGE

  A very simple example is as follows (please excuse the formatting):

  First, define a class that has methods getFoo and setFoo in
  accordance with the Beans convention.

package GenericBean;

sub new {
my ($proto) = @_;
my $class = ref($proto) || $proto;
my $self  = {};
bless ($self, $class);
return $self;
}

sub getFoo {
my ($self) = @_;
return $self-{_foo};
}

sub setFoo {
my ($self, $foo) = @_;
$self-{_foo} = $foo;
}

sub getBar {
my ($self) = @_;
return $self-{_bar};
}

sub setBar {
my ($self, $bar) = @_;
$self-{_bar} = $bar;
}


  First, create an instance of the GenericBean class:

use GenericBean;

my $genericBean = new GenericBean();

  Now, you can use the Beans classes to reflect on a run-time instance
  of that class.  Since Perl doesn't have exactly the same notion of
  Class that Java does, the semantics are a little different.

use Beans::Class;
use Beans::Introspector;
use Beans::BeanInfo;
use Beans::PropertyDescriptor;
use Beans::Method;

my $genericBeanClass =
Beans::Class::getClassFromInstance($genericBean);
  
my $beanInfo =
Beans::Introspector::getBeanInfo($genericBeanClass);

my @propertyDescriptors = $beanInfo-getPropertyDescriptors();

  You can iterate over the property descriptors and actually write
  values to each property:

my $i;

foreach my $propertyDescriptor (@propertyDescriptors) {
 my $writeMethod = $propertyDescriptor-getWriteMethod();
 $writeMethod-invoke($genericBean, $i);
 $i++;
}

  Or, read those values back out again:

foreach my $propertyDescriptor (@propertyDescriptors) {
 my $readMethod = $propertyDescriptor-getReadMethod();
 my $propertyValue = $readMethod-invoke($genericBean);
 my $propertyName = $propertyDescriptor-getName();
 print "$propertyName: $propertyValue\n";
}
  

SEE ALSO

  Sun's Java Beans tutorial:

http://www.javasoft.com/docs/books/tutorial/javabeans/index.html

  Sun's JDK 1.3 API documentation for the java.beans package:

http://java.sun.com/j2se/1.3/docs/api/java/beans/package-summary.html






Re: [OT] Bean::*?

2000-06-01 Thread DeWitt Clinton

On Wed, May 31, 2000 at 04:16:09PM -0700, Perrin Harkins wrote:

 Maybe I'm just being thick, but if you leave out the event driven parts of
 the spec, what's left that isn't just an ordinary Perl class?  You can
 easily make classes to hold data using Class::Struct or
 Class::MethodMaker.

In all this time, I don't think I've seen you simply being thick even
once.  :) I agree, if you take out the event driven stuff, beans
aren't that exciting.  They are basically just property bags with a
way to reflect over those properties.  However, if I build the
beginning of the bean model in Perl, then someone might come along and
add the event model.

 Is it just that you want a safer way to do introspection than looking at
 the data structure the class is made from?  Maybe you want the
 Class::Fields module from CPAN.

Trust me -- this has nothing to do with "safe."  :)  The bean model is 
simply a convention -- if a class has a getFoo and a setFoo, then it
has the property Foo.  Like I said, I don't want to debate whether or
not it is a good standard.  The entire point is to have a consistent
framework between Java and Perl.  Chances are, the underlying
mechanism for doing the reflection will leverage something from the
Class:: package. 

There are a million ways I'd rather see this solved in Perl.  Or in
Java for that matter.  But java.beans are pretty popular, and they
provide the underpinnings for EJB, which, for better or for worse, is
very prevalent in e-commerce.  I'd rather support a mediocre standard
then invent a proprietary one.  And beans aren't *that* mediocre.

-DeWitt



[OT] Bean::*?

2000-05-31 Thread DeWitt Clinton

Hi all,

Okay, this is a rather ridiculous question.  I spent the weekend
implementing a property based object model in Java.  However, after I
finished, someone in my company laughingly pointed out that I had just
re-invented Java beans.  While there were some minor advantages to the
model I came up with, the overwhelmingly richer feature set of the
java.beans.* framework made my efforts redundant at best.

One of the advantages, however, is that the object model I was working
on would translate very nicely to Perl.  In fact, the language
independent nature of this (other than OO being a requirement) was the
real reason I bothered with it at all.

My question is this -- has anyone written an implementation of the
Java Bean standard in Perl?  I don't think we need to have a major
debate about the shortcomings of beans on the mod_perl list.  But I am
wondering if anyone has pulled this off yet.  If we left out the event
driven parts of the bean spec, and just focused on the property side
of things, this could be a rather short project to get the core down.

I'd like to see the Perl version be API compatable with the Java
version.  There is no need to take it much further.

Thoughts?  Flames?  :)

-DeWitt



Re: mod_perl and IPC

2000-05-22 Thread DeWitt Clinton

On Mon, May 22, 2000 at 12:06:20PM -0700, Karyn Ulriksen wrote:

 IPC::Cache has worked like a champ for me on a heavily loaded webserver.
 It's simple and (I'm lead to believe) it's a compiled module.  You interact
 with it like a hash file, but if you're clever you can work it out for your
 usage without semaphores.  It also behaves itself very nicely (no memory
 leaks that I have run into).

Hi, I'm the author of IPC::Cache and frankly, I'm a little surprised
it's worked so well for you.  :) I've noticed some performance issues
with it.  The good news, however, is that Sam Tregar, author of
IPC::SharedCache, and I have been working on a major upgrade.  In the
next release, IPC::Cache will be back-ended against IPC::SharedCache
module.  It will now scale and perform much better.

The other alternative is to use File::Cache (also available on CPAN).
File::Cache actually has better performance characteristics than
IPC::Cache.  Since switching between the two usually entails simply
replacing the string "IPC::Cache" with "File::Cache" it may be worth
checking out.

I could use a few beta testers of the new IPC::Cache module.  If
anyone wants to help, please let me know.

-DeWitt



Re: mod_perl and IPC

2000-05-22 Thread DeWitt Clinton

On Mon, May 22, 2000 at 04:35:54PM -0400, Drew Taylor wrote:

 Could I ask what the problems you were having? I can't use
 IPC::SharedCache currently because I get the error "Couldn't find free
 segment" or something like that when trying to initialize the cache. The
 system engineers are supposed to be changing this kernel parameter, but
 I haven't heard back from them yet.

The problem had to do with large numbers of objects in the cache.
Rather than rephrase what Sam wrote to me, I'll just excerpt it here:

  In case you're wondering why I'm saying IPC::Cache needs help, I
  remember discussing two points.  First, it's storing every cached
  object in the same giant IPC segment and copying it into local memory
  on every cache access.  This essentially means that you are gaining
  *no* memory-usage advantage by using shared memory.  Also, it wastes a
  lot of time copying large hunks of memory on every access.  Second,
  you're doing reads inside an exclusive lock.  This means that no two
  programs can be reading at the same time.  This wastes the potential
  for efficient multi-processing with shared memory.

He was absolutely correct about this.  Since he is actively
maintaining and improving IPC::SharedCache, I figured it makes sense
to simply provide the straightforward IPC::Cache style interface
around his code.

Right now, things are in a holding pattern because I'm finding a limit
on the number of objects I can put in the cache (less than 100, so it
is an issue).  Hopefully Sam will offer some insight here.

-DeWitt



Re: mod_perl and IPC

2000-05-22 Thread 'DeWitt Clinton'

On Mon, May 22, 2000 at 03:46:41PM -0700, Karyn Ulriksen wrote:

 I know what beta testing is, so I hope you didn't think I was trying to yank
 your chain!  I knew that if it got ugly I could easily take that server
 offline and easily back out the code.

Ack!  I'm sorry for what I think is a miscommunication.  I think the
cordial tone was lost in e-mail.  :)  I was being sincere, I definitely
appreciate your testing the code.

 As it was, it got even uglier since I last wrote in.  Again this is FYI...
 for input on the beta and all...  The load balancing service I'm using
 masked the more serious problem of the data failing with get passed
 somewhere in the bowels of Apache/mod_perl where it was before.  I'll tinker
 with it in the wee hours to see if it is failing on the set or the retrieve.
 I reviewed the syntax and there doesn't appear to be any changes between new
 and old (did I miss anything?... it's a little late in the day and I had a
 long weekend :) and it's possible I did and I didn't review for any
 caveats... ).

I'm pretty sure you are doing exactly the right thing.  It is
certainly possible that the new version doesn't work at all.   I'd
love to figure out a way to make this version work, because it does
address some of the concerns that Sam pointed out.  However, if there
is degraded performance (which may be an understatement) then I won't
release version 0.04 like this.

Again, thanks for the highly valuable feedback!

-DeWitt





Re: Implementing security in CGI

2000-04-20 Thread DeWitt Clinton

Hi, 
 
Interesting thread and interesting question.   
 
It makes sense to start with the requirements for what it means to 
implement those secure features.  My requirements have an obvious 
e-commerce bias, and should probably be heavily reviewed by anyone 
thinking of using this design for online banking or government work. 

First, I'd like to introduce the notion of a secure session.  Secure
sessions have some subtle differences from the traditional notion of
the session, and I'll point those out when they appear.

The secure session has the following properties:

*) The user is able to initiate a secure session by providing proper
credentials (i.e., a username and password pair) via a login process.

*) The user is able to terminate the secure session via a logout
process.

*) Secure sessions must be able to time out automatically.

*) Secure sessions must *never* transmit sensitive data (such as the
password) over insecure channels.

*) The secure session, while it requires the use of a secure protocol
(such as HTTPS), should not require the use of cookies.  Cookies,
however, can be employed to extend the functionality of the system.  

Additionally, I feel that one of the essential requirements to any
enterprise quality, highly scalable, and fault-tolerant system is the
ability to store session state on a tier other than the front-end.
This usually means storing state in a shared database, but there are
other options.

A very effective method of meeting the requirements above is to use a
two token architecture.  The first token is a user identifier, which
can be passed over insecure channels.  This user identification token
can be used to restore state that isn't particularly sensitive, such
as a shopping cart or user preferences.  The second token is a secure
identifier, which is never passed over insecure channels.  The secure
identifier is required to access sensitive data, such as credit card
data. (It is possible to create an architecture that uses *only* the
secure token, but there are significant benefits in terms of
flexibility afforded by using two tokens, so I won't go into the one
token model here.)

The fundamental goal of the secure token is to a) keep it safe from
prying eyes, and b) use it is a requirement for accessing secure data.

Tokens can be passed in one of two ways.  First, the token can be
passed as part of the URL.  Second, the token can be passed in a
cookie.  I've found it very useful to create an initialization
procedure inside applications that check both places for these tokens,
and abstract the particular mechanism used to pass them.  Note,
however, that is important to remember how the token was passed -- if
it came from a cookie, it is cosmetic to not pass it around in the
URL.  I also recommend creating a library used in your templates that
has a function to build URLs that automatically append the tokens if
necessary.  Since it is critical that secure tokens are never passed
in the clear, it really helps to have that function, because it can
prevent the secure token from being sent via an insecure page in a URL
(i.e., only setting secure tokens in URLs that contain "https").

The user identification token is typically used to restore state from
the database.  It is created whenever a page is viewed and the client 
has either not provided one (via the URL or a cookie), or the token
they provide doesn't exist in the database.  This token should then be 
set in both the URL and in a cookie.  On the subsequent page request,
if the user identification token is found in a cookie, it is probably
safe to stop putting it in the URL.  However, if the token is found
only in the URL (probably meaning the client does not use cookies) the 
server should not try writing the cookie again.  

The advantage of the cookie is such that the user can insecurely
identify themselves across browser sessions.  This is perfectly
acceptable when used to restore some insensitive state data (such as a
name or shopping cart).  However, the system still functions without
the cookie.  The user can now login to associate their client with
particular stored user information.

The login process is as follows:

1) The client connects to a secure page that presents a form
requesting a username and password.  (There has been much discussion
about whether a HTTP form that POSTS to a secure page will encrypt the 
posted data.  There is too much ambiguity here -- I recommend using
HTTPS on the login form as well.)  The ACTION of the form must also
be a secure page.

2) On receiving the username and password, the server compares an
encrypted version of the password with the stored encrypted password
associated with that customer.  If the passwords do not match, the
user is returned to the login page. (For highly secure sites, storing
a count of the number of failed logins may be desirable.)

3) The server compares the client supplied user identification token
with the one associated with that username in 

Re: Caching DB queries amongst multiple httpd child processes

2000-02-03 Thread DeWitt Clinton

On Thu, 3 Feb 2000, Peter Skipworth wrote:

 Does anyone have any experience in using IPC shared memory or similar in
 caching data amongst multiple httpd daemons ?

Well, I just released IPC::Cache version 0.02 to CPAN.  Two major caveats,
however -- first, it is not currently in production on our site
(www.eziba.com) and won't be for another week or two.  Second, it relies
on IPC::ShareLite as a backend.  

However, it does offer a very straightforward interface, and attempts to
manage the memory segments and clean up after them.  We are running Linux
2.2 on the front end, btw, and that may account for why ShareLite seems
rather reliable for us.

I'd love it if you could download it and let me know it works for you.  
We are planning on maintaining this, and would appreciate any feedback you
can give about how to improve it.

You can get IPC::Cache here:

   http://www.cpan.org/modules/by-module/IPC/IPC-Cache-0.02.tar.gz   

-DeWitt



Version 0.02 of IPC::Cache available

2000-02-01 Thread DeWitt Clinton

Hi,

[I sent this out a while back, but it never made it through.  Odd.]

Based on some great feedback from this list, I was able to remove the
dependency on IPC::Shareable.  Not that I dislike IPC::Shareable -- to the
contrary, it does what it does rather well -- but I was really looking for
something more lightweight.

I discovered Maurice Aubrey's excellent IPC::ShareLite module, which
exactly filled the role I needed.  

IPC::Cache now relies on the efficient ShareLite shared memory routines,
but the interface did not change at all.

You can find it at CPAN now:

   http://www.cpan.org/modules/by-module/IPC/IPC-Cache-0.02.tar.gz

Thanks for the feedback!

-DeWitt






IPC::Cache version 0.01 available

2000-01-30 Thread DeWitt Clinton

Hi,

A week or two ago, in the squid performance thread, I mentioned that I was
looking for ways to eliminate squid from our production servers.  I noted
that we are using squid to save an expensive trip to the database to
retrieve mostly static files.  At that time I said that I planned to write
a simple cache using shared memory that could be used with mod_perl to
keep data around on the first tier.  This weekend I finally had an
opportunity to do just that.

IPC::Cache is rather simple, considering that all the hard work is being
done in the IPC::Shareable the Storable modules.  I just added some basic
namespace semantics and the ability to have data expire after a period of
time.  I'm planning on submitting it to CPAN, but I think it makes sense
to make it available on this list before I do.

I'd appreciate it if anyone who tries it out could send me feedback about
whether it seems to do the trick.  

For now, you can get it at:

   http://unto.net/archives/IPC-Cache-0.01.tar.gz

Apologies if that site is a little slow -- I'll move it over to our real
servers and to CPAN in the near future.

Thanks!

-DeWitt





Re: numerous segv's, bus error

2000-01-26 Thread DeWitt Clinton


[apologies if people get this twice -- it seemed to bounce the first time]

 We've been getting many segv's and bus errors on apache processes since
 adding modperl.  Env: Apache 1.3.11 (though we had the same thing
 under 1.3.9.), Perl 5.005003, modperl 1.21, hpux 11.00.

If you compare this to my earlier post this week, I believe we are having
the same problem. 

I've tried rolling back to 1.3.9 as well and had the same problem.  I'm
running Linux 2.3.39 and 2.2.15, btw -- it's interesting to note that you
are having this problem on another system.  I've tried every possible
configuration of static/dynamic I can think of, too.  Setting
PerlFreshRestart to "off" didn't seem to do the trick for me either.  I
tried using the latest cvs snapshot of mod_perl with no luck.

I agree -- this is a problem with the cleanup.  In a way I'm glad it's not
just me, although I'm really sorry you are being bit by this too!

Hopefully someone will have an idea.

-DeWitt




Re: squid performance

2000-01-16 Thread DeWitt Clinton

On Fri, 14 Jan 2000 [EMAIL PROTECTED] wrote:

 Hi, I am switching my modperl site to squid in httpd acclerator mode
 and everything works as advertised, but was very surprised to find
 squid 10x slower than apache on a cached 6k gif as measured by
 apache bench... 100 requests/second vs almost 1000 for apache.

I've experienced the same results with accelerating small static files.  
However, we still use squid on our site (www.eziba.com) because our httpd
processes are so heavy, and to save a trip to the database for documents
that are cachable, such as product images.  We've found that the net
performance gain to be noticeable.  However, if all you are serving is
static files that live on the same tier as the apache server, you'll
probably be better of without squid.

On that topic, is there an alternative to squid?  We are using it
exclusively as an accelerator, and don't need 90% of it's admittedly
impressive functionality.  Is there anything designed exclusively for this
purpose?  Perhaps a modperl module that implements caching of files based
on Expires headers?

-DeWitt







Re: squid performance

2000-01-16 Thread DeWitt Clinton

On Sun, 16 Jan 2000, Steven Lembark wrote:

 given you have the core to support it...  try using libmm and
 a tied hash to just stash the stuff until it's asked for.  

Actually, we are currently developing a generic object caching interface
that will support a TTL based on expiration dates, and implementing this
in both Java and perl.  Since the perl version will be used most in a
multi-process environment (like apache/modperl) we'll back-end this on
mmap.  The java version will be used in both our middle tier (a
single-process threaded server) and in a servlet implementation of our
front-end, which also runs in a single process.

Right now, I do cache data in each httpd process, but since I haven't
implemented a TTL, I rely on the apache process dying naturally to clean
up the data.  Obviously, this creates *huge* httpd processes (50+ MB) with
non-shared but redundant data, and isn't particularly predictable.  Our
site is a little different that most ecommerce sites, because we have a
very low depth of inventory, and have to keep all the state in the
database, so that customers will rarely see out-of-date product data.

Ultimately, I hope to open up our entire source tree under the GPL anyway,
but I'll try to get the generic caching code out there earlier, if
possible.  I'll keep this list posted.  Unless someone has already done
this, that is.

-DeWitt









RE: Potential bug in HTML::Embperl 1.2.0

1999-12-09 Thread DeWitt Clinton

On Thu, 9 Dec 1999, Gerald Richter wrote:

 The problem is, that Embperl must know, that you want to run in a Safe
 compartement. Just add
 
 'options' = HTML::Embperl::optSafeNamespace
 
 to the Excute parameters and it should work.

Right.  That makes sense, and it is definitely a good idea to run it with
that option, but unfortunately it doesn't address the issue demonstrated
by the example code.   If anyone has a second, I'd like to see someone
else run the script and see if I'm not going crazy and imagining things.

Thanks,

-DeWitt





RE: Potential bug in HTML::Embperl 1.2.0

1999-12-09 Thread DeWitt Clinton

On Thu, 9 Dec 1999, Gerald Richter wrote:

 Rereading your first mail, seems to me that there is an additional problem
 with Embperl's cleanup. This has nothing todo with Safe namespaces. Embperl
 will per default tidyup your namespace and therefor execute something like
 $vars = undef at the end of the first request.
 
 Try adding the additional parameter
 
 'cleanup' = -1
 
 which will disable cleanup at all, add $CLEANUP{'vars'} = 0 inside your
 template, which should disable the cleanup only for that variable.

That will work for this trivial example, but we have 107 templates on our
site, each with 10 to 100 variables, and changing them all would be
difficult, to say the least.  Worse, if we missed one, we'd have those
subtle bugs that are only occasionally apparent, but can cause real
damage.  Imagine if your credit card number was inadvertently passed to
another customer in the template because we forgot to undef a variable?  

I think the right approach is to probably track down the cause of the
variables not being undefined in the first place.  I suppose I'll start
hunting down the reason why this is occurring.  Wish me luck.  Any
pointers as to where to focus are appreciated, of course.  :)

Thanks,

-DeWitt








RE: Potential bug in HTML::Embperl 1.2.0

1999-12-09 Thread DeWitt Clinton

On Thu, 9 Dec 1999, Gerald Richter wrote:

 If I had understand you right, the first call via execute to your template
 works and the second not, right?
 
 That's not a bug! Embperl cleans up all your global variables inside your
 page after every request. $vars in your template is such a global variable,
 so Embperl undef it and therfore the content gets lost and it won't work the
 second time.

Oddly, that's not quite what is happening.  The content isn't lost
altogether, else the simple join() I include wouldn't work either.

Moreover, even if I re-initialize the data passed into my printTemplate()
routine, the second call is still using some of the old data.  

Note that this issue is only appearing on array data, and only in the
foreach command.  

When I call the template containing a [$ foreach $] for the second time,
every value in that array *in the foreach loop* is returned as the last
value of the array, as called the first time around.

Here's a little test.  Try putting this in a template, and Executing on it
twice by the same process.  No input data, no parameters, just the
template on it's own.

  [- @array = (1, 2, 3) -]
  [+ join(",", @array) +]
  [$ foreach $arrayElement (@array) $]
  [+ $arrayElement +]
  [$ endforeach $]

That's it.  The first time it will print (I stripped out the newlines):

1,2,3
1
2
3

The second time it will print:

1,2,3
3
3
3


Apparently, this has nothing to do with Safe() or anything else.  Just
that the value given to $arrayElement in the [$ foreach $] doesn't seem to
be correct on subsequent executions, that's all.

Sorry to be sticking with this one, but I want to hear from just one
person that I'm not losing my mind.  I really do appreciate the help,
though.  :)

-DeWitt





RE: Potential bug in HTML::Embperl 1.2.0

1999-12-09 Thread DeWitt Clinton

On Thu, 9 Dec 1999, Gerald Richter wrote:

 So the solution to your problem should be, to use a fixed package name for
 all calls to the _same_ page.

Brilliant!  This does address exactly the problem I raised, and makes
perfect sense.

I had to do one absolutely wacky thing, though.  Because I call Execute
from within templates (indirectly, actually, through our own Template
module that wraps Embperl), I needed to extend your idea a little.

Essentially, I needed to use a unique Safe namespace for each request, but
somehow have it carry over between the included Execute calls.  So, I
share the Safe compartment's root across calls to Execute.  But simply
doing that means that when the same file is included and executed by
different templates then the package names will mismatch and data won't be
shared across included files correctly.  I solved this by creating a
virtual filename that is unique on a per file *and* a per "root"
namespace, but is common across requests, to allow for caching.

Does this make any sense to anyone?  Probably not.  :)  I'll go away now,
but thanks everyone, especially Gerald, for the help!  

-DeWitt