[OT] RE: Getting a Cache::SharedMemoryCache started
Hi all, I eventually found the problem with getting shm to work. I had the Openwall Linux kernel patch enabled with the "Destroy shared memory segments not in use" option set, which most Perl IPC functions seem to not like. Apache and PostgreSQL seem to work fine with this patch in place, however. Removed the option, and IPC::ShareLite compiles and tests fine so my SharedMemoryCache now works as exampled. Thanks Matt
RE: Getting a Cache::SharedMemoryCache started
Thanks for the pointers, unfortunately I've got a problem with the Shared > cache in that I need IPC::ShareLite, no problem, except it won't test ok, > I get: > > PERL_DL_NONLAZY=1 /usr/bin/perl -Iblib/arch -Iblib/lib > -I/usr/lib/perl5/i386-linux -I/usr/lib/perl5 test.pl > 1..8 > ok 1 > ok 2 > IPC::ShareLite store() error: Identifier removed at test.pl line 33 It took me forever, but I finally figured out HOW to deal w/ share memory! As in, what is the status, what can I change, how can I make garbage go away? On Linux (and Solaris, probably others) there is a program called ipcs(8). This will give you a listing of all shared memory segments, message queues, and semaphore arrays. An example (Linux): -- Shared Memory Segments keyshmid owner perms bytes nattchstatus 0x00280267 1 root 644 1048576 0 -- Semaphore Arrays key semid owner perms nsems status 0x00280269 0 root 666 14 -- Message Queues key msqid owner perms used-bytes messages NOW, if I wanted to make these go away (AFTER checking that nothing critical is actually USING these segments!), I would use the companion program ipcrm(8). ipcrm shm 1 ipcrm sem 0 the shm 1 line corresponds to the lines above that show shmid 1 under shared memory. Likewise, the sem 0 is for the semaphore w/ semid 0. See the manpages! Anyway, whenever I've seen that message, it means that something is barfing in shared memory. Usually, something got left behind instead of being cleaned up. A quick removal usually takes care of it. BUT!!! Don't do something silly (and dangerous) like deleting ORACLE's shared memory segments, while it's running. Fortunately, ipcs(8) shows the owners of shared memory segments, so this should be reasonable simple to identify. Hope this helps! (is this OT? :-) L8r, Rob
Re: Getting a Cache::SharedMemoryCache started
On Tue, 27 Mar 2001, DeWitt Clinton wrote: > Now, this implies that a module, such as the Cache::MemoryCache, which > currently shares data for a particular process by creating a global > hash, would only share data for one instance of the Perl interpreter > associated with that thread. If I wanted to extend the cache to share > data between multiple threads, I would create what Doug calls a > "solar" variable to hold that hash, correct? I don't think so. A solar variable is used for sharing data between threads with the same parent interpreter, and only for that, but your Cache::* modules will work with any combination of threads and processes since the data is really stored outside of the interpreter in shared memory or files. Even if you use globals, they are only global to the current thread. - Perrin
Re: Getting a Cache::SharedMemoryCache started
On Tue, Mar 27, 2001 at 11:57:26PM +0100, Matt wrote: > Thanks for the pointers, unfortunately I've got a problem with the > Shared cache in that I need IPC::ShareLite, no problem, except it > won't test ok, I'm not sure what the IPC::ShareLite problem is, maybe Maurice Aubrey does... In any case, consider switching to Cache::FileCache instead. Preliminary benchmarks show that FileCache is just as fast as the SharedMemoryCache, at least on my Linux 2.4 system. And it is significantly faster on large caches (either a large number of keys or large values). -DeWitt
Re: Getting a Cache::SharedMemoryCache started
Thanks for the pointers, unfortunately I've got a problem with the Shared cache in that I need IPC::ShareLite, no problem, except it won't test ok, I get: PERL_DL_NONLAZY=1 /usr/bin/perl -Iblib/arch -Iblib/lib -I/usr/lib/perl5/i386-linux -I/usr/lib/perl5 test.pl 1..8 ok 1 ok 2 IPC::ShareLite store() error: Identifier removed at test.pl line 33 I know this isn't strictly "mod_"perl, but seems as people are using the Cache::Cache package, maybe someone knows the problem. I'm running Linux 2.2.17 smp, Perl 5.6.0 from Slackware 7.1 btw Thanks Matt
Re: Getting a Cache::SharedMemoryCache started
On Tue, Mar 27, 2001 at 02:38:34PM -0800, Perrin Harkins wrote: > On Tue, 27 Mar 2001, DeWitt Clinton wrote: > > > I imagine that nearly most Perl libraries are not thread safe, of > > course. But code that will be used in mod_perl environments needs > > to be, right? > > You can read all about it here: > http://www.apache.org/~dougm/modperl_2.0.html > > The gist is that you probably don't need to change any Perl code, but some > XS modules on CPAN may have to change. Great pointer, Perrin. So, according to Doug: The Perl ``ithreads'' implementation ensures that Perl code is thread safe, at least with respect to the Apache threads in which it is running. However, it does not ensure that extensions which call into third-party C/C++ libraries are thread safe. In the case of non-threadsafe extensions, if it is not possible to fix those routines, care will need to be taken to serialize calls into such functions (either at the xs or Perl level). Another issue is that ``global'' variables are only global to the interpreter in which they are created. Some research has been done on the concept of solar variables which are global across all interpreter instances. It has not been decided if this feature would best fit built into the Perl core or as an extension, but fear not, the feature will be provided in one form or another. Now, this implies that a module, such as the Cache::MemoryCache, which currently shares data for a particular process by creating a global hash, would only share data for one instance of the Perl interpreter associated with that thread. If I wanted to extend the cache to share data between multiple threads, I would create what Doug calls a "solar" variable to hold that hash, correct? Neat! Cheers, -DeWitt
Re: Getting a Cache::SharedMemoryCache started
On Tue, 27 Mar 2001, DeWitt Clinton wrote: > Which reminds me of something. These cache objects are not currently > thread safe. When should I start expecting multi-threaded > apache/mod_perl to become mainstream enough to warrant an overhaul of > the code? I imagine that nearly most Perl libraries are not thread > safe, of course. But code that will be used in mod_perl environments > needs to be, right? You can read all about it here: http://www.apache.org/~dougm/modperl_2.0.html The gist is that you probably don't need to change any Perl code, but some XS modules on CPAN may have to change. - Perrin
Re: Getting a Cache::SharedMemoryCache started
On Tue, Mar 27, 2001 at 05:19:15PM -0500, Pierre Phaneuf wrote: > DeWitt Clinton wrote: > > > The other question is whether or not to share the cache instance > > itself globally. Technically, this is up to you. Personally I > > wouldn't bother, considering the overhead of instantiating the > > cache is so low that I would rather keep it local to the handler > > (as I did above). I tend to only share data globally in mod_perl > > handlers very judiciously, and even then I wrap the data in an > > object that provides accessor methods. For an example of an > > object that wraps shared data, check out Cache::MemoryCache, which > > simply caches data inside the process. > > Wouldn't this cause the cache to be torn down and rebuilt for every > request??? The cache instance itself would be, but the data behind it (a global hash in the case of the MemoryCache, and a IPC::ShareLite store for the SharedMemoryCache ) would stay around. Since the instance isn't that heavy, I feel better instantiating them fresh every request. If you need optimal performance, however, you may want to keep the instance itself global. Which reminds me of something. These cache objects are not currently thread safe. When should I start expecting multi-threaded apache/mod_perl to become mainstream enough to warrant an overhaul of the code? I imagine that nearly most Perl libraries are not thread safe, of course. But code that will be used in mod_perl environments needs to be, right? -DeWitt
Re: Getting a Cache::SharedMemoryCache started
DeWitt Clinton wrote: > The other question is whether or not to share the cache instance > itself globally. Technically, this is up to you. Personally I > wouldn't bother, considering the overhead of instantiating the cache > is so low that I would rather keep it local to the handler (as I did > above). I tend to only share data globally in mod_perl handlers very > judiciously, and even then I wrap the data in an object that provides > accessor methods. For an example of an object that wraps shared data, > check out Cache::MemoryCache, which simply caches data inside the > process. Wouldn't this cause the cache to be torn down and rebuilt for every request??? -- "The camel has not evolved to smell good. Neither has Perl." -- Larry Wall
Re: Getting a Cache::SharedMemoryCache started
On Tue, Mar 27, 2001 at 09:50:33PM +0100, Matt wrote:
> I want to cache some values in my handlers so they don't keep having
> to look stuff up, and so I figured a Cache::SharedMemoryCache would
> be the way to go.
It all depends on what you want to cache and why. If you wanted to
save a trip to the database on each request, a really simple mod_perl
handler could look something like this:
package MyApplication;
use strict;
use Apache::Request;
use Cache::SharedMemoryCache;
sub handler
{
my ( $request );
my $cache = new Cache::SharedMemoryCache( );
my $data = $cache->get( 'data' );
if ( not defined $data )
{
$data = expensive_database_request( );
$cache->set( 'data', $data );
}
$request->content_type( 'text/plain' );
$request->send_http_header( );
$request->print( "data: $data" );
}
The first time this handler is run, the data will not be found in the
cache. The second time, however, the data will be returned from the
cache, saving a trip to the database.
The other question is whether or not to share the cache instance
itself globally. Technically, this is up to you. Personally I
wouldn't bother, considering the overhead of instantiating the cache
is so low that I would rather keep it local to the handler (as I did
above). I tend to only share data globally in mod_perl handlers very
judiciously, and even then I wrap the data in an object that provides
accessor methods. For an example of an object that wraps shared data,
check out Cache::MemoryCache, which simply caches data inside the
process.
Hope this helps a bit. Cheers,
-DeWitt
