1+ for smarter cache
On 12/2/05, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote:
>
> > > So, are you interested in PHP realization with main idea explained?
> > Yes
>
> Ok, hope it will be interesting:
>
> In my CMS there is (there was?) such an abstraction: node. Node - it's
> everything - it's a news, article or any other content unit.
>
> Node has all CRUD operations through the functions: get, add, edit,
> delete.
>
> There is also such a module - cache: it track two things - dependence
> and time. Dependence, in fact, that's actually, one file, which is
> `touch`ed automaticly every time, any non-read node operations is
> perfomed (it is just a cache_update_dependence() call from node
> functions).
>
> Then, any page, which may want to use cache, has the following lines
> (it can be put in one place/class, in a normal designed,
> object-oriented, enviroment):
> ============
> //Predefine cache data
> $CacheID = basename($_SERVER['SCRIPT_NAME']);
> $CurrTs = time(); $CacheExpireTs = mktime(0, 0, 1, date('m', $CurrTs),
> (date('d', $CurrTs) + 1), date('Y', $CurrTs)); //first second of the
> next day
> $CacheDependencies = array('taxonomy_data', 'taxonomy_function',
> 'taxonomy_hierarchy', 'taxonomy_nodes', 'source_data', 'source_nodes',
> 'nodes');} //show nid
>
> #Check cache
> if ($CachedContent = cache_get($CacheID, $ScriptArgs,
> $CacheDependencies)) {echo $CachedContent;} //Return cached page
> content
> else { //Get and construct page content
> //content generation here
> }//if($CachedContent
> ============
>
> Dependencies - it's just an empty files, with the same names, as an
> existen tables.
>
> What cache do behind the scene:
> - it check, cache validness: if table was modified (and touched by the
> node functions) recently or if cache should be deleted by time-out - it
> is droped and Null is returned; otherwise - cache content, which is
> readen from the file, which name is an md5 sum from the script
> arguments (special function for that).
> - it check cache existens;
> - it also empty cache, by cron, by calling cache_clean() function,
> which delete all cache object, starting from the oldest one, untill
> predefined (in settings) size will be reached (so we will always have
> the most recent XXX Mbytes of cache).
>
> cache.inc source code below.
> Please, do not blame me too much - yes, this is not an object oriented
> code, and it is not perfect, by I just want, that the idea of such a
> cache will not dyu with the code (when it was the time, when I need a
> cache, I do not find any cache sollutions based on the DB changes
> track, so I have to invent it on my own).
>
> cache.inc (comments were written once and was not modified with the
> code updates)
> ==================================================================
> <?php
> /* Work algorithm: track cache validness using expiration time & (db)
> dependencies
> Rules:
> - cache is decided to be dirty if cache timestamp is invalid _or_ if
> one of the dependencies is out of the date
> - correct dependencies is a caller problem and should be specified at
> the cache_get function
> - database tables (dependencies) update is core functions problem
> - the only thing cache take on itself - it's section handle (including
> table (dep) completion: 'taxonomy_data' dep will be completed to the
> 'SECTION._taxonomy_data' )
>
> Some details: expiration is computed using file modification time
> (cache expire timestamp == file modification time), so, cache creation
> time is not available.
> Algorithm:
> - two stuff is tracked: cache expire time & dependence update time
> - cache has three dates: creation, last acces and expire, that is
> inode change time of file, last file access time and file modification
> accordingly
> - dependencies tracked in a following algorithm: if dep. access time
> is more than cache creation, than cache is obsolete, and it's dropped
> */
> //ToDo:
> // - tune tables updates track
> // - implement function cache
>
> ini_set('ignore_user_abort', 1); //partly loaded pages is useless and
> should never be in cache (suppose, that if script is failed, than it
> will never store cache, cause it's one of the last deal)
>
> //Convert function args to the fs storable hash (md5 used);
> func_get_args - for function and script_args for script should be used;
> valid for no more than a two level array.
> function args_hash($Args = NULL) {
> if (!is_array($Args) && !is_string($Args)) {global $ErrMsg;
> func_error_handler(array(__FUNCTION__, func_get_args(),
> $ErrMsg['InvalidArgs'])); return FALSE;}
>
> if (is_string($Args)) {$ArgsString = $Args;} //just a line
> else if (is_array($Args)) {
> $ArgsString = '';
> while(list($Key, $Val) = each($Args)) { //no more than a two
> level
> array reconstructor
> if (!is_array($Val)) {$ArgsString .= $Key.$Val;}
> else if (is_array($Val)) {while(list($SubKey,
> $SubVal) = each($Val))
> {$ArgsString .= $SubKey.$SubVal;}}
> }//while
> } else {
> func_error_handler(array(__FUNCTION__, func_get_args(),
> $ErrMsg['InvalidArgs'])); return FALSE;
> }//if(is_string
>
> return md5($ArgsString);
> }//function
>
>
> //Just check enviroment to the errors; return FALSE & log event, that
> may "polute" cache
> function cache_env_dirty() {
>
> if ($ErrorMessage = error_description_read()) {
> debug_log('Cache', __FUNCTION__.": cache interrupted because
> of the
> unhandled error.");
> $GLOBALS['CashSwitch'] = 0;
> return $ErrorMessage;
> } else {
> return FALSE;
> } //if($ErrorMessage
>
> }//function
>
>
> //Should be called by any function, that insert/update any info to the
> database; section append is optional, cause other db function know full
> table name
> function cache_update_dependence($Dependence = NULL, $SectionAppend =
> 0) {
> global $CacheStatDir;
>
> if ($SectionAppend) {if (!defined('SECTION')) {$Section =
> 'UnSectioned';} else {$Section = strtolower(SECTION);}}
>
> if (!is_string($Dependence) || (($SectionAppend != 0) &&
> ($SectionAppend != 1))) {global $ErrMsg;
> func_error_handler(array(__FUNCTION__, func_get_args(),
> $ErrMsg['InvalidArgs'])); return FALSE;}
>
> if (!is_dir($CacheStatDir)) { //dependence dir should exist allready
> global $ErrMsg;
> func_error_handler(array(__FUNCTION__, func_get_args(),
> "$CacheStatDir {$ErrMsg['UnexistenDirectory']}.
> {$ErrMsg['CachePreventCorruptionAndOff']}"));
> $GLOBALS['CashSwitch'] = 0;
> return FALSE;
> }//if(!is_dir
>
> $DependenceFullName =
> ($SectionAppend)?"{$Section}_{$Dependence}":$Dependence;
> $DependenceFullPath = "$CacheStatDir/$DependenceFullName";
>
> $CurrTime = time();
> $Result = touch($DependenceFullPath, $CurrTime, $CurrTime);
>
> if ($Result) {
> debug_log('Cache', __FUNCTION__.": '$Dependence' dependence
> updated;
> access & modification time is set to ".timestamp2date($CurrTime, 1, '
> ')." ($CurrTime)");
> return TRUE;
> } else {
> global $ErrMsg;
> func_error_handler(array(__FUNCTION__, func_get_args(),
> "$Dependence
> {$ErrMsg['DepUpdateFail']}.
> {$ErrMsg['CachePreventCorruptionAndOff']}"));
> $GLOBALS['CashSwitch'] = 0;
> return FALSE;
> }//if($Result
>
> }//function
>
>
> //===>> Function store_page_cache <<===
> function cache_store($Item = NULL, $Args = NULL, $Content = NULL,
> $Expire = NULL) {
> global $CacheFilesRoot, $ErrMsg;
> settype($Expire, 'int');
>
> if (!defined('SECTION')) {$Section = 'UnSectioned';} else {$Section =
> strtolower(SECTION);}
>
> //Arguments check
> if (!is_string($Item) || (!is_string($Args) && !is_array($Args)) ||
> !is_string($Content) || !is_int($Expire))
> {func_error_handler(array(__FUNCTION__, func_get_args(),
> $ErrMsg['InvalidArgs'])); return FALSE;}
>
> //Check unhandled errors
> if ($UnhandledError = cache_env_dirty())
> {func_error_handler(array(__FUNCTION__, func_get_args(),
> "[i]'$UnhandledError'[/i].
> {$ErrMsg['CachePreventCorruptionAndOff']}.")); return FALSE;}
>
> //Global cache switch check
> if (!$GLOBALS['CashSwitch']) {return FALSE;}
>
> $ArgsHash = args_hash($Args);
> $ItemDir = "$CacheFilesRoot/$Section/$Item";
> $TargetFile = "$ItemDir/$ArgsHash";
>
> //Destination dir check
> if (!is_dir($ItemDir)) {if (!mk_r_dir($ItemDir))
> {func_error_handler(array(__FUNCTION__, func_get_args(), "$ItemDir
> {$ErrMsg['DirCantBeCreated']}")); return FALSE;}}
>
> if (file_store($TargetFile, $Content) && touch($TargetFile, $Expire))
> {
> debug_log('Cache', __FUNCTION__."($Item, $ArgsHash, content
> (length):
> ".strlen($Content).", expire: ".timestamp2date($Expire, 1, ' ')."
> ($Expire)) - done.");
> return TRUE; //content stored, cach marked - all is Ok
> } else {
> func_error_handler(array(__FUNCTION__, func_get_args(),
> "$TargetFile
> {$ErrMsg['FileCantBeCreatedUpdated']}")); return FALSE;
> }//if(file_store
>
> }//function
>
>
> //Return cache
> function cache_get($Item = NULL, $Args = NULL, $Dependencies = array())
> {
> global $CacheFilesRoot, $CacheStatDir;
>
> $FuncArgs = func_get_args(); debug_log('Cache',
> __FUNCTION__."(".array2string($FuncArgs).")");
>
> if (!defined('SECTION')) {$Section = 'UnSectioned';} else {$Section =
> strtolower(SECTION);}
>
> if (!is_string($Item) || (!is_string($Args) && !is_array($Args)) ||
> !is_array($Dependencies)) {global $ErrMsg;
> func_error_handler(array(__FUNCTION__, func_get_args(),
> $ErrMsg['InvalidArgs'])); return FALSE;}
>
> if (!$GLOBALS['CashSwitch']) {return FALSE;} //cache is switched off
>
> $ArgsHash = args_hash($Args);
> $DependenciesAmount = count($Dependencies);
> $Dir = $CacheFilesRoot."/".$Section."/".$Item;
> $CacheFile = $Dir."/".$ArgsHash;
>
> if (!is_dir($Dir) || !is_file($CacheFile)) {debug_log('Cache',
> __FUNCTION__."($Item, $ArgsHash, $DependenciesAmount) there is no cache
> file: '$CacheFile'"); return FALSE;}
>
> debug_log('Cache', __FUNCTION__."('$Item', '$ArgsHash',
> '$DependenciesAmount') - file '$CacheFile'");
>
> $CheckingCurrTimestamp = time();
> $CacheExpirationTs = filemtime($CacheFile);
>
> if ($CacheExpirationTs <= $CheckingCurrTimestamp) {
> debug_log('Cache', __FUNCTION__."($Item, $ArgsHash,
> $DependenciesAmount) cache expired at
> ".timestamp2date($CacheExpirationTs, 1, ' '));
> cache_drop($Item, $ArgsHash);
> return FALSE;
> } else if ($DependenciesAmount) {
>
> if (!is_dir($CacheStatDir)) { //dependence dir should exist
> allready
> func_error_handler(array(__FUNCTION__,
> func_get_args(),
> "$CacheStatDir {$ErrMsg['UnexistenDirectory']}.
> {$ErrMsg['CachePreventCorruptionAndOff']}"));
> $GLOBALS['CashSwitch'] = 0;
> return FALSE;
> }//if(!is_dir
>
> //Return FALSE, if any of the dep is updated
> for ($i = 0; $i < $DependenciesAmount; $i++) {
> $CurrDep =
> "$CacheStatDir/{$Section}_{$Dependencies[$i]}";
> $CacheCreationTs = filectime($CacheFile);
> $CurrDepUpdateTs = fileatime($CurrDep);
> if ($CacheCreationTs <= $CurrDepUpdateTs) {
> debug_log('Cache', __FUNCTION__."($Item,
> $ArgsHash,
> $DependenciesAmount) $CurrDep dependence expired at
> ".timestamp2date($CurrDepUpdateTs, 1, ' '));
> cache_drop($Item, $ArgsHash);
> return FALSE;
> }//if(filemtime
> }//for
>
> }//if($DependenciesAmount
>
> //If all about code is passed, then expire & dependencies is Ok
> debug_log('Cache', __FUNCTION__."($Item, $ArgsHash,
> $DependenciesAmount) returning cache content");
> return file_read($CacheFile);
> }//function
>
> //===>> Function drop_page_cache <<===
> function cache_drop($Item = NULL, $Args = NULL) {
> global $CacheFilesRoot;
>
> if (!defined('SECTION')) {$Section = 'UnSectioned';} else {$Section =
> strtolower(SECTION);}
>
> if (!is_string($Item) || (!is_string($Args) && !is_array($Args)))
> {global $ErrMsg; func_error_handler(array(__FUNCTION__,
> func_get_args(), $ErrMsg['InvalidArgs'])); return FALSE;}
>
> $ArgsHash = args_hash($Args);
>
> $Dir = $CacheFilesRoot."/".$Section."/".$Item;
> $CacheFile = $Dir."/".$ArgsHash;
>
> debug_log('Cache', __FUNCTION__."($Item, $ArgsHash) dropping cache");
> return file_delete($CacheFile);
> }//function
>
> //Do not forget to clear vars, before reusing
> function dir_size_atime_info($DirName = NULL) {
> if (!is_string($DirName)) {global $ErrMsg;
> func_error_handler(array(__FUNCTION__, func_get_args(),
> $ErrMsg['InvalidArgs'])); return FALSE;}
>
> static $FullSize = 0, $Amount = 0, $Id = 0, $Files = array();
>
> //static vars has to be cleaned by hand
> if ($DirName == 'CLEARVARS') {
> $FullSize = 0; $Amount = 0; $Id = 0; $Files = array();
> // debug_log('Cache', __FUNCTION__.": variables cleared
> [$FullSize,
> $Amount, $Id, ".str_replace("\n", '', print_r($Files, 1))."]");
> return TRUE;
> }//if($DirName
>
> if (!is_dir($DirName)) {return array('amount' => $Amount, 'size' =>
> $FullSize, 'data' => $Files);}
>
> if (!$DirHandle = opendir($DirName)) {global $ErrMsg;
> func_error_handler(array(__FUNCTION__, func_get_args(), "$DirName
> {$ErrMsg['FileCantBeOpened']}")); return FALSE;}
> while (FALSE != ($File = readdir($DirHandle))) {
> $FullFilePath = $DirName.'/'.$File;
> if (($File != '.') && ($File != '..') &&
> !is_link($FullFilePath)) {
> if (is_dir($FullFilePath)) {
> dir_size_atime_info($FullFilePath);
> } else if(is_file($FullFilePath)) {
> $FileStatInfo = stat($FullFilePath);
> $Files['file'][$Id] = $FullFilePath; //file
> $Files['size'][$Id] = $FileStatInfo['size'];
> //size
> $Files['atime'][$Id] =
> $FileStatInfo['atime']; //access time
> $Id++; $Amount++; $FullSize +=
> $FileStatInfo['size'];
> // debug_log('Cache', __FUNCTION__." full size:
> $FullSize");
> unset($FileStatInfo);
> }//if
> }//if
> } //while (skip '.', '..' and links)
> closedir($DirHandle);
>
> return array('amount' => $Amount, 'size' => $FullSize, 'data' =>
> $Files);
> }//function
>
> function cache_clean($SectionToClean = NULL) {
> if (!is_string($SectionToClean)) {global $ErrMsg;
> func_error_handler(array(__FUNCTION__, func_get_args(),
> $ErrMsg['InvalidArgs'])); return FALSE;}
> global $CacheSize, $CacheFilesRoot;
>
> $FullPath = $CacheFilesRoot.'/'.$SectionToClean;
>
> $NodeLimit = mbytes_to_bytes($CacheSize[$SectionToClean]);
> $NodeCacheInfo = dir_size_atime_info($FullPath);
> // debug_log('Cache', __FUNCTION__."() accepted
> {$NodeCacheInfo['amount']} files; size: {$NodeCacheInfo['size']}
> (".bytes_to_mbytes($NodeCacheInfo['size'])." Mb).");
> dir_size_atime_info('CLEARVARS');
>
> $Counter = 0;
> $Odds = $NodeLimit - $NodeCacheInfo['size'];
>
> if (($Odds <= 0) && ($NodeCacheInfo['size'] != 0)) {
> $Odds *= -1;
> debug_log('Cache', "[$SectionToClean] ".'Limit exceed for
> '.$Odds.'
> bytes ('.bytes_to_mbytes($Odds).' Mb), '.$NodeCacheInfo['amount'].'
> elements to process (curr limit: '.bytes_to_mbytes($NodeLimit).' Mb,
> curr occuped: '.bytes_to_mbytes($NodeCacheInfo['size']).' Mb)');
>
> asort($NodeCacheInfo['data']['atime']);
>
> for (; $Value = current($NodeCacheInfo['data']['atime']),
> $Odds > 0;
> $Value = next($NodeCacheInfo['data']['atime']), $Counter++) {
> $Key = key($NodeCacheInfo['data']['atime']); $Size =
> $NodeCacheInfo['data']['size'][$Key]; $FileName =
> $NodeCacheInfo['data']['file'][$Key];
> if (file_delete($FileName)) {$Odds -= $Size;}
> debug_log('Cache', "[$SectionToClean] Id: $Counter,
> key: ".$Key." =>
> atime: $Value, size: $Size, file name: $FileName; odds: $Odds");
> }//for
>
> } else {
> debug_log('Cache', "[$SectionToClean] ".'There is cache space
> ('.bytes_to_mbytes($NodeCacheInfo['size']).' Mb, limit:
> '.bytes_to_mbytes($NodeLimit).' Mb), no need to clean
> ('.$NodeCacheInfo['amount'].' files analyzed).');
> }//if
>
> clearstatcache();
>
> return $Counter;
> }//function
> ?>
> ==================================================================
>
> Hope it will be usefull and will find some reflection in existen CMSs.
> Please, feel free to ask me any questions regarding my CMS and such a
> cache implementation.
>
> Regards,
> /Alexander.
>
>
--
Diego F. Toritto.
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-developers
-~----------~----~----~----~------~----~------~--~---
