On Mittwoch, 14. Januar 2009, zehet...@molgen.mpg.de wrote: > I think it's objects creating new objects whose memory is than never > released even if the first object is unset.(most likely the php problem as > described in the link you sent me in your original reply). I tried to keep > track where which objects are created but I became quickly rather confused. > I could release a few extra K bytes but not enough to make any real > difference. > However, after I found in a moment of enlightnment a way to retrieve the > data I need without using any objects at all I didn't try any further.
Ok, I am about to look into the runs of SMW more closely. I think I will be able to safe some memory/time by reusing property objects (quite a lot of those are created during a single run now). Another issue during querying is that SMW prefetches database contents when a property table is accessed. If pages contain a lot of data (of which only few parts might be needed), then this prefetching might be rather inefficient. I will debug this soon. Cheers, Markus > > Cheers, > Gu > > Quoting Markus Krötzsch <mar...@semantic-mediawiki.org>: > > Ok, thanks for this first analysis. The built-in MW profiling actually > > reports > > memory usage as well. This might further increase our understanding of > > the issue. But ultimately, some detailed report on memory usage for > > allocated data > > structures in PHP would be most valuable to find out what really occupies > > those additional bytes. I have some guesses but it will take me some time > > until I can look into the details. > > > > -- Markus > > > > On Donnerstag, 8. Januar 2009, zehet...@molgen.mpg.de wrote: > > > Hi, > > > > > > I added some memory_get_usage() calls to the code > > > > > > 'before' is the return value before the call to > > > SMWQueryProcessor::getResultFromQueryString (with different offset > > > values) '#1' value in getInstanceQueryResult() after > > > $result = new SMWQueryResult() > > > '#2' value in getInstanceQueryResult() after the foreach loops > > > assigning $row[] = new SMWResultArray > > > > > > Offset:0 before:15,295,804 > > > #1: 16,775,940 > > > #2: 17,468,932 > > > Offset:10 before:17,557,768 > > > #1: 17,581,712 > > > #2: 18,729,208 > > > Offset:20 before:18,701,124 > > > #1: 18,733,052 > > > #2: 20,044,556 > > > > > > Offset:480 before:128,938,860 > > > #1: 128,981,036 > > > #2: 131,807,872 > > > Offset:490 before:131,779,456 > > > #1: 131,821,492 > > > <b>Fatal error</b>: Allowed memory size of 134217728 bytes > > > > > > Basically there are app. 240K memory used up for my query after calling > > > getResultFromQueryString.and at present that limits me to less than 500 > > > returned results. > > > > > > It would of course be nice if there would be a way to release some of > > > that memory again after the result has been processed (but I guess > > > first one would have to know where it went, to the SMWQueryResult > > > object or > > > > somewhere > > > > > else?) > > > > > > Another question is, if (as I only just need the page titles from the > > > > query > > > > > result) there is a simpler way to do that than calling > > > getResultFromQueryString? > > > > > > Cheers, > > > Gu > > > > > > Quoting Markus Krötzsch <mar...@semantic-mediawiki.org>: > > > > Hi, > > > > > > > > I am not aware of a good way of debugging memory issues in PHP yet. > > > > Can anyone > > > > recommend a free profiler or the like? As for your problem, there is > > > > a general > > > > tendency of PHP to hoard memory during (very) long runs. This is a > > > > known bug > > > > > > > > that apparently can only be avoided by re-engineering the object > > > > dependencies > > > > > > > > in affected code [1]. However, without a proper memory profiling, I > > > > have no feasible approach for finding out where this should be done. > > > > > > > > Of course, it is also possible that some other problem causes memory > > > > to be eaten without PHP being to blame. Usually this is caused by > > > > look-up arrays that are used as caches in PHP. For example, MediaWiki > > > > has a LinkCache that can grow without control; if you do long runs > > > > that may (indirectly) use this > > > > > > > > cache, then it is required to purge it manually from time to time. > > > > Again, > > > > > > there might be further such problems, though I have tried to limit > > > > all caches' > > > > size in SMW. Ideas for better debugging are welcome. > > > > > > > > Regards, > > > > > > > > Markus > > > > > > > > [1] http://paul-m-jones.com/?p=262 > > > > > > > > On Mittwoch, 7. Januar 2009, zehet...@molgen.mpg.de wrote: > > > > > Hi, > > > > > > > > > > I wonder if anyone can tell me if the functions used when I call > > > > > SMWQueryProcessor::getResultFromQueryString should release all the > > > > > memory after they finished? > > > > > > > > > > I thought I could avoid a memory problem by calling > > > > > getResultFromQueryString with different 'offset' values to retrieve > > > > > > > > smaller > > > > > > > > > chunks of articles. However, at some point I still run out of > > > > > memory > > > > > > > > during > > > > > > > > > the loop, even if I just call the function and ignore the result. > > > > > > > > > > $maxresults = 500; > > > > > $limit = $params['limit']; # is 10 > > > > > for ($ofs = 0; $ofs < $maxresults ; $ofs += $limit) { > > > > > $params['offset'] = $ofs; > > > > > $dummy = > > > > > SMWQueryProcessor::getResultFromQueryString($querystring, $params, > > > > > $printouts, SMW_OUTPUT_WIKI); > > > > > } > > > > > > > > > > When $ofs reaches 490 I run out of memory (Fatal error: Allowed > > > > > memory > > > > > > > > size > > > > > > > > > of 134217728 bytes exhausted). > > > > > > > > > > The test query is '[[MyProp::~*;?;?]][[Category:MainArticle]]' with > > > > > about 19000 articles matching (as using [[MyProp::?;?;?]] gives the > > > > > error 'The value of property "MyProp" was not understood', I have > > > > > to replace one '?' with '~*'). > > > > > > > > > > MediaWiki 1.13.1 > > > > > PHP 5.2.1 > > > > > MySQL 6.0.45 > > > > > SMW 1.5a SVN > > > > > > > > > > Thanks, > > > > > Gu > > > > ------------------------------------------------------------------------- > > > > > >-- > > > > > > > > >--- Check out the new SourceForge.net Marketplace. > > > > > It is the best place to buy or sell services for > > > > > just about anything Open Source. > > > > > http://p.sf.net/sfu/Xq1LFB > > > > > _______________________________________________ > > > > > Semediawiki-devel mailing list > > > > > Semediawiki-devel@lists.sourceforge.net > > > > > https://lists.sourceforge.net/lists/listinfo/semediawiki-devel > > > > > > > > -- > > > > Markus Krötzsch > > > > Semantic MediaWiki http://semantic-mediawiki.org > > > > http://korrekt.org mar...@semantic-mediawiki.org > > > > -- > > Markus Krötzsch > > Semantic MediaWiki http://semantic-mediawiki.org > > http://korrekt.org mar...@semantic-mediawiki.org -- Markus Krötzsch Semantic MediaWiki http://semantic-mediawiki.org http://korrekt.org mar...@semantic-mediawiki.org
signature.asc
Description: This is a digitally signed message part.
------------------------------------------------------------------------------ This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword
_______________________________________________ Semediawiki-devel mailing list Semediawiki-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/semediawiki-devel
