Schoenwaelder Oliver wrote: > Hi list members, > > I have some problems with a script using hashes. > I use hashes for years but never had this kind of problem before... > I have a ascii file with 6.5 MB of data. This file is tokenized by > Parse:Lex module.
i never use this module before so i don't know how efficient it will be and what algr. it uses. > The tokens are then stored in a two level hash: > $TokenHash{$TokenType}->{$TokenID}=$TokenValue. not that much of a problem in Perl. hash algr. in Perl is pretty efficient in terms of speed and storage. yes, hash is a little waste of memory (apply to most other languages) but it's not that expensive as you might think of. > The file contains 536,332 tokens which will lead to 79 keys for > %TokenHash. I'm evaluating the hash with two loops, one for each level. > Due to that I need to move back and forth through the _sorted_ hash while > being in the loop I can't use the built-in procedures like "foreach $key1 > (keys %TokenHash)...". So I decided to use Tie::LLHash. never use Tie::LLHash either. did you contact the author of this module? > Now I'm amazed by the memory consumption. The script uses up to 300MB for could be a combination of your parsing module and Tie::LLHash that cause this. by the way, what platform are you using? > processing this small file which will lead to a 3.5 MB file at the end. I > developed and tested my script with a 2K subset of the original file and > therefore I haven't encountered the problem during tests. > A simple "if (not exists $TokenHash{$TokenType}->{$TokenID}) {}" uses > 110MB of memory. I encountered this when storing the elements into the > hash was commented out. Just tokenization of the file uses 4M of memory. > So in my opinion it's hash/hash operations related. > In production the files to be processed will be up to several 100MB of > size, so memory usage is really an issue for me. > I also tried with "simple/built-in" hashes just to be sure that the module > isn't the problem. But I got the same strange results. And I tried to use > multi-dimension arrays, but they also use up to 50MB of memory. > > Anything I need to consider? Anybody with the same experience? i regularly parse log files up to > 50G with pretty bad nested deep down hash/ref but never encounter the problem you are referring to. for a 6.5M of text file to consum over 300M of memory is unbelieveable. if the problem comes down to the modules you are using, they would be useless and many users would have complain about them. david -- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]