Yes, CG-3 slowly grows in memory use for big corpora. I don't really know why. None of the memory checker tools I've tried could show me exactly what's going on - there are no leaks and nothing look unusual.
Problem is that thoroughly debugging it requires several hours of runtime for each check. -- Tino Didriksen On Sun, 4 Nov 2018 at 14:57, mansur <6688...@gmail.com> wrote: > I also noticed another problem... > > When I parse a very big file I see that cg-proc process' memory > consumption constantly grows. About 0,1% per 2 seconds. I get out of memory > in my machine (8Gb total) during the night... > > # top's outpus > 32509 root 20 0 1201684 1,1g 6540 R 92,1 14,2 24:35.61 > cg-proc > 32508 root 20 0 79972 54880 3628 R 80,8 0,7 21:10.69 > lt-proc > 32490 root 20 0 1165972 1,1g 6080 R 29,1 13,7 8:02.71 > cg-proc > > # command I'm using: > cat file.txt | sed -r 's/$/\n/' | apertium -n -d ./apertium-tat tat-tagger > | cg-proc ./apertium-tat/dev/mansur.bin > file.txt > > I also noticed that the speed of memory usage growing depends on what I > use in the example above: > 1) 's/$/\n/' > 2) 's/$/\n\n\n\n\n\n\n\n\n\n/' > 3) 's/$/\n. . . . . . . . . .\n. . . . . . . . . .\n. . . . . . . . . .\n. > . . . . . . . . .\n. . . . . . . . . .\n. . . . . . . . . .\n. . . . . . . > . . .\n. . . . . . . . . .\n. . . . . . . . . .\n. . . . . . . . . .\n/' > > With the 3rd example memory consumption grows much slower that with the > 1st. > > Does cg-proc process have problems with memory leaking or I'm somehow > using it the wrong way? What can I do here to process my file successfully? > > With best wishes, > Mansur >
_______________________________________________ Apertium-stuff mailing list Apertium-stuff@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/apertium-stuff
