From: Python-list <python-list-bounces+sjeik_appie=hotmail....@python.org> on behalf of Mayling ge <maylinge0...@gmail.com> Sent: Tuesday, July 4, 2017 9:01 AM To: python-list Subject: memory leak with re.match Hi,
My function is in the following way to handle file line by line. There are multiple error patterns defined and need to apply to each line. I use multiprocessing.Pool to handle the file in block. The memory usage increases to 2G for a 1G file. And stays in 2G even after the file processing. File closed in the end. If I comment out the call to re_pat.match, memory usage is normal and keeps under 100Mb. am I using re in a wrong way? I cannot figure out a way to fix the memory leak. And I googled . def line_match(lines, errors) <snip> lines = list(itertools.islice(fo, line_per_proc)) ===> do you really need to listify the iterator? if not lines: break result = p.apply_async(line_match, args=(errors, lines)) ===> the signature of line_match is (lines, errors), in args you do (errors, lines) -- https://mail.python.org/mailman/listinfo/python-list
