Thanks. I actually comment out all handling code. The loop ends with the re_pat.match and nothing followed. Sent from Mail Master On 07/05/2017 08:31, [1]Cameron Simpson wrote:
On 04Jul2017 17:01, Mayling ge <maylinge0...@gmail.com> wrote: > My function is in the following way to handle file line by line. There are > multiple error patterns defined and need to apply to each line. I use > multiprocessing.Pool to handle the file in block. > > The memory usage increases to 2G for a 1G file. And stays in 2G even after > the file processing. File closed in the end. > > If I comment out the call to re_pat.match, memory usage is normal and > keeps under 100Mb. [...] > > def line_match(lines, errors) > for error in errors: > try: > re_pat = re.compile(error['pattern']) > except Exception: > print_error > continue > for line in lines: > m = re_pat.match(line) > # other code to handle matched object [...] > Notes: I omit some code as I think the significant difference is > with/without re_pat.match(...) Hmm. Does the handling code (omitted) keep the line or match object in memory? If leaving out the "m = re_pat.match(line)" triggers the leak, and presuming that line itself doesn't leak, then I would start to suspect the handling code is not letting go of the match object "m" or of the line (which is probably attached to the match object "m" to support things like m.group() and so forth). So you might need to show us the handling code. Cheers, Cameron Simpson <c...@zip.com.au> References Visible links 1. mailto:c...@zip.com.au -- https://mail.python.org/mailman/listinfo/python-list