#20335: ReferenceChecker causes OOM -------------------------------+--------------------- Reporter: iwakeh | Owner: Type: defect | Status: new Priority: High | Milestone: Component: Metrics/CollecTor | Version: Severity: Normal | Resolution: Keywords: | Actual Points: Parent ID: | Points: Reviewer: | Sponsor: -------------------------------+--------------------- Changes (by karsten):
* priority: Medium => High Comment: I ''believe'' that's a smart thing to do. I'm not sure if that would break something. But I think it doesn't solve the issue here. Or, I ran out of memory with a similar patch applied. And I think I know why: {{{ -rw-r--r-- 1 karsten staff 2.1G Oct 18 11:52 /Users/karsten/src/collector/recent/relay- descriptors/votes/2016-10-18-09-22-46-votes }}} With the recent change to append all votes to a single file, we're creating a 2.1G large file when doing the first synchronization run. However, metrics-lib wants to read that entire file to memory which probably doubles in size when it's parsed. Ugh! Note that the CollecTor's reference checker is just one application that breaks when input files are too large. Other applications using metrics- lib would similarly be affected. A longer-term fix might be to change metrics-lib to read through large files and pause when it has parsed enough descriptors that the application still has to process. A short-term fix would be to stick with the current naming scheme for files in the `recent/` directory, as that would have avoided this issue. Fun times! -- Ticket URL: <https://trac.torproject.org/projects/tor/ticket/20335#comment:10> Tor Bug Tracker & Wiki <https://trac.torproject.org/> The Tor Project: anonymity online _______________________________________________ tor-bugs mailing list tor-bugs@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs