On 9/01/2007 8:25 AM, Carroll, Barry wrote: >> -----Original Message----- >> From: [EMAIL PROTECTED] [mailto:python- >> [EMAIL PROTECTED] On Behalf Of John Machin >> Sent: Saturday, January 06, 2007 11:09 PM >> To: python-list@python.org >> Subject: Re: File Closing Problem in 2.3 and 2.4, Not in 2.5 >> >> Martin v. Löwis wrote: >>> Carroll, Barry schrieb: >>>> What I want to know is: >>>> >>>> * has anyone else encountered a problem like this, * how was the >>>> problem corrected, * can the fix be retro-fitted to 2.5 and 2.4? >>> From your description, I suspect an error in your code. Your description >>> indicates that you don't expect to have more than five files open >>> simultaneously. Yet, the error message "Too many open files" occurs when >>> you open many more files (in the order of hundreds of files). >>> >>> It is very unlikely that there is a bug in Python where it would fail to >>> close a file when .close() is explicitly invoked on it (as your >>> description suggests that you do), so if you get that error message, it >>> can only mean that you fail to close some files. >> I don't understand: the OP's description suggests nothing of the sort >> to me. What he said was: >> """ >> In this way, a tree of Parser instances is created, each with a single >> open file object. (BTW, recursive and circular references are not >> allowed.) When each Parser instance comes to the end of its table, the >> instance is explicitly destroyed, presumably destroying any objects it >> holds, AND closing its open file. >> """ >> which I interpret as: he is doing del parser_instance, and *presuming* >> (incorrectly) that attributes of parser_instance (including an open >> file object) are magically whisked away instantly, instead of >> later/maybe. He later says he explicitly closed the files, which fixed >> what he alleges (incorrectly) to be a bug. >> >> To the OP: >> (1) The del statement doesn't "destroy" anything. It unbinds the name >> from the object in the current namespace, and decrements the object's >> reference count. Only if the reference count is then zero will the >> janitor be called in. >> (2) Check the reference count on the parser_instance just before you >> del it. You could be retaining a reference somewhere. >> (3) Explicitly close all non-lightweight objects like files (even >> read-only ones) and sockets rather than hoping they will go away. >> >> HTH, >> John > > John: > > Thank you. I was afraid I had been totally misunderstood. We have retained > the explicit file close in the current version of the Parser code. > > I still have a question, however. Python 2.3 and 2.4 both produce the "Too > many open files" error on Windows XP Pro, but Python 2.5 does not. Running > on the 2.5 interpreter, our program process all the table invocations with no > errors, a total of over 650 file opens. Also, Python 2.3 on RedHat Fedora > core 1 runs our program to completion with no errors. (I know, we are > running pretty ancient SW here. Constraints unrelated to this question make > that necessary.) Do you know why the error would appear on some combinations > of OS and interpreter and not on others? > > Thanks again for your help. > > Regards, > > Barry
Hi Barry, Please always reply on-list if the communication is non-private -- and meaningful :-) I don't know; here are some things you could check out: (1) Windows XP versus *x: The limit on open file handles is 512 in Windows XP; write yourself a reference-preserving [hint: append to a list] multiple file opener on *x and see how many you get open. I have vague recollections of the number 1024 being mentioned. (2) Python 2.4 versus 2.5: Read the "what's new" doc, fire up your googler, ... IMHO, the attitude should be to program defensively so that you need neither to know nor care about such implementation-specific concerns. Operating systems these days give you enough file handles to do any reasonable job. It's many moons since MS-DOS 2.n with its max 20 open file handles made folks implement LRU caches for file handles. Simple rules about unwanted objects include: (1) If an object has a free() or close() method, then call it! (2) Unbind *all* references to the object. Have you checked out my point 2 (holding multiple references to your Parser objects)? Cheers, John -- http://mail.python.org/mailman/listinfo/python-list
