[issue1652] subprocess should have an option to restore SIGPIPE to default action
Changes by Robert Cronk cron...@gmail.com: -- nosy: +rcronk ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1652 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4749] Issue with RotatingFileHandler logging handler on Windows
Robert Cronk cron...@gmail.com added the comment: I changed the script to use subprocess (attached file) and got the same rollover errors as before. I had to change cd and del to be cd.bat and del.bat which contained cd %1 and del %1 respectively since it appears subprocess can't run internal commands like cd and del (unless you specify shell = True, which I thought might defeat the purpose of the test). I will search around for this bug to see if it's already been entered. If the python developers decide not to fix this by wrapping os.system (and I guess subprocess.Popen too) with locks to prevent this error, then I agree that it should at least be well documented. -- Added file: http://bugs.python.org/file14260/subprclg.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4749 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1425127] os.remove OSError: [Errno 13] Permission denied
Robert Cronk cron...@gmail.com added the comment: Could this problem be associated with issue4749? It was found that something goes wrong when two cmd children processes are spawned from different threads, when the first exits, it is closing file handles shared with the first (or something like that) and it's causing a problem with logging in issue4749. That bug has been closed since it's not a problem with logging so I'm searching for other similar bugs to see if we can create a new bug that documents the cause and link to these other bugs that are all showing different symptoms of the bug. Thoughts? -- nosy: +rcronk ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1425127 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4749] Issue with RotatingFileHandler logging handler on Windows
Robert Cronk cron...@gmail.com added the comment: I found Issue1425127 which may be a different symptom of this core problem. I suggested that we create a bug that documents the core problem here as described by Vinay in msg89174 and links to these two bugs (along with any others we find) as examples of the types of symptoms that can come from this bug. At that point, the merits of working around this bug in python versus documenting the problem for python users to work around it can be discussed and a decision can be made. Thoughts? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4749 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4749] Issue with RotatingFileHandler logging handler on Windows
Robert Cronk cron...@gmail.com added the comment:
One more possibly related bug is issue2320 where subprocesses are
spawned from multiple threads. They found an interesting workaround
that I found seems to help our problem too: on Windows, if close_fds
is true then no handles will be inherited by the child process. So if
I change the subprocess.Popen lines in my most recently uploaded script
to:
rc = subprocess.Popen('cd.bat . blah.txt', close_fds=True).wait()
rc = subprocess.Popen('del.bat blah.txt', close_fds=True).wait()
The rollover logging failure goes away. Does os.system have a similar
option? If not, then that would mean I'd have to convert all os.system
calls to subprocess.Popen and add the close_fds=True parameter to it.
The problem with this is that Note that on Windows, you cannot set
close_fds to true and also redirect the standard handles by setting
stdin, stdout or stderr and I am using both stdin and stdout on my
main subprocess.Popen call and I have a couple of os.system calls as
well. Thoughts?
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4749
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2320] Race condition in subprocess using stdin
Robert Cronk cron...@gmail.com added the comment: Could this problem be associated with issue4749? It was found that something goes wrong when two cmd children processes are spawned from different threads, when the first exits, it is closing file handles shared with the first (or something like that) and it's causing a problem with logging in issue4749. That bug has been closed since it's not a problem with logging itself so I'm searching for other similar bugs to see if we can create a new bug that documents the cause and link to these other bugs that are all showing different symptoms of the bug. Thoughts? -- nosy: +rcronk ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue2320 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4749] Issue with RotatingFileHandler logging handler on Windows
Robert Cronk cron...@gmail.com added the comment: Thanks Vinay. I ran the newest revised script with virus protection turned off and got the same failures as before (see console output below). If you comment out the os.system() calls, everything works just fine. Uncomment them and logging breaks. The os.system() calls shouldn't break logging, especially since they are accessing a completely unrelated file - not the log file. If you could go back to my original script that fails for you, you should be able to chase down why these unrelated os.system() calls are breaking logging. Feel free to add the joins and replace the tabs if you want to, but don't put locks around the os.system() calls because that just makes the problem go away and leaves us with nothing to do. Does that make sense? Regardless of what the os.system() calls are doing, they shouldn't break logging in any way whatsoever. So if my original script breaks for you, please use that broken state to debug the problem and find out why an os.system() call is breaking logging. --- C:\logthredrevised.py Traceback (most recent call last): File C:\Python26\lib\logging\handlers.py, line 72, in emit self.doRollover() File C:\Python26\lib\logging\handlers.py, line 129, in doRollover os.rename(self.baseFilename, dfn) WindowsError: [Error 32] The process cannot access the file because it is being used by another process 2009-06-09 09:54:47,828 DEBUG3704 This is log message 0 from thread- 00 and I think this should be a little longer, so I'll add some more data here because p erhaps the size of the individual log message is a factor. Who knows for sure u ntil this simple test fails. Traceback (most recent call last): File C:\Python26\lib\logging\handlers.py, line 71, in emit if self.shouldRollover(record): File C:\Python26\lib\logging\handlers.py, line 145, in shouldRollover self.stream.seek(0, 2) #due to non-posix-compliant Windows feature ValueError: I/O operation on closed file 2009-06-09 09:54:47,967 DEBUG9052 This is log message 0 from thread- 01 and I think this should be a little longer, so I'll add some more data here because p erhaps the size of the individual log message is a factor. Who knows for sure u ntil this simple test fails. Traceback (most recent call last): File C:\Python26\lib\logging\handlers.py, line 71, in emit if self.shouldRollover(record): File C:\Python26\lib\logging\handlers.py, line 145, in shouldRollover self.stream.seek(0, 2) #due to non-posix-compliant Windows feature ValueError: I/O operation on closed file 2009-06-09 09:54:48,062 DEBUG7480 This is log message 0 from thread- 02 and I think this should be a little longer, so I'll add some more data here because p erhaps the size of the individual log message is a factor. Who knows for sure u ntil this simple test fails. Traceback (most recent call last): File C:\Python26\lib\logging\handlers.py, line 71, in emit if self.shouldRollover(record): File C:\Python26\lib\logging\handlers.py, line 145, in shouldRollover self.stream.seek(0, 2) #due to non-posix-compliant Windows feature ValueError: I/O operation on closed file . . . -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4749 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4749] Issue with RotatingFileHandler logging handler on Windows
Robert Cronk cron...@gmail.com added the comment: I'll thoroughly look through every piece of software that's running to see if I can turn eveything off that might be causing the problem. Were you able to reproduce the problem with my original script? I'm sure you have all of your virus/searching/etc. stuff turned off, so if it reproduced for you, then those things aren't the problem. Question: If you take my original script and add joins at the end so it's proper and modify nothing else about it, does it show the rollover error? If so, then I think that is where we need to start since you then have a simple test that manifests the problem without virus scanners (etc.) that might taint the issue. Then you can chase this without having to assume what my system has running on it. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4749 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4749] Issue with RotatingFileHandler logging handler on Windows
Robert Cronk cron...@gmail.com added the comment: I turned off anti-virus again as well as file indexing and google desktop too and still got the errors when I disabled the locks around the os.system() calls. Vinay - when the locks aren't around the os.system() calls, do you get the rotating log errors? I'm still confused at how the os.system calls could be affecting the logging at all. The os.system calls aren't touching the log files. Why would it cause them to fail when the os.system calls fail? It seems that when the os.system calls succeed (because of the locks) then the logging succeeds but when the os.system calls fail (because the locks are disabled), then logging fails. If, as you suggest, this is a race condition that is being exposed by the os.system calls failing because they don't have locks around them, then that would be the perfect situation (if you can reproduce it) to debug it. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4749 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4749] Issue with RotatingFileHandler logging handler on Windows
Robert Cronk cron...@gmail.com added the comment: Vinay - that's great news! Are you going to create a new bug for this issue with a proper title? It would seem to me that the fix for this would be to put locks internal to the os.system() call around where it spawns cmd so multiple spawns don't occur simultaneously. Is the proper procedure at this point to open a new bug with a more correct title that points back to this bug for reference? Can you do that Vinay? I don't know if I have the authority (nor experience with the bug tracking system) to do that. Thanks. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4749 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4749] Issue with RotatingFileHandler logging handler on Windows
Robert Cronk cron...@gmail.com added the comment:
I have a small script that reproduces the problem. I couldn't
reproduce it until I added some os.system() calls in the threads that
were logging. Here's the output using python 2.6.1:
Traceback (most recent call last):
File C:\Python26\lib\logging\handlers.py, line 74, in emit
if self.shouldRollover(record):
File C:\Python26\lib\logging\handlers.py, line 146, in
shouldRollover
self.stream.seek(0, 2) #due to non-posix-compliant Windows feature
ValueError: I/O operation on closed file
Here is the script - let me know if I'm doing things incorrectly:
import os, threading, logging, logging.handlers
class LoggerThread(threading.Thread):
def __init__(self, numLoops):
threading.Thread.__init__(self)
self.numLoops = numLoops
def run(self):
for x in range(0, self.numLoops):
os.system('cd.blah.txt')
os.system('del blah.txt')
logging.debug('This is log message ' + str(x) + ' from ' +
self.name + ' and I think this should be a little longer, so I\'ll add
some more data here because perhaps the size of the individual log
message is a factor. Who knows for sure until this simple test fails.')
if __name__==__main__:
logSize = 2048
numberOfLogs = 10
files = logging.handlers.RotatingFileHandler('logthred.log', 'a',
logSize, numberOfLogs)
console = logging.StreamHandler()
# set a format
fileFormatter = logging.Formatter('%(asctime)s %(levelname)-8s %
(thread)-4s %(message)s')
consoleFormatter = logging.Formatter('%(asctime)s %(levelname)-8s %
(thread)-4s %(message)s')
# tell the handler to use this format
files.setFormatter(fileFormatter)
console.setFormatter(consoleFormatter)
# add the handlers to the root logger
logging.getLogger('').addHandler(files)
logging.getLogger('').addHandler(console)
logging.getLogger('').setLevel(logging.DEBUG)
numThreads = 10
numLoops = 100
# Create and execute threads
for x in range(0, numThreads):
LoggerThread(numLoops).start()
--
Added file: http://bugs.python.org/file14226/logthred.py
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4749
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4749] Issue with RotatingFileHandler logging handler on Windows
Robert Cronk cron...@gmail.com added the comment: P.S. The above script and failure is running on winxp sp3. Also, if you comment out the two os.system() calls, it works just fine. They seem like they should be unrelated to the logging though. You'll also see some errors about access to the blah.txt file which makes sense since multiple threads are hitting that file at the same time. I don't know if this is about using os.system() itself from multiple threads while logging or if it's about having an error condition during the os.system() call on top of that. Anyway, let me know what you think or if I've done something wrong, let me know how to fix it and that might be good documentation for others running into this problem. Thanks. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4749 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4749] Issue with RotatingFileHandler logging handler on Windows
Robert Cronk cron...@gmail.com added the comment: import sys print sys.version 2.6.1 (r261:67517, Dec 4 2008, 16:51:00) [MSC v.1500 32 bit (Intel)] I have seen this behavior in older versions as well. Interesting to see it fail in linux as well. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4749 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4749] Issue with RotatingFileHandler logging handler on Windows
Robert Cronk cron...@gmail.com added the comment: P.S. Frans - It's good to get these other data points from you. So this is reproducible from another person and on different versions of python AND on different platforms! I wasn't expecting that at all. Thanks Frans. Is there a way we can reopen this bug? I couldn't find a way to change its status now that we seem to have a reproducible case. Perhaps Vinay is authorized to do so. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4749 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4749] Issue with RotatingFileHandler logging handler on Windows
Robert Cronk cron...@gmail.com added the comment: Thanks Lowell - good information. You have many more versions of Python laying around than I do. ;) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4749 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4749] Issue with RotatingFileHandler logging handler on Windows
Robert Cronk cron...@gmail.com added the comment: I just upgraded to 2.6.2 windows from python.org and it fails as well: Python 2.6.2 (r262:71605, Apr 14 2009, 22:40:02) [MSC v.1500 32 bit (Intel)] on win32 I hope Vinay can track this down in case it's a race condition that's just moving around between versions or track it down to something concrete that has actually been purposefully fixed somewhere else that fixes this too. I guess we'll see. Lowell - what's the difference between my 2.6.2 shown above and yours (release26-maint, Apr 19 2009)? Yours is on ubuntu and mine is on windows, but I'm not familiar with the two version types (r262:71605 vs. release26-maint). Yours is 5 days newer - is it a patch? What was changed in it that might affect this problem? Just wondering aloud until Vinay can help make sense of all of this. Thanks everyone. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4749 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4749] Issue with RotatingFileHandler logging handler on Windows
Robert Cronk cron...@gmail.com added the comment: I didn't care about the os.system() call contention because that's what caused the logging problem and that blah.txt file contention should not cause logging to fail. I also had the join calls originally but took them out to simplify the code since it seemed to run correctly either way - if this were production code, I'd have left them in. The revised script works for me on windows 2.6.2 (the version I upgraded to) but I think it just puts locks around the problem and masks the true problem out. It appears something in os.system() is crashing logging and that shouldn't happen. If locks need to be placed, they should be placed around the problem within os.system() or within logging, if needed. Please take the locks off the os.system() calls and see why logging is failing when those calls are made. Remember, this is a test script I wrote from scratch with the express purpose of making logging fail from multiple threads so you could catch it in a debugger or something. Changing the script to make it work misses the point. -- status: pending - open ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4749 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4749] Issue with RotatingFileHandler logging handler on Windows
Robert Cronk cron...@gmail.com added the comment: I have had this problem with 2.6.1 on windows from multiple _threads_ instead of multiple processes. Is that not supported either? If not, what is the workaround for logging from multiple threads? Shouldn't it be easy to use a semaphore within the logging code to make it thread safe at least? I have had this problem on multiple PC's with multiple versions of python, and multiple versions of windows. -- nosy: +rcronk ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4749 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4749] Issue with RotatingFileHandler logging handler on Windows
Robert Cronk cron...@gmail.com added the comment: I will go through the code and make sure I am not mistaken, but I do believe I have a single process, multiple threads, and only one handler for this file and I'm getting the same types of error messages shown on this page. I'm probably doing something wrong. I'll post what I find soon. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4749 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
