[ python-Bugs-1512163 ] mailbox (2.5b1): locking doesn't work (esp. on FreeBSD)
Bugs item #1512163, was opened at 2006-06-25 16:38 Message generated for change (Comment added) made by baikie You can respond by visiting: https://sourceforge.net/tracker/?func=detailatid=105470aid=1512163group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Python Library Group: Python 2.5 Status: Open Resolution: None Priority: 5 Private: No Submitted By: David Watson (baikie) Assigned to: A.M. Kuchling (akuchling) Summary: mailbox (2.5b1): locking doesn't work (esp. on FreeBSD) Initial Comment: fcntl/flock() locking of mailboxes is actually disabled due to a typo: --- mailbox.py.orig +++ mailbox.py @@ -15,7 +15,7 @@ import rfc822 import StringIO try: -import fnctl +import fcntl except ImportError: fcntl = None However, once enabled, it doesn't work on FreeBSD - the lock() method always raises ExternalClashError (tested on 5.3). This is because flock(), lockf() and fcntl locking share the same underlying mechanism on this system (and probably others), and so the second lock can never be acquired. -- Comment By: David Watson (baikie) Date: 2006-11-12 23:52 Message: Logged In: YES user_id=1504904 I'd forgotten about this for a while, but I see the docs didn't actually get changed for 2.5. I was going to submit a patch for them today and suggest using subclasses as described below, but then I realized that subclassing wouldn't work, since the code makes use of the module functions _lock_file() and _unlock_file() outside of the lock()/unlock() methods. A possible solution is attached; the Mailbox class is given methods _acquire_lock() and _release_lock() wrapping _lock_file() and _unlock_file(), and these are used instead (thus, they can be overridden as methods, but if someone had replaced _lock_file() in the module namespace instead, then their code would still work). Also attached is a minimal doc patch to describe the new (shorter) list of locking methods. I've called the system-call locking fcntl() since that's the usual term when dealing with mailboxes, and the name fcntl.lockf() is misleading (it calls fcntl() directly, not lockf(), and the Linux man pages say that In general, the relation between lockf() and fcntl() is unspecified.). -- Comment By: Neal Norwitz (nnorwitz) Date: 2006-06-27 04:21 Message: Logged In: YES user_id=33168 Is the optional parameter necessary if you use inheritance (assuming there's a class/object in there)? Maybe it would be better to have a subclass? Would that be better for us to provide or just to add something to the docs that a user could make a subclass to work around locking issues? I haven't looked at the issues, but if we really need an API change, I'm probably ok with it since it would seem to be quite small. If an API change is really required (doc isn't sufficient), I'd like it to go in ASAP. We could doc for 2.5 and if there's a problem fix for 2.6. -- Comment By: A.M. Kuchling (akuchling) Date: 2006-06-27 00:24 Message: Logged In: YES user_id=11375 Hm, this looks messy; I doubt the module can know if you want lockf() or fcntl() locking. Maybe Mailbox.lock() should grow an optional parameter that lets you specify the locking mechanism to use -- lockf, fcntl, or both. (But defaulting to what?) On the other hand, maybe it's too late to go making API changes for beta2 (but I suspect it would be OK in this case). -- Comment By: David Watson (baikie) Date: 2006-06-26 23:56 Message: Logged In: YES user_id=1504904 Yeah, it works fine now (locks successfully when the mailbox isn't locked by another process, and fails when it is locked), although I was reminded that the Python install process didn't work properly (I've submitted another bug report). However, I should point out that flock() locking is required when working with qmail on (for example) Linux - flock() is the only locking mechanism it uses, and as noted below, flock() on Linux is independent from fcntl/lockf(). -- Comment By: A.M. Kuchling (akuchling) Date: 2006-06-26 14:27 Message: Logged In: YES user_id=11375 Typo fixed in rev. 47099; flock() calls removed in rev. 47100; a test for lock conflicts added in rev. 47101. David, please try out the current SVN head and let me know if matters have improved. I'll wait to see the buildbot reports before closing this bug. (Surprisingly, we don't seem to have a FreeBSD buildbot, though we do have an OpenBSD one that didn't seem to have a problem with the original version of the mailbox code.)
[ python-Bugs-1512163 ] mailbox (2.5b1): locking doesn't work (esp. on FreeBSD)
Bugs item #1512163, was opened at 2006-06-25 11:38 Message generated for change (Comment added) made by akuchling You can respond by visiting: https://sourceforge.net/tracker/?func=detailatid=105470aid=1512163group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Python Library Group: Python 2.5 Status: Open Resolution: None Priority: 5 Submitted By: David Watson (baikie) Assigned to: A.M. Kuchling (akuchling) Summary: mailbox (2.5b1): locking doesn't work (esp. on FreeBSD) Initial Comment: fcntl/flock() locking of mailboxes is actually disabled due to a typo: --- mailbox.py.orig +++ mailbox.py @@ -15,7 +15,7 @@ import rfc822 import StringIO try: -import fnctl +import fcntl except ImportError: fcntl = None However, once enabled, it doesn't work on FreeBSD - the lock() method always raises ExternalClashError (tested on 5.3). This is because flock(), lockf() and fcntl locking share the same underlying mechanism on this system (and probably others), and so the second lock can never be acquired. -- Comment By: A.M. Kuchling (akuchling) Date: 2006-06-26 08:56 Message: Logged In: YES user_id=11375 Interesting. The man page for flock() on my Linux system says of flock() in a certain version: This yields true BSD semantics: there is no interaction between the types of lock placed by flock(2) and fcntl(2), and flock(2) does not detect deadlock. Apparently this is out of date, and placing two locks is harmful. I think it's best to just use one set of locking primitives, and that one should be lockf(); the flock() calls should be removed. -- Comment By: Neal Norwitz (nnorwitz) Date: 2006-06-25 15:52 Message: Logged In: YES user_id=33168 Andrew, are you helping maintain this module? David is correct that fcntl is mispelled in current svn. Not sure what to do about the locking issue. David, would it be possible for you to work on a patch to correct this problem? I'm not sure if any developer has access to a FreeBSD box similar to yours. (There are a couple that use FreeBSD though.) It would be great to come up with tests for this if the current ones don't stress this situation. -- You can respond by visiting: https://sourceforge.net/tracker/?func=detailatid=105470aid=1512163group_id=5470 ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[ python-Bugs-1512163 ] mailbox (2.5b1): locking doesn't work (esp. on FreeBSD)
Bugs item #1512163, was opened at 2006-06-25 11:38 Message generated for change (Comment added) made by akuchling You can respond by visiting: https://sourceforge.net/tracker/?func=detailatid=105470aid=1512163group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Python Library Group: Python 2.5 Status: Open Resolution: None Priority: 5 Submitted By: David Watson (baikie) Assigned to: A.M. Kuchling (akuchling) Summary: mailbox (2.5b1): locking doesn't work (esp. on FreeBSD) Initial Comment: fcntl/flock() locking of mailboxes is actually disabled due to a typo: --- mailbox.py.orig +++ mailbox.py @@ -15,7 +15,7 @@ import rfc822 import StringIO try: -import fnctl +import fcntl except ImportError: fcntl = None However, once enabled, it doesn't work on FreeBSD - the lock() method always raises ExternalClashError (tested on 5.3). This is because flock(), lockf() and fcntl locking share the same underlying mechanism on this system (and probably others), and so the second lock can never be acquired. -- Comment By: A.M. Kuchling (akuchling) Date: 2006-06-26 09:27 Message: Logged In: YES user_id=11375 Typo fixed in rev. 47099; flock() calls removed in rev. 47100; a test for lock conflicts added in rev. 47101. David, please try out the current SVN head and let me know if matters have improved. I'll wait to see the buildbot reports before closing this bug. (Surprisingly, we don't seem to have a FreeBSD buildbot, though we do have an OpenBSD one that didn't seem to have a problem with the original version of the mailbox code.) -- Comment By: A.M. Kuchling (akuchling) Date: 2006-06-26 08:56 Message: Logged In: YES user_id=11375 Interesting. The man page for flock() on my Linux system says of flock() in a certain version: This yields true BSD semantics: there is no interaction between the types of lock placed by flock(2) and fcntl(2), and flock(2) does not detect deadlock. Apparently this is out of date, and placing two locks is harmful. I think it's best to just use one set of locking primitives, and that one should be lockf(); the flock() calls should be removed. -- Comment By: Neal Norwitz (nnorwitz) Date: 2006-06-25 15:52 Message: Logged In: YES user_id=33168 Andrew, are you helping maintain this module? David is correct that fcntl is mispelled in current svn. Not sure what to do about the locking issue. David, would it be possible for you to work on a patch to correct this problem? I'm not sure if any developer has access to a FreeBSD box similar to yours. (There are a couple that use FreeBSD though.) It would be great to come up with tests for this if the current ones don't stress this situation. -- You can respond by visiting: https://sourceforge.net/tracker/?func=detailatid=105470aid=1512163group_id=5470 ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[ python-Bugs-1512163 ] mailbox (2.5b1): locking doesn't work (esp. on FreeBSD)
Bugs item #1512163, was opened at 2006-06-25 16:38 Message generated for change (Comment added) made by baikie You can respond by visiting: https://sourceforge.net/tracker/?func=detailatid=105470aid=1512163group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Python Library Group: Python 2.5 Status: Open Resolution: None Priority: 5 Submitted By: David Watson (baikie) Assigned to: A.M. Kuchling (akuchling) Summary: mailbox (2.5b1): locking doesn't work (esp. on FreeBSD) Initial Comment: fcntl/flock() locking of mailboxes is actually disabled due to a typo: --- mailbox.py.orig +++ mailbox.py @@ -15,7 +15,7 @@ import rfc822 import StringIO try: -import fnctl +import fcntl except ImportError: fcntl = None However, once enabled, it doesn't work on FreeBSD - the lock() method always raises ExternalClashError (tested on 5.3). This is because flock(), lockf() and fcntl locking share the same underlying mechanism on this system (and probably others), and so the second lock can never be acquired. -- Comment By: David Watson (baikie) Date: 2006-06-26 23:56 Message: Logged In: YES user_id=1504904 Yeah, it works fine now (locks successfully when the mailbox isn't locked by another process, and fails when it is locked), although I was reminded that the Python install process didn't work properly (I've submitted another bug report). However, I should point out that flock() locking is required when working with qmail on (for example) Linux - flock() is the only locking mechanism it uses, and as noted below, flock() on Linux is independent from fcntl/lockf(). -- Comment By: A.M. Kuchling (akuchling) Date: 2006-06-26 14:27 Message: Logged In: YES user_id=11375 Typo fixed in rev. 47099; flock() calls removed in rev. 47100; a test for lock conflicts added in rev. 47101. David, please try out the current SVN head and let me know if matters have improved. I'll wait to see the buildbot reports before closing this bug. (Surprisingly, we don't seem to have a FreeBSD buildbot, though we do have an OpenBSD one that didn't seem to have a problem with the original version of the mailbox code.) -- Comment By: A.M. Kuchling (akuchling) Date: 2006-06-26 13:56 Message: Logged In: YES user_id=11375 Interesting. The man page for flock() on my Linux system says of flock() in a certain version: This yields true BSD semantics: there is no interaction between the types of lock placed by flock(2) and fcntl(2), and flock(2) does not detect deadlock. Apparently this is out of date, and placing two locks is harmful. I think it's best to just use one set of locking primitives, and that one should be lockf(); the flock() calls should be removed. -- Comment By: Neal Norwitz (nnorwitz) Date: 2006-06-25 20:52 Message: Logged In: YES user_id=33168 Andrew, are you helping maintain this module? David is correct that fcntl is mispelled in current svn. Not sure what to do about the locking issue. David, would it be possible for you to work on a patch to correct this problem? I'm not sure if any developer has access to a FreeBSD box similar to yours. (There are a couple that use FreeBSD though.) It would be great to come up with tests for this if the current ones don't stress this situation. -- You can respond by visiting: https://sourceforge.net/tracker/?func=detailatid=105470aid=1512163group_id=5470 ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[ python-Bugs-1512163 ] mailbox (2.5b1): locking doesn't work (esp. on FreeBSD)
Bugs item #1512163, was opened at 2006-06-25 11:38 Message generated for change (Comment added) made by akuchling You can respond by visiting: https://sourceforge.net/tracker/?func=detailatid=105470aid=1512163group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Python Library Group: Python 2.5 Status: Open Resolution: None Priority: 5 Submitted By: David Watson (baikie) Assigned to: A.M. Kuchling (akuchling) Summary: mailbox (2.5b1): locking doesn't work (esp. on FreeBSD) Initial Comment: fcntl/flock() locking of mailboxes is actually disabled due to a typo: --- mailbox.py.orig +++ mailbox.py @@ -15,7 +15,7 @@ import rfc822 import StringIO try: -import fnctl +import fcntl except ImportError: fcntl = None However, once enabled, it doesn't work on FreeBSD - the lock() method always raises ExternalClashError (tested on 5.3). This is because flock(), lockf() and fcntl locking share the same underlying mechanism on this system (and probably others), and so the second lock can never be acquired. -- Comment By: A.M. Kuchling (akuchling) Date: 2006-06-26 19:24 Message: Logged In: YES user_id=11375 Hm, this looks messy; I doubt the module can know if you want lockf() or fcntl() locking. Maybe Mailbox.lock() should grow an optional parameter that lets you specify the locking mechanism to use -- lockf, fcntl, or both. (But defaulting to what?) On the other hand, maybe it's too late to go making API changes for beta2 (but I suspect it would be OK in this case). -- Comment By: David Watson (baikie) Date: 2006-06-26 18:56 Message: Logged In: YES user_id=1504904 Yeah, it works fine now (locks successfully when the mailbox isn't locked by another process, and fails when it is locked), although I was reminded that the Python install process didn't work properly (I've submitted another bug report). However, I should point out that flock() locking is required when working with qmail on (for example) Linux - flock() is the only locking mechanism it uses, and as noted below, flock() on Linux is independent from fcntl/lockf(). -- Comment By: A.M. Kuchling (akuchling) Date: 2006-06-26 09:27 Message: Logged In: YES user_id=11375 Typo fixed in rev. 47099; flock() calls removed in rev. 47100; a test for lock conflicts added in rev. 47101. David, please try out the current SVN head and let me know if matters have improved. I'll wait to see the buildbot reports before closing this bug. (Surprisingly, we don't seem to have a FreeBSD buildbot, though we do have an OpenBSD one that didn't seem to have a problem with the original version of the mailbox code.) -- Comment By: A.M. Kuchling (akuchling) Date: 2006-06-26 08:56 Message: Logged In: YES user_id=11375 Interesting. The man page for flock() on my Linux system says of flock() in a certain version: This yields true BSD semantics: there is no interaction between the types of lock placed by flock(2) and fcntl(2), and flock(2) does not detect deadlock. Apparently this is out of date, and placing two locks is harmful. I think it's best to just use one set of locking primitives, and that one should be lockf(); the flock() calls should be removed. -- Comment By: Neal Norwitz (nnorwitz) Date: 2006-06-25 15:52 Message: Logged In: YES user_id=33168 Andrew, are you helping maintain this module? David is correct that fcntl is mispelled in current svn. Not sure what to do about the locking issue. David, would it be possible for you to work on a patch to correct this problem? I'm not sure if any developer has access to a FreeBSD box similar to yours. (There are a couple that use FreeBSD though.) It would be great to come up with tests for this if the current ones don't stress this situation. -- You can respond by visiting: https://sourceforge.net/tracker/?func=detailatid=105470aid=1512163group_id=5470 ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[ python-Bugs-1512163 ] mailbox (2.5b1): locking doesn't work (esp. on FreeBSD)
Bugs item #1512163, was opened at 2006-06-25 08:38 Message generated for change (Comment added) made by nnorwitz You can respond by visiting: https://sourceforge.net/tracker/?func=detailatid=105470aid=1512163group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Python Library Group: Python 2.5 Status: Open Resolution: None Priority: 5 Submitted By: David Watson (baikie) Assigned to: A.M. Kuchling (akuchling) Summary: mailbox (2.5b1): locking doesn't work (esp. on FreeBSD) Initial Comment: fcntl/flock() locking of mailboxes is actually disabled due to a typo: --- mailbox.py.orig +++ mailbox.py @@ -15,7 +15,7 @@ import rfc822 import StringIO try: -import fnctl +import fcntl except ImportError: fcntl = None However, once enabled, it doesn't work on FreeBSD - the lock() method always raises ExternalClashError (tested on 5.3). This is because flock(), lockf() and fcntl locking share the same underlying mechanism on this system (and probably others), and so the second lock can never be acquired. -- Comment By: Neal Norwitz (nnorwitz) Date: 2006-06-26 20:21 Message: Logged In: YES user_id=33168 Is the optional parameter necessary if you use inheritance (assuming there's a class/object in there)? Maybe it would be better to have a subclass? Would that be better for us to provide or just to add something to the docs that a user could make a subclass to work around locking issues? I haven't looked at the issues, but if we really need an API change, I'm probably ok with it since it would seem to be quite small. If an API change is really required (doc isn't sufficient), I'd like it to go in ASAP. We could doc for 2.5 and if there's a problem fix for 2.6. -- Comment By: A.M. Kuchling (akuchling) Date: 2006-06-26 16:24 Message: Logged In: YES user_id=11375 Hm, this looks messy; I doubt the module can know if you want lockf() or fcntl() locking. Maybe Mailbox.lock() should grow an optional parameter that lets you specify the locking mechanism to use -- lockf, fcntl, or both. (But defaulting to what?) On the other hand, maybe it's too late to go making API changes for beta2 (but I suspect it would be OK in this case). -- Comment By: David Watson (baikie) Date: 2006-06-26 15:56 Message: Logged In: YES user_id=1504904 Yeah, it works fine now (locks successfully when the mailbox isn't locked by another process, and fails when it is locked), although I was reminded that the Python install process didn't work properly (I've submitted another bug report). However, I should point out that flock() locking is required when working with qmail on (for example) Linux - flock() is the only locking mechanism it uses, and as noted below, flock() on Linux is independent from fcntl/lockf(). -- Comment By: A.M. Kuchling (akuchling) Date: 2006-06-26 06:27 Message: Logged In: YES user_id=11375 Typo fixed in rev. 47099; flock() calls removed in rev. 47100; a test for lock conflicts added in rev. 47101. David, please try out the current SVN head and let me know if matters have improved. I'll wait to see the buildbot reports before closing this bug. (Surprisingly, we don't seem to have a FreeBSD buildbot, though we do have an OpenBSD one that didn't seem to have a problem with the original version of the mailbox code.) -- Comment By: A.M. Kuchling (akuchling) Date: 2006-06-26 05:56 Message: Logged In: YES user_id=11375 Interesting. The man page for flock() on my Linux system says of flock() in a certain version: This yields true BSD semantics: there is no interaction between the types of lock placed by flock(2) and fcntl(2), and flock(2) does not detect deadlock. Apparently this is out of date, and placing two locks is harmful. I think it's best to just use one set of locking primitives, and that one should be lockf(); the flock() calls should be removed. -- Comment By: Neal Norwitz (nnorwitz) Date: 2006-06-25 12:52 Message: Logged In: YES user_id=33168 Andrew, are you helping maintain this module? David is correct that fcntl is mispelled in current svn. Not sure what to do about the locking issue. David, would it be possible for you to work on a patch to correct this problem? I'm not sure if any developer has access to a FreeBSD box similar to yours. (There are a couple that use FreeBSD though.) It would be great to come up with tests for this if the current ones don't stress this situation.
[ python-Bugs-1512163 ] mailbox (2.5b1): locking doesn't work (esp. on FreeBSD)
Bugs item #1512163, was opened at 2006-06-25 16:38 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detailatid=105470aid=1512163group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Python Library Group: Python 2.5 Status: Open Resolution: None Priority: 5 Submitted By: David Watson (baikie) Assigned to: Nobody/Anonymous (nobody) Summary: mailbox (2.5b1): locking doesn't work (esp. on FreeBSD) Initial Comment: fcntl/flock() locking of mailboxes is actually disabled due to a typo: --- mailbox.py.orig +++ mailbox.py @@ -15,7 +15,7 @@ import rfc822 import StringIO try: -import fnctl +import fcntl except ImportError: fcntl = None However, once enabled, it doesn't work on FreeBSD - the lock() method always raises ExternalClashError (tested on 5.3). This is because flock(), lockf() and fcntl locking share the same underlying mechanism on this system (and probably others), and so the second lock can never be acquired. -- You can respond by visiting: https://sourceforge.net/tracker/?func=detailatid=105470aid=1512163group_id=5470 ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[ python-Bugs-1512163 ] mailbox (2.5b1): locking doesn't work (esp. on FreeBSD)
Bugs item #1512163, was opened at 2006-06-25 08:38 Message generated for change (Comment added) made by nnorwitz You can respond by visiting: https://sourceforge.net/tracker/?func=detailatid=105470aid=1512163group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Python Library Group: Python 2.5 Status: Open Resolution: None Priority: 5 Submitted By: David Watson (baikie) Assigned to: A.M. Kuchling (akuchling) Summary: mailbox (2.5b1): locking doesn't work (esp. on FreeBSD) Initial Comment: fcntl/flock() locking of mailboxes is actually disabled due to a typo: --- mailbox.py.orig +++ mailbox.py @@ -15,7 +15,7 @@ import rfc822 import StringIO try: -import fnctl +import fcntl except ImportError: fcntl = None However, once enabled, it doesn't work on FreeBSD - the lock() method always raises ExternalClashError (tested on 5.3). This is because flock(), lockf() and fcntl locking share the same underlying mechanism on this system (and probably others), and so the second lock can never be acquired. -- Comment By: Neal Norwitz (nnorwitz) Date: 2006-06-25 12:52 Message: Logged In: YES user_id=33168 Andrew, are you helping maintain this module? David is correct that fcntl is mispelled in current svn. Not sure what to do about the locking issue. David, would it be possible for you to work on a patch to correct this problem? I'm not sure if any developer has access to a FreeBSD box similar to yours. (There are a couple that use FreeBSD though.) It would be great to come up with tests for this if the current ones don't stress this situation. -- You can respond by visiting: https://sourceforge.net/tracker/?func=detailatid=105470aid=1512163group_id=5470 ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
