Re: [Python-Dev] Does Python need a file locking module (slightly higher level)?

[skip]

>> I'm always daunted by the prospect of trying to implement file
>> locking.

Barry> If you want something like this, you might start by looking at
Barry> Mailman's LockFile.py.

If I interpret the Python documentation for os.link correctly, the scheme
used by Mailman won't work on Windows (os.link isn't advertised as being
available there).  Nevertheless, the pointer to the Linux open(2) man page
was a good start.  Implementing something for Unix-y systems is not too
difficult using that advice.

Jean-Paul Calderone sent me a pointer to Twisted's file locking module.  I'm
still trying to figure out exactly what it does on Windows.  Something about
making and populating directories.  (os.mkdir is the replacement for
os.link?)

Benji York referred me to zc.lockfile.  That appears to use fcntl.flock.
Based on Jean-Paul's response it seems the jury is still out on whether
fcntl.flock works over NFS.  zc.lockfile at least has something specifically
for Windows.  Whether or not msvcrt.locking() works on networked file
systems remains to be seen.

Mailman's lockfile makes provision to block for a user-specified period of
time.  The other's push the waiting back onto the calling code.

It's not clear that any of these implementations is going to be perfect.
Maybe none ever will be.  In his reply Jean-Paul made this comment:

It might be nice to have something like that in the standard library,
but it's very simple once you know what to do.

I'm not so sure about the "very simple" part, especially if you aren't
familiar with all the ins and outs of the different platforms.  The fact
that the first three bits of code I was referred to were implemented by
three significant Python tools/platforms and that all are different in some
significant ways suggests that there is some both an underlying need for a
file locking mechanism but with a lack of consensus about the best way to
implement the mother-of-all-file-locking schemes for Python.  Maybe the best
place for this is in the distribution.  PEP?

Skip
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Does Python need a file locking module (slightly higher level)?

[Jean-Paul Calderone]

On Tue, 23 Oct 2007 01:11:39 +0100, Jon Ribbens <[EMAIL PROTECTED]> wrote:
>On Tue, Oct 23, 2007 at 12:29:35PM +1300, Greg Ewing wrote:
>> [EMAIL PROTECTED] wrote:
>> > Does fcntl.flock work over NFS and SMB and on Windows?
>>
>> I don't think file locking will ever work over NFS, since
>> it's a stateless protocol by design, and locking would
>> require maintaining state on the server.
>
>You can do file locking over NFS, that's one of the reasons people
>use fcntl. It uses an RPC side channel separate to the main NFS
>protocol.

You can do it.  It just doesn't work.  (You could say the same about
regular read and write operations for many NFS implementations, though)

Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Does Python need a file locking module (slightly higher level)?

[Jon Ribbens]

On Tue, Oct 23, 2007 at 12:29:35PM +1300, Greg Ewing wrote:
> [EMAIL PROTECTED] wrote:
> > Does fcntl.flock work over NFS and SMB and on Windows?
> 
> I don't think file locking will ever work over NFS, since
> it's a stateless protocol by design, and locking would
> require maintaining state on the server.

You can do file locking over NFS, that's one of the reasons people
use fcntl. It uses an RPC side channel separate to the main NFS
protocol.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Does Python need a file locking module (slightly higher level)?

[Jon Ribbens]

On Tue, Oct 23, 2007 at 12:16:41PM +1300, Greg Ewing wrote:
> [EMAIL PROTECTED] wrote:
> > This interface follows the completely stupid semantics of System V and
> > IEEE Std 1003.1-1988 (``POSIX.1'') that require that all locks
> > associated with a file for a given process are removed when any file
> > descriptor for that file is closed by that process  Flock(2) is
> > recommended for applications that want to ensure the integrity of their
> > locks when using library routines or wish to pass locks to their
> > children.
> > 
> > I guess the BSD folks were a bit upset when they wrote that.
> 
> That sounds more like something written by the GNU folks than
> the BSD folks.

It's from BSD. The earliest I can find it is 4.4BSD. It's still in the
latest versions of OpenBSD, NetBSD and FreeBSD.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 2.5.1 ported to z/OS and EBCDIC

[Terry Reedy]

"Lauri Alanko" <[EMAIL PROTECTED]> wrote in message 
news:[EMAIL PROTECTED]
| All this just shows that treating plain octet sequences as "strings"
| simply won't work in the long run. You have to have separate type for
| _textual_ data (i.e. Unicode strings, in Python), and encode and decode
| between those and octet sequences using some _explicit_ encoding. Of
| course, all non-English-speaking people have been keenly aware of this
| already for ages. The relative universality of ASCII is an exception
| amongst encodings rather than the norm. It's only reasonable to require
| English text to require the same attention to encodings as all the other
| languages.

Setting aside any debate over 'reasonable', I believe the above pretty well 
matchess the design for Py3, except that Guido prefers 'byte' to 'octet'. 
You may end up preferring Py3 for your port.  3.0a2 should be out in a few 
weeks.

| PICKLING
| In the end, for now, I made protocol 0 textual, and disabled support for
| protocol versions > 0 on non-ASCII platforms.

I believe that the plan for pickle in 3.0 is to read any protocol but only 
write the latest (2?), but perhaps this was only a suggestion.




___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 2.5.1 ported to z/OS and EBCDIC

[Greg Ewing]

Lauri Alanko wrote:
> In the end, for now, I made protocol 0 textual,

That could be a mistake. I believe there are some objects,
such as array.array, that use a binary format for pickling
even in protocol 0.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Does Python need a file locking module (slightly higher level)?

[Greg Ewing]

[EMAIL PROTECTED] wrote:
> Does fcntl.flock work over NFS and SMB and on Windows?

I don't think file locking will ever work over NFS, since
it's a stateless protocol by design, and locking would
require maintaining state on the server.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Does Python need a file locking module (slightly higher level)?

[Greg Ewing]

[EMAIL PROTECTED] wrote:
> This interface follows the completely stupid semantics of System V and
> IEEE Std 1003.1-1988 (``POSIX.1'') that require that all locks
> associated with a file for a given process are removed when any file
> descriptor for that file is closed by that process  Flock(2) is
> recommended for applications that want to ensure the integrity of their
> locks when using library routines or wish to pass locks to their
> children.
> 
> I guess the BSD folks were a bit upset when they wrote that.

That sounds more like something written by the GNU folks than
the BSD folks.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Patch for adding offset to mmap

[Travis Oliphant]

Hi all,

I think the latest patch for fixing Issue 708374 (adding offset to mmap)
should be committed to SVN.

I will do it, if nobody opposes the plan.  I think it is a very
important addition and greatly increases the capability of the mmap module.

Thanks,

-Travis Oliphant


P.S.  Initially sent this to the wrong group (I've been doing that a lot 
lately --- too many groups seen through gmane...).  Apologies for 
multiple postings.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Does Python need a file locking module (slightly higher level)?

[Benji York]

Barry Warsaw wrote:
> On Oct 22, 2007, at 8:15 AM, [EMAIL PROTECTED] wrote:
> 
>> I'm always daunted by the prospect of trying to implement file  
>> locking.

> If you want something like this, you might start by looking at  
> Mailman's LockFile.py.

Also related is the very simple zc.lockfile: 
http://pypi.python.org/pypi/zc.lockfile
-- 
Benji York
http://benjiyork.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Does Python need a file locking module (slightly higher level)?

[Barry Warsaw]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On Oct 22, 2007, at 8:15 AM, [EMAIL PROTECTED] wrote:

> I'm always daunted by the prospect of trying to implement file  
> locking.
> This just came up again in SpamBayes where we have never protected our
> pickle files from corruption when multiple processes access them
> simultaneously.  The presence of networked file systems and
> platform-independent locks make it a nasty little problem.  Maybe  
> I'm just
> showing my age.  Does fcntl.flock work over NFS and SMB and on  
> Windows?  If
> this is still as much of a mess as I remember, should Python provide a
> simple file locking module in the standard distribution?

If you want something like this, you might start by looking at  
Mailman's LockFile.py.  It has a particular set of semantics (such as  
lock breaking) that you might not be interested in, but it is, or can  
be, mostly de-Mailmanized for use as a general library module.  In  
the particular use case it is designed for, it's been quite stable  
for many years.  Essentially it provides an NFS-safe lock file  
implementation.

- -Barry

http://codebrowse.launchpad.net/~mailman-coders/mailman/3.0/annotate/ 
barry%40python.org-20071011032203-w1j8qrmtlpkrvay4? 
file_id=mailmanlockfile.py-20070507165525-0o0kligrooe34vyc-172

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.7 (Darwin)

iQCVAwUBRxyngHEjvBPtnXfVAQKPOwP+JhuIC2LiOsHDPLtAft4bSMaYC1qfVJqG
q6SXFc8yJauE9zKttPcn9kkbgONj3RYbDJ9qW4aVA7fJfHEiRDbW8omp/e7rTELl
fIonBDnIk5XEo5bL/JslMudgInOa6BY7yGzCKjaRRy19wSmOZ8ptroXfOvLgqF+e
n7WVkh82sD8=
=aFzw
-END PGP SIGNATURE-
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Python 2.5.1 ported to z/OS and EBCDIC

[Lauri Alanko]

Hello.

Based on Jean-Yves Mengant's work on previous versions, I have ported
Python 2.5.1 to z/OS. A patch against current svn head is attached to
. The same patch should work with very
little changes also against pristine 2.5.1 sources. (The only failing
hunk is for Modules/makesetup, and it is quite trivial.)

I have no opinion on whether the patch should eventually be incorporated
into the main distribution. The port was motivated by internal reasons,
and I'm merely offering it as a community service to anyone else who
might be interested. If Jean-Yves wishes to distribute it from his
z/OS-page, that is fine with me. In general, anyone can do what they
want with the patch, but please give credit.

I'll describe some of the porting issues below.


CHARACTER SETS
==

The biggest, major difficulty with z/OS is of course the character set.
There are lots of ASCII-dependencies in Python code, and z/OS uses
CP1047, an EBCDIC variant, which is utterly incompatible with ASCII.

There are two possible approaches in this situation. One is to keep on
using ASCII as the execution character set (and also as the default
encoding of string objects), and to add conversion support to everywhere
where we do text-based I/O, so that communication with the external
world still happens in EBCDIC. This was feasible since the z/OS C
compiler does support ASCII as the execution character set. (The source
character set would still remain EBCDIC, though. If you've ever wondered
why the C standard makes a distinction between these, here's a prime
example of a situation where they're different.)

However, I decided against this approach. The I/O conversions would have
been deeply magical, and would have required classic "text mode vs.
binary mode" -crap, which would be rather confusing.

Instead, I followed Jean-Yves' example and kept Python as a "native"
EBCDIC application: input, 8-bit data is treated by default as EBCDIC
everywhere. This only required fixing various ASCII-specific bits in the
code, e.g. stuff like this (in PyString_DecodeEscape):

-   else if (c < ' ' || c >= 0x7f)
+   else if (!isprint((unsigned char) c))

Of course, now this allows unescaped printing of characters if they are
printable in the platform's encoding even if they wouldn't be printable
in ASCII. I'm not sure if this is desirable or not. It would be simple
to fix this so that only characters in the ASCII _character set_ are
displayed varbatim.

A result of making strings EBCDIC-native is that it breaks any code that
depends on string literals being in ASCII. This probably applies to most
network protocol implementations written in Python. On the other hand,
making string literals use ASCII would break code that does ordinary
text processing on local files. Damned if you do, damned if you don't.

The real issue is that strings in Python are rather underspecified.
String objects are really just octet sequences without any _inherent_
textual interpretation for them. This is apparent from the fact that
strings are what are read from and written to binary files, and also
what unicode strings are encoded to and decoded from. However, Python
syntax allows specifying an octet sequence with a _character_ sequence
(i.e. a string literal), and the relationship between the source
characters and the resulting octets has been left implicit. So
programmers aren't really encouraged to think about character set issues
and the end result is code that only works on a platform that uses ASCII
everywhere.

Python already has the property that the meaning of a source file
depends on its encoding: if I write a string literal with some latin-1
characters, the resulting octet sequence depends on whether my source
was encoded in latin-1 or utf-8. I'm not sure if this is a good idea,
but my approach with the z/OS port continues the tradition: when your
source is in EBCDIC, the string literals get encoded in EBCDIC.

All this just shows that treating plain octet sequences as "strings"
simply won't work in the long run. You have to have separate type for
_textual_ data (i.e. Unicode strings, in Python), and encode and decode
between those and octet sequences using some _explicit_ encoding. Of
course, all non-English-speaking people have been keenly aware of this
already for ages. The relative universality of ASCII is an exception
amongst encodings rather than the norm. It's only reasonable to require
English text to require the same attention to encodings as all the other
languages.


UNICODE
---

The biggest hurdle by far (at least LoC-wise) in the porting was
Unicode. The code assumed that the execution character set was not only
ASCII, but ISO-8859-1, since there was lots of casting back and forth
between Py_UNICODE and char. I added the following conversion operations
into unicodeobject.h:

#ifdef Py_CHARSET_ASCII
# define Py_UNICODE_FROM_CHAR(c) ((Py_UNICODE)(unsigned char)(c))
# define Py_UNICODE_AS

[Python-Dev] Does Python need a file locking module (slightly higher level)?

[skip]

I'm always daunted by the prospect of trying to implement file locking.
This just came up again in SpamBayes where we have never protected our
pickle files from corruption when multiple processes access them
simultaneously.  The presence of networked file systems and
platform-independent locks make it a nasty little problem.  Maybe I'm just
showing my age.  Does fcntl.flock work over NFS and SMB and on Windows?  If
this is still as much of a mess as I remember, should Python provide a
simple file locking module in the standard distribution?

Side note: While reading the fcntl man page on my Mac I came across this
text in the description of F_GETLK/F_SETLK/F_SETLKW.

This interface follows the completely stupid semantics of System V and
IEEE Std 1003.1-1988 (``POSIX.1'') that require that all locks
associated with a file for a given process are removed when any file
descriptor for that file is closed by that process  Flock(2) is
recommended for applications that want to ensure the integrity of their
locks when using library routines or wish to pass locks to their
children.

I guess the BSD folks were a bit upset when they wrote that.

Skip
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com