Re: [Python-Dev] 2.4 vs Windows vs bsddb [correction]
I wrote: > You omitted to state an assumption that sizeof(errTxt) >= 4, since size_t > (and the constant 4) are unsigned. Sorry, the constant '4' is signed, but sizeof(errTxt) - 4 can nevertheless wrap around unless sizeof(errTxt) >= 4. -- David Hopwood <[EMAIL PROTECTED]> ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 2.4 vs Windows vs bsddb
Tim Peters wrote:
> Given that, the assert() in question looks fine to me:
>
> if (_db_errmsg[0] && bytes_left < (sizeof(errTxt) - 4)) {
> bytes_left = sizeof(errTxt) - bytes_left - 4 - 1;
> assert(bytes_left >= 0);
>
> We can't get into the block unless
>
> bytes_left < sizeof(errTxt) - 4
>
> is true. Subtracting bytes_left from both sides, then swapping LHS and RHS:
>
> sizeof(errTxt) - bytes_left - 4 > 0
>
> which implies
>
> sizeof(errTxt) - bytes_left - 4 >= 1
>
> Subtracting 1 from both sides:
>
> sizeof(errTxt) - bytes_left - 4 - 1 >= 0
>
> And since the LHS of that is the new value of bytes_left, it must be true that
>
> bytes_left >= 0
>
> Either that, or the original author (and me, just above) made an error
> in analyzing what must be true at this point.
You omitted to state an assumption that sizeof(errTxt) >= 4, since size_t
(and the constant 4) are unsigned. Also bytes_left must initially be nonnegative
so that the subexpression 'sizeof(errTxt) - bytes_left' cannot overflow.
--
David Hopwood <[EMAIL PROTECTED]>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pep 353: Py_ssize_t advice
Martin v. Löwis wrote: > David Abrahams schrieb: > >>(C++ allows restating of typedefs; if C allows it, that should be >>something like): > > C also allows this; [...] This is nitpicking, since you agreed the change to the PEP, but are you sure that C allows this? >From C99 + TC1 + TC2 (http://www.open-std.org/JTC1/SC22/WG14/www/standards): # 6.2.2 Linkages of identifiers # # 6 The following identifiers have no linkage: an identifier declared #to be anything other than an object or a function; [...] (i.e. typedef identifiers have no linkage) # 6.7 Declarations # # Constraints # 3 If an identifier has no linkage, there shall be no more than one #declaration of the identifier (in a declarator or type specifier) #with the same scope and in the same name space, except for tags as #specified in 6.7.2.3. # 6.7.2.3 Tags # # Constraints # 1 A specific type shall have its content defined at most once. (There is nothing else in 6.7.2.3 that applies to typedefs.) Since 6.7 (3) and 6.7.2.3 (1) are constraints, I read this as saying that a C99 implementation must produce a diagnostic if a typedef is redeclared in the same scope. If the program is run despite the diagnostic, its behaviour is undefined. Several C compilers I've used in the past have needed the idempotence guard on typedefs, in any case. -- David Hopwood <[EMAIL PROTECTED]> ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Unicode Imports
Martin v. Löwis wrote: > David Hopwood schrieb: > >>On Windows, file system pathnames can contain arbitrary Unicode characters >>(well, almost). Despite the existence of "ANSI" filesystem APIs, and >>regardless of what 'sys.getfilesystemencoding()' returns, the underlying >>file system encoding for NTFS and FAT filesystems is UTF-16LE. >> >>Thus, either: >> - the fact that sys.getfilesystemencoding() returns a non-Unicode encoding >> on Windows is a bug, or >> - any program that relies on sys.getfilesystemencoding() being able to >> encode arbitrary Windows pathnames has a bug. >> >>We need to decide which of these is the case. > > There is a third option: > - the operating system has a bug This behaviour is by design. If it is a bug, then it is a "won't ever fix -- no way, no how" bug, that Python must accomodate if it is to properly support Unicode on Windows. > It is actually this option that rules out the other two. > sys.getfilesystemencoding() returns "mbcs" on Windows, which means > CP_ACP. The file system encoding is an encoding that converts a > file name into a byte string. Unfortunately, on Windows, there are > file names which cannot be converted into a byte string in a standard > manner. This is an operating system bug (or mis-design; they should > have chosen UTF-8 as the byte encoding of file names, instead of > making it depend on the system locale, but they of course did so > for backwards compatibility with Windows 3.1 and 9x). Although UTF-8 was invented (in September 1992) technically before the release of the first version of NT supporting NTFS (NT 3.1 in July 1993), it had not been invented before the decision to use Unicode in NTFS, or in Windows NT's file APIs, had been made. (I believe OS/2 HPFS had not supported Unicode, even though NTFS was otherwise almost identical to it.) At that time, the decision to use Unicode at all was quite forward-looking; the final version of Unicode 1.0 had only been published in June 1992 (although it had been approved earlier; see <http://www.unicode.org/history/>). UTF-8 was only officially added to the Unicode standard in an appendix of Unicode 2.0 (published July 1996), and only given essentially equal status to UTF-16 and UTF-32 in Unicode 3.0 (September 1999). > As a side note: every encoding in Python is a Unicode encoding; > so there aren't any "non-Unicode encodings". It was clear from context that I meant "encoding capable of representing all Unicode characters". > Programs that rely on sys.getfilesystemencoding() being able to > represent arbitrary file names on Windows might have a bug; > programs that rely on sys.getfilesystemencoding() being able > to encode all elements of sys.path do not (at least not for > Python 2.5 and earlier). Elements of sys.path can be Unicode strings in Python 2.5, and should be pathnames supported by the underlying OS. Where is it documented that there is any further restriction on them? And why should there be any further restriction on them? -- David Hopwood <[EMAIL PROTECTED]> ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Unicode Imports
Nick Coghlan wrote: > David Hopwood wrote: >> Martin v. Löwis wrote: >>> Nick Coghlan schrieb: >>> >>>> So this is taking something that *already works properly on POSIX >>>> systems* and making it work on Windows as well. >>> >>> I doubt it does without side effects. For example, an application that >>> would go through sys.path, and encode everything with >>> sys.getfilesystemencoding() currently works, but will break if the patch >>> is applied and non-mbcs strings are put on sys.path. >> >> Huh? It won't break on any path for which it is not already broken. >> >> You seem to be saying "Paths with non-mbcs strings shouldn't work on >> Windows, because they haven't worked in the past." > > I think MvL is looking at it from the point of view of consumers of the > list of strings in sys.path, such as PEP 302 importer and loader > objects, and tools like module_finder. Currently, the list of values in > sys.path is limited to: > > 1. 8-bit strings > 2. Unicode strings containing only characters which can be encoded using > the default file system encoding On Windows, file system pathnames can contain arbitrary Unicode characters (well, almost). Despite the existence of "ANSI" filesystem APIs, and regardless of what 'sys.getfilesystemencoding()' returns, the underlying file system encoding for NTFS and FAT filesystems is UTF-16LE. Thus, either: - the fact that sys.getfilesystemencoding() returns a non-Unicode encoding on Windows is a bug, or - any program that relies on sys.getfilesystemencoding() being able to encode arbitrary Windows pathnames has a bug. We need to decide which of these is the case. -- David Hopwood <[EMAIL PROTECTED]> ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Unicode Imports
Martin v. Löwis wrote: > Nick Coghlan schrieb: > >>So this is taking something that *already works properly on POSIX >>systems* and making it work on Windows as well. > > I doubt it does without side effects. For example, an application that > would go through sys.path, and encode everything with > sys.getfilesystemencoding() currently works, but will break if the patch > is applied and non-mbcs strings are put on sys.path. Huh? It won't break on any path for which it is not already broken. You seem to be saying "Paths with non-mbcs strings shouldn't work on Windows, because they haven't worked in the past." -- David Hopwood <[EMAIL PROTECTED]> ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: Problem withthe API for str.rpartition()
Barry Warsaw wrote: > The bias with these terms is clearly the English left-to-right > order. Actually, that brings up an interesting question: what would > happen if you called rpartition on a unicode string representing > Hebrew, Arabic, or other RTL language? Do partition and rpartition > suddenly switch directions? What happens is that rpartition searches the string backwards in logical order (i.e. left to right as the text is written, assuming it only contains Hebrew or Arabic letters, and not numbers or a mixture of scripts). But this is not "switching directions"; it's still searching backwards. You really don't want to think of bidirectional text in terms of presentation, when you're doing processing that should be independent of presentation. > If not, then I think left-sep-right are fine. If so, then yeah, we > probably need something else. +1 for (upto, sep, rest) -- and I think it should be in that order for both partition and rpartition. -- David Hopwood <[EMAIL PROTECTED]> ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals, threads, blocking C functions
Jean-Paul Calderone wrote: > On Mon, 04 Sep 2006 17:24:56 +0100, David Hopwood <[EMAIL PROTECTED]> wrote: > >>Jean-Paul Calderone wrote: >> >>>PyGTK would presumably implement its pending call callback by writing a >>>byte to a pipe which it is also passing to poll(). >> >>But doing that in a signal handler context invokes undefined behaviour >>according to POSIX. > > write(2) is explicitly listed as async-signal safe in IEEE Std 1003.1, 2004. I stand corrected. I must have misremembered this. -- David Hopwood <[EMAIL PROTECTED]> ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals, threads, blocking C functions
Jean-Paul Calderone wrote: > PyGTK would presumably implement its pending call callback by writing a > byte to a pipe which it is also passing to poll(). But doing that in a signal handler context invokes undefined behaviour according to POSIX. -- David Hopwood <[EMAIL PROTECTED]> ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Signals, threads, blocking C functions
Gustavo Carneiro wrote: > OK, let's review what we know about current python, signals, and threads: > > 1. Python launches threads without touching sigprocmask; > 2. Python installs signal handlers for all signals; > 3. Signals can be delivered to any thread, let's assume (because > of point #1 and not others not mentioned) that we have no control over > which threads receive which signals, might as well be random for all > we know; > 4. Python signal handlers do almost nothing: just sets a flag, > and calls Py_AddPendingCall, to postpone the job of handling a signal > until a "safer" time. > 5. The function Py_MakePendingCalls() should eventually get > called at a "safer" time by user or python code. > 6. It follows that until Py_MakePendingCalls() is called, the > signal will not be handled at all! > > Now, back to explaining the problem. > > 1. In PyGTK we have a gobject.MainLoop.run() method, which blocks > essentially forever in a poll() system call, and only wakes if/when it > has to process timeout or IO event; > 2. When we only have one thread, we can guarantee that e.g. > SIGINT will always be caught by the thread running the > g_main_loop_run(), so we know poll() will be interrupted and a EINTR > will be generated, giving us control temporarily back to check for > python signals; > 3. When we have multiple thread, we cannot make this assumption, > so instead we install a timeout to periodically check for signals. > > We want to get rid of timeouts. Now my idea: add a Python API to say: > "dear Python, please call me when you start having pending calls, > even if from a signal handler context, ok?" What can be safely done from a signal handler context is *very* limited. Calling back arbitrary Python code is certainly not safe. Reliable asynchronous interruption of arbitrary code is a difficult problem, but POSIX and POSIX implementations botch it particularly badly. I don't know how to implement what you want here, but I'd endorse the comments of Nick Maclaren and Antony Baxter against making precipitate changes. -- David Hopwood <[EMAIL PROTECTED]> ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adding an rslice() builtin?
Nick Coghlan wrote: > A discussion on the py3k list reminded me that translating a forward slice > into a reversed slice is significantly less than obvious to many people. Not > only do you have to negate the step value and swap the start and stop values, > but you also need to subtract one from each of the step values, and ensure > the > new start value was actually in the original slice: > >reversed(seq[start:stop:step]) becomes > seq[(stop-1)%abs(step):start-1:-step] > > An rslice builtin would make the latter version significantly easier to read: > >seq[rslice(start, stop, step)] Or slice.reversed(). -- David Hopwood <[EMAIL PROTECTED]> ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] gcc 4.2 exposes signed integer overflows
Jack Howarth wrote: > I believe some of the others here might be interested in > some other postings on the gcc mailing list regarding this issue > (which weren't cross-posted here)... > > http://gcc.gnu.org/ml/gcc/2006-08/msg00516.html > http://gcc.gnu.org/ml/gcc/2006-08/msg00517.html > > It makes clear that the impact of this change in gcc was considered > and the reasoning on their decision to follow the standard so closely. > > http://gcc.gnu.org/ml/gcc/2005-06/msg01238.html > > Just so other beside Guido see those. This discussion just highlights how different the perspective of much of the C community (especially its language committee and compiler writers) is on error handling and language safety, relative to the perspective of the community that uses Python and other memory-safe languages. The C perspective is that programs have to be correct, and if they're not correct, it basically doesn't matter what happens. The safe language perspective is that of course we try to write correct programs, but if a program is not correct, then it *really matters* what the error behaviour is. Ideally, it should be something that facilitates debugging and that doesn't silently propagate an incorrect result. Undefined behaviour is right out. For example, from <http://gcc.gnu.org/ml/gcc/2006-08/msg00522.html>: # The gcc developers always face two competing constituencies: do not # change the compiler behaviour so that existing code which relies on # undefined behaviour continues to work, and do change the compiler # behaviour so that new code which does not rely on undefined behaviour # runs faster. In general, being compiler developers, we come down on # the side of making code which does not rely on undefined behaviour run # faster. ... and rarely even consider whether the behaviour *should* have been undefined or not. (There are many people who use C but do not agree with that perspective on error handling -- I'm one of them. These people often devise elaborate error handling frameworks, and compile at low optimization levels when it doesn't hurt performance.) -- David Hopwood <[EMAIL PROTECTED]> ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] gcc 4.2 exposes signed integer overflows
Thomas Wouters wrote: > On 8/26/06, David Hopwood <[EMAIL PROTECTED]> wrote: > >> CPython should be fixed anyway. The correct fix is >> "if (y == -1 && x < 0 && (unsigned long)x == -(unsigned long)x)". > > Why not just "... && x == LONG_MIN"? Because the intent is to check that x / y does not overflow a long, and x == LONG_MIN would not cause an overflow on 1's complement or sign-magnitude systems. (CPython has probably only been tested on 2's complement systems anyway, but if we're going to be pedantic about depending only on things in the C standard...) -- David Hopwood <[EMAIL PROTECTED]> ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] gcc 4.2 exposes signed integer overflows
Guido van Rossum wrote: > On 8/26/06, Jack Howarth <[EMAIL PROTECTED]> wrote: > >> I discovered that gcc 4.2 exposes a flaw with >>signed integer overflows in python. This bug and the >>necessary fix has been discussed in detail on the gcc >>mailing list. I have filed a detailed bug report and >>the recommended patch proposed by the gcc developers. >>This problem should be addressed BEFORE python 2.5 is >>released. The bug report is... >> >>[ 1545668 ] gcc trunk (4.2) exposes a signed integer overflows >> >>in the python sourceforge bug tracker. Thanks in advance >>for attempting to fix this before Python 2.5 is released. > > I'm not sure I follow why this isn't considered a regression in GCC. > Clearly, on all current hardware, x == -x is also true for the most > negative int (0x8000 on a 32-bit box). Why is GCC attempting to > break our code (and then blaming us for it!) by using the C standard's > weaselwords that signed integer overflow is undefined, despite that it > has had a traditional meaning on 2's complement hardware for many > decades? If GCC starts to enforce everything that the C standard says > is undefined then very few programs will still work... +1. It's clearly undefined behaviour, but that doesn't mean that it is a good idea for gcc to be making changes that could silently break a lot of code, in pursuit of a microoptimization that is unlikely to significantly help performance. CPython should be fixed anyway. The correct fix is "if (y == -1 && x < 0 && (unsigned long)x == -(unsigned long)x)". -- David Hopwood <[EMAIL PROTECTED]> ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] dict containment annoyance
tomer filiba wrote:
>>>> a={1:2, 3:4}
>>>> [] in a
>
> Traceback (most recent call last):
> File "", line 1, in ?
> TypeError: list objects are unhashable
>
> imo, the expression should just evaluate to False instead of raising an
> exception. it's a question of semantics -- i asked whether the object
> (a list, in this case) is contained in the dict. i didn't ask whether
> it's hashable or not.
However, if it isn't hashable, asking whether it is in a dict is very
likely to be a programming error.
--
David Hopwood <[EMAIL PROTECTED]>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] unicode hell/mixing str and unicode as dictionary keys
Martin v. Löwis wrote: > David Hopwood schrieb: >>Michael Foord wrote: >>>David Hopwood wrote:[snip..] >>> >>>>>>we should, of course, continue to use the one we always used (for >>>>>>"ascii", there is no difference between the two). >>>>> >>>>>+1 >>>>> >>>>>This seems the most (only ?) logical solution. >>>> >>>>No; always considering Unicode and non-ASCII byte strings to be distinct >>>>is just as logical. >> >>I think you must have misread my comment: > > Indeed. The misunderstanding originates from your sentence starting with > "no", when, in fact, you seem to be supporting the proposal I made. I had misunderstood what the existing Python behaviour is. I now think the current behaviour (which uses "B.decode(system_encoding) == U") is definitely a bad idea, especially in cases where the system encoding is not US-ASCII, but I agree that it can't be changed for 2.5. -- David Hopwood <[EMAIL PROTECTED]> ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] unicode hell/mixing str and unicode as dictionary keys
Michael Foord wrote: > David Hopwood wrote:[snip..] > >>>> we should, of course, continue to use the one we always used (for >>>> "ascii", there is no difference between the two). >>> >>> +1 >>> >>> This seems the most (only ?) logical solution. >> >> No; always considering Unicode and non-ASCII byte strings to be distinct >> is just as logical. > > Except there has been an implicit promise in Python for years now that > ascii byte-strings will compare equally to the unicode equivalent: lots > of code assumes this. I think you must have misread my comment: No; always considering Unicode and *non-ASCII* byte strings to be distinct is just as logical. This says nothing about comparing Unicode and ASCII byte strings. -- David Hopwood <[EMAIL PROTECTED]> ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] unicode hell/mixing str and unicode as dictionary keys
Michael Foord wrote: > Martin v. Löwis wrote: > >>[snip..] >>Expanding this view to Unicode should mean that a unicode >>string U equals a byte string B if >>U.encode(system_encode) == B or B.decode(system_encoding) == U, >>and that they don't equal otherwise (e.g. if the conversion >>fails with a "not convertible" exception). I disagree. Unicode strings should always be considered distinct from non-ASCII byte strings. Implicitly encoding or decoding in order to perform a comparison is a bad idea; it is expensive and will often do the wrong thing. The programmer should explicitly encode the Unicode string or decode the byte string before comparison (which one of these is correct is application-dependent). >>Which of the two conversions is selected is arbitrary; [...] It would not be arbitrary. In the common case where the byte encoding uses "precomposed" characters, using "U.encode(system_encoding) == B" will tend to succeed in more cases than "B.decode(system_encoding) == U", because alternative representations of the same abstract character in Unicode will be mapped to the same precomposed character. (Whether these are cases in which the comparison *should* succeed is, as I said above, application-dependent.) The special case of considering US-ASCII strings to compare equal to the corresponding Unicode string, is more reasonable than this would be for a general byte encoding, because: - it can be done with no (or only a trivial) conversion, - US-ASCII has no precomposed characters or combining marks, so it does not have multiple encodings for the same abstract character, - Unicode has a US-ASCII subset that uses exactly the same encoding model as US-ASCII (whereas in general, a byte encoding might use an arbitrarily different encoding model to Unicode, as for example is the case for ISCII). >>we should, of course, continue to use the one we always used (for >>"ascii", there is no difference between the two). > > +1 > > This seems the most (only ?) logical solution. No; always considering Unicode and non-ASCII byte strings to be distinct is just as logical. -- David Hopwood <[EMAIL PROTECTED]> ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New miniconf module
Nick Coghlan wrote: > Sylvain Fourmanoit wrote: >>Armin Rigo wrote: >> >>>If it goes in that direction, I'd suggest to rename the module to give >>>it a name closer to existing persistence-related modules already in the >>>stdlib. >> >>I am not especially fond of the current miniconf name either; I didn't >>find something more suitable, yet evocative of what it does; I would be >>glad to hear any suggestion you or the rest of the developers would have. > > pyson :) Following the pattern of JSON, it would be "PYON" (PYthon Object Notation). -- David Hopwood <[EMAIL PROTECTED]> ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bad interaction of __index__ and sequence repeat
Martin v. Löwis wrote: > Travis Oliphant wrote: > >>I say it's a bug that should be fixed. Don't clear the error, raise it. > > Several people have said this, but I don't think it can work. > > If you raise an OverflowError in __index__, the slicing code cannot know > whether this meant as overflow or underflow (in a signed sense). Why not use IndexError for an underflow, and OverflowError for an overflow? -- David Hopwood <[EMAIL PROTECTED]> ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bad interaction of __index__ and sequence repeat
Armin Rigo wrote: > Hi, > > There is an oversight in the design of __index__() that only just > surfaced :-( It is responsible for the following behavior, on a 32-bit > machine with >= 2GB of RAM: > > >>> s = 'x' * (2**100) # works! > >>> len(s) > 2147483647 > > This is because PySequence_Repeat(v, w) works by applying w.__index__ in > order to call v->sq_repeat. However, __index__ is defined to clip the > result to fit in a Py_ssize_t. Clipping the result sounds like it would *never* be a good idea. What was the rationale for that? It should throw an exception. -- David Hopwood <[EMAIL PROTECTED]> ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Internal namespace proposal
Richard Jones wrote: > On 27/07/2006, at 12:19 PM, David Hopwood wrote: > >> A restricted interpreter refuses access to any object attribute or >> method with a name beginning with '_' (by throwing a new exception type >> 'InternalAccessException'), unless the access is from a method and its >> static target is that method's first argument variable. >> >> Also, a restricted interpreter refuses access to any module-global >> variable or module-global function with a name beginning with '_' (by >> throwing 'InternalAccessException'), unless the access is statically >> from the same module. > > Note that this is a rule that Zope enforces in its restricted environment. Is that documented anywhere? -- David Hopwood <[EMAIL PROTECTED]> ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Internal namespace proposal
Greg Ewing wrote: > David Hopwood wrote: > >> Inheritance should be defined as though the code of inherited methods and >> attributes were copied into the subclass (with global accesses updated to >> point to the original module). > > You'll have to propose an implementation strategy for that > which works without actually copying all the code, though. The only difference between the copying semantics and the current semantics, is in the visibility of module-global internal variables and functions. It's sufficient to keep track of whether each class could access a variable or function that is internal to its module. If it does, then it cannot be subclassed from a different module. (It must not be possible to access internal variables/ functions reflectively.) The effect of this is that if a programmer intends a class to be subclassable from outside the module, they must make sure that all of the variables/functions it depends on are public. Anyone performing a security review of the module then does not have to consider inheritance from a different module when deciding which variables/functions might be accessible. There is a slightly more flexible version of this approach that is just as secure, provided that functions are restricted to be stateless. If it is possible to prove statically that a class does not access any internal *variables* of a module (regardless of whether it accesses internal functions), then it is safe to allow the class to be subclassed from another module. This is because the subclassing module could have copied all of the code of the original module (assuming it is written purely in Python); the only possible sources of authority in a capability language are from access to variables or primitives, not code. If a class is not written in Python, then we cannot analyse whether it accesses internal variables. In that case the class will be part of the TCB, and we have to trust the class writer to mark whether it can be safely subclassed from another module. >> Since x._hidden is not accessible from attack.py, the attack fails. > > But if _hidden were an attribute of the A instance that > you were trying to protect, it would succeed. No, attack.py could only access a _hidden attribute in an instance of B. This is harmless, because it could just as well define the _hidden attribute of B itself, rather than by subclassing. -- David Hopwood <[EMAIL PROTECTED]> ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Internal namespace proposal
Greg Ewing wrote: > David Hopwood wrote: > >> A restricted interpreter refuses access to any object attribute or >> method with a name beginning with '_' (by throwing a new exception type >> 'InternalAccessException'), unless the access is from a method and its >> static target is that method's first argument variable. > > What's to stop > > def my_naughty_method(self): > self = some_protected_object > self._a_special_attribute = some_naughty_value That's a good point -- I didn't describe the intended restriction correctly. The reason for not just saying "... and its target is the object that was passed as that method's first argument", was that I want it to be possible to reject programs with internal accesses that cannot be statically recognized as legal. (This does not mean that a particular implementation would have to rely on static detection of non-internal accesses, or on a trusted compiler.) How about this: A restricted interpreter shall refuse access to any object attribute or method with a name beginning with '_', unless the access is from a method, and its target is the object that was passed as that method's first argument. If such an access is detected at run-time, then it shall be reported by throwing a new exception type 'InternalAccessException'. In addition, a program containing an access to an object attribute or method with a name beginning with '_', where the access is not from a method, or the target of the access is not the method's first argument variable, or there is an assignment to that variable in the method, is an illegal program. >> __init__ is an internal method. This is as it should be, because it >> should not be possible to call __init__ on an existing object ... from outside the object, that is ... >> ; only to have __init__ implicitly called when a new object is constructed. > > What about calling an inherited __init__ method? > Your proposed rule would seem to disallow > > BaseClass.__init__(self, ...) No, this call would be allowed because it is to self. -- David Hopwood <[EMAIL PROTECTED]> ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Internal namespace proposal
David Hopwood wrote: > The intention was not to require the restrictions to be compiler-enforced; > only to *allow* them to be compiler-enforced. > > Code like this, for example: > > def someMethod(self, x): > if self == x: "if self is x:", I meant. > foo(x._internal) > > should not have to work. -- David Hopwood <[EMAIL PROTECTED]> ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Internal namespace proposal
Armin Rigo wrote: > Hi David, > > Your proposal is too vague to be useful. In Python I would not feel > that any compiler-enforced restrictions are going to be too restrictive, > and so I believe that your approach is not viable, but I cannot give you > many concrete examples of why before you come up with a more concrete > specification. The intention was not to require the restrictions to be compiler-enforced; only to *allow* them to be compiler-enforced. Code like this, for example: def someMethod(self, x): if self == x: foo(x._internal) should not have to work. > More importantly, this is going to depend critically on a restricted > interpreter in the first place, in the sense of Brett, but the safety of > your proposal depends on the restricted interpreter forbidding many > operations that it would not otherwise forbid for its original goal. > For example, in Brett's use case there is no need to prevent reading the > 'func_globals' attribute of function objects, but if that's allowed, > then accessing any _attribute of any module is easy. I disagree that there is no need to prevent reading func_globals. func_globals is clearly incompatible with capability security (as are func_dict and func_closure; also the other function attributes should be read-only). Functions in a capability language should be opaque. I don't see that there is any problem with the proposal depending on a restricted interpreter to prevent access via loopholes such as func_globals, since that is the main intended context of its use. Remember that Brett's document stated that protection could only be obtained at interpreter granularity, rather than object granularity, primarily because objects have no way to prevent access to their private state. My intention in describing the basic idea of enforcing the PEP 8 convention for internal attributes/methods, was to get precisely this kind of feedback on potential problems. I have already obtained useful feedback (including yours), and will prepare a more concrete proposal based on it. > About special methods: how do built-in functions like str(), int(), and > so on, know in which context they are called? Surely you don't propose > that '2+3' should be invalid because it accesses the hidden attribute '2 > .__add__' ? This and other examples have convinced me that names starting and ending with double underscores should not automatically be considered internal. There are a few such names that should be internal (e.g. __dict__), but it is reasonable to treat those as special cases. > How would you formulate a rule in term on Python's attribute look-up > algorithm to prevent the following trivial attack? : > > x.py: > > # supposedly secure? > > _hidden = [1,2,3] > > class A: > def __init__(self): > self._authorized = ... > def read(self): > if not self._authorized: > raise Forbidden > return _hidden > > attack.py: > > import x > class B(x.A): > def __init__(self): > self._authorized = True > > b = B() > print b.read() # => [1,2,3] Inheritance should be defined as though the code of inherited methods and attributes were copied into the subclass (with global accesses updated to point to the original module). IOW, B acts as though it is defined like this: attack.py: class B(x.A): def __init__(self): self._authorized = True def read(self): if not self._authorized: raise Forbidden return x._hidden Since x._hidden is not accessible from attack.py, the attack fails. > On any real-life example I'm sure that hacks like overriding selected > methods on the instance itself would allow an attacker to confuse the > remaining methods enough to leak hidden information. Yes, Java was subject to many attacks of this type. However, a code-copying semantics for inheritance prevents all of them, by ensuring that a class cannot do anything by inheritance that it could not do without it. > Here is a metaclass attack against the rule "self._attr is only allowed > if syntactically inside the class definition of the exact class of > self": > > class SupposedlySecure(object): > _hidden = [1,2,3] > > class MetaAttack(type): > def read(self): > return self._hidden # seen as an instance attribute > > class Attack(SupposedlySecure): > __metaclass__ = MetaAttack > > print Attack.read() Metaclasses are a reflective feature; almost all such features would have to be limited in restricted interpreters. -- David Hopwood <[EMAIL PROTECTED]> ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Internal namespace proposal
bclass Y both name an attribute __foo, then we will end up with two attributes _X__foo and _Y__foo in instances of Y, which is the desired behaviour for private attributes. In the case of an attribute called _foo, OTOH, there can be only one such attribute per object, which is the desired behaviour for protected attributes. The name mangling also ensures that an object will not *accidentally* access a private attribute inherited from a superclass. However, in the same example, an instance of Y can still deliberately access the copy of the attribute inherited from X by specifying _X__foo. There is no security problem here, because Y cannot do anything as a result that it could not have done by copying X's code, rather than inheriting from it. Notice that this is only true because we restrict an object to only accessing its own internal attributes and methods; if we followed C++'s semantics where an object can access protected members of any superclass, this would break security. (Java solves this problem by applying a more complicated access rule for protected members, which I considered to be unintuitive. More details on request.) __dict__ is an internal attribute. This means that an object can only directly reflect on itself. I know that there are other means of reflection (e.g. using the 'inspect' module); blocking these or making them safe is a separate issue. If desired, it would be safe to add a 'publicdict' attribute to each object, or a 'publicdict(object)' built-in. This would return a *read-only* dict, probably created lazily if needed, giving access only to public (non-internal) attributes and methods. __init__ is an internal method. This is as it should be, because it should not be possible to call __init__ on an existing object; only to have __init__ implicitly called when a new object is constructed. __repr__ and __str__ are internal under these rules, and probably shouldn't be. Existing classes may expose private state in the strings returned by __repr__ or __str__, but in principle, there is nothing unsafe about being able to convert the public state of an object to a string. OTOH, this functionality is usually accessed via the built-ins 'repr' and 'str', which we could perhaps allow to access '__repr__' and '__str__' as a special case. -- David Hopwood <[EMAIL PROTECTED]> ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New miniconf module
Sylvain Fourmanoit wrote: > I wrote a data persistence module called miniconf, aimed at making > easy to create and safely retrieve configuration info from external, > human-readable sources using Python syntax. I feel it would eventually > make a nice addition to the standard library. >From a security point of view, this is a great improvement on the existing pickle, marshal, and shelve modules. Those modules could not be safely imported from restricted code. miniconf, OTOH, appears to have an interface compatible with capability security. (I have not checked that the compiler.ast module used in its implementation is safe.) However: +Limitations +=== + +miniconf has a few limitations one should be aware of: [...] +- It is not preemptiple: concurrent calls to dump() or load() will + have unpredictable results and must be avoided. This limitation should be fixed before the module is added to the standard library, IMHO. -- David Hopwood <[EMAIL PROTECTED]> ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] new security doc using object-capabilities
Phillip J. Eby wrote: > At 12:04 PM 7/25/2006 +1200, Greg Ewing wrote: >>Phillip J. Eby wrote: >> >>>When I say "name checker" I mean the Zope type that allows you to specify >>>a list of names that are allowed for a given object. This allowing is not >>>based on identity or code signing or anything like that. It's just a list >>>of attribute names: i.e. a capability mask over an existing object. >> >>But this is backwards from what a true object-capability >>system should be like if it's properly designed. Instead >>of starting with too-powerful objects and trying to >>hide some of their powers, the different powers should >>be separated into different objects in the first place. > > And what about code that needs to pass on a subset of a capability? You > need the ability to create such capability-restricted subsets anyway, no > matter how "pure" a system you start with. That is true, but doing so for every secure object has significant costs. A simple wrapper is not sufficient, because it would not prevent a wrapped object from returning a reference to itself, bypassing the wrapper. To solve this problem you need to use the more complex Membrane pattern, which also wraps the results of method calls on a wrapped object, for example. In fact Zope's approach does implement a membrane, but this does not really dent the argument that Greg Ewing was making. A pure capability system incurs the complexity and performance costs of wrappers or membranes only in cases where they are needed, not for every object, and the complexity is only in user code, not in the system's security kernel. -- David Hopwood <[EMAIL PROTECTED]> ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] new security doc using object-capabilities
Phillip J. Eby wrote: > At 11:07 PM 7/23/2006 +0100, David Hopwood wrote: >> Phillip J. Eby wrote: >> > At 01:00 PM 7/23/2006 -0700, Brett Cannon wrote: >> > >> >> I obviously don't want to change the feel of Python, but if I have to >> >> remove the constructor for code objects to prevent evil bytecode or >> >> __subclasses__() from object to prevent poking around stuff, then so be >> >> it. For this project, security is [trumping] backwards-compatibility when >> >> the latter is impossible in order to have the former. I will obviously >> >> try to minimize it, but something that works at such a basic level of the >> >> language is just going to require some changes for it to work. >> > >> > Zope 3's sandboxing machinery manages to handle securing these things >> > without any language changes. So, declaring it "impossible" to manage >> > without backward compatibility seems inappropriate, or at least >> > incorrect. >> >> ... if Zope's sandboxing is secure. I haven't done a security review >> of it, but your argument assumes that it is. > > What argument is that? You said "Zope 3's sandboxing machinery manages to handle securing these things without any language changes." This assertion assumes that Zope 3's sandboxing machinery is secure. > I'm merely suggesting that coming up with a > completely new way to secure Python without a serious consideration of > existing practical prior art (with many years' deployment experience on > the public internet!) seems ill-advised with respect to achieving > practical goals. > > Brett's securing_python.txt don't refer to or cite Zope in any way, This is indeed an omission that should be corrected, in order to explain why this project is not using Zope or following Zope's approach, and what the differences are. (I've explained some of them below.) > but rather relies on broad and unsupported assertions about what can or > can't be done with Python. I hope he isn't doing the same in his > thesis, as this is rather like writing about one's new theory of how to > have a worldwide ball-kicking contest without making any reference as to > how one's theory compares with the World Cup. > > I'm not saying Zope is better or worse. I'm simply saying that in a > business context, a failure to compare and contrast a proposed "build" > solution to show how it would be better than a well-established > available "buy" solution would be called something like "lack of due > diligence". I think in the academic context it might be called > something like "failure to cite", but the general idea is the same, > i.e., not doing your homework. :) > > In other words, if the solution being proposed is better than what Zope > does, the appropriate thing in business is to show the reasons why, and > the appropriate thing in science is to state a hypothesis regarding the > differences, and then perform an experiment to either prove or disprove it. I completely agree with this. >> In any case, Zope's sandboxing is not capability-based. > > You're right: you haven't done a review of it. :) I haven't done a detailed security review. However, I wouldn't have commented on it without knowing what its basic approach is. From <http://www.zope.org/Wikis/DevSite/Projects/ComponentArchitecture/TransitionToSecurityProxies>: # When an object is passed to untrusted code, it is wrapped in a security # proxy unless it is already wrapped. Security proxies mediate all accesses # to the wrapped object. Operations on security proxies return security # proxies as well. Security proxies passed from untrusted code to trusted # code remain wrapped, so untrusted code can't trick trusted code into # performing operations that the untrusted code could not perform. Restricting the actions of "trusted" code when called by "untrusted" code is precisely what a capability system does *not* do. Indeed, capability systems have no binary distinction between "trusted" and "untrusted" code (outside the system TCB, which is as small as possible); the only security distinction between protection domains is in what capabilities they hold. The main reason why capability systems do not have any such restriction is that it reduces the system's ability to support fine-grained delegation of authority. We want to be able to grant an object just the authority it needs for any particular task. For that to work, code that is otherwise untrusted must be able to use any capability it is given, even if it is a very powerful capabil
Re: [Python-Dev] new security doc using object-capabilities
Brett Cannon wrote: > On 7/23/06, Armin Rigo <[EMAIL PROTECTED]> wrote: > >> Hi David, hi Brett, >> >> On Sun, Jul 23, 2006 at 02:18:48AM +0100, David Hopwood wrote: >> > If I understand correctly, the proposal is that any incompatible >> > changes to the language would apply only in "sandboxed" interpreters. >> > So there is no reason why support for these couldn't go into the >> > main branch. >> >> That's what I originally thought too, but Brett writes: >> >> Implementation Details >> >> >> An important point to keep in mind when reading about the >> implementation details for the security model is that these are >> general changes and are not special to any type of interpreter, >> sandboxed or otherwise. That means if a change to a built-in type is >> suggested and it does not involve a proxy, that change is meant >> Python-wide for *all* interpreters. >> >> So that's why I'm starting to worry that Brett is proposing to change >> the regular Python language too. > > Yes, I am proposing changing some constructors and methods on some built-in > types for the regular languages. That's it. No new keywords or major > semantic changes and such. If I make changes just for sandboxed > interpreters it changes the general approach of the security model by then > requiring an identity check to see if the interpreter is sandboxed or not. I assume that the extent of incompatible changes would be limited as much as possible. So the only checks would be in operations that are directly affected by whatever incompatible changes are made. The performance and complexity costs of this are likely to be small -- or at least should not be assumed to be large before having hammered out a more detailed design. Suppose, for the sake of argument, that we introduced private methods and attributes. If an attribute in an existing standard library class was changed to be private, then code depending on it would break. But if there were a notion of a "compatibility private" attribute that acts as private only in a sandboxed interpreter, then no code running in an unprotected interpreter would break. -- David Hopwood <[EMAIL PROTECTED]> ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] new security doc using object-capabilities
Phillip J. Eby wrote: > At 01:00 PM 7/23/2006 -0700, Brett Cannon wrote: > >>I obviously don't want to change the feel of Python, but if I have to >>remove the constructor for code objects to prevent evil bytecode or >>__subclasses__() from object to prevent poking around stuff, then so be >>it. For this project, security is [trumping] backwards-compatibility when >>the latter is impossible in order to have the former. I will obviously >>try to minimize it, but something that works at such a basic level of the >>language is just going to require some changes for it to work. > > Zope 3's sandboxing machinery manages to handle securing these things > without any language changes. So, declaring it "impossible" to manage > without backward compatibility seems inappropriate, or at least > incorrect. ... if Zope's sandboxing is secure. I haven't done a security review of it, but your argument assumes that it is. In any case, Zope's sandboxing is not capability-based. -- David Hopwood <[EMAIL PROTECTED]> ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] new security doc using object-capabilities
Armin Rigo wrote: > Re-hi, > > On Wed, Jul 19, 2006 at 03:35:45PM -0700, Brett Cannon wrote: > >>http://svn.python.org/view/python/branches/bcannon-sandboxing/securing_python.txt?rev=50717&view=log. > > I'm not sure I understand what you propose to fix holes like > constructors and __subclasses__: it seems that you want to remove them > altogether (and e.g. make factory functions instead). That would > completely break all programs, right? I mean, there is no way such > changes would go into mainstream CPython. If I understand correctly, the proposal is that any incompatible changes to the language would apply only in "sandboxed" interpreters. So there is no reason why support for these couldn't go into the main branch. Of course we want to minimize the changes that will need to be made to programs and libraries to make them work in a sandboxed interpreter, but not at the expense of security. Some incompatible changes will be necessary. -- David Hopwood <[EMAIL PROTECTED]> ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
