date:20060815

Re: [Python-Dev] Type of range object members

2006-08-15 Thread Neal Norwitz


On 8/15/06, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote:


That penalty is already paid today. Much code dealing with
ints has a type test whether it's an int or a long. If
int and long become subtypes of each other or of some abstract
type, performance will decrease even more because a subtype
test is quite expensive if the object is neither int nor
long (it has to traverse the entire base type hierarchy to
find out its not inherited from int).


I was playing around with a little patch to avoid that penalty.  It
doesn't take any additional memory, just a handful of bits we aren't
using. :-)

For the more common builtin types, it stores whether it's a subclass
in tp_flags, so there's no function call necessary and it's a constant
time operation.  It was faster when doing simple stuff.  Haven't
thought much whether this is really worthwhile or not.

n
Index: Include/stringobject.h
===
--- Include/stringobject.h	(revision 51237)
+++ Include/stringobject.h	(working copy)
@@ -55,8 +55,9 @@
 PyAPI_DATA(PyTypeObject) PyBaseString_Type;
 PyAPI_DATA(PyTypeObject) PyString_Type;
 
-#define PyString_Check(op) PyObject_TypeCheck(op, &PyString_Type)
 #define PyString_CheckExact(op) ((op)->ob_type == &PyString_Type)
+#define PyString_Check(op) (PyString_CheckExact(op) || \
+ PyType_FastSubclass((op)->ob_type, Py_TPFLAGS_STRING_SUBCLASS))
 
 PyAPI_FUNC(PyObject *) PyString_FromStringAndSize(const char *, Py_ssize_t);
 PyAPI_FUNC(PyObject *) PyString_FromString(const char *);
Index: Include/dictobject.h
===
--- Include/dictobject.h	(revision 51237)
+++ Include/dictobject.h	(working copy)
@@ -90,8 +90,9 @@
 
 PyAPI_DATA(PyTypeObject) PyDict_Type;
 
-#define PyDict_Check(op) PyObject_TypeCheck(op, &PyDict_Type)
 #define PyDict_CheckExact(op) ((op)->ob_type == &PyDict_Type)
+#define PyDict_Check(op) (PyDict_CheckExact(op) || \
+ PyType_FastSubclass((op)->ob_type, Py_TPFLAGS_DICT_SUBCLASS))
 
 PyAPI_FUNC(PyObject *) PyDict_New(void);
 PyAPI_FUNC(PyObject *) PyDict_GetItem(PyObject *mp, PyObject *key);
Index: Include/unicodeobject.h
===
--- Include/unicodeobject.h	(revision 51237)
+++ Include/unicodeobject.h	(working copy)
@@ -390,8 +390,9 @@
 
 PyAPI_DATA(PyTypeObject) PyUnicode_Type;
 
-#define PyUnicode_Check(op) PyObject_TypeCheck(op, &PyUnicode_Type)
 #define PyUnicode_CheckExact(op) ((op)->ob_type == &PyUnicode_Type)
+#define PyUnicode_Check(op) (PyUnicode_CheckExact(op) || \
+ PyType_FastSubclass((op)->ob_type, Py_TPFLAGS_UNICODE_SUBCLASS))
 
 /* Fast access macros */
 #define PyUnicode_GET_SIZE(op) \
Index: Include/intobject.h
===
--- Include/intobject.h	(revision 51237)
+++ Include/intobject.h	(working copy)
@@ -27,8 +27,9 @@
 
 PyAPI_DATA(PyTypeObject) PyInt_Type;
 
-#define PyInt_Check(op) PyObject_TypeCheck(op, &PyInt_Type)
 #define PyInt_CheckExact(op) ((op)->ob_type == &PyInt_Type)
+#define PyInt_Check(op) (PyInt_CheckExact(op) || \
+		 PyType_FastSubclass((op)->ob_type, Py_TPFLAGS_INT_SUBCLASS))
 
 PyAPI_FUNC(PyObject *) PyInt_FromString(char*, char**, int);
 #ifdef Py_USING_UNICODE
Index: Include/listobject.h
===
--- Include/listobject.h	(revision 51237)
+++ Include/listobject.h	(working copy)
@@ -40,8 +40,9 @@
 
 PyAPI_DATA(PyTypeObject) PyList_Type;
 
-#define PyList_Check(op) PyObject_TypeCheck(op, &PyList_Type)
 #define PyList_CheckExact(op) ((op)->ob_type == &PyList_Type)
+#define PyList_Check(op) (PyList_CheckExact(op) || \
+		PyType_FastSubclass((op)->ob_type, Py_TPFLAGS_LIST_SUBCLASS))
 
 PyAPI_FUNC(PyObject *) PyList_New(Py_ssize_t size);
 PyAPI_FUNC(Py_ssize_t) PyList_Size(PyObject *);
Index: Include/object.h
===
--- Include/object.h	(revision 51237)
+++ Include/object.h	(working copy)
@@ -517,6 +517,18 @@
 /* Objects support nb_index in PyNumberMethods */
 #define Py_TPFLAGS_HAVE_INDEX (1L<<17)
 
+/* These flags are used to determine if a type is a subclass. */
+/* Uses bits 30-27. */
+#define Py_TPFLAGS_FAST_SUBCLASS_MASK (0x7800)
+#define Py_TPFLAGS_FAST_SUBCLASS (1L<<27)
+#define Py_TPFLAGS_INT_SUBCLASS		(Py_TPFLAGS_FAST_SUBCLASS | 0x7000)
+#define Py_TPFLAGS_LONG_SUBCLASS	(Py_TPFLAGS_FAST_SUBCLASS | 0x6000)
+#define Py_TPFLAGS_LIST_SUBCLASS	(Py_TPFLAGS_FAST_SUBCLASS | 0x5000)
+#define Py_TPFLAGS_TUPLE_SUBCLASS	(Py_TPFLAGS_FAST_SUBCLASS | 0x4000)
+#define Py_TPFLAGS_STRING_SUBCLASS	(Py_TPFLAGS_FAST_SUBCLASS | 0x3000)
+#define Py_TPFLAGS_UNICODE_SUBCLASS	(Py_TPFLAGS_FAST_SUBCLASS | 0x2000)
+#define Py_TPFLAGS_DICT_SUBCLASS	(Py_TPFLAGS_FAST_SUBCLASS | 0x1000)
+
 #define Py_TPFLAGS_DEFAULT  ( \
  Py_TPFLAGS_HAVE_GE

Re: [Python-Dev] Type of range object members

2006-08-15 Thread Martin v. Löwis

Greg Ewing schrieb:
>> There isn't? Actually a lot of APIs currently assumen that.
> 
> Also it means you'd pay a penalty every time you
> access it, whereas presumably short ints are the
> case we want to optimise for speed as well.

That penalty is already paid today. Much code dealing with
ints has a type test whether it's an int or a long. If
int and long become subtypes of each other or of some abstract
type, performance will decrease even more because a subtype
test is quite expensive if the object is neither int nor
long (it has to traverse the entire base type hierarchy to
find out its not inherited from int).

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Type of range object members

2006-08-15 Thread Martin v. Löwis

Greg Ewing schrieb:
> Guido van Rossum wrote:
>> I worry that dropping the special allocator will be too slow.
> 
> Surely there's some compromise that would allow
> recently-used ints to be kept around, but reclaimed
> if memory becomes low?

Hardly. The efficiency of the special-case allocator also
comes from the fact that it doesn't ever have to release
memory. Just try changing it to see what I mean.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Type of range object members

2006-08-15 Thread Martin v. Löwis

Greg Ewing schrieb:
> Martin v. Löwis wrote:
>> We had this discussion before; if you use ob_size==0 to indicate
>> that it's an int, this space isn't needed in a long int.
> 
> What about int subclasses?

It's what Guido proposes.

It would still leave two types (perhaps three) at the C level,
so C code might have to continue making conditional code depending
on which of these it is. Also, Python code that dispatches by type
still needs to make the distinction.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Type of range object members

2006-08-15 Thread Tim Peters

[Greg Ewing]
> Surely there's some compromise that would allow
> recently-used ints to be kept around, but reclaimed
> if memory becomes low?

Not without losing /something/ we currently enjoy.  The current int
scheme has theoretically optimal memory use too, consuming 12 bytes
per int object on a 32-bit box.  That's minimal, because the object's
type pointer, refcount, and integer value each require 4 bytes on a
32-bit box.

It does this by allocating "big blocks" and carving them into 12-byte
slices itself.  But as with any "big block" scheme, a live integer
anywhere in a block prevents the entire block from being freed, and
objects can't /move/ in CPython either (so, e.g., if a single live
integer is keeping a big block alive, we can't move it into some other
block).  That's the same kind of problem obmalloc has with its "very
big block" arenas, and for which only heuristic help was added (with
considerable increase in complexity) for 2.5 (before 2.5, obmalloc
never freed an arena, same as ints and floats never free their
versions of big blocks).
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] uuid module - byte order issue

2006-08-15 Thread Oren Tirosh

On 04/08/06, Ka-Ping Yee <[EMAIL PROTECTED]> wrote:
> On Thu, 3 Aug 2006, Oren Tirosh wrote:
> > The UUID module uses network byte order, regardless of the platform
> > byte order. On little-endian platforms like Windows the ".bytes"
> > property of UUID objects is not compatible with the memory layout
>
> RFC 4122 says:
>
> In the absence of explicit application or presentation protocol
> specification to the contrary, a UUID is encoded as a 128-bit
> object, as follows:
>
> The fields are encoded as 16 octets, with the sizes and order of
> the fields defined above, and with each field encoded with the
> Most Significant Byte first (known as network byte order).

RFC 4122 defines a canonical byte order for UUIDs but also makes
explicit reference to the fact that UUIDs are stored locally in native
byte order. The final step in the RFC 4122 UUID generation algorithm
is:

>   o  Convert the resulting UUID to local byte order.

So this is not another case of the
Microsoft-implements-RFC-incorrectly syndrome. After all, they are one
of the co-authors of the document.

Compatibility with Windows "GUIDs" may be one of the most important
use cases for the UUID module. It's important to resolve this or users
will have unpleasant surprises. I did.

alternatives:

1. Default is big endian byte order. Little endian is explicit.
2. Default is native byte order. Little endian and big endian are explicit.
3. No default. Little endian and big endian are both explicit.

All three are relevant for both the constructor and retrieving the
byte representation.

  Oren
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Dicts are broken Was: unicode hell/mixing str and unicode asdictionarykeys

2006-08-15 Thread mal

Martin v. Löwis wrote:
> M.-A. Lemburg schrieb:
>> Python just doesn't know the encoding of the 8-bit string, so can't
>> make any assumptions on it. As result, it raises an exception to inform
>> the programmer.
> 
> Oh, Python does make an assumption what the encoding is: it assumes
> it is the system encoding (i.e. "ascii"). Then invoking the ascii
> codec raises an exception, because the string clearly isn't ascii.

Right, and as consequence, Python raises an exception to let the
programmer correct the problem.

The subsequent solution to the problem may result in the
string being decoded into Unicode and the two resulting Unicode
objects being unequal, or it may also result in them being equal.
Python doesn't have this knowledge, so always returning false
is clearly wrong.

Hiding programmer errors is not making life easier in the
long run, so I'm -1 on having the equality comparison return
False.

Instead we should generate a warning in Python 2.5 and introduce
the exception in Python 2.6.

>> Note that you do have to interpret the string as characters
>> > if you compare it to Unicode and there's nothing wrong with
>> > that.
> 
> Consider this:
> py> int(3+4j)
> Traceback (most recent call last):
>   File "", line 1, in ?
> TypeError: can't convert complex to int; use int(abs(z))
> py> 3 == 3+4j
> False
>
> So even though the conversion raises an exception, the
> values are determined to be not equal. Again, because int
> is a nearly true subset of complex, the conversion goes
> the other way, but *if* it would use the complex->int
> conversion, then the TypeError should be taken as
> a guarantee that the objects don't compare equal.

In the above example, you clearly know that the two are
unequal due to the relationship between complex numbers
having an imaginary part and integers..

The same is true for the overflow case:

>>> 2**1 == 1.23
False
>>> float(2**1)
Traceback (most recent call last):
  File "", line 1, in ?
OverflowError: long int too large to convert to float

(Note that in Python 2.3 this used to raise an exception as well.)

However, this is not the case for 8-bit string vs. Unicode,
since you cannot use such extra knowledge if you find that ASCII
encoding assumption obviously doesn't match the string
in question.

> Expanding this view to Unicode should mean that a unicode
> string U equals a byte string B if
> U.encode(system_encode) == B or B.decode(system_encoding) == U,
> and that they don't equal otherwise 

Agreed.

Note that Python always coerces to the "bigger" type. As a result,
the second option is what is actually implemented in Python.

> (e.g. if the conversion
> fails with a "not convertible" exception). 

I disagree with this part.

Failure to decode a string doesn't imply inequality. It implies
that the programmer needs to step in and correct the problem by
making an explicit and conscious decision.

The alternative would be to decide that equal comparisons should never
be allowed to raise exceptions and instead have the equal comparison
return False. In which case, we'd have the revert the dict patch
altogether and instead silence all exceptions that
are generated during the equal comparison (not only in the dict
implementation), replacing them with a False return value.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Aug 07 2006)
>>> Python/Zope Consulting and Support ...http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! 

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Four issue trackers submitted for Infrastructue Committee's tracker search

2006-08-15 Thread Brett Cannon

Back in June, the Python Software Foundation's Infrastructure Committee asked for help in the search for a new issue tracker to replace SourceForge (see http://wiki.python.org/moin/CallForTrackers
for details). We asked people who wished to help with the search to install their favourite issue tracker and import a data dump of our issues from SourceForge. We placed a deadline of August 7th to have the installation up and going. We realized that this was a large request to make of the community, but we needed to make sure that we got to evaluate any candidate trackers with the amount of issues that Python has so as to get a real-world feel.
Not surprisingly, the community stepped up to the challenge with four different test tracker installations by the deadline! We have one for JIRA, Roundup, Trac, and Launchpad. The current schedule calls for the Infrastructure Committee to spend the next month evaluating the four trackers and making a decision on which one to go with. This might slip, though, since this overlaps with traditional American and European vacation time. But the slippage will not go beyond October 1.
On behalf of the Infrastructure committee I thank the people who took the time and effort to set up a test tracker to help improve the development of Python.- Brett Cannon Chairman, PSF Infrastructure Committee
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

51 matches

Mail list logo