Re: [Python-Dev] repeated keyword arguments
On Jun 28, 12:56 am, Guido van Rossum [EMAIL PROTECTED] wrote: No, it could just be a harmless typo in a long argument list. to elaborate on this point a little, i came across this error when i ported my code to 2.4. i used the optparse class which takes 10's of kwargs, and it turned out i'd given the same parameter twice (the infamous copy-paste problem). so on the one hand, it was a harmless typo (because the latest instance was taken). on the other hand, it's a good thing i tested it on 2.4, or i'd never notice the repeated argument, which may have led to weird runtime errors (if the order of the params was changed). i'd be in favor of fixing this in 2.5, just to eliminate possibly hard- to-debug runtime errors. since it's a syntax error, it would be early- noticed when the code is first run/imported, and it wouldn't require the original author of the code to fix. -tomer ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] repeated keyword arguments
the following code works on python 2.5:
def f(**kwargs):
... print kwargs
...
f(a=5,b=7,a=8)
{'a': 8, 'b': 7}
but fails on python2.4, saying that a is given twice.
is this a bug or a feature?
-tomer
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] forceful exit
hi i'm having trouble when forking child processes to serve sockets. the skeleton is something like the following: def work(): try: while True: s = listener.accept()[0] log(hello %s, s) if os.fork() == 0: try: serve_client(sock) finally: sys.exit() (1) else: log(forked child) sock.close() except KeyboardInterrupt: log(got ctrl+c) finally: log(server terminated) listener.close() the problem is that sys.exit() raises an exception, which propagates all the way up and is handled by the finallies... the real code does more than just printing logs, so i can't allow that. i'm forced to resort to os._exit, which does the trick but doesn't perform cleanup, which made me realize there's no clean way in python to *force* exit. i think there ought to be something like sys.forcedexit(), which collects all objects nicely and then exits immediately without letting anything propagate up. this will also solve another problem i came across, with threads. turns out only the main thread can kill the process -- if another thread issues sys.exit, it only kills itself. there's no clean way for a thread to terminate the process... i think there must be. i can contribute a patch in a matter of days... i think it's pretty straight forward (calling Py_Finalize and then exit). aye or nay? -tomer ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] misbehaving __contains__
On Jan 23, 2008 3:19 AM, Raymond Hettinger [EMAIL PROTECTED] wrote: [Steven Bethard] We've already lost this if anyone really wants to break it:: class C(object): ... def __iter__(self): ... return iter(xrange(3)) ... def __contains__(self, item): ... return False [snip] I'm more concerned about introducing an API where well-meaning attempts to code a __contains__ override will implicitly shoot the toes off of client code that innocently assumes a somewhat sensible invariant relation between looping over elements in a container and those elements being in the container. The code for sets and frozensets makes that assumption, for example. And, when the code does break, the errors will be darned hard to find. well, i don't see how limiting __contains__ to returning booleans solves the problem you mentioned. what you're talking about is *semantics*, and that's always up to the programmer to get right. allowing __contains__ to return a FooBar object won't change that a bit -- but it will allow me to construct expression objects more easily. and, statements like if x in y: pass will work as expected, assuming the object returned by y.__contains__(x) has a proper __bool__/__nonzero__ method. we're just deferring the type-cast to the point it's actually needed. as per performance, i don't believe changing a slot method to return PyObject* instead of int would change anything... am i wrong? -tomer ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] misbehaving __contains__
i'm using python to create expression objects, for more intuitive usage as predicates, for instance: x = (Arg(0) 17) (Arg(1).foo == bar) instead of x = And(Gt(Arg(0), 17), Eq(GetAttr(Arg(1), foo), bar)) so now i can use x.eval(18, spam) and get the result of the expression. i'm doing it by overriding all the operators to return expression objects, instead of evaluating the result immediately. it works fine, but i encountered a problem with making __contains__ behave so. it turns out __contains__ coerces the return value into a bool. this might seem logical at first, but is not consistent with the result of the language. consider the following code: class Foo(object): ... def __contains__(self, key): ... return 17 ... def __eq__(self, other): ... return 19 ... f=Foo() f == 8 19 8 in f True as you can see, __eq__ does not coerces the result to bool, so why should __contains__ do that? i've looked at PySequence_Contains in objects/abstract.c, but i can't quite understand where the coercion is made. is because the function is typed as int? if so, i believe it should either be changed to PyObject, or have cmp_outcome (in ceval.c) not use this API directly, and rather return the result as it is returned from __contains__. i see no reason it should behave differently than __eq__ and the rest of comparison operators. -tomer ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] functional continuations
yes, well, as i suspected and as PJE pointed out, extension or builtin functions would make life difficult, but these are difficulties i would be willing to accept. however, my attempts are futile anyways, as the evaluation stack is cleared before the exception propagates up to the frames above... which would explain the NULL deref exceptions i was getting :) -tomer On Dec 16, 2007 12:54 AM, Greg Ewing [EMAIL PROTECTED] wrote: tomer filiba wrote: the idea i came up with is, using exceptions for functional continuations: after all, the exception's traceback holds the entire context... The problem with this is that, if the call chain has passed through a C-implemented function at some point, the traceback *doesn't* contain the entire context -- some of it is in the C stack frames that got unwound during the propagation of the exception. So you may be able to make this work where all the code involved is pure Python, but it can't work in general, unless you redesign the whole interpreter the way the original Stackless did. Having said that, it might still be a useful thing to have, as the pure-Python case is probably fairly common. But I'd want to know what the failure mode is like when the pure-Python case doesn't hold. Do you get a clear indication of what is wrong, or does it misbehave in some obscure way? A test case for this would be to call map() with a function that tries to suspend itself using this mechanism. What happens when you try to resume it? -- Greg -- An NCO and a Gentleman ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] functional continuations
i'm working on some minimalistic asynchronous framework for python, somewhat like twisted or stackless, but for different purposes. i came to the conclusion i want to be able to freeze functions, and resume them later, when some condition is matched. the idea i came up with is, using exceptions for functional continuations: after all, the exception's traceback holds the entire context... so with some help from black magic (i.e., ctypes), i can resurrect dead stackframes. i will have a reactor in the background which manages the execution flow, and blocking operations (I/O, etc.) will be wrapped with a thin wrapper, such as class socket2(socket): def recv(self, count = None): raise WaitFor(self, r) return socket.recv(self, count) now, when read()ing from file2 objects, first an exception will propagate up to the reactor, which will catch it, create a continuation object (which holds the traceback) and register it with the desired event. then, when the event is triggered, the reactor will resurrect the continuation object to the point where it stopped (f_lasti + 2), and execution would continue normally from there. the idea itself is similar to generator objects, and requires some messing-around with f_back and f_lasti which is quite ugly. on the other hand, the beauty of it is, it doesn't require any special design patterns (unlike twisted) or replacing the interpreter (unlike stackless), so it can be transparently used with any third-party libs. all it takes is wrapping I/O objects to raise WaitFor(...) prior to doing blocking I/O. moreover, unlike generators, i gain the ability to propagate up to the reactor (like normal exceptions), so the code in between the I/O and the reactor needn't be aware of anything (only avoiding unconstrained try-except blocks). i wanted to get some feedback on the issue (i tried c.l.p, but they didn't understand me well enough): * has it been tried before? * do you suppose it will work? are there any drawbacks i didn't anticipate? * is it useful enough to be added to the core interpreter (like generators)? of course if it is added, it could be implemented with a dedicated mechanism, rather than abusing exceptions for that. -tomer ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] functional continuations
yeah, i did think native functions wouldn't fit well with it, but then
again, i don't plan to have any c-side functions invoking python
callbacks. i can live with that. but, alas, ceval::EvalFrameEx will
clear the execution stack upon returning [1], so this couldn't work
anyway [2].
[1]
while (STACK_LEVEL() b-b_level) {
v = POP();
Py_XDECREF(v);
}
[2] hrrrmpff
-tomer
On Dec 15, 2007 6:57 PM, Phillip J. Eby [EMAIL PROTECTED] wrote:
At 01:04 AM 12/15/2007 -0800, tomer filiba wrote:
* do you suppose it will work? are there any drawbacks i didn't
anticipate?
Yes. :)
Specifically, think about what happens when a C function is in the
call stack, e.g.:
def f1():
return map(f2, [1,2,3])
def f2(ob):
raise WaitFor(something)
return ob
print f1()
If I understand you correctly, when this program is run, it will
print 1, rather than [1, 2, 3], because there is no way for you
to keep track of the internal state of the 'map()' call.
And this isn't just a problem for map() -- even something as simple
as a property access passes through C code whose state can get lost.
I don't think this approach is practical; you'd be better off using a
co-routine stack and trampoline, since the nature of generators
naturally forces all the C code out of the picture.
--
An NCO and a Gentleman
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] import file extensions
On 9/14/07, Martin v. Löwis [EMAIL PROTECTED] wrote: The best way would be to not use import, but provide a separate function (e.g. calling it require). yepp, that's probably the cleanest and quickest solution. i needed to see all the alternatives to realize this though. sorry. -- An NCO and a Gentleman ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] classes and cell variables question
On 12/29/06, Jeremy Hylton [EMAIL PROTECTED] wrote: def spam(): x = 5 class eggs(object): x = 6 def spam(self): return x return eggs spam()().spam() should return 5. the question that arises is -- is this what we wanted? if i had to read such code, where i can't (easily) tell where x is bound to, i wouldn't be content, to say the least. -tomer ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Non-blocking (asynchronous) timer without thread?
The main goal is to prevent threads overhead and problems with race conditions and deadlocks. check out stackless python -- http://www.stackless.com/ -tomer ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] classes and cell variables question
to my understanding of the object model, the code of snippet 1
and snippet 2 should be equivalent. a class is just a special function
that returns its locals automatically and passes them to the metaclass
constructor:
--- snippet 1 ---
class foo(object):
x = 5
def f(self):
print f, x
--- snippet 2 ---
def bar():
x = 5
def g(self):
print g, x
return locals()
barcls = type(barcls, (object,), bar())
but there's one big difference. classes don't create cell variables
to hold bound external variables. the bar version works, because
x is a bound cell variable, but the foo version fails, as it attempts
to access x as a global.
. barcls().g()
g 5
. foo().f()
f
Traceback (most recent call last):
File stdin, line 1, in module
File stdin, line 4, in f
NameError: global name 'x' is not defined
for reference, i attached the code of all four functions below.
my question is, how come classes don't create cell variables, like
normal functions? was this done on purpose? does it have to
do with inheritance? if so, what's wrong with my bar version?
[1]
# code of class foo
# 2 0 LOAD_NAME0 (__name__)
# 3 STORE_NAME 1 (__module__)
# 3 6 LOAD_CONST 0 (5)
# 9 STORE_NAME 2 (x)
#
# 4 12 LOAD_CONST 1 (code object f at
009E5608, file stdin, line 4)
# 15 MAKE_FUNCTION0
# 18 STORE_NAME 3 (f)
# 21 LOAD_LOCALS
# 22 RETURN_VALUE
[2]
# code of foo.f:
# 5 0 LOAD_CONST 1 ('f')
# 3 PRINT_ITEM
# 4 LOAD_GLOBAL 0 (x)
# 7 PRINT_ITEM
# 8 PRINT_NEWLINE
# 9 LOAD_CONST 0 (None)
# 12 RETURN_VALUE
[3]
# code of bar:
# 2 0 LOAD_CONST 1 (5)
# 3 STORE_DEREF 0 (x)
#
# 3 6 LOAD_CLOSURE 0 (x)
# 9 BUILD_TUPLE 1
# 12 LOAD_CONST 2 (code object g at
009F6698, file stdin, line 3)
# 15 MAKE_CLOSURE 0
# 18 STORE_FAST 0 (g)
#
# 5 21 LOAD_GLOBAL 0 (locals)
# 24 CALL_FUNCTION0
# 27 RETURN_VALUE
[4]
# code of bar.g:
# 4 0 LOAD_CONST 1 ('g')
# 3 PRINT_ITEM
# 4 LOAD_DEREF 0 (x)
# 7 PRINT_ITEM
# 8 PRINT_NEWLINE
# 9 LOAD_CONST 0 (None)
# 12 RETURN_VALUE
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] classes and cell variables question
If you don't follow this reasoning, please write a counter-proposal so that people have something to shoot down. ? i just wanted to be sure it was done on purpose, and what were the reasons for that. -tomer On 12/20/06, Martin v. Löwis [EMAIL PROTECTED] wrote: tomer filiba schrieb: my question is, how come classes don't create cell variables, like normal functions? Not sure what you mean by how come? Why is the implementation reacting as it is? Because the body of class is compiled as a global code fragment, not as a nested one. Or why is the implementation the way it is? For several reasons, one being that classes predate nested functions. was this done on purpose? Yes. Attributes defined inside a class are assumed to be accessed through attribute access *only*. So you write self.foo() to invoke a method, instead of invoking foo() Python treats methods and data attributes uniformly, so the same applies to data variables. does it have to do with inheritance? if so, what's wrong with my bar version? It also has to do with inheritance. If you do self.foo, it looks 1. in the object itself 2. in the class of the object 3. in the bases of the class of the object (recursively) It would be counter-intuitive if some things (i.e. things defined in the class itself) could be accessed directly, whereas other things (ie. attributes of the object itself, and things in the base classes) would need to be accessed through qualification. It would also be counter-intuitive if you find methods in an unqualified manner, but then can't call them because you didn't give a self argument. If you don't follow this reasoning, please write a counter-proposal so that people have something to shoot down. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] infinities
i found several places in my code where i use positive infinity (posinf) for various things, i.e., def readline(self, limit = -1): if limit 0: limit = 1e1 # posinf chars = [] while limit 0: ch = self.read(1) chars.append(ch) if not ch or ch == \n: break limit -= 1 return .join(chars) i like the concept, but i hate the 1e1 stuff... why not add posint, neginf, and nan to the float type? i find it much more readable as: if limit 0: limit = float.posinf posinf, neginf and nan are singletons, so there's no problem with adding as members to the type. -tomer ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] infinities
sys.maxint makes more sense there. no, it requires *infinity* to accomplish x - y == x; y != 0, for example: while limit 0: limit -= len(chunk) with limit = posinf, the above code should be equivalent to while True. There is already a PEP 754 for float constants okay, that would suffice. but why isn't it part of stdlib already? the pep is three years old... it should either be rejected or accepted. meanwhile, there are lots of missing API functions in the floating-point implementation... besides, all the suggested APIs should be part of the float type, not a separate module. here's what i want: f = 5.0 f.is_infinity() False float.PosInf 1.#INF -tomer On 11/26/06, Bob Ippolito [EMAIL PROTECTED] wrote: On 11/26/06, tomer filiba [EMAIL PROTECTED] wrote: i found several places in my code where i use positive infinity (posinf) for various things, i.e., def readline(self, limit = -1): if limit 0: limit = 1e1 # posinf chars = [] while limit 0: ch = self.read(1) chars.append(ch) if not ch or ch == \n: break limit -= 1 return .join(chars) i like the concept, but i hate the 1e1 stuff... why not add posint, neginf, and nan to the float type? i find it much more readable as: if limit 0: limit = float.posinf posinf, neginf and nan are singletons, so there's no problem with adding as members to the type. sys.maxint makes more sense there. Or you could change it to while limit != 0 and set it to -1 (though I probably wouldn't actually do that)... There is already a PEP 754 for float constants, which is implemented in the fpconst module (see CheeseShop). It's not (yet) part of the stdlib though. -bob ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] infinities
Um, you do realize that you're not going to be able to fit sys.maxint strings into a list, right? i can multiply by four, thank you. of course i don't expect anyone to read a string *that* long. besides, this *particular example* isn't important, it was just meant to show why someone might want to use it. why are people being so picky about the details of an example code? first of all, a while True loop is not limited by sys.maxint, so i see no reason why i couldn't get the same result by subtracting from infinity. that may seem blunt, but it's a good way have the same code handle both cases (limited and unlimited reading). all i was asking for was a better way to express and handle infinity (and nan), instead of the poor-man's version of nan = 2e/3e. float.posinf or float.isinf(5.0) seem the right way to me. for some reference, it seemed the right way to other people too: http://msdn2.microsoft.com/en-gb/library/system.double_methods.aspx http://java.sun.com/j2se/1.4.2/docs/api/java/lang/Float.html the third-party fp module is nice, but it ought to be part of the float type, or at least part of stdlib. - - - - - - if it were up to me, *literals* producing infinity would be a syntax error (of course i would allow computations to result in infinity). for the reason why, consider this: 1e1 == 2e2 True -tomer ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __dir__, part 2
okay, everything's fixed. i updated the patch and added a small test to: Lib/test/test_builtins.py::test_dir -tomer On 11/7/06, Nick Coghlan [EMAIL PROTECTED] wrote: tomer filiba wrote: cool. first time i build the entire interpreter, 'twas fun :) currently i retained support for __members__ and __methods__, so it doesn't break anything and is compatible with 2.6. i really hope it will be included in 2.6 as today i'm using ugly hacks in RPyC to make remote objects appear like local ones. having __dir__ solves all of my problems. besides, it makes a lot of sense of define __dir__ for classes that define __getattr__. i don't think it should be pushed back to py3k. here's the patch: http://sourceforge.net/tracker/index.php?func=detailaid=1591665group_id=5470atid=305470 As I noted on the tracker, PyObject_Dir is a public C API function, so it's behaviour needs to be preserved as well as the behaviour of calling dir() from Python code. So the final form of the patch will likely need to include stronger tests for that section of the API, as well as updating the documentation in various places (the dir and PyObject_Dir documentation, obviously, but also the list of magic methods in the language reference). +1 on targeting 2.6, too. Cheers, Nick. -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- http://www.boredomandlaziness.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __dir__, part 2
as well as updating the documentation in various places (the dir and PyObject_Dir documentation, obviously, but also the list of magic methods in the language reference). oops, i meant everything except that -tomer On 11/7/06, tomer filiba [EMAIL PROTECTED] wrote: okay, everything's fixed. i updated the patch and added a small test to: Lib/test/test_builtins.py::test_dir -tomer On 11/7/06, Nick Coghlan [EMAIL PROTECTED] wrote: tomer filiba wrote: cool. first time i build the entire interpreter, 'twas fun :) currently i retained support for __members__ and __methods__, so it doesn't break anything and is compatible with 2.6. i really hope it will be included in 2.6 as today i'm using ugly hacks in RPyC to make remote objects appear like local ones. having __dir__ solves all of my problems. besides, it makes a lot of sense of define __dir__ for classes that define __getattr__. i don't think it should be pushed back to py3k. here's the patch: http://sourceforge.net/tracker/index.php?func=detailaid=1591665group_id=5470atid=305470 As I noted on the tracker, PyObject_Dir is a public C API function, so it's behaviour needs to be preserved as well as the behaviour of calling dir() from Python code. So the final form of the patch will likely need to include stronger tests for that section of the API, as well as updating the documentation in various places (the dir and PyObject_Dir documentation, obviously, but also the list of magic methods in the language reference). +1 on targeting 2.6, too. Cheers, Nick. -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- http://www.boredomandlaziness.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] __dir__, part 2
so, if you remember, i suggested adding __dir__ to objects, so as to make dir() customizable, remove the deprecated __methods__ and __members__, and make it symmetrical to other built-in functions. you can see the original post here: http://mail.python.org/pipermail/python-dev/2006-July/067095.html which was generally accepted by the forum: http://mail.python.org/pipermail/python-dev/2006-July/067139.html so i went on, now that i have some spare time, to research the issue. the current dir() works as follows: (*) builtin_dir calls PyObject_Dir to do the trick (*) if the object is NULL (dir with no argument), return the frame's locals (*) if the object is a *module*, we're just using it's __dict__ (*) if the object is a *type*, we're using it's __dict__ and __bases__, but not __class__ (so as not to show the metaclass) (*) otherwise, it's a normal object, so we take it's __dict__, along with __methods__, __members__, and dir(__class__) (*) create a list of keys from the dict, sort, return we'll have to change that if we were to introduce __dir__. my design is: (*) builtin_dir, if called without an argument, returns the frame's locals (*) otherwise, it calls PyObject_Dir(self), which would dispatch self.__dir__() (*) if `self` doesn't have __dir__, default to object.__dir__(self) (*) the default object.__dir__ implementation would do the same as today: collect __dict__, __members__, __methods__, and dir(__class__). by py3k, we'll remove looking into __methods__ and __members__. (*) type objects and module objects would implement __dir__ to their liking (as PyObject_Dir does today) (*) builtin_dir would take care of sorting the list returned by PyObject_Dir so first i'd want you people to react on my design, maybe you'd find flaws whatever. also, should this become a PEP? and last, how do i add a new method slot? does it mean i need to change all type-object definitions throughout the codebase? do i add it to some protocol? or directly to the object protocol? -tomer ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __dir__, part 2
cool. first time i build the entire interpreter, 'twas fun :) currently i retained support for __members__ and __methods__, so it doesn't break anything and is compatible with 2.6. i really hope it will be included in 2.6 as today i'm using ugly hacks in RPyC to make remote objects appear like local ones. having __dir__ solves all of my problems. besides, it makes a lot of sense of define __dir__ for classes that define __getattr__. i don't think it should be pushed back to py3k. here's the patch: http://sourceforge.net/tracker/index.php?func=detailaid=1591665group_id=5470atid=305470 here's a demo: class foo(object): ... def __dir__(self): ... return [kan, ga, roo] ... f = foo() f __main__.foo object at 0x00A90C78 dir() ['__builtins__', '__doc__', '__name__', 'f', 'foo'] dir(f) ['ga', 'kan', 'roo'] dir(foo) ['__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__getattribute__ ', '__hash__', '__init__', '__module__', '__new__', '__reduce__', '__reduce_ex__ ', '__repr__', '__setattr__', '__str__', '__weakref__'] class bar(object): ... __members__ = [bow, wow] ... b=bar() dir(b) ['__class__', '__delattr__', '__dict__', '__doc__', '__getattribute__', '__hash_ _', '__init__', '__members__', '__module__', '__new__', '__reduce__', '__reduce_ ex__', '__repr__', '__setattr__', '__str__', '__weakref__', 'bow', 'wow'] -tomer On 11/6/06, Guido van Rossum [EMAIL PROTECTED] wrote: Sounds like a good plan, though I'm not sure if it's worth doing in 2.6 -- I'd be happy with doing this just in 3k. I'm not sure what you mean by adding a method slot -- certainly it's possible to define a method __foo__ and call it directly without having a special tp_foo in the type object, and I recommend doing it that way since the tp_foo slots are just there to make things fast; in this case I don't see a need for dir() to be fast. --Guido On 11/6/06, tomer filiba [EMAIL PROTECTED] wrote: so, if you remember, i suggested adding __dir__ to objects, so as to make dir() customizable, remove the deprecated __methods__ and __members__, and make it symmetrical to other built-in functions. you can see the original post here: http://mail.python.org/pipermail/python-dev/2006-July/067095.html which was generally accepted by the forum: http://mail.python.org/pipermail/python-dev/2006-July/067139.html so i went on, now that i have some spare time, to research the issue. the current dir() works as follows: (*) builtin_dir calls PyObject_Dir to do the trick (*) if the object is NULL (dir with no argument), return the frame's locals (*) if the object is a *module*, we're just using it's __dict__ (*) if the object is a *type*, we're using it's __dict__ and __bases__, but not __class__ (so as not to show the metaclass) (*) otherwise, it's a normal object, so we take it's __dict__, along with __methods__, __members__, and dir(__class__) (*) create a list of keys from the dict, sort, return we'll have to change that if we were to introduce __dir__. my design is: (*) builtin_dir, if called without an argument, returns the frame's locals (*) otherwise, it calls PyObject_Dir(self), which would dispatch self.__dir__() (*) if `self` doesn't have __dir__, default to object.__dir__(self) (*) the default object.__dir__ implementation would do the same as today: collect __dict__, __members__, __methods__, and dir(__class__). by py3k, we'll remove looking into __methods__ and __members__. (*) type objects and module objects would implement __dir__ to their liking (as PyObject_Dir does today) (*) builtin_dir would take care of sorting the list returned by PyObject_Dir so first i'd want you people to react on my design, maybe you'd find flaws whatever. also, should this become a PEP? and last, how do i add a new method slot? does it mean i need to change all type-object definitions throughout the codebase? do i add it to some protocol? or directly to the object protocol? -tomer ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] weakref enhancements
this may still be premature, but i see people misunderstood the purpose. weakattrs are not likely to be used externally, out of the scope of the object. they are just meant to provide an easy to use means for not holding cyclic references between parents and children. many graph-like structures, i.e., rpyc's node and proxies, are interconnected in both ways, and weakattrs help to solve that: i don't want a proxy of a node to keep the node alive. weakmethods are used very similarly. nodes have a method called getmodule, that performs remote importing of modules. i expose these modules as a namespace object, so you could do: mynode.modules.sys or mynode.modules.xml.dom.minidom.parseString instead of mynode.getmodule(xml.dom.minidom).parseString here's a sketch: def ModuleNamespace: def __init__(self, importerfunc): self.importerfunc = importerfunc class Node: def __init__(self, stream): self.modules = ModuleNamespace(self.getmodule) @ weakmethod def getmodule(self, name): i define this getmodule method as a *weakmethod*, so the mere existence of the ModuleNamespace instance will not keep the node alive. when the node loses all external references, the ModuleNamespace should just commit suicide, and allow the node to be reclaimed. yes, you can do all of these with today's weakref, but it takes quite a lot of hassle to manually set up weakproxies every time. -tomer On 9/29/06, Raymond Hettinger [EMAIL PROTECTED] wrote: [Alex Martelli] I've had use cases for weakrefs to boundmethods (and there IS a Cookbook recipe for them), Weakmethods make some sense (though they raise the question of why bound methods are being kept when the underlying object is no longer in use -- possibly as unintended side-effect of aggressive optimization). I'm more concerned about weakattr which hides the weak referencing from client code when it is usually the client that needs to know about the refcounts: n = SomeClass(x) obj.a = n del n # hmm, what happens now? If obj.a is a weakattr, then n get vaporized; otherwise, it lives. It is clearer and less error-prone to keep the responsibility with the caller: n = SomeClass(x) obj.a = weakref.proxy(n) del n # now, it is clear what happens The wiki-space example shows objects that directly assign a copy of self to an attribute of self. Even in that simplified, self-referential example, it is clear that correct functioning (when __del__ gets called) depends knowing whether or not assignments are creating references. Accordingly, the code would be better-off if the weak-referencing assignment was made explicit rather than hiding the weak-referencing wrapper in a descriptor. Raymond ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] weakref enhancements
i'd like to suggest adding weak attributes and weak methods to the std weakrefmodule. weakattrs are weakly-referenced attributes. when the value they reference is no longer strongly-referenced by something else, the weakattrs nullify themselves. weakmethod is a method decorator, like classmethod et al, that returns weaklybound methods. weakmethod's im_self is a weakref.proxy to `self`, which meansthe mere method will not keep the entire instance alive. instead, you'll get a ReferenceError.i think these two features are quite useful, and being part of the stdlib, would provide programmers with easy-to-use solutions to object-aliveness issues.more info, examples, and suggested implementation: * http://sebulba.wikispaces.com/recipe+weakattr* http://sebulba.wikispaces.com/recipe+weakmethod -tomer ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] weakref enhancements
I'm sceptical that these would find use in practice. [..] Also, I question the utility of maintaining a weakref to a method or attribute instead of holding one for the object or class. As long as the enclosing object or class lives, so too will their methods and attributes. So what is the point of a tighter weakref granualarity?i didn't just came up with them out of boredom, i have had specific use cases for these, mainly in rpyc3000... but since the rpyc300 code base is still far from completion, i don't want to give examples at this early stage.however, these two are theoretically useful, so i refactored them out of my code into recipes.-tomerOn 9/28/06, Raymond Hettinger [EMAIL PROTECTED] wrote: Also, I question the utility of maintaining a weakref to a method or attribute instead of holding one for the object or class.Strike that paragraph -- the proposed weakattrs have references away from the object, not to the object.Raymond ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] dict containment annoyance
a={1:2, 3:4} [] in aTraceback (most recent call last): File stdin, line 1, in ?TypeError: list objects are unhashableimo, the _expression_ should just evaluate to False instead of raising an exception.
it's a question of semantics -- i asked whether the object (a list, in this case)is contained in the dict. i didn't ask whether it's hashable or not. if the object isunhashable, then surely it's not contained in the dict, therefore the answer is
False, rather than an exception.dict.__contains__ should just swallow these exceptions, and return False.-tomer
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PyThreadState_SetAsyncExc bug?
while working on a library for raising exceptions in the context of another thread, i've come across a bug in PyThreadState_SetAsyncExc. if i raise an instance, sys.exc_info() confuses the exception value for the exception type, and the exception value is set None. if i raise the type itself, the interpreter creates an instance internally, but then i can't pass arguments to the exception. code: = import threading import ctypes def _async_raise(tid, excobj): res = ctypes.pythonapi.PyThreadState_SetAsyncExc(tid, ctypes.py_object(excobj)) if res == 0: raise ValueError(nonexistent thread id) elif res 1: # if it returns a number greater than one, you're in trouble, # and you should call it again with exc=NULL to revert the effect ctypes.pythonapi.PyThreadState_SetAsyncExc(tid, None) raise SystemError(PyThreadState_SetAsyncExc failed) class Thread(threading.Thread): def raise_exc(self, excobj): assert self.isAlive(), thread must be started for tid, tobj in threading._active.items(): if tobj is self: _async_raise(tid, excobj) break # the thread was alive when we entered the loop, but was not found # in the dict, hence it must have been already terminated. should we raise # an exception here? silently ignore? def terminate(self): self.raise_exc(SystemExit()) if __name__ == __main__: import time import sys i_am_active = False def f(): global i_am_active i_am_active = True try: try: while True: time.sleep(0.01) except IOError, ex: print IOError handler except TypeError, ex: print TypeError handler print ex=, repr(ex) typ, val, tb = sys.exc_info() print typ=, repr(typ) print val=, repr(val) print tb=, tb finally: i_am_active = False t1 = Thread(target = f) t1.start() time.sleep(1) t1.raise_exc(TypeError(blah blah)) while i_am_active: time.sleep(0.01) print !! thread terminated output: = TypeError handler ex= None typ= exceptions.TypeError instance at 0x00C15D28 # should be the type val= None # should be the instance tb= traceback object at 0x00C159E0 !! thread terminated if i change: t1.raise_exc(TypeError(blah blah)) to: t1.raise_exc(TypeError) i get: = TypeError handler ex= exceptions.TypeError instance at 0x00C159B8 typ= class exceptions.TypeError at 0x00B945A0 val= exceptions.TypeError instance at 0x00C159B8 tb= traceback object at 0x00C15D00 !! thread terminated but then of course i can't pass arguments to the exception -tomer ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PyThreadState_SetAsyncExc bug?
so it should be fixed, or at least checked for conformness by the code.-tomerOn 8/11/06, Tim Peters
[EMAIL PROTECTED] wrote:[tomer filiba] while working on a library for raising exceptions in the context
of another thread, i've come across a bug in PyThreadState_SetAsyncExc. if i raise an instance, sys.exc_info() confuses the exception value for the exception type, and the exception value is set None. if i raise the
type itself, the interpreter creates an instance internally, but then i can't pass arguments to the exception.That appears to be the way it was designed; i.e., AFAICT, it's workingas intended.This follows from the code in
ceval.c that raises theexception:if (tstate-async_exc != NULL) {x = tstate-async_exc;tstate-async_exc = NULL;
PyErr_SetNone(x);Py_DECREF(x);why = WHY_EXCEPTION;goto on_error;
}PyErr_SetNone(x) there gives no possibility that setting an /instance/could work as you hope -- `x` has to be an exception type, andtstate-async_exc is simply the `exc` argument that was passed to
PyThreadState_SetAsyncExc().
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] patching pydoc?
i have a problem with pydoc in rpyc. i wanted help(obj), where obj
is a NetProxy object, to work as if it were local.
i followed the code starting from site.help to pydoc.doc, which is the
ultimate function that generates and prints the text. i expected there
would be some function in the middle that prepares the text, and
another that writes it to the pager, but to my disappointment pydoc.doc
does both.
this means i can't transfer the document to my local machine (it's
printed directly to the remote console).
therefore, i would like to split this behavior into two parts:
* render_doc - a function that returns the document text
* doc - a function that calls render_doc and sends it to the pager
this way no existing code breaks (no existing function signatures
are changed) and i gain help on remote objects.
i hope people would be in favor, as it's not such a big change anyway.
is it possible to add to 2.5?
-tomer
this is the code of pydoc, starting at line 1457
def doc(thing, title='Python Library Documentation: %s', forceload=0):
Display text documentation, given an object or a path to an object.
try:
object, name = resolve(thing, forceload)
desc = describe(object)
module = inspect.getmodule(object)
if name and '.' in name:
desc += ' in ' + name[:name.rfind('.')]
elif module and module is not object:
desc += ' in module ' + module.__name__
if not (inspect.ismodule(object) or
inspect.isclass(object) or
inspect.isroutine(object) or
isinstance(object, property)):
# If the passed object is a piece of data or an instance,
# document its available methods instead of its value.
object = type(object)
desc += ' object'
pager(title % desc + '\n\n' + text.document(object, name))
except (ImportError, ErrorDuringImport), value:
print value
this is the suggested code
def render_doc(thing, title='Python Library Documentation: %s', forceload=0):
generate the text
object, name = resolve(thing, forceload)
desc = describe(object)
module = inspect.getmodule(object)
if name and '.' in name:
desc += ' in ' + name[:name.rfind('.')]
elif module and module is not object:
desc += ' in module ' + module.__name__
if not (inspect.ismodule(object) or
inspect.isclass(object) or
inspect.isroutine(object) or
isinstance(object, property)):
# If the passed object is a piece of data or an instance,
# document its available methods instead of its value.
object = type(object)
desc += ' object'
return title % desc + '\n\n' + text.document(object, name)
def doc(*args, **kwargs):
Display text documentation, given an object or a path to an object.
try:
text = render_doc(*args, **kwargs)
pager(text)
except (ImportError, ErrorDuringImport), value:
print value
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Fwd: patching pydoc?
submitted patch:
https://sourceforge.net/tracker/?func=detailatid=305470aid=1530482group_id=5470
-tomer
-- Forwarded message --
From: tomer filiba [EMAIL PROTECTED]
Date: Jul 28, 2006 3:35 PM
Subject: patching pydoc?
To: python-dev@python.org
i have a problem with pydoc in rpyc. i wanted help(obj), where obj
is a NetProxy object, to work as if it were local.
i followed the code starting from site.help to pydoc.doc, which is the
ultimate function that generates and prints the text. i expected there
would be some function in the middle that prepares the text, and
another that writes it to the pager, but to my disappointment pydoc.doc
does both.
this means i can't transfer the document to my local machine (it's
printed directly to the remote console).
therefore, i would like to split this behavior into two parts:
* render_doc - a function that returns the document text
* doc - a function that calls render_doc and sends it to the pager
this way no existing code breaks (no existing function signatures
are changed) and i gain help on remote objects.
i hope people would be in favor, as it's not such a big change anyway.
is it possible to add to 2.5?
-tomer
this is the code of pydoc, starting at line 1457
def doc(thing, title='Python Library Documentation: %s', forceload=0):
Display text documentation, given an object or a path to an object.
try:
object, name = resolve(thing, forceload)
desc = describe(object)
module = inspect.getmodule(object)
if name and '.' in name:
desc += ' in ' + name[:name.rfind('.')]
elif module and module is not object:
desc += ' in module ' + module.__name__
if not (inspect.ismodule(object) or
inspect.isclass(object) or
inspect.isroutine(object) or
isinstance(object, property)):
# If the passed object is a piece of data or an instance,
# document its available methods instead of its value.
object = type(object)
desc += ' object'
pager(title % desc + '\n\n' + text.document(object, name))
except (ImportError, ErrorDuringImport), value:
print value
this is the suggested code
def render_doc(thing, title='Python Library Documentation: %s', forceload=0):
generate the text
object, name = resolve(thing, forceload)
desc = describe(object)
module = inspect.getmodule(object)
if name and '.' in name:
desc += ' in ' + name[:name.rfind('.')]
elif module and module is not object:
desc += ' in module ' + module.__name__
if not (inspect.ismodule(object) or
inspect.isclass(object) or
inspect.isroutine(object) or
isinstance(object, property)):
# If the passed object is a piece of data or an instance,
# document its available methods instead of its value.
object = type(object)
desc += ' object'
return title % desc + '\n\n' + text.document(object, name)
def doc(*args, **kwargs):
Display text documentation, given an object or a path to an object.
try:
text = render_doc(*args, **kwargs)
pager(text)
except (ImportError, ErrorDuringImport), value:
print value
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] patching pydoc?
Giving the amount of hair-tearing over uuid and __index__, this seems like an especially bad day to ask for a new-feature variance in a time of feature freeze ;-). yeah, i guess so. * Would making pager() a parameter of doc() make sense? not at all. * I presume you gain the new functionality by directly calling the factored-out render_doc and printing thru your own pager. Does everyone? i don't quite get the question, but yes, i plan to call render_doc instead of doc, getting the doc text, and printing it on my own console using a pager/whatever. about the signature, okay, it can easily be fixed. i didn't think about that too much. about the other point -- ImportError etc -- i tried to go dumb as much as possible and not make any change that would break something. pydoc is poorly written anyway, and as ron adam noted, there is a need for at least a cleaned-up version of pydoc, or even a redesigned package. and it ought to be a package. it's a module of ~2000 lines. that's way too much for a module. i made the minimal changes needed to separate the generate text part from the generate and display part. and i kept all the semantics in tact (including who handles the errors) and therefore, i don't expect it to break any tests (after restoring the original doc signature, that is). i'd hate to wait for 2.6 for such a small change. today my solution is to replace pydoc.pager by a fake pager that hands over the result. this is of course not thread safe, as multiple threads calling help() would mix their outputs. not very likely, but i'd love to see this code go away asap. thanks for the comments, -tomer From: Terry Reedy tjreedy at udel.edu Subject: Re: patching pydoc? Newsgroups: gmane.comp.python.devel Date: 2006-07-28 18:29:50 GMT (5 hours and 27 minutes ago) tomer filiba tomerfiliba at gmail.com wrote in message news:1d85506f0607280635q3a693682l230c7821dc6f408f at mail.gmail.com... ... therefore, i would like to split this behavior into two parts: * render_doc - a function that returns the document text * doc - a function that calls render_doc and sends it to the pager this way no existing code breaks (no existing function signatures are changed) and i gain help on remote objects. i hope people would be in favor, as it's not such a big change anyway. is it possible to add to 2.5? Giving the amount of hair-tearing over uuid and __index__, this seems like an especially bad day to ask for a new-feature variance in a time of feature freeze ;-). Some quick questions: * I presume you gain the new functionality by directly calling the factored-out render_doc and printing thru your own pager. Does everyone? * Would making pager() a parameter of doc() make sense? * Is pager() the only part of the original doc() that can generate ImportError, ErrorDuringImport? If not, the try/except should be in render_doc also or instead. * Why generalize the doc() signature? Bad calls will be traced as caught in render_doc instead of doc. Couldn't that potentially break a bad_call test? Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] exception too expensive?
i thought avoiding a second dict lookup should be faster, but it turned out to be completely wrong. it's only marginally faster, but if an exception occurs,it's x10 slower.## the code# import time b = dict((i, 7) for i in range(1000)) def try_lookup(k):... try:... return b[k]... except KeyError:... return 17... def if_lookup(k):... if k in b:... return b[k]... return 17 ... def test(f, k, count=10):... t=time.time()... while count:... count -=1... f(k)... return time.time()-t...## try_lookup with valid key # test(try_lookup, 56, 100)0.608950958252 test(try_lookup, 56, 100)0.6086757507324 test(try_lookup, 56, 100)0.608950958252~0.61 sec ## if_lookup with valid key# test(if_lookup, 56, 100)0.6876376037598 test(if_lookup, 56, 100)0.6873623962402 test(if_lookup, 56, 100) 0.67200016975402832~0.68 sec## try_lookup with invalid key# test(try_lookup, 9456, 100)7.06236239624 test(try_lookup, 9456, 100)6.81236239624 test(try_lookup, 9456, 100)6.8440001010894775~6.90 sec## if_lookup with invalid key# test(if_lookup, 9456, 100)0.625 test(if_lookup, 9456, 100) 0.6413242492676 test(if_lookup, 9456, 100)0.65599989891052246~0.64 sec==before someone says why don't you use dict.get, it's not applicable inmy code. i have a cache -- either object already exists in the cache, or i create and store it in the cache -- so get() and setdefault() are not useful.so first of all, i'd recommend not using the try_lookup method; use the if_lookup instead... unless you can be sure the changes a KeyError will be raised are marginal.second, isn't there anything that can be done to improve the performanceof exceptions? imo, exceptions should be cheap. i understand it has to do with creating the exception instance everytime, but can't something be done to improve that? for example, hold a cache exception instances and just memcpy them when needed (just a wild idea)?-tomer ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] weakattr
I like the added functionality offered with weakattrs as defined. I'm not terribly in love with the syntax of their creation, and I'm curious as to how it plays with __slots__ weakattrs are data descriptors, just like properties etc. they are part of the class, not the instance, so there shouldn't be any trouble with mixing those with __slots__ moreover, adding those to stdlib is very staight-forward. we don't even need to introduce a new module. if people show interest, i'll write a bit of a longer doc string and add some unit tests (although there's not much to test :) ) Toss it out in python-list, I think some people over there would be able to offer more feedback. will do... although i doubt they will offer any -tomer On 7/2/06, Josiah Carlson [EMAIL PROTECTED] wrote: tomer filiba [EMAIL PROTECTED] wrote: weakattr (weak attributes) are attributes that are weakly referenced by their containing object. they are very useful for cyclic references -- an object that holds a reference to itself. I like the added functionality offered with weakattrs as defined. I'm not terribly in love with the syntax of their creation, and I'm curious as to how it plays with __slots__ (not quite having the time to look at its implementation right now), but it is quite explicit, so I can get past that. It would allow us to say, stop using __del__, use weakattrs, but I'm not sure how well that would work, generally. Toss it out in python-list, I think some people over there would be able to offer more feedback. - Josiah ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] weakattr
weakattr (weak attributes) are attributes that are weakly referencedby their containing object. they are very useful for cyclic references --an object that holds a reference to itself. when a cyclic reference is found by the GC, the memory may be freed, but __del__ is not called, because it's impossible to tell which __del__ to call first. this is an awkward asymmetry with no clean solution: most such objects provide a close or dispose method that must be called explicitly.weakattrs to solve this problem, by providing a magical attributethat disappears when the attribute is no longer strongly-referenced. you can find the code, as well as some examples, on this link http://sebulba.wikispaces.com/recipe+weakattr since the stdlib already comes with weakref.py, which provideshigher level concepts over the builtin _weakref module, i'd like tomake weakattr a part of it. it's only ~20 lines of code, and imho saves the trouble of explicitly releasing the resource of un__del__able objects.i think it's useful. here's a snippet: from weakref import weakattr class blah(object):... yada = weakref() ... o1 = blah() o2 = blah() o1.yada = o2 o2.yada = o1o1.yada is a *weakref* to o2, so that when o2 is no longer strongly-referenced... del o2 o1.yada magically disappears as well. o1.yada... AttributeError(...)since the programmer explicitly defined yada as a weakatt, he/she knows it might disappear. it might look awkward at first, but that's exactly the *desired* behavior (otherwise we'd just use the regular strong attributes).another thing to note is that weakattrs are likely to be gone onlywhen the object's __del__ is already invoked, so the only code that needs to take such precautions is __del__ (which already has some constraints)for example: class blah(object):... me = weakattr().. def __init__(self):... self.me = self.. def something(self):... # we can rest assure me exists at this stage... print self.me.. def __del__(self):... # by the time __del__ is called, me is removed ... print me exists?, hasattr(self, me)... b = blah() b.me__main__.blah object at 0x00C0EC10 b.something()__main__.blah object at 0x00C0EC10 del b import gc gc.collect()me exists? False0-tomer ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] suggestion: except in list comprehension
a friend of mine suggested this, and i thought i should share it with the mailing list.many times when you would want to use list/generator comprehensions, you have tofall back to the old for/append way, because of exceptions. so it may be a good idea to allow an exception handling mechanism in these language constructs.since list comprehensions are expressions, an exceptions thrown inside one meansthe entire list is discarded. you may want to provide some, at least fundamental, error handling, like if this item raises an exception, just ignore it, or terminate theloop in that case and return whatever you got so far.the syntax is quite simple:[ expr for expr in expr [if cond] [except exception-class-or-tuple: action] ] where action is be one of continue or break:* continue would mean ignore this item* break would mean return what you got so farfor example: a = [i, fart, in, your, general, 5, direction, you, silly, english, kniggits]give me every word that starts with y, ignoring all errors b = [x for x in a if x.startswith(y) except: continue]# returns [your, you] or only AttributeError to be more speciifcb = [x for x in a if x.startswith(y) except AttributeError: continue]# returns [your, you]and with break b = [x for x in a if x.startswith(y) except AttributeError: continue] # returns only [your] -- we didn't get past the 5 in order to do something like this today, you have to resort to the old way,b = []for x in a: try: if x.startswith(y):b.append(x) except ...: pass which really breaks the idea behind list comprehension. so it's true that this example i provided can be done with a more complex if condition (first doing hasattr), but it's not always possible, and how would you do it if the error occurs at the first part of the _expression_? y = [4, 3, 2, 1, 0, -1, -2, -3] [1.0 / x for x in y except ZeroDivisionError: break][0.25, 0.333, 0.5, 1.0] [1.0 / x for x in y except ZeroDivisionError: continue] [0.25, 0.333, 0.5, 1.0, -1.0, -0.5, -0.333]again, in this example you can add if x != 0, but it's not always possible to tell whichelement will fail. for example:filelist = [a, b, c, \\/invalid file name:?*, e] openfiles = [open(filename) for filename in filelist except IOError: continue]the example is hypothetical, but in this case you can't tell *prior to the exception*that the operation is invalid. the same goes for generator expressions, of course. -tomer ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] suggestion: except in list comprehension
first, i posted it two days ago, so it's funny it got posted only now... the moderators are sleeping on the job :)anyway. Note that of the continue cases you offer, all of them are merely simple
if conditionyes, i said that explicitly, did you *read* my mail?but i also said it's not always possible. you *can't* always tell prior to doing somethingthat it's bound to fail. you may have os.path.isfile
, but not an arbitrary is_this_going_to_fail(x)and doing [1.0 / x for x in y if z(x != 0)]is quite an awkward way to say break... if then_break(cond) instead of
if cond then break- - - - -anyway, i guess my friend may have better show-cases, as he's the one who found theneed for it. i just thought i should bring this up here. if you think better show cases
would convince you, i can ask him. If you couldn't guess; -1, you can get equivalent behavior without complicating the generator _expression_/list comprension syntax.so how come list comprehensions aren't just a complication to the syntax?
you can always do them the old way,x = []for : x.append(...)but i since people find {shorter / more to-the-point / convenience} reason enough to have introduced the list-comprehension syntax in the first place, there's also a case
for adding exception handling to it.if the above snippet deserves a unique syntax, why not extend it to cover this as well?x = []
for : try: x.append(...) except: ...and it's such a big syntactic change.don't worry, i'm not going to argue it past this.-tomer
On 4/26/06, Josiah Carlson [EMAIL PROTECTED] wrote:
tomer filiba [EMAIL PROTECTED] wrote: [ expr for expr in expr [if cond] [except exception-class-or-tuple: action] ]
Note that of the continue cases you offer, all of them are merely simpleif condition (though the file example could use a better test thanos.path.isfile).[x for x in a if x.startswith(y) except AttributeError: continue]
[x for x in a if hasattr(x, 'startswith') and x.startswith(y)][1.0 / x for x in y except ZeroDivisionError: continue][1.0 / x for x in y if x != 0][open(filename) for filename in filelist except IOError: continue]
[open(filename) for filename in filelist if os.path.isfile(filename)]The break case can be implemented with particular kind of instanceobject, though doesn't have the short-circuiting behavior...
class StopWhenFalse:def __init__(self):self.t = 1def __call__(self, t):if not t:self.t = 0return 0return self.tz = StopWhenFalse()
Assuming you create a new instance z of StopWhenFalse before doing thelist comprehensions...[x for x in a if z(hasattr(x, 'startswith') and x.startswith(y))][1.0 / x for x in y if z(x != 0)]
[open(filename) for filename in filelist if z(os.path.isfile(filename))]If you couldn't guess; -1, you can get equivalent behavior withoutcomplicating the generator _expression_/list comprension syntax.
- Josiah
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] proposal: evaluated string
many times, templating a string is a tidious task. using the % operator, either with tuples or dicts,is difficult to maintain, when the number of templated arguments is large. and string.Template,although more easy to read, is less intutive and cumbersome:
import stringt = string.Template(hello $name)print t.substitute({name : john})as you can see, it is is redundant, as you must repeat the dict keys in two places, and imagine
maintaining such a template with 20 parameters! if you change the one argument's name in the template, you must go and fix all locations that use that template. not nice at all.i'm suggesting something like boo's string interpolation:
http://boo.codehaus.org/String+Interpolationbut i chose to call it evaluated string.like raw strings (r), which are baiscally a syntactic sugar, evaluated strings will be marked by 'e',
for instance, e, which may be combined with the 'r' or 'u', that exist today.the evaluated string will be evaluated based on the current scope (locals and globals), just likenormal expressions. the difference is, the results of the expressions will be str()ed into the
evaluated string directly. these expressions will be embedded into the string surrounded byspecial delimiters, and an unclosed delimited or a syntax error will be reported at just like \x??raises ValueError: invalid \x escape.
i'm not sure which delimiters to use, but i think only { } is sufficient (no need for ${ } like in boo)some examples:===name = johnprint ehello {name}
a = 3b = 7print ethe function is y = {a}x + {b}for x in range(10): print ey({x}) = {a*x+b}import time, sysprint ethe time is {time.asctime()} and you are running on {
sys.platform}===in order to implement it, i suggest a new type, estr. doing a = ehellowill be equivalent to a = estr(hello, locals(), globals()), just like
uhellois equivalent tounicode(hello) (if we ignore \u escaping for a moment)and just like unicode literals introduce the \u escape, estr literals would introduce \{ and \}
to escape delimiters.the estr object will be evaluated with the given locals() and globals() only at __repr__ or __str__,which means you can work with it like a normal string:a = ehello {name}
b = ehow nice to meet you at this lovely day of {time.localtime().tm_year}c = a + bc is just the concatenation of the two strings, and it will will be evaluated as a whole when youstr()/repr() it. of course the internal representation of the object shouldnt not as a string,
rather a sequence of static (non evaluated) and dynamic (need evaluation) parts, i.e.:[hello, name, how nice to meet you at this lovely day of, time.localtime().tm_year],
so evaluating the string will be fast (just calling eval() on the relevant parts)also, estr objects will not support getitem/slicing/startswith, as it's not clear what the indexes are...
you'd have to first evaluate it and then work with the string:str(ehello)[2:]estr will have a counterpart type called eunicode. some rules:estr + str = estrestr + estr = estrestr + unicode = eunicode
estr + eunicode = eunicodeeunicode + eunicode = eunicodethere are no backwards compatibility issues, as e is an invalid syntax today, and as for clarity, i'm sure editors like emacs and the like can be configured to highlight the strings enclosed by {}
like normal expressions.i know it may cause the perl-syndrome, where all the code of the program is pushed into strings,but templating/string interpolation is really a fundamental requirement of scripting languages,
and the perl syndrome can be prevented with two precautions:* compile the code with eval flag instead of exec. this would prevent abominations likee{import time\ndef f(a):\n\tprint 'blah'}
* do not allow the % operator to work on estr's, to avoid awful things likeehow are %s {%s} % (you, name)one templating mechanism at a time, please :)perhaps there are other restrictions to impose, but i couldnt think of any at the moment.
here's a proposed implementation:class estr(object): # can't derive from basestring! def __init__(self, raw, locals, globals): self.elements = self._parse(raw) self.locals = locals
self.globals = globals def _parse(self, raw): i = 0 last_index = 0 nesting = 0 elements = [] while i len(raw): if raw[i] == {:
if nesting == 0: elements.append((False, raw[last_index : i])) last_index = i + 1 nesting += 1 elif raw[i] == }:
nesting -= 1 if nesting == 0: elements.append((True, raw[last_index : i])) last_index = i + 1 if nesting 0: raise ValueError(too many '}' (at index %d) % (i,))
i += 1 if nesting 0: raise ValueError(missing '}' before end) if last_index i: elements.append((False, raw[last_index : i])) return elements
def __add__(self, obj): if type(obj) == estr: elements = self.elements + obj.elements else: elements = self.elements + [(False, obj)] # the new object inherits the current one's namespace (?)
newobj = estr(, self.locals, self.globals) newobj.elements = elements return newobj def __mul__(self, count): newobj =
Re: [Python-Dev] proposal: evaluated string
just like r does the escaping for you.but estr types must be implemented so the evaluate with the current scope (locals and globals),not the score they were defined it, so unless you want to do nasty tricks with sys._getframe, which doesn't work on all implementations of python, you'll need it as a builtin-tomerOn 4/20/06, Josiah Carlson [EMAIL PROTECTED] wrote:tomer filiba [EMAIL PROTECTED] wrote: the evaluated string will be evaluated based on the current scope (locals and globals), just like normal expressions. the difference is, the results of the expressions will be str()ed into the evaluated string directly. these expressions will be embedded into the string surrounded by-1You are basically suggesting that e... be a replacement for ...%locals() .That doesn't seem to me to be sufficiently compelling(yes, I do use string interpolation). - Josiah ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] proposal: evaluated string
We already have a slew of templating utilities (see Cheetah for example).first of all -- i know there's a bunch of templating engines, but i think it should be a
built-in feature of the language. like boo does. and estr is stronger than simple $name substitution, like Template does.
Be sure to stay aware of the security risks if the fill-in values are user specified.that's one major benefit of having it as a builtin type -- you dont have security risks,as the _expression_ itself is embedded in your code, not something you get from the
outside:name = raw_input(what's you name?)print ehello {name}does not get the *_expression_* from the user, only the *values*, so unless the usercauses a buffer overflow with a huge string, he can't run code. the estr object is part
of *your* code, which you trust. If you need this, then consider using a third-party templating module.
that 50-liner estr class i presented does just that.Using the key twice is basic to templating (once of specify where to
make the substitution and once to specify its value). This is nodifferent from using variable names in regular code: a=1; ... ; b =a+2 # variable-a is used twice.but when it's defined once as an argument to a function, once in the template,
and once in the dict, that's three times, where it can be only two.def f(name): print ehello {name}
Also, the example is misleading because real-apps are substitutevariables, not constants. IOW, the above code fragment is sematicallyequivalent to: print hello john.what do you mean by that?
3) it is lessflexible than the class constructor which can be subclassed and
extended as needed.do you often subclass str? it's a built-in type, part of the language, subclassing it doesntmake much sense. after all it's the language compiler that instanciates these types,
i.e.,when you do hello, the compiler creates an instance of str() with that value, not youdirectly, and that's the case here.-tomerOn 4/20/06,
Raymond Hettinger [EMAIL PROTECTED] wrote:
If you don't like the $name style of template markup and preferdelimiters instead, then check-out the recipe at:
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/3053The link should have been: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/305306
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] adding Construct to the standard library?
Giovanni Bajo: Both ctypes and construct provide a way to describe a binary-packed structure in Python terms: and this is an overload of functionalityso does struct, so why not just use struct? there's a receipe at the python cookbook that adds naming ability to fields, i.e.6s.destincation 6s.source H.typesomething like that, so when you parse you get named attributes instead ofa tuple. so why did ctypes wrote another mechanism? because you can't extend it and you can't nest it. but ctypes, just as well, provides the mechanisms for its requirements -- defining C structs, not arbitrarily complex structures. so of courseit doesnt, and shouldnt, support variable-length fields or data pointers, which are common in file formats, protocols, and other complex data structures -- whatyou can't do with a C struct you don't need to do with ctypes.now i'll save me a mail and put this also here: Greg Ewing: It does seem rather silly to have about 3 or 4 different incompatible ways to do almost exactly the same thing (struct, ctypes, NumPy and now Construct).* struct is designed for packing and unpacking, but is very limited * ctypes is not oriented at packing/unpacking, it only provided amechanism to handle its requirements, which are domain specificand not general purpose.* i never checked how NumPy packs arrays, etc., but it's also domain-specific, as it's a math library, not a generic packer/unpacker.and trust me those are not the only ones. the reason people have to *reinvent the wheel* every time is the lack of a *generic* parsing/building library. (i prefer the term parsing over unpacking. check my blog for moredetails)yes, putting bytes together isn't too complicated, and because peopledon't have a built-in mechanism for that, they tend to just oh, well, it can't be too complicated, i'll just write one for me, and this yields manyflavors of packers/unpackers, all incompatible.Construct is the first library, that i'm aware of, that is dedicated to parsing/building, instead of doing it as a side-kick domain-specificmechanism.Construct is a *superset* of all those packers and unpackers, andhad it been part of stdlib, people would have used it instead. of course it's only been released a month ago, and couldnt have been already included in the stdlib, i still think it has a room there. existing projects can be ported without too much effort, and new ones could benefit from it as well.-tomerOn 4/19/06, Giovanni Bajo [EMAIL PROTECTED] wrote: tomer filiba [EMAIL PROTECTED] wrote: the point is -- ctypes can define C types. not the TCP/IP stack. Construct can do both. it's a superset of ctype's typing mechanism. but of course both have the right to *coexist* -- ctypes is oriented at interop with dlls, and provides the mechanisms needed for that. Construst is about data structures of all sorts and kinds. ctypes is a very helpful library as a builtin, and so is Construct. the two don't compete on a spot in the stdlib. I don't agree. Both ctypes and construct provide a way to describe abinary-packed structure in Python terms: and this is an overload offunctionality. When I first saw Construct, the thing that crossed my head was: hey, yet another syntax to describe a binary-packed structure in Python.ctypes uses its description to interoperate with native libraries, whileConstruct uses its to interoperate with binary protocols. I didn't see a good reason why you shouldn't extend ctypes so to provide features that it iscurrently missing. It looks like it could be easily extended to do so.Giovanni Bajo ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] bug with __dict__?
overriding __getattr__ and __setattr__ has several negative side effects, for example:* inside__getattr__/__setattr__ you have to use self.__dict__[attr] instead of self.attr* it's easy to get stack overflow exceptions when you do something wrong * you must remember to call the super's [get/set]attr or the MRO is broken* when deriving from a class that overrides one of the speical methods, you usuallyneed to override the special function of your class as well, to allow some local storage for yourselfso i had an idea -- why not just replace __dict__? this does not affect the MRO. i wrote an AttrDict class, which is like dict, only it allows you to acces its keys as attributes. i later saw something like this on the python cookbook as well. class AttrDict(dict): def __init__(self, *a, **k): dict.__init__(self, *a, **k) self.__dict__ = selfthis code basically causes __getattr/setattr__ to use __getitem/setitem__: a =AttrDict()a[blah] = 5a.yadda = 6print a.blahprint a[yadda]which is exactly what i wanted. now, i thought, instead of overriding __getattr/setattr__, i'd just write a class that overrides __getitem/setitem__. for example: # old wayclass A(object): def __getattr__(self, name): return 5a = A()print a.xyz # 5# new wayclass mydict(dict): def __getitem__(self, key): return 5 class A(object): def __init__(self): self.__dict__ = mydict()a = A()print a.xyz # should print 5but lo and behold, python does not call my overriden method. it just bypasses it and always calls the original dict's method. i made several tests, trying to see if it calls __contains__,placed hooks on __getattribute__, but nothing from *my* overriden methods is ever called. this is probably because the C implementation calls PyDict_GetItem or whatever directly... i think it's a bug. i should be able to override the __getitem__ of my __dict__. it breaks the polymorphism of python-level objects! on the one hand, i'm allowed to change __dict__ to anyobject that derives from dict, but i'm not allowed to override it's methods! python must either disable assigning to __dict__, or be willing to call overriden methods, notsilently ignore mine.-tomer ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] adding Construct to the standard library?
Indeed, I wish I had known about this a year ago; it would have saved me a lot of work. Of course it probably didn't exist a year ago... :( well, yeah. many people need parsing-abilities, but they resort to ad-hoc parsers using struct/some ad-hoc implementation of their own. there clearly is a need for a generic, strong, and extensible parsing/building mechanism. Well, declarative is less flexible. OTOH declarative is nice in the way it is more readable and allows more optimisations. i don't think less flexible is the term. it's certainly different, but if you need something specific, you can always subclass a construct on your own. other than that, being declarative means easy to read/write/maintain/debug/upgrage (to a newer version of the library). IMHO, at least in theory Construct could have small but fast C extension to take care of the encoding and decoding, which is the critical path. Everything else, like the declaration part, can be python, as it is usually done once on application startup. well, i expected the encodings package to have a str.encode(bin) and str.decode(bin)... for some reason there's no such codec. it's a pity. This is a very nice library indeed. But the number one feature that I need in something like this would be to use C. That's because of my application specific requirements, where i have observed that reapeatedly using struct.pack/unpack and reading bytes from a stream represents a considerable CPU overhead, whereas the same thing in C would be ultra fast.well, you must have the notion of a stream, i.e., go back and forth, be able to read/write bits/bytes at arbitrary locations, etc. i thought of moving the library to pyrex, and compiling it, but the number of critical parts is very small -- basically only the Repeater class could be improved by writing it in C. i mean, most of the time is consumed at creating objects in the objects tree, etc. for example, the Struct class simply iterates over the nested construsts and parses each of the in that sequence. doing a pythonic iteration of a C-level iteration over a pythonic object is practically the same. If you agree to go down this path I might even be able to volunteer some of my time to help, but it's not my decision. well, mainly i'm looking for ideas. just moving it to c wouldnt be too helpful. some ideas i have:* making the stream work with bytes instead of bits, so that memory consumption would decrease 8-fold... but then parsing unaligned fields (either by size of position) is gonna be a headache * unifying the context tree with the parsing/building tree, to create less objects on the fly (but it has some issues)* using lambda functions for meta expressions, instead of eval(string) -- perhaps it's faster, but lambda is getting deprecated by python3k :( apart from that, i'm rely on inheritance in many places. if some classes are written in C and some in python, i'm not sure how it could work (can a C class inherit a pythonic one? would it be easy to extend?). and, that means users would have to compile the C sources, while now all they have to do is extract a zip file. and then i'd have to write makefiles, and maintain those also... it's getting dirty. i like the painless unzip-and-use installation. so if you have ideas, i'd be happy to hear those.thanks,-tomerOn 4/18/06, Gustavo Carneiro [EMAIL PROTECTED] wrote: why include Construct?* the struct module is very nice, but very limited and non-pythonic as well * pure python (no platform/security issues) IMHO this is a drawback. More on this below. * lots of people need to parse and build binary data structures, it's not an esoteric library * license: public domain* quite a large user base for such a short time (proves the need of the community) Indeed, I wish I had known about this a year ago; it would have saved me a lot of work. Of course it probably didn't exist a year ago... :( * easy to use and extend (follows the componentization pattern) * declarative: you don't need to write executable code for most cases Well, declarative is less flexible. OTOH declarative is nice in the way it is more readable and allows more optimisations. why not:* the code is (very) young. stable and all, but less than a month on the loose. * new features may still be added / existing ones may be changed in a non-backwards-compatible manner so why am i saying this now, instead of waiting a few months for it to maturet? well, i wanted to get feedback. those of you who have seen/used the library, please tell me what you think:* is it suitable for a standard library? * what more features would you want?* any changes you think are necessary? This is a very nice library indeed. But the number one feature that I need in something like this would be to use C. That's because of my application specific requirements, where i have observed that reapeatedly using struct.pack/unpack and reading bytes from a stream represents a considerable CPU overhead, whereas the same thing in C would be
Re: [Python-Dev] adding Construct to the standard library?
ctypes, as the name implies, is relevant to *C data structures* only.you cannot extend it and you cannot define complex things with it, at least noteasily. * ctypes doesn't have a way (that I'm aware of) to specify theendianness of types like c_short - so my example, when run on Windows(intel architecture) gives type = 8, rather than type = 2048 (from thewiki). But the wiki example doesn't explicitly specify endianness, so maybe that's a limitation in Construct as well?in ctypes, the endianness of numbers is determined by the platform, since they are passed to a C (platform-dependent) function. you share your addressspace with the dll you load -- so both python and the dll live on the same platform.so except of writing data to files or sockets, you shouldn't care about the byte ordering. in Construct you have UBInt16 and ULInt16, for big and little ordering. and UInt16 is an alias to UBInt16 (because network ordering is more common in protocols) * ctypes doesn't have an easy way to parse a string based on astructure definition - hence my str_to_ctype function. But that's atrivial helper function to write, so that's not a big issue.sorry, but if you mean people must use memmove in order to parse string, you better slap yourself. this is a python mailing list, not a C one. we don't have a concept of addressof() or physically moving data. we use objects andreferences. no offense, but so that's not a big issue makes me think you don't belong to this mailing list.I'm not sure I understand the other wiki examples - but the ones I do, look doable in ctypes.i gues you should also look at http://pyconstruct.wikispaces.com/demos to geta better understanding, but i only uploaded it a couple of hours ago. sorry for that. anyway, on the projects page i explain thoroughly why there is room for yet anotherparsing/building library. but for the example you mentioned above, the ethernet header, struct is good enough:struct.pack(6s6sH, 123456, ABCDEF, 0x0800) but --how would you parse a pascal-string (length byte followed by data of that length)using ctypes? how would you read a 61 bit, unaligned field? how would you convert \x00\x11P\x88kW to 00-11-50-88-6B-57, the way people would like to see MAC addresses? yeah, the MAC address is only a representation issue, but adapters can do much more powerful things. plus, people usually prefer seeing IP instead of 0x0800 in their parsed objects. how would you define mappings in ctypes? Personally, I'd rather see the ctypes facilities for structure packingand unpacking be better documented, and enhanced if necessary, rather than having yet another way of doing the same thing added to thestdlib.the stdlib is too messy already. it must be revised anyway, since it's full of shitnobody uses. the point is -- ctypes can define C types. not the TCP/IP stack. Construct can do both. it's a superset of ctype's typing mechanism. but of course both have the right to *coexist* -- ctypes is oriented at interop with dlls, and provides the mechanisms needed for that. Construst is about data structures of all sorts and kinds. ctypes is a very helpful library as a builtin, and so is Construct. the two don't competeon a spot in the stdlib.-tomer On 4/18/06, Paul Moore [EMAIL PROTECTED] wrote: On 4/17/06, tomer filiba [EMAIL PROTECTED] wrote: after several people (several 10) contacted me and said IMHO 'construct' is a good candidate for stdlib, i thought i should give it a try. of course i'm not saying it should be included right now, but in 6 months time, or such a timeframe (aiming at python 2.6? some 2.5.x release?)Now that ctypes is part of the standard library, that provides a structured datatype facility. Here's an example demonstrating thefirst example from the Construct wiki: from ctypes import * def str_to_ctype(s, typ):...t = typ() ...memmove(addressof(t), s, sizeof(t))...return t... class ethernet_header(Structure):..._fields_ = [(destination, c_char * 6),...(source, c_char * 6), ...(type, c_short)]... s = ABCDEF123456\x08\x00 e = str_to_ctype(s, ethernet_header) e.source'123456' e.destination 'ABCDEF' e.type8I'm not sure I understand the other wiki examples - but the ones I do,look doable in ctypes.There are a couple of things to note:* ctypes doesn't have a way (that I'm aware of) to specify the endianness of types like c_short - so my example, when run on Windows(intel architecture) gives type = 8, rather than type = 2048 (from thewiki). But the wiki example doesn't explicitly specify endianness, so maybe that's a limitation in Construct as well?* ctypes doesn't have an easy way to parse a string based on astructure definition - hence my str_to_ctype function. But that's atrivial helper function to write, so that's not a big issue. Personally, I'd rather see the ctypes facilities for structure packingand unpacking be better documented, and enhanced if necessary, ratherthan having yet another way of doing the same thing added to the stdlib.Paul
[Python-Dev] a flattening operator?
DISCLAIMERi'm not going to defend and fight for this idea too much. i only bringit up because it bothers me. i'm sure some people here would kill me for even suggesting this, and i really don't want to be killed right now,so i bring it up as something you should think about. nothing more./DISCLAIMERNOTEPEP-225 has some weird ideas which may or may not be related to this, but i don't understand how this magical ~ operator can do everything from tuple flattening to list arithmetics, replacing map(), changing the order of operations, deep-copying, list comprehension, rich comparison, and whatever not. so i don't consider this a serious PEP. looks more like an april fool's joke to me, and it seems those japanese celebrate it on september for some reason. /NOTE[reposted from comp.lang.python]as we all know, * (asterisk) can be used to inline or flatten a tuple into an argument list, i.e.: def f(a, b, c):... x = (1,2,3) f(*x) so... mainly for symmetry's sake, why not make a flattening operator that also works outside the context of function calls? for example: a = (1,2,3) b = (4,5) c = (*a, *b) # == (1,2,3,4,5) yeah, a + b would also give you the same result, but it could be used like format-strings, for templating tuples, i.e. c = (*a, 7, 8, *b) i used to have a concrete use-case for this feature some time ago, but i can't recall it now. sorry. still, the main argument is symmetry: it's a syntactic sugar, but it can be useful sometimes, so why limit it to function calls only? allowing it to be a generic operator would make things like this possible: f(*args, 7) # an implied last argument, 7, is always passed to the function today you have to do f(*(args + (7,))) which is quite ugly. and if you have to sequences, one being a list and the other being a tuple, e.g. x = [1,2] y = (3,4) you can't just x+y them. in order to concat them you'd have to use casting like f(*(tuple(x) + y)) instead of f(*x, *y) isn't the latter more elegant? just an idea. i'm sure people could come up with more creative use-cases of a standard flattening operator. but even without the creative use cases -- isn't symmetry strong enough an argument? why are function calls more important than regular expressions? and the zen supports my point: (*) Beautiful is better than ugly -- f(*(args + (7,))) is ugly (*) Flat is better than nested -- less parenthesis (*) Sparse is better than dense -- less noise (*) Readability counts -- again, less noise (*) Special cases aren't special enough to break the rules -- then why are function calls so special to add a unique syntactic sugar for them? the flattening operator would work on any sequence (having __iter__ or __next__), not just tuples and lists. one very useful feature i can think of is expanding generators, i.e.: print xrange(10) # == xrange(10) print *xrange(10) # == (0, 1, 2, 3, 4, 5, 6, 7, 8, 9) i mean, python already supports this half-way: def f(*args): ... print args ... f(*xrange(10)) (0, 1, 2, 3, 4, 5, 6, 7, 8, 9) so... why can't i just do print *xrange(10) directly? defining a function just to expand a generator? well, i could use list(xrange(10))to expand it, but it's less intuitive. the other way is with list-comprehension,[x for x in xrange(10)]which is just, but isn't *xrange(10) more to-the-point? also, There should be one-- and preferably only one --obvious way to do it... so which one? (*) list(xrange(10)) (*) [x for x in xrange(10)] (*) mylist.extend(xrange(10)) (*) f(*xrange(10)) they all expand generators, but which is the preferable way? and imagine this: f(*xrange(10), 7) this time you can't do *(xrange(10) + (7,)) as generators do not support addition... you'd have to do *(tuple(xrange(10)) + (7,)) which is getting quite long already. so as you can see, there are many inconsistencies between function-call expressions and regular expressions, that impose artificial limitations on the language. after all, the code is already in there to support the function-call version... all it takes is adding support for regular expressions.so, what do you think? isn't symmetry worth it?-tomer ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] adding Construct to the standard library?
hello folksafter several people (several 10) contacted me and said IMHO 'construct' is a good candidate for stdlib,i thought i should give it a try. of course i'm not saying it should be included right now, but in 6 months time, or such a timeframe (aiming at python 2.6? some 2.5.x release?)a little intro:Construct ( http://pyconstruct.sourceforge.net/) is a library for declaratively defining data structures at the bit-level. these constructs can be used to parse raw data into objects, or build objects into raw data. you can see a couple of examples at http://pyconstruct.wikispaces.com/examples being data structures they are not limited to simple structures -- they can be linked lists, for example, or an enitreefl32 file, with sections and pointers (included in the distribution). currently i'm writing a parser of ext2 file systems, to allow inspecting file systems without mounting.why include Construct?* the struct module is very nice, but very limited and non-pythonic as well* pure python (no platform/security issues)* lots of people need to parse and build binary data structures, it's not an esoteric library * license: public domain* quite a large user base for such a short time (proves the need of the community)* easy to use and extend (follows the componentization pattern)* declarative: you don't need to write executable code for most cases why not:* the code is (very) young. stable and all, but less than a month on the loose. * new features may still be added / existing ones may be changed in a non-backwards-compatible mannerso why am i saying this now, instead of waiting a few months for it to maturet? well, i wanted to get feedback. those of you who have seen/used the library, please tell me what you think:* is it suitable for a standard library?* what more features would you want?* any changes you think are necessary?i'm starting this now, in order to have a mature version in the (near) future. thanks,-tomer ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
