Re: multithreading

[Bryan]

Kiuhnm wrote:
> My question is this: can I use 'threading' without interfering with the
> program which will import my module?

Yes. The things to avoid are described at the bottom of:
http://docs.python.org/library/threading.html

On platforms without threads, 'import threading' will fail. There's a
standard library module dummy_threading which offers fake versions of
the facilities in threading. It suggests:

try:
import threading as _threading
except ImportError:
import dummy_threading as _threading


--Bryan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python Gotcha's?

[Chris Angelico]

On Sun, Apr 8, 2012 at 2:19 PM, Chris Angelico  wrote:
> Agreed. Putting an expression first feels weird; in every high level
> language I know of, the word "if" is followed by the condition, and
> then by what to do if true, and then what to do if false - not true,
> then condition, then false.

Clarification: I'm talking primarily about statement-if here. Not many
languages have an expression-if that isn't derived from either LISP or
C; both of those still have the three parts in the same order, so it
comes to the same thing. Python switches them around compared to that.

ChrisA
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python Gotcha's?

[Chris Angelico]

On Sun, Apr 8, 2012 at 2:01 PM, John Nagle  wrote:
> 4.  The syntax for expression-IF is just weird.

Agreed. Putting an expression first feels weird; in every high level
language I know of, the word "if" is followed by the condition, and
then by what to do if true, and then what to do if false - not true,
then condition, then false.

> 6.  Multiple inheritance is a mess.  Especially "super".

Can you name any language in which multiple inheritance is NOT a mess?

Okay, so I'm a bit cynical. But MI is its own problem, and I think the
Python 3 implementation is about as good as it's worth hoping for.

ChrisA
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python Gotcha's?

[John Nagle]

On 4/4/2012 3:34 PM, Miki Tebeka wrote:

Greetings,

I'm going to give a "Python Gotcha's" talk at work.
If you have an interesting/common "Gotcha" (warts/dark corners ...) please 
share.

(Note that I want over http://wiki.python.org/moin/PythonWarts already).

Thanks,
--
Miki


A few Python "gotchas":

1.  Nobody is really in charge of third party packages.  In the
Perl world, there's a central repository, CPAN, and quality
control.  Python's "pypi" is just a collection of links.  Many
major packages are maintained by one person, and if they lose
interest, the package dies.

2.  C extensions are closely tied to the exact version of CPython
you're using, and finding a properly built version may be difficult.

3.  "eggs".  The "distutils" system has certain assumptions built into
it about where things go, and tends to fail in obscure ways.  There's
no uniform way to distribute a package.

4.  The syntax for expression-IF is just weird.

5.  "+" as concatenation.  This leads to strange numerical
semantics, such as (1,2) + (3,4) is (1,2,3,4).  But, for
"numarray" arrays, "+" does addition.  What does a mixed
mode expression of a numarray and a tuple do?  Guess.

5.  It's really hard to tell what's messing with the
attributes of a class, since anything can store into
anything.  This creates debugging problems.

6.  Multiple inheritance is a mess.  Especially "super".

7.  Using attributes as dictionaries can backfire.  The
syntax of attributes is limited.  So turning XML or HTML
structures into Python objects creates problems.

8.  Opening a URL can result in an unexpected prompt on
standard input if the URL has authentication.  This can
stall servers.

9.  Some libraries aren't thread-safe.  Guess which ones.

10. Python 3 isn't upward compatible with Python 2.

John Nagle


--
http://mail.python.org/mailman/listinfo/python-list

Re: Python Gotcha's?

[Chris Angelico]

On Sun, Apr 8, 2012 at 1:47 PM, Tim Roberts  wrote:
> No, but there certainly is a justification for expecting JAVASCRIPT Object
> Notation (which is, after all, what JSON stands for) to follow Javascript's
> syntax rules.  And Javascript happens to follow the same quoting rules as
> Python.
>
> Now, I fully understand that it is the way it is.  I'm merely pointing out
> that his was not an unreasonable expectation.

I agree, it would make sense. But that's only because of the name; if
it had been called "Just Simple Object Notation" or something
stupider, then nobody would expect it to be the same as anything. And
these days, nobody's confused by the fact that Java and Javascript are
completely different beasts.

ChrisA
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python Gotcha's?

[Tim Roberts]

Roy Smith  wrote:
>
>There's absolutely no reason why JSON should follow Python syntax rules. 

No, but there certainly is a justification for expecting JAVASCRIPT Object
Notation (which is, after all, what JSON stands for) to follow Javascript's
syntax rules.  And Javascript happens to follow the same quoting rules as
Python.

Now, I fully understand that it is the way it is.  I'm merely pointing out
that his was not an unreasonable expectation.
-- 
Tim Roberts, t...@probo.com
Providenza & Boekelheide, Inc.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: PEP 274

[Alec Taylor]

I have it in 2.7.3

On Sun, Apr 8, 2012 at 2:35 AM, Terry Reedy  wrote:
> On 4/7/2012 7:20 AM, Rodrick Brown wrote:
>>
>> This proposal was suggested in 2001 and is only now being
>> implemented. Why the extended delay?
>
>
> It was implemented in revised form 3 years ago in 3.0.
>
> --
> Terry Jan Reedy
>
> --
> http://mail.python.org/mailman/listinfo/python-list
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Multiprocessing & Logging

[Vinay Sajip]

Thibaut  gmail.com> writes:

> This is exactly what I wanted, it seems perfect. However I still have a 
> question, from what I understood,
> I have to configure logging AFTER creating the process, to avoid 
> children process to inherits the logging config.
> 
> Unless there is a way to "clean" logging configuration in children 
> processes, so they only have one handler : the QueueHandler.
> 
> I looked at the logging code and it doesn't seems to have an easy way to 
> do this. The problem of configuring the logging
> after the process creation is that... I can't log during process 
> creation. But if it's too complicated, I will just do this.
> 

I've updated the 3.2 / 3.3 logging cookbook with an example of what I mean.
There is a gist of the example script at

https://gist.github.com/2331314/

and the cookbook example should show once the docs get built on docs.python.org.

Regards,

Vinay Sajip


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Pass a list of variables to a procedure

[Keith Burns]

Thank you, Chris!

Sent from my iPhone

On Apr 7, 2012, at 3:24 PM, Chris Rebert  wrote:

> On Sat, Apr 7, 2012 at 2:15 PM, KRB  wrote:
>> Hi there,
>>
>> I would like to be able to pass a list of variables to a procedure, and have 
>> the output assigned to them.
>
> You cannot pass a variable itself to a function; you can only pass a
> variable's value. Which is to say that Python doesn't use
> pass-by-reference.
> Without using black magic, a Python function cannot rebind variables
> in its caller's scope. Mutable values can be mutated however.
> Details: http://effbot.org/zone/call-by-object.htm
>
>> For instance:
>>
>> x=0
>> y=0
>> z=0
>>
>> vars =[x,y,z]
>> parameters=[1,2,3]
>>
>> for i in range(1,len(vars)):
>> *** somefunction that takes the parameter "1", does a computation and 
>> assigns the output to "x", and so on and so forth.
>>
>> Such that later in the program I can
>> print x,y,z
>>
>> I hope that makes sense, otherwise I have to do:
>> x=somefunction(1)
>> y=somefunction(2)
>> z=somefunction(3)
>> etc etc
>
> Just use sequence (un)packing:
>
> def somefunction(*parameters):
># one would normally use a list comprehension here;
># for simplicity, I'm not
>results = []
>for parameter in parameters:
>result = do_some_calculation(parameter)
>results.append(result)
>return results
>
> #…later...
> x, y, z = somefunction(1, 2, 3)
>
>
> Relevant docs:
> http://docs.python.org/tutorial/datastructures.html#tuples-and-sequences
> http://docs.python.org/tutorial/controlflow.html#tut-unpacking-arguments
>
> Cheers,
> Chris
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Why does this hang sometimes?

[Jason Friedman]

> I am just playing around with threading and subprocess and found that
> the following program will hang up and never terminate every now and
> again.
>
> import threading
> import subprocess
> import time
>
> def targ():
>    p = subprocess.Popen(["/bin/sleep", "2"])
>    while p.poll() is None:
>        time.sleep(1)
>
> t1 = threading.Thread(target=targ)
> t2 = threading.Thread(target=targ)
> t1.start()
> t2.start()
>
> t1.join()
> t2.join()
>
>
> I found this bug, and while it sounds similar it seems that it was
> closed during python 2.5 (I'm using 2.7.2):
> http://bugs.python.org/issue1404925

I can confirm hanging on my installation of 2.7.2.  I also ran this
code 100 times on 3.2.2 without experiencing a hang.  Is version 3.x a
possibility for you?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Pass a list of variables to a procedure

[Nobody]

On Sat, 07 Apr 2012 14:15:09 -0700, KRB wrote:

> I would like to be able to pass a list of variables to a procedure, and
> have the output assigned to them.

Use a dictionary or an object.

If the variables are globals (i.e. attributes of the current module), you
can pass the result of globals() into the function, or have the function
return a dictionary which the caller merges into globals().

There isn't no way to do anything similar for local variables. The parser
has to "see" the name being used a local variable in order for it to
actually exist as a local variable.

> otherwise I have to do:
> x=somefunction(1)
> y=somefunction(2)
> z=somefunction(3)

Or you could do:

x, y, z = [somefunction(i) for i in [1,2,3]]

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Pass a list of variables to a procedure

[Chris Rebert]

On Sat, Apr 7, 2012 at 2:15 PM, KRB  wrote:
> Hi there,
>
> I would like to be able to pass a list of variables to a procedure, and have 
> the output assigned to them.

You cannot pass a variable itself to a function; you can only pass a
variable's value. Which is to say that Python doesn't use
pass-by-reference.
Without using black magic, a Python function cannot rebind variables
in its caller's scope. Mutable values can be mutated however.
Details: http://effbot.org/zone/call-by-object.htm

> For instance:
>
> x=0
> y=0
> z=0
>
> vars =[x,y,z]
> parameters=[1,2,3]
>
> for i in range(1,len(vars)):
> *** somefunction that takes the parameter "1", does a computation and assigns 
> the output to "x", and so on and so forth.
>
> Such that later in the program I can
> print x,y,z
>
> I hope that makes sense, otherwise I have to do:
> x=somefunction(1)
> y=somefunction(2)
> z=somefunction(3)
> etc etc

Just use sequence (un)packing:

def somefunction(*parameters):
# one would normally use a list comprehension here;
# for simplicity, I'm not
results = []
for parameter in parameters:
result = do_some_calculation(parameter)
results.append(result)
return results

#…later...
x, y, z = somefunction(1, 2, 3)


Relevant docs:
http://docs.python.org/tutorial/datastructures.html#tuples-and-sequences
http://docs.python.org/tutorial/controlflow.html#tut-unpacking-arguments

Cheers,
Chris
-- 
http://mail.python.org/mailman/listinfo/python-list

Pass a list of variables to a procedure

[KRB]

Hi there,

I would like to be able to pass a list of variables to a procedure, and have 
the output assigned to them.

For instance:

x=0
y=0
z=0

vars =[x,y,z]
parameters=[1,2,3]

for i in range(1,len(vars)):
*** somefunction that takes the parameter "1", does a computation and assigns 
the output to "x", and so on and so forth.

Such that later in the program I can
print x,y,z 

I hope that makes sense, otherwise I have to do:
x=somefunction(1)
y=somefunction(2)
z=somefunction(3)
etc etc

Appreciate any help
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: multithreading

[Adam Skutt]

On Apr 7, 5:06 pm, Kiuhnm  wrote:
> On 4/7/2012 22:09, Bryan wrote:>> For instance, let's say I want to make this 
> code thread-safe:
>
> >> --->
> >> myDict = {}
>
> >> def f(name, val):
> >>       if name not in myDict:
> >>           myDict[name] = val
> >>       return myDict[name]
> >> <---
>
> > First, don't re-code Python's built-ins. The example is a job for
> > dict.setdefault().
>
> [...]
>
> That was just an example for the sake of the discussion.
> My question is this: can I use 'threading' without interfering with the
> program which will import my module?

'import threading' ought to work everywhere, but that's not enough to
tell you whether whatever you're trying to do will actually work.
However, you shouldn't need to do it unless your application is meant
to /only/ be used in applications that have done 'import threading'
elsewhere.  Otherwise, you probably have a pretty serious design
issue.

Global state is bad.  TLS state is little better, even if it's common
in a lot of python modules.  Non-thread-safe object instances is
usually fine.  Object construction needs to be thread-safe, but that's
also the default behavior.  You need not worry about it unless you're
doing very unusual things.

Plainly, most of the time you shouldn't need to do anything to support
multiples threads beyond avoiding global state.  In fact, you should
stop and give some serious thought to your design if you need to do
anything else.

Adam
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: multithreading

[Kiuhnm]

On 4/7/2012 22:09, Bryan wrote:

For instance, let's say I want to make this code thread-safe:

--->
myDict = {}

def f(name, val):
  if name not in myDict:
  myDict[name] = val
  return myDict[name]
<---


First, don't re-code Python's built-ins. The example is a job for
dict.setdefault().

[...]

That was just an example for the sake of the discussion.
My question is this: can I use 'threading' without interfering with the 
program which will import my module?


Kiuhnm
--
http://mail.python.org/mailman/listinfo/python-list

Re: Distribute app without source?

[Bill Felton]

On Apr 7, 2012, at 1:22 PM, Terry Reedy wrote:

> On 4/7/2012 9:07 AM, Bill Felton wrote:
>> Thanks in advance for any insights!
>> 
>> My partner and I have developed an application primarily
> > intended for internal use within our company.  However, we face the
> > need  to expose the app to certain non-employees.
>> We would like to do so without exposing our source code.
> 
> To really do that, make it a web service, so the code only lives on your 
> server. That also takes care of

That's not really an alternative for us, for a variety of reasons.  It's not 
clear at the moment if it would
ever be an alternative, given our app and its structure and intended use.

As recommended elsewhere in this thread, cx_freeze seems likely to do us some 
good.
Initial testing has been fairly positive, although there are still some issues 
we're wrestling with.
If those continue, I'll post questions to the cx_freeze list.

Thanks everyone.

regards,
Bill

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: multithreading

[Bryan]

Kiuhnm wrote:
> I'm about to write my first module and I don't know how I should handle
> multithreading/-processing.
> I'm not doing multi-threading inside my module. I'm just trying to make
> it thread-safe so that users *can* do multi-threading.

There are a couple conventions to follow. Trying to anticipate the
threading needs of users and then locking everything for the worst
case is a bad idea.

So what are the conventions? Unless documented otherwise, classes
don't guarantee that each instance can be used by more then one
thread. Most of the classes in Python's standard library are not one-
instance-multiple-threads safe. An important documented exception is
queue.queue.

Classes should be safe for instance-per-thread multi-threading, unless
documented otherwise. Likewise, functions should be thread safe under
the assumption that their arguments are not shared between threads,
which brings us to your example:

> For instance, let's say I want to make this code thread-safe:
>
> --->
> myDict = {}
>
> def f(name, val):
>      if name not in myDict:
>          myDict[name] = val
>      return myDict[name]
> <---

First, don't re-code Python's built-ins. The example is a job for
dict.setdefault(). Language built-ins are already thread-safe (at
least in CPython), though not meant as thread synchronization
primitives.

Second, the example suggests no obvious reason for the single global
variabel. It could be a class for which users can make any number of
instances.

Third, there are cases where you want a single global. Most of the
time I'd recommend warning users about threading assumptions.

-Bryan
-- 
http://mail.python.org/mailman/listinfo/python-list

ordering with duck typing in 3.1

[Jon Clements]

Any reason you can't derive from int instead of object? You may also want to 
check out functions.total_ordering on 2.7+
-- 
http://mail.python.org/mailman/listinfo/python-list

documentation for asyncmongo?

[Laszlo Nagy]

Is there any kind of API documentation for asyncmongo? On GITHub they 
say "asyncmongo syntax strives to be similar to pymongo 
.".


However, many basic things do not work or they are not similar.

http://api.mongodb.org/python/2.1.1/tutorial.html

Example from pymongo:


 db.collection_names()

[u'posts', u'system.indexes']

The same in asyncmongo:

TypeError: 'Cursor' object is not callable

Even the connection is different: pymongo.Connect versus 
asyncmongo.Client. It has a "pool_id" parameter and what the hack is 
that? Is there no documentation for asyncmongo at all?


Thanks,

   Laszlo

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Distribute app without source?

[Terry Reedy]

On 4/7/2012 9:07 AM, Bill Felton wrote:

Thanks in advance for any insights!

My partner and I have developed an application primarily

> intended for internal use within our company.  However, we face the
> need  to expose the app to certain non-employees.

We would like to do so without exposing our source code.


To really do that, make it a web service, so the code only lives on your 
server. That also takes care of



Our targets include users of Windows and Mac OS, but not UNIX.


You could also distribute .pyc files compiled from obfuscated .py files.


We are using Python 3.2 and tkinter.  It appears, and limited testing
bears out, that py2app, and presumably py2exe, are not options given
lack of 3.x support.  PyInstaller does not support the 64-bit version
we are using.


Any such thing will include the contents of .pyc files somehow, even if 
harder to get at.



Does it make sense for us to try to use pyInstaller with a 32-bit
install of Python 3.2?


If you app runs within 2 GB, I would think yes.

--
Terry Jan Reedy

--
http://mail.python.org/mailman/listinfo/python-list

Re: Python Script Works Locally But Not Remotely with SSH

[Andrew Berg]

On 4/7/2012 11:59 AM, goldtech wrote:
 I thought if I SSH
> even from a Linux to a Windows machine whatever I say on the SSH
> client command line would be the same as me doing a command on the
> "DOS" command-line in Windows. I incorrectly thought SSH is just a
> tunnel for text...
It gives you whatever shell the SSH server serves (could be cmd, could a
Windows build of bash, could be whatever). I think the other posters
think you are trying to view Firefox on the Linux client's desktop. If
that is indeed the case, you will need an X server on the XP machine (I
think; it might not be that simple since it's a Windows process).
AFAICT, you want to open Firefox and have it show on the XP machine's
desktop. If that's the case, see Jerry Hill's post and configure Windows
to either run the SSH server as the logged in user or to allow it to
interact with the logged in user's desktop.

-- 
CPython 3.2.2 | Windows NT 6.1.7601.17640
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python Script Works Locally But Not Remotely with SSH

[goldtech]

Thank you for the help.

I guess I didn't understand what's really going on. I thought if I SSH
even from a Linux to a Windows machine whatever I say on the SSH
client command line would be the same as me doing a command on the
"DOS" command-line in Windows. I incorrectly thought SSH is just a
tunnel for text...

But that's not what's going on, there's  a lot more to it. Thanks, I
will get into it more to understand this.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: PEP 274

[Terry Reedy]

On 4/7/2012 7:20 AM, Rodrick Brown wrote:

This proposal was suggested in 2001 and is only now being
implemented. Why the extended delay?


It was implemented in revised form 3 years ago in 3.0.

--
Terry Jan Reedy

--
http://mail.python.org/mailman/listinfo/python-list

Re: 'string_escape' in python 3

[Ian Kelly]

On Sat, Apr 7, 2012 at 8:30 AM, Nicholas Cole  wrote:
> On Sat, Apr 7, 2012 at 12:10 AM, Ian Kelly  wrote:
> import codecs
> codecs.getdecoder('unicode_escape')(s)[0]
>> 'Hello: this is a test'
>>
>> Cheers,
>> Ian
>
> Thanks, Ian.  I had assumed that if a unicode string didn't have a
> .decode method, then I couldn't use a decoder on it, so it hadn't
> occurred to me to try that

Just a warning, I'm not really sure whether this behavior is intended
or not.  I just tried it and found that it seems to work.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Multiprocessing & Logging

[Vinay Sajip]

Thibaut  gmail.com> writes:

> This is exactly what I wanted, it seems perfect. However I still have a 
> question, from what I understood,
> I have to configure logging AFTER creating the process, to avoid 
> children process to inherits the logging config.
> 
> Unless there is a way to "clean" logging configuration in children 
> processes, so they only have one handler : the QueueHandler.
> 
> I looked at the logging code and it doesn't seems to have an easy way to 
> do this. The problem of configuring the logging
> after the process creation is that... I can't log during process 
> creation. But if it's too complicated, I will just do this.

You may be able to have a "clean" configuration: for example, dictConfig()
allows the configuration dictionary to specify whether existing loggers are
disabled. So the details depend on the details of your desired configuration.

One more point: I suggested that you subclass QueueListener, but you don't
actually need to do this. For example, you can do something like:

class DelegatingHandler(object):
def handle(self, record):
logger = logging.getLogger(record.name)
logger.handle(record)

And then instantiate the QueueListener with an instance of DelegatingHandler.
QueueListener doesn't need actual logging handlers, just something with a handle
method which takes a record.

Regards,

Vinay Sajip



-- 
http://mail.python.org/mailman/listinfo/python-list

Let's have hex

[cotpi]

A hexual game: http://cotpi.com/letushavehex/1/
--
http://mail.python.org/mailman/listinfo/python-list

Re: Multiprocessing & Logging

[Thibaut]

Le 07/04/2012 16:47, Vinay Sajip a écrit :

Thibaut  gmail.com>  writes:


Ok, I understand what happenned. In fact, configuring the logging before
forking works fine. Subprocess inherits the configuration, as I thought.

The problem was that I didn't passed any handler to the QueueListener
constructor. The when the listener recieved an message, it wasn't handled.

I'm not sure how the logging module works, but what handlers should I
pass the QueueListener constructor ? I mean, maybe that I would like
some messages (depending of the logger) to be logged to a file, while
some others message would just be printed to stdout.

This doesn't seem to be doable with a QueueListener. Maybe I should
implement my own system, and pass a little more informations with the
record sent in the queue : the logger name for example.

Then, in the main process I would do a logging.getLogger(loggername) and
log the record using this logger (by the way it was configured).

What do you think ?


You probably need different logging configurations in different processes. In
your multiprocessing application, nominate one of the processes as a logging
listener. It should initialize a QueueListener subclass which you write. All
other processes should just configure a QueueHandler, which uses the same queue
as the QueueListener.

All the processes with QueueHandlers just send their records to the queue. The
process with the QueueListener picks these up and handles them by calling the
QueueListener's handle() method.

The default implementation of QueueListener.handle() is:

 def handle(self, record):
 record = self.prepare(record)
 for handler in self.handlers:
 handler.handle(record)

where self.handlers is just the handlers you passed to the QueueListener
constructor. However, if you want a very flexible configuration where different
loggers have different handlers, this is easy to arrange. Just configure logging
in the listener process however you want, and then, in your QueueListener
subclass, do something like this:

class MyQueueListener(logging.handlers.QueueListener):
 def handle(self, record):
 record = self.prepare(record)
 logger = logging.getLogger(record.name)
 logger.handle(record)

This will pass the events to whatever handlers are configured for a particular
logger.

I will try to update the Cookbook in the logging docs with this approach, and a
working script.

Background information is available here: [1][2]

Regards,

Vinay Sajip

[1]
http://plumberjack.blogspot.co.uk/2010/09/using-logging-with-multiprocessing.html
[2]
http://plumberjack.blogspot.co.uk/2010/09/improved-queuehandler-queuelistener.html



This is exactly what I wanted, it seems perfect. However I still have a 
question, from what I understood,
I have to configure logging AFTER creating the process, to avoid 
children process to inherits the logging config.


Unless there is a way to "clean" logging configuration in children 
processes, so they only have one handler : the QueueHandler.


I looked at the logging code and it doesn't seems to have an easy way to 
do this. The problem of configuring the logging
after the process creation is that... I can't log during process 
creation. But if it's too complicated, I will just do this.


Thanks again for your help Vinay,
--
http://mail.python.org/mailman/listinfo/python-list

Re: Multiprocessing & Logging

[Vinay Sajip]

Thibaut  gmail.com> writes:

> Ok, I understand what happenned. In fact, configuring the logging before 
> forking works fine. Subprocess inherits the configuration, as I thought.
> 
> The problem was that I didn't passed any handler to the QueueListener 
> constructor. The when the listener recieved an message, it wasn't handled.
> 
> I'm not sure how the logging module works, but what handlers should I 
> pass the QueueListener constructor ? I mean, maybe that I would like 
> some messages (depending of the logger) to be logged to a file, while 
> some others message would just be printed to stdout.
> 
> This doesn't seem to be doable with a QueueListener. Maybe I should 
> implement my own system, and pass a little more informations with the 
> record sent in the queue : the logger name for example.
> 
> Then, in the main process I would do a logging.getLogger(loggername) and 
> log the record using this logger (by the way it was configured).
> 
> What do you think ?
> 

You probably need different logging configurations in different processes. In
your multiprocessing application, nominate one of the processes as a logging
listener. It should initialize a QueueListener subclass which you write. All
other processes should just configure a QueueHandler, which uses the same queue
as the QueueListener.

All the processes with QueueHandlers just send their records to the queue. The
process with the QueueListener picks these up and handles them by calling the
QueueListener's handle() method.

The default implementation of QueueListener.handle() is:

def handle(self, record):
record = self.prepare(record)
for handler in self.handlers:
handler.handle(record)

where self.handlers is just the handlers you passed to the QueueListener
constructor. However, if you want a very flexible configuration where different
loggers have different handlers, this is easy to arrange. Just configure logging
in the listener process however you want, and then, in your QueueListener
subclass, do something like this:

class MyQueueListener(logging.handlers.QueueListener):
def handle(self, record):
record = self.prepare(record)
logger = logging.getLogger(record.name)
logger.handle(record)

This will pass the events to whatever handlers are configured for a particular
logger.

I will try to update the Cookbook in the logging docs with this approach, and a
working script.

Background information is available here: [1][2]

Regards,

Vinay Sajip

[1]
http://plumberjack.blogspot.co.uk/2010/09/using-logging-with-multiprocessing.html
[2]
http://plumberjack.blogspot.co.uk/2010/09/improved-queuehandler-queuelistener.html

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: ordering with duck typing in 3.1

[mwilson]

Thomas Rachel wrote:

> Am 07.04.2012 14:23 schrieb andrew cooke:
> 
>> class IntVar(object):
>>
>>  def __init__(self, value=None):
>>  if value is not None: value = int(value)
>>  self.value = value
>>
>>  def setter(self):
>>  def wrapper(stream_in, thunk):
>>  self.value = thunk()
>>  return self.value
>>  return wrapper
>>
>>  def __int__(self):
>>  return self.value
>>
>>  def __lt__(self, other):
>>  return self.value<  other
>>
>>  def __eq__(self, other):
>>  return self.value == other
>>
>>  def __hash__(self):
>>  return hash(self.value)
> 
>> so what am i missing?
> 
> If I don't confuse things, I think you are missing a __gt__() in your
> IntVar() class.
> 
> This is because first, a '2 < three' is tried with 2.__lt__(three). As
> this fails due to the used types, it is reversed: 'three > 2' is
> equivalent. As your three doesn't have a __gt__(), three.__gt__(2) fails
> as well.

Practically, yes.  Just that that's not what the documentation says.  Looks 
like Python no longer tries to cobble together missing relations based on 
the "usual" properties of ordering.

Mel.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: ordering with duck typing in 3.1

[John O'Hagan]

On Sat, 7 Apr 2012 05:23:25 -0700 (PDT)
andrew cooke  wrote:

> 
> hi,
> 
> please, what am i doing wrong here?  the docs say
> http://docs.python.org/release/3.1.3/library/stdtypes.html#comparisons "in
> general, __lt__() and __eq__() are sufficient, if you want the conventional
> meanings of the comparison operators" but i am seeing
> 
> >   assert 2 < three
> E   TypeError: unorderable types: int() < IntVar()
> 
> with this test:
> 
> 
> class IntVar(object):
> 
> def __init__(self, value=None):
> if value is not None: value = int(value)
> self.value = value
> 
> def setter(self):
> def wrapper(stream_in, thunk):
> self.value = thunk()
> return self.value
> return wrapper
> 
> def __int__(self):
> return self.value
> 
> def __lt__(self, other):
> return self.value < other
> 
> def __eq__(self, other):
> return self.value == other
> 
> def __hash__(self):
> return hash(self.value)
> 
> 
> class DynamicTest(TestCase):
> 
> def test_lt(self):
> three = IntVar(3)
> assert three < 4
> assert 2 < three
> assert 3 == three
> 
> so what am i missing?
> 

I think that quote from the docs is just to point out that you only need those
two (== and <) to derive any of the other comparisons; but not to imply that a
class that only defines those two will automatically possess the others.
However, you can do that, with functools.total_ordering.

Regards,

John

-- 
http://mail.python.org/mailman/listinfo/python-list

multithreading

[Kiuhnm]

I'm about to write my first module and I don't know how I should handle 
multithreading/-processing.
I'm not doing multi-threading inside my module. I'm just trying to make 
it thread-safe so that users *can* do multi-threading.


For instance, let's say I want to make this code thread-safe:

--->
myDict = {}

def f(name, val):
if name not in myDict:
myDict[name] = val
return myDict[name]
<---

I could use threading.Lock() but I don't know if that might interfere 
with some other modules imported by the user.
In some languages you can't mix multi-threading libraries. Is Python one 
of them?


Kiuhnm
--
http://mail.python.org/mailman/listinfo/python-list

Re: ordering with duck typing in 3.1

[Rob Williscroft]

andrew cooke wrote in
news:33019705.1873.1333801405463.JavaMail.geo-discussion-forums@ynmm9 in
gmane.comp.python.general: 

> 
> hi,
> 
> please, what am i doing wrong here?  the docs say
> http://docs.python.org/release/3.1.3/library/stdtypes.html#comparisons
> "in general, __lt__() and __eq__() are sufficient, if you want the
> conventional meanings of the comparison operators" but i am seeing 
> 
>>   assert 2 < three
> E   TypeError: unorderable types: int() < IntVar()
> 
> with this test:
> 
> 
> class IntVar(object):
> 

> def __lt__(self, other):
> return self.value < other
> 
 
> so what am i missing?
> 

The part of the docs you are relying on uses the wording "in general",
IOW, it is not saying that defining __eq__ and __lt__ will always be 
sufficient.

In this case the expression "2 < three" is calling int.__lt__, which
doesn't know how to comapre to an instance of your class so returns 
NotImplemented.

At this point if you had defined a __gt__ method the interpreter would then 
try and call that having first switched the arguments around.  But you 
didn't so a TypeError is raised.

I'm afraid I couldn't find anywhere in the docs where that behaviour is 
described, I suspect I only know it from lurking on usenet for a number of 
years.

The best description that I could find of the behaviour you are seeing is 
at:

http://docs.python.org/py3k/reference/expressions.html#not-in

There is a paragraph that contains:

"... the == and != operators always consider objects of different types to 
be unequal, while the <, >, >= and <= operators raise a TypeError when 
comparing objects of different types that do not implement these operators 
for the given pair of types. ..."

Perhapse the docs could be reworded to note that, to define a full set of 
comparisons between *different* types, you need to define a full set of 
special methods.

Some links I found along the way:

http://docs.python.org/release/3.1.3/library/constants.html?
highlight=__lt__#NotImplemented

http://code.activestate.com/recipes/576685/
http://docs.python.org/py3k/library/functools.html#functools.total_ordering

Rob.


-- 
http://mail.python.org/mailman/listinfo/python-list

how i loved lisp cons and UML and Agile and Design Patterns and Pythonic and KISS and YMMV and stopped worrying

[Xah Lee]

OMG, how i loved lisp cons and macros and UML and Agile eXtreme
Programing and Design Patterns and Anti-Patterns and Pythonic and KISS
and YMMV and stopped worrying.

〈World Multiconference on Systemics, Cybernetics and Informatics???〉
http://xahlee.org/comp/WMSCI.html

highly advanced plain text format follows, as a amenity for tech
geekers.

---
World Multiconference on Systemics, Cybernetics and Informatics ???

Xah Lee, 2010-04-04

Starting in 2004, i regularly receive email asking me to participate a
conference, called “World Multiconference on Systemics, Cybernetics
and Informatics” (WMSCI). Here's one of such email i got today:


Dear Xah Lee:

As you know the Nobel Laureate Herbert Simon affirmed that design is
an essential ingredient of the Artificial Sciences Ranulph Glanville,
president of the American Society for Cybernetics and expert in design
theory, affirms that “Research is a variety of design. So do research
as design. Design is key to research. Research has to be designed.” An
increasing number of authors are stressing the relationships between
Design and Research. Design is a mean for Research, and Research is a
mean for Design. Design and research are related via cybernetic loops
in the context of means-ends logic. Consequently, we invite you to
submit a paper/abstract and/ot to organize an invited session in the
International Symposium on Design and Research in the Artificial and
the Natural Sciences: DRANS 2010 (http://www.sysconfer.org/drans)
which is being organized in the context of The 14th World Multi-
Conference on Systemics, Cybernetics and Informatics: WMSCI 2010
(http://www.sysconfer.org/wmsci), 2010 in Orlando, Florida, USA.

…

Here's the first email i got from them from my mail archive:

 From: sci2...@iiis.org
Subject: Inviting you to participate in SCI 2005
Date: October 20, 2004 1:39:48 PM PDT
To: x...@xahlee.org

 Dear Dr. Xah Lee:

On behalf of the SCI 2005 Organizing Committee, I would like to invite
you to participate in the 9th World Multi-Conference on Systemics,
Cybernetics and Informatics (http://www.iiisci.org/sci2005), which
will take place in Orlando, Florida, USA, from July 10-13, 2005.

Full text wmsci.txt.

I do not know this organization. I don't know how they got my email or
how they know that i'm involved in the computer science community.
(surely from trawling email addresses in science forums) Though, after
getting a few of their emails, one clearly gets a sense that it is a
scam, soliciting innocent idiotic academicians (many PhDs are
idiots.).

Here's what Wikipedia has to say about them: World Multiconference on
Systemics, Cybernetics and Informatics. Here's a juicy quote:


WMSCI attracted publicity of a less favorable sort in 2005 when three
graduate students at MIT succeeded in getting a paper accepted as a
“non-reviewed paper” to the conference that had been randomly
generated by a computer program called SCIgen.[8] Documents generated
by this software have been used to submit papers to other similar
conferences. Compare to the Sokal affair.

WMSCI has been accused of using spam to advertise its conferences.[8]

Now and then, whenever i got their email, the curiosity in me do
lookup the several terms they used in the email, partly to check the
validity. For example, in this one, it mentions Herbert Simon. Another
one i recall i got recently mentioned Science 2.0. Both of the terms i
haven't heard of before.

One'd think that it is easy to tell scam from real science, but with
today's science proliferation, it's actually not that easy. Even if
you are a academic, it's rather common that many new science terms you
never heard of, because there are tremendous growth of new disciplines
or cross disciplines, along with new jargons. Cross-discipline is
rather common and natural, unlike in the past where science is more or
less clearly delineated hierarchy like Physics, Math, Chemistry,
Biology, etc and their sub-branches. However, many of today's new
areas is a bit questionable, sometimes a deliberate money-making
scheme, which i suppose is the case for WMSCI. Many of these, use
terms like “post-modern”, “science 2.0” to excuse themselves from the
rather strict judgment of classic science. Many of these terms such as
“systemics”, “cybernetics”, “infomatics” are vague. Depending on the
context, it could be a valid emerging science discipline, but it could
also be pure new-age hogwash. And sometimes, nobody really knows
today. Fledgling scientific fields may started off as pseudo-science
but later became well accepted with more solid theories. (e.g.
evolutionary psychology)

In the past 2 decades, there are quite a few cases where peer reviewed
papers published in respected journals are exposed as highly
questionable or deliberate hoax, arose massive debate on the peer
review system. The peer-review system itself can't hold all the blame,
but part of it has to do with the incredible growth of sciences and
li

Re: 'string_escape' in python 3

[Nicholas Cole]

On Sat, Apr 7, 2012 at 12:10 AM, Ian Kelly  wrote:
 import codecs
 codecs.getdecoder('unicode_escape')(s)[0]
> 'Hello: this is a test'
>
> Cheers,
> Ian

Thanks, Ian.  I had assumed that if a unicode string didn't have a
.decode method, then I couldn't use a decoder on it, so it hadn't
occurred to me to try that

N.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Distribute app without source?

[Andrew Berg]

On 4/7/2012 8:07 AM, Bill Felton wrote:
> We are using Python 3.2 and tkinter.  It appears, and limited testing bears 
> out, that py2app, and presumably py2exe, are not options given lack of 3.x 
> support.
cx_Freeze supports Python 3.2. It works fine for my purposes, but I have
not done any serious work with it. I don't know about tkinter, but I was
able to freeze a simple Qt application.

-- 
CPython 3.2.2 | Windows NT 6.1.7601.17640
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: ordering with duck typing in 3.1

[Thomas Rachel]

Am 07.04.2012 14:23 schrieb andrew cooke:


class IntVar(object):

 def __init__(self, value=None):
 if value is not None: value = int(value)
 self.value = value

 def setter(self):
 def wrapper(stream_in, thunk):
 self.value = thunk()
 return self.value
 return wrapper

 def __int__(self):
 return self.value

 def __lt__(self, other):
 return self.value<  other

 def __eq__(self, other):
 return self.value == other

 def __hash__(self):
 return hash(self.value)



so what am i missing?


If I don't confuse things, I think you are missing a __gt__() in your 
IntVar() class.


This is because first, a '2 < three' is tried with 2.__lt__(three). As 
this fails due to the used types, it is reversed: 'three > 2' is 
equivalent. As your three doesn't have a __gt__(), three.__gt__(2) fails 
as well.



Thomas
--
http://mail.python.org/mailman/listinfo/python-list

Distribute app without source?

[Bill Felton]

Thanks in advance for any insights!

My partner and I have developed an application primarily intended for internal 
use within our company.  However, we face the need  to expose the app to 
certain non-employees.
We would like to do so without exposing our source code.
Our targets include users of Windows and Mac OS, but not UNIX.

We are using Python 3.2 and tkinter.  It appears, and limited testing bears 
out, that py2app, and presumably py2exe, are not options given lack of 3.x 
support.  PyInstaller does not support the 64-bit version we are using.  

Does it make sense for us to try to use pyInstaller with a 32-bit install of 
Python 3.2?  Are there other options?  

Any assistance or input will be welcome.

regards,
Bill

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Multiprocessing & Logging

[Thibaut]

Le 07/04/2012 11:22, Thibaut DIRLIK a écrit :

Hi,

I'm currently writing a multiprocess applications with Python 3.2 and 
multiprocessing module. My subprocesses will use a QueueHandler to log 
messages (by sending them to the main process, which uses a 
QueueListener). However, if logging is already configured when I 
create the subprocesses, they will inherit the configuration, and 
opened file descriptors used for logging in the main process.


However, when I tried this with a basicConfig configuration, which 
prints to a file, the messages are only written once to the file. I 
don't understand why. Normally, the mainprocess contains a logging 
handler to log to the file. This handler will be copied into the child 
processes. Child processes will then have two handlers : a 
QueueHandler, and a FileHandler. They should write to the file handler 
and then send the message to the main process QueueListener, which 
should write the message AGAIN to the FileHandler.


But that's not the case. Any rea

How can I totally "unset" the logging configuration in child processes 
and only enable the QueueHandler (foor all configured loggers, not 
only the root one) ?


Thanks for your help,



Ok, I understand what happenned. In fact, configuring the logging before 
forking works fine. Subprocess inherits the configuration, as I thought.


The problem was that I didn't passed any handler to the QueueListener 
constructor. The when the listener recieved an message, it wasn't handled.


I'm not sure how the logging module works, but what handlers should I 
pass the QueueListener constructor ? I mean, maybe that I would like 
some messages (depending of the logger) to be logged to a file, while 
some others message would just be printed to stdout.


This doesn't seem to be doable with a QueueListener. Maybe I should 
implement my own system, and pass a little more informations with the 
record sent in the queue : the logger name for example.


Then, in the main process I would do a logging.getLogger(loggername) and 
log the record using this logger (by the way it was configured).


What do you think ?


--
http://mail.python.org/mailman/listinfo/python-list

ordering with duck typing in 3.1

[andrew cooke]

hi,

please, what am i doing wrong here?  the docs say 
http://docs.python.org/release/3.1.3/library/stdtypes.html#comparisons "in 
general, __lt__() and __eq__() are sufficient, if you want the conventional 
meanings of the comparison operators" but i am seeing

>   assert 2 < three
E   TypeError: unorderable types: int() < IntVar()

with this test:


class IntVar(object):

def __init__(self, value=None):
if value is not None: value = int(value)
self.value = value

def setter(self):
def wrapper(stream_in, thunk):
self.value = thunk()
return self.value
return wrapper

def __int__(self):
return self.value

def __lt__(self, other):
return self.value < other

def __eq__(self, other):
return self.value == other

def __hash__(self):
return hash(self.value)


class DynamicTest(TestCase):

def test_lt(self):
three = IntVar(3)
assert three < 4
assert 2 < three
assert 3 == three

so what am i missing?

thanks,
andrew
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: PEP 274

[Rodrick Brown]

This proposal was suggested in 2001 and is only now being implemented. Why the 
extended delay? 

Sent from my iPhone

On Apr 7, 2012, at 3:32 AM, Alec Taylor  wrote:

> Has been withdrawn... and implemented
> 
> http://www.python.org/dev/peps/pep-0274/
> -- 
> http://mail.python.org/mailman/listinfo/python-list
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: escaping/encoding/formatting in python

[Chris Angelico]

On Sat, Apr 7, 2012 at 3:36 PM, Nobody  wrote:
> The delimiter can be chosen either by analysing the string
> or by choosing something a string at random and relying upon a collision
> being statistically improbable.

The same techniques being available to MIME multi-part encoders, and
for the same reason. Nestable structures can be quite annoying to
parse.

ChrisA
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python Training

[Mohan kumar]

On Apr 7, 2:21 pm, Mohan kumar  wrote:
> Hi,
> We are an IT Training company located in Bangalore, Chennai and
> Coimbatore. We provide Python training with Placement Assistance. For
> more details, email to mo...@cgonsoft.com

We also provide Online Training.
-- 
http://mail.python.org/mailman/listinfo/python-list

Python Training

[Mohan kumar]

Hi,
We are an IT Training company located in Bangalore, Chennai and
Coimbatore. We provide Python training with Placement Assistance. For
more details, email to mo...@cgonsoft.com
-- 
http://mail.python.org/mailman/listinfo/python-list

Multiprocessing & Logging

[Thibaut DIRLIK]

Hi,

I'm currently writing a multiprocess applications with Python 3.2 and
multiprocessing module. My subprocesses will use a QueueHandler to log
messages (by sending them to the main process, which uses a QueueListener).
However, if logging is already configured when I create the subprocesses,
they will inherit the configuration, and opened file descriptors used for
logging in the main process.

However, when I tried this with a basicConfig configuration, which prints
to a file, the messages are only written once to the file. I don't
understand why. Normally, the mainprocess contains a logging handler to log
to the file. This handler will be copied into the child processes. Child
processes will then have two handlers : a QueueHandler, and a FileHandler.
They should write to the file handler and then send the message to the main
process QueueListener, which should write the message AGAIN to the
FileHandler.

But that's not the case. Any rea

How can I totally "unset" the logging configuration in child processes and
only enable the QueueHandler (foor all configured loggers, not only the
root one) ?

Thanks for your help,
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Learning new APIs/classes (beginner question)

[Martin Jones]

On Apr 7, 1:52 am, Steven D'Aprano  wrote:

> Sounds like this library is documented the same way most third party
> libraries are: as an afterthought, by somebody who is so familiar with
> the software that he cannot imagine why anyone might actually need
> documentation.
>
> I feel your pain.
>

Thanks Steven, I suspected this might be the case, but wasn't sure if
I was missing something obvious. Maybe I'll start on a different
project using better-documented or just the build-in libraries.

Many thanks,

Martin.
-- 
http://mail.python.org/mailman/listinfo/python-list