[issue29168] multiprocessing pickle error

2017-01-06 Thread Vinay Sajip
Vinay Sajip added the comment: I'm closing this on the basis that I don't believe there's a bug here, but you can of course re-open if you have evidence to the contrary. -- resolution: -> not a bug status: open -> closed ___ Python tracker

[issue29168] multiprocessing pickle error

2017-01-05 Thread Davin Potts
Davin Potts added the comment: Simon: You do have a lock object inside the logging handler object. That is what prevents your pickling of the logging handler object. The background information Serhiy shared is focused on why pickling a _thread.RLock object is problematic and unsupportable.

[issue29168] multiprocessing pickle error

2017-01-05 Thread Vinay Sajip
Vinay Sajip added the comment: > Have a look at the sample.py program. It fails on Python 3.4.3 on my Linux system (just renamed to mp_sample.py): python3 mp_sample.py starting (without multiprocessing pool)... worker process 0 worker process 3 worker process 4 worker process 6 worker process

[issue29168] multiprocessing pickle error

2017-01-05 Thread Simon Schuler
Simon Schuler added the comment: I don't have any lock object. I just use the multiprocessing pool and a QueueHandler in order to be able to log from all processes. -- ___ Python tracker __

[issue29168] multiprocessing pickle error

2017-01-05 Thread Serhiy Storchaka
Serhiy Storchaka added the comment: _thread.RLock objects were wrongly pickleable before 3.6. >>> import pickle, pickletools, _thread >>> lock = _thread.RLock() >>> lock.acquire() True >>> lock >>> pickle.loads(pickle.dumps(lock)) >>> pickletools.dis(pickletools.optimize(pickle.dumps(lock)))

[issue29168] multiprocessing pickle error

2017-01-05 Thread Simon Schuler
Simon Schuler added the comment: I want to handle the logging of the main and all my started processes. They should all log to the same Queue. Have a look at the sample.py program. In addition there is a inconsistency in using a multiprocessing pool or just the process class directly. The sam

[issue29168] multiprocessing pickle error

2017-01-05 Thread Vinay Sajip
Vinay Sajip added the comment: It may be that _thread.RLock objects were pickleable before 3.6 but aren't now, and if so, then this is unrelated to logging except by coincidence. Why do you want to pickle the *handler*? It's not especially designed for sending over the wire, and that's not sup

[issue29168] multiprocessing pickle error

2017-01-05 Thread Davin Potts
Changes by Davin Potts : -- nosy: +vinay.sajip ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.pyt

[issue29168] multiprocessing pickle error

2017-01-05 Thread Davin Potts
Davin Potts added the comment: In your first example (not sample.py), if I make the following change: #queue = multiprocessing.Manager().Queue(-1) queue = 42 I am still able to reproduce the exception you observe. This suggests the issue has nothing to do with the use of multiprocessing but

[issue29168] multiprocessing pickle error

2017-01-05 Thread Simon Schuler
Simon Schuler added the comment: Attached is a sample program to illustrate the problem. When I use a multiprocessing pool the exception is raised. -- Added file: http://bugs.python.org/file46160/sample.py ___ Python tracker

[issue29168] multiprocessing pickle error

2017-01-05 Thread Simon Schuler
New submission from Simon Schuler: Hello, the following code doesn't work any longer in the new Python version 3.6. import sys import os import subprocess from multiprocessing import Pool, Value, Queue import multiprocessing import logging import logging.handlers import pickle queue = multipro