[issue28965] Multiprocessing spawn/forkserver fails to pass Queues
New submission from Sean Murphy:
Python fails to pass a Queue when calling Process with
multiprocessing.set_start_method set to "spawn" or "forkserver".
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python3.5/multiprocessing/spawn.py", line 106, in spawn_main
exitcode = _main(fd)
File "/usr/lib/python3.5/multiprocessing/spawn.py", line 116, in _main
self = pickle.load(from_parent)
File "/usr/lib/python3.5/multiprocessing/synchronize.py", line 111, in
__setstate__
self._semlock = _multiprocessing.SemLock._rebuild(*state)
FileNotFoundError: [Errno 2] No such file or directory
Here is a minimized example:
```
#!/usr/bin/env python3
import multiprocessing
def check_child(q):
print("Queue", q)
if __name__ == '__main__':
multiprocessing.set_start_method('spawn')
# multiprocessing.set_start_method('fork')
# multiprocessing.set_start_method('forkserver')
q = multiprocessing.Queue(-1)
print("q", q)
proc = multiprocessing.Process(target=check_child, args=(q,))
proc.start()
```
Also, this fails when the Queue is implicitly passed to the child.
```
class Blerg():
def __init__(self):
self.q = multiprocessing.Queue(-1)
def print_queue(self):
print("Queue", self.q)
if __name__ == '__main__':
multiprocessing.set_start_method('spawn')
blerg = Blerg()
blerg.print_queue()
proc = multiprocessing.Process(target=blerg.print_queue)
proc.start()
```
$ python3 --version
Python 3.5.2
Windows (which defaults to "spawn" style multiprocessing) does not seem to have
this issue (at least in 2.7.12).
--
components: Library (Lib)
messages: 283150
nosy: Sean Murphy
priority: normal
severity: normal
status: open
title: Multiprocessing spawn/forkserver fails to pass Queues
type: crash
versions: Python 3.5
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue28965>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
Python and wireshark.
All. Is there any way to use python with Wireshark/Tshark? I am not able to use the GUI due to my vision impairment. So I am thinking of using Wireshark libraries and Python to provide a text console environment. Tshark does give you command line capability. I am more seeking for the ability of changing things on the fly, rather then creating complex command line parameters. Thoughts? Sean -- https://mail.python.org/mailman/listinfo/python-list
Module depositary
Hi All. I am aware that active state python has a commercial module depositary which you can get modules from. Under PERL you had CPAN. Is there anything like this for Python? Sean -- https://mail.python.org/mailman/listinfo/python-list
Re: nested dictionaries and functions in data structures.
Thanks for that. I will have a play and see how I can apply your example.
On 07/01/2014, at 11:19 PM, Jean-Michel Pichavant jeanmic...@sequans.com
wrote:
- Original Message -
Thanks for that. It resolved the issue and it was so simple compared
to everything else I saw on the net.
Only outstanding thing I have to work out is how to execute functions
from a dictionary. I will continue searching on the net.
Sean
This may help you (untested code)
class Workers:
@staticmethod
def ospf(*args, **kwargs):
print args, kwargs
return 'Bar'
@staticmethod
def bhp(*args, **kwargs):
return 'Foo'
outputs = {'ospf' : {1 : {'param1' : 'foo', 'param2' : 'foo2'}}
for command, value in outputs.items():
for counter, kwargs in value:
func = getattr(Workers, command) # get the function
func(**kwargs) # the actual call
It would be possible to do it without the *args, **kwargs magic. You can
write ospf and bhp with the explicit parameters, however you must make sure
you pass the correct number of parameter in the call.
JM
-- IMPORTANT NOTICE:
The contents of this email and any attachments are confidential and may also
be privileged. If you are not the intended recipient, please notify the
sender immediately and do not disclose the contents to any other person, use
it for any purpose, or store or copy the information in any medium. Thank you.
--
https://mail.python.org/mailman/listinfo/python-list
nested dictionaries and functions in data structures.
Hello all.
I have some questions again. :-)
I wish to be able to place a function within a data structure. I would like to
use a dictionary because I could pass it a key and then the function could be
called. I couldn't find anything on the net to show me how to do this. More
then likely, not using the right search terms.
For example:
funct_call = { 'bhp' : 'bhp_counters (addict[key1][key2])', 'ospf' :
'ospf_counters (addict[key1][key2])'}
I am sure there is a way to do this.
The other issue is I cannot nest dictionaries. I have seen examples and when I
apply them to the code below. They do not provide the result I want. The
program does 3 actions.
1. Opens all the files in the directory. Each file begins with data_. The 6
character in the file name is the occurrence of the output. Ranging from 1 to
9. The8th character plus the remaining part of the file is the output of the
command. For example:
data_1_ospf.txt
The commands stored in this file are related to OSPF. When I build the nested
dictionary I want to have OSPF as the primary key. Nested under OSPF is the
number of times the command has been captured. The file content stored as an
array and is the value of the 2nd key. data structure could look like this:
outputs = { 'ospf' : { '1' : lines_of_file[], '2' : lines of_file[]} }
Below is the code I have used to try and achieve the above. I just don't get
the nested dictionary effect I am after. Again, I am trying to use standard
core which some of the examples I have seen appeared to use. I am aware of
collection module.
#! /usr/bin/env python
# Identifying if memory leaks are occurring.
# goal is to import output to Excel.
# created on 28 Dec 2013 By Sean Murphy
import os, sys
from os.path import exists
# main code begins
if len(sys.argv) = 2:
# storing file names into variable from command line.
filenames = sys.argv[1:]
else:
filenames = os.listdir(os.getcwd())
#print (Error, must provide at least one file name\n)
#quit()
outputs = {} # empty dictionary (hash)
capture = # key used for the capture version
command = # key for the command output
for filename in filenames:
if exists(filename):
fp = open(filename, r)
capture = filename[6]
command = filename[8:]
# nested dictionary. Command and then number of captures.
outputs = {command : { capture :[fp.readlines() } }
fp.close()
else:
print (error %s doesn't exists\n % filename)
quit()
print (%r\n % outputs.keys())
for key in sorted(outputs):
print (outputs[key].keys ())
Cheers
Sean
--
https://mail.python.org/mailman/listinfo/python-list
Re: nested dictionaries and functions in data structures.
Thanks for that. It resolved the issue and it was so simple compared to
everything else I saw on the net.
Only outstanding thing I have to work out is how to execute functions from a
dictionary. I will continue searching on the net.
Sean
On 07/01/2014, at 9:21 PM, Jean-Michel Pichavant jeanmic...@sequans.com wrote:
- Original Message -
Hello all.
I have some questions again. :-)
I wish to be able to place a function within a data structure. I
would like to use a dictionary because I could pass it a key and
then the function could be called. I couldn't find anything on the
net to show me how to do this. More then likely, not using the right
search terms.
For example:
funct_call = { 'bhp' : 'bhp_counters (addict[key1][key2])', 'ospf' :
'ospf_counters (addict[key1][key2])'}
I am sure there is a way to do this.
The other issue is I cannot nest dictionaries. I have seen examples
and when I apply them to the code below. They do not provide the
result I want. The program does 3 actions.
1. Opens all the files in the directory. Each file begins with
data_. The 6 character in the file name is the occurrence of the
output. Ranging from 1 to 9. The8th character plus the remaining
part of the file is the output of the command. For example:
data_1_ospf.txt
The commands stored in this file are related to OSPF. When I build
the nested dictionary I want to have OSPF as the primary key.
Nested under OSPF is the number of times the command has been
captured. The file content stored as an array and is the value of
the 2nd key. data structure could look like this:
outputs = { 'ospf' : { '1' : lines_of_file[], '2' : lines of_file[]}
}
Below is the code I have used to try and achieve the above. I just
don't get the nested dictionary effect I am after. Again, I am
trying to use standard core which some of the examples I have seen
appeared to use. I am aware of collection module.
#! /usr/bin/env python
# Identifying if memory leaks are occurring.
# goal is to import output to Excel.
# created on 28 Dec 2013 By Sean Murphy
import os, sys
from os.path import exists
# main code begins
if len(sys.argv) = 2:
# storing file names into variable from command line.
filenames = sys.argv[1:]
else:
filenames = os.listdir(os.getcwd())
#print (Error, must provide at least one file name\n)
#quit()
outputs = {} # empty dictionary (hash)
capture = # key used for the capture version
command = # key for the command output
for filename in filenames:
if exists(filename):
fp = open(filename, r)
capture = filename[6]
command = filename[8:]
# nested dictionary. Command and then number of captures.
outputs = {command : { capture :[fp.readlines() } }
fp.close()
else:
print (error %s doesn't exists\n % filename)
quit()
print (%r\n % outputs.keys())
for key in sorted(outputs):
print (outputs[key].keys ())
Cheers
Sean
outputs keeps track of the last loop only because you're assigning a new dict
on every loop. You need to update the current dict instead.
try to replace
outputs = {command : { capture :fp.readlines() } }
with (untested code)
if command not in outputs:
outputs[command] = {}
outputs[command][capture] = fp.readlines()
JM
-- IMPORTANT NOTICE:
The contents of this email and any attachments are confidential and may also
be privileged. If you are not the intended recipient, please notify the
sender immediately and do not disclose the contents to any other person, use
it for any purpose, or store or copy the information in any medium. Thank you.
--
https://mail.python.org/mailman/listinfo/python-list
Dos cursor and input management.
Hi all. I am after a module that manages keyboard input. I am aware of raw_input for python 2.x and input for 3.x. They don't quite achieve what I want. I want to except a single key without printing it to the screen and then the key would perform an action. Sudo code: print line of text wait for key press If key press equals delete line. Delete list element. else if key press equals edit display line for interactive edit. else move to next line The module must work under dos for now. Eventually Mac. Sean -- https://mail.python.org/mailman/listinfo/python-list
Re: Dos cursor and input management.
Dennis, Loose terminology. The Command terminal within windows which the app will be executed in. Not native DOS. On 06/01/2014, at 3:48 AM, Denis McMahon denismfmcma...@gmail.com wrote: On Sun, 05 Jan 2014 20:25:11 +1100, Sean Murphy wrote: The module must work under dos for now. Eventually Mac. Do you mean a windows command line terminal window, or some *nix shell? As far as I know, dos as an operating system hasn't been around since version 6.22 or thereabouts, although I believe ms windows provides a dos shell like interface on top of the windows os. I'm not aware that any version of python is supported on dos, but I may be wrong, there may be some 15 year old hardware running dos somewhere that also has a working python install. I associate dos with machines of the pre-pentium era, although I suspect that might not be quite accurate either. -- Denis McMahon, denismfmcma...@gmail.com -- https://mail.python.org/mailman/listinfo/python-list -- https://mail.python.org/mailman/listinfo/python-list
Re: Strange behaviour with a for loop.
Chris, Thanks for the tip on the function. I was not aware of that function, Grin. Creating the function as you mention makes a lot of sense. I am doing a lot of little bits and pieces focusing on things I need to eventually build a script that is going to compile data from a router and config it. I have hundreds of other questions, if I don't find answers on the net before hand. Sean On 04/01/2014, at 6:52 PM, Cameron Simpson c...@zip.com.au wrote: On 04Jan2014 16:54, Sean Murphy mhysnm1...@gmail.com wrote: Thanks everyone. Mark thanks for the correction on the ':'. Since I didn't cut and copy, rather typed it out. Errors crept in. :-) another question in relation to slicing strings. If you want to get a single character, just using the index position will get it. If I use the following, shouldn't it also work? when I use Python 3.3, it didn't provide anything. a = test.txt print a[3] result is: 't As expected, yes? print a[3:1] Nothing is printed. print a[3:2] Nothing is printed. These are not requests for 1 and 2 character strings. They are requests for the character in the span from, respectively, 3 to 1 and from 3 to 2. Important: counting FORWARDS. So: zero length strings. print a[3:-1] t.tx is printed. Why doesn't the positive number of characters to be splice return anything while the negative value does? -1 is shorthand for len(a)-1 It is often convenient to refer to a position from the end of the array instead of the start. So this means: [3:7], so positions 3,4,5,6. sorry about these basic questions. I do like the splice feature within Python. Also what is the best method of testing for a blank string? Well, and empty string: a == '' or len(a) == 0. And, because an if tests the nonzeroness of a single argument and an empty string has length zero, you can also go: if a: print long string, a else: print empty string OTOH, if you mean a blank string to mean containing only whitespace, you can use the string method isspace, which tests that all characters are whitespace and that the string is not empty. The doco for isspace() actually says: Return true if there are only whitespace characters in the string and there is at least one character, false otherwise. So you might write: if not a or a.isspace(): print blank string:, repr(a) Really you'd want to put that it a (trivial) function: def isblank(s): ''' Test that the string `s` is entirely blank. ''' return not s or s.isspace() That way you can write isblank() all through your program and control the precise meaning by modifying the function. Cheers, -- The perl5 internals are a complete mess. It's like Jenga - to get the perl5 tower taller and do something new you select a block somewhere in the middle, with trepidation pull it out slowly, and then carefully balance it somewhere new, hoping the whole edifice won't collapse as a result. - Nicholas Clark, restating an insight of Simon Cozens -- https://mail.python.org/mailman/listinfo/python-list
Debugging on the Mac question.
Team, I am a Vision Impaired programmer on the Mac and Window platforms. I have started to learn Python. The biggest road block I have is the ability of debugging my simple scripts. The IDLE program does not work with the screen readers I use on the Mac or Windows. A screen reader is a program that grabs the text on the screen and converts it into speech output, at least this is the 5 feet explanation. I cannot see the screen at all. I have looked at eclipse and it doesn't work with Voice-Over (the screen reader on the Mac). I have java issues on my windows machine preventing me running this app. If I use $python -d script.py the debugger doesn't seem to trigger on the mac. So how I can perform a debug on a script so I can step through it, set up break points, watch variables, etc. It is really annoying me, since under Perl I just added the -d switch and had a full debugger that worked at the console level. Sean -- https://mail.python.org/mailman/listinfo/python-list
Re: Debugging on the Mac question.
PETER, thanks Peter, I have already found the PDB module and have had a play with it. It will do for now. On 03/01/2014, at 8:08 PM, Paul Rudin paul.nos...@rudin.co.uk wrote: Sean Murphy mhysnm1...@gmail.com writes: I am a Vision Impaired programmer on the Mac and Window platforms. I have started to learn Python. The biggest road block I have is the ability of debugging my simple scripts. The IDLE program does not work with the screen readers I use on the Mac or Windows. A screen reader is a program that grabs the text on the screen and converts it into speech output, at least this is the 5 feet explanation. I cannot see the screen at all. I have looked at eclipse and it doesn't work with Voice-Over (the screen reader on the Mac). I have java issues on my windows machine preventing me running this app. If I use $python -d script.py the debugger doesn't seem to trigger on the mac. So how I can perform a debug on a script so I can step through it, set up break points, watch variables, etc. It is really annoying me, since under Perl I just added the -d switch and had a full debugger that worked at the console level. For command line debugging see http://docs.python.org/3/library/pdb.html. More generally you might want to investigate http://emacspeak.sourceforge.net/ (disclaimer - I have never used this, but from what you say you might find it useful). -- https://mail.python.org/mailman/listinfo/python-list -- https://mail.python.org/mailman/listinfo/python-list
Strange behaviour with a for loop.
Hello all. This is a newly question. But I wish to understand why the below code is providing different results. import os, sys if len(sys.argv) 2: filenames = sys.argv[1:] else print (no parameters provided\n) sys.edit() for filename in filenames: print (filename is: %s\n %filename) The above code will return results like: filename is test.txt If I modify the above script slightly as shown below, I get a completely different result. if len(sys.argv) 2: filenames = sys.argv[1] else print (no parameters provided\n) sys.exit() for filename in filenames: print (filename is: %s\n % filename) The result is the filename is spelled out a character at a time. The bit I am missing is something to do with splicing or referencing in Python. Why am I getting different results? In other languages I would have got the whole content of the element when using the index of the array (list). Sean filename is: t filename -- https://mail.python.org/mailman/listinfo/python-list
Re: Strange behaviour with a for loop.
Thanks everyone. Mark thanks for the correction on the ':'. Since I didn't cut and copy, rather typed it out. Errors crept in. :-) another question in relation to slicing strings. If you want to get a single character, just using the index position will get it. If I use the following, shouldn't it also work? when I use Python 3.3, it didn't provide anything. a = test.txt print a[3] result is: 't print a[3:1] Nothing is printed. print a[3:2] Nothing is printed. print a[3:-1] t.tx is printed. Why doesn't the positive number of characters to be splice return anything while the negative value does? sorry about these basic questions. I do like the splice feature within Python. Also what is the best method of testing for a blank string? end of paragraph line 1 new paragraph of line 1. The above example text is what I want to test for. I am planning to either load the whole file in as a single chunk of memory using fp.read() or store it into an array by using fp.readlines(). The first option I see being useful because you can create a regular expression to test for multiple '\n'. While in an array (list) I would have to test for a blank line which I assume would be . Any suggestions on this would be welcomed. Sean print a[ On 04/01/2014, at 4:38 PM, Mark Lawrence breamore...@yahoo.co.uk wrote: On 04/01/2014 04:03, Sean Murphy wrote: Hello all. This is a newly question. But I wish to understand why the below code is providing different results. import os, sys if len(sys.argv) 2: filenames = sys.argv[1:] else print (no parameters provided\n) sys.edit() for filename in filenames: print (filename is: %s\n %filename) The above code will return results like: filename is test.txt If I modify the above script slightly as shown below, I get a completely different result. if len(sys.argv) 2: filenames = sys.argv[1] else print (no parameters provided\n) sys.exit() for filename in filenames: print (filename is: %s\n % filename) The result is the filename is spelled out a character at a time. The bit I am missing is something to do with splicing or referencing in Python. Why am I getting different results? In other languages I would have got the whole content of the element when using the index of the array (list). Sean filename is: t filename As you've already had answers I'd like to point out that your test for len(sys.argv) is wrong, else is missing a colon and sys.edit() is very unlikely to work :) -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list -- https://mail.python.org/mailman/listinfo/python-list
Re: Strange behaviour with a for loop.
Hi everyone. Worked out what I was doing wrong with the string splicing. The offset number was lower then the index number, so it was failing. E.G: On 04/01/2014, at 4:54 PM, Sean Murphy mhysnm1...@gmail.com wrote: Thanks everyone. Mark thanks for the correction on the ':'. Since I didn't cut and copy, rather typed it out. Errors crept in. :-) another question in relation to slicing strings. If you want to get a single character, just using the index position will get it. If I use the following, shouldn't it also work? when I use Python 3.3, it didn't provide anything. a = test.txt print a[3] print a[4:1] --- index is 4 and offset is one. This is invalid. So I suspect the offset number still starts at the beginning of the string and counts forward or another way to look at it you are slicing from element x to element y. If element y is less then element x, return nothing. Does this make sense? I should have used: print a[4:6]) to get: t.t The 2nd part of my original question still stands. I will expand upon this a bit more to give more context. I want to print from the beginning of the paragraph to the end. Each paragraph ends with \n\n\n. If I use \n\n\n in lines this does return true for the string. But I don't have a starting position and ending position. The list method which I mention before can be sliced by going back one element. Any suggestion on this would be welcomed. I want to achieve this using standard core python objects/methods. Sean result is: 't print a[3:1] Nothing is printed. print a[3:2] Nothing is printed. print a[3:-1] t.tx is printed. Why doesn't the positive number of characters to be splice return anything while the negative value does? sorry about these basic questions. I do like the splice feature within Python. Also what is the best method of testing for a blank string? end of paragraph line 1 new paragraph of line 1. The above example text is what I want to test for. I am planning to either load the whole file in as a single chunk of memory using fp.read() or store it into an array by using fp.readlines(). The first option I see being useful because you can create a regular expression to test for multiple '\n'. While in an array (list) I would have to test for a blank line which I assume would be . Any suggestions on this would be welcomed. Sean print a[ On 04/01/2014, at 4:38 PM, Mark Lawrence breamore...@yahoo.co.uk wrote: On 04/01/2014 04:03, Sean Murphy wrote: Hello all. This is a newly question. But I wish to understand why the below code is providing different results. import os, sys if len(sys.argv) 2: filenames = sys.argv[1:] else print (no parameters provided\n) sys.edit() for filename in filenames: print (filename is: %s\n %filename) The above code will return results like: filename is test.txt If I modify the above script slightly as shown below, I get a completely different result. if len(sys.argv) 2: filenames = sys.argv[1] else print (no parameters provided\n) sys.exit() for filename in filenames: print (filename is: %s\n % filename) The result is the filename is spelled out a character at a time. The bit I am missing is something to do with splicing or referencing in Python. Why am I getting different results? In other languages I would have got the whole content of the element when using the index of the array (list). Sean filename is: t filename As you've already had answers I'd like to point out that your test for len(sys.argv) is wrong, else is missing a colon and sys.edit() is very unlikely to work :) -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list -- https://mail.python.org/mailman/listinfo/python-list
