Re: [Tutor] Multiprocessing with many input input parameters
On Fri, Jul 12, 2019 at 11:53:16AM +, Shall, Sydney via Tutor wrote: > Thanks Mike, > > But I am still not clear. Neither is your question. > do I write: > > def f([x,y,z]) ? > How exactly do one write the function and how does one ensure that each > positional argument is accounted for. Is your function taking three seperate arguments, or a single argument that must be a list of exactly three items? (1) Three seperate arguments: def function(a, b, c): # Write the body of the function. ... That's all you need do, as the interpreter will ensure it is only called with three arguments. (2) One list argument with three items: def function(alist): if isinstance(alist, list): if len(alist) < 3: raise ValueError("too few arguments in alist") elif len(alist) > 3: raise ValueError("too many arguments in alist") else: a, b, c = alist # Unpack the three items from the list. # Write the body of the function here. ... else: raise TypeError('expected a list') Python will ensure your function is only called with a single argument. The rest is up to you. -- Steven ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Multiprocessing with many input input parameters
Right, I meant tuple, not list. a = ('A string') b = ('A List Member',) print(a[0]) print(b[0]) The output for this is: A A List Member @mike -Original Message- From: Cameron Simpson Sent: Friday, July 12, 2019 7:59 PM To: Mike Barnett Cc: Shall, Sydney ; tutor@python.org Subject: Re: [Tutor] Multiprocessing with many input input parameters On 11Jul2019 15:40, Mike Barnett wrote: >If you're passing parameters as a list, then you need a "," at the end of the >items. Otherwise if you have something like a string as the only item, the >list will be the string. > >list_with_one_item = ['item one',] Actually, this isn't true. This is a one element list, no trailing coma required: [5] Mike has probably confused this with tuples. Because tuples are delineated with parentheses, there is ambiguity between a tuple's parentheses and normal "group these terms together" parentheses. So: x = 5 + 4 * (9 + 7) Here we just have parentheses causing the assignment "9 + 7" to occur before the multiplication by 4. And this is also legal: x = 5 + 4 * (9) where the parentheses don't add anything special in terma of behaviour. Here is a 2 element tuple: (9, 7) How does one write a one element tuple? Like this: (9,) Here the trailing comma is _required_ to syntacticly indicate that we intend a 1 element tuple instead of a plain "9 in parentheses") as in the earlier assignment statement. I'm not sure any of this is relevant to Sydney's question though. Cheers, Cameron Simpson ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Multiprocessing with many input input parameters
On 7/12/19 5:53 AM, Shall, Sydney via Tutor wrote: > Thanks Mike, > > But I am still not clear. > > do I write: > > def f([x,y,z]) ? > How exactly do one write the function and how does one ensure that each > positional argument is accounted for. The concept of packing will be useful, you can use the * operator to pack and unpack. A trivial example to get you started: >>> a = [1, 2, 3, 4] >>> print(a) [1, 2, 3, 4] >>> print(*a) 1 2 3 4 In the first print we print the list, in the second we print the result of unpacking the list - you see it's now four numbers rather than one list of four numbers. In a function definition you can pack with the * operator: >>> def f(*args): ... print(type(args)) ... print(len(args)) ... print(args) ... >>> >>> >>> f(1, 2, 3, 4) 4 (1, 2, 3, 4) Here we called the function with four arguments, but it received those packed into the one argument args, which is a tuple of length 4. Python folk conventionally name the argument which packs the positional args that way - *args - but the name "args" has no magic, its familiarity just aids in recognition. By packing your positional args you don't error out if you're not called with the exact number you expect (or if you want to accept differing numbers of args), and then you can do what you need to with what you get. The same packing concept works for dictionaries as well, here the operator is **. >>> def fun(a, b, c): ... print(a, b, c) ... >>> d = {'a':2, 'b':4, 'c':10} >>> fun(**d) 2 4 10 What happened here is in unpacking, the keys in the dict are matched up with the names of the function parameters, and the values for those keys are passed as the parameters. If your dict doesn't match, it fails: >>> d = {'a':2, 'b':4, 'd':10} >>> fun(**d) Traceback (most recent call last): File "", line 1, in TypeError: fun() got an unexpected keyword argument 'd' Dictionary unpacking looks like: >>> def fun(**kwargs): ... print(f"{kwargs}") ... >>> >>> fun(a=1, b=2, c=3) {'a': 1, 'b': 2, 'c': 3} again the name 'kwargs' is just convention. There are rules for how to mix regular positional args, unpacked positional args (or varargs), and keyword ares but don't want to go on forever here... ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Multiprocessing with many input input parameters
Hi Sydney, On Wed, 10 Jul 2019 at 16:45, Shall, Sydney via Tutor wrote: > > I am a relative beginner. > > My program models cell reproduction. I have written a program that models > this and it works. > > Now I want to model a tissue with several types of cells. I did this by > simply rerunning the program with different inputs (cell characteristics). > But now I want to send and receive signals between the cells in each > population. This requires some sort of concurrent processing with halts at > appropriate points to pass and receive signals. You say that this "requires some sort of concurrent processing" but I think that you are mistaken there. I have a lot of experience in mathematical modelling and dynamic simulation including some multi-cell models and I have never personally come across a situation where any significant benefit could be obtained from using concurrent processing for different parts of a single simulation. Those situations do exist but you haven't said anything to make me think that yours is an exceptional case. A simple way to do this (not the only way!) is something like: # Some data structure that stores which cells are sending messages messages_from_cells = {} for t in range(num_timesteps): # Calculate new state of all cells based only on the old states of all cells and messages. new_cells = {} for c in range(len(cells)): new_cells[c] = update_cell(cells[c], messages_from_cells) # Update all cells synchronously: cells = new_cells # Update messages based on new cell states: for c in range(len(cells)): messages_from_cells = update_messages(cells[c], messages_from_cells) You just need to figure out a data structure (I've suggested a dict above) that would somehow store what messages are being passed between which cells. You can update each cell based on the current messages and then update the messages ready for the next timestep. Concurrent execution is not needed: I have simulated concurrency by using two separate loops over the cells. The result is as if each cell was updated concurrently. Another approach is that at each timestep you can choose a cell randomly and update that one keeping all the others constant. It really depends what kind of model you are using. In a simulation context like this there are two different reasons why you might conceivably want to use concurrent execution: 1. Your simulations are CPU-bound and are slow and you need to make them run faster by using more cores. 2. Your simulation needs more memory then an individual computer has and you need to run it over a cluster of many computers. Python's multiprocessing module can help with the first problem: it can theoretically make your simulations run faster. However it is hard to actually achieve any speedup that way. Most likely there are other ways to make your code run faster that are easier than using concurrent execution and can deliver bigger gains. Multiprocessing used well might make your code 10x faster but I will bet that there are easier ways to make your code 100x faster. Multiprocessing makes the second problem worse: it actually uses more memory on each computer. To solve problem 2 is very hard but can be done. I don't think either problem applies to you though. There is a situation where I have used multiprocessing to make simulations faster. In practice I rarely want to do just one simulation; I want to do thousands with different parameters or because they are stochastic and I want to average them. Running these thousands of simulations can be made faster easily with multiprocessing because it is an "embarrassingly parallel" problem. You need to get your simulations working without multiprocessing first though. This is a much easier way to solve problem 1 (in so far as using more cores can help). Side note: you've misspelt Professor in your email signature: > Prodessor. Sydney Shall -- Oscar ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Multiprocessing with many input input parameters
On Sat, Jul 13, 2019 at 09:59:16AM +1000, Cameron Simpson wrote: > Mike has probably confused this with tuples. Because tuples are > delineated with parentheses, there is ambiguity between a tuple's > parentheses and normal "group these terms together" parentheses. There are no "tuple parentheses". Tuples are created by the *comma*, not the parens. The only exception is the empty tuple, since you can't have a comma on its own. x = ()# Zero item tuple. x = 1,# Single item tuple. x = 1, 2 # Two item tuple. Any time you have a tuple, you only need to put parens around it to dismbiguate it from the surrounding syntax: x = 1, 2, (3, 4, 5), 6 # Tuple containing a tuple. function(0, 1, (2, 3), 4) # Tuple as argument to a function. or just to make it more clear to the human reader. > Here is a 2 element tuple: > > (9, 7) > > How does one write a one element tuple? Like this: > > (9,) To be clear, in both cases you could drop the parentheses and still get a tuple: 9, 7 9, provided that wasn't in a context where the comma was interpreted as something with higher syntactic precedence, such as a function call: func(9, 7)# Two integer arguments, not one tuple argument. func((9, 7)) # One tuple argument. -- Steven ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Multiprocessing with many input input parameters
On 11Jul2019 15:40, Mike Barnett wrote: If you're passing parameters as a list, then you need a "," at the end of the items. Otherwise if you have something like a string as the only item, the list will be the string. list_with_one_item = ['item one',] Actually, this isn't true. This is a one element list, no trailing coma required: [5] Mike has probably confused this with tuples. Because tuples are delineated with parentheses, there is ambiguity between a tuple's parentheses and normal "group these terms together" parentheses. So: x = 5 + 4 * (9 + 7) Here we just have parentheses causing the assignment "9 + 7" to occur before the multiplication by 4. And this is also legal: x = 5 + 4 * (9) where the parentheses don't add anything special in terma of behaviour. Here is a 2 element tuple: (9, 7) How does one write a one element tuple? Like this: (9,) Here the trailing comma is _required_ to syntacticly indicate that we intend a 1 element tuple instead of a plain "9 in parentheses") as in the earlier assignment statement. I'm not sure any of this is relevant to Sydney's question though. Cheers, Cameron Simpson ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Multiprocessing with many input input parameters
Thanks Mike, But I am still not clear. do I write: def f([x,y,z]) ? How exactly do one write the function and how does one ensure that each positional argument is accounted for. Dr. Sydney Shall Department of Haematological Medicine King's College London 123 Coldharbour Lane London SE5 9NU ENGLAND E-Mail: sydney.shall (Correspondents outside the College should add @KCL.AC.UK) TEL: +44 (0)208 48 59 01 From: Mike Barnett Sent: 11 July 2019 16:40 To: Shall, Sydney Cc: tutor@python.org Subject: RE: [Tutor] Multiprocessing with many input input parameters If you're passing parameters as a list, then you need a "," at the end of the items. Otherwise if you have something like a string as the only item, the list will be the string. list_with_one_item = ['item one',] @mike -Original Message- From: Shall, Sydney Sent: Wednesday, July 10, 2019 11:44 AM To: tutor@python.org Subject: [Tutor] Multiprocessing with many input input parameters I am using MAC OS X 10.14.5 on a MAC iBook I use Python 3.7.0 from Anaconda, with Spyder 3.3.3 I am a relative beginner. My program models cell reproduction. I have written a program that models this and it works. Now I want to model a tissue with several types of cells. I did this by simply rerunning the program with different inputs (cell characteristics). But now I want to send and receive signals between the cells in each population. This requires some sort of concurrent processing with halts at appropriate points to pass and receive signals. I thought to use multiprocessing. I have read the documentation and reproduced the models in the docs. But I cannot figure out how to feed in the data for multiple parameters. I have tried using Pool and it works fine, but I can only get it to accept 1 input parameter, although multiple data inputs with one parameter works nicely. So, my questions are; 1. Is multiprocessing the suitable choice. 2. if yes, how does one write a function with multiple input parameters. Thank s in advance. Sydney Prodessor. Sydney Shall Department of Haematological Medicine King's College London 123 Coldharbour Lane London SE5 9NU ENGLAND E-Mail: sydney.shall (Correspondents outside the College should add @KCL.AC.UK) TEL: +44 (0)208 48 59 01 ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Multiprocessing with many input input parameters
On 12/07/2019 01:51, DL Neil wrote: > older articles! We haven't discussed hardware. Most modern PC CPUs offer > multiple "cores". Assuming (say) four cores, asyncio is capable of > running up to four processes concurrently - realising attendant > acceleration of the entirety. Just to pick up on this point because I often see it being cited. The number of concurrent processes running to achieve performance gain is only very loosely tied to the number of cores. We ran concurrent processes for many years before multi-core processors were invented with significant gains. Indeed any modern computer runs hundreds of "concurrent" processes on a small umber of cores and the OS switches between them. What the number of cores affects is the number of processes actually executing at the same time. If you just want to chunk up the processing of a large amount of data and run the exact same code multiple times then there is no point in having more than the number of cores. But if your concurrent processes are doing different tasks on different data then the number of cores is basically irrelevant. And especially if they are performing any kind of I/O operations since they are likely to be parked by the OS for most of the time anyway. Of course, there is a point where creating extra processes becomes counter-effective since that is an expensive operation, and especially if the process will be very short lived or only execute for tiny lengths of time (such as handling a network event by passing it on to some other process). But for most real world uses of multiprocessing the number of cores is not a major factor in deciding how many processes to run. I certainly would not hesitate to run 10xN where N is the number of cores. Beyond that you might need to think carefully. In Sydney's scenario is sounds like the processes are different and explicitly halt to perform I/O so the cores issue should not be a problem. -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Multiprocessing with many input input parameters
Sydney, 1 There have been many projects to look at cells, division, multiplication, ... It is worth researching the Python eco-system in the expectation of saving yourself time and effort! 2 The latest releases of Python (such as you quote) offer updated asyncio module(s) for multiprocessing, ie be careful if you are reading older articles! We haven't discussed hardware. Most modern PC CPUs offer multiple "cores". Assuming (say) four cores, asyncio is capable of running up to four processes concurrently - realising attendant acceleration of the entirety. (admittedly, I tend to limit my ambitions to number_of_cores - 1) On 12/07/19 3:40 AM, Mike Barnett wrote: If you're passing parameters as a list, then you need a "," at the end of the items. Otherwise if you have something like a string as the only item, the list will be the string. list_with_one_item = ['item one',] @mike -Original Message- From: Shall, Sydney Sent: Wednesday, July 10, 2019 11:44 AM To: tutor@python.org Subject: [Tutor] Multiprocessing with many input input parameters I am using MAC OS X 10.14.5 on a MAC iBook I use Python 3.7.0 from Anaconda, with Spyder 3.3.3 I am a relative beginner. My program models cell reproduction. I have written a program that models this and it works. Now I want to model a tissue with several types of cells. I did this by simply rerunning the program with different inputs (cell characteristics). But now I want to send and receive signals between the cells in each population. This requires some sort of concurrent processing with halts at appropriate points to pass and receive signals. I thought to use multiprocessing. I have read the documentation and reproduced the models in the docs. But I cannot figure out how to feed in the data for multiple parameters. I have tried using Pool and it works fine, but I can only get it to accept 1 input parameter, although multiple data inputs with one parameter works nicely. So, my questions are; 1. Is multiprocessing the suitable choice. 2. if yes, how does one write a function with multiple input parameters. Thank s in advance. Sydney Prodessor. Sydney Shall Department of Haematological Medicine King's College London 123 Coldharbour Lane London SE5 9NU ENGLAND E-Mail: sydney.shall (Correspondents outside the College should add @KCL.AC.UK) TEL: +44 (0)208 48 59 01 ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor -- Regards =dn ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Multiprocessing with many input input parameters
If you're passing parameters as a list, then you need a "," at the end of the items. Otherwise if you have something like a string as the only item, the list will be the string. list_with_one_item = ['item one',] @mike -Original Message- From: Shall, Sydney Sent: Wednesday, July 10, 2019 11:44 AM To: tutor@python.org Subject: [Tutor] Multiprocessing with many input input parameters I am using MAC OS X 10.14.5 on a MAC iBook I use Python 3.7.0 from Anaconda, with Spyder 3.3.3 I am a relative beginner. My program models cell reproduction. I have written a program that models this and it works. Now I want to model a tissue with several types of cells. I did this by simply rerunning the program with different inputs (cell characteristics). But now I want to send and receive signals between the cells in each population. This requires some sort of concurrent processing with halts at appropriate points to pass and receive signals. I thought to use multiprocessing. I have read the documentation and reproduced the models in the docs. But I cannot figure out how to feed in the data for multiple parameters. I have tried using Pool and it works fine, but I can only get it to accept 1 input parameter, although multiple data inputs with one parameter works nicely. So, my questions are; 1. Is multiprocessing the suitable choice. 2. if yes, how does one write a function with multiple input parameters. Thank s in advance. Sydney Prodessor. Sydney Shall Department of Haematological Medicine King's College London 123 Coldharbour Lane London SE5 9NU ENGLAND E-Mail: sydney.shall (Correspondents outside the College should add @KCL.AC.UK) TEL: +44 (0)208 48 59 01 ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor