Re: [Tutor] Multiprocessing with many input input parameters
Sydney, 1 There have been many projects to look at cells, division, multiplication, ... It is worth researching the Python eco-system in the expectation of saving yourself time and effort! 2 The latest releases of Python (such as you quote) offer updated asyncio module(s) for multiprocessing, ie be careful if you are reading older articles! We haven't discussed hardware. Most modern PC CPUs offer multiple "cores". Assuming (say) four cores, asyncio is capable of running up to four processes concurrently - realising attendant acceleration of the entirety. (admittedly, I tend to limit my ambitions to number_of_cores - 1) On 12/07/19 3:40 AM, Mike Barnett wrote: If you're passing parameters as a list, then you need a "," at the end of the items. Otherwise if you have something like a string as the only item, the list will be the string. list_with_one_item = ['item one',] @mike -Original Message- From: Shall, Sydney Sent: Wednesday, July 10, 2019 11:44 AM To: tutor@python.org Subject: [Tutor] Multiprocessing with many input input parameters I am using MAC OS X 10.14.5 on a MAC iBook I use Python 3.7.0 from Anaconda, with Spyder 3.3.3 I am a relative beginner. My program models cell reproduction. I have written a program that models this and it works. Now I want to model a tissue with several types of cells. I did this by simply rerunning the program with different inputs (cell characteristics). But now I want to send and receive signals between the cells in each population. This requires some sort of concurrent processing with halts at appropriate points to pass and receive signals. I thought to use multiprocessing. I have read the documentation and reproduced the models in the docs. But I cannot figure out how to feed in the data for multiple parameters. I have tried using Pool and it works fine, but I can only get it to accept 1 input parameter, although multiple data inputs with one parameter works nicely. So, my questions are; 1. Is multiprocessing the suitable choice. 2. if yes, how does one write a function with multiple input parameters. Thank s in advance. Sydney Prodessor. Sydney Shall Department of Haematological Medicine King's College London 123 Coldharbour Lane London SE5 9NU ENGLAND E-Mail: sydney.shall (Correspondents outside the College should add @KCL.AC.UK) TEL: +44 (0)208 48 59 01 ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor -- Regards =dn ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
[Tutor] Reading .csv data vs. reading an array
Hello,
Quick background on what I'm trying to achieve.
I have a data set from a digital storage oscilloscope. It includes sampled
data points for several electrical events that I'd like to break down and
decode.
The scope generates a single file with all of the events concatenated. The
data set is a comma-delimited file with time and volts. There's a header
between events that will allow me to recognize the beginning of a new
event. This 'raw' file is roughly 4 GBytes of data. Too much for any
editor to handle.
So, I've written a script that will go through and 'chunk' out each event
and save it to a .csv file. Now, I have smaller files to work with but
they are still a bit too large for most applications. These files are
roughly 50 MByte.
In order to extract the desired information from the files, I ended up
reading through each row in the .csv file and finding the events of
interest (rising edges) and saved off data on either side of the event into
arrays, which I saved to .csv files. I then wrote a script that further
processed the information to generate actual bit-concatenated words..
So, here's where it gets interesting. And, I'm presuming that someone out
there knows exactly what is going on and can help me get past this hurdle.
When I read through the .csv file and collect the events and the bits, I
get the expected result. A 16-bit output
To improve efficiency, I then took this snippet of code and included it
into a larger script that will help skip a few manual steps and automate
the process on the original 4 GByte file. In this larger script, I save
the data that would have normally gone to the .csv files into an array and
I work on the array within the script. Everything should be the same.. or
so I thought.
When I use the same code, except reading an array, I get results that are
basically 10x the size of the correct output.
I've checked the contents of the array against the contents of the .csv
file (the sources in each of these cases) and they are identical to each
other. Same size, dimensions and data.
My guess, at this point, is that the way a loop reading a .csv file and the
way a loop reads an array are somehow slightly different and my code isn't
accounting for this.
The other possibility is that I've become code-blind to a simple mistake
which my brain keeps overlooking...
Thank you in advance for your time,
Example code when reading from file:
risingdetected = False
for row in csvReader:
voltage = float(row[1])
# print("test v ", voltage)
if(voltage > (avg + triglevel) and not(risingdetected)):
# we've found an edge
risingdetected = True
edgearray.append([float(row[0]), float(row[1])])
# print(edgearray)
elif(voltage < (avg + triglevel) and risingdetected):
# print(voltage)
# we've found the falling edge of the signal
risingdetected = False
print("edge array: ", edgearray)# displays the array
arraysize = len(edgearray)# roughly a count <= 33
print("edge array size: ", arraysize)# display size
Example code when reading array inside a script:
(note that I simplified things since it didn't make sense to have a bunch
of repeated math going on for what ended up being the same result.
Specifically, triggervolts, which is the avg + the trigger level)
[I also added in the else statement to see if there were cases where it was
falling through.. which there are. But that should be the case in both
instances]:
risingdetected = False
for row in range (len(TrigWind)):
# grab the voltage from this entry
voltage = float(TrigWind[row][1])
# if we've not already detected a rising edge
# and we're over the trigger level
if((voltage > triggervolts) and not(risingdetected)):
# set the rising edge detected to help control flow
risingdetected = True
# write the edge entry into the edge array
edgearray.append([float(TrigWind[row][0]),
float(TrigWind[row][1])])
# We've detected a rising edge, now we're looking for a falling
edge
elif((voltage < triggervolts) and risingdetected):
# we're done with the pulse, time to wait for the next one..
risingdetected = False
else:
print("Error")
print("Edge array: ", edgearray)# display the array results
arraysize = len(edgearray)# ends up being about twice the size
of the .csv version
print("Size of edagearray: ", arraysize)# show the size of the
array
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Multiprocessing with many input input parameters
If you're passing parameters as a list, then you need a "," at the end of the items. Otherwise if you have something like a string as the only item, the list will be the string. list_with_one_item = ['item one',] @mike -Original Message- From: Shall, Sydney Sent: Wednesday, July 10, 2019 11:44 AM To: tutor@python.org Subject: [Tutor] Multiprocessing with many input input parameters I am using MAC OS X 10.14.5 on a MAC iBook I use Python 3.7.0 from Anaconda, with Spyder 3.3.3 I am a relative beginner. My program models cell reproduction. I have written a program that models this and it works. Now I want to model a tissue with several types of cells. I did this by simply rerunning the program with different inputs (cell characteristics). But now I want to send and receive signals between the cells in each population. This requires some sort of concurrent processing with halts at appropriate points to pass and receive signals. I thought to use multiprocessing. I have read the documentation and reproduced the models in the docs. But I cannot figure out how to feed in the data for multiple parameters. I have tried using Pool and it works fine, but I can only get it to accept 1 input parameter, although multiple data inputs with one parameter works nicely. So, my questions are; 1. Is multiprocessing the suitable choice. 2. if yes, how does one write a function with multiple input parameters. Thank s in advance. Sydney Prodessor. Sydney Shall Department of Haematological Medicine King's College London 123 Coldharbour Lane London SE5 9NU ENGLAND E-Mail: sydney.shall (Correspondents outside the College should add @KCL.AC.UK) TEL: +44 (0)208 48 59 01 ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Make a linked list subscriptable?
On 7/10/19 6:30 PM, Sarah Hembree wrote: > How might I best make a linked list subscriptable? Below is skeleton code > for a linked list (my > actual is much more). I've done __iter__ and __next__ but I would like to > be able to do start:stop:stride I just can't figure out how. Suggestions or > just hints please? As a learning exercise this can be interesting, but as to practical applications, one would like to ask "why"? If index into the list is important, then choose a regular list; the "interesting" part of a linked list, which is "next node" is then available as index + 1. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Make a linked list subscriptable?
On 10Jul2019 20:30, Sarah Hembree wrote: How might I best make a linked list subscriptable? Below is skeleton code for a linked list (my actual is much more). I've done __iter__ and __next__ but I would like to be able to do start:stop:stride I just can't figure out how. Suggestions or just hints please? Well, you could write a method to find and return element `n`, counting from 0 (the root Node). For start stop stride you could do the extremely simple approach: iterate over a range() of the start stop stride and call the get-element-n method. This will be very slow though (a complete scan for every element). Instead, you could write a method accepting a start, stop and stride. Call the find-element-n method to get to start, lets call that `n`. While n < stop, step forward `stride` elements and also bump `n` by stride. That steps you along each element indicated by the start:stop:stride. You'll notice that this only works for a positive stride. For a negative stride you're probably better off handing it to range(), getting a list of values back from that as a list, and reversing the list (which then has ascending values). Collect the elements indicated by the list (because you can traverse the linkedlist forwards). Then reverse the collected elements and return them. Now, you _could_ accumulate each element in a list and return that at the end. _Or_ you could make the function a generator and just yield each element as found. To turn all this into a subscriptable list, define the __getitem__ method as a function accepting an index. If that index is an int, just return that element. If the index is a slice (start:stop:stride), call the more complicated function to return multiple elements. Cheers, Cameron Simpson ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
[Tutor] Make a linked list subscriptable?
How might I best make a linked list subscriptable? Below is skeleton code for a linked list (my actual is much more). I've done __iter__ and __next__ but I would like to be able to do start:stop:stride I just can't figure out how. Suggestions or just hints please? # -*- coding: utf8 -*- class Node: def __init__(self, val=None, nxt=None): self.val = val self.next = nxt class LinkedList: def __init__(self, node=None): self.root = node self.length = 0 def prepend(self, data): """ add data to head/root of list """ if self.root == None: self.root = Node(data) self.length = self.length + 1 else: n = Node(data) n.next = self.root self.root = n self.length = self.length + 1 def pop(self): """ Remove first data node """ t = self.root.val if self.root: self.root = self.root.next self.length = self.length - 1 return t def __repr__(self): tmp = self.root s = '' while tmp: s = s + str(tmp.val) + '> ' tmp = tmp.next return s[:-2] ll = LinkedList() [ll.prepend(x) for x in range(14,-1,-1)] >>> ll 0> 1> 2> 3> 4> 5> 6> 7> 8> 9> 10> 11> 12> 13> 14 --- We not only inherit the Earth from our Ancestors, we borrow it from our Children. Aspire to grace. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
