Hello,
Quick background on what I'm trying to achieve.
I have a data set from a digital storage oscilloscope. It includes sampled
data points for several electrical events that I'd like to break down and
decode.
The scope generates a single file with all of the events concatenated. The
data set is a comma-delimited file with time and volts. There's a header
between events that will allow me to recognize the beginning of a new
event. This 'raw' file is roughly 4 GBytes of data. Too much for any
editor to handle.
So, I've written a script that will go through and 'chunk' out each event
and save it to a .csv file. Now, I have smaller files to work with but
they are still a bit too large for most applications. These files are
roughly 50 MByte.
In order to extract the desired information from the files, I ended up
reading through each row in the .csv file and finding the events of
interest (rising edges) and saved off data on either side of the event into
arrays, which I saved to .csv files. I then wrote a script that further
processed the information to generate actual bit-concatenated words..
So, here's where it gets interesting. And, I'm presuming that someone out
there knows exactly what is going on and can help me get past this hurdle.
When I read through the .csv file and collect the events and the bits, I
get the expected result. A 16-bit output
To improve efficiency, I then took this snippet of code and included it
into a larger script that will help skip a few manual steps and automate
the process on the original 4 GByte file. In this larger script, I save
the data that would have normally gone to the .csv files into an array and
I work on the array within the script. Everything should be the same.. or
so I thought.
When I use the same code, except reading an array, I get results that are
basically 10x the size of the correct output.
I've checked the contents of the array against the contents of the .csv
file (the sources in each of these cases) and they are identical to each
other. Same size, dimensions and data.
My guess, at this point, is that the way a loop reading a .csv file and the
way a loop reads an array are somehow slightly different and my code isn't
accounting for this.
The other possibility is that I've become code-blind to a simple mistake
which my brain keeps overlooking...
Thank you in advance for your time,
Example code when reading from file:
risingdetected = False
for row in csvReader:
voltage = float(row[1])
# print("test v ", voltage)
if(voltage > (avg + triglevel) and not(risingdetected)):
# we've found an edge
risingdetected = True
edgearray.append([float(row[0]), float(row[1])])
# print(edgearray)
elif(voltage < (avg + triglevel) and risingdetected):
# print(voltage)
# we've found the falling edge of the signal
risingdetected = False
print("edge array: ", edgearray) # displays the array
arraysize = len(edgearray) # roughly a count <= 33
print("edge array size: ", arraysize) # display size
Example code when reading array inside a script:
(note that I simplified things since it didn't make sense to have a bunch
of repeated math going on for what ended up being the same result.
Specifically, triggervolts, which is the avg + the trigger level)
[I also added in the else statement to see if there were cases where it was
falling through.. which there are. But that should be the case in both
instances]:
risingdetected = False
for row in range (len(TrigWind)):
# grab the voltage from this entry
voltage = float(TrigWind[row][1])
# if we've not already detected a rising edge
# and we're over the trigger level
if((voltage > triggervolts) and not(risingdetected)):
# set the rising edge detected to help control flow
risingdetected = True
# write the edge entry into the edge array
edgearray.append([float(TrigWind[row][0]),
float(TrigWind[row][1])])
# We've detected a rising edge, now we're looking for a falling
edge
elif((voltage < triggervolts) and risingdetected):
# we're done with the pulse, time to wait for the next one..
risingdetected = False
else:
print("Error")
print("Edge array: ", edgearray) # display the array results
arraysize = len(edgearray) # ends up being about twice the size
of the .csv version
print("Size of edagearray: ", arraysize) # show the size of the
array
_______________________________________________
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
