Re: [Tutor] read text file in zip archive, process, plot
* Kent Johnson <[EMAIL PROTECTED]> [070415 20:30]:
> Washakie wrote:
> > Thanks so much! Now another task, associated with this one... what I
> > actually ultimately want is to just pull out several fields from the
> > text file in the zip archive (there is actually only one file).. so, my
> > goal is the to create a file that looks like:
> >
> > t[0], x[0], y[0]
> > t[1], x[1], y[1]
> > t[2], x[2], y[2]
> > t[3], x[3], y[3]
> > ...
> > t[:], x[:], y[:]
> >
> > Note, these would actually be strings representing the values... and I'm
> > not trying to do the header yet... :s
>
> You could just write them out as you process them. The same loop that
> reads the input lines can write the output lines.
Consider using the csv module.
out = file("output", "w")
cout = csv.writer(out)
cout.writerow((1,2,3,"abc", "def"))
> >
> > I've tried OUT = [t,x,y ]
> > but when I write it out, it seems to go through all of t first, then x,
> > then y...
> > i'm having a really hard time switching my thinking away from matlab
> > array processing which I think is part of my problem!!
out = file("output", "w")
cout = csv.writer(out)
cout.writerows(zip(t, x, y))
out.close()
Andreas
___
Tutor maillist - [EMAIL PROTECTED]
http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] read text file in zip archive, process, plot
Washakie wrote: > Thanks so much! Now another task, associated with this one... what I > actually ultimately want is to just pull out several fields from the > text file in the zip archive (there is actually only one file).. so, my > goal is the to create a file that looks like: > > t[0], x[0], y[0] > t[1], x[1], y[1] > t[2], x[2], y[2] > t[3], x[3], y[3] > ... > t[:], x[:], y[:] > > Note, these would actually be strings representing the values... and I'm > not trying to do the header yet... :s You could just write them out as you process them. The same loop that reads the input lines can write the output lines. > > I've tried OUT = [t,x,y ] > but when I write it out, it seems to go through all of t first, then x, > then y... > i'm having a really hard time switching my thinking away from matlab > array processing which I think is part of my problem!! Have you looked at NumPy? It has a native array type. http://numpy.scipy.org/ http://www.scipy.org/NumPy_for_Matlab_Users Kent ___ Tutor maillist - [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] read text file in zip archive, process, plot
Thanks so much! Now another task, associated with this one... what I actually ultimately want is to just pull out several fields from the text file in the zip archive (there is actually only one file).. so, my goal is the to create a file that looks like: t[0], x[0], y[0] t[1], x[1], y[1] t[2], x[2], y[2] t[3], x[3], y[3] ... t[:], x[:], y[:] Note, these would actually be strings representing the values... and I'm not trying to do the header yet... :s I've tried OUT = [t,x,y ] but when I write it out, it seems to go through all of t first, then x, then y... i'm having a really hard time switching my thinking away from matlab array processing which I think is part of my problem!! I read a discussion last night that I thought was interesting: http://mail.python.org/pipermail/python-list/2002-December/174954.html It was this point that made me start to think I need to 'rethink' how I'm going about things... but argh!! ... how to teach an old dog new tricks! -j ___ Tutor maillist - [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] read text file in zip archive, process, plot
John W wrote:
> Kent and Alan: better?
> .j
>
> import zipfile
> import os
> import pylab as P
> iFile = raw_input("Which file to process?")
>
> def openarchive(filename):
> """ open the cmet archive and read contents of file into memory """
> z = zipfile.ZipFile(filename, "r")
> for filename in z.namelist():
> print filename
> contents = z.read(filename)
> data = contents.split('\n')
> return data
This will just return the data of the first file in the zip, is that
what you want? The return statement will exit from the loop. If there
can be more than one file in the zip, you should call plotflight(data)
within the loop instead of returning the data.
If you only want the first file, you could omit the loop and write
filename = z.namelist()[0]
> def plotflight(data):
> """ Plot flight data t vs. hPa with Pylab """
> firstline = data[0].strip().split(' ')
> stind = int(firstline[0])
> hdrline = stind - 1
You don't use hdrline
> x = []
> t = []
> for l in data[stind:-1]:
> l = l.split() # Split on whitespace
> tval = float(l[0])
> t.append(tval)
> xval = float(l[24])
> x.append(xval)
I split out tval and xval for clarity but I would actually write this as
t.append(float(l[0]))
and skip the temp variables.
> #Create scatter plot using pylab function
> P.scatter(t,x)
> P.xlabel('time (s) after 00 UTC')
> P.ylabel('Altitude (hPa)')
> P.title('Flight elevation vs. Time')
> P.grid(True)
> #print out figure
> if os.path.exists("flight_track.png"):
> os.remove("flight_track.png")
> P.savefig('flight_track')
> P.close()
>
> flightdata = openarchive(iFile)
> plotflight(flightdata)
>
>
>
>
> ___
> Tutor maillist - [EMAIL PROTECTED]
> http://mail.python.org/mailman/listinfo/tutor
___
Tutor maillist - [EMAIL PROTECTED]
http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] read text file in zip archive, process, plot
* Washakie Wyoming <[EMAIL PROTECTED]> [070415 14:25]: > Thank you both! That seems to make my code much more clear... I thought > it was foolish writing files, but I just couldn't determine how to > parse the data! > > Kent, one thing, regarding: > x = [] > t = [] > for l in data[stind:-1]: > l = l.split() # Split on whitespace > tval = float(l[0]) > t.append(tval) > xval = float(l[24]) > x.append(xval) > > note that I had to change to [stind:-1] I believe I must have a blank > line at the end of the file... That depends upon the platform. Under Unix, files are meant to be ending in a \n, so you get an empty line at the end. But nobody enforces that. So personally, I'd probably add something like this to the loop: l = l.split() if not l: continue This way the loop ignores empty lines. Andreas ___ Tutor maillist - [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/tutor
[Tutor] read text file in zip archive, process, plot
Kent and Alan: better?
.j
import zipfile
import os
import pylab as P
iFile = raw_input("Which file to process?")
def openarchive(filename):
""" open the cmet archive and read contents of file into memory """
z = zipfile.ZipFile(filename, "r")
for filename in z.namelist():
print filename
contents = z.read(filename)
data = contents.split('\n')
return data
def plotflight(data):
""" Plot flight data t vs. hPa with Pylab """
firstline = data[0].strip().split(' ')
stind = int(firstline[0])
hdrline = stind - 1
x = []
t = []
for l in data[stind:-1]:
l = l.split() # Split on whitespace
tval = float(l[0])
t.append(tval)
xval = float(l[24])
x.append(xval)
#Create scatter plot using pylab function
P.scatter(t,x)
P.xlabel('time (s) after 00 UTC')
P.ylabel('Altitude (hPa)')
P.title('Flight elevation vs. Time')
P.grid(True)
#print out figure
if os.path.exists("flight_track.png"):
os.remove("flight_track.png")
P.savefig('flight_track')
P.close()
flightdata = openarchive(iFile)
plotflight(flightdata)
___
Tutor maillist - [EMAIL PROTECTED]
http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] read text file in zip archive, process, plot
Thank you both! That seems to make my code much more clear... I thought it was foolish writing files, but I just couldn't determine how to parse the data! Kent, one thing, regarding: x = [] t = [] for l in data[stind:-1]: l = l.split() # Split on whitespace tval = float(l[0]) t.append(tval) xval = float(l[24]) x.append(xval) note that I had to change to [stind:-1] I believe I must have a blank line at the end of the file... .j Interested in getting caught up on today's news? Click here to checkout USA TODAY Headlines. http://track.juno.com/s/lc?s=198954&u=http://www.usatoday.com/news/front.htm?csp=24 ___ Tutor maillist - [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] read text file in zip archive, process, plot
Washakie Wyoming wrote:
> Greetings all!
>
> I'm including here a first python program! Very nice. It's written to
> read in a text file which resides in a .zip archive, extract two fields
> and plot them. It uses some bits from pylab for the plotting.
>
> I'm writing to ask for ways to improve it. I feel like the writing and
> reading from 'jnk' files is definitely a hack! Any suggestions? I would
> be greatly appreciative of anyone provided comments on how to more
> efficiently tackle this problem. (NOTE: this is not a school
> assignment... just some side learning out of interest in what seems to
> be a great language!).
>
> Thanks!
>
> Here's my program (a zip is attached - if that's allowed???):
>
> import zipfile
> import os
> import pylab as P
>
> z = zipfile.ZipFile("flight_data.zip", "r")
> for filename in z.namelist():
> print filename
> contents = z.read(filename)
> f = open('jnk','w')
> f.write(contents)
> f.close()
> print "file closed, now reading"
> #now open files to process
> f = open('jnk','r')
> fn = open('newjnk','w')
> data = f.readlines()
Alan has shown you a better way to do this.
> firstline = data[0].strip().split(' ')
> stind = int(firstline[0])
> hdrline = stind - 1
> #print data[stind:len(data)]
> for l in data[stind:len(data)]:
You don't need the end index when it is the length of the data:
for l in data[stind:]:
> #l = l.replace(' ',',')
> fn.writelines(l)
> f.close()
> fn.close()
> #print 'file closed now'
>
> jnk2 = P.load('newjnk')
> t = jnk2[:,0]
> x = jnk2[:,24]
It looks like you are using newjnk just as a way to get P.load() to
parse the data for you. You can do this yourself with something like
x = []
t = []
for l in data[stind:]:
l = l.split() # Split on whitespace
tval = float(l[0])
t.append(tval)
xval = float(l[24])
x.append(xval)
Kent
___
Tutor maillist - [EMAIL PROTECTED]
http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] read text file in zip archive, process, plot
"Washakie Wyoming" <[EMAIL PROTECTED]> wrote
> I'm writing to ask for ways to improve it. I feel like the
> writing and reading from 'jnk' files is definitely a hack!
Yup!
I'm not sure why you do it, at least the first time.
You read the file into memory, why not just process it there?
You can split the file into lines using split('\n') and you get
the same list that readlines generates from the jnk file.
So you can replace all of this:
f = open('jnk','w')
f.write(contents)
f.close()
print "file closed, now reading"
#now open files to process
f = open('jnk','r')
fn = open('newjnk','w')
data = f.readlines()
with
data = contents.split('\n')
fn = open('newjnk','w')
I'm not familiar with pyLab so I don;t know if you can
bypass the newjnk file Also you seem to overwrite
the newjnk file before you use it. open('w') will overwrite
the existing file. Is that what you want? Or should you
be appending the data? Or maybe you want to create
a new file for each file in the zipfile? So you create a
new filename like:
fn = open(filename + '.jnk', 'w')
Othewise your pylab section only ever processes the
last file in the zipfile.
> Any suggestions?
The other thing is that I'd introduce a couple of functions,
one to process each file from the archive, the second to
do the pylab stuff. So your outer loop will shrink to
something like:
for filename in z.namelist():
prepareFile(filename)
display(filename)
It helps to keep the logic separated which makes
maintenance easier in the future.
HTH,
Alan G.
Here's my program (a zip is attached - if that's allowed???):
import zipfile
import os
import pylab as P
z = zipfile.ZipFile("flight_data.zip", "r")
for filename in z.namelist():
print filename
contents = z.read(filename)
f = open('jnk','w')
f.write(contents)
f.close()
print "file closed, now reading"
#now open files to process
f = open('jnk','r')
fn = open('newjnk','w')
data = f.readlines()
firstline = data[0].strip().split(' ')
stind = int(firstline[0])
hdrline = stind - 1
#print data[stind:len(data)]
for l in data[stind:len(data)]:
#l = l.replace(' ',',')
fn.writelines(l)
f.close()
fn.close()
#print 'file closed now'
jnk2 = P.load('newjnk')
t = jnk2[:,0]
x = jnk2[:,24]
P.scatter(t,x)
P.xlabel('time (s) after 00 UTC')
P.ylabel('Altitude (hPa)')
P.title('Flight elevation vs. Time')
P.grid(True)
if os.path.exists("flight_track.png"):
os.remove("flight_track.png")
P.savefig('flight_track')
P.close()
os.remove("jnk")
os.remove("newjnk")
Interested in getting caught up on today's news?
Click here to checkout USA TODAY Headlines.
http://track.juno.com/s/lc?s=198954&u=http://www.usatoday.com/news/front.htm?csp=24
> ___
> Tutor maillist - [EMAIL PROTECTED]
> http://mail.python.org/mailman/listinfo/tutor
>
___
Tutor maillist - [EMAIL PROTECTED]
http://mail.python.org/mailman/listinfo/tutor
