Re: [Numpy-discussion] Reading footer lines
On Tue, Aug 13, 2013 at 6:01 AM, Daπid wrote: > Alternatively, you could also use seek to put the pointer a certain > distance from the end of the file and start from there, That's what I'd do if the file(s) may be too large to simply dump into memory. > but this could cause problems in Windows. what problems? this is pretty straightforward stuff! -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Reading footer lines
On 13 August 2013 14:20, Resmi wrote:
> As a workaround, I've tried using os.system along with grep. And I get the
> following output :
>
os.system("grep -e 'tx' 'data.dat' ")
> ## tx =2023.06
> 0
>
> Why is there a 0 in the output? The file has no blank lines.
That 0 corresponds to the exit status of os.system, as there were no
errors, it returns a 0.
My idea would be to read the file and store it into a StringIO. That
can be fed to np.loadtxt to extract the numbers, and look for your
number. My (untested) attempt:
import StringIO
with datafile open as f:
... raw_data = StringIO.StringIO()
... raw_data.write(f.read())
data = np.loadfromtxt(raw_data)
numbers = [float(line.split['='][1].strip()) for line in
raw_data.readlines() if line[0] == '#']
If there is only one number there and it is after all the data, you
could skip them all. From the shape of data you know there are N lines
of data:
for _ in range(N):
... raw_data.readline()
while True:
... line = raw_data.readline()
... if line[0] == '#':
.. number = float(line.split['='][1].strip())
.. break
Alternatively, you could also use seek to put the pointer a certain
distance from the end of the file and start from there, but this could
cause problems in Windows.
David.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Reading footer lines
Hi Resmi On Tue, Aug 13, 2013 at 2:20 PM, Resmi wrote: > I've a list of long files of numerical data ending with footer lines > (beginning with #). I am using numpy.loadtxt to read the numbers, and > loadtxt ignores these footer lines. I want the numpy code to read one of the > footer lines and extract words from it. Is there a way to use loadtxt for > this? If there weren't many files I could have used the line number (which > keep varying between the files) of the footer line along with linecache. > Nevertheless there should be a generic way to do this in numpy? If the data-set is not too big, you can read it into memory with "f.readlines()" and then do a bit of preparsing on the resulting array before sending the rest of it to np.loadtxt (which allows an array of strings as input). Stéfan ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Reading footer lines
On Tue, Aug 13, 2013 at 1:20 PM, Resmi wrote:
> Hi,
>
> I've a list of long files of numerical data ending with footer lines
> (beginning with #). I am using numpy.loadtxt to read the numbers, and
> loadtxt ignores these footer lines. I want the numpy code to read one of the
> footer lines and extract words from it. Is there a way to use loadtxt for
> this? If there weren't many files I could have used the line number (which
> keep varying between the files) of the footer line along with linecache.
> Nevertheless there should be a generic way to do this in numpy?
>
> As a workaround, I've tried using os.system along with grep. And I get the
> following output :
>
os.system("grep -e 'tx' 'data.dat' ")
> ## tx =2023.06
> 0
>
> Why is there a 0 in the output? The file has no blank lines.
The os.system function call returns the integer return value
(error level) of the command invoked, by convention zero
for success but the value is set by the tool (here grep).
> Since I want the value 2023.06 in the numpy code for later use I tried to
> pipe the output to a variable:-
> test = os.system(command)
> But that isn't working as test is getting assigned the value 0.
> Tried subprocess.call(['grep','-e','tx','data.dat']) which is also ending up
> in the same fashion.
>
> It'll be great if I can get to know (i) a way to read the footer lines (ii)
> to sort out the operation of os.system and subprocess.call output
> processing.
Don't use os.system, instead you should use subprocess:
http://docs.python.org/2/library/subprocess.html
The standard library module commands would also work
but is not cross platform, and is also deprecated in favour
of subprocess: http://docs.python.org/2/library/commands.html
Peter
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Reading footer lines
Hi,
I've a list of long files of numerical data ending with footer lines
(beginning with #). I am using numpy.loadtxt to read the numbers, and
loadtxt ignores these footer lines. I want the numpy code to read one of
the footer lines and extract words from it. Is there a way to use loadtxt
for this? If there weren't many files I could have used the line
number (which keep varying between the files) of the footer line along with
*linecache.*
Nevertheless there should be a generic way to do this in numpy?
As a workaround, I've tried using os.system along with grep. And I get the
following output :
*>>> os.system("grep -e 'tx' 'data.dat' ")*
* ## tx =2023.06 *
*0*
Why is there a 0 in the output? The file has no blank lines.
Since I want the value 2023.06 in the numpy code for later use I tried to
pipe the output to a variable:-
*test = os.system(command)*
But that isn't working as *test* is getting assigned the value 0.
Tried *subprocess.call(['grep','-e','tx','data.dat'])* which is also ending
up in the same fashion.
It'll be great if I can get to know (i) a way to read the footer lines (ii)
to sort out the operation of os.system and subprocess.call output
processing.
Thanks,
Resmi
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
