how to read mixed from multiple csv file
Hi,
My 20 csv files has string header, and first two columns are string (e.g.,
1999-01-02, 01:00:00) among the 50 columns. Other columns store numerical
values (int, or float)
I need to do data analysis for these data. For example, extract the each
month data from each of the cvs files (each csv file stores 1 year data)
and there are 20 year data.
in addition, I want to store the data in disk so that I can retrieve data
quickly, just like save and load in Matlab.
Currently, I use structured array
data = []
i = 0
for s in range(1991, 2011):
fileName = folder +_{_sY}0101_{_sY}1231_725300.csv.format(_sY=s)
data.append(np.genfromtxt(fileName, delimiter=,, dtype=None,
names=True))
i += 1
np.save(alldata, data)
However, when I load data np.load(alldata.npy), it is becomes 0-d array
which is different from original one.
My question is that
(1) How to store or save the data?
(2) as you can see, I use list to store all the 20 ndarrays, I do not feel
it is a good way. Is there any suggestion for the data structure I should
use?
Thanks in advance.
--
http://mail.python.org/mailman/listinfo/python-list
Re: why is this group being spammed?
Hi, can I subscribe this by gmail? On Sat, Jul 24, 2010 at 3:16 PM, Mark Lawrence breamore...@yahoo.co.ukwrote: On 24/07/2010 18:01, Dennis Lee Bieber wrote: On Sat, 24 Jul 2010 07:32:30 -0700 (PDT), be.krulbe.k...@gmail.com declaimed the following in gmane.comp.python.general: But maybe owner of this group do no care in that case we *all* get spammed! There is NO OWNER of comp.lang.python; and turning a comp.* group into moderated takes a fairly long time assuming you can find someone willing to be the moderators -- what would likely happen is that comp.lang.python.moderated would be created and then take a few months to be picked up by the servers, and a few more months for the real users to leave comp.lang.python to use it. Oh, and also the hassle of the mailing-listusenet gateway (does it pass traffic to both groups, drop the existing one, etc.). And readers on Google may still see spam if Google takes posts stuff before it passes through the moderation board. For the benefit of those who might have missed it, I'll repeat that I'm reading this from gmane.comp.python.general and see little or no spam. Regards. Mark Lawrence. -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Re: why is this group being spammed?
I subscribe for this mailing list at http://mail.python.org/mailman/listinfo/python-list On Sat, Jul 24, 2010 at 3:22 PM, Jia Hu huji...@gmail.com wrote: Hi, can I subscribe this by gmail? On Sat, Jul 24, 2010 at 3:16 PM, Mark Lawrence breamore...@yahoo.co.ukwrote: On 24/07/2010 18:01, Dennis Lee Bieber wrote: On Sat, 24 Jul 2010 07:32:30 -0700 (PDT), be.krulbe.k...@gmail.com declaimed the following in gmane.comp.python.general: But maybe owner of this group do no care in that case we *all* get spammed! There is NO OWNER of comp.lang.python; and turning a comp.* group into moderated takes a fairly long time assuming you can find someone willing to be the moderators -- what would likely happen is that comp.lang.python.moderated would be created and then take a few months to be picked up by the servers, and a few more months for the real users to leave comp.lang.python to use it. Oh, and also the hassle of the mailing-listusenet gateway (does it pass traffic to both groups, drop the existing one, etc.). And readers on Google may still see spam if Google takes posts stuff before it passes through the moderation board. For the benefit of those who might have missed it, I'll repeat that I'm reading this from gmane.comp.python.general and see little or no spam. Regards. Mark Lawrence. -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
numpy installation
Hello: I tried to install numpy 1.4.1 from source under ubuntu following instruction at http://docs.scipy.org/doc/numpy/user/install.html I type python setup.py build –help-fcompiler and it says gnu95 is found. Then I run python setup.py build –fcompiler=gnu95. There is error. Does anyone know how to fix it?? Thank you. -- http://mail.python.org/mailman/listinfo/python-list
Re: How to pass the shell in Python
sub = subprocess.Popen(shell command, shell=True) If you have to wait the shell finishes its commands and then continue the next Python code. You can add another line: sub.wait() On Tue, Jul 20, 2010 at 7:57 AM, S.Selvam s.selvams...@gmail.com wrote: On Tue, Jul 20, 2010 at 7:11 AM, Ranjith Kumar ranjitht...@gmail.comwrote: Hi Folks, Can anyone tell me how to run shell commands using python script. For simple work, i generally use os.system call. -- Regards, S.Selvam I am because we are -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Re: why is this group being spammed?
I use Gmail. When I receive spams, I will click Report Spam. In addition, I will not empty the spam box of my email immediately. When I receive about 25 spams, I will click Filter messages like these to filter all the spams and let gmail automatically delete them. On Mon, Jul 19, 2010 at 1:20 PM, John Bokma j...@castleamber.com wrote: be.krul be.k...@gmail.com writes: why is this group being spammed? Do you report those spammers? While Google is extremely lazy with dealing with spammers, if sufficient people report them action might be taken. Also make sure to report those spammers with their ISP; posts via GG contain the posting IP address. Even trolls can be hurt, if enough people report them: http://www.xahlee.org/Periodic_dosage_dir/t2/harassment.html -- John Bokma j3b Hacking Hiking in Mexico - http://johnbokma.com/ http://castleamber.com/ - Perl Python Development -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Re: why is this group being spammed?
Hi noboby: How can you make the current email and name hidden? On Mon, Jul 19, 2010 at 2:42 PM, Nobody nob...@nowhere.com wrote: On Sun, 18 Jul 2010 15:18:59 -0700, sturlamolden wrote: why is this group being spammed? There used to be bots that issued cancel messages against spam, but I don't think they are actively maintained anymore. Mostly because cancel messages are invariably ignored nowadays. -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
timing
Hello: If I want to calculate the runtime of a section of a program. How can I do it? Thank you, Jia -- http://mail.python.org/mailman/listinfo/python-list
Re: timing
Thank you, it is so straightforward. On Fri, Jul 16, 2010 at 9:58 PM, Chris Rebert c...@rebertia.com wrote: On Fri, Jul 16, 2010 at 5:52 PM, Jia Hu huji...@gmail.com wrote: Hello: If I want to calculate the runtime of a section of a program. How can I do it? Taking you extremely literally: from time import time start = time() run_section_here() end = time() runtime = end-start Assuming you're doing this in order to optimize: http://docs.python.org/library/profile.html Or in particularly simple cases: http://docs.python.org/library/timeit.html Cheers, Chris -- http://blog.rebertia.com -- http://mail.python.org/mailman/listinfo/python-list
Re: write a .txt file
Hi:
Do you mean the following code?
#!/usr/bin/python
# OS: Ubuntu
import subprocess
fileName = open ('final.txt', 'a')
fileName.write ('%s %s %s \n' % (12,25,9))
fileName.flush() # add
fileName.close() # add
desLrr = subprocess.Popen('ls -a final.txt', shell=True) # change to ls
-a
fileName=open ('final.txt', 'a')
fileName.seek(0,2)
fileName.write ('%s %s %s \n' % (85,25,12))
fileName.close()
I run that, the result showed the list of file is located after the number
list 85 25 12
Is that possible that I put the fileName.flush() in a wrong position ?
I am new to Python and do not quite understand the flush concept.
Thank you.
Jia
On Tue, Jul 13, 2010 at 2:16 AM, Cameron Simpson c...@zip.com.au wrote:
On 12Jul2010 21:28, Jia Hu huji...@gmail.com wrote:
| I have a problem about how to generate a specific.txt file. I use the
| following code:
|
| #!/usr/bin/python
| # OS: Ubuntu
| import subprocess
| fileName = open ('final.txt', 'a')
| fileName.write ('%s %s %s \n' % (12,25,9))
String still in Python's buffer, not yet in the file. Add:
fileName.flush()
to ensure data in file before running echo.
| desLrr = subprocess.Popen('echo hello final.txt ', shell=True)
Command dispatched, but not yet waited for. So your Python program
proceeds
and writes to the file _before_ the echo command gets to run.
Wait for desLrr to complete before proceeding.
| fileName.seek(0,2)
| fileName.write ('%s %s %s \n' % (85,25,12))
| fileName.close()
And the rest is ok.
Cheers,
--
Cameron Simpson c...@zip.com.au DoD#743
http://www.cskk.ezoshosting.com/cs/
We tend to overestimate the short-term impact of technological change and
underestimate its long-term impact. - Amara's Law
--
http://mail.python.org/mailman/listinfo/python-list
Re: write a .txt file
On Tue, Jul 13, 2010 at 3:17 AM, Cameron Simpson c...@zip.com.au wrote:
On 13Jul2010 02:46, Jia Hu huji...@gmail.com wrote:
| Hi:
|
| Do you mean the following code?
|
| #!/usr/bin/python
| # OS: Ubuntu
| import subprocess
| fileName = open ('final.txt', 'a')
| fileName.write ('%s %s %s \n' % (12,25,9))
|
| fileName.flush() # add
| fileName.close() # add
You should not need the .close().
Thank you.
| desLrr = subprocess.Popen('ls -a final.txt', shell=True) # change to
ls
| -a
You're still not waiting for the Popen subprocess to finish before
continuing.
| fileName=open ('final.txt', 'a')
You shouldn't need the open() if you skip the .close()
| fileName.seek(0,2)
If you _do_ do an open(..., a) then you don't need the seek - append
mode will write at the end for you.
| fileName.write ('%s %s %s \n' % (85,25,12))
| fileName.close()
|
| I run that, the result showed the list of file is located after the
number
| list 85 25 12
| Is that possible that I put the fileName.flush() in a wrong position ?
| I am new to Python and do not quite understand the flush concept.
First: please don't top post. Post below the relevant points as I do
above, like a conversation. That way it is clear exactly what each
remark is for.
OK, it is a good suggestion. By the way, do you add the pipe | or
they are automatically generated ?
Second: flush is a general idea used with buffers.
When you .write() to a file the data does not normally go directly to
disc. This is because a real write to the disc is expensive in time
(the CPU is much much faster than a hard drive).
Also, a OS-level write, which transfers the data from your program into
the OS's buffers (where it queues, for eventual writing to the disc)
is _also_ a comparitively expensive operation. So when you .write()
in python (or C using stdio, and in many other languages) the data is
normally kept in a memory buffer inside your program. Only when that
buffer is filled is an OS-level write done. In this way, OS calls are
few. This is a performance gain.
So, in your program, when you .write() your first line of numbers
the data has not been handed to the OS at all, and therefore the
final.txt file has not seen it. A .flush() call is an explicit request
to empty the buffer, making an OS-level write immediately. In your
program, this is necessary to get the data to final.txt before the
echo or ls commands are run.
If you .close() a file, that also empties the buffer. But it also closes
the file! SO you need to re-open it. For your purposes, a .flush() is
enough and leaves the file open for you to continue using it later.
Regarding the .seek(): after the echo or ls command has run
the file has grown. Python's file object does not know the file has
grown because it was not involved. So a .seek(0 is needed to position
Python's .write() location to the end of the file instead of where it
though things were.
Um. Because you have opened the file in 'a' mode you should not need the
.seek() - an append mode file will always write to the end of the
file.
So: your first problem only requires a .flush(), to empty the buffer
before echo or ls runs. However, you still have not waited for the
echo or ls to finish - Popen kicks the command off, but it will run
at the same time as your program! Call:
desLrr.wait()
I initially thought only desLrr.wait() is needed even if without
fileName.flush() , that is
because the first line of number will transfer to the memory buffer and then
echo, ls is
also kept in the memory buffer and then wait() to make them finish. But when
I actually did this,
I find this method is not very correct. The first few words generated
by ls are missing.
When I add fileName.flush() again and got a correct output.
Should I always write to the file at one time and then run flush or
close() before the next
write to the file (e.g. echo or the second line of number) ?
Thank you.
to wait for the echo or ls to finish before continuing!
Cheers,
--
Cameron Simpson c...@zip.com.au DoD#743
http://www.cskk.ezoshosting.com/cs/
Quick Guide to Resources to break crypto systems of given strength:
Difficulty Resources required
2**0 Pencil and paper
2**8 Pencil and paper and a lot of patience
2**16 Warez d00d with a Commodore 64
2**32 Amateur with an average PC
2**40 Smart amateur with a good PC
2**56 Network of workstations, custom-built hardware
2**64 Thousands of PCs working together for several years
2**80 NSA, Microsoft, the Illuminati
2**128Cubic kilometers of sci-fi nanotech
2**160Dyson spheres
Eddy L O Jansson and Matthew Skala, from:
http://hem.passagen.se/eddy1/reveng/cp4/cp4break.html
--
http://mail.python.org/mailman/listinfo/python-list
how to delete \n
Hi, I just want to delete \n at each line. My operating system is ubuntu
9.1. The code is as follows
#!/usr/bin/python
import string
fileName=open('Direct_Irr.txt', 'r') # read file
directIrr = fileName.readlines()
fileName.close()
for line in directIrr:
line.rstrip('\n')
print directIrr
But I found there is still \n . Could someone help me why it is not
correct?
Thank you
--
http://mail.python.org/mailman/listinfo/python-list
Re: how to delete \n
Thank you. It works now.
if I use 'print' to print the whole list, 'print' will add newline
at the end of the list but not each item in the list. right?
For the code:
for line in fileName:
line = line.rstrip('\n')
I think this will affect 'fileName' because it assign the value to 'line' ?
But when I print fileName, \n still exists at each item in the list.
Because each line in my txt file are numeric values. is there any other
better way to get each
numerical value at each line? for example using numpy or changing string
list to numerical list?
Thank you for help.
On Mon, Jul 12, 2010 at 4:45 PM, Chris Rebert c...@rebertia.com wrote:
On Mon, Jul 12, 2010 at 1:27 PM, Jia Hu huji...@gmail.com wrote:
Hi, I just want to delete \n at each line. My operating system is
ubuntu
9.1. The code is as follows
#!/usr/bin/python
import string
fileName=open('Direct_Irr.txt', 'r') # read file
directIrr = fileName.readlines()
fileName.close()
for line in directIrr:
line.rstrip('\n')
print directIrr
But I found there is still \n . Could someone help me why it is not
correct?
.rstrip() returns a *new* string without trailing whitespace (which
you are currently then throwing away); it does *not* modify string
objects in-place. Python strings objects are entirely immutable and
unmodifiable; all operations on them merely produce /new/ strings.
Assuming you still want to use .readlines(), you'd do:
directIrr = fileName.readlines()
fileName.close()
directIrr = [line.rstrip('\n') for line in directIrr]
print directIrr
For how third line works, google python list comprehensions.
Cheers,
Chris
--
http://blog.rebertia.com
--
http://mail.python.org/mailman/listinfo/python-list
write a .txt file
Hello:
I have a problem about how to generate a specific.txt file. I use the
following code:
#!/usr/bin/python
# OS: Ubuntu
import subprocess
fileName = open ('final.txt', 'a')
fileName.write ('%s %s %s \n' % (12,25,9))
desLrr = subprocess.Popen('echo hello final.txt ', shell=True)
fileName.seek(0,2)
fileName.write ('%s %s %s \n' % (85,25,12))
fileName.close()
The above code generates the following result:
12 25 9
85 25 12
hello
What I want is:
12 25 9
hello
85 25 12
Could someone provies some suggestions about this problem?
Thank you.
--
http://mail.python.org/mailman/listinfo/python-list
Re: write a .txt file
If I change 'echo hello final.txt to ls -a final.txt and add
fileName.close() before subprocess.
The result is still like:
12 25 9
85 25 12
# list of files
.
How can I put the list before the number list 85 25 12?
Thank you.
On Tue, Jul 13, 2010 at 12:47 AM, Jia Hu huji...@gmail.com wrote:
Thank you. I will try that.
The code was simplified to show my problem. I do not use fileName as
a variable in my code. Thanks for your reminding.
On Mon, Jul 12, 2010 at 9:58 PM, MRAB pyt...@mrabarnett.plus.comwrote:
Jia Hu wrote:
Hello:
I have a problem about how to generate a specific.txt file. I use the
following code:
#!/usr/bin/python
# OS: Ubuntu
import subprocess
fileName = open ('final.txt', 'a')
fileName.write ('%s %s %s \n' % (12,25,9))
desLrr = subprocess.Popen('echo hello final.txt ', shell=True)
fileName.seek(0,2)
fileName.write ('%s %s %s \n' % (85,25,12))
fileName.close()
The above code generates the following result:
12 25 9
85 25 12
hello
What I want is:
12 25 9
hello
85 25 12
Could someone provies some suggestions about this problem?
Thank you.
When echo is called the file is still open, so the might not be able
to write to it. Try closing the file before starting the subprocess and
then reopen it afterwards.
By the way, it slightly confusing that you call the variable fileName,
but it doesn't contain the name of a file! :-)
--
http://mail.python.org/mailman/listinfo/python-list
--
http://mail.python.org/mailman/listinfo/python-list
