Re: [Tutor] Appending an extra column in a data file
Hi Sayan, > Is it that? We can have different plots and hence animation from a single > data file.Very good!...if you have some time in your disposal, it will be > kind of you to illumine in this matter...may be some useful link will do. > > I have written this small code from scratch,there's no 'get it to work' > business...:) > > Here is the total code...not attaching as you might have problem opening > random files from internet:) I will leave the complete code snippet for the list memory. A couple things. With the code you have now, you do not even have to save any files. You can simply implant the plotting into the initial while loop. Right now, your saved files serve only as a temporary holding file for the data you had just created. Since your initial data is not read from another file, but simply generated in your code, there is no reason to write any thing to disk, except the figures you are saving. One thing to notice, your "t" and "i" while loops iterate over different counters. You have t = t + 0.01 i = i + 0.001 so your second loop will produce many calls that don't match anything (the files you created don't exist). The previous error is because the pp_za and pv_za are already 1D numpy arrays of dimension 999. That is why combining them with a single float, "np.array([pp_za,pv_za,t])", did not work - the dimension of "pp_za" and "t" don't match, so writing them with numpy.savetxt() gave an error. How do you want to include the "t" information in the figure? Do you want to make different plots with a time axis? Or, it sounds like you are talking about a 3D plot? If you want to store the time with the data, then make a t_array like so ''' inside your t while loop ''' # make an array filled with ones the same dimension as your pp_init, and then multiply each entry by "t" t_array = t * numpy.ones_like(pp_init) ''' then pack your data as ''' np.array([pp_za,pv_za,t_array]) but to have a single data file containing all of this: t_array = np.arange(0,1,.001) # initialize all the times you care about pp_init = np.linspace(0,1.99799799,999) #just taking from your code # initialize an array of zeros # first dimension spans time # second dimension is 3 to store value of pp_za, pv_za, t # third dimension is actual length of data all_data = np.zeros([len(t_array),3,len(pp_init)]) for i,t in enumerate(t_array): # make your pp_za and pv_za as in your code t_tmp = t * numpy.ones_like(pp_init) all_data[i] = np.array([pp_za,pv_za,t]) ''' now you have a single object (np.array) which contains all of your data. You could either make each of the figures in the above loop, or you can use a second loop to make them. I see Oscar has given you some advice on the plotting - so you could use this single object and his example to do what you want. Andre ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Appending an extra column in a data file
Hi Sayan, > Yup it's exactly what I want! > > I want many data files,not one...to make an animation out of it. For a data > file t is constant. You should not need many data files to make an animation. If you write your loops correctly, you can take a single data file with all the data. If you are using code you did not write, and just want to get it to work, I understand that, but would encourage you also to figure out how to do it all with single data file. Less files makes a files system and user happier. > the solution you have just mentioned i.e np.array([t,pp_za,pv_za]) is giving > the following error: > > > Traceback (most recent call last): > File "ZA_Phase_Plot.py", line 38, in > np.savetxt(fname, np.array([pp_za,pv_za,t]).T, '%f') > File "/usr/lib/python2.7/dist-packages/numpy/lib/npyio.py", line 979, in > savetxt > fh.write(asbytes(format % tuple(row) + newline)) > TypeError: float argument required, not numpy.ndarray > > What to do? :-| There are pieces of the code snipet you sent which are not defined, so it is not clear why you are getting this error. How long is the file "ZA_Phase_Plot.py"? At least you can send from line 0 to where you did before, so all the variables are defined. That should help. Andre ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Appending an extra column in a data file
Hi Sayan, > Well,this is the concerned snippet of the code: > > > while t < 1: > > > pp_za = pp_init + t*K*np.sin(K*pp_init) > > # Periodic Boundary Condition > > for i in range(0,999): > > if pp_za[i] < 0: > pp_za[i] = 2 - abs(pp_za[i]) > if pp_za[i] > 2: > pp_za[i] = pp_za[i] % 2 > > pv_za = +K*np.sin(K*pp_init) > > fname = 'file_' + str(t) + '.dat' > > # Generating dataset for Phase Space diagram > > np.savetxt(fname, np.array([pp_za,pv_za]).T, '%f') > > t = t + 0.01 To answer your question, to add "t" to the array, you can simply replace the current np.array command with either of the following np.array([pp_za,pv_za,t]) or np.array([t,pp_za,pv_za]) depending if you want "t" is the first element of the data or the last. Also, you are "transposing" the array with the ".T", so this will write the data in sequential lines, rather than a single line (row) with multiple entries. I just bring it up because it is not the "obvious" way to pack the data. Also, it looks like you are generating a separate file for each "t". Is this what you want? I assume you want a single data file that stores this info, not many data files. Andre > > And the sample data generated is : > > 0.105728 0.098678 > 0.126865 0.118406 > 0.147998 0.138128 > 0.169126 0.157845 > 0.190247 0.177556 > 0.211362 0.197259 > 0.232469 0.216955 > 0.253567 0.236643 > 0.274657 0.256321 > 0.295737 0.275989 > 0.316806 0.295646 > > Precisely what I want is, I want to add the 't' (over which the while loop is > run) in the 3 rd column. 't' is float and constant for each looping. If you > know what a phase plot is(doesn't matter if you don't), I want to see the > phase plot evolving with time. Thus the need of a time axis and hence plot > with V(velocity-Yaxis), X ( Position -X axis) and t (Time-Z axis). > > I hope your first example might help.Though,I think,I need some tweaking to > fit to my needs. > > Regards, > Sayan > > > On 10 April 2013 22:38, Andre' Walker-Loud wrote: > Hi Sayan, > > > Thank Andre for your prompt answer. > > No problem. > > > I'll figure out the plotting issue once the dat files are made. So it's the > > primary concern. > > For an example I am attaching a dat file herewith. The two columns here are > > 2 numpy arrays.I want to add a third column, to be precise, I want to print > > a parameter value on the third column of the file. > > Let me try again. > The reason the matplotlib list would be a better place is this is a general > python list, and most people are not familiar with numpy. However, most > people who use matplotlib are familiar with numpy. > > I am hoping you can describe precisely the structure of the data. Maybe show > a little code on how it is created, or how you access it. I am not keen to > open "random" files from the internet. As two examples of how I think your > code might be packed > > 1/ > ''' > x = numpy.zeros([10]) # 1D numpy array of dimension 10 > y = numpy.zeros([10]) # 1D numpy array of dimension 10 > your_data = [] > your_data.append(x) > your_data.append(y) > ''' > > so now your data is a table with two entries, and each entry is a numpy array. > You have in mind adding a third entry to the table with just floats. > > 2/ > ''' > your_data = numpy.zeros([10,10]) # initialize a 2D numpy array with all zeros > for i in range(your_data.shape[0]): > for j in range(your_data.shape[1]): > your_data[i,j] = data[i][j] # I am assuming the data is imported > already and called data and is in a python list/table format > ''' > > Now you want to make a new column or row for your data file, which contains > floats. Well, all the entries inside the numpy array are already floats, so > it is not clear to me why you want a new column that is not a numpy array. So > it would be nice if you could precisely describe what you currently have and > what you want. > > > Hope this helps, > > Andre > > > > > > -- > > > -- > Sayan Chatterjee > Dept. of Physics and Meteorology > IIT Kharagpur > Lal Bahadur Shastry Hall of Residence > Room AB 205 > Mob: +91 9874513565 > blog: www.blissprofound.blogspot.com > > Volunteer , Padakshep > www.padakshep.org ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Appending an extra column in a data file
Hi Sayan, > Thank Andre for your prompt answer. No problem. > I'll figure out the plotting issue once the dat files are made. So it's the > primary concern. > For an example I am attaching a dat file herewith. The two columns here are 2 > numpy arrays.I want to add a third column, to be precise, I want to print a > parameter value on the third column of the file. Let me try again. The reason the matplotlib list would be a better place is this is a general python list, and most people are not familiar with numpy. However, most people who use matplotlib are familiar with numpy. I am hoping you can describe precisely the structure of the data. Maybe show a little code on how it is created, or how you access it. I am not keen to open "random" files from the internet. As two examples of how I think your code might be packed 1/ ''' x = numpy.zeros([10]) # 1D numpy array of dimension 10 y = numpy.zeros([10]) # 1D numpy array of dimension 10 your_data = [] your_data.append(x) your_data.append(y) ''' so now your data is a table with two entries, and each entry is a numpy array. You have in mind adding a third entry to the table with just floats. 2/ ''' your_data = numpy.zeros([10,10]) # initialize a 2D numpy array with all zeros for i in range(your_data.shape[0]): for j in range(your_data.shape[1]): your_data[i,j] = data[i][j] # I am assuming the data is imported already and called data and is in a python list/table format ''' Now you want to make a new column or row for your data file, which contains floats. Well, all the entries inside the numpy array are already floats, so it is not clear to me why you want a new column that is not a numpy array. So it would be nice if you could precisely describe what you currently have and what you want. Hope this helps, Andre ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Appending an extra column in a data file
Hi Sayan, This question would be better suited to the matplotlib list. Also, a more precise description of your existing data, and what you hope it would look like would make it easier to help answer your question. Eg., from your description, it is not clear if your existing data is in a table, and in the table you have numpy arrays, or if you have one numpy array of rank 2 with data in it. Regards, Andre On Apr 10, 2013, at 8:44 AM, Sayan Chatterjee wrote: > Dear All, > > I have some data files with numpy arrays stored in it in 2 columns. I want to > add a third column with just floating point numbers(not numpy array) and plot > them later with Matplotlib in 3d. How is it done? Could you please illuminate > me? > > Bests, > Sayan > > -- > > > -- > Sayan Chatterjee > Dept. of Physics and Meteorology > IIT Kharagpur > Lal Bahadur Shastry Hall of Residence > Room AB 205 > Mob: +91 9874513565 > blog: www.blissprofound.blogspot.com > > Volunteer , Padakshep > www.padakshep.org > ___ > Tutor maillist - Tutor@python.org > To unsubscribe or change subscription options: > http://mail.python.org/mailman/listinfo/tutor ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] release a script with non-standard dependencies
Hi Rail, > I'm attempting to write a script with command-line arguments. Based on some > reading I have done online and some advice from this mailing list I'm going > to use docopt and schema modules. The problem I'm facing is that I'd like to > be able to give this script to anyone who needs it by just using one file, > i.e. the script itself. Does anybody know if there is a way to achieve this > without pasting code from docopt and schema into my script file. > > Any help is appreciated. I would like to know the answer to your question as well. Hopefully, the people you work with are comfortable installing something like docopt - but I know that is often not the case. If you are unsuccessful, and consider going back to OptParse, I just wanted to point you to using Argparse instead. It is a bit better, and more importantly, is not deprecated (Optparse is no longer updated). However, it shares the same issues the creator of docopt railed against in his presentation that Modulok sent a link to earlier. Regards, Andre ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] data analysis with python
Hi David, > I'm trying to use python for analysing data from building energy simulations > and was wondering whether there is way to do this without using anything sql > like. > > The simulations are typically run for a full year, every hour, i.e. there are > 8760 rows and about 100+ variables such as external air temperature, internal > air temperature, humidity, heating load, ... making roughly a million data > points. I've got the data in a csv file and also managed to write it in a > sqlite db. > > I would like to make requests like the following: > > Show the number of hours the aircon is running at 10%, 20%, ..., 100% > Show me the average, min, max air temperature, humidity, solar gains, > when the aircon is running at 10%, 20%,...,100% > > Eventually I'd also like to generate an automated html or pdf report with > graphs. Creating graphs is actually somewhat essential. > > I tried sql and find it horrible, error prone, too much to write, the logic > somehow seems to work different than my brain and I couldn't find particulary > good documentation (particulary the documentation of the api is terrible, in > my humble opinion). I heard about zope db which might be an alternative. > Would you mind pointing me towards an appropriate way to solve my problem? Is > there a way for me to avoid having to learn sql or am I doomed? I would recommend learning hdf5http://www.hdfgroup.org/HDF5/ and the python utility to interface with it pytableshttp://www.pytables.org/moin and numpy and scipy are great for data analysis (python libraries) - numpy handles things like linear algebra, scipy has many built in scientific functions. And then matplotlib for plotting (very similar functions to matlab if you are familiar with it). Lastly, a very nice interface is "iPython", which is basically an enhanced python interpreter designed for/by science types. All of these tools are installed for you with the Enthought Python Distribution (full dist is free if you have a .edu address, otherwise they provide a light version with basic libraries, and you can install others you like) http://www.enthought.com/ If you have any specific questions on these (I know that is a lot to look into right away) let me know. Cheers, Andre ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
[Tutor] good coding habits/practices
Hi All, I have a general question. I have never taken a formal programming course. I have learned everything I know from office mates (earlier on) and now mostly Google, as I need it. Most everything I write is only for my own consumption. I write scripts/programs to do science research, and have never manage to make the time to take a proper class, or even work through a good python tutorial. On a matplotlib thread, someone pointed us to a PyCon talk posted on YouTube [for those interested - link below] http://www.youtube.com/watch?v=pXhcPJK5cMc This particular talk focussed on argument passing in python. I found it very enjoyable and educational. One thing I learned - I am really not aware of the coding standards out there. For example, I was not aware precisely of the argument passing standards discussed in the above talk. However, I am aware that if I were aware of these things, my ability to do my research would be improved. Especially, just being more familiar with proper programming. Also, just because of personal principles, even if my code is only ever really consumed by me, I would like it to meet higher standards of "good code" than it currently does. However, since I am still hunting for a permanent job, I can not take the time for a full proper course on programming [I still need to use the learn as I need it model]. So finally the question: is there a good SUCCINCT guide to things like the POSIX standards, and other programming standards {IEEE ... I believe), and a guide to good programming practices that I can peruse on the web to try and improve my programming skills as I work? Also - recommended (shorter) tutorials would be nice. With my current work schedule, things that can be worked through on the order of an hour or so [each lesson that is]. Thanks, Andre ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] html checker
Dear Matthew, > I don't need to hear how bad my programs are...either you are gonna help or > your not...if you have questions about what i have wrote or why i wrote > something someway ask...dont just jump to conclusions > > I forgot to include that i had to write a "stack" function in a "pythonbasic" > file to import > http://pastie.org/4892792 > > Im here because i want to learn not to be fucking bashed...a few of the > functions were copied from my book as they were required for other parts of > the lab...so again if you have a problem with those other functions maybe you > should take it up with the author... > > anyway i just don't know where to start to get the htmlChecker function to > run...i am not a programmer by any means but is required to learn for my > major...this is why im stuggling...i have networking experience not > programming Bob was not trying to bash you. You clearly must be frustrated with your program, as your response was very intense and negative. Try not to project your frustration with your program out on the people who are trying to help you. Recall, those responding to the list are purely volunteers - they do what they do because they enjoy helping others learn about python - or feel like giving back to new people learning since they received help from this list and others. In addition to helping you with the programming, the senior responders also try to teach you how to think about your problems better, including what information to include in such emails, so that they can help you. Bob was pointing out that you have not included enough information in your original post for those on the list to help you. May I suggest that you relieve your frustration on something local, and then try and see what information Bob was asking for that you can provide, so that people on the list can provide you with guidance and help. Cheers, Andre ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] OT: Netiquette
To the python tutor list, One comment I would like to make in light of the current email exchanges is that generally speaking, one of the things I have greatly appreciated about the python user community is how friendly and helpful they have been. This particularly strikes me when I see time and again, new users ask the same old questions in their own way, and time and again, the same very helpful experienced users reply with helpful comments - they are clearly committed to trying to help people understand python (and programming), generally, with great patience. I hope that new subscribers to the list do not go away with the thought that bickering is common - quite the contrary, as I stated above, I have found the python tutor list (and other python lists I subscribe to) mostly free of this problem. Of course, in any community, problems like the current one are unavoidable. People bring with them their own communication styles, and when communication is solely restricted to electronic forms, we are bound to misinterpret others meaning and intentions, and sometimes that gets the better of us. Also, as most of us know from experience, sometimes a good fight is what is needed, to clear the air - especially true with "family". Regards, Andre On Sep 26, 2012, at 3:59 AM, Walter Prins wrote: > Dwight, > > On 26 September 2012 09:26, Dwight Hutto wrote: >> The only face I personally want to see of him >>> because of this is his back. >>> >> >> You wanna see my ass, because that's what you want homo. Butt just >> look, you can't touch. > > The personal attacks and innuendo are really not acceptable and you're > apparently deliberately twisting/misinterpreting Mark's words there. > Wy out of line and quite disingenuous. Would you respond so > aggressively to people in person? No? Well why do you think it's OK > to be abusive on the internet? (If you do think it's OK to be this > abusive to people in person, then you're sadly mistaken.) Grow up. > Walk away. Learn to be polite to people you don't know. This is not > the school playground and you're not 5 years old. Do some careful > introspection. > > Walter > ___ > Tutor maillist - Tutor@python.org > To unsubscribe or change subscription options: > http://mail.python.org/mailman/listinfo/tutor ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] np array.any() question
> To summarize, > with a numpy array called a, the OP is getting an error doing: > > (3 < a < 5).any() > >> ... > > I think not. The last expression you give will even return true if one > of the values is > 3 and a DIFFERENT value is < 5. And i suspect the > OP only wants a TRUE if at least one item in the array is between 3 and 5. > > I know nothing about numpy, but does it have a way to do an element-wise > AND of two bool arrays of the same size? Maybe it's a function call and > not 'and' Or maybe it's the & operator or something. Yes, numpy does have a function like this: numpy.all() [ "and" function ] numpy.any() [ "or" function ] > I find the operator overloading i've seen in numpy to be very confusing, > so I haven't tried to download it and try it out. Maybe if I read the > docs directly, instead of just seeing examples and problems here, I'd > think differently. here you go :) http://docs.scipy.org/doc/numpy/reference/generated/numpy.all.html#numpy.all http://docs.scipy.org/doc/numpy/reference/generated/numpy.any.html#numpy.any you can ask if any satisfy the conditions and return true if 1+ are true: >>> a = numpy.array([ 7., 1., 1., 1., 2., 4., 0., 7., 6., >>> 10.]) >>> numpy.any([ 3 < a, a < 5]) True >>> numpy.all([ 3 < a, a < 5]) False or you can ask for an array back telling you which are true and which false: >>> numpy.any([ 3 < a, a < 5], axis=0) array([ True, True, True, True, True, True, True, True, True, True], dtype=bool) >>> numpy.any([ 3 < a, a < 5], axis=0) array([False, False, False, False, False, True, False, False, False, False], dtype=bool) Note you are not limited to just comparing two arrays, but can compare as many as you like. To the OP: in this case, google (insert whatever search engine you like) is your friend. A good first bet is if there is some operation you like in python, other people like it also, and someone has taken the time to make a numpy version which acts element-wise on equal sized arrays. Andre ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Why isn't my simple if elif script not working?
> > On Jul 17, 2012, at 10:29 PM, Alexandre Zani wrote: > > > On Tue, Jul 17, 2012 at 10:21 PM, Andre' Walker-Loud > wrote: >> Hi Santosh, >> >> On Jul 17, 2012, at 10:09 PM, Santosh Kumar wrote: >> >>> Here is my script: >>> >>> name = raw_input("What's your name? ") >>> >>> if name == "Santosh": >>> print "Hey!! I have the same name." >>> elif name == "John Cleese" or "Michael Palin": >>> print "I have no preference about your name. Really!!" >>> else: >>> print "You have a nice name." >>> >>> >>> The if part works well. The elif part works well too. The problem is >>> even if you enter strings other than "Santosh", "John Cleese" and >>> "Michael Palin" you still get the print from elif part, not from else >>> part. >> >> you just have to be careful with the multiple boolean line in the elif. You >> can use either >> >> elif name == ("John Cleese" or "Michael Palin"): > > That won't work. > >>>> "John Cleese" == ("John Cleese" or "Michael Palin") > True >>>> "Michael Palin" == ("John Cleese" or "Michael Palin") > False >>>> ("John Cleese" or "Michael Palin") > 'John Cleese' > > Python will look at the expression ("John Cleese" or "Michael Palin") > since bool("John Cleese") is True, the expression immediately > evaluates to "John Cleese" and the elif clause becomes equivalent to > name == "John Cleese" thanks - I fell into the same trap! I tried it in my terminal but it worked for the same reason as the OP. Andre ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Why isn't my simple if elif script not working?
Hi Santosh, On Jul 17, 2012, at 10:09 PM, Santosh Kumar wrote: > Here is my script: > > name = raw_input("What's your name? ") > > if name == "Santosh": >print "Hey!! I have the same name." > elif name == "John Cleese" or "Michael Palin": >print "I have no preference about your name. Really!!" > else: >print "You have a nice name." > > > The if part works well. The elif part works well too. The problem is > even if you enter strings other than "Santosh", "John Cleese" and > "Michael Palin" you still get the print from elif part, not from else > part. you just have to be careful with the multiple boolean line in the elif. You can use either elif name == ("John Cleese" or "Michael Palin"): or elif name == "John Cleese" or name == "Michael Palin": With your elif line, it is asking "does name == John Cleese" or "Michael Palin", and so if the name is not John Cleese, then I believe it prints "Michael Palin" while reporting True, so satisfying the elif clause. Cheers, Andre ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] summary stats grouped by month year
dear anonymous questioner, > Excellent, thank you so much. I don't understand all the steps at this stage > so I will need some time to go through it carefully but it works perfectly. > Thanks again! words of caution - you will notice that the way I constructed the data file - I assumed what the input file would look like (they come chronologically and all data are present - no missing years or months). While this might be safe for a data file you constructed, and one that is short enough to read - this is generally a very bad habit - hence my encouraging you to figure out how to make dictionaries. Imagine how you would read a file that you got from a colleague, which was so long you can not look by eye and see that it is intact, or perhaps you know that in 1983, the month of June is missing as well as some other holes in the data. And perhaps your colleague decided, since those data are missing, just don't write the data to the file, so instead of having 1983 May 2.780009889 1983 June nan 1983 July 0.138150181 you have 1983 May 2.780009889 1983 July 0.138150181 now the little loop I showed you will fail to place the data in the correct place in your numpy array, and you will get all your averaging and other analysis wrong. Instead - you can create dictionaries for the years and months. Then when you read in the data, you can grab this info to correctly place it in the right spot # years and months are your dictionaries years = {'1972':0,'1973':1,} months = {'Jan':0,'Feb':1,...,'Dec':11} data = open(your_data).readlines() for line in data: year,month,dat = line.split() y = int(('%('+year+')s') % years) m = int(('%('+month+')s') % months) rain_fall[y,m] = float(dat) [also - if someone knows how to use the dictionaries more appropriately here - please chime in] then you also have to think about what to do with the empty data sets. The initialization rain_fall = np.zeros([n_years,n_months]) will have placed zeros everywhere - and if the data is missing - it won't get re-written. So that will make your analysis bogus also - so you have to walk through and replace the zeros with something else, like 'nan'. And then you could think about replacing missing data by averages - eg. replace a missing June entry by the average over all the non-zero June data. I was just hoping to give you a working example that you could use to make a functioning well thought out example that can handle the exceptions which will arise (like missing data, or a data file with a string where a float should be etc') Have fun! Andre On May 8, 2012, at 5:48 PM, questions anon wrote: > On Tue, May 8, 2012 at 4:41 PM, Andre' Walker-Loud > wrote: > Hello anonymous questioner, > > first comment - you may want to look into hdf5 data structures > > http://www.hdfgroup.org/HDF5/ > > and the python tools to play with them > > pytables - http://www.pytables.org/moin > h5py - http://code.google.com/p/h5py/ > > I have personally used pytables more - but not for any good reason. If you > happen to have the Enthought python distribution - these come with the > package, as well as an installation of hdf5 > > hdf5 is a very nice file format for storing large amounts of data (binary) > with descriptive meta-data. Also, numpy plays very nice with hdf5. Given > all your questions here, I suspect you would benefit from learning about > these and learning to play with them. > > Now to your specific question. > > > I would like to calculate summary statistics of rainfall based on year and > > month. > > I have the data in a text file (although could put in any format if it > > helps) extending over approx 40 years: > > YEAR MONTHMeanRain > > 1972 Jan12.7083199 > > 1972 Feb14.17007142 > > 1972 Mar14.5659302 > > 1972 Apr1.508517302 > > 1972 May2.780009889 > > 1972 Jun1.609619287 > > 1972 Jul0.138150181 > > 1972 Aug0.214346148 > > 1972 Sep1.322102228 > > > > I would like to be able to calculate the total rain annually: > > > > YEAR Annualrainfall > > 1972400 > > 1973300 > > 1974350 > > > > 2011 400 > > > > and also the monthly mean rainfall for all years: > > > > YEAR MonthlyMeanRain > > Jan 13 > > Feb 15 > > Mar 8 > > . > > Dec 13 > > > > > > Is this something I can easily do? > > Yes - this should be very easy. Imagine importing all this data into a numpy > array > > === > import numpy as np > > data = open(your_data).readlin
Re: [Tutor] summary stats grouped by month year
Hello anonymous questioner, first comment - you may want to look into hdf5 data structures http://www.hdfgroup.org/HDF5/ and the python tools to play with them pytables - http://www.pytables.org/moin h5py - http://code.google.com/p/h5py/ I have personally used pytables more - but not for any good reason. If you happen to have the Enthought python distribution - these come with the package, as well as an installation of hdf5 hdf5 is a very nice file format for storing large amounts of data (binary) with descriptive meta-data. Also, numpy plays very nice with hdf5. Given all your questions here, I suspect you would benefit from learning about these and learning to play with them. Now to your specific question. > I would like to calculate summary statistics of rainfall based on year and > month. > I have the data in a text file (although could put in any format if it helps) > extending over approx 40 years: > YEAR MONTHMeanRain > 1972 Jan12.7083199 > 1972 Feb14.17007142 > 1972 Mar14.5659302 > 1972 Apr1.508517302 > 1972 May2.780009889 > 1972 Jun1.609619287 > 1972 Jul0.138150181 > 1972 Aug0.214346148 > 1972 Sep1.322102228 > > I would like to be able to calculate the total rain annually: > > YEAR Annualrainfall > 1972400 > 1973300 > 1974350 > > 2011 400 > > and also the monthly mean rainfall for all years: > > YEAR MonthlyMeanRain > Jan 13 > Feb 15 > Mar 8 > . > Dec 13 > > > Is this something I can easily do? Yes - this should be very easy. Imagine importing all this data into a numpy array === import numpy as np data = open(your_data).readlines() years = [] for line in data: if line.split()[0] not in years: years.append(line.split()[0]) months = ['Jan','Feb',,'Dec'] rain_fall = np.zeros([len(n_year),len(months)]) for y,year in enumerate(years): for m,month in enumerate(months): rain_fall[y,m] = float(data[ y * 12 + m].split()[2]) # to get average per year - average over months - axis=1 print np.mean(rain_fall,axis=1) # to get average per month - average over years - axis=0 print np.mean(rain_fall,axis=0) === now you should imagine doing this by setting up dictionaries, so that you can request an average for year 1972 or for month March. That is why I used the enumerate function before to walk the indices - so that you can imagine building the dictionary simultaneously. years = {'1972':0, '1973':1, } months = {'Jan':0,'Feb':1,...'Dec':11} then you can access and store the data to the array using these dictionaries. print rain_fall[int('%(1984)s' % years), int('%(March)s' % months)] Andre > I have started by simply importing the text file but data is not represented > as time so that is probably my first problem and then I am not sure how to > group them by month/year. > > textfile=r"textfile.txt" > f=np.genfromtxt(textfile,skip_header=1) > > Any feedback will be greatly appreciated. > > ___ > Tutor maillist - Tutor@python.org > To unsubscribe or change subscription options: > http://mail.python.org/mailman/listinfo/tutor ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] How to exit this loop in the interpreter
Hi Greg, > I'm new to python, is schooling a python-developer thing? Whether I ask for > help at forums, IRC or here, someone schools me. Should I stick with Java? No, the python mailing list is for the most part, very friendly and helpful (in my experience). Mark's message that you not hijack the thread is so that other people in the future, when trying to find similar problems to yours (or this thread) do not get confused by disjoined topics in the same thread. So instead of replying all to a random email from the group, start a new thread by posting an original email to the list. And if you don't get a reply immediately, be patient. Many of the people who may know the answer to your questions may live in a time-zone 12 hours different from yours. My interpretation of Mark's message was a jest-full way of reminding you (letting you know) appropriate behavior for the list. Further information on list behavior/rules can be found starting with the link sent at the bottom of every mail from the list http://mail.python.org/mailman/listinfo/tutor Certainly a way to discourage people from replying to your questions are - steal a thread - when asked not to hijack a thread, continue to do so anyways - "spam" the list with the same question multiple times in a short period of time Regards, Andre ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] finding a maximum between the absolute difference of several columns
> import numpy as np >> Vhel_fdiff3 = np.array([abs(Vmatch3_1 - Vmatch3_2), abs(Vmatch3_1 - >> Vmatch3_3), abs(Vmatch3_3 - Vmatch3_2)]) >> your_answer = Vhel_fdiff3.max(axis=0) > > or > > import numpy as np > a = np.array([Vmatch3_1-Vmatch3_2, Vmatch3_1-Vmatch3_3, Vmatch3_3- > Vmatch3_2]) > print np.max(np.abs(a), axis=0) yes, this one is better than mine. It is more flexible, leaving more options for what you may want to do with it afterwards, andre ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] finding a maximum between the absolute difference of several columns
Hi Elaina, > Vhel_fdiff3=[] > for i in xrange(len(Vmatch3_1)): > Vhel_fdiff3.append(max([abs(Vmatch3_1[i] - Vmatch3_2[i]),abs(Vmatch3_1[i] > - Vmatch3_3[i]),abs(Vmatch3_3[i] - Vmatch3_2[i])])) > > #print Vhel_fdiff3 > #appending writes a list, but numpy which makes the with_ work needs an array > Vhel_fdiff3=array(Vhel_fdiff3) You can skip making the list and directly make a numpy array. """ import numpy as np ... Vhel_fdiff3 = np.zeros_like(Vmatch3_1) for i in xrange(len(Vmatch3_1): Vhel_fdiff3[i] = max(abs(Vmatch3_1[i] - Vmatch3_2[i]),abs(Vmatch3_1[i] - Vmatch3_3[i]),abs(Vmatch3_3[i] - Vmatch3_2[i])) """ But, more importantly, you can do it all in numpy - have a read about the "max" function http://scipy.org/Numpy_Example_List#head-7918c09eea00e59ec399064e7d5e1e672d242f60 Given an 2-dimensional array, you can find the max for each row or each column by specifying which axis you want to compare against. Almost surely, the built in numpy function will be faster than your loop - even if you precompile it by importing your functions in another file. You can do something like """ import numpy as np Vhel_fdiff3 = np.array([abs(Vmatch3_1 - Vmatch3_2), abs(Vmatch3_1 - Vmatch3_3), abs(Vmatch3_3 - Vmatch3_2)]) #create an array of the absolute values of the differences of each pair of data # Vhel_fdiff3 is a 2D array # the first index runs over the three pairs of differences # the second index runs over the elements of each pair # to get the maximum difference, you want to compare over the first axis (0) your_answer = Vhel_fdiff3.max(axis=0) """ but you may not want the abs value of the differences, but to actually keep the differences, depending on which one has the maximum abs difference - that would require a little more coding. Andre ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Request for advice on Python code
Hi Elaina, just reading your question (and responses). There is an issue with what you wrote in the email, which may help. > for row in dat: > > gcoords=ICRSCoordinates(dat['ra-drad'],dat['dec-drad'],radians=True).convert(GalacticCoordinates) This line is identical to the one that broke before, and now you are just looping over it, so it is not surprising to fail in the same way. If a loop will work, you would need the following instead """ for row in dat: gcoords = ICRSCoordinates(row['ra-drad'],row['dec-drad'],radians=True).convert(GalacticCoordinates) """ notice the "dat" --> "row" inside the loop. Now if that works, you can get it to work along the following lines Example: suppose the output (gcoords) is a scalar float, and you would like to store these data in an array. Then you can do """ import numpy as np #just copying the style from a previous reply gcoords = np.zeros([len(dat)]) #this creates a numpy array of the length of dat, and fills it with all zeros for i,row in enumerate(dat): gcoords[i] = ICRSCoordinates(row['ra-drad'],row['dec-drad'],radians=True).convert(GalacticCoordinates) """ If the loop doesn't work, then you need to investigate the astrophysics library you are using, as previously mentioned, and hopefully there is someone who is familiar with it you can ask questions to. Actually, even if the loop does work, you want to be more familiar with the astrophysics library. You certainly would not want to produce results with it (and base some conclusions on them/publish them) without being 100% sure you know what it is doing. Good luck, Andre ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Lists/raw_input
Hi Michael, I bet there is a better way (which I would like to see), but here is what I have come up with for my own uses. """ ints = [] u_in = False while u_in == False: try: u_input = raw_input('please enter 5 integers, space separated\n ') for n in u_input.split(): ints.append(int(n)) u_in = True except: print('input error, try again') """ The while loop is set up in case your user inputs a float or string instead of just integers. You can also imagine putting checks in case you want exactly 5 integers, etc. Cheers, Andre On Feb 6, 2012, at 9:40 PM, Michael Lewis wrote: > I want to prompt the user only once to enter 5 numbers. I then want to create > a list out of those five numbers. How can I do that? > > I know how to do it if I prompt the user 5 different times, but I only want > to prompt the user once. > > Thanks. > > -- > Michael > > ___ > Tutor maillist - Tutor@python.org > To unsubscribe or change subscription options: > http://mail.python.org/mailman/listinfo/tutor ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Pythonic way of concatenation of elements in an array
Hi Steven, > (5) When assembling strings from substrings, never use repeated concatenation > using + as that can be EXTREMELY slow. Use str.join to build the string in > one assignment, instead of multiple assignments. > > Your code shown above is *very* inefficient and will be PAINFULLY slow if m > is very large. To understand why, you should read this article: > > http://www.joelonsoftware.com/articles/fog000319.html > > In this case, you can replace your snippet with this: > > result = '-'.join(str(item) for item in m[1:]) This was an interesting article. I have only had one programming class, and that was 15 years ago or so, so these are not issues I am aware of. I often find myself joining strings (and have mostly used + to do it). An alternate method I use is, for eg. >print('here is a string %s which has many variables %s %s %s I have to sort >out' %(s1,s2,s3,s4)) where the various strings (s1 - s4) have been determined elsewhere, perhaps in a loop. Is this method any better at combining strings than the +? My first guess would be no, but that is really just a guess. Thanks, Andre ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Why do you have to close files?
>> Exercise 17, extra credit 6 Learn python the hard way: Find out why >> you had to do output.close() in the code. >> >> >> Code: >> >> output.close() >> input.close() >> >> >> I don't get it. If you don't close input and output it works exactly >> the same as if you would close them, so why do you have to do >> output.close() and input.close()? >> >> Also does it matter if you do: input.close() and then output.close()? >> Is there an order to follow? >> > There's no order to follow, and it's really more about cleaning up after > yourself than being a necessity. If you were writing to real files, your > operating system would limit how many open files you could have at any time, > so you want to make sure you close file handles you're no longer using. Also - from experience, output.close() can be very important. eg. - I often use python to 1- create an initialization file 2 - use this initialization file with some executable code These are steps I perform in the same script. One time, I forgot to do "output.close()" before handing my file to the executable. Then for two weeks, I was wondering what was wrong with my code?!? It compiled damn it, and if I ran my job (this is on a big computing cluster) in interactive mode, everything worked!?! What was going wrong? a Well, when I finally realized I forgot to "close()" my file, I felt rather silly. The problem is python does not actually write the file to disk until you execute "output.close()". When you end your session, if you forgot to do "output.close()" it will write it to disk for you (it has all the time for me at least). But in my case, it was the same python "session" in which I was trying to both write the file, and then use it. Andre ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] databases
>> You might look at http://www.sqlalchemy.org/, since it works with most >> databases. However if you are new on databases, I think you should start >> learning databases basics, choose a database management system (DBMS) >> that fit your needs and learn as much as you can about that specific >> DBMS. Then you can use SQLAlchemy or specific packagas (MySQL-Python, >> PyGreSQL,...) to acces to that DBMS. In addition if you are going to use >> frameworks, most of them already have their own tools for DB manipulation. >> > > Just wanted to chime in, because I wish someone had told me this sooner, but > if you're using a Mac, try to steer clear of mysql-python. Setting it up on a > Mac is absolutely infuriating. > > If your databases are MySQL-based and you're using a Mac, I'd recommend > setting up a Linux VM to access them with Python (or not using Python at > all). Good luck! Thanks for the warning Alejandro. Turns out, they live on a linux cluster - but I log in via a Mac, and likely will copy locally to play around with. I figured I could either hack the c++ code built already to manipulate them, or use this as an excuse to learn about databases via python. Andre ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] databases
> > package that can open a databases without knowing there format? > > The Python DB API is pretty good at covering all the common databases but > sadly everyone has some slight variances so you do need to know which product > you will be using. > > As an example the SQLite package that comes in the standard library - and is > a good starter - doesn't require login credentials but Oracle, MySQL etc do. > Also Sqllite is stored in a single file accessed via a code library whereas > most other SQL databases use multiple files and a server frontend. (That's > why there's a connect() function - to connect to the server... in SQLite > connect just opens the file!) > > If you are a database noob I'd keep it simple and stick with SQLite, it's > powerful enough for most beginner type projects and misses out some of the > more complex features of the other packages. Provided you aren't expecting to > scale up to 10's of millions of records it will do just fine. Once you > understand SQLite moving to MySQL or Firebird or whatever will be an easy > next step. So, in case I wasn't clear, the databases are already made by someone else, and the format is beyond my control. I need/want to learn to manipulate them. Most likely they are similar to the Berkeley database (but I don't know what this means yet). Thanks for the help, Andre ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] databases
> > What type of databases? sql server, mysql, sqllite? Hi James, well this already helps. I don't even know. Do I have to know ahead of time? Or is there a general database package that can open a databases without knowing there format? Thanks, Andre ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
[Tutor] databases
Hi All, I am completely new to databases as well as using python to access/create databases. I started googling about it and found so much info, I wasn't sure where to begin to answer my first question. So I thought I would query this group by asking a question I am sure has been asked before - but you all are so friendly, I thought I would give it a try. I have databases (many of them) which I want to manipulate, averaging data, merging databases, etc. Do I need to install separate modules to access the databases? Do I need to know the specific style the databases were created in to open manipulate them with python (2.7)? I ask this because with xml files, I was able to just use from xml.dom import minidom and then by trial and error in an interactive session, I could figure out how to walk through the xml file to find what I wanted. I am wondering if I can do something similar with a database, or if there are more pre-defined formats. I do not actually know what format my databases are in (someone else wrote the c-code to create them). While waiting for suggestions, I have started to read Alan Gauld's tutorial on the subject http://www.alan-g.me.uk/tutor/index.htm Thanks, Andre ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] reclassify values in an array
> thank you for all of the resonses, I have attempted all of the suggestions. > It is a numpy array so I can try another list if you would prefer but I > thought I would show the error anyway. > the error I am receiving is > ValueError: The truth value of an array with more than one element is > ambiguous. Use a.any() or a.all() this is telling you that "value" is not a scalar element, but it has multiple dimensions. do the following: >>> big_array=N.ma.concatenate(all_FFDI) >>> print big_array.shape >>> print big_array[0].shape the first print will report the dimensionality of your big_array. The second print statement will tell you the dimensionality of the 0th element of big_array. This is the first value of "value" in your for loop. Only if the shape is given as "( )", a single (scalar) element, can you compare it to an integer or float. example >>> a = N.zeros([3]) #this builds a one-dimensional array with 3 elements and >>> puts zeros for each value >>> a array([ 0., 0., 0.]) >>> a.shape (3,) >>> a[0].shape ( ) >>> a[0] 0. imagine you have a 2-dimensional array >>> b = N.zeros([3,3]) # a 3 by 3 array of zeros >>> b.shape (3, 3) >>> b[0] array([ 0., 0., 0.]) >>> for i,value in enumerate(b): ... print value [ 0., 0., 0.] [ 0., 0., 0.] [ 0., 0., 0.] you are trying to compare the "value" [ 0., 0., 0.], to an integer. This is why your code fails - your big_array is a multi-dimensional array. The above example is what I mean by "you should play around with the python interpreter". By doing these things (above) you will begin to learn the structure of these objects (defined in this case with numpy). Regards, Andre ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] reclassify values in an array
Hi anonymous questioner, Like Alan, I suspect you are using numpy as import numpy as N there is probably a numpy email list where this would be more appropriate (personally I don't object, but I don't want to speak for all the subscribers). the 2nd attempt is closer to the right answer. To help yourself answer the question, try > for i, value in enumerate(big_array): print i,value and see what you get. Are you allowed to compare value with 100? Then, when performing the sum, you are asking to sum over axis=0. I assume you are trying to sum all the individual elements, rather than sum the rows. asking to sum over axis=0 is telling numpy to treat each row as an object, and sum all those objects, preserving all other dimensions of your array. In your case, you have a 2 dimensional array, so summing over axis=0 is taking all the rows of your array (matrix) and summing them to produce a new row. Specifically, it will take the first entry of each row, and add them to make the first entry of the summed row, then likewise for each additional entry. In math language, you are doing r_j = sum_i big_array_{i,j} if you do big_array.sum() then it will sum all of the individual elements sum = sum_i sum_j big_array_{i,j} play around more with the interactive interpreter. If you try these things, and they fail, reproduce your code from the top to bottom, adding only one line at a time, and see what happens (at least for these simple short code snippets). That should help you improve your understanding faster - which I assume is one of your goals :) Andre On Aug 31, 2011, at 4:17 PM, questions anon wrote: > Dear All, > I have been going round in circles trying to solve something that sounds > simple. I have a huge array and I would like to reclassify the values. > Firstly just make them zeros and ones, for example if the values in the array > are less than 100 make them 0 and if greater than 100 make them 1. And then > finally sum them together. > I have attempted a few methods, see code below. Any feedback will be greatly > appreciated. > > Attempt 1: > big_array=N.ma.concatenate(all_FFDI) > for i in big_array: > if i<100: > i=0 > elif i>=100: > i=1 > print big_array > sum=big_array.sum(axis=0) > print "the sum is", sum > > > Attempt 2: > big_array=N.ma.concatenate(all_FFDI) > for i, value in enumerate(big_array): > if value==100: > big_array[i]=0 > elif value>=100: > big_array[i]=1 > print big_array > sum=big_array.sum(axis=0) > print "the sum is", sum > > ___ > Tutor maillist - Tutor@python.org > To unsubscribe or change subscription options: > http://mail.python.org/mailman/listinfo/tutor ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] select particular directories and files
Hello anonymous questioner, > ok, thanks for your help Andre your welcome. hope you continue enjoying python. Andre > On Tue, Aug 30, 2011 at 2:37 PM, Andre' Walker-Loud > wrote: > to have multiple dirs is simple, > >> for dir in glob.glob(MainFolder + '*/01/') + glob.glob(MainFolder + '*/02/') >> + glob.glob(MainFolder + '*/03/'): > > > there may be a better way, but this should work. > > > By the way, not to discourage you from asking questions on the list, but many > of these things can be deduced quickly by trial and error using the python > interpreter (the interactive python shell). That is one thing I really like > about python, you really can just play around with it. And with all things, > if you figure it out on your own, then you will likely feel better about > that, and also, you will retain the knowledge better and gain more confidence > in trying new things. > > > Cheers, > > Andre > > > > > > On Aug 29, 2011, at 9:18 PM, questions anon wrote: > >> It worked! thank you. This is the code I ended with: >> >> for dir in glob.glob(MainFolder + '*/01/'): >> print dir >> for ncfile in glob.glob(dir + '*.nc'): >> print ncfile >> >> can you choose more than one folder with glob? >> i.e. I tried: >> for dir in glob.glob(MainFolder + '*/01/', '*/02/', '*/03/'): >> but I received: >> TypeError: glob() takes exactly 1 argument (3 given) >> >> >> thanks >> >> >> On Tue, Aug 30, 2011 at 1:40 PM, Andre' Walker-Loud >> wrote: >> hello, >> >> yes, I would also try adding a wild card in the dir search >> >>>> for dir in glob.glob(MainFolder + '01\*'): >> >> >> to check if this is helps, in an interpreter (rather than script) try >> >> dirs = glob.glob(MainFolder + '\01\*'): >> print dirs >> >> >> if you get "[]" then this was not the answer, but if you get a list of >> directories, then this should work. >> >> Well, it should work with the correction >> >> >>>> for ncfile in glob.glob(dir+'\*.nc'): >> >> >> >> Cheers, >> >> Andre >> >> >> >> >> >> >> On Aug 29, 2011, at 8:35 PM, questions anon wrote: >> >>> thanks, that was an error by me. but that still doesn't help me select the >>> dir and files! >>> Could it be because I am trying to select folders within other folders to >>> then get a file from each of those folders? >>> >>> >>> On Tue, Aug 30, 2011 at 1:30 PM, Andre' Walker-Loud >>> wrote: >>> Dear Anonymous Questioner, >>> >>> I am not sure how the Windows environment works, but in linux, I would >>> replace >>> >>>> for ncfile in glob.glob('.nc'): >>> >>> >>> with >>> >>>> for ncfile in glob.glob('*.nc'): >>> >>> >>> ie, add the "wild card" '*' character to grab all files which end in '.nc' >>> >>> >>> Andre >>> >>> >>> >>> >>> On Aug 29, 2011, at 7:23 PM, questions anon wrote: >>> >>>> Thanks for responding >>>> When I try glob.glob I receive no errors but nothing prints. >>>> >>>> MainFolder=r"E:/Sample/" >>>> for dir in glob.glob(MainFolder + '01'): >>>> print "my selected directories are:", dir >>>> for ncfile in glob.glob('.nc'): >>>> print "my selected netcdf files are:", ncfile >>>> >>>> any suggestions? thanks >>>> >>>> >>>> On Tue, Aug 30, 2011 at 10:07 AM, Emile van Sebille wrote: >>>> On 8/29/2011 4:52 PM questions anon said... >>>> >>>> I am trying to select particular files within >>>> a particular subdirectory, >>>> >>>> You might find glob a better starting point: >>>> >>>> ActivePython 2.6.6.15 (ActiveState Software Inc.) based on >>>> Python 2.6.6 (r266:84292, Aug 24 2010, 16:01:11) [MSC v.1500 32 bit >>>> (Intel)] on win32 >>>> Type "help", "copyright", "credits&qu
Re: [Tutor] select particular directories and files
to have multiple dirs is simple, > for dir in glob.glob(MainFolder + '*/01/') + glob.glob(MainFolder + '*/02/') > + glob.glob(MainFolder + '*/03/'): there may be a better way, but this should work. By the way, not to discourage you from asking questions on the list, but many of these things can be deduced quickly by trial and error using the python interpreter (the interactive python shell). That is one thing I really like about python, you really can just play around with it. And with all things, if you figure it out on your own, then you will likely feel better about that, and also, you will retain the knowledge better and gain more confidence in trying new things. Cheers, Andre On Aug 29, 2011, at 9:18 PM, questions anon wrote: > It worked! thank you. This is the code I ended with: > > for dir in glob.glob(MainFolder + '*/01/'): > print dir > for ncfile in glob.glob(dir + '*.nc'): > print ncfile > > can you choose more than one folder with glob? > i.e. I tried: > for dir in glob.glob(MainFolder + '*/01/', '*/02/', '*/03/'): > but I received: > TypeError: glob() takes exactly 1 argument (3 given) > > > thanks > > > On Tue, Aug 30, 2011 at 1:40 PM, Andre' Walker-Loud > wrote: > hello, > > yes, I would also try adding a wild card in the dir search > >>> for dir in glob.glob(MainFolder + '01\*'): > > > to check if this is helps, in an interpreter (rather than script) try > > dirs = glob.glob(MainFolder + '\01\*'): > print dirs > > > if you get "[]" then this was not the answer, but if you get a list of > directories, then this should work. > > Well, it should work with the correction > > >>> for ncfile in glob.glob(dir+'\*.nc'): > > > > Cheers, > > Andre > > > > > > > On Aug 29, 2011, at 8:35 PM, questions anon wrote: > >> thanks, that was an error by me. but that still doesn't help me select the >> dir and files! >> Could it be because I am trying to select folders within other folders to >> then get a file from each of those folders? >> >> >> On Tue, Aug 30, 2011 at 1:30 PM, Andre' Walker-Loud >> wrote: >> Dear Anonymous Questioner, >> >> I am not sure how the Windows environment works, but in linux, I would >> replace >> >>> for ncfile in glob.glob('.nc'): >> >> >> with >> >>> for ncfile in glob.glob('*.nc'): >> >> >> ie, add the "wild card" '*' character to grab all files which end in '.nc' >> >> >> Andre >> >> >> >> >> On Aug 29, 2011, at 7:23 PM, questions anon wrote: >> >>> Thanks for responding >>> When I try glob.glob I receive no errors but nothing prints. >>> >>> MainFolder=r"E:/Sample/" >>> for dir in glob.glob(MainFolder + '01'): >>> print "my selected directories are:", dir >>> for ncfile in glob.glob('.nc'): >>> print "my selected netcdf files are:", ncfile >>> >>> any suggestions? thanks >>> >>> >>> On Tue, Aug 30, 2011 at 10:07 AM, Emile van Sebille wrote: >>> On 8/29/2011 4:52 PM questions anon said... >>> >>> I am trying to select particular files within >>> a particular subdirectory, >>> >>> You might find glob a better starting point: >>> >>> ActivePython 2.6.6.15 (ActiveState Software Inc.) based on >>> Python 2.6.6 (r266:84292, Aug 24 2010, 16:01:11) [MSC v.1500 32 bit >>> (Intel)] on win32 >>> Type "help", "copyright", "credits" or "license" for more information. >>> >>> import glob >>> >>> help(glob.glob) >>> Help on function glob in module glob: >>> >>> glob(pathname) >>>Return a list of paths matching a pathname pattern. >>> >>>The pattern may contain simple shell-style wildcards a la fnmatch. >>> >>> >>> for filename in glob.glob(r'C:\WSG\GL\2011-08\*.txt'): >>>print filename >>> ... >>> C:\WSG\GL\2011-08\2011-01-WIP-Details.txt >>> C:\WSG\GL\2011-08\2011-02-WIP-Details.txt >>> C:\WSG\GL\2011-08\2011-03-WIP-Details.txt >>> C:\WSG\GL\2011-08\2011-04-WIP-Details.txt >>>
Re: [Tutor] select particular directories and files
hello, yes, I would also try adding a wild card in the dir search >> for dir in glob.glob(MainFolder + '01\*'): to check if this is helps, in an interpreter (rather than script) try dirs = glob.glob(MainFolder + '\01\*'): print dirs if you get "[]" then this was not the answer, but if you get a list of directories, then this should work. Well, it should work with the correction >> for ncfile in glob.glob(dir+'\*.nc'): Cheers, Andre On Aug 29, 2011, at 8:35 PM, questions anon wrote: > thanks, that was an error by me. but that still doesn't help me select the > dir and files! > Could it be because I am trying to select folders within other folders to > then get a file from each of those folders? > > > On Tue, Aug 30, 2011 at 1:30 PM, Andre' Walker-Loud > wrote: > Dear Anonymous Questioner, > > I am not sure how the Windows environment works, but in linux, I would replace > >> for ncfile in glob.glob('.nc'): > > > with > >> for ncfile in glob.glob('*.nc'): > > > ie, add the "wild card" '*' character to grab all files which end in '.nc' > > > Andre > > > > > On Aug 29, 2011, at 7:23 PM, questions anon wrote: > >> Thanks for responding >> When I try glob.glob I receive no errors but nothing prints. >> >> MainFolder=r"E:/Sample/" >> for dir in glob.glob(MainFolder + '01'): >> print "my selected directories are:", dir >> for ncfile in glob.glob('.nc'): >> print "my selected netcdf files are:", ncfile >> >> any suggestions? thanks >> >> >> On Tue, Aug 30, 2011 at 10:07 AM, Emile van Sebille wrote: >> On 8/29/2011 4:52 PM questions anon said... >> >> I am trying to select particular files within >> a particular subdirectory, >> >> You might find glob a better starting point: >> >> ActivePython 2.6.6.15 (ActiveState Software Inc.) based on >> Python 2.6.6 (r266:84292, Aug 24 2010, 16:01:11) [MSC v.1500 32 bit (Intel)] >> on win32 >> Type "help", "copyright", "credits" or "license" for more information. >> >>> import glob >> >>> help(glob.glob) >> Help on function glob in module glob: >> >> glob(pathname) >>Return a list of paths matching a pathname pattern. >> >>The pattern may contain simple shell-style wildcards a la fnmatch. >> >> >>> for filename in glob.glob(r'C:\WSG\GL\2011-08\*.txt'): >>print filename >> ... >> C:\WSG\GL\2011-08\2011-01-WIP-Details.txt >> C:\WSG\GL\2011-08\2011-02-WIP-Details.txt >> C:\WSG\GL\2011-08\2011-03-WIP-Details.txt >> C:\WSG\GL\2011-08\2011-04-WIP-Details.txt >> C:\WSG\GL\2011-08\2011-05-WIP-Details.txt >> C:\WSG\GL\2011-08\2011-06-WIP-Details.txt >> C:\WSG\GL\2011-08\2011-07 - bankToRec.txt >> C:\WSG\GL\2011-08\2011-07 - vsdsToRec.txt >> C:\WSG\GL\2011-08\2011-07-WIP-Details.txt >> C:\WSG\GL\2011-08\5790-00 RECONCILIATION.txt >> C:\WSG\GL\2011-08\BankRecUtils.txt >> C:\WSG\GL\2011-08\CapitalizationExamples.txt >> C:\WSG\GL\2011-08\DEALLOCATE-2011-04.txt >> C:\WSG\GL\2011-08\dump glsmf1 data for 2004-2010.txt >> C:\WSG\GL\2011-08\MAR DEALLOCATION.txt >> C:\WSG\GL\2011-08\Notes.txt >> C:\WSG\GL\2011-08\shipping safety net util.txt >> C:\WSG\GL\2011-08\UNBILLED WIP.txt >> C:\WSG\GL\2011-08\Vacation Accrual - post-bonus-changes.txt >> C:\WSG\GL\2011-08\Vacation Accrual - pre-bonus-changes.txt >> C:\WSG\GL\2011-08\vacation accrual notes.txt >> >>> >> >> >> >> >> I have been able to do both but not together! >> When I try to select my files within the dir loop nothing comes up, but >> when I leave the files outside the dir loops it selects all the files >> not just the ones in the dirs I have selected. >> The code I am using is: >> >> import os >> >> MainFolder=r"D:/samples/" >> >> for (path, dirs, files) in os.walk(MainFolder): >> for dir in dirs: >> if dir=='01': >> print "selected directories are:",dir >> >> for ncfile in dir: >> if ncfile[-3:]=='.nc': >> print "ncfiles are:", ncfile >> >> Any feedback will be greatly appreciated!! >> >> >> ___ >> Tutor maillist - Tutor@python.org >> To unsubscribe or change subscription options: >> http://mail.python.org/mailman/listinfo/tutor >> >> >> ___ >> Tutor maillist - Tutor@python.org >> To unsubscribe or change subscription options: >> http://mail.python.org/mailman/listinfo/tutor >> >> ___ >> Tutor maillist - Tutor@python.org >> To unsubscribe or change subscription options: >> http://mail.python.org/mailman/listinfo/tutor > > ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] select particular directories and files
Dear Anonymous Questioner, I am not sure how the Windows environment works, but in linux, I would replace > for ncfile in glob.glob('.nc'): with > for ncfile in glob.glob('*.nc'): ie, add the "wild card" '*' character to grab all files which end in '.nc' Andre On Aug 29, 2011, at 7:23 PM, questions anon wrote: > Thanks for responding > When I try glob.glob I receive no errors but nothing prints. > > MainFolder=r"E:/Sample/" > for dir in glob.glob(MainFolder + '01'): > print "my selected directories are:", dir > for ncfile in glob.glob('.nc'): > print "my selected netcdf files are:", ncfile > > any suggestions? thanks > > > On Tue, Aug 30, 2011 at 10:07 AM, Emile van Sebille wrote: > On 8/29/2011 4:52 PM questions anon said... > > I am trying to select particular files within > a particular subdirectory, > > You might find glob a better starting point: > > ActivePython 2.6.6.15 (ActiveState Software Inc.) based on > Python 2.6.6 (r266:84292, Aug 24 2010, 16:01:11) [MSC v.1500 32 bit (Intel)] > on win32 > Type "help", "copyright", "credits" or "license" for more information. > >>> import glob > >>> help(glob.glob) > Help on function glob in module glob: > > glob(pathname) >Return a list of paths matching a pathname pattern. > >The pattern may contain simple shell-style wildcards a la fnmatch. > > >>> for filename in glob.glob(r'C:\WSG\GL\2011-08\*.txt'): >print filename > ... > C:\WSG\GL\2011-08\2011-01-WIP-Details.txt > C:\WSG\GL\2011-08\2011-02-WIP-Details.txt > C:\WSG\GL\2011-08\2011-03-WIP-Details.txt > C:\WSG\GL\2011-08\2011-04-WIP-Details.txt > C:\WSG\GL\2011-08\2011-05-WIP-Details.txt > C:\WSG\GL\2011-08\2011-06-WIP-Details.txt > C:\WSG\GL\2011-08\2011-07 - bankToRec.txt > C:\WSG\GL\2011-08\2011-07 - vsdsToRec.txt > C:\WSG\GL\2011-08\2011-07-WIP-Details.txt > C:\WSG\GL\2011-08\5790-00 RECONCILIATION.txt > C:\WSG\GL\2011-08\BankRecUtils.txt > C:\WSG\GL\2011-08\CapitalizationExamples.txt > C:\WSG\GL\2011-08\DEALLOCATE-2011-04.txt > C:\WSG\GL\2011-08\dump glsmf1 data for 2004-2010.txt > C:\WSG\GL\2011-08\MAR DEALLOCATION.txt > C:\WSG\GL\2011-08\Notes.txt > C:\WSG\GL\2011-08\shipping safety net util.txt > C:\WSG\GL\2011-08\UNBILLED WIP.txt > C:\WSG\GL\2011-08\Vacation Accrual - post-bonus-changes.txt > C:\WSG\GL\2011-08\Vacation Accrual - pre-bonus-changes.txt > C:\WSG\GL\2011-08\vacation accrual notes.txt > >>> > > > > > I have been able to do both but not together! > When I try to select my files within the dir loop nothing comes up, but > when I leave the files outside the dir loops it selects all the files > not just the ones in the dirs I have selected. > The code I am using is: > > import os > > MainFolder=r"D:/samples/" > > for (path, dirs, files) in os.walk(MainFolder): > for dir in dirs: > if dir=='01': > print "selected directories are:",dir > > for ncfile in dir: > if ncfile[-3:]=='.nc': > print "ncfiles are:", ncfile > > Any feedback will be greatly appreciated!! > > > ___ > Tutor maillist - Tutor@python.org > To unsubscribe or change subscription options: > http://mail.python.org/mailman/listinfo/tutor > > > ___ > Tutor maillist - Tutor@python.org > To unsubscribe or change subscription options: > http://mail.python.org/mailman/listinfo/tutor > > ___ > Tutor maillist - Tutor@python.org > To unsubscribe or change subscription options: > http://mail.python.org/mailman/listinfo/tutor ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] numpy.mean across multiple netcdf files
Hello Anonymous Questioner, First - you will probably be told this isn't the correct email list for this question - this is a general python tutorial list, while your question is numpy specific, so if this doesn't help, you should probably look to another email list. There are a couple general comments to make regarding your question. Basically, you want to load all the data into one big numpy array. Currently, it seems your data array has 3 indices, from this line > TSFC=ncfile.variables['T_SFC'][4::24,:,:] you can make a larger array with one more index, where the new index runs over files. So instead of looping over the files as you do, you can use the loop to build the larger array. I am sure there is a more elegant way to do this, but here is something that is along the lines of what you want > big_array = [] > for ncfile in files: > big_array.append(ncfile.variables['T_SFC'][4::24,:,:]) if all of your data you are taking from each file is the same size for all the files, then you can declare the whole thing an array > big_array = N.array(big_array) and then you can take averages very easily by specifying which index, or indices you want to average over. If life is not so kind, and your files are of different sizes, then you have to be more careful. But to give a precise answer to your problem would require knowledge of what your data files actually look like. Also, a bit of caution with averaging (in case you are unfamiliar). Say you have 5 data sets, and each one has a different amount of data in it. If you first take the average of each set, you can not simply take the average of those sets to produce the global average. You have to weight each set by the amount of data you averaged. Say the length of the sets are [N0,N1,N2,N3,N4] with average values [a0,a1,a2,a3,a4], then to produce the global average, you need to take avg = (1 / (N0+N1+N2+N3+N4) ) * (N0*a0 + N1*a1 + N2*a2 + N3*a3 + N4*a4) a few lines of algebra can demonstrate this produces the average you would get by combining all the data in all the sets and taking one big average. Regards, Andre On Aug 10, 2011, at 6:57 PM, questions anon wrote: > I have many ncfiles each containing one month of hourly temperature data. > I have worked out how to loop through a number of ncfiles and calculate the > mean for each file at a particular time and even plot this and output as a > *.png. > What I am unsure of is how to calculate the mean at a particular time for all > files. > Do I somehow output the mean for each folder and then calculate the mean of > all the means? Or is there some way to loop through and calculate it directly? > Below is the code I have working that calculates the mean for each file. Any > help will be greatly appreciated!! > > from netCDF4 import Dataset > import numpy as N > > MainFolder=r"D:/temp_samples/" > print MainFolder > for (path, dirs, files) in os.walk(MainFolder): > for dir in dirs: > print dir > path=path+'/' > > for ncfile in files: > if ncfile[-3:]=='.nc': > print "dealing with ncfiles:", ncfile > ncfile=os.path.join(path,ncfile) > ncfile=Dataset(ncfile, 'r+', 'NETCDF4') > TSFC=ncfile.variables['T_SFC'][4::24,:,:] #this chooses a > particular time for each day for the whole month > LAT=ncfile.variables['latitude'][:] > LON=ncfile.variables['longitude'][:] > TIME=ncfile.variables['time'][:] > fillvalue=ncfile.variables['T_SFC']._FillValue > ncfile.close() > > #calculate summary stats > TSFCmean=N.mean(TSFC, axis=0) > print "The TSFCmean array is:", TSFCmean > > # there is a lot more code that follows this to plot the mean for each ncfile > but I have cut that out as I think it is irrelevant to my question > > ___ > Tutor maillist - Tutor@python.org > To unsubscribe or change subscription options: > http://mail.python.org/mailman/listinfo/tutor ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] [OT] Re: Floating Point Craziness
On Jun 13, 2011, at 1:44 PM, Emile van Sebille wrote: > On 6/12/2011 1:55 PM Andre' Walker-Loud said... >> Hi Alan, >> >>>> * Or you just get used to the fact that some numbers are not exact in >>>> floating point. >>> >>> This got me thinking. How many decimal places do you need to >>> accurately, say, aim a laser somewhere in a 180 degree arc accurately >>> enough to hit a dime on the surface of the moon? >> >> Here is a quick back of the envelope estimate for you. (While I am still >> learning the Python, I can answer this one!) >> >> The angle subtended by a dime on the earth is (approximately) given by >> >> sin( theta ) = d / sqrt( R^2 + d^2 ) >> >> where >> >> d = 1 cm (the diameter of a dime) >> R = 384,403 km (the average distance from the center of the earth to the >> center of the moon - the moon traverses an elliptical path about the earth) >> >> To make the approximation simple, take advantage of the series expansion for >> sin (theta) and 1 / sqrt(R^2 + d^2) >> >> first >> >> d / sqrt( R^2 + d^2 ) = d / R * 1 / sqrt(1 + d^2 / R^2 ) >>~= d / R * (1 - 1/2 * d^2 / R^2 + ...) >> >> now >> >> d / R = 1 * e-2 / (384403 * e3) >>~= 3 * e-11 >> >> so the d^2 / R^2 correction will be very small, and won't effect the >> determination. So we now have >> >> sin (theta) ~= d / R >> >> This will be a very small angle. The next approximation to make is for >> small angles >> >> sin (theta) ~= theta + ... >> >> leaving us with >> >> theta ~= d / R >> >> >> To be approximate, assume the precision you need is equal to the size of the >> dime. This means you need an precision of >> >> d theta ~= d/R ~= 3 * e-11 ( = 3 * 10^{-11} if you aren't familiar with the >> "e" notation) >> >> this is the minimum precision you would need in both the "x" and "y" >> direction to accurately hit the dime on the moon with your laser (at its >> average distance). >> >> Corrections to this estimate will come from the fact that the moon's radius >> is ~1737 km and the earth's radius is ~6370 km, so you are actually this >> much closer (R is this much smaller). >> >> Of course both the earth is spinning and the moon is moving relative to us, >> so you would have to account for those extra corrections as well. >> >> >> Hope that wasn't too much info, >> > > > Of course not. I enjoyed it. However, don't you need to work divergence in, > as per wikipedia, "...At the Moon's surface, the beam is only about 6.5 > kilometers (four miles) wide[6] and scientists liken the task of aiming the > beam to using a rifle to hit a moving dime 3 kilometers (two miles) away." > > (http://en.wikipedia.org/wiki/Lunar_Laser_Ranging_experiment) Of course, I was performing just the 'theoretical' calculation. It is up to others to dtermine if it is actually practical :) Andre > > Emile > > ___ > Tutor maillist - Tutor@python.org > To unsubscribe or change subscription options: > http://mail.python.org/mailman/listinfo/tutor ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] [OT] Re: Floating Point Craziness
Hi Alan, >> * Or you just get used to the fact that some numbers are not exact in >> floating point. > > This got me thinking. How many decimal places do you need to > accurately, say, aim a laser somewhere in a 180 degree arc accurately > enough to hit a dime on the surface of the moon? Here is a quick back of the envelope estimate for you. (While I am still learning the Python, I can answer this one!) The angle subtended by a dime on the earth is (approximately) given by sin( theta ) = d / sqrt( R^2 + d^2 ) where d = 1 cm (the diameter of a dime) R = 384,403 km (the average distance from the center of the earth to the center of the moon - the moon traverses an elliptical path about the earth) To make the approximation simple, take advantage of the series expansion for sin (theta) and 1 / sqrt(R^2 + d^2) first d / sqrt( R^2 + d^2 ) = d / R * 1 / sqrt(1 + d^2 / R^2 ) ~= d / R * (1 - 1/2 * d^2 / R^2 + ...) now d / R = 1 * e-2 / (384403 * e3) ~= 3 * e-11 so the d^2 / R^2 correction will be very small, and won't effect the determination. So we now have sin (theta) ~= d / R This will be a very small angle. The next approximation to make is for small angles sin (theta) ~= theta + ... leaving us with theta ~= d / R To be approximate, assume the precision you need is equal to the size of the dime. This means you need an precision of d theta ~= d/R ~= 3 * e-11 ( = 3 * 10^{-11} if you aren't familiar with the "e" notation) this is the minimum precision you would need in both the "x" and "y" direction to accurately hit the dime on the moon with your laser (at its average distance). Corrections to this estimate will come from the fact that the moon's radius is ~1737 km and the earth's radius is ~6370 km, so you are actually this much closer (R is this much smaller). Of course both the earth is spinning and the moon is moving relative to us, so you would have to account for those extra corrections as well. Hope that wasn't too much info, André ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Making a script part of the terminal
Hi Michael, You have to do three (four) things. 1 - make a directory where you want your executables to live (after you have created them) on my machine, I copy everything to /usr/local/walkloud/bin/ which requires sudo, but you can put them anywhere in your $PATH 2 - in your .tcshrc or .cshrc or equivalent file (to figure it out, type echo $SHELL to figure out your shell if you have no idea what I am talking about), you must append your path. eg. with tcshrc (in the .tcshrc file) - the .tcshrc file is located in your $HOME dir. setenv PATH /usr/local/walkloud/bin:$PATH 3 - if you haven't, in the directory where your script lives chmod +x your_script 4 - cp your script to this directory in 1- launch a new terminal and it should work. Andre On May 20, 2011, at 11:10 AM, michael scott wrote: > Thank you for the reply, but thats not exactly what I mean. Perhaps I should > say, how do I install a program to my computer, so that I can use it by its > self without running it with python. No matter what directory I'm in I can > type mozilla in and it runs, no matter what directory I'm in if I type sudo > natutilus it will run, no matter what directory I'm in if I type gedit it > will run. > > So I'm trying to achieve this with the script I wrote. I don't know the > terminology to ask the question correctly, so forgive me. > > > What is it about you... that intrigues me so? > > > From: James Reynolds > To: michael scott > Cc: tutor@python.org > Sent: Fri, May 20, 2011 1:57:57 PM > Subject: Re: [Tutor] Making a script part of the terminal > > We just had a similar question yesterday. > > Just make sure Python is on your PATH. CD to the directory where your file is > located and then you can just type "pythonmyfile.py" where myfile is the name > of your file. > > On Fri, May 20, 2011 at 1:43 PM, michael scott wrote: > Okay, my title might be undescriptive, let me try to explain it better. I > want to take a script I've written and make it usable by typing its name in > the terminal. Perfect example is the python interpreter. You just type in the > word python to the terminal and then the interpreter runs. I know other > programs can do this as well (like mozilla or nautilus or rhythmbox). So how > do I make my scripts executable from the terminal? > > > What is it about you... that intrigues me so? > > > ___ > Tutor maillist - Tutor@python.org > To unsubscribe or change subscription options: > http://mail.python.org/mailman/listinfo/tutor > > > ___ > Tutor maillist - Tutor@python.org > To unsubscribe or change subscription options: > http://mail.python.org/mailman/listinfo/tutor ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] numpy import failure
>> Andre > Thanks again for your responses. I am using MS Windows 7, and it turns out > there was a problem with build and re-installing EPD fixed it. I appreciate > the help, I am so new at this, I don't know if things aren't working because > it is me, or because something else is wrong (I feel like a person who has > known how to drive for years, but never bothered to look under the hood of > the car). Anyway, thanks again. welcome to the club! Andre On Feb 28, 2011, at 7:34 AM, Ezra Kahn wrote: > On 2/27/2011 9:36 PM, tutor-requ...@python.org wrote: >> Date: Sun, 27 Feb 2011 18:58:46 -0800 >> From: Andre' Walker-Loud >> To: Ezra Kahn >> Cc:tutor@python.org >> Subject: Re: [Tutor] numpy import failure >> Message-ID:<75198436-f849-4994-8d48-1726c2ee7...@gmail.com> >> Content-Type: text/plain; charset=us-ascii >> >> Hi Ezra, >> >> Are you using Mac OSX or LINUX or ... >> >> If you have a preexisting python installation, it may be that when you >> launch python, it loads the older version, and not the new EPD version. >> When you launch python, what do you see? For example, on my Mac OSX, >> launched from Terminal, I get >> >> % python >> Enthought Python Distribution --http://www.enthought.com >> Version: 6.2-2 (32-bit) >> >> Python 2.6.5 |EPD 6.2-2 (32-bit)| (r265:79063, May 28 2010, 15:13:03) >> [GCC 4.0.1 (Apple Inc. build 5488)] on darwin >> Type "help", "copyright", "credits" or "license" for more information. >>>>> >>> >> whereas, if I use an older version, I get >> >> % python >> Python 2.6.6 (r266:84374, Aug 31 2010, 11:00:51) >> [GCC 4.0.1 (Apple Inc. build 5493)] on darwin >> Type "help", "copyright", "credits" or "license" for more information. >>>>> >>> >> You need to get the equivalent of the first option. If you are definitely >> launching the Enthought python (from your new installation) and getting this >> error, then there is a problem with the package build on your machine. >> >> >> Andre > Thanks again for your responses. I am using MS Windows 7, and it turns out > there was a problem with build and re-installing EPD fixed it. I appreciate > the help, I am so new at this, I don't know if things aren't working because > it is me, or because something else is wrong (I feel like a person who has > known how to drive for years, but never bothered to look under the hood of > the car). Anyway, thanks again. > > Ezra > ___ > Tutor maillist - Tutor@python.org > To unsubscribe or change subscription options: > http://mail.python.org/mailman/listinfo/tutor ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] numpy import failure
Hi Ezra, Are you using Mac OSX or LINUX or ... If you have a preexisting python installation, it may be that when you launch python, it loads the older version, and not the new EPD version. When you launch python, what do you see? For example, on my Mac OSX, launched from Terminal, I get % python Enthought Python Distribution -- http://www.enthought.com Version: 6.2-2 (32-bit) Python 2.6.5 |EPD 6.2-2 (32-bit)| (r265:79063, May 28 2010, 15:13:03) [GCC 4.0.1 (Apple Inc. build 5488)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> whereas, if I use an older version, I get % python Python 2.6.6 (r266:84374, Aug 31 2010, 11:00:51) [GCC 4.0.1 (Apple Inc. build 5493)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> You need to get the equivalent of the first option. If you are definitely launching the Enthought python (from your new installation) and getting this error, then there is a problem with the package build on your machine. Andre On Feb 26, 2011, at 8:45 PM, Ezra Kahn wrote: > I am a total newb, learning to ween myself off of Matlab. I am working off > of EPD6.1, and I cannot get numpy to import. Python keeps sending me back > this: > > Traceback (most recent call last): > File "", line 1, in >import numpy > ImportError: No module named numpy > > How do I trouble shoot this? > > Thanks > > Ezra > ___ > Tutor maillist - Tutor@python.org > To unsubscribe or change subscription options: > http://mail.python.org/mailman/listinfo/tutor ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] specifying precision with scientific notation
Hi Wayne, Yes - that helps. I missed the correct combination of parentheses. Now I am trying to improve my aesthetics, as Evert suggested earlier > Personally, I would print this as 7.63e-03 +- 0.83e-03, which shows the > precision a bit better. But that's just a matter of aesthetics, and would > make things even more complicated (then again, since you are printing numbers > and formatting them, you are concerned about aesthetics). I agree with him here, and would like to match this style. I have been trying to figure out if you can specify the "mantissa" and exponent in scientific notation, by reading in the string.format section of python, but have been unsuccessful. Is there an easy way to do this? To be precise, I now have > a = 0.0762921383941; ea = 0.000830132912068 > p = int(("%.1e" % (a / ea)).split('e')[-1]) > print(('%.' + str(int(2+p)) +'e +- %.1e') % (a, ea)) 7.629e-02 +- 8.3e-04 and instead I would like this to print 7.629e-02 +- 0.083e-02 I could imagine writing a little function that does all this, but am hoping there is a quick (and dirty) way to just force the scientific notation to into this format - I guess by forcing the power of the exponent. Thanks, Andre On Oct 5, 2010, at 6:39 PM, Wayne Werner wrote: > On Tue, Oct 5, 2010 at 7:51 PM, Andre' Walker-Loud > wrote: > Hi Alan, > > The point I can not get to work is > > > fmt = "%.%de + %.1e" % n else: > > when I try this, I get (python 2.6.5, OS X 10.6) > > > n = 3; fmt = "%.%de + %.1e" % n; fmt > '%de + 3.0e+00' > > But something like the "%.%de " %n is exactly what I am looking for - if I > could get it to work. > > > a = 0.00762921383941 > ea = 0.000830132912068 > > > p = int(("%.1e" % (a / ea)).split('e')[-1]) # grab the power in the > scientific n > otation (is there a better way?) > if p >= 0: > print(('%.' + str(int(2+p)) +'e +- %.1e') % (a, ea)) > else: > print('%.2e +- %.1e' % (a, ea)) > #(desired output): 7.63e-03 +- 8.3e-04 > > > This works for me - I added some extra parenthesis because the original was > giving me this error: > > Traceback (most recent call last): > File "exponent.py", line 7, in > print('%.' + str(int(2+p)) +'e +- %.1e' % (a, ea)) > TypeError: not all arguments converted during string formatting > > HTH, > Wayne > ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] specifying precision with scientific notation
Hi Alan, The point I can not get to work is > fmt = "%.%de + %.1e" % n else: when I try this, I get (python 2.6.5, OS X 10.6) > n = 3; fmt = "%.%de + %.1e" % n; fmt '%de + 3.0e+00' But something like the "%.%de " %n is exactly what I am looking for - if I could get it to work. Thanks, Andre On Oct 5, 2010, at 3:43 PM, Alan Gauld wrote: > "Andre' Walker-Loud" wrote > >>> a = 0.00762921383941 >>> ea = 0.000830132912068 >>> a / ea >> 9.190352205653852 >> >> By default, I will print the uncertainty ("ea") with two significant digits. >> In this example, the central value is about 10 times larger than the >> uncertainty, so I want to print it with 3 significant figures. > > I don't understand why the difference but if the deciding factor is related > to the ratio why bother with all the string stuff? Just use the ratio > directly... > >>> p = int(("%.1e" % (a / ea)).split('e')[-1]) # grab the power in the >>> scientific notation (is there a better way?) >>> if p >= 0: > > Why not just > > limit = 10 > n = 3 if a/ea <= limit else n = 2 # or whatever expression is needed to > calculate n > fmt = "%.%de + %.1e" % n else: > > print fmt % (a, ea) > > > But I suspect I'm missing something in your reasoning about what size of n > you want. > > -- > Alan Gauld > Author of the Learn to Program web site > http://www.alan-g.me.uk/ > > > > ___ > Tutor maillist - Tutor@python.org > To unsubscribe or change subscription options: > http://mail.python.org/mailman/listinfo/tutor ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
[Tutor] specifying precision with scientific notation
Hi All, I want to print scientific numbers with a specified number of decimal places. However, I want the number printed to be dynamically determined by the data. Example: > a = 0.00762921383941 > ea = 0.000830132912068 > a / ea 9.190352205653852 By default, I will print the uncertainty ("ea") with two significant digits. In this example, the central value is about 10 times larger than the uncertainty, so I want to print it with 3 significant figures. So I want to do something like > p = int(("%.1e" % (a / ea)).split('e')[-1]) # grab the power in the > scientific notation (is there a better way?) > if p >= 0: > print('%.' + str(int(2+p)) +'e +- %.1e' % (a, ea)) > else: print('%.2e +- %.1e' % (a, ea)) (desired output): 7.63e-03 +- 8.3e-04 but this fails. And I haven't figured out how to get this to work. Seems like it should be simple. Any help? Thanks, Andre ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] optional sys.argv parsing
Hi Sander, On Thu, 2009-10-29 at 17:30 -0400, Andre Walker-Loud wrote: I have a simple question. I am writing a little program that will make some plots of data files. I want to have optional args to pass, for example to specify the plot ranges. I have never written a script/ code that takes optional args (but I have used plenty) - so I am feeling a little sluggish writing a good sys.argv reader. I have the following few lines I made, but I am wondering if any of you have suggestions for how to make this better (ie more slick, more readable, more general etc) You are a perfect candidate for the optparse module [1] which will do the heavy lifting for you. Example code relating to your code below. Thanks - yes this is exactly what I need. I will play around with this. Thanks again, Andre -- from optparse import OptionParser parser = OptionParser() parser.add_option('-f', '--file', action='store', type='string', dest='filename', help='Explain your filename') parser.add_option('-x', action='store', type='int', dest='x', help='Explain your x value') parser.print_help() Usage: [options] Options: -h, --helpshow this help message and exit -f FILENAME, --file=FILENAME Explain your filename -x X Explain your x value args = ['-f','somefilename','-x', '25'] opts, args = parser.parse_args(args) opts.x 25 opts.filename 'somefilename' type(opts.x) - Greets Sander [1] http://docs.python.org/library/optparse.html import sys if len(sys.argv) < 2: print('no data file specified') sys.exit(-1) elif len(sys.argv) > 2: if sys.argv.count('-x') > 1: print('error: multiple instances of "-x xmin xmax"') sys.exit(-1) elif sys.argv.count('-x') == 1: xrange = sys.argv.index('-x') if sys.argv.count('-y') > 1: print('error: multiple instances of "-y ymin ymax"') sys.exit(-1) elif sys.argv.count('-y') == 1: yrange = sys.argv.index('-y') else: xrange = 0 yrange = 0 if xrange != 0: xmin = float(sys.argv[xrange+1]) xmax = float(sys.argv[xrange+2]) else: xmin = "x-min determined from data file" xmax = "x-max determined from data file" if yrange != 0: ymin = float(sys.argv[yrange+1]) ymax = float(sys.argv[yrange+2]) else: ymin = "y-min determined from data file" ymax = "y-max determined from data file" ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
[Tutor] optional sys.argv parsing
Hi All, I have a simple question. I am writing a little program that will make some plots of data files. I want to have optional args to pass, for example to specify the plot ranges. I have never written a script/ code that takes optional args (but I have used plenty) - so I am feeling a little sluggish writing a good sys.argv reader. I have the following few lines I made, but I am wondering if any of you have suggestions for how to make this better (ie more slick, more readable, more general etc) Thanks, Andre >>> import sys if len(sys.argv) < 2: print('no data file specified') sys.exit(-1) elif len(sys.argv) > 2: if sys.argv.count('-x') > 1: print('error: multiple instances of "-x xmin xmax"') sys.exit(-1) elif sys.argv.count('-x') == 1: xrange = sys.argv.index('-x') if sys.argv.count('-y') > 1: print('error: multiple instances of "-y ymin ymax"') sys.exit(-1) elif sys.argv.count('-y') == 1: yrange = sys.argv.index('-y') else: xrange = 0 yrange = 0 if xrange != 0: xmin = float(sys.argv[xrange+1]) xmax = float(sys.argv[xrange+2]) else: xmin = "x-min determined from data file" xmax = "x-max determined from data file" if yrange != 0: ymin = float(sys.argv[yrange+1]) ymax = float(sys.argv[yrange+2]) else: ymin = "y-min determined from data file" ymax = "y-max determined from data file" ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] creating interactive program with python
Hi Alan, 1) I am trying to trying to figure out how I would set up an interactive python script, however, I am having trouble googling this, since a search for "interactive python" just returns instructions on how to type "python" in a terminal to get an "interactive" session Check out the Talking to the User topic in my tutorial. It covers interactive prompting and use of command line arguments. Wow! That was sooo easy, thanks! I was expecting to have to sweat for a while to figure it out. I really appreciate the multiple example style of your tutorial! 3) I see iPython (which I have never used) is an enhanced interactive python shell. Perhaps this could do what I want - does anyone have experience with iPython? Several here use it but that is a development tool not the final application. However you might like to investigate the cmd module... It could be very helpful. I will check it out. Thanks, Andre 4) does anyone have strong recommendations for or against using matplotlib? Is it low maintenance for producing graphs, or does it take lots of user interaction? All the example pictures I see are very pretty, but it is not clear how much effort is needed to generate them - or is it just a "well know to those who know it well" thing? I haven't used it seriously but those that do seem to find it works OK. HTH, -- Alan Gauld Author of the Learn to Program web site http://www.alan-g.me.uk/ ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
[Tutor] creating interactive program with python
Hi All, I am thinking of creating a data analysis suite with python, and I want it to be interactive - ie the program asks the user (me) for input, like which data file to use, how many parameters to minimize, etc. There are a few features I want it to have, like graphing, both the initial data as well as results of my analysis, etc. If you are familiar with it, Matlab is close to what I would like, a sort of all in one tool - but that is expensive commercial software. So I have a few questions. And I should also say I would classify myself as a medium level python scripter, but I have not used the more advanced programming features, and I am a beginning programmer. That being said, I am interested in expanding my python knowledge, especially getting into the more advanced features. Even though this is ultimately just a tool to do my job (limiting my time to play around and learn new things) I generally find python so easy to use (and fun) that I am interested in this little project of mine (and also it will help me sever my dependency on Mathematica, if you are familiar with that program). Thanks Andre 1) I am trying to trying to figure out how I would set up an interactive python script, however, I am having trouble googling this, since a search for "interactive python" just returns instructions on how to type "python" in a terminal to get an "interactive" session I have in mind a while loop that asks the user for arguments, and then some switch making sure I have passed enough sys.args to proceed, but I am not sure how to prompt the user for arguments and count them locally (not a global number of sys.argv which is what I am used to now). So any advice on where to look at an example of this would be great. 2) I have come across the "Scipy Superpack" http://macinscience.org/?page_id=6 which contains Scipy, Numpy, Matplotlib, iPython and PyMC. So it seems the installation would be convenient since it is all bundled together. Anyone have experience with this package? 3) I see iPython (which I have never used) is an enhanced interactive python shell. Perhaps this could do what I want - does anyone have experience with iPython? Even if iPython can do everything I want (and more I am sure) I would still like to write my own basic little interactive script to expand my python knowledge. 4) does anyone have strong recommendations for or against using matplotlib? Is it low maintenance for producing graphs, or does it take lots of user interaction? All the example pictures I see are very pretty, but it is not clear how much effort is needed to generate them - or is it just a "well know to those who know it well" thing? ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] python precision output?
> if you want accuracy and are willing to sacrifice the total range of > numbers that Python's IEEE754 double-precision floats give you, then > use the decimal.Decimal class instead -- better precision, smaller > range. > > however, if you wish to stick with floats, use the string format > operator and tell it you want 17 places after the decimal point: > x=7./13 x > 0.53846153846153844 str(x) > '0.538461538462' '%.17f' % x > '0.53846153846153844' > > hope this helps! This did the trick! thanks, Andre > -- wesley > - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - > "Core Python Programming", Prentice Hall, (c)2007,2001 > http://corepython.com > > wesley.j.chun :: wescpy-at-gmail.com > python training and technical consulting > cyberweb.consulting : silicon valley, ca > http://cyberwebconsulting.com ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
[Tutor] python precision output?
Hi there, I am using python to do some scripting. In particular, I am using it to run some jobs which require precision inputs. I do this by having python write an input file, which I then feed to some other program. The problem I am having is getting python to write number into this input file, keeping 16 digits of precision. I have played around interactively, and see that python default prints 17 digits of precision to the screen, but when I use a replace command to write into the input file, it only prints 12 digits of precision. The relevant snipit of my script is value = float( int(ai) * 6 * math.pi / (int(L)*int(T)) replace = {'VALUE':str(value)} ini_file = open('generic_ini').read() f=open('my_input.xml','w') f.write(ini_file % replace) f.close() where, "ai", "L" and "T" are process dependent numbers defined in my script, and the output "my_input.xml", is just an xml file I later feed to another program, and this is why I replace 'VALUE' with a string. To reiterate, I need str(value) to be written to my file with 16 digits of precision...??? Thanks, Andre ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] using python to execute from Dir A in Dir B
Thank you everyone for the help. I have two solutions, but I would love one that uses the subprocess.Popen() - I have no experience with this module.class - if someone with more experience would feel inclined to provide an example, I would be very much appreciative. SOLUTION 1 (my brute force method) create a csh file which does what I want, # filename: hack.csh #!/bin/csh set RUN_DIR = $1 set EXEC_DIR = $2 cd ${RUN_DIR} ${EXEC_DIR}/my.exe # then I call this in my python code from some arbitrary directory # python script... os.system('./hack.csh /scratch exec_dir') # SOLUTION 2 (courtesy of my computer-superhero friend - again simple) in my python script... # import os curdir = os.path.abspath('.') # in case I need to get back to where I am - in my case no os.chdir('RUN_DIR') # in my case RUN_DIR = /scratch if I wanted to get back to my old directory - then add os.chdir(curdir) # so both of these methods are sort of brute force - being completely unfamiliar with the subprocess module, again is someone would like to provide an example, or at least more hints than 'you should use subprocess.Popen()' I thank you in advance. Cheers, Andre On Oct 4, 2007, at 5:13 PM, Alan Gauld wrote: > "Andre Walker-Loud" <[EMAIL PROTECTED]> wrote > >> If I were using CSH, I could do all this very simply by having these >> lines in my script >> >> ### .csh file >> >> cd /scratch >> my_exe.csh > > The best answer is to use subprocess as Kent suggested > but you can also use os.chdir(path) before using os.system() > But system() is deprecated in favour of the subprocess module. > > Of course you could also modify your script to take a > path as a command line argument and use that to direct > the output explicitly... > > Alan G. > > > ___ > Tutor maillist - Tutor@python.org > http://mail.python.org/mailman/listinfo/tutor ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
[Tutor] using python to execute from Dir A in Dir B
Hi All, lets say I am in Dir A (out of my control because I have submitted a job to a queuing system) and I have a python script which is running in this directory - the one I submitted to the queue what I need to do is have my python script run another executable, but it must do it from a directory different from the one I am in, it needs to run in the /scratch/ directory (again not my choice) Is this possible, and is it easy? the way I have been using python to run other executables is os.system('path/my.executable input') but this dumps the output in the directory I am currently in, and not the directory the my.executable lives in - I am hoping there is a simple way to tell python to run the executable in its directory, and have its output live there as well. If I were using CSH, I could do all this very simply by having these lines in my script ### .csh file cd /scratch my_exe.csh I am hoping this idea makes sense and is simple with python... Thanks, Andre ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor