Re: [Tutor] Civil discourse from a newbie's perspective
First of all, a HUGE thanks to the volunteers that share their knowledge in this forum for free! When I first became aware of this and other volunteer forums I was amazed that there are people out there willing to spend their valuable time freely helping those of us in need. Thanks also to everyone that has participated on this post, it has been an enlightening read. In my case, I am a complete beginner not only to python but programming in general, a complete beginner to internet forums (this is my third post ever) and I am also not a native English speaker. So, I feel triply ignorant when I post a question. Not only do I find it difficult to articulate my problem, as being a beginner to programming, I basically don't know what the hell I'm talking about, but after reading other posts where people have been corrected on their formatting I am also very insecure about this. So, I would second Leam's suggestion to add recommended behaviour to the welcome email to this forum and if it is not too much to ask, some guidelines on appropriate formating when participating in this forum. If this information has already been given in previous posts, maybe a permanent link on the forum website to these could be setup so troublesome, ignorant beginners like myself can be referred to it and valuable volunteer's time is not wasted on making these corrections over and over again. On Tue, Oct 2, 2012 at 10:04 AM, leam hall leamh...@gmail.com wrote: My own struggles to better communicate, and to have my message heard, supports the concerns raised here. The Python community is a very good one and we are only made better by treating people well. it is easy to go to other lists where I am a newbie and find top posting preferred and other behavior encouraged. Does the welcome e-mail cover any of the recommended behavior? Are there easier ways to request participation within guidelines? Leam -- Mind on a Mission http://leamhall.blogspot.com/ ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Help for Python Beginner with extracting and manipulating data from thousands of ASCII files
On Mon, Oct 1, 2012 at 12:33 AM, Alan Gauld alan.ga...@btinternet.comwrote: On 30/09/12 23:07, Cecilia Chavana-Bryant wrote: Hola again Python Tutor! With a friend's help I have the following code to extract reflectance data from an ASCII data file, do a bit of data manipulation to calibrate the data and then write the calibrated file into an out file. snip I have successfully calibrated one ASCII file at a time with this code. However, I have 1,000s of files that I need to calibrate so I would like some help to modify this code so it can: 1. Use one calibration file (Cal_FileP17.txt) on data files created from July to the 18th Sep and a different calibration file (Cal_FileP19.txt) for data files created from the 19th of Sep onwards. 2. Find all the .txt files in a folder called ASCII_files, which is subdivided into 12 different folders and calibrate all these files Number 2 is easier to solve and the os.walk() and glob.glob() functions should provide all the tools you need. Number 1 is more tricky since there is no obvious way to determine the arbitrary start/stop dates you specify. So I'd suggest you need to generalise the requirement to take a start/stop date as well as the calibration file name and the input data file pattern. Use those as input parameters to a function that generates the list of files to process and then calls your existing code (wrapped in a new function) and possibly provide default values for all/some of the parameters. Another option is to add the start/end dates to the calibration file if you have control of that, but personally I'd stick with input parameters... Many thanks Alan for your reply. I have added start and end dates as part of the header information for the calibration files in the date format: 01/07/2011. So, I now need to write some code to take this into consideration, any suggestions? -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ __**_ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/**mailman/listinfo/tutorhttp://mail.python.org/mailman/listinfo/tutor ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Help for Python Beginner with extracting and manipulating data from thousands of ASCII files
On Mon, Oct 1, 2012 at 1:16 AM, Dave Angel d...@davea.name wrote:
On 09/30/2012 06:07 PM, Cecilia Chavana-Bryant wrote:
Hola again Python Tutor!
With a friend's help I have the following code to extract reflectance
data
from an ASCII data file, do a bit of data manipulation to calibrate the
data and then write the calibrated file into an out file.
import numpy
# import glob - use if list_of_files is used
dataFile = 1SH0109.001.txt
#list_of_files = glob.glob('./*.txt') to replace dataFile to search for
all
text files in ASCII_files folder?
First, an important observation. This code has no functions defined in
it. Thus it's not reusable. So every time you make a change, you'll be
breaking the existing code and then trying to make the new version work.
The decision of one file versus many is usually handled by writing a
function that deals with one file. Test it with a single file. Then
write another function that uses glob to build a list of files, and call
the original one in a loop.
As you work on it, you should discover that there are a half dozen other
functions that you need, rather than one big one.
Many thanks for this advise this helps me to get started with trying to
write functions for the different procedures and then think about many
files.
caliFile1 = Cal_File_P17.txt # calibration file to be used on data
files
created from July to 18 Sep
caliFile2 = Cal_File_P19.txt # calibration file to be used on data
files
created from 19 Sep onwards
outFile = Cal_ + dataFile # this will need to change if list_of_files
is
used
fileDate = data[6][16:26] # location of the creation date on the data
files
Show us the full traceback from the runtime error you get on this line.
The option of using 2 different calibration files is an idea that I
haven't tested yet as I am a bit lost in how to do this. I have gotten as
far as adding start and end dates on both calibration files as part of the
header information for each file.
#extract data from data file
fdData = open(dataFile,rt)
data = fdData.readlines()
fdData.close()
#extract data from calibration file
fdCal = open(caliFile,rt)
Show us the full traceback from the runtime error here, as well.
In the original code which uses only one calibration file this and the
rest of the code works without error.
calibration = fdCal.readlines()
fdCal.close()
#create data table
k=0 #just a counter
dataNum = numpy.ndarray((2151,2))
#the actual data (the numbers) in the data file begin at line 30
for anItem in data[30:]:
theNums = anItem.replace(\r\n,).split(\t)
dataNum[k,0] = int(theNums[0])
dataNum[k,1] = float(theNums[1])
k+=1 #advance the counter
#create the calibration table
k = 0
calNum = numpy.ndarray((2151,2))
for anItem in calibration[5:]:
theNums = anItem.replace(\r\n,).split(\t)
calNum[k,0] = int(theNums[0])
calNum[k,1] = float(theNums[1])
k+=1
#calibrate the data
k=0
calibratedData = numpy.ndarray((2151,2))
for aNum in dataNum:
calibratedData[k,0] = aNum[0] #first column is the wavelength
calibratedData[k,1] = (aNum[1] * dataNum[k,1]) * 100.0 #second column
is the measurement to be calibrated.
k+=1
#write the calibrated data
fd = open(outFile,wt)
Error traceback ?
#prior to writing the calibrated contents, write the headers for data
files
and calibration files
fd.writelines(data[0:30])
fd.writelines(calibration[0:5])
for aNum in calibratedData:
#Write the data in the file in the following format:
# An integer with 3 digits, tab character, Floating point
number
fd.write(%03d\t%f\n % (aNum[0],aNum[1]))
#close the file
fd.close()
Are the individual files small? By doing readlines() on them, you're
assuming you can hold all of both the data file and the calibration file
in memory.
Both the calibration and the data files are small. The original excel
calibration files have been saved as Tab delimited text files and the
data files are ASCII files with 2151 rows and 2 columns.
I have successfully calibrated one ASCII file at a time with this code.
Unless I'm missing something, this code does not run. I didn't try it,
though, just inspected it quickly.
However, I have 1,000s of files that I need to calibrate so I would like
some help to modify this code so it can:
1. Use one calibration file (Cal_FileP17.txt) on data files created from
July to the 18th Sep and a different calibration file (Cal_FileP19.txt)
for
data files created from the 19th of Sep onwards.
2. Find all the .txt files in a folder called ASCII_files, which is
subdivided into 12 different folders and calibrate all these files
I have googled and tried thinking about how to make changes and I've
managed to get myself a bit more confused. Thus, I would like some
guidance
on how to tackle/think about this process and how to get started
[Tutor] What are all those letters after terminal commands?
Hola, I'm going through the 'Command line crash course' by Zed Shaw, thanks to the people that recommended this book, its quite a good course, I can see what the author was going for with the title but if it wasn't for your recommendations, it would have put me off. At the beginning of Chapter 8 - Moving around (pushd, popd) on Source: 13 exercise 8 I found this command: mkdir -p i/like/icecream. I am guessing that the -p stands for directory path? I have seen other such letters sometimes with or without the ' - ' before them (I think) in commands so my question is, what are these letters for? what are they called? and could someone please point me to where I can find a list of these with descriptions of what they do. I have tried googling with no positive results as I don't know what they are called or I get just the information for the command they are used with. Many thanks in advance for the help, Cecilia ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
[Tutor] Hello Python Tutor - help please!
Dear all, I am just returning to my doctoral studies after a 7-month medical leave and desperately trying to catch up for lost time. I am COMPLETELY new to programming, well, I did try learning C for 3 weeks 3 yrs ago (with very little success) but had to stop and then spent 2 years in the Amazon climbing trees (lots more enjoyable than learning to programme!) and collecting loads of field data that I now need to post-process and analyse. By the way, the 3 weeks I spent trying to learn C really ended up being spent trying to get to grips with using a terminal for the first time in my life. Since getting back to work, I was advised to try learning Python instead of C as it is a much easier first language to learn. I have been trying, but again, to not great success. I started following A Primer on Scientific programming with Python but I kept getting lost and stuck, specially on the exercises. I have also been advised that I should not try to learn programming by following guides but by trying to write the programmes I need to analyse my data. Although I can understand the logic behind this last bit of advise (it gives context and direction to the learning process) I have also gotten stuck trying this approach as I do not know how to programme!. Thus, I was hoping that some of you can remember how you got started and point me towards any really good interactive learning guides/materials and/or have a good learning strategy for a complete beginner. I have searched the web and was overwhelmed by choice of tutorials and guides. I have skimmed through a couple of tutorials but then fail to see how all that relates to my own work and I get stuck with what seems like basic important concepts so I don't progress. I then think I should try to make some progress with my own data analysing and go back to trying to learn to write a programme for my specific needs and get stuck again because this requires more advanced skills then the basic programming concepts I have been reading about on the learning guides. So, I am now feeling VERY frustrated and have no idea what on Earth I am doing! Can anyone please offer guidance in my learning process? I don't know how and what I should be spending my time learning first and/or if I should focus my learning towards the skill areas I will require to write my specific programmes, although I have no idea what these are. I would like advise on finding some really good interactive(let you know if your solution to an exercise is correct or not) and or video tutorials that give you feedback on the solutions you write to exercises. Many thanks in advance for all your help, it will be much appreciated! Cecilia Chavana-Bryant DPhil Candidate - Remote sensing and tropical phenology Environmental Change Institute School of Geography and the Environment University of Oxford South Parks Road, Oxford, OX1 3QY Web: http://www.eci.ox.ac.uk/teaching/doctoral/chavanabryantcecilia.php Tel Direct: +44 (0)1865 275861 Fax: +44 (0)1865 275885 ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Hello Python Tutor - help please!
Hola Bill, Many thanks for your reply to my post, you seem to understand the predicament I am in very well. Unfortunately, I am currently working from home and do not have someone close by to help and this is why I came to this space. This is also why I asked for advise about interactive tutorials or video tutorials. I have found that I keep getting lost with the more traditional tutorials where you just read and then do exercises. Following the guide I mentioned on my initial post I got through the first 2 chapters but I found them quite hard going. I don't know if this makes me not a complete beginner but I certainly do not feel like I learned much from reading them. Maybe it is the trying to learn the computer ecosystem of terminal commands at the same time that is making this learning process so tough. With respect to my field data, during my 2 yrs of fieldwork I collected a large amount of data which is currently stored in excel files. My research involves remote sensing (data from Earth-observation satellites) and I work with data from the MODIS NASA satellite which monitors the health of forest canopies using reflectance data. My research is based in the Amazon. I have collected field data to monitor the leaf dynamics of canopy leaves during the dry season. Dry season is the time of year when many tropical trees change their old leaves for new ones. New leaves are more photosynthetically active (absorb more carbon from and release more oxygen into the atmosphere) so the leaf exchange of such a large forest region as the Amazon can have huge effects on regional and global carbon and water cycles and thus on global climate (apologies if I'm giving you loads more information than you need or requested?!). My data involves a large amount of data on leaf demography (we demographically surveyed more than 120,000 leaves), and thousands of morphological and reflectance measurements. I will have to reorganise this data and create a few easily manipulable datasets so I can sort data according to leaf age, canopy position, date, etc. Then I will have to do statistical analyses on the data. I will also have to model some of the data. Many thanks for taking the time to respond to my post so comprehensively and for your good wishes. Cecilia Chavana-Bryant DPhil Candidate - Remote sensing and tropical phenology Environmental Change Institute School of Geography and the Environment University of Oxford South Parks Road, Oxford, OX1 3QY Web: http://www.eci.ox.ac.uk/teaching/doctoral/chavanabryantcecilia.php Tel Direct: +44 (0)1865 275861 Fax: +44 (0)1865 275885 From: William R. Wing (Bill Wing) [w...@mac.com] Sent: 22 August 2012 15:17 To: Cecilia Chavana-Bryant Cc: William R. Wing (Bill Wing) Subject: Re: [Tutor] Hello Python Tutor - help please! On Aug 22, 2012, at 6:10 AM, Cecilia Chavana-Bryant cecilia.chavana-bry...@ouce.ox.ac.ukmailto:cecilia.chavana-bry...@ouce.ox.ac.uk wrote: Dear all, I am just returning to my doctoral studies after a 7-month medical leave and desperately trying to catch up for lost time. I am COMPLETELY new to programming, well, I did try learning C for 3 weeks 3 yrs ago (with very little success) but had to stop and then spent 2 years in the Amazon climbing trees (lots more enjoyable than learning to programme!) and collecting loads of field data that I now need to post-process and analyse. By the way, the 3 weeks I spent trying to learn C really ended up being spent trying to get to grips with using a terminal for the first time in my life. Could you say a few words about what the field data is, and how you hope to analyze it. That is, are you headed in the direction of plotting species density on maps, or the time evolution of something, or doing statistics? Since getting back to work, I was advised to try learning Python instead of C as it is a much easier first language to learn. I have been trying, but again, to not great success. I started following A Primer on Scientific programming with Python but I kept getting lost and stuck, specially on the exercises. I have also been advised that I should not try to learn programming by following guides but by trying to write the programmes I need to analyse my data. Although I can understand the logic behind this last bit of advise (it gives context and direction to the learning process) I have also gotten stuck trying this approach as I do not know how to programme!. Thus, I was hoping that some of you can remember how you got started and point me towards any really good interactive learning guides/materials and/or have a good learning strategy for a complete beginner. I have searched the web and was overwhelmed by choice of tutorials and guides. I have skimmed through a couple of tutorials but then fail to see how all that relates to my own work and I get stuck with what seems like basic important concepts so I don't progress. I then think I
[Tutor] Thanks!!
Hola, Just a big THANK YOU!!! to everyone that has replied to my post. I have been so confused about how to get started and since I am working from home, it has been a very frustrating and lonely experience so far. Many, many thanks for your empathy, encouragement and availability!! Cecilia Chavana-Bryant DPhil Candidate - Remote sensing and tropical phenology Environmental Change Institute School of Geography and the Environment University of Oxford South Parks Road, Oxford, OX1 3QY Web: http://www.eci.ox.ac.uk/teaching/doctoral/chavanabryantcecilia.php Tel Direct: +44 (0)1865 275861 Fax: +44 (0)1865 275885 ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
[Tutor] Hello Python Tutor - help please!
Steven, (now from my new account without all the long-winded signature) can files be attached to posts in this forum? Cecilia ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
