[Tutor] help for a beginner
Hi, I am a beginner to progrmming and want to learn basic python. I am a scientist (with less math background) with absolutely no programming experience. Are there any web based tutorials/videos/books to learn python. Any help is deeply appreciated, thanks Murail -- Murali Dharan Bashyam, PhD, MNAScI Staff Scientist and Chief, Laboratory of Molecular Oncology, Centre for DNA Fingerprinting and Diagnostics (CDFD), Tuljaguda complex, Nampally, Hyderabad 51, INDIA Ph: 91-40-24749383 Fax: 91-40-24749448 ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] help for a beginner
On Sat, Jan 12, 2013 at 12:28 AM, MDB bashya...@gmail.com wrote: Hi, I am a beginner to progrmming and want to learn basic python. I am a scientist (with less math background) with absolutely no programming experience. Are there any web based tutorials/videos/books to learn python. The best book for beginners in my experience is Python for Absolute Beginners, I liked it's approach in making learning a bit fun. A good online resource is http://www.alan-g.me.uk/ Any help is deeply appreciated, thanks Murail -- Murali Dharan Bashyam, PhD, MNAScI Staff Scientist and Chief, Laboratory of Molecular Oncology, Centre for DNA Fingerprinting and Diagnostics (CDFD), Tuljaguda complex, Nampally, Hyderabad 51, INDIA Ph: 91-40-24749383 Fax: 91-40-24749448 ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor -- Vishwajeet Singh +91-9657702154 | dextrou...@gmail.com | http://bootstraptoday.com Twitter: http://twitter.com/vishwajeets | LinkedIn: http://www.linkedin.com/in/singhvishwajeet ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] help for a beginner
Murail, Check out Udacity.com and the CS101 course. Great video lectures reinforced by homework and problems (with answers) that you can do yourself. Also has a very good forum and active user base to ask questions. It is a good starting point for a beginner and teaches the basics behind how to build a simple web crawler using Python. Cheers, Graham On Fri, Jan 11, 2013 at 1:58 PM, MDB bashya...@gmail.com wrote: Hi, I am a beginner to progrmming and want to learn basic python. I am a scientist (with less math background) with absolutely no programming experience. Are there any web based tutorials/videos/books to learn python. Any help is deeply appreciated, thanks Murail -- Murali Dharan Bashyam, PhD, MNAScI Staff Scientist and Chief, Laboratory of Molecular Oncology, Centre for DNA Fingerprinting and Diagnostics (CDFD), Tuljaguda complex, Nampally, Hyderabad 51, INDIA Ph: 91-40-24749383 Fax: 91-40-24749448 ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Help for Python Beginner with extracting and manipulating data from thousands of ASCII files
Hi Cecilia, I'm sending this again as the first message was sent only to you (I hadn't realised that your own message was sent only to me as well). If you want to reply please reply-all to this message. On 1 October 2012 17:42, Cecilia Chavana-Bryant cecilia.chav...@gmail.com wrote: On Mon, Oct 1, 2012 at 11:02 AM, Oscar Benjamin oscar.j.benja...@gmail.com wrote: On Sep 30, 2012 11:10 PM, Cecilia Chavana-Bryant cecilia.chav...@gmail.com wrote: fileDate = data[6][16:26] # location of the creation date on the data files What format does fileDate have? I guess it's a string of text from the file. If you can convert it to a datetime (or date) object it will be easy to compare with the dates as required for your calibration file. Can you show us how it looks e.g. '12-Nov-2012' or '12/11/12' or something else? Date is in the following format: dd/mm/ The standard way to work with dates is to turn the date strings into Python datetime objects. You can read about those here: http://docs.python.org/library/datetime.html datetime objects can be create directly: from datetime import datetime start_date = datetime(year=2012, month=11, day=3) print start_date 2012-11-03 00:00:00 You can also create them from a string: datestring = '10/11/2012' experiment_date = datetime.strptime(datestring, '%d/%m/%Y') print experiment_date 2012-11-10 00:00:00 Once you have two datetime objects you can compare them directly: experiment_date start_date True print experiment_date - start_date 7 days, 0:00:00 Using this you can check whether the date from the data file is in between the start and end dates for each of the calibration files and then choose which calibration file to use. Oscar ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Help for Python Beginner with extracting and manipulating data from thousands of ASCII files
On Sep 30, 2012 11:10 PM, Cecilia Chavana-Bryant
cecilia.chav...@gmail.com wrote:
Hola again Python Tutor!
Hi Cecilia
With a friend's help I have the following code to extract reflectance data
from an ASCII data file, do a bit of data manipulation to calibrate the data
and then write the calibrated file into an out file.
import numpy
# import glob - use if list_of_files is used
dataFile = 1SH0109.001.txt
#list_of_files = glob.glob('./*.txt') to replace dataFile to search for all
text files in ASCII_files folder?
caliFile1 = Cal_File_P17.txt # calibration file to be used on data files
created from July to 18 Sep
caliFile2 = Cal_File_P19.txt # calibration file to be used on data files
created from 19 Sep onwards
outFile = Cal_ + dataFile # this will need to change if list_of_files is
used
fileDate = data[6][16:26] # location of the creation date on the data files
The variable data used in the line above is not created until the
lines below run. I think you need to move this line down. What format
does fileDate have? I guess it's a string of text from the file. If
you can convert it to a datetime (or date) object it will be easy to
compare with the dates as required for your calibration file. Can you
show us how it looks e.g.
'12-Nov-2012'
or
'12/11/12'
or something else?
#extract data from data file
fdData = open(dataFile,rt)
data = fdData.readlines()
fdData.close()
Python has a slightly better way of writing code like this:
with open(dataFile, 'rt') as fdata:
data = fdata.readlines()
This way you don't need to remember to close the file. In fact Python
will even remember to close it if there is an error.
#extract data from calibration file
fdCal = open(caliFile,rt)
calibration = fdCal.readlines()
fdCal.close()
Where is caliFile set? If your going to load all the data files you
might as well load both calibration files here at the beginning.
#create data table
k=0 #just a counter
dataNum = numpy.ndarray((2151,2))
Does dataNum store integers or floating point numbers? Numpy won't let
you do both in the same array. You should always specify the type of
the numpy array that you want to create:
dataNum = numpy.ndarray((2152, 2), float)
or
dataNum = numpy.ndarray((2152, 2), int)
As it happens you are creating an array floats. That means that when
you try to store an integer in the array below it gets converted to a
float.
#the actual data (the numbers) in the data file begin at line 30
for anItem in data[30:]:
theNums = anItem.replace(\r\n,).split(\t)
dataNum[k,0] = int(theNums[0])
dataNum[k,1] = float(theNums[1])
k+=1 #advance the counter
You should look into using numpy.fromfile. This function is
specifically designed for this purpose.
For example:
with open(dataFile) as fdata:
header_lines = [fdata.readline() for _ in range(30)]
dataNum = numpy.fromfile(fdata, float, sep='\t')
Oscar
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Help for Python Beginner with extracting and manipulating data from thousands of ASCII files
On Mon, Oct 1, 2012 at 12:33 AM, Alan Gauld alan.ga...@btinternet.comwrote: On 30/09/12 23:07, Cecilia Chavana-Bryant wrote: Hola again Python Tutor! With a friend's help I have the following code to extract reflectance data from an ASCII data file, do a bit of data manipulation to calibrate the data and then write the calibrated file into an out file. snip I have successfully calibrated one ASCII file at a time with this code. However, I have 1,000s of files that I need to calibrate so I would like some help to modify this code so it can: 1. Use one calibration file (Cal_FileP17.txt) on data files created from July to the 18th Sep and a different calibration file (Cal_FileP19.txt) for data files created from the 19th of Sep onwards. 2. Find all the .txt files in a folder called ASCII_files, which is subdivided into 12 different folders and calibrate all these files Number 2 is easier to solve and the os.walk() and glob.glob() functions should provide all the tools you need. Number 1 is more tricky since there is no obvious way to determine the arbitrary start/stop dates you specify. So I'd suggest you need to generalise the requirement to take a start/stop date as well as the calibration file name and the input data file pattern. Use those as input parameters to a function that generates the list of files to process and then calls your existing code (wrapped in a new function) and possibly provide default values for all/some of the parameters. Another option is to add the start/end dates to the calibration file if you have control of that, but personally I'd stick with input parameters... Many thanks Alan for your reply. I have added start and end dates as part of the header information for the calibration files in the date format: 01/07/2011. So, I now need to write some code to take this into consideration, any suggestions? -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ __**_ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/**mailman/listinfo/tutorhttp://mail.python.org/mailman/listinfo/tutor ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Help for Python Beginner with extracting and manipulating data from thousands of ASCII files
On Mon, Oct 1, 2012 at 1:16 AM, Dave Angel d...@davea.name wrote:
On 09/30/2012 06:07 PM, Cecilia Chavana-Bryant wrote:
Hola again Python Tutor!
With a friend's help I have the following code to extract reflectance
data
from an ASCII data file, do a bit of data manipulation to calibrate the
data and then write the calibrated file into an out file.
import numpy
# import glob - use if list_of_files is used
dataFile = 1SH0109.001.txt
#list_of_files = glob.glob('./*.txt') to replace dataFile to search for
all
text files in ASCII_files folder?
First, an important observation. This code has no functions defined in
it. Thus it's not reusable. So every time you make a change, you'll be
breaking the existing code and then trying to make the new version work.
The decision of one file versus many is usually handled by writing a
function that deals with one file. Test it with a single file. Then
write another function that uses glob to build a list of files, and call
the original one in a loop.
As you work on it, you should discover that there are a half dozen other
functions that you need, rather than one big one.
Many thanks for this advise this helps me to get started with trying to
write functions for the different procedures and then think about many
files.
caliFile1 = Cal_File_P17.txt # calibration file to be used on data
files
created from July to 18 Sep
caliFile2 = Cal_File_P19.txt # calibration file to be used on data
files
created from 19 Sep onwards
outFile = Cal_ + dataFile # this will need to change if list_of_files
is
used
fileDate = data[6][16:26] # location of the creation date on the data
files
Show us the full traceback from the runtime error you get on this line.
The option of using 2 different calibration files is an idea that I
haven't tested yet as I am a bit lost in how to do this. I have gotten as
far as adding start and end dates on both calibration files as part of the
header information for each file.
#extract data from data file
fdData = open(dataFile,rt)
data = fdData.readlines()
fdData.close()
#extract data from calibration file
fdCal = open(caliFile,rt)
Show us the full traceback from the runtime error here, as well.
In the original code which uses only one calibration file this and the
rest of the code works without error.
calibration = fdCal.readlines()
fdCal.close()
#create data table
k=0 #just a counter
dataNum = numpy.ndarray((2151,2))
#the actual data (the numbers) in the data file begin at line 30
for anItem in data[30:]:
theNums = anItem.replace(\r\n,).split(\t)
dataNum[k,0] = int(theNums[0])
dataNum[k,1] = float(theNums[1])
k+=1 #advance the counter
#create the calibration table
k = 0
calNum = numpy.ndarray((2151,2))
for anItem in calibration[5:]:
theNums = anItem.replace(\r\n,).split(\t)
calNum[k,0] = int(theNums[0])
calNum[k,1] = float(theNums[1])
k+=1
#calibrate the data
k=0
calibratedData = numpy.ndarray((2151,2))
for aNum in dataNum:
calibratedData[k,0] = aNum[0] #first column is the wavelength
calibratedData[k,1] = (aNum[1] * dataNum[k,1]) * 100.0 #second column
is the measurement to be calibrated.
k+=1
#write the calibrated data
fd = open(outFile,wt)
Error traceback ?
#prior to writing the calibrated contents, write the headers for data
files
and calibration files
fd.writelines(data[0:30])
fd.writelines(calibration[0:5])
for aNum in calibratedData:
#Write the data in the file in the following format:
# An integer with 3 digits, tab character, Floating point
number
fd.write(%03d\t%f\n % (aNum[0],aNum[1]))
#close the file
fd.close()
Are the individual files small? By doing readlines() on them, you're
assuming you can hold all of both the data file and the calibration file
in memory.
Both the calibration and the data files are small. The original excel
calibration files have been saved as Tab delimited text files and the
data files are ASCII files with 2151 rows and 2 columns.
I have successfully calibrated one ASCII file at a time with this code.
Unless I'm missing something, this code does not run. I didn't try it,
though, just inspected it quickly.
However, I have 1,000s of files that I need to calibrate so I would like
some help to modify this code so it can:
1. Use one calibration file (Cal_FileP17.txt) on data files created from
July to the 18th Sep and a different calibration file (Cal_FileP19.txt)
for
data files created from the 19th of Sep onwards.
2. Find all the .txt files in a folder called ASCII_files, which is
subdivided into 12 different folders and calibrate all these files
I have googled and tried thinking about how to make changes and I've
managed to get myself a bit more confused. Thus, I would like some
guidance
on how to tackle/think about this process and how to get started.
Re: [Tutor] Help for Python Beginner with extracting and manipulating data from thousands of ASCII files
On 30/09/12 23:07, Cecilia Chavana-Bryant wrote: Hola again Python Tutor! With a friend's help I have the following code to extract reflectance data from an ASCII data file, do a bit of data manipulation to calibrate the data and then write the calibrated file into an out file. snip I have successfully calibrated one ASCII file at a time with this code. However, I have 1,000s of files that I need to calibrate so I would like some help to modify this code so it can: 1. Use one calibration file (Cal_FileP17.txt) on data files created from July to the 18th Sep and a different calibration file (Cal_FileP19.txt) for data files created from the 19th of Sep onwards. 2. Find all the .txt files in a folder called ASCII_files, which is subdivided into 12 different folders and calibrate all these files Number 2 is easier to solve and the os.walk() and glob.glob() functions should provide all the tools you need. Number 1 is more tricky since there is no obvious way to determine the arbitrary start/stop dates you specify. So I'd suggest you need to generalise the requirement to take a start/stop date as well as the calibration file name and the input data file pattern. Use those as input parameters to a function that generates the list of files to process and then calls your existing code (wrapped in a new function) and possibly provide default values for all/some of the parameters. Another option is to add the start/end dates to the calibration file if you have control of that, but personally I'd stick with input parameters... -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Help for Python Beginner with extracting and manipulating data from thousands of ASCII files
On 09/30/2012 06:07 PM, Cecilia Chavana-Bryant wrote:
Hola again Python Tutor!
With a friend's help I have the following code to extract reflectance data
from an ASCII data file, do a bit of data manipulation to calibrate the
data and then write the calibrated file into an out file.
import numpy
# import glob - use if list_of_files is used
dataFile = 1SH0109.001.txt
#list_of_files = glob.glob('./*.txt') to replace dataFile to search for all
text files in ASCII_files folder?
First, an important observation. This code has no functions defined in
it. Thus it's not reusable. So every time you make a change, you'll be
breaking the existing code and then trying to make the new version work.
The decision of one file versus many is usually handled by writing a
function that deals with one file. Test it with a single file. Then
write another function that uses glob to build a list of files, and call
the original one in a loop.
As you work on it, you should discover that there are a half dozen other
functions that you need, rather than one big one.
caliFile1 = Cal_File_P17.txt # calibration file to be used on data files
created from July to 18 Sep
caliFile2 = Cal_File_P19.txt # calibration file to be used on data files
created from 19 Sep onwards
outFile = Cal_ + dataFile # this will need to change if list_of_files is
used
fileDate = data[6][16:26] # location of the creation date on the data files
Show us the full traceback from the runtime error you get on this line.
#extract data from data file
fdData = open(dataFile,rt)
data = fdData.readlines()
fdData.close()
#extract data from calibration file
fdCal = open(caliFile,rt)
Show us the full traceback from the runtime error here, as well.
calibration = fdCal.readlines()
fdCal.close()
#create data table
k=0 #just a counter
dataNum = numpy.ndarray((2151,2))
#the actual data (the numbers) in the data file begin at line 30
for anItem in data[30:]:
theNums = anItem.replace(\r\n,).split(\t)
dataNum[k,0] = int(theNums[0])
dataNum[k,1] = float(theNums[1])
k+=1 #advance the counter
#create the calibration table
k = 0
calNum = numpy.ndarray((2151,2))
for anItem in calibration[5:]:
theNums = anItem.replace(\r\n,).split(\t)
calNum[k,0] = int(theNums[0])
calNum[k,1] = float(theNums[1])
k+=1
#calibrate the data
k=0
calibratedData = numpy.ndarray((2151,2))
for aNum in dataNum:
calibratedData[k,0] = aNum[0] #first column is the wavelength
calibratedData[k,1] = (aNum[1] * dataNum[k,1]) * 100.0 #second column
is the measurement to be calibrated.
k+=1
#write the calibrated data
fd = open(outFile,wt)
Error traceback ?
#prior to writing the calibrated contents, write the headers for data files
and calibration files
fd.writelines(data[0:30])
fd.writelines(calibration[0:5])
for aNum in calibratedData:
#Write the data in the file in the following format:
# An integer with 3 digits, tab character, Floating point number
fd.write(%03d\t%f\n % (aNum[0],aNum[1]))
#close the file
fd.close()
Are the individual files small? By doing readlines() on them, you're
assuming you can hold all of both the data file and the calibration file
in memory.
I have successfully calibrated one ASCII file at a time with this code.
Unless I'm missing something, this code does not run. I didn't try it,
though, just inspected it quickly.
However, I have 1,000s of files that I need to calibrate so I would like
some help to modify this code so it can:
1. Use one calibration file (Cal_FileP17.txt) on data files created from
July to the 18th Sep and a different calibration file (Cal_FileP19.txt) for
data files created from the 19th of Sep onwards.
2. Find all the .txt files in a folder called ASCII_files, which is
subdivided into 12 different folders and calibrate all these files
I have googled and tried thinking about how to make changes and I've
managed to get myself a bit more confused. Thus, I would like some guidance
on how to tackle/think about this process and how to get started. Please, I
am not asking for someone to do my work and write the code for me, I would
like some guidance on how to approach this and get started.
Many thanks in advance for your help,
Cecilia
--
DaveA
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor
