Danny,
What is the best way to track the time uses in I/O and CPU. Here is the
code. It checks some text files for a user name and collects memory and
inode usage then adds them together and checks against a set limit...if the
limit is reached it calls a mail script to send an email warning. The
program is used like a script with variables passed from the command line.
(maybe I should just write this in bash?)
import sys
import time
import glob
import os.path
import subprocess
class memoryUsage:
def
__init__(self,userName,gpfs,emailAddress,memLimit=500000,inodeLimit=20000,
searchMonth="None",searchDay="None",searchYear="None"):
self.BASEDIR = "/GPFS/Usage"
self.gpfs=gpfs.lower()
self.emailAddress=emailAddress
self.memLimit=memLimit
self.inodeLimit=inodeLimit
self.userName=userName
self.searchMonth=searchMonth
self.searchDay=searchDay
self.searchYear=searchYear
self.memTotal = 0
self.inodeTotal = 0
self.gpfsList = []
date = time.strftime("%b %d %Y")
print "date =" ,date
date = date.split()
if self.searchMonth == "None":
self.searchMonth=date[0]
if self.searchDay == "None":
self.searchDay = date[1]
if self.searchYear == "None":
self.searchYear = date[2]
if emailAddress == "None":
print "You must enter an email (-e)"
print "ex. memoryCheck.py -n ertlj -g all -e
[EMAIL PROTECTED]"
sys.exit()
def getGPFSList(self):
## if self.gpfs is set to all get all availabel otherwise just add
the one asked for
if self.gpfs == "all":
self.gpfsPathList = glob.glob(self.BASEDIR +"/gpfs*")
for each in self.gpfsPathList:
(base,gpfs) = os.path.split(each)
self.gpfsList.append(gpfs)
else:
self.gpfsList.append(self.gpfs)
def makeFilePath(self,gpfs):
## make the full path to the file tha contans the memory and inode
usage for each node
self.filePath =
os.path.join(self.BASEDIR,gpfs,self.searchYear,
self.searchMonth,self.searchDay)
def extractUserData(self):
## look in each file and search for the name...if found get the
memory and inode usage
## and return them
print self.filePath
fullList = open(self.filePath,"r").readlines()
for line in fullList:
#print "line", line
singleList = line.split()
try:
if singleList[1] == self.userName:
print line
return singleList[2],singleList[3]
except:
pass
return 0,0
def add(self,memAmount,inodeAmount):
self.memTotal = self.memTotal + int(memAmount)
self.inodeTotal = self.inodeTotal + int(inodeAmount)
def sendEmail(self,message):
if self.emailAddress != "None":
messagePath="/home/ertlj/localBin/blank"
emailComand = "/u/curr/bin/smail.pl -f %s -t %s -s '%s'" %
(messagePath,self.emailAddress,message)
p = subprocess.Popen(emailComand, shell=True)
self.sts = os.waitpid(p.pid, 0)
else:
sys.exit("No valid email address given...can not email size
warning")
if __name__ == "__main__":
from optparse import OptionParser
parser = OptionParser()
parser.add_option("-n", "--userName",
default=os.getenv("LOGNAME","None"), help="The user name to look for")
parser.add_option("-g", "--gpfs", default="all", help="Enter the gpfs
you want to search or 'all'")
parser.add_option("-e", "--email", default="None", help="The email the
report if any will be sent to")
parser.add_option("-l", "--memLimit", default=5000000, help="The memory
size in KB that will triger a report")
parser.add_option("-i", "--inodeLimit",default=50000, help="The inode
size that will trigger a report")
parser.add_option("-d", "--day", default="None", help="The day of the
month you want to check")
parser.add_option("-m", "--month",default="None" ,help="The month of the
year you want to check")
parser.add_option("-y", "--year",default="None", help="The year you want
to check")
(options, args) = parser.parse_args()
myUse =
memoryUsage(options.userName,options.gpfs,options.email,options.memLimit,
options.inodeLimit,options.month,options.day,options.year)
myUse.getGPFSList()
for each in myUse.gpfsList:
myUse.makeFilePath(each)
(memAmount,inodeAmount) = myUse.extractUserData()
myUse.add(memAmount,inodeAmount)
print "Your memory usage is %s KB and your inode usage is %s" %
(myUse.memTotal,myUse.inodeTotal)
print "Your memory limit is %s KB and your inode limit is %s" %
(myUse.memLimit, myUse.inodeLimit)
if myUse.memLimit < myUse.memTotal or myUse.inodeLimit <
myUse.inodeTotal:
print "You have excedded your limit"
myUse.sendEmail("%s memory/inode limit reached on gpfs " %
myUse.userName)
-----Original Message-----
From: Danny Yoo [mailto:[EMAIL PROTECTED]
Sent: Tuesday, May 02, 2006 1:32 PM
To: Ertl, John
Cc: tutor@python.org
Subject: Re: [Tutor] question about run time
> I have been using python for sometime...and occasionally I noticed
> significant delay before the code would run but unitl now I have been
> able to write it off to other things. Now I have a short script that I
> wrote to check some files and print out a few lines.
>
> I have noticed that usually the first time I fire it up in the morning
> or after a long time of not running it, it takes 10-15 seconds to run
> and the output to the screen is very slow...maybe 1 second per line.
> If I run it soon after that it runs and the output is on the screen in
> less then a second. I would think this has to do with compiling but I
> am not sure. Any ideas how to speed this up?
>
> I am running python 2.4 on a RHE3.0 cluster.
^^^^^^^^^^^^^^
Hi John,
One thing to check is to see if the program is spending the majority of
its time doing input and output (I/O Bound), or if it's really doing heavy
computations (CPU bound). Knowing this might provide clues as to why
you're seeing this kind of jerky performance.
Also, you may want to check with your cluster folks on the possible
effects the cluster's architecture may have on program startup. You're
running on a slightly specialized platform, so I wouldn't be surprised if
the cluster architecture is contributing something special.
Finally, if you want to share that script for people to comment on, that
might help.
Good luck!
_______________________________________________
Tutor maillist - Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor
