Hi!
I need to open about 1200 urls from my database. And to check that those
urls are really exists.
I used urllib2.urlopen for it. But it's a little bit slow. I thought that
it's might be a good idea to do it in a threads. So it can add some
performance to my code.
Unfortunately I can't get started with the treading module. There are no
simple examples in the python docs. Not sure how to start.
What I want to have:
for my list of urls I want to start few threads, each one of them should do
some validation.
My code without threads:
def queryDb(query, db_name = sys.argv[1]):
db = MySQLdb.connect(db = db_name, read_default_file="~/.my.cnf")
c = db.cursor()
c.execute(query)
return c
def openUrl(url):
error = ''
try:
urllib2.urlopen(url)
except (urllib2.URLError, urllib2.HTTPError,httplib.BadStatusLine),
error:
pass
return str(error)
def isValuePresent(dbField):
if str(dbField).find('None') is not -1:
return False
else:
return True
def displayResults(resultTable):
for query in resultTable:
print query
def processUrl(url, guid, parent_guid):
def testUrls():
error = ''
resultTable = []
errorTable = ['None']
products = queryDb(query = """SELECT url, guid, parent from product;""")
i = 0
for product in products.fetchall():
url = product[0]
guid = product[1]
parent_guid = product[2]
error_code = ''
#resultTable = {}
#print product[0]
if isValuePresent(url) is False:
#print "!!!!!", "this one is none", product[0]
"""Checking if this is a variant and it has parent with url"""
if isValuePresent(parent_guid) is True:
#print parent_guid, isValuePresent(parent_guid)
for parent in queryDb(query="""SELECT url from product where
guid = '%s';""" %parent_guid).fetchall():
#print parent[0]
'''Checking if parent product contains url'''
if isValuePresent(parent[0]) is True:
url = guid + "the parent url is: " + parent[0]
error = openUrl(parent[0])
else:
error = 'The variant with guid ' + guid + ' and his
parent do not contain urls!'
"""So the product is parent, and it's expected that it's child
will contain guids"""
_______________________________________________
Tutor maillist - Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor
