Hello I have a working Python script that SELECTs rows from a database to fetch a company's name from a web-based database.
Since this list is quite big and the site is the bottleneck, I'd like to run multiple instances of this script, and figured a solution would be to pick rows at random from the dataset, check in my local database if this item has already been taken care of, and if not, download details from the remote web site. If someone's done this before, should I perform the randomization in the SQL query (SQLite using the APSW wrapper http://code.google.com/p/apsw/), or in Python? Thank you. Here's some simplified code: sql = 'SELECT id,label FROM companies WHERE activity=1' rows=list(cursor.execute(sql)) for row in rows: id = row[0] label = row[1] print strftime("%H:%M") url = "http://www.acme.com/details.php?id=%s" % id req = urllib2.Request(url, None, headers) response = urllib2.urlopen(req).read() name = re_name.search(response) if name: name = name.group(1) sql = 'UPDATE companies SET name=? WHERE id=?' cursor.execute(sql, (name,id) ) -- http://mail.python.org/mailman/listinfo/python-list