On Sun, 25 May 2008 18:42:06 -0700, notnorwegian wrote:
> def scrapeSites(startAddress):
> site = startAddress
> sites = set()
> iterator = iter(sites)
> pos = 0
> while pos < 10:#len(sites):
> newsites = scrapeSite(site)
> joinSets(sites, newsites)
You change the size of "sites" here, which means you can no longer use
the iterator.
> wtf? im not multithreading or anything so how can the size change here?
How about:
def scrape_sites(start_address):
sites = set([start_address])
sites_scraped = set()
# This will run until it doesn't find any new sites, which may be
# a long time
while len(sites) > 0:
new_sites = set()
for site in sites:
new_sites.update(scrape_site(site))
sites_scraped.update(sites)
sites = new_sites.difference(sites_scraped)
return sites
--
http://mail.python.org/mailman/listinfo/python-list
