Josh Rosenberg added the comment:
Note: While this particular use case wouldn't be fixed (map returns in order,
not as completed), applying the fix from #29842 would make many similar use
cases both simpler to implement and more efficient/possible.
That said, no action has been taken on
Tim Peters added the comment:
Note that you can consume multiple gigabytes of RAM with this simpler program
too, and for the same reasons:
"""
import concurrent.futures as cf
bucket = range(30_000_000)
def _dns_query(target):
from time import sleep
sleep(0.1)
def run():
with
Tim Peters added the comment:
If your `bucket` has 30 million items, then
for element in bucket:
executor.submit(kwargs['function']['name'], element, **kwargs)
is going to create 30 million Future objects (and all the under-the-covers
objects needed to manage their concurrency)
Dem added the comment:
It seems that even without the as_completed call it has the same problem.
```
# -*- coding: utf-8 -*-
import dns.resolver
import concurrent.futures
from pprint import pprint
from json import json
bucket = json.load(open('30_million_strings.json','r'))
def
New submission from Dem :
I have a list of 30 million strings, and I want to run a dns query to all of
them. I do not understand how this operation can get memory intensive. I would
assume that the threads would exit after the job is done, and there is also a
timeout of 1 minute as well
