> The number of nodes is very large: millions for sure, maybe tens > of millions. If considering (2), take note of my BOLD text above, which > means I can't remove nodes as I iterate through them in the main loop.
Since your use of 'node' is pretty vague and I don't have a good sense of what tests you are running and how long they would take, I'm only speculating, but a single loop might be the wrong way to go about that. If you are going to be frequently running tests and switching nodes on/off, have you considered a separate set of processes to do both? For example: A set of some number of "tester" threads, that loop through and test, recording thier results (somewhere). You could then have a separate loop that runs every so often, checks all the current test values, and runs through the nodes once, switching them on or off. I know it's not exactly what you asked, but depending on what your nodes are exactly, you can avoid a lot of other problems down the road. What if your single loop dies or gets hung on a test? With a separate approach, you'll have a lot more resilience too.. if there's some problem with a single tester or node, it won't keep the rest of the program from continuing to function. -- http://mail.python.org/mailman/listinfo/python-list
