On Mon, Jun 24, 2019, at 17:24, Chengi Liu wrote:
> How do I use more multiple CPU cores? I guess atleast the fact that twisted
> app wont be blocking is good, but what would be a good way to use multiple
> cores.
It *really* depends on what is the function that is CPU heavy, and why it is
CPU heavy.
You have a lot of options here, and they're all subtly different.
But I'll start with noting that this is *no longer* a Twisted question: using
multiple cores from Python is genuinely a problem for which there is no single
good solution, just a number of different ones with trade-offs. As I mentioned,
you can use `dask`. I think that might be the simplest solution for you -- dask
has a lot of good tutorials, and if your only goal is using multiple CPUs, it
has an "auto-setup". Another option is to see if you can use a library that
natively releases the Global Interpreter Lock. If, for example, your function
is numeric computation heavy, than numpy does release the GIL. You can also use
ampoule, though the set-up becomes non-trivial -- you will need to either
change your process initialization or maintain a process pool from the main
process. Either option can become non-trivial fast. Finally, the option of
writing the CPU heavy code in Cython, and then releasing the GIL explicitly
around the area it is not needed, exists.
To sum up:
* Dask
* Something that releases the GIL, like numpy
* Setting up an ampoule process pool
* Move CPU heavy logic to Cython, and release GIL explicitly
As I said, none of these options are really Twisted-related (the only one
vaguely connected is the ampoule one, but even then, the fact that you'll use
Twisted to manage the pool is incidental). There are even more exotic options,
of course, but I think this e-mail might already be confusing enough.
Moshe Z.
_______________________________________________
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
