On Mon, Jun 24, 2019, at 17:24, Chengi Liu wrote:
> How do I use more multiple CPU cores? I guess atleast the fact that twisted
> app wont be blocking is good, but what would be a good way to use multiple
> cores.
It *really* depends on what is the function that is CPU heavy, and why it is
CPU
> On Jun 24, 2019, at 4:26 PM, Chengi Liu wrote:
>
> t = defer.ensureDeferred(network_get(i))
The meaning of "ensureDeferred" is, "ensure [that this thing, which may be a
coroutine or Deferred, is a] Deferred". You don't need to use it with things
decorated with
I think I got around the second problem (thanks for sharing the react
documentation)
How do I use more multiple CPU cores? I guess atleast the fact that twisted
app wont be blocking is good, but what would be a good way to use multiple
cores.
I do see that meejah suggested using multiple python
On Mon, Jun 24, 2019, at 16:27, Chengi Liu wrote:
> cpu_res *= yield *defer.*gatherResults*(cpus)
Remember: This will not block the reactor (Good!) but will still limit you to
one CPU core (in general, some caveats, but true.) If you are CPU bound, this
is woefully underutilizing modern CPU
The stack overflow link:
https://stackoverflow.com/questions/56696562/running-long-blocking-calculations-in-parallel-in-twisted
On Mon, Jun 24, 2019 at 7:26 PM Chengi Liu wrote:
> Thanks Moshe & Meejah & Gelin for the suggestions and advice. This is
> super helpful.
>
> I think, I am able to
Thanks Moshe & Meejah & Gelin for the suggestions and advice. This is super
helpful.
I think, I am able to move forward with this.
Let me just summarize this..
My usecase is.. fetch the data.. and then you assemble the data. Fetching
data is network bound, assembling data is like CPU bound.
Can
As a clarification to the above, parallelization of Python code across
cores is not unique to Twisted; all Python code has this same
limitation.
To use multiple cores with Python code, you need multiple Python
processes (as has been pointed out). One way to achieve this is to have
the multiple
Hi Cheng,
deferToThread *returns* a deferred, and sends the work to a thread pool. The
other two functions are useful for manipulating deferreds or other promises,
but will not send work to a separate thread, so they cannot be used instead.
The simple example is pretty simple, it's pretty much
Thanks for pointing that out. Is there a dead simply example on how to
parallelize a function using ampoule.
Again, I have a function.. I have a for loop.. the function takes in an
argument from loop.. but the for loop is extremely parallelizable.
Also, I was going thru this awesome twisted
Hi
If your purpose is merely not to block the reactor, you can run your
code in a separate thread by using deferToThread. If you want to benefit
from multi cores, you may consider use https://github.com/twisted/ampoule
or other similar tools.
Regards
gelin yan
Hi,
I tried posting on SO, but havent been able to solve the problem.
https://stackoverflow.com/questions/56696562/running-long-blocking-calculations-in-parallel-in-twisted
I have a very noob question.
I have a function, which takes a second or two to process (CPU bounded) and
then there is a
11 matches
Mail list logo
