[
https://issues.apache.org/jira/browse/BEAM-690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16606366#comment-16606366
]
Vojtech Janota edited comment on BEAM-690 at 9/6/18 8:38 PM:
-------------------------------------------------------------
This issue affects us quite a lot and so I tried to implement the solution as
described above (PR linked). It doesn't seem to be possible to make the CPU
load really low unless the back off uses quite significant values in which case
the delay introduced by the throttling mechanism becomes a problem of its own.
After spending some time with this, I'm not sure if the issue can be solved
solely within the DirectRunner itself. For example in our scenario simple
mechanism that would allow to pause/resume the pipeline in the DirectRunner
would do the trick without having to sacrifice neither performance nor
resources.
was (Author: janotav):
This issue affects us quite a lot and so I tried to implement the solution as
described above (PR linked). It doesn't seem to be possible to make the CPU
load really low unless the back off uses quite significant values in which case
the delay introduced by the throttling mechanism becomes a problem of its own.
After spending some time with this, I'm not sure if the issue can be solved
solely within the DirectRunner itself. For example in our scenario simple
mechanism that would allow to pause/resume the pipeline in the DirectRunner
would do the trick without having to sacrifice either performance or resources.
> Backoff in the DirectRunner Monitor if no work is Available
> -----------------------------------------------------------
>
> Key: BEAM-690
> URL: https://issues.apache.org/jira/browse/BEAM-690
> Project: Beam
> Issue Type: Bug
> Components: runner-direct
> Reporter: Thomas Groh
> Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> When a Pipeline has no elements available to process, the Monitor Runnable
> will be repeatedly scheduled. Given that there is no work to be done, this
> will loop over the steps in the transform looking for timers, and prompt the
> sources to perform additional work, even though there is no work to be done.
> This consumes the entirety of a single core.
> Add a bounded backoff to rescheduling the monitor runnable if no work has
> been done since it last ran. This will reduce resource consumption on
> low-throughput Pipelines.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
