[ https://issues.apache.org/jira/browse/SPARK-9968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14696987#comment-14696987 ]
Apache Spark commented on SPARK-9968: ------------------------------------- User 'tdas' has created a pull request for this issue: https://github.com/apache/spark/pull/8204 > BlockGenerator lock structure can cause lock starvation of the block updating > thread > ------------------------------------------------------------------------------------ > > Key: SPARK-9968 > URL: https://issues.apache.org/jira/browse/SPARK-9968 > Project: Spark > Issue Type: Sub-task > Components: Streaming > Reporter: Tathagata Das > Assignee: Tathagata Das > > When the rate limiter is actually limiting the rate at which data is inserted > into the buffer, the synchronized block of BlockGenerator.addData stays > blocked for long time. This causes the thread switching the buffer and > generating blocks (synchronized with addData) to starve and not generate > blocks for seconds. The correct solution is to not block on the rate limiter > within the synchronized block for adding data to the buffer. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org