[
https://issues.apache.org/jira/browse/SAMZA-203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13944665#comment-13944665
]
Chris Riccomini commented on SAMZA-203:
---------------------------------------
Also, as an alternative fix, I found that simply setting the
systems.kafka.samza.fetch.threshold to 75000 made sure that there were always
messages buffered in the queue (i.e. BlockingEnvelopeMap.poll always
immediately returns data), and therefore BrokerProxy's Thread.sleeps never
throttle the processing.
As part of this, I noticed a bug in KafkaConfig:
{code}
val CONSUMER_FETCH_THRESHOLD = SystemConfig.SYSTEM_PREFIX +
".samza.fetch.threshold"
{code}
Should be:
{code}
val CONSUMER_FETCH_THRESHOLD = SystemConfig.SYSTEM_PREFIX +
"samza.fetch.threshold"
{code}
> Bad performance in BrokerProxy when restoring changelogs
> --------------------------------------------------------
>
> Key: SAMZA-203
> URL: https://issues.apache.org/jira/browse/SAMZA-203
> Project: Samza
> Issue Type: Bug
> Components: kv
> Affects Versions: 0.6.0
> Reporter: Chris Riccomini
> Fix For: 0.7.0
>
>
> As part of SAMZA-126, we introduced a Thread.sleep call in BrokerProxy's
> fetchMessages method. The goal was to skip fetch requests on SimpleConsumer
> when the topicAndPartitionsToFetch variable was empty. Since we had no
> topic/partitions to fetch, we slowed the thread down by calling
> Thread.sleep(sleepMSWhileNoTopicPartitions), which defaults to 1000ms.
> We now see that we are only getting about 1mb/s when restoring changelogs.
> This is very slow. Upon investigation, it appears that the BrokerProxy thread
> is sleeping 90% of the time during restore, and the main SamzaContainer
> thread is polling for more messages about 60% of the time.
> The reason for the poor restore performance is that the BrokerProxy sleeps
> for 1 second every time the message queue for the restore topic is not empty.
> Effectively, the proxy starts throttling the reads. If I comment out the
> Thread.sleep line in the BrokerProxy, I get about 64mb/s network usage on my
> loopback (one broker running locally), 10mb/s disk read, and 70mb/s disk
> write on my MacBook Air SSD--the write appears to be the bottleneck (since
> we're writing all the values to the LevelDB store). This is much much faster
> than before.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
