[
https://issues.apache.org/jira/browse/SAMZA-833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15046268#comment-15046268
]
Tao Feng commented on SAMZA-833:
--------------------------------
Just skim through the code for this bug. GroupByContainerCountFactory will take
config.getContainerCount into consideration. And in JobConfig.scala, if user
specifies either yarn.container.count or job.container.count larger than 1, the
getContainerCount will return larger than 1 value which causes the mismatch
case. Is the description correct?
A quick fix I could think of is in getContainerCount class only set the
count.toInt if it is using YarnJobFactory. If the fixed sounds reasonable, I
could provide a quick patch for this. Thanks.
> ProcessJob mishandling containers
> ---------------------------------
>
> Key: SAMZA-833
> URL: https://issues.apache.org/jira/browse/SAMZA-833
> Project: Samza
> Issue Type: Bug
> Reporter: Jake Maes
>
> As a result of SAMZA-465 and SAMZA-805, ProcessJobFactory now passes the full
> config to the ProcessJob and no longer forces the container count to 1. This
> causes the ProcessJob to actually read the container count config and if it
> is not 1, it produces some unexpected behavior.
> Specifically we've had reports of ProcessJobs dropping messages because the
> container count is > 1, so the grouper assigns partitions to more than 1
> container, but only one container actually runs.
> The goal of this ticket is to either force the container count to 1 for
> ProcessJob, or fix how multiple containers run with ProcessJob. But we should
> not allow the scenario where messages are dropped.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
