[
https://issues.apache.org/jira/browse/SAMZA-833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15047529#comment-15047529
]
Tao Feng commented on SAMZA-833:
--------------------------------
Hi [~jmakes], thanks for the suggetions. [~nickpan47] also suggests that we
could just throw exception if job.container.count or yarn.countainer.count are
larger than 1 if user is using ProcessJobFactory. What do you think?
> ProcessJob mishandling containers
> ---------------------------------
>
> Key: SAMZA-833
> URL: https://issues.apache.org/jira/browse/SAMZA-833
> Project: Samza
> Issue Type: Bug
> Reporter: Jake Maes
> Assignee: Tao Feng
>
> As a result of SAMZA-465 and SAMZA-805, ProcessJobFactory now passes the full
> config to the ProcessJob and no longer forces the container count to 1. This
> causes the ProcessJob to actually read the container count config and if it
> is not 1, it produces some unexpected behavior.
> Specifically we've had reports of ProcessJobs dropping messages because the
> container count is > 1, so the grouper assigns partitions to more than 1
> container, but only one container actually runs.
> The goal of this ticket is to either force the container count to 1 for
> ProcessJob, or fix how multiple containers run with ProcessJob. But we should
> not allow the scenario where messages are dropped.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
