[
https://issues.apache.org/jira/browse/SAMZA-2749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alan Zhang updated SAMZA-2749:
------------------------------
Description:
Within Samza (the main class to pay attention to is OffsetManager.scala), there
is a bug in which a startpoint can be deleted before the startpoint actually
gets used for message consumption. If a container gets into this situation,
then the result is that the startpoint is ignored and consumption will continue
from the previous processed message from before the startpoint was applied.
# Load last processed offsets and startpoints
# Use startpoints to register starting offsets for consumers
# Message processing starts, but messages for only some of the partitions are
received
# Write checkpoint using last processed offsets
# If a partition did not get messages, then the last processed offset is still
the offset from before the startpoint.
# Delete startpoints
# Container dies (e.g. due to running out of memory)
# On restart, load last processed offsets (startpoints have been deleted)
# The partitions that did have messages in the previous deployment will have
the correct checkpoint.
# The partitions that did not have messages will have the checkpoint set to
the offset from before the startpoint was applied. This is unexpected, and it
means that bootstrapping is not happening for this partition.
> Startpoint bug fix
> ------------------
>
> Key: SAMZA-2749
> URL: https://issues.apache.org/jira/browse/SAMZA-2749
> Project: Samza
> Issue Type: Bug
> Reporter: Alan Zhang
> Assignee: Alan Zhang
> Priority: Major
>
> Within Samza (the main class to pay attention to is OffsetManager.scala),
> there is a bug in which a startpoint can be deleted before the startpoint
> actually gets used for message consumption. If a container gets into this
> situation, then the result is that the startpoint is ignored and consumption
> will continue from the previous processed message from before the startpoint
> was applied.
> # Load last processed offsets and startpoints
> # Use startpoints to register starting offsets for consumers
> # Message processing starts, but messages for only some of the partitions
> are received
> # Write checkpoint using last processed offsets
> # If a partition did not get messages, then the last processed offset is
> still the offset from before the startpoint.
> # Delete startpoints
> # Container dies (e.g. due to running out of memory)
> # On restart, load last processed offsets (startpoints have been deleted)
> # The partitions that did have messages in the previous deployment will have
> the correct checkpoint.
> # The partitions that did not have messages will have the checkpoint set to
> the offset from before the startpoint was applied. This is unexpected, and it
> means that bootstrapping is not happening for this partition.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
