[
https://issues.apache.org/jira/browse/KAFKA-4393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-4393:
-----------------------------------
Description:
Currently, Kafka Streams does not handle invalid/negative timestamps returned
from the {{TimestampExtractor}} gracefully, but fails with an exception,
because negative timestamps cannot get handled in a meaningful way for any time
based (ie, window) operators like window aggregates and joins.
We want to change Stream to a auto-drop behavior for negative timestamps for
those records (without any further user notification about dropped record)
instead of an exception. To guard the user from silently dropping messages (and
kept current fail-fast behavior), we change the default extractor
{{ConsumerRecordTimestampExtractor}} to check the extracted meta-data record
timestamp and raise an exception if it is negative. Furthermore, we add a
"drop-and-log" extractor, as this seems to be a common behavior user might want
to have. For any other behavior, users can still provide a custom TS-Extractor
implementation.
was:
Currently, Kafka Streams does not handle invalid/negative timestamps returned
from the {{TimestampExtractor}} gracefully, but fails with an exception. Input
record timestamps determine output record timestamps and {{KafkaProducer}} does
not allow negative timestamps to get written.
Besides the exception issues describe above, negative timestamp can also not
get handled in a meaningful way for any time based (ie, window) operators like
window aggregates and joins.
Thus, we want to change Stream to a auto-drop behavior for negative timestamps
and not process those records at all (without any further user notification
about dropped record). To guard the user from silently dropping messages, we
change the default extractor {{ConsumerRecordTimestampExtractor}} to check the
extracted meta-data record timestamp and raise an exception if it is negative.
Furthermore, we add a "drop-and-log" extractor, as this seems to be a common
behavior user might want to have. For any other behavior, users can still
provide a custom TS-Extractor implementation.
> Improve invalid/negative TS handling
> ------------------------------------
>
> Key: KAFKA-4393
> URL: https://issues.apache.org/jira/browse/KAFKA-4393
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Reporter: Matthias J. Sax
> Assignee: Matthias J. Sax
> Fix For: 0.10.2.0
>
>
> Currently, Kafka Streams does not handle invalid/negative timestamps returned
> from the {{TimestampExtractor}} gracefully, but fails with an exception,
> because negative timestamps cannot get handled in a meaningful way for any
> time based (ie, window) operators like window aggregates and joins.
> We want to change Stream to a auto-drop behavior for negative timestamps for
> those records (without any further user notification about dropped record)
> instead of an exception. To guard the user from silently dropping messages
> (and kept current fail-fast behavior), we change the default extractor
> {{ConsumerRecordTimestampExtractor}} to check the extracted meta-data record
> timestamp and raise an exception if it is negative. Furthermore, we add a
> "drop-and-log" extractor, as this seems to be a common behavior user might
> want to have. For any other behavior, users can still provide a custom
> TS-Extractor implementation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
