Matthias J. Sax created KAFKA-6048:
--------------------------------------
Summary: Support negative record timestamps
Key: KAFKA-6048
URL: https://issues.apache.org/jira/browse/KAFKA-6048
Project: Kafka
Issue Type: Improvement
Components: clients, core, streams
Affects Versions: 1.0.0
Reporter: Matthias J. Sax
Kafka does not support negative record timestamps, and this prevents the
storage of historical data in Kafka. In general, negative timestamps are
supported by UNIX system time stamps:
>From https://en.wikipedia.org/wiki/Unix_time
{quote}
The Unix time number is zero at the Unix epoch, and increases by exactly 86,400
per day since the epoch. Thus 2004-09-16T00:00:00Z, 12,677 days after the
epoch, is represented by the Unix time number 12,677 × 86,400 = 1095292800.
This can be extended backwards from the epoch too, using negative numbers; thus
1957-10-04T00:00:00Z, 4,472 days before the epoch, is represented by the Unix
time number −4,472 × 86,400 = −386380800.
{quote}
Allowing for negative timestamps would require multiple changes:
- while brokers in general do support negative timestamps, broker use {{-1}}
as default value if a producer uses an old message format (this would not be
compatible with supporting negative timestamps "end-to-end" as {{-1}} cannot be
used as "unknown" anymore): we could introduce a message flag indicating a
missing timestamp (and let producer throw an exception if
{{ConsumerRecord#timestamp()}} is called. Another possible solution might be,
to require topics that are used by old producers to be configured with
{{LogAppendTime}} semantics and rejecting writes to topics with {{CreateTime}}
semantics for older message formats
- {{KafkaProducer}} does not allow to send records with negative timestamp and
thus this would need to be fixed
- Streams API does drop records with negative timestamps (or fails by default)
-- also, some internal store implementation for windowed stores assume that
there are not negative timestamps to do range queries
There might be other gaps we need to address. This is just a summary of issue
coming to my mind atm.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
