[
https://issues.apache.org/jira/browse/KAFKA-4935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ismael Juma updated KAFKA-4935:
-------------------------------
Description:
With the new message format proposed in KIP-98, the record level CRC has been
moved to the the batch header.
Because we expose the record-level CRC in `RecordMetadata` and
`ConsumerRecord`, we currently compute it eagerly based on the key, value and
timestamp even though these methods are rarely used. Ideally, we'd deprecate
the relevant methods in `RecordMetadata` and `ConsumerRecord` while making the
CRC computation lazy. This seems pretty hard to achieve in the Producer without
increasing memory retention, but it may be possible to do in the Consumer.
An alternative option is to return the batch CRC from the relevant methods.
was:
With the new message format proposed in KIP-98, the record level CRC has been
moved to the the batch header. The consumer record API still allows retrieval
of CRC per record, and in this case we compute the CRC on the fly. This
degrades performance, and also such a computation becomes unnecessary with the
batch level CRC.
We can address this by deprecating the record level CRC api. We can also work
around this by disabling record level checks when the check crc config is set
to false.
> Consider disabling record level CRC checks for message format V2
> ----------------------------------------------------------------
>
> Key: KAFKA-4935
> URL: https://issues.apache.org/jira/browse/KAFKA-4935
> Project: Kafka
> Issue Type: Improvement
> Reporter: Apurva Mehta
> Fix For: 0.11.0.0
>
>
> With the new message format proposed in KIP-98, the record level CRC has been
> moved to the the batch header.
> Because we expose the record-level CRC in `RecordMetadata` and
> `ConsumerRecord`, we currently compute it eagerly based on the key, value and
> timestamp even though these methods are rarely used. Ideally, we'd deprecate
> the relevant methods in `RecordMetadata` and `ConsumerRecord` while making
> the CRC computation lazy. This seems pretty hard to achieve in the Producer
> without increasing memory retention, but it may be possible to do in the
> Consumer.
> An alternative option is to return the batch CRC from the relevant methods.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
