[
https://issues.apache.org/jira/browse/KAFKA-8722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ChenLin updated KAFKA-8722:
---------------------------
Description:
In our production environment, when we consume kafka's topic data in an
operating program, we found an error:
org.apache.kafka.common.KafkaException: Record for partition
rl_dqn_debug_example-49 at offset 2911287689 is invalid, cause: Record is
corrupt (stored crc = 3580880396, computed crc = 1701403171)
at
org.apache.kafka.clients.consumer.internals.Fetcher.parseRecord(Fetcher.java:869)
at
org.apache.kafka.clients.consumer.internals.Fetcher.parseCompletedFetch(Fetcher.java:788)
at
org.apache.kafka.clients.consumer.internals.Fetcher.fetchedRecords(Fetcher.java:480)
at
org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1188)
at
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1046)
at kafka.consumer.NewShinyConsumer.receive(BaseConsumer.scala:88)
at kafka.tools.ConsoleConsumer$.process(ConsoleConsumer.scala:120)
at kafka.tools.ConsoleConsumer$.run(ConsoleConsumer.scala:75)
at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:50)
at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala)
By looking at the code, I found that in some cases kafka would not verify the
data and write it to disk, so we fixed it.
We found that when record.offset is not equal to the offset we are expecting,
kafka will set the variable inPlaceAssignment to false. When inPlaceAssignment
is false, no data validation will be performed:
!image-2019-07-27-14-50-58-300.png!
!image-2019-07-27-14-50-08-128.png!
was:
In our production environment, when we consume kafka's topic data in an
operating program, we found an error:
org.apache.kafka.common.KafkaException: Record for partition
rl_dqn_debug_example-49 at offset 2911287689 is invalid, cause: Record is
corrupt (stored crc = 3580880396, computed crc = 1701403171)
By looking at the code, I found that in some cases kafka would not verify the
data and write it to disk, so we fixed it.
> In some cases, the crc check does not cause dirty data to be written.
> ---------------------------------------------------------------------
>
> Key: KAFKA-8722
> URL: https://issues.apache.org/jira/browse/KAFKA-8722
> Project: Kafka
> Issue Type: Improvement
> Components: log
> Affects Versions: 0.10.2.2
> Reporter: ChenLin
> Priority: Major
> Fix For: 0.10.2.2
>
> Attachments: image-2019-07-27-14-50-08-128.png,
> image-2019-07-27-14-50-58-300.png
>
>
> In our production environment, when we consume kafka's topic data in an
> operating program, we found an error:
> org.apache.kafka.common.KafkaException: Record for partition
> rl_dqn_debug_example-49 at offset 2911287689 is invalid, cause: Record is
> corrupt (stored crc = 3580880396, computed crc = 1701403171)
> at
> org.apache.kafka.clients.consumer.internals.Fetcher.parseRecord(Fetcher.java:869)
> at
> org.apache.kafka.clients.consumer.internals.Fetcher.parseCompletedFetch(Fetcher.java:788)
> at
> org.apache.kafka.clients.consumer.internals.Fetcher.fetchedRecords(Fetcher.java:480)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1188)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1046)
> at kafka.consumer.NewShinyConsumer.receive(BaseConsumer.scala:88)
> at kafka.tools.ConsoleConsumer$.process(ConsoleConsumer.scala:120)
> at kafka.tools.ConsoleConsumer$.run(ConsoleConsumer.scala:75)
> at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:50)
> at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala)
>
> By looking at the code, I found that in some cases kafka would not verify the
> data and write it to disk, so we fixed it.
> We found that when record.offset is not equal to the offset we are expecting,
> kafka will set the variable inPlaceAssignment to false. When
> inPlaceAssignment is false, no data validation will be performed:
> !image-2019-07-27-14-50-58-300.png!
> !image-2019-07-27-14-50-08-128.png!
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
