[jira] [Commented] (KAFKA-7045) Consumer may not be able to consume all messages when down-conversion is required
[ https://issues.apache.org/jira/browse/KAFKA-7045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16770503#comment-16770503 ] Matthias J. Sax commented on KAFKA-7045: Moving all major/minor/trivial tickets that are not merged yet out of 2.2 release. > Consumer may not be able to consume all messages when down-conversion is > required > - > > Key: KAFKA-7045 > URL: https://issues.apache.org/jira/browse/KAFKA-7045 > Project: Kafka > Issue Type: Bug > Components: consumer, core >Affects Versions: 0.11.0.0, 0.11.0.1, 0.11.0.2, 1.0.0, 1.0.1, 1.1.0, 2.0.0 >Reporter: Dhruvil Shah >Priority: Major > Fix For: 2.2.0 > > Attachments: log-cleaner-test.zip > > > When down-conversion is required, the consumer might fail consuming messages > under certain conditions. Couple such cases are outlined below: > (1) When consuming from a compacted topic, it is possible that the consumer > wants to fetch messages that fall in the middle of a batch but the messages > have been compacted by the cleaner. For example, let's say we have the > following two segments. The brackets indicate a single batch of messages and > the numbers within are the message offsets. > Segment #1: [0, 1, 2], [3, 4, 5], [6, 7, 8] > Segment #2: [9, 10, 11], [12, 13, 14] > If the cleaner were to come in now and clean up messages with offsets 7 and > 8, the segments would look like the following: > Segment #1: [0, 1, 2], [3, 4, 5], [6] > Segment #2: [9, 10, 11], [12, 13, 14] > A consumer attempting to fetch messages at offset 7 will start reading the > batch starting at offset 6. During down-conversion, we will drop the record > starting at 6 it is less than the current fetch start offset. However, there > are no messages in the log following offset 6. In such cases, we return the > `FileRecords` itself which would cause the consumer to throw an exception > because it does not understand the stored message format. > (2) When consuming from a topic with transactional messages, down-conversion > usually drops control batches because these do not exist in V0 and V1 message > formats. If there are no message batches following the control batch in the > particular segment (or if we are at the end of the log), we would again get > no records after down-conversion and will return the `FileRecords`. Because > the consumer is not able to interpret control batches, it will again throw an > exception. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7045) Consumer may not be able to consume all messages when down-conversion is required
[ https://issues.apache.org/jira/browse/KAFKA-7045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16749277#comment-16749277 ] Dhruvil Shah commented on KAFKA-7045: - Unassigning as I don't have time to work on this at the moment. > Consumer may not be able to consume all messages when down-conversion is > required > - > > Key: KAFKA-7045 > URL: https://issues.apache.org/jira/browse/KAFKA-7045 > Project: Kafka > Issue Type: Bug > Components: consumer, core >Affects Versions: 0.11.0.0, 0.11.0.1, 0.11.0.2, 1.0.0, 1.0.1, 1.1.0, 2.0.0 >Reporter: Dhruvil Shah >Priority: Major > Fix For: 2.2.0, 2.1.1 > > Attachments: log-cleaner-test.zip > > > When down-conversion is required, the consumer might fail consuming messages > under certain conditions. Couple such cases are outlined below: > (1) When consuming from a compacted topic, it is possible that the consumer > wants to fetch messages that fall in the middle of a batch but the messages > have been compacted by the cleaner. For example, let's say we have the > following two segments. The brackets indicate a single batch of messages and > the numbers within are the message offsets. > Segment #1: [0, 1, 2], [3, 4, 5], [6, 7, 8] > Segment #2: [9, 10, 11], [12, 13, 14] > If the cleaner were to come in now and clean up messages with offsets 7 and > 8, the segments would look like the following: > Segment #1: [0, 1, 2], [3, 4, 5], [6] > Segment #2: [9, 10, 11], [12, 13, 14] > A consumer attempting to fetch messages at offset 7 will start reading the > batch starting at offset 6. During down-conversion, we will drop the record > starting at 6 it is less than the current fetch start offset. However, there > are no messages in the log following offset 6. In such cases, we return the > `FileRecords` itself which would cause the consumer to throw an exception > because it does not understand the stored message format. > (2) When consuming from a topic with transactional messages, down-conversion > usually drops control batches because these do not exist in V0 and V1 message > formats. If there are no message batches following the control batch in the > particular segment (or if we are at the end of the log), we would again get > no records after down-conversion and will return the `FileRecords`. Because > the consumer is not able to interpret control batches, it will again throw an > exception. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7045) Consumer may not be able to consume all messages when down-conversion is required
[ https://issues.apache.org/jira/browse/KAFKA-7045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16509076#comment-16509076 ] Dhruvil Shah commented on KAFKA-7045: - Attached topic data that reproduces case (1) described in the JIRA where messages in the first segment starting offset 70 have been cleaned out. These messages were part of the batch starting at offset 69. Credit to [~omkreddy] for finding this issue and providing a reproducible test case. > Consumer may not be able to consume all messages when down-conversion is > required > - > > Key: KAFKA-7045 > URL: https://issues.apache.org/jira/browse/KAFKA-7045 > Project: Kafka > Issue Type: Bug > Components: consumer, core >Affects Versions: 0.11.0.0, 0.11.0.1, 1.0.0, 0.11.0.2, 1.1.0, 2.0.0, 1.0.1 >Reporter: Dhruvil Shah >Priority: Major > Fix For: 2.1.0 > > Attachments: log-cleaner-test.zip > > > When down-conversion is required, the consumer might fail consuming messages > under certain conditions. Couple such cases are outlined below: > (1) When consuming from a compacted topic, it is possible that the consumer > wants to fetch messages that fall in the middle of a batch but the messages > have been compacted by the cleaner. For example, let's say we have the > following two segments. The brackets indicate a single batch of messages and > the numbers within are the message offsets. > Segment #1: [0, 1, 2], [3, 4, 5], [6, 7, 8] > Segment #2: [9, 10, 11], [12, 13, 14] > If the cleaner were to come in now and clean up messages with offsets 7 and > 8, the segments would look like the following: > Segment #1: [0, 1, 2], [3, 4, 5], [6] > Segment #2: [9, 10, 11], [12, 13, 14] > A consumer attempting to fetch messages at offset 7 will start reading the > batch starting at offset 6. During down-conversion, we will drop the record > starting at 6 it is less than the current fetch start offset. However, there > are no messages in the log following offset 6. In such cases, we return the > `FileRecords` itself which would cause the consumer to throw an exception > because it does not understand the stored message format. > (2) When consuming from a topic with transactional messages, down-conversion > usually drops control batches because these do not exist in V0 and V1 message > formats. If there are no message batches following the control batch in the > particular segment (or if we are at the end of the log), we would again get > no records after down-conversion and will return the `FileRecords`. Because > the consumer is not able to interpret control batches, it will again throw an > exception. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7045) Consumer may not be able to consume all messages when down-conversion is required
[
https://issues.apache.org/jira/browse/KAFKA-7045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16509067#comment-16509067
]
Dhruvil Shah commented on KAFKA-7045:
-
Relevant code from 1.x release that sends {{FileRecords}} when down-conversion
does not yield any records:
{code:java}
public ConvertedRecords downConvert(byte toMagic, long
firstOffset, Time time) {
ConvertedRecords convertedRecords = downConvert(batches,
toMagic, firstOffset, time);
if (convertedRecords.recordsProcessingStats().numRecordsConverted() == 0) {
// This indicates that the message is too large, which means that the
buffer is not large
// enough to hold a full record batch. We just return all the bytes in
this instance.
// Even though the record batch does not have the right format version,
we expect old clients
// to raise an error to the user after reading the record batch size
and seeing that there
// are not enough available bytes in the response to read it fully.
Note that this is
// only possible prior to KIP-74, after which the broker was changed to
always return at least
// one full record batch, even if it requires exceeding the max fetch
size requested by the client.
return new ConvertedRecords<>(this, RecordsProcessingStats.EMPTY);
} else {
return convertedRecords;
}
}
{code}
> Consumer may not be able to consume all messages when down-conversion is
> required
> -
>
> Key: KAFKA-7045
> URL: https://issues.apache.org/jira/browse/KAFKA-7045
> Project: Kafka
> Issue Type: Bug
> Components: consumer, core
>Affects Versions: 0.11.0.0, 0.11.0.1, 1.0.0, 0.11.0.2, 1.1.0, 2.0.0, 1.0.1
>Reporter: Dhruvil Shah
>Priority: Major
> Fix For: 2.1.0
>
>
> When down-conversion is required, the consumer might fail consuming messages
> under certain conditions. Couple such cases are outlined below:
> # When consuming from a compacted topic, it is possible that the consumer
> wants to fetch messages that fall in the middle of a batch but the messages
> have been compacted by the cleaner. For example, let's say we have the
> following two segments. The brackets indicate a single batch of messages and
> the numbers within are the message offsets.
> Segment #1: [0, 1, 2], [3, 4, 5], [6, 7, 8]
> Segment #2: [9, 10, 11], [12, 13, 14]
> If the cleaner were to come in now and clean up messages with offsets 7 and
> 8, the segments would look like the following:
> Segment #1: [0, 1, 2], [3, 4, 5], [6]
> Segment #2: [9, 10, 11], [12, 13, 14]
> A consumer attempting to fetch messages at offset 7 will start reading the
> batch starting at offset 6. During down-conversion, we will drop the record
> starting at 6 it is less than the current fetch start offset. However, there
> are no messages in the log following offset 6. In such cases, we return the
> `FileRecords` itself which would cause the consumer to throw an exception
> because it does not understand the stored message format.
> (1) When consuming from a topic with transactional messages, down-conversion
> usually drops control batches because these did not exist in V0 and V1
> message formats. If there are no message batches following the control batch
> in the particular segment (or if we are at the end of the log), we would
> again get no records after down-conversion and will return the `FileRecords`.
> Because the consumer is not able to interpret control batches, it will again
> throw an exception.
> (2) When consuming from a topic with transactional messages, down-conversion
> usually drops control batches because these did not exist in V0 and V1
> message formats. If there are no message batches following the control batch
> in the particular segment (or if we are at the end of the log), we would
> again get no records after down-conversion and will return the `FileRecords`.
> Because the consumer is not able to interpret control batches, it will again
> throw an exception.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
