lucasbru commented on code in PR #20665:
URL: https://github.com/apache/kafka/pull/20665#discussion_r2426280921
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamTask.java:
##########
@@ -470,17 +470,24 @@ private OffsetAndMetadata findOffsetAndMetadata(final
TopicPartition partition)
Optional<Integer> leaderEpoch =
partitionGroup.headRecordLeaderEpoch(partition);
final long partitionTime =
partitionGroup.partitionTimestamp(partition);
if (offset == null) {
- try {
- if (nextOffsetsAndMetadataToBeConsumed.containsKey(partition))
{
- final OffsetAndMetadata offsetAndMetadata =
nextOffsetsAndMetadataToBeConsumed.get(partition);
- offset = offsetAndMetadata.offset();
- leaderEpoch = offsetAndMetadata.leaderEpoch();
- } else {
- // This indicates a bug and thus we rethrow it as fatal
`IllegalStateException`
- throw new IllegalStateException("Stream task " + id + "
does not know the partition: " + partition);
+ final OffsetAndMetadata offsetAndMetadata =
nextOffsetsAndMetadataToBeConsumed.get(partition);
+ if (offsetAndMetadata == null) {
+ try {
+ offset = mainConsumer.position(partition);
Review Comment:
Yes, this is also what I understood when digging into it last week. I'm
actually not sure that logic is fully watertight - what if my regular
expression matches a disjoint subset of topics - none of them will have the
metadata, so I have lost it, right? It seems a bit of a best-effort thing to
me... In which we could also consider not committing the "partition we never
consumed from" here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
