[ https://issues.apache.org/jira/browse/STORM-643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14300943#comment-14300943 ]
ASF GitHub Bot commented on STORM-643: -------------------------------------- GitHub user vesense opened a pull request: https://github.com/apache/storm/pull/405 Update PartitionManager.java fix bug [STORM-643] KafkaUtils repeat fetch messages which offset is out of range You can merge this pull request into a Git repository by running: $ git pull https://github.com/vesense/storm 0.9.3-branch Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/405.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #405 ---- commit 9280a948efaf28ab4de019060435d46731abd375 Author: vesense <best.wang...@163.com> Date: 2015-02-02T06:24:14Z Update PartitionManager.java fix bug [STORM-643] KafkaUtils repeat fetch messages which offset is out of range ---- > KafkaUtils repeat fetch messages which offset is out of range > ------------------------------------------------------------- > > Key: STORM-643 > URL: https://issues.apache.org/jira/browse/STORM-643 > Project: Apache Storm > Issue Type: Bug > Components: storm-kafka > Affects Versions: 0.9.2-incubating, 0.9.3 > Reporter: Xin Wang > Assignee: Xin Wang > Priority: Critical > > KafkaUtils repeat fetch messages which offset is out of range. > This happened when failed list(SortedSet<Long> failed) is not empty and some > offset in it is OutOfRange. > [worker-log] > 2015-02-01 10:24:27.231+0800 s.k.KafkaUtils [WARN] Got fetch request with > offset out of range: [20919071816]; retrying with default start offset time > from configuration. configured start offset time: [-2] > 2015-02-01 10:24:27.232+0800 s.k.PartitionManager [WARN] Using new offset: > 20996130717 > 2015-02-01 10:24:27.333+0800 s.k.KafkaUtils [WARN] Got fetch request with > offset out of range: [20919071816]; retrying with default start offset time > from configuration. configured start offset time: [-2] > 2015-02-01 10:24:27.334+0800 s.k.PartitionManager [WARN] Using new offset: > 20996130717 > ... > [FIX] > storm.kafka.PartitionManager.fill(): > ... > try { > msgs = KafkaUtils.fetchMessages(_spoutConfig, _consumer, _partition, > offset); > } catch (UpdateOffsetException e) { > _emittedToOffset = KafkaUtils.getOffset(_consumer, _spoutConfig.topic, > _partition.partition, _spoutConfig); > LOG.warn("Using new offset: {}", _emittedToOffset); > // fetch failed, so don't update the metrics > //fix bug: remove this offset from failed list when it is OutOfRange > if (had_failed) { > failed.remove(offset); > } > return; > } > ... > also: Log "retrying with default start offset time from configuration. > configured start offset time: [-2]" is incorrect. -- This message was sent by Atlassian JIRA (v6.3.4#6332)