[
https://issues.apache.org/jira/browse/STORM-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348558#comment-16348558
]
Alexandre Vermeerbergen commented on STORM-2914:
------------------------------------------------
Hello [~Srdo]
I have build storm-kafka-client.jar from your 1.x branch, tests are starting
now.
While it's too early to say if it's OK or not, I noticed the following warning
in one of our topologies's log:
2018-02-01 13:08:25.016 o.a.k.c.u.AppInfoParser
Thread-4-podOrTenantsFromKafkaSpout-executor[2 2] [INFO] Kafka version :
0.10.2.1 2018-02-01 13:08:25.016 o.a.k.c.u.AppInfoParser
Thread-4-podOrTenantsFromKafkaSpout-executor[2 2] [INFO] Kafka commitId :
e89bffd6b2eff799 2018-02-01 13:08:25.076 o.a.s.k.s.KafkaSpout
Thread-4-podOrTenantsFromKafkaSpout-executor[2 2] [INFO] Partitions revoked.
[consumer-group=StormPodsyncTopology_RealTimeSupervision_zDHZGCMKTHGySBqVw9DiqQ,
consumer=org.apache.kafka.clients.consumer.KafkaConsumer@5cb084d9,
topic-partitions=[]] 2018-02-01 13:08:25.077 o.a.s.k.s.KafkaSpout
Thread-4-podOrTenantsFromKafkaSpout-executor[2 2] [INFO] Partitions
reassignment. [task-ID=2,
consumer-group=StormPodsyncTopology_RealTimeSupervision_zDHZGCMKTHGySBqVw9DiqQ,
consumer=org.apache.kafka.clients.consumer.KafkaConsumer@5cb084d9,
topic-partitions=[podsync_RealTimeSupervision_zDHZGCMKTHGySBqVw9DiqQ-15,
podsync_RealTimeSupervision_zDHZGCMKTHGySBqVw9DiqQ-4,
podsync_RealTimeSupervision_zDHZGCMKTHGySBqVw9DiqQ-3,
podsync_RealTimeSupervision_zDHZGCMKTHGySBqVw9DiqQ-6,
podsync_RealTimeSupervision_zDHZGCMKTHGySBqVw9DiqQ-5,
podsync_RealTimeSupervision_zDHZGCMKTHGySBqVw9DiqQ-0,
podsync_RealTimeSupervision_zDHZGCMKTHGySBqVw9DiqQ-2,
podsync_RealTimeSupervision_zDHZGCMKTHGySBqVw9DiqQ-1,
podsync_RealTimeSupervision_zDHZGCMKTHGySBqVw9DiqQ-12,
podsync_RealTimeSupervision_zDHZGCMKTHGySBqVw9DiqQ-11,
podsync_RealTimeSupervision_zDHZGCMKTHGySBqVw9DiqQ-14,
podsync_RealTimeSupervision_zDHZGCMKTHGySBqVw9DiqQ-13,
podsync_RealTimeSupervision_zDHZGCMKTHGySBqVw9DiqQ-8,
podsync_RealTimeSupervision_zDHZGCMKTHGySBqVw9DiqQ-7,
podsync_RealTimeSupervision_zDHZGCMKTHGySBqVw9DiqQ-10,
podsync_RealTimeSupervision_zDHZGCMKTHGySBqVw9DiqQ-9]] 2018-02-01 13:08:25.083
o.a.k.c.c.i.AbstractCoordinator Thread-4-podOrTenantsFromKafkaSpout-executor[2
2] [INFO] Discovered coordinator
ec2-34-242-207-227.eu-west-1.compute.amazonaws.com:9092 (id: 2147483644 rack:
null) for group
StormPodsyncTopology_RealTimeSupervision_zDHZGCMKTHGySBqVw9DiqQ. 2018-02-01
13:08:25.116 o.a.s.k.s.i.CommitMetadata
Thread-4-podOrTenantsFromKafkaSpout-executor[2 2] [WARN] Failed to deserialize
[OffsetAndMetadata\{offset=63683,
metadata='{topic-partition=podsync_RealTimeSupervision_zDHZGCMKTHGySBqVw9DiqQ-15,
offset=63682, numFails=0,
thread='Thread-4-podOrTenantsFromKafkaSpout-executor[2 2]'}'}]. Error likely
occurred because the last commit for this topic-partition was done using an
earlier version of Storm. Defaulting to behavior compatible with earlier
version 2018-02-01 13:08:25.125 o.a.s.k.s.i.CommitMetadata
Thread-4-podOrTenantsFromKafkaSpout-executor[2 2] [WARN] Failed to deserialize
[OffsetAndMetadata\{offset=67701,
metadata='{topic-partition=podsync_RealTimeSupervision_zDHZGCMKTHGySBqVw9DiqQ-4,
offset=67700, numFails=0,
thread='Thread-4-podOrTenantsFromKafkaSpout-executor[2 2]'}'}]. Error likely
occurred because the last commit for this topic-partition was done using an
earlier version of Storm. Defaulting to behavior compatible with earlier
version 2018-02-01 13:08:25.130 o.a.s.k.s.i.CommitMetadata
Thread-4-podOrTenantsFromKafkaSpout-executor[2 2] [WARN] Failed to deserialize
[OffsetAndMetadata\{offset=69382,
metadata='{topic-partition=podsync_RealTimeSupervision_zDHZGCMKTHGySBqVw9DiqQ-3,
offset=69381, numFails=0,
thread='Thread-4-podOrTenantsFromKafkaSpout-executor[2 2]'}'}]. Error likely
occurred because the last commit for this topic-partition was done using an
earlier version of Storm. Defaulting to behavior compatible with earlier
version 2018-02-01 13:08:25.134 o.a.s.k.s.i.CommitMetadata
Thread-4-podOrTenantsFromKafkaSpout-executor[2 2] [WARN] Failed to deserialize
[OffsetAndMetadata\{offset=78943,
metadata='{topic-partition=podsync_RealTimeSupervision_zDHZGCMKTHGySBqVw9DiqQ-6,
offset=78942, numFails=0,
thread='Thread-4-podOrTenantsFromKafkaSpout-executor[2 2]'}'}]. Error likely
occurred because the last commit for this topic-partition was done using an
earlier version of Storm. Defaulting to behavior compatible with earlier
version 2018-02-01 13:08:25.135 o.a.s.k.s.i.CommitMetadata
Thread-4-podOrTenantsFromKafkaSpout-executor[2 2] [WARN] Failed to deserialize
[OffsetAndMetadata\{offset=76603,
metadata='{topic-partition=podsync_RealTimeSupervision_zDHZGCMKTHGySBqVw9DiqQ-5,
offset=76602, numFails=0,
thread='Thread-4-podOrTenantsFromKafkaSpout-executor[2 2]'}'}]. Error likely
occurred because the last commit for this topic-partition was done using an
earlier version of Storm. Defaulting to behavior compatible with earlier
version 2018-02-01 13:08:25.136 o.a.s.k.s.i.CommitMetadata
Thread-4-podOrTenantsFromKafkaSpout-executor[2 2] [WARN] Failed to deserialize
[OffsetAndMetadata\{offset=63190,
metadata='{topic-partition=podsync_RealTimeSupervision_zDHZGCMKTHGySBqVw9DiqQ-0,
offset=63189, numFails=0,
thread='Thread-4-podOrTenantsFromKafkaSpout-executor[2 2]'}'}]. Error likely
occurred because the last commit for this topic-partition was done using an
earlier version of Storm. Defaulting to behavior compatible with earlier
version 2018-02-01 13:08:25.137 o.a.s.k.s.i.CommitMetadata
Thread-4-podOrTenantsFromKafkaSpout-executor[2 2] [WARN] Failed to deserialize
[OffsetAndMetadata\{offset=73541,
metadata='{topic-partition=podsync_RealTimeSupervision_zDHZGCMKTHGySBqVw9DiqQ-2,
offset=73540, numFails=0,
thread='Thread-4-podOrTenantsFromKafkaSpout-executor[2 2]'}'}]. Error likely
occurred because the last commit for this topic-partition was done using an
earlier version of Storm. Defaulting to behavior compatible with earlier
version 2018-02-01 13:08:25.140 o.a.s.k.s.i.CommitMetadata
Thread-4-podOrTenantsFromKafkaSpout-executor[2 2] [WARN] Failed to deserialize
[OffsetAndMetadata\{offset=70191,
metadata='{topic-partition=podsync_RealTimeSupervision_zDHZGCMKTHGySBqVw9DiqQ-1,
offset=70190, numFails=0,
thread='Thread-4-podOrTenantsFromKafkaSpout-executor[2 2]'}'}]. Error likely
occurred because the last commit for this topic-partition was done using an
earlier version of Storm. Defaulting to behavior compatible with earlier
version 2018-02-01 13:08:25.142 o.a.s.k.s.i.CommitMetadata
Thread-4-podOrTenantsFromKafkaSpout-executor[2 2] [WARN] Failed to deserialize
[OffsetAndMetadata\{offset=70966,
metadata='{topic-partition=podsync_RealTimeSupervision_zDHZGCMKTHGySBqVw9DiqQ-12,
offset=70965, numFails=0,
thread='Thread-4-podOrTenantsFromKafkaSpout-executor[2 2]'}'}]. Error likely
occurred because the last commit for this topic-partition was done using an
earlier version of Storm. Defaulting to behavior compatible with earlier
version 2018-02-01 13:08:25.143 o.a.s.k.s.i.CommitMetadata
Thread-4-podOrTenantsFromKafkaSpout-executor[2 2] [WARN] Failed to deserialize
[OffsetAndMetadata\{offset=71945,
metadata='{topic-partition=podsync_RealTimeSupervision_zDHZGCMKTHGySBqVw9DiqQ-11,
offset=71944, numFails=0,
thread='Thread-4-podOrTenantsFromKafkaSpout-executor[2 2]'}'}]. Error likely
occurred because the last commit for this topic-partition was done using an
earlier version of Storm. Defaulting to behavior compatible with earlier
version 2018-02-01 13:08:25.148 o.a.s.k.s.i.CommitMetadata
Thread-4-podOrTenantsFromKafkaSpout-executor[2 2] [WARN] Failed to deserialize
[OffsetAndMetadata\{offset=74705,
metadata='{topic-partition=podsync_RealTimeSupervision_zDHZGCMKTHGySBqVw9DiqQ-14,
offset=74704, numFails=0,
thread='Thread-4-podOrTenantsFromKafkaSpout-executor[2 2]'}'}]. Error likely
occurred because the last commit for this topic-partition was done using an
earlier version of Storm. Defaulting to behavior compatible with earlier
version 2018-02-01 13:08:25.149 o.a.s.k.s.i.CommitMetadata
Thread-4-podOrTenantsFromKafkaSpout-executor[2 2] [WARN] Failed to deserialize
[OffsetAndMetadata\{offset=70042,
metadata='{topic-partition=podsync_RealTimeSupervision_zDHZGCMKTHGySBqVw9DiqQ-13,
offset=70041, numFails=0,
thread='Thread-4-podOrTenantsFromKafkaSpout-executor[2 2]'}'}]. Error likely
occurred because the last commit for this topic-partition was done using an
earlier version of Storm. Defaulting to behavior compatible with earlier
version 2018-02-01 13:08:25.153 o.a.s.k.s.i.CommitMetadata
Thread-4-podOrTenantsFromKafkaSpout-executor[2 2] [WARN] Failed to deserialize
[OffsetAndMetadata\{offset=70662,
metadata='{topic-partition=podsync_RealTimeSupervision_zDHZGCMKTHGySBqVw9DiqQ-8,
offset=70661, numFails=0,
thread='Thread-4-podOrTenantsFromKafkaSpout-executor[2 2]'}'}]. Error likely
occurred because the last commit for this topic-partition was done using an
earlier version of Storm. Defaulting to behavior compatible with earlier
version 2018-02-01 13:08:25.154 o.a.s.k.s.i.CommitMetadata
Thread-4-podOrTenantsFromKafkaSpout-executor[2 2] [WARN] Failed to deserialize
[OffsetAndMetadata\{offset=64628,
metadata='{topic-partition=podsync_RealTimeSupervision_zDHZGCMKTHGySBqVw9DiqQ-7,
offset=64627, numFails=0,
thread='Thread-4-podOrTenantsFromKafkaSpout-executor[2 2]'}'}]. Error likely
occurred because the last commit for this topic-partition was done using an
earlier version of Storm. Defaulting to behavior compatible with earlier
version 2018-02-01 13:08:25.155 o.a.s.k.s.i.CommitMetadata
Thread-4-podOrTenantsFromKafkaSpout-executor[2 2] [WARN] Failed to deserialize
[OffsetAndMetadata\{offset=72981,
metadata='{topic-partition=podsync_RealTimeSupervision_zDHZGCMKTHGySBqVw9DiqQ-10,
offset=72980, numFails=0,
thread='Thread-4-podOrTenantsFromKafkaSpout-executor[2 2]'}'}]. Error likely
occurred because the last commit for this topic-partition was done using an
earlier version of Storm. Defaulting to behavior compatible with earlier
version 2018-02-01 13:08:25.156 o.a.s.k.s.i.CommitMetadata
Thread-4-podOrTenantsFromKafkaSpout-executor[2 2] [WARN] Failed to deserialize
[OffsetAndMetadata\{offset=61852,
metadata='{topic-partition=podsync_RealTimeSupervision_zDHZGCMKTHGySBqVw9DiqQ-9,
offset=61851, numFails=0,
thread='Thread-4-podOrTenantsFromKafkaSpout-executor[2 2]'}'}]. Error likely
occurred because the last commit for this topic-partition was done using an
earlier version of Storm. Defaulting to behavior compatible with earlier
version 2018-02-01 13:08:25.159 o.a.s.k.s.KafkaSpout
Thread-4-podOrTenantsFromKafkaSpout-executor[2 2] [INFO] Initialization complete
I don't know yet whether or not it'll be repeated forever, I'll check later.
Is this something expected from your perspective?
Could it be just a "transition" effect ?
Best regards,
Alexandre Vermeerbergen
> Remove enable.auto.commit support from storm-kafka-client
> ---------------------------------------------------------
>
> Key: STORM-2914
> URL: https://issues.apache.org/jira/browse/STORM-2914
> Project: Apache Storm
> Issue Type: Improvement
> Components: storm-kafka-client
> Affects Versions: 2.0.0, 1.2.0
> Reporter: Stig Rohde Døssing
> Assignee: Stig Rohde Døssing
> Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The enable.auto.commit option causes the KafkaConsumer to periodically commit
> the latest offsets it has returned from poll(). It is convenient for use
> cases where messages are polled from Kafka and processed synchronously, in a
> loop.
> Due to https://issues.apache.org/jira/browse/STORM-2913 we'd really like to
> store some metadata in Kafka when the spout commits. This is not possible
> with enable.auto.commit. I took at look at what that setting actually does,
> and it just causes the KafkaConsumer to call commitAsync during poll (and
> during a few other operations, e.g. close and assign) with some interval.
> Ideally I'd like to get rid of ProcessingGuarantee.NONE, since I think
> ProcessingGuarantee.AT_MOST_ONCE covers the same use cases, and is likely
> almost as fast. The primary difference between them is that AT_MOST_ONCE
> commits synchronously.
> If we really want to keep ProcessingGuarantee.NONE, I think we should make
> our ProcessingGuarantee.NONE setting cause the spout to call commitAsync
> after poll, and never use the enable.auto.commit option. This allows us to
> include metadata in the commit.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
