[
https://issues.apache.org/jira/browse/STORM-697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthew Tieman updated STORM-697:
---------------------------------
Description:
It would be nice expose the offset and partition of messages consumed from
Kafka to the Scheme generating the Tuples. This is useful for
auditing/replaying data from arbitrary points on a Kafka topic. There are a few
ways to accomplish this.
One is to define a new extension of Scheme, say MessageAndMetadataScheme, and
add another case in KafkaUtils.generateTuples to determine if the config's
schema is an instance of MessageAndMetadataScheme.
Another is to overload KafkaUtils.generateTuples to accept and instance of
MessageAndRealOffset. Next, add a flag to SpoutConfig indicating that the
scheme should accept a byte array of MessageAndRealOffset, this flag will be
used to determine which generateTuples method to call.
was:
It would be nice expose the offset and partition of messages consumed from
Kafka to the Scheme generating the Tuples. This is useful for
auditing/replaying data from arbitrary points on a Kafka topic. There are a few
ways to accomplish this.
One is to define a new extension of Scheme, say MessageAndMetadataScheme, and
add another case in KafkaUtils.generateTuples to determine if the config's
schema is an instance of MessageAndMetadataScheme.
Another is add a flag to SpoutConfig indicating that the scheme should accept a
byte array of MessageAndRealOffset.
> Support for Emitting Kafka Message Offset and Partition
> -------------------------------------------------------
>
> Key: STORM-697
> URL: https://issues.apache.org/jira/browse/STORM-697
> Project: Apache Storm
> Issue Type: Improvement
> Components: storm-kafka
> Reporter: Matthew Tieman
> Assignee: Matthew Tieman
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> It would be nice expose the offset and partition of messages consumed from
> Kafka to the Scheme generating the Tuples. This is useful for
> auditing/replaying data from arbitrary points on a Kafka topic. There are a
> few ways to accomplish this.
> One is to define a new extension of Scheme, say MessageAndMetadataScheme, and
> add another case in KafkaUtils.generateTuples to determine if the config's
> schema is an instance of MessageAndMetadataScheme.
> Another is to overload KafkaUtils.generateTuples to accept and instance of
> MessageAndRealOffset. Next, add a flag to SpoutConfig indicating that the
> scheme should accept a byte array of MessageAndRealOffset, this flag will be
> used to determine which generateTuples method to call.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
