GitHub user mjtieman opened a pull request:
https://github.com/apache/storm/pull/454
Storm 697: Support for Emitting Kafka Message Offset and Partition
It would be nice expose the offset and partition of messages consumed from
Kafka to the Scheme generating the Tuples. This is useful for
auditing/replaying data from arbitrary points on a Kafka topic, saving the
partition and offset of each message of a discrete stream instead of persisting
the entire message.
* Added new sheme to that accepts a Partition and the message offset in its
deserialization method.
* Defined an overload of `KafkaUtils.generateTuples` to accept a
`Partition` and offset in addition to the message byte[].
* Added a flag in `KafkaConfig` to indicate if the metadata, partition and
offset, should be available during tuple generation.
* Wrote a simple String implementation of the new scheme,
`StringMessageAndMetadataScheme`, following the same pattern as
`StringKeyValueScheme`.
* Unit tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mjtieman/storm STORM-697
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/454.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #454
----
commit 5b4c28a088ffc62ebcc28e8c28a25d096aa1eb78
Author: matt.tieman <matt.tie...@inin.com>
Date: 2015-03-03T16:46:30Z
STORM-697: Added tupleMetaData flag
commit 6e4fde20af8d285cdf4829e4c2c4aef4cd45d89d
Author: matt.tieman <matt.tie...@inin.com>
Date: 2015-03-03T16:47:38Z
STORM-697: Overload of generateTuples to accept the Partition and offset
commit 6e768665320d08815c53f27e706ef2ae1ff5af78
Author: matt.tieman <matt.tie...@inin.com>
Date: 2015-03-03T16:48:57Z
STORM-697: Test for MessageMetadataSchemeAsMultiScheme and generateTuples
with metadata using SchemeAsMultiScheme
commit 2f119c6e2edace030afeb9ee0885010f1de7fc28
Author: matt.tieman <matt.tie...@inin.com>
Date: 2015-03-03T16:50:04Z
STORM-697: Added scheme to include Partition and offset when generating
tuple. >>>
The MessageMetadataScheme interface extends Sheme and defines a
deserialization method that accepts the message byte[], Partition, and the
offset.
MessageMetadataSchemeAsMultiScheme follows the same pattern as
KeyValueSchemeAsMultiScheme, extending SchemeAsMultiScheme and providing a
deserialization method named for the method defined by
MessageMetadataScheme.
StringMessageAndMetadataScheme provides an implementation of
MessageMetadataScheme, following the same pattern as StringKeyValueScheme.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
