[
https://issues.apache.org/jira/browse/STORM-1015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14718078#comment-14718078
]
ASF GitHub Bot commented on STORM-1015:
---------------------------------------
GitHub user hsun-cnnxty opened a pull request:
https://github.com/apache/storm/pull/705
[STORM-1015] Allow Kafka offsets to be saved using Kafka's consumer offset
management api
Not sure when it will be reviewed. So I chose to implement it based on
master branch. Could be ported to other branches if needed.
-thanks
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/hsun-cnnxty/storm kafka-storage
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/705.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #705
----
commit 30ac6ecf81a9ffb842c4cd047516d178dcbbdb01
Author: Hang Sun <[email protected]>
Date: 2015-08-28T04:45:04Z
Added new option to allow Kafka spout to save offset and other state using
Kafka's offset management api
----
> Store Kafka offsets with Kafka's consumer offset management api
> ---------------------------------------------------------------
>
> Key: STORM-1015
> URL: https://issues.apache.org/jira/browse/STORM-1015
> Project: Apache Storm
> Issue Type: Improvement
> Components: storm-kafka
> Affects Versions: 0.11.0
> Reporter: Hang Sun
> Priority: Minor
> Labels: consumer, kafka, offset
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> Current Kafka spout stores the offsets (and some other states) inside ZK with
> its proprietary format. This does not work well with other Kafka offset
> monitoring tools such as Burrow, KafkaOffsetMonitor etc. In addition, the
> performance does not scale well compared with offsets managed by Kafka's
> built-in offset management api. I have added a new option for Kafka to store
> the same data using Kafka's built-in offset management capability. The change
> is completely backward compatible with the current ZK storage option. The
> feature can be turned on by a single configuration option. Hope this will
> help people who wants to explore the option of using Kafka's built-in offset
> management api.
> -thanks
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
