[
https://issues.apache.org/jira/browse/KAFKA-3914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15356795#comment-15356795
]
ASF GitHub Bot commented on KAFKA-3914:
---------------------------------------
GitHub user dguy opened a pull request:
https://github.com/apache/kafka/pull/1576
KAFKA-3914: Global discovery of state stores
@guozhangwang @enothereska @mjsax @miguno please take a look. A few things
that need to be clarified
1. I've added StreamsConfig.USER_ENDPOINT_CONFIG, but should we have
separate configs for host and port or is this one config ok?
2. `HostState` in the KIP has a byte[] field - not sure why and what it
would be populated with
3. I've changed the API to return `List<KafkaStreamsInstance>` as opposed
to `Map<HostInfo, Set<TaskMetadata>>` as i find this far more intuitive to work
with.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dguy/kafka kafka-3914v2
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/1576.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1576
----
commit f22ca0e680063d42240d8ceebe140ecd9e3fab48
Author: Damian Guy <[email protected]>
Date: 2016-06-29T07:40:11Z
Connect stateStores to source topics.
Add user endpoint config to StreamsConfig.
Collect Map<HostState, Set<TopicPartition>> in StreamPartitionAssignor
API for retrieving Map<HostState, TaskMetadata>
commit 2030da06a22b4160760a2d29a7afd394bd29972b
Author: Damian Guy <[email protected]>
Date: 2016-06-30T09:09:50Z
change api to use List and KafkaStreamsInstance rather than Map..
----
> Global discovery of state stores
> --------------------------------
>
> Key: KAFKA-3914
> URL: https://issues.apache.org/jira/browse/KAFKA-3914
> Project: Kafka
> Issue Type: Sub-task
> Components: streams
> Affects Versions: 0.10.1.0
> Reporter: Damian Guy
> Assignee: Damian Guy
> Fix For: 0.10.1.0
>
>
> - Update group membership data to include discovery endpoints
> - Enable discovery
> We need to attach some host and port information to
> {{org.apache.kafka.streams.processor.internals.assignment.SubscriptionInfo}}
> This can then be used in
> {{org.apache.kafka.streams.processor.internals.StreamPartitionAssignor}} to
> build a {{Map<HostState, Set<TopicPartition>>}} that should be added to
> {{org.apache.kafka.streams.processor.internals.assignment.AssignmentInfo}}
> during the {{assign}} call.
> When {{StreamPartitionAssignor.onAssignment}} is called we should hold on to
> a copy of the {{Map<HostState, Set<TopicPartition>>}} for use by the
> discovery methods
> To enable discovery, implement the below methods on {{KafkaStreams}}
> {code}
> /**
> * @return metadata about all tasks
> */
> Map<HostState, Set<TaskMetadata>> getAllTasks();
> /**
> * @param storeName requested store name
> * @return metadata about all tasks that include
> * storeName in this KStreams instance
> */
> Map<HostState, Set<TaskMetadata>> getAllTasksWithStore(String storeName);
> /**
> * @param key requested key
> * @param storeName requested store name
> * @return metadata about all tasks that include
> * storeName and key in this KStreams instance
> */
> <K> Map<HostState, Set<TaskMetadata>> getAllTasksWithKey(String storeName, K
> key);
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
