Re: Q about PartitionAssignor
It is used to carry data metadata that leader wants to propagate to other members while doing the rebalance. For example, in Kafka Stream userData contains the mapping of stream tasks to partition groups; in Kafka Connector different connectors can also use this field to fill in app-specific assignments. In fact with userData, you have another way of sticky partitioning: set the full assignment in the userData so all members can remember the whole mapping and hence you do not need another persistent storage for leader failover. Guozhang On Wed, Nov 18, 2015 at 3:25 PM, hsy...@gmail.com wrote: > Thanks Guozhang, what is userData for in the Subscription? > > On Wed, Nov 18, 2015 at 12:05 PM, Guozhang Wang > wrote: > > > Currently the whole KafkaConsumer interface is tagged as " > > @InterfaceStability.Unstable", meaning that the API may change in the > > future. We have been very careful to make any dramatic public API changes > > but still cannot guarantee this will not happen. > > > > Member-Id is assigned by the server-side coordinator upon accepting the > > consumer to join the specified group, hence it cannot be manually set. > But > > the memberId will not change as long as the consumer is still part of the > > members of the group, so you want to do some sticky assignment you can > just > > remember the memberId -> partitions map on the consumer side in some > > persistent storage so that even when the leader who does the assignment > has > > failed over other new leaders can still access the past assignment > history. > > > > Guozhang > > > > > > > > On Wed, Nov 18, 2015 at 9:02 AM, hsy...@gmail.com > > wrote: > > > > > Hey guys, > > > > > > I saw the PartitionAssignor is not in public doc API and the package > name > > > is internals. > > > > > > Does it mean this API is not stable and could be changed even in minor > > > release? > > > > > > And in the assign method signature, the key for the "subscription" map > is > > > memberId, what is memberId, can I manually set the id to identify > member? > > > I want to do some sticky assignment. > > > > > > > > > Thanks! > > > > > > Best, > > > Siyuan > > > > > > > > > > > -- > > -- Guozhang > > > -- -- Guozhang
Re: Q about PartitionAssignor
Thanks Guozhang, what is userData for in the Subscription? On Wed, Nov 18, 2015 at 12:05 PM, Guozhang Wang wrote: > Currently the whole KafkaConsumer interface is tagged as " > @InterfaceStability.Unstable", meaning that the API may change in the > future. We have been very careful to make any dramatic public API changes > but still cannot guarantee this will not happen. > > Member-Id is assigned by the server-side coordinator upon accepting the > consumer to join the specified group, hence it cannot be manually set. But > the memberId will not change as long as the consumer is still part of the > members of the group, so you want to do some sticky assignment you can just > remember the memberId -> partitions map on the consumer side in some > persistent storage so that even when the leader who does the assignment has > failed over other new leaders can still access the past assignment history. > > Guozhang > > > > On Wed, Nov 18, 2015 at 9:02 AM, hsy...@gmail.com > wrote: > > > Hey guys, > > > > I saw the PartitionAssignor is not in public doc API and the package name > > is internals. > > > > Does it mean this API is not stable and could be changed even in minor > > release? > > > > And in the assign method signature, the key for the "subscription" map is > > memberId, what is memberId, can I manually set the id to identify member? > > I want to do some sticky assignment. > > > > > > Thanks! > > > > Best, > > Siyuan > > > > > > -- > -- Guozhang >
Re: Q about PartitionAssignor
Currently the whole KafkaConsumer interface is tagged as " @InterfaceStability.Unstable", meaning that the API may change in the future. We have been very careful to make any dramatic public API changes but still cannot guarantee this will not happen. Member-Id is assigned by the server-side coordinator upon accepting the consumer to join the specified group, hence it cannot be manually set. But the memberId will not change as long as the consumer is still part of the members of the group, so you want to do some sticky assignment you can just remember the memberId -> partitions map on the consumer side in some persistent storage so that even when the leader who does the assignment has failed over other new leaders can still access the past assignment history. Guozhang On Wed, Nov 18, 2015 at 9:02 AM, hsy...@gmail.com wrote: > Hey guys, > > I saw the PartitionAssignor is not in public doc API and the package name > is internals. > > Does it mean this API is not stable and could be changed even in minor > release? > > And in the assign method signature, the key for the "subscription" map is > memberId, what is memberId, can I manually set the id to identify member? > I want to do some sticky assignment. > > > Thanks! > > Best, > Siyuan > -- -- Guozhang
Q about PartitionAssignor
Hey guys, I saw the PartitionAssignor is not in public doc API and the package name is internals. Does it mean this API is not stable and could be changed even in minor release? And in the assign method signature, the key for the "subscription" map is memberId, what is memberId, can I manually set the id to identify member? I want to do some sticky assignment. Thanks! Best, Siyuan