Re: Kafka 0.9 consumer API question

2015-12-17 Thread hsy...@gmail.com
Hi Rajiv, I think it makes sense to return a read-only assignments. What we can improve here is we can have addPartition method for consumer. Then we don't have to do any operations on the assignments returned by assignment method BTW, I think you can implement PartitionAssignor interface to

Re: Kafka 0.9 consumer API question

2015-12-15 Thread Jason Gustafson
Hey Rajiv, I agree the Set/List inconsistency is a little unfortunate (another annoying one is pause() which uses a vararg). I think we should probably add the following variants: assign(Collection) subscribe(Collection) pause(Collection) I can open a JIRA to fix this. As for returning the

Kafka 0.9 consumer API question

2015-12-15 Thread Rajiv Kurian
We are trying to use the Kafka 0.9 consumer API to poll specific partitions. We consume partitions based on our own logic instead of delegating that to Kafka. One of our use cases is handling a change in the partitions that we consume. This means that sometimes we need to consume additional

Re: Kafka 0.9 consumer API question

2015-12-15 Thread Jason Gustafson
Hey Rajiv, My point was that you could maintain the assignment set yourself in a field, which would eliminate the need to copy the set returned by assignment(). Then it's just one copy to convert it to a list, and we can fix this by adding the assign() variant I suggested above. By the way,

Re: Kafka 0.9 consumer API question

2015-12-15 Thread Rajiv Kurian
Hi Jason, The copying is not a problem in terms of performance. It's just annoying to write the extra code. My point with the copy is that since the client is already making a copy when it returns the set to me, why would it matter if I modify the copy. Creating an unmodifiable set on top of a

Re: Kafka 0.9 consumer API question

2015-12-15 Thread Rajiv Kurian
Right I could do that. Thanks for creating the JIRA! On Tue, Dec 15, 2015 at 3:01 PM, Jason Gustafson wrote: > Hey Rajiv, > > My point was that you could maintain the assignment set yourself in a > field, which would eliminate the need to copy the set returned by >