Hey Neha, Basically partitionsForTopic will invoke Metadata.fetch(topic).partitionsFor. So it will block on the first request for a given topic while metadata is loaded. Each subsequent request will return whatever metadata is present, the logic for refreshing metadata will remain exactly as it is. It *is* expected that the user call this method on every single call and because we no longer control the partitioner interface there will be no way to control this.
Overall having looked at this more I think I am pretty neutral on the serialization/partitioning thing. We do give up a lot by removing those interfaces... Regardless of whether you prefer org.apache or not, the server code doesn't have that package. So we will be inconsistent with our own code, which would be extremely weird. For example the maven co-ords would be different... -Jay On Sat, Feb 1, 2014 at 2:52 PM, Neha Narkhede <neha.narkh...@gmail.com>wrote: > 1. Change send to use java.util.concurrent.Future in send(): > Future<RecordPosition> send(ProducerRecord record, Callback callback) > The cancel method will always return false and not do anything. Callback > will then be changed to > interface Callback { > public void onCompletion(RecordPosition position, Exception exception); > } > so that the exception is available there too and will be null if no error > occurred. I'm not planning on changing the name callback because I haven't > thought of a better one. > > +1 > > 2. We will change the way serialization works to proposal 1A in the > previous discussion. That is the Partitioner and Serializer interfaces will > disappear. ProducerRecord will change to: > class ProducerRecord { > public byte[] key() {...} > public byte[] value() {...} > public Integer partition() {...} // can be null > } > In order to allow correctly choosing a partition the Producer interface > will include a new method: > List<PartitionInfo> partitionsForTopic(String topic); > PartitionInfo will be changed to include the actual Node objects not just > the Node ids. > > Mostly agree with this but wanted to confirm my understanding of > partitionsForTopic. > So, is that something a user has to remember to invoke in order to detect > newly > added partitions to a topic? Will each invocation of partitionsForTopic > lead to a > TopicMetadataRequest RPC? > > 3. I will make the producer implement java.io.Closable but not throw any > exceptions as there doesn't really seem to be any disadvantage to this and > the interface may remind people to call close. > > +1 > > Non-changes > > 1. I don't plan to change the build system. The SBT=>gradle change is > basically orthoganol and we should debate it in the context of its ticket. > > +1 > > 2. I'm going to stick with my oddball kafka.* rather than > org.apache.kafka.* package name and non-getter methods unless everyone > complains. > > -1. I prefer org.apache.kafka for consistency with other top level Apache > projects > > 3. I'm not going to introduce a zookeeper dependency in the client as I > don't see any advantage. > > +1. I see the point about non-java clients not having an easy way to talk > to zookeeper, > but don't really think the connections to zookeeper is the problem. > > 4. There were a number of reasonable suggestions on the Callback execution > model. I'm going to leave it as is, though. Because we are moving to use > Java Future we can't include functionality like ListenableFuture. I think > the simplestic callback model where callbacks are executed in the i/o > thread should be good enough for most cases and other cases can use their > own threading. > > +1 > > > On Sat, Feb 1, 2014 at 9:06 AM, Joel Koshy <jjkosh...@gmail.com> wrote: > > > . > > > > > > > In order to allow correctly choosing a partition the Producer > interface > > > > will include a new method: > > > > List<PartitionInfo> partitionsForTopic(String topic); > > > > PartitionInfo will be changed to include the actual Node objects not > > just > > > > the Node ids. > > > > > > Why are the node id's alone insufficient? You can still do > > > round-robin/connection-limiting, etc. with just the node id right? > > > > > > Actually no - we do need the full node info. > > > > > > > > > > -- > > Sent from Gmail Mobile > > >