Hey Neha,
Basically partitionsForTopic will invoke
Metadata.fetch(topic).partitionsFor. So it will block on the first request
for a given topic while metadata is loaded. Each subsequent request will
return whatever metadata is present, the logic for refreshing metadata will
remain exactly as it is. It *is* expected that the user call this method on
every single call and because we no longer control the partitioner
interface there will be no way to control this.
Overall having looked at this more I think I am pretty neutral on the
serialization/partitioning thing. We do give up a lot by removing those
interfaces...
Regardless of whether you prefer org.apache or not, the server code doesn't
have that package. So we will be inconsistent with our own code, which
would be extremely weird. For example the maven co-ords would be
different...
-Jay
On Sat, Feb 1, 2014 at 2:52 PM, Neha Narkhede <neha.narkh...@gmail.com>wrote:
> 1. Change send to use java.util.concurrent.Future in send():
> Future<RecordPosition> send(ProducerRecord record, Callback callback)
> The cancel method will always return false and not do anything. Callback
> will then be changed to
> interface Callback {
> public void onCompletion(RecordPosition position, Exception exception);
> }
> so that the exception is available there too and will be null if no error
> occurred. I'm not planning on changing the name callback because I haven't
> thought of a better one.
>
> +1
>
> 2. We will change the way serialization works to proposal 1A in the
> previous discussion. That is the Partitioner and Serializer interfaces will
> disappear. ProducerRecord will change to:
> class ProducerRecord {
> public byte[] key() {...}
> public byte[] value() {...}
> public Integer partition() {...} // can be null
> }
> In order to allow correctly choosing a partition the Producer interface
> will include a new method:
> List<PartitionInfo> partitionsForTopic(String topic);
> PartitionInfo will be changed to include the actual Node objects not just
> the Node ids.
>
> Mostly agree with this but wanted to confirm my understanding of
> partitionsForTopic.
> So, is that something a user has to remember to invoke in order to detect
> newly
> added partitions to a topic? Will each invocation of partitionsForTopic
> lead to a
> TopicMetadataRequest RPC?
>
> 3. I will make the producer implement java.io.Closable but not throw any
> exceptions as there doesn't really seem to be any disadvantage to this and
> the interface may remind people to call close.
>
> +1
>
> Non-changes
>
> 1. I don't plan to change the build system. The SBT=>gradle change is
> basically orthoganol and we should debate it in the context of its ticket.
>
> +1
>
> 2. I'm going to stick with my oddball kafka.* rather than
> org.apache.kafka.* package name and non-getter methods unless everyone
> complains.
>
> -1. I prefer org.apache.kafka for consistency with other top level Apache
> projects
>
> 3. I'm not going to introduce a zookeeper dependency in the client as I
> don't see any advantage.
>
> +1. I see the point about non-java clients not having an easy way to talk
> to zookeeper,
> but don't really think the connections to zookeeper is the problem.
>
> 4. There were a number of reasonable suggestions on the Callback execution
> model. I'm going to leave it as is, though. Because we are moving to use
> Java Future we can't include functionality like ListenableFuture. I think
> the simplestic callback model where callbacks are executed in the i/o
> thread should be good enough for most cases and other cases can use their
> own threading.
>
> +1
>
>
> On Sat, Feb 1, 2014 at 9:06 AM, Joel Koshy <jjkosh...@gmail.com> wrote:
>
> > .
> > >
> > > > In order to allow correctly choosing a partition the Producer
> interface
> > > > will include a new method:
> > > > List<PartitionInfo> partitionsForTopic(String topic);
> > > > PartitionInfo will be changed to include the actual Node objects not
> > just
> > > > the Node ids.
> > >
> > > Why are the node id's alone insufficient? You can still do
> > > round-robin/connection-limiting, etc. with just the node id right?
> >
> >
> > Actually no - we do need the full node info.
> >
> >
> >
> >
> > --
> > Sent from Gmail Mobile
> >
>
