It *is* expected that the user call this method on
every single call and because we no longer control the partitioner
interface there will be no way to control this.
Make sense. This will ensure that new partitions are detected as defined by
topic.metadata.refresh.interval.ms.
There are a quite a few things that are inconsistent between the server and
client. For example, java vs scala.
Regardless, if we change the client to 'org.apache.kafka', the server
packaging would obviously have to change
as well to point to 'org.apache.kafka'. This feedback, to remain consistent
with other Apache projects, was punted
on when we first entered Apache, for no particular reason.
On Sun, Feb 2, 2014 at 9:42 AM, Jay Kreps <jay.kr...@gmail.com> wrote:
> Hey Neha,
>
> Basically partitionsForTopic will invoke
> Metadata.fetch(topic).partitionsFor. So it will block on the first request
> for a given topic while metadata is loaded. Each subsequent request will
> return whatever metadata is present, the logic for refreshing metadata will
> remain exactly as it is. It *is* expected that the user call this method on
> every single call and because we no longer control the partitioner
> interface there will be no way to control this.
>
> Overall having looked at this more I think I am pretty neutral on the
> serialization/partitioning thing. We do give up a lot by removing those
> interfaces...
>
> Regardless of whether you prefer org.apache or not, the server code doesn't
> have that package. So we will be inconsistent with our own code, which
> would be extremely weird. For example the maven co-ords would be
> different...
>
> -Jay
>
>
>
> On Sat, Feb 1, 2014 at 2:52 PM, Neha Narkhede <neha.narkh...@gmail.com
> >wrote:
>
> > 1. Change send to use java.util.concurrent.Future in send():
> > Future<RecordPosition> send(ProducerRecord record, Callback callback)
> > The cancel method will always return false and not do anything. Callback
> > will then be changed to
> > interface Callback {
> > public void onCompletion(RecordPosition position, Exception
> exception);
> > }
> > so that the exception is available there too and will be null if no error
> > occurred. I'm not planning on changing the name callback because I
> haven't
> > thought of a better one.
> >
> > +1
> >
> > 2. We will change the way serialization works to proposal 1A in the
> > previous discussion. That is the Partitioner and Serializer interfaces
> will
> > disappear. ProducerRecord will change to:
> > class ProducerRecord {
> > public byte[] key() {...}
> > public byte[] value() {...}
> > public Integer partition() {...} // can be null
> > }
> > In order to allow correctly choosing a partition the Producer interface
> > will include a new method:
> > List<PartitionInfo> partitionsForTopic(String topic);
> > PartitionInfo will be changed to include the actual Node objects not just
> > the Node ids.
> >
> > Mostly agree with this but wanted to confirm my understanding of
> > partitionsForTopic.
> > So, is that something a user has to remember to invoke in order to detect
> > newly
> > added partitions to a topic? Will each invocation of partitionsForTopic
> > lead to a
> > TopicMetadataRequest RPC?
> >
> > 3. I will make the producer implement java.io.Closable but not throw any
> > exceptions as there doesn't really seem to be any disadvantage to this
> and
> > the interface may remind people to call close.
> >
> > +1
> >
> > Non-changes
> >
> > 1. I don't plan to change the build system. The SBT=>gradle change is
> > basically orthoganol and we should debate it in the context of its
> ticket.
> >
> > +1
> >
> > 2. I'm going to stick with my oddball kafka.* rather than
> > org.apache.kafka.* package name and non-getter methods unless everyone
> > complains.
> >
> > -1. I prefer org.apache.kafka for consistency with other top level Apache
> > projects
> >
> > 3. I'm not going to introduce a zookeeper dependency in the client as I
> > don't see any advantage.
> >
> > +1. I see the point about non-java clients not having an easy way to talk
> > to zookeeper,
> > but don't really think the connections to zookeeper is the problem.
> >
> > 4. There were a number of reasonable suggestions on the Callback
> execution
> > model. I'm going to leave it as is, though. Because we are moving to use
> > Java Future we can't include functionality like ListenableFuture. I think
> > the simplestic callback model where callbacks are executed in the i/o
> > thread should be good enough for most cases and other cases can use their
> > own threading.
> >
> > +1
> >
> >
> > On Sat, Feb 1, 2014 at 9:06 AM, Joel Koshy <jjkosh...@gmail.com> wrote:
> >
> > > .
> > > >
> > > > > In order to allow correctly choosing a partition the Producer
> > interface
> > > > > will include a new method:
> > > > > List<PartitionInfo> partitionsForTopic(String topic);
> > > > > PartitionInfo will be changed to include the actual Node objects
> not
> > > just
> > > > > the Node ids.
> > > >
> > > > Why are the node id's alone insufficient? You can still do
> > > > round-robin/connection-limiting, etc. with just the node id right?
> > >
> > >
> > > Actually no - we do need the full node info.
> > >
> > >
> > >
> > >
> > > --
> > > Sent from Gmail Mobile
> > >
> >
>
