Re: Helping out

[Jun Rao]

Yes. We will need to think a bit more for the more general case when there
are messages for multiple topics.

Thanks,

Jun


On Wed, Jul 3, 2013 at 11:18 AM, S Ahmed  wrote:

>
> https://github.com/apache/kafka/blob/0.8/core/src/main/scala/kafka/producer/async/DefaultEventHandler.scala#L152
>
>
>
> On Wed, Jul 3, 2013 at 2:18 PM, S Ahmed  wrote:
>
> > Jun,
> >
> > Looks like the update would happen around here?
> >
> >
> >
> >
> > On Wed, Jul 3, 2013 at 12:53 PM, Jun Rao  wrote:
> >
> >> Sam,
> >>
> >> Interested in taking a look at
> >> KAFKA-959
> >> ?
> >>
> >> Thanks,
> >>
> >> Jun
> >>
> >>
> >> On Sun, Jun 23, 2013 at 11:01 PM, Sam Meder  >> >wrote:
> >>
> >> > Hey,
> >> >
> >> > I now have roughly a day a week I can dedicate to working on Kafka,
> so I
> >> > am looking for issues in the 0.8.1 batch that you think might be good
> >> > starting points. Input would be much appreciated.
> >> >
> >> > Speaking of issues, I think it would be good to either fix
> >> > https://issues.apache.org/jira/browse/KAFKA-946 for 0.8 or just drop
> >> the
> >> > code from the release.
> >> >
> >> > /Sam
> >>
> >
> >
>

Re: Helping out

[Wang Guozhang]

Yes. And as you can see in that function, when no key is specified, it
chooses a partition for each message in a round-robin manner, while we want
to make all the messages in the same batch with the same topic to be sent
to one partition.

So if the message is the first seen with the topic in the batch, we can
choose one partition, but we want to stick to that partition with all the
rest messages with the same topic in this batch.

Guozhang


On Wed, Jul 3, 2013 at 11:18 AM, S Ahmed  wrote:

>
> https://github.com/apache/kafka/blob/0.8/core/src/main/scala/kafka/producer/async/DefaultEventHandler.scala#L152
>
>
>
> On Wed, Jul 3, 2013 at 2:18 PM, S Ahmed  wrote:
>
> > Jun,
> >
> > Looks like the update would happen around here?
> >
> >
> >
> >
> > On Wed, Jul 3, 2013 at 12:53 PM, Jun Rao  wrote:
> >
> >> Sam,
> >>
> >> Interested in taking a look at
> >> KAFKA-959
> >> ?
> >>
> >> Thanks,
> >>
> >> Jun
> >>
> >>
> >> On Sun, Jun 23, 2013 at 11:01 PM, Sam Meder  >> >wrote:
> >>
> >> > Hey,
> >> >
> >> > I now have roughly a day a week I can dedicate to working on Kafka,
> so I
> >> > am looking for issues in the 0.8.1 batch that you think might be good
> >> > starting points. Input would be much appreciated.
> >> >
> >> > Speaking of issues, I think it would be good to either fix
> >> > https://issues.apache.org/jira/browse/KAFKA-946 for 0.8 or just drop
> >> the
> >> > code from the release.
> >> >
> >> > /Sam
> >>
> >
> >
>



-- 
-- Guozhang

Re: Helping out

[S Ahmed]

https://github.com/apache/kafka/blob/0.8/core/src/main/scala/kafka/producer/async/DefaultEventHandler.scala#L152



On Wed, Jul 3, 2013 at 2:18 PM, S Ahmed  wrote:

> Jun,
>
> Looks like the update would happen around here?
>
>
>
>
> On Wed, Jul 3, 2013 at 12:53 PM, Jun Rao  wrote:
>
>> Sam,
>>
>> Interested in taking a look at
>> KAFKA-959
>> ?
>>
>> Thanks,
>>
>> Jun
>>
>>
>> On Sun, Jun 23, 2013 at 11:01 PM, Sam Meder > >wrote:
>>
>> > Hey,
>> >
>> > I now have roughly a day a week I can dedicate to working on Kafka, so I
>> > am looking for issues in the 0.8.1 batch that you think might be good
>> > starting points. Input would be much appreciated.
>> >
>> > Speaking of issues, I think it would be good to either fix
>> > https://issues.apache.org/jira/browse/KAFKA-946 for 0.8 or just drop
>> the
>> > code from the release.
>> >
>> > /Sam
>>
>
>

Re: Helping out

[S Ahmed]

Jun,

Looks like the update would happen around here?




On Wed, Jul 3, 2013 at 12:53 PM, Jun Rao  wrote:

> Sam,
>
> Interested in taking a look at
> KAFKA-959
> ?
>
> Thanks,
>
> Jun
>
>
> On Sun, Jun 23, 2013 at 11:01 PM, Sam Meder  >wrote:
>
> > Hey,
> >
> > I now have roughly a day a week I can dedicate to working on Kafka, so I
> > am looking for issues in the 0.8.1 batch that you think might be good
> > starting points. Input would be much appreciated.
> >
> > Speaking of issues, I think it would be good to either fix
> > https://issues.apache.org/jira/browse/KAFKA-946 for 0.8 or just drop the
> > code from the release.
> >
> > /Sam
>

Re: Helping out

[Jun Rao]

Sam,

Interested in taking a look at
KAFKA-959
?

Thanks,

Jun


On Sun, Jun 23, 2013 at 11:01 PM, Sam Meder wrote:

> Hey,
>
> I now have roughly a day a week I can dedicate to working on Kafka, so I
> am looking for issues in the 0.8.1 batch that you think might be good
> starting points. Input would be much appreciated.
>
> Speaking of issues, I think it would be good to either fix
> https://issues.apache.org/jira/browse/KAFKA-946 for 0.8 or just drop the
> code from the release.
>
> /Sam

Re: Helping out

[Jay Kreps]

I actually don't think quotas really needs to do cross-broker
communication. I think we can just enforce it at the topic-partition level
and have the leader do the check (even if the user expresses the quota at
the topic level we will just turn it into a per-partition limit by dividing
by the number of partitions). If any one is interested in this I would be
happy to help work through a design and help get it in.

-Jay


On Tue, Jun 25, 2013 at 5:43 PM, S Ahmed  wrote:

> Quotas is a cool idea, is it going to do cross broker communication or it
> doesn't need to talk to other brokers?
>
>
> On Mon, Jun 24, 2013 at 12:41 PM, Jay Kreps  wrote:
>
> > Hey Sam,
> >
> > That's great. I would be happy to act as a point of contact if any
> > questions come up or you need a reviewer for patches.
> >
> > We haven't had an official discussion of roadmap but trunk already has
> > support for keyed messages and log compaction and a cleanup of the log
> > code. I think the goal would be to make this release as small and
> > quick as possible with any bigger disruptive features going into 0.9.
> > Here are some ideas of things that I think would fit well:
> > 1. Any good housekeeping issues. 0.8 was a pretty major rewrite and
> > getting that done and stable required ignoring a lot of basic issues
> > that impact the out-of-the-box experience---good errors,
> > documentation, logging, etc. We haven't filed a lot of JIRAs for this,
> > but anything you see that falls under this umbrella would be a good
> > target. Doing a few of these is often a good way to learn the code
> > base.
> > 2. Quotas. Jonathan had proposed working on this, but I don't think he
> > ever got the time. https://issues.apache.org/jira/browse/KAFKA-656
> > 3. Phase 2 of this wiki:
> > https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management
> >
> > Folks--any other ideas?
> >
> > -Jay
> >
> >
> >
> > On Sun, Jun 23, 2013 at 11:01 PM, Sam Meder 
> > wrote:
> > > Hey,
> > >
> > > I now have roughly a day a week I can dedicate to working on Kafka, so
> I
> > am looking for issues in the 0.8.1 batch that you think might be good
> > starting points. Input would be much appreciated.
> > >
> > > Speaking of issues, I think it would be good to either fix
> > https://issues.apache.org/jira/browse/KAFKA-946 for 0.8 or just drop the
> > code from the release.
> > >
> > > /Sam
> >
>

Re: Helping out

[Jay Kreps]

Actually the intern ended up working on offset storage
(https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management)
in case any one is still interested in quotas.

-Jay

On Tue, Jun 25, 2013 at 8:54 PM, Jun Rao  wrote:
> As for offset management, we have a summer intern starting this week and
> our plan is to have the intern work on this.
>
> Thanks,
>
> Jun
>
>
> On Tue, Jun 25, 2013 at 7:53 AM, Sam Meder wrote:
>
>> On Jun 24, 2013, at 9:41 AM, Jay Kreps  wrote:
>> > Hey Sam,
>> >
>> > That's great. I would be happy to act as a point of contact if any
>> > questions come up or you need a reviewer for patches.
>> >
>>
>> Great, I have a few patches outstanding, so if you could take a look:
>>
>> https://issues.apache.org/jira/browse/KAFKA-943
>> https://issues.apache.org/jira/browse/KAFKA-946
>> https://issues.apache.org/jira/browse/KAFKA-956
>>
>> > We haven't had an official discussion of roadmap but trunk already has
>> > support for keyed messages and log compaction and a cleanup of the log
>> > code. I think the goal would be to make this release as small and
>> > quick as possible with any bigger disruptive features going into 0.9.
>>
>> So the plan is to base 0.8.1 on trunk?
>>
>> > Here are some ideas of things that I think would fit well:
>> > 1. Any good housekeeping issues. 0.8 was a pretty major rewrite and
>> > getting that done and stable required ignoring a lot of basic issues
>> > that impact the out-of-the-box experience---good errors,
>> > documentation, logging, etc. We haven't filed a lot of JIRAs for this,
>> > but anything you see that falls under this umbrella would be a good
>> > target. Doing a few of these is often a good way to learn the code
>> > base.
>>
>> Mostly what I have been trying to do - some classes could probably also
>> use some splitting up. I'll continue down this path.
>>
>> > 2. Quotas. Jonathan had proposed working on this, but I don't think he
>> > ever got the time. https://issues.apache.org/jira/browse/KAFKA-656
>> > 3. Phase 2 of this wiki:
>> > https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management
>> >
>>
>> For us 3 seems definitely more valuable than 2. We'd also really like to
>> have support for deleting topics (i.e. continue work on
>> https://issues.apache.org/jira/browse/KAFKA-330) and would potentially
>> want to change the build so it creates separate server and client artifacts.
>>
>> /Sam
>>
>>
>> > Folks--any other ideas?
>> >
>> > -Jay
>> >
>> >
>> >
>> > On Sun, Jun 23, 2013 at 11:01 PM, Sam Meder 
>> wrote:
>> >> Hey,
>> >>
>> >> I now have roughly a day a week I can dedicate to working on Kafka, so
>> I am looking for issues in the 0.8.1 batch that you think might be good
>> starting points. Input would be much appreciated.
>> >>
>> >> Speaking of issues, I think it would be good to either fix
>> https://issues.apache.org/jira/browse/KAFKA-946 for 0.8 or just drop the
>> code from the release.
>> >>
>> >> /Sam
>>
>>

Re: Helping out

[Jun Rao]

Sam,

Also, just want to say thank you for fixing kafka-938. That's an awesome
fix!

Jun


On Sun, Jun 23, 2013 at 11:01 PM, Sam Meder wrote:

> Hey,
>
> I now have roughly a day a week I can dedicate to working on Kafka, so I
> am looking for issues in the 0.8.1 batch that you think might be good
> starting points. Input would be much appreciated.
>
> Speaking of issues, I think it would be good to either fix
> https://issues.apache.org/jira/browse/KAFKA-946 for 0.8 or just drop the
> code from the release.
>
> /Sam

Re: Helping out

[Jun Rao]

As for offset management, we have a summer intern starting this week and
our plan is to have the intern work on this.

Thanks,

Jun


On Tue, Jun 25, 2013 at 7:53 AM, Sam Meder wrote:

> On Jun 24, 2013, at 9:41 AM, Jay Kreps  wrote:
> > Hey Sam,
> >
> > That's great. I would be happy to act as a point of contact if any
> > questions come up or you need a reviewer for patches.
> >
>
> Great, I have a few patches outstanding, so if you could take a look:
>
> https://issues.apache.org/jira/browse/KAFKA-943
> https://issues.apache.org/jira/browse/KAFKA-946
> https://issues.apache.org/jira/browse/KAFKA-956
>
> > We haven't had an official discussion of roadmap but trunk already has
> > support for keyed messages and log compaction and a cleanup of the log
> > code. I think the goal would be to make this release as small and
> > quick as possible with any bigger disruptive features going into 0.9.
>
> So the plan is to base 0.8.1 on trunk?
>
> > Here are some ideas of things that I think would fit well:
> > 1. Any good housekeeping issues. 0.8 was a pretty major rewrite and
> > getting that done and stable required ignoring a lot of basic issues
> > that impact the out-of-the-box experience---good errors,
> > documentation, logging, etc. We haven't filed a lot of JIRAs for this,
> > but anything you see that falls under this umbrella would be a good
> > target. Doing a few of these is often a good way to learn the code
> > base.
>
> Mostly what I have been trying to do - some classes could probably also
> use some splitting up. I'll continue down this path.
>
> > 2. Quotas. Jonathan had proposed working on this, but I don't think he
> > ever got the time. https://issues.apache.org/jira/browse/KAFKA-656
> > 3. Phase 2 of this wiki:
> > https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management
> >
>
> For us 3 seems definitely more valuable than 2. We'd also really like to
> have support for deleting topics (i.e. continue work on
> https://issues.apache.org/jira/browse/KAFKA-330) and would potentially
> want to change the build so it creates separate server and client artifacts.
>
> /Sam
>
>
> > Folks--any other ideas?
> >
> > -Jay
> >
> >
> >
> > On Sun, Jun 23, 2013 at 11:01 PM, Sam Meder 
> wrote:
> >> Hey,
> >>
> >> I now have roughly a day a week I can dedicate to working on Kafka, so
> I am looking for issues in the 0.8.1 batch that you think might be good
> starting points. Input would be much appreciated.
> >>
> >> Speaking of issues, I think it would be good to either fix
> https://issues.apache.org/jira/browse/KAFKA-946 for 0.8 or just drop the
> code from the release.
> >>
> >> /Sam
>
>

Re: Helping out

[S Ahmed]

Quotas is a cool idea, is it going to do cross broker communication or it
doesn't need to talk to other brokers?


On Mon, Jun 24, 2013 at 12:41 PM, Jay Kreps  wrote:

> Hey Sam,
>
> That's great. I would be happy to act as a point of contact if any
> questions come up or you need a reviewer for patches.
>
> We haven't had an official discussion of roadmap but trunk already has
> support for keyed messages and log compaction and a cleanup of the log
> code. I think the goal would be to make this release as small and
> quick as possible with any bigger disruptive features going into 0.9.
> Here are some ideas of things that I think would fit well:
> 1. Any good housekeeping issues. 0.8 was a pretty major rewrite and
> getting that done and stable required ignoring a lot of basic issues
> that impact the out-of-the-box experience---good errors,
> documentation, logging, etc. We haven't filed a lot of JIRAs for this,
> but anything you see that falls under this umbrella would be a good
> target. Doing a few of these is often a good way to learn the code
> base.
> 2. Quotas. Jonathan had proposed working on this, but I don't think he
> ever got the time. https://issues.apache.org/jira/browse/KAFKA-656
> 3. Phase 2 of this wiki:
> https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management
>
> Folks--any other ideas?
>
> -Jay
>
>
>
> On Sun, Jun 23, 2013 at 11:01 PM, Sam Meder 
> wrote:
> > Hey,
> >
> > I now have roughly a day a week I can dedicate to working on Kafka, so I
> am looking for issues in the 0.8.1 batch that you think might be good
> starting points. Input would be much appreciated.
> >
> > Speaking of issues, I think it would be good to either fix
> https://issues.apache.org/jira/browse/KAFKA-946 for 0.8 or just drop the
> code from the release.
> >
> > /Sam
>

Re: Helping out

[Jay Kreps]

Yeah I can take a look at those.

The plan for branching is to do all development off trunk and cut
branches for releases. 0.8 was sort of a one-off in this respect.
Going forward the idea is that all new code will go against trunk.

Topic delete would also be a great one.

I would recommend we hold off on splitting the jars. I think post
0.8.1 (0.9?) would be a better target. The reason is that we have a
bunch of major surgery to do there--move to nio, fix approach to
request definition, redo producer and consumer api, etc--and I think
it will be a lot easier to do all the surgery at once.

-Jay

On Tue, Jun 25, 2013 at 7:53 AM, Sam Meder  wrote:
> On Jun 24, 2013, at 9:41 AM, Jay Kreps  wrote:
>> Hey Sam,
>>
>> That's great. I would be happy to act as a point of contact if any
>> questions come up or you need a reviewer for patches.
>>
>
> Great, I have a few patches outstanding, so if you could take a look:
>
> https://issues.apache.org/jira/browse/KAFKA-943
> https://issues.apache.org/jira/browse/KAFKA-946
> https://issues.apache.org/jira/browse/KAFKA-956
>
>> We haven't had an official discussion of roadmap but trunk already has
>> support for keyed messages and log compaction and a cleanup of the log
>> code. I think the goal would be to make this release as small and
>> quick as possible with any bigger disruptive features going into 0.9.
>
> So the plan is to base 0.8.1 on trunk?
>
>> Here are some ideas of things that I think would fit well:
>> 1. Any good housekeeping issues. 0.8 was a pretty major rewrite and
>> getting that done and stable required ignoring a lot of basic issues
>> that impact the out-of-the-box experience---good errors,
>> documentation, logging, etc. We haven't filed a lot of JIRAs for this,
>> but anything you see that falls under this umbrella would be a good
>> target. Doing a few of these is often a good way to learn the code
>> base.
>
> Mostly what I have been trying to do - some classes could probably also use 
> some splitting up. I'll continue down this path.
>
>> 2. Quotas. Jonathan had proposed working on this, but I don't think he
>> ever got the time. https://issues.apache.org/jira/browse/KAFKA-656
>> 3. Phase 2 of this wiki:
>> https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management
>>
>
> For us 3 seems definitely more valuable than 2. We'd also really like to have 
> support for deleting topics (i.e. continue work on 
> https://issues.apache.org/jira/browse/KAFKA-330) and would potentially want 
> to change the build so it creates separate server and client artifacts.
>
> /Sam
>
>
>> Folks--any other ideas?
>>
>> -Jay
>>
>>
>>
>> On Sun, Jun 23, 2013 at 11:01 PM, Sam Meder  
>> wrote:
>>> Hey,
>>>
>>> I now have roughly a day a week I can dedicate to working on Kafka, so I am 
>>> looking for issues in the 0.8.1 batch that you think might be good starting 
>>> points. Input would be much appreciated.
>>>
>>> Speaking of issues, I think it would be good to either fix 
>>> https://issues.apache.org/jira/browse/KAFKA-946 for 0.8 or just drop the 
>>> code from the release.
>>>
>>> /Sam
>

Re: Helping out

[Sam Meder]

On Jun 24, 2013, at 9:41 AM, Jay Kreps  wrote:
> Hey Sam,
> 
> That's great. I would be happy to act as a point of contact if any
> questions come up or you need a reviewer for patches.
> 

Great, I have a few patches outstanding, so if you could take a look:

https://issues.apache.org/jira/browse/KAFKA-943
https://issues.apache.org/jira/browse/KAFKA-946
https://issues.apache.org/jira/browse/KAFKA-956

> We haven't had an official discussion of roadmap but trunk already has
> support for keyed messages and log compaction and a cleanup of the log
> code. I think the goal would be to make this release as small and
> quick as possible with any bigger disruptive features going into 0.9.

So the plan is to base 0.8.1 on trunk?

> Here are some ideas of things that I think would fit well:
> 1. Any good housekeeping issues. 0.8 was a pretty major rewrite and
> getting that done and stable required ignoring a lot of basic issues
> that impact the out-of-the-box experience---good errors,
> documentation, logging, etc. We haven't filed a lot of JIRAs for this,
> but anything you see that falls under this umbrella would be a good
> target. Doing a few of these is often a good way to learn the code
> base.

Mostly what I have been trying to do - some classes could probably also use 
some splitting up. I'll continue down this path.

> 2. Quotas. Jonathan had proposed working on this, but I don't think he
> ever got the time. https://issues.apache.org/jira/browse/KAFKA-656
> 3. Phase 2 of this wiki:
> https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management
> 

For us 3 seems definitely more valuable than 2. We'd also really like to have 
support for deleting topics (i.e. continue work on 
https://issues.apache.org/jira/browse/KAFKA-330) and would potentially want to 
change the build so it creates separate server and client artifacts.

/Sam


> Folks--any other ideas?
> 
> -Jay
> 
> 
> 
> On Sun, Jun 23, 2013 at 11:01 PM, Sam Meder  
> wrote:
>> Hey,
>> 
>> I now have roughly a day a week I can dedicate to working on Kafka, so I am 
>> looking for issues in the 0.8.1 batch that you think might be good starting 
>> points. Input would be much appreciated.
>> 
>> Speaking of issues, I think it would be good to either fix 
>> https://issues.apache.org/jira/browse/KAFKA-946 for 0.8 or just drop the 
>> code from the release.
>> 
>> /Sam

Re: Helping out

[Jay Kreps]

Hey Sam,

That's great. I would be happy to act as a point of contact if any
questions come up or you need a reviewer for patches.

We haven't had an official discussion of roadmap but trunk already has
support for keyed messages and log compaction and a cleanup of the log
code. I think the goal would be to make this release as small and
quick as possible with any bigger disruptive features going into 0.9.
Here are some ideas of things that I think would fit well:
1. Any good housekeeping issues. 0.8 was a pretty major rewrite and
getting that done and stable required ignoring a lot of basic issues
that impact the out-of-the-box experience---good errors,
documentation, logging, etc. We haven't filed a lot of JIRAs for this,
but anything you see that falls under this umbrella would be a good
target. Doing a few of these is often a good way to learn the code
base.
2. Quotas. Jonathan had proposed working on this, but I don't think he
ever got the time. https://issues.apache.org/jira/browse/KAFKA-656
3. Phase 2 of this wiki:
https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management

Folks--any other ideas?

-Jay



On Sun, Jun 23, 2013 at 11:01 PM, Sam Meder  wrote:
> Hey,
>
> I now have roughly a day a week I can dedicate to working on Kafka, so I am 
> looking for issues in the 0.8.1 batch that you think might be good starting 
> points. Input would be much appreciated.
>
> Speaking of issues, I think it would be good to either fix 
> https://issues.apache.org/jira/browse/KAFKA-946 for 0.8 or just drop the code 
> from the release.
>
> /Sam