[jira] [Commented] (KAFKA-835) Update 0.8 configs on the website

[Swapnil Ghike (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13698668#comment-13698668
 ] 

Swapnil Ghike commented on KAFKA-835:
-

This was committed http://kafka.apache.org/08/configuration.html 



> Update 0.8 configs on the website
> -
>
> Key: KAFKA-835
> URL: https://issues.apache.org/jira/browse/KAFKA-835
> Project: Kafka
>  Issue Type: Sub-task
>Affects Versions: 0.8
>Reporter: Neha Narkhede
>Assignee: Swapnil Ghike
> Attachments: config-changes.patch, config-changes-v2.patch, 
> config-changes-v3.patch, config-changes-v4.patch
>
>
> We renamed existing configs and also added new configs in 0.8.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: criteria for fixes on 0.8 beta

[Jun Rao]

My vote is that a patch can go into 0.8 if (1) it fixes a critical issue or
(2) the change is trivial and it makes the 0.8 experience better (e.g.,
improving log4j readability). kafka-946 may fall into (2).

Thanks,

Jun


On Tue, Jul 2, 2013 at 7:34 PM, Joe Stein  wrote:

> << How about for now: by default we will not incorporate fixes into 0.8
> unless there is a compelling argument (e.g., regression/clear bug with no
> good workaround) to do so.
>
> +1
>
>
> On Tue, Jul 2, 2013 at 8:48 PM, Joel Koshy  wrote:
>
> > Good question. Some fixes are clearly critical (e.g., consumer
> > deadlocks) that would impact everyone and need to go into 0.8.
> > Unfortunately the criticality of most other fixes is subjective and
> > I'm not sure how feasible it is to develop a global criteria. It
> > probably needs to be determined through consensus whether it needs to
> > go into 0.8 or not. How about for now: by default we will not
> > incorporate fixes into 0.8 unless there is a compelling argument
> > (e.g., regression/clear bug with no good workaround) to do so.
> >
> > Joel
> >
> >
> > On Tue, Jul 2, 2013 at 5:11 PM, Jay Kreps  wrote:
> > > What should the criteria for fixes on 0.8 be? This seems like a
> > reasonable
> > > candidate but I don't think we discussed what we would be taking so I
> > > thought I would ask...
> > >
> > > https://issues.apache.org/jira/browse/KAFKA-946
> > >
> > > -Jay
> >
>
>
>
> --
>
> /*
> Joe Stein
> http://www.linkedin.com/in/charmalloc
> Twitter: @allthingshadoop 
> */
>

[jira] [Created] (KAFKA-957) MirrorMaker needs to preserve the key in the source cluster

[Jun Rao (JIRA)]

Jun Rao created KAFKA-957:
-

 Summary: MirrorMaker needs to preserve the key in the source 
cluster
 Key: KAFKA-957
 URL: https://issues.apache.org/jira/browse/KAFKA-957
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8
Reporter: Jun Rao


Currently, MirrorMaker only propagates the message to the target cluster, but 
not the associated key.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (KAFKA-718) kafka-run-class.sh should use reasonable gc settings

[Jay Kreps (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13698576#comment-13698576
 ] 

Jay Kreps commented on KAFKA-718:
-

That would be great.

> kafka-run-class.sh should use reasonable gc settings
> 
>
> Key: KAFKA-718
> URL: https://issues.apache.org/jira/browse/KAFKA-718
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8
>Reporter: Jay Kreps
>Assignee: Neha Narkhede
>
> Our start script seems to use the default "stop the world" collector. It 
> would be good to default to well tuned gc settings including gc logging, CMS, 
> etc. Whatever we are using in prod and perf lab...
> Many people who want to use kafka basically don't know java well so they 
> won't succeed in figuring this stuff out on their own and just think it is 
> broken and timing out if we don't have good defaults.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (KAFKA-718) kafka-run-class.sh should use reasonable gc settings

[Ashwanth Fernando (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13698560#comment-13698560
 ] 

Ashwanth Fernando commented on KAFKA-718:
-

Hi, This bug seems to be open for a long time. I can take this up if its not 
been paid attention.

> kafka-run-class.sh should use reasonable gc settings
> 
>
> Key: KAFKA-718
> URL: https://issues.apache.org/jira/browse/KAFKA-718
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8
>Reporter: Jay Kreps
>Assignee: Neha Narkhede
>
> Our start script seems to use the default "stop the world" collector. It 
> would be good to default to well tuned gc settings including gc logging, CMS, 
> etc. Whatever we are using in prod and perf lab...
> Many people who want to use kafka basically don't know java well so they 
> won't succeed in figuring this stuff out on their own and just think it is 
> broken and timing out if we don't have good defaults.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (KAFKA-616) Implement acks=0

[Jay Kreps (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jay Kreps resolved KAFKA-616.
-

Resolution: Fixed
  Assignee: Neha Narkhede

Neha did it :-)

> Implement acks=0
> 
>
> Key: KAFKA-616
> URL: https://issues.apache.org/jira/browse/KAFKA-616
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8
>Reporter: Jay Kreps
>Assignee: Neha Narkhede
>  Labels: newbie
>
> For completeness it would be nice to handle the case where acks=0 in the 
> produce request. The meaning of this would be that the broker immediately 
> responds without blocking even on the local write. The advantage of this is 
> that it would often isolate the producer from any latency in the local write 
> (which we have occasionally seen).
> Since we don't block on the append the response would contain a placeholder 
> for all the fields--e.g. offset=-1 and no error.
> This should be pretty easy to implement, just an if statement in 
> KafkaApis.handleProduceRequest to send the response immediately in this case 
> (and again to avoid sending a second response later).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (KAFKA-555) Add a key-based log retention strategy

[Jay Kreps (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jay Kreps resolved KAFKA-555.
-

Resolution: Duplicate
  Assignee: Jay Kreps

Did it.

> Add a key-based log retention strategy
> --
>
> Key: KAFKA-555
> URL: https://issues.apache.org/jira/browse/KAFKA-555
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Jay Kreps
>Assignee: Jay Kreps
>
> Currently we have two log retention strategies: one based on time and one 
> based on log size. These work well for "event" type data--i.e. data that 
> consists only of appends. However if the events model changes to an 
> underlying keyed data set, a more convenient retention strategy would delete 
> keys that had been overwritten rather than retaining whole segments.
> The proposed implementation would be a background process that scanned log 
> segments and recopied only keys that hadn't been overwritten. Some more 
> details are in this wiki:
> https://cwiki.apache.org/confluence/display/KAFKA/Keyed+Messages+Proposal

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (KAFKA-739) Handle null values in Message payload

[Jay Kreps (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jay Kreps resolved KAFKA-739.
-

Resolution: Fixed

> Handle null values in Message payload
> -
>
> Key: KAFKA-739
> URL: https://issues.apache.org/jira/browse/KAFKA-739
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jay Kreps
>Assignee: Jay Kreps
> Fix For: 0.8.1
>
> Attachments: KAFKA-739-v1.patch, KAFKA-739-v2.patch, 
> KAFKA-739-v3.patch, KAFKA-739-v4.patch, KAFKA-739-v5.patch
>
>
> Add tests for null message payloads in producer, server, and consumer.
> Ensure log cleaner treats these as deletes.
> Test that null keys are rejected on dedupe logs.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (KAFKA-835) Update 0.8 configs on the website

[Jay Kreps (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13698548#comment-13698548
 ] 

Jay Kreps commented on KAFKA-835:
-

Also if you could "rebase" off current site html...

> Update 0.8 configs on the website
> -
>
> Key: KAFKA-835
> URL: https://issues.apache.org/jira/browse/KAFKA-835
> Project: Kafka
>  Issue Type: Sub-task
>Affects Versions: 0.8
>Reporter: Neha Narkhede
>Assignee: Swapnil Ghike
> Attachments: config-changes.patch, config-changes-v2.patch, 
> config-changes-v3.patch, config-changes-v4.patch
>
>
> We renamed existing configs and also added new configs in 0.8.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (KAFKA-835) Update 0.8 configs on the website

[Jay Kreps (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13698547#comment-13698547
 ] 

Jay Kreps commented on KAFKA-835:
-

Swapnil, is this still outstanding? If so I can apply...

> Update 0.8 configs on the website
> -
>
> Key: KAFKA-835
> URL: https://issues.apache.org/jira/browse/KAFKA-835
> Project: Kafka
>  Issue Type: Sub-task
>Affects Versions: 0.8
>Reporter: Neha Narkhede
>Assignee: Swapnil Ghike
> Attachments: config-changes.patch, config-changes-v2.patch, 
> config-changes-v3.patch, config-changes-v4.patch
>
>
> We renamed existing configs and also added new configs in 0.8.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (KAFKA-893) Unable to access kafka via Tomcat

[Jay Kreps (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13698545#comment-13698545
 ] 

Jay Kreps commented on KAFKA-893:
-

I think this happens if you have the wrong version of scala on your classpath.

> Unable to access kafka via Tomcat
> -
>
> Key: KAFKA-893
> URL: https://issues.apache.org/jira/browse/KAFKA-893
> Project: Kafka
>  Issue Type: Bug
>  Components: tools
> Environment: Apache Tomcat 7
>Reporter: arathi m
>
> Iam using kafka_2.8.0-0.8-SNAPSHOT.jar and Apache Tomcat 7. Upon deploying, 
> the Consumer example (provided at Kafka quickstart) in a Tomcat web app, I 
> get the following error 
> java.lang.VerifyError: class scala.Tuple2$mcLL$sp overrides final method 
> _1.()Ljava/lang/Object;
>   java.lang.ClassLoader.defineClass1(Native Method)
>   java.lang.ClassLoader.defineClass(ClassLoader.java:791)
>   java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>   
> org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2888)
>   
> org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:1172)
>   
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1680)
>   
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1558)
>   kafka.consumer.ConsumerConfig.(ConsumerConfig.scala:75)
>   TestServlet.kafkaconsumer(TestServlet.java:44)
>   TestServlet.doGet(TestServlet.java:20)
>   javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
>   javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
> Please suggest what should be done to fix this error.
> Thanks in advance.
> Arathi

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (KAFKA-918) Change log.retention.hours to be log.retention.mins

[Jay Kreps (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13698541#comment-13698541
 ] 

Jay Kreps commented on KAFKA-918:
-

This make sense, should be a pretty easy fix. We would definitely take a patch 
against trunk if could still support the older property as a fallback for 
compatibility.

> Change log.retention.hours to be log.retention.mins
> ---
>
> Key: KAFKA-918
> URL: https://issues.apache.org/jira/browse/KAFKA-918
> Project: Kafka
>  Issue Type: New Feature
>  Components: config
>Affects Versions: 0.7.2
>Reporter: Jason Weiss
>  Labels: features
>
> We stood up a cluster that is processing over 350,000 events per second, with 
> each event a fixed payload size of 2K. The storage required to process that 
> much data over an hour is beyond what we wanted to pay for at AWS. 
> Additionally, we don't have a requirement to keep the files around for an 
> extended period after processing.
> It would be tremendously valuable for us to be able to define the 
> log.retention in minutes, not hours. For example, we would prefer to only 
> keep 30 minutes of logs around.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Helping out

[Jay Kreps]

I actually don't think quotas really needs to do cross-broker
communication. I think we can just enforce it at the topic-partition level
and have the leader do the check (even if the user expresses the quota at
the topic level we will just turn it into a per-partition limit by dividing
by the number of partitions). If any one is interested in this I would be
happy to help work through a design and help get it in.

-Jay


On Tue, Jun 25, 2013 at 5:43 PM, S Ahmed  wrote:

> Quotas is a cool idea, is it going to do cross broker communication or it
> doesn't need to talk to other brokers?
>
>
> On Mon, Jun 24, 2013 at 12:41 PM, Jay Kreps  wrote:
>
> > Hey Sam,
> >
> > That's great. I would be happy to act as a point of contact if any
> > questions come up or you need a reviewer for patches.
> >
> > We haven't had an official discussion of roadmap but trunk already has
> > support for keyed messages and log compaction and a cleanup of the log
> > code. I think the goal would be to make this release as small and
> > quick as possible with any bigger disruptive features going into 0.9.
> > Here are some ideas of things that I think would fit well:
> > 1. Any good housekeeping issues. 0.8 was a pretty major rewrite and
> > getting that done and stable required ignoring a lot of basic issues
> > that impact the out-of-the-box experience---good errors,
> > documentation, logging, etc. We haven't filed a lot of JIRAs for this,
> > but anything you see that falls under this umbrella would be a good
> > target. Doing a few of these is often a good way to learn the code
> > base.
> > 2. Quotas. Jonathan had proposed working on this, but I don't think he
> > ever got the time. https://issues.apache.org/jira/browse/KAFKA-656
> > 3. Phase 2 of this wiki:
> > https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management
> >
> > Folks--any other ideas?
> >
> > -Jay
> >
> >
> >
> > On Sun, Jun 23, 2013 at 11:01 PM, Sam Meder 
> > wrote:
> > > Hey,
> > >
> > > I now have roughly a day a week I can dedicate to working on Kafka, so
> I
> > am looking for issues in the 0.8.1 batch that you think might be good
> > starting points. Input would be much appreciated.
> > >
> > > Speaking of issues, I think it would be good to either fix
> > https://issues.apache.org/jira/browse/KAFKA-946 for 0.8 or just drop the
> > code from the release.
> > >
> > > /Sam
> >
>

Re: Helping out

[Jay Kreps]

Actually the intern ended up working on offset storage
(https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management)
in case any one is still interested in quotas.

-Jay

On Tue, Jun 25, 2013 at 8:54 PM, Jun Rao  wrote:
> As for offset management, we have a summer intern starting this week and
> our plan is to have the intern work on this.
>
> Thanks,
>
> Jun
>
>
> On Tue, Jun 25, 2013 at 7:53 AM, Sam Meder wrote:
>
>> On Jun 24, 2013, at 9:41 AM, Jay Kreps  wrote:
>> > Hey Sam,
>> >
>> > That's great. I would be happy to act as a point of contact if any
>> > questions come up or you need a reviewer for patches.
>> >
>>
>> Great, I have a few patches outstanding, so if you could take a look:
>>
>> https://issues.apache.org/jira/browse/KAFKA-943
>> https://issues.apache.org/jira/browse/KAFKA-946
>> https://issues.apache.org/jira/browse/KAFKA-956
>>
>> > We haven't had an official discussion of roadmap but trunk already has
>> > support for keyed messages and log compaction and a cleanup of the log
>> > code. I think the goal would be to make this release as small and
>> > quick as possible with any bigger disruptive features going into 0.9.
>>
>> So the plan is to base 0.8.1 on trunk?
>>
>> > Here are some ideas of things that I think would fit well:
>> > 1. Any good housekeeping issues. 0.8 was a pretty major rewrite and
>> > getting that done and stable required ignoring a lot of basic issues
>> > that impact the out-of-the-box experience---good errors,
>> > documentation, logging, etc. We haven't filed a lot of JIRAs for this,
>> > but anything you see that falls under this umbrella would be a good
>> > target. Doing a few of these is often a good way to learn the code
>> > base.
>>
>> Mostly what I have been trying to do - some classes could probably also
>> use some splitting up. I'll continue down this path.
>>
>> > 2. Quotas. Jonathan had proposed working on this, but I don't think he
>> > ever got the time. https://issues.apache.org/jira/browse/KAFKA-656
>> > 3. Phase 2 of this wiki:
>> > https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management
>> >
>>
>> For us 3 seems definitely more valuable than 2. We'd also really like to
>> have support for deleting topics (i.e. continue work on
>> https://issues.apache.org/jira/browse/KAFKA-330) and would potentially
>> want to change the build so it creates separate server and client artifacts.
>>
>> /Sam
>>
>>
>> > Folks--any other ideas?
>> >
>> > -Jay
>> >
>> >
>> >
>> > On Sun, Jun 23, 2013 at 11:01 PM, Sam Meder 
>> wrote:
>> >> Hey,
>> >>
>> >> I now have roughly a day a week I can dedicate to working on Kafka, so
>> I am looking for issues in the 0.8.1 batch that you think might be good
>> starting points. Input would be much appreciated.
>> >>
>> >> Speaking of issues, I think it would be good to either fix
>> https://issues.apache.org/jira/browse/KAFKA-946 for 0.8 or just drop the
>> code from the release.
>> >>
>> >> /Sam
>>
>>

Re: criteria for fixes on 0.8 beta

[Joe Stein]

<< How about for now: by default we will not incorporate fixes into 0.8
unless there is a compelling argument (e.g., regression/clear bug with no
good workaround) to do so.

+1


On Tue, Jul 2, 2013 at 8:48 PM, Joel Koshy  wrote:

> Good question. Some fixes are clearly critical (e.g., consumer
> deadlocks) that would impact everyone and need to go into 0.8.
> Unfortunately the criticality of most other fixes is subjective and
> I'm not sure how feasible it is to develop a global criteria. It
> probably needs to be determined through consensus whether it needs to
> go into 0.8 or not. How about for now: by default we will not
> incorporate fixes into 0.8 unless there is a compelling argument
> (e.g., regression/clear bug with no good workaround) to do so.
>
> Joel
>
>
> On Tue, Jul 2, 2013 at 5:11 PM, Jay Kreps  wrote:
> > What should the criteria for fixes on 0.8 be? This seems like a
> reasonable
> > candidate but I don't think we discussed what we would be taking so I
> > thought I would ask...
> >
> > https://issues.apache.org/jira/browse/KAFKA-946
> >
> > -Jay
>



-- 

/*
Joe Stein
http://www.linkedin.com/in/charmalloc
Twitter: @allthingshadoop 
*/

Re: criteria for fixes on 0.8 beta

[Joel Koshy]

Good question. Some fixes are clearly critical (e.g., consumer
deadlocks) that would impact everyone and need to go into 0.8.
Unfortunately the criticality of most other fixes is subjective and
I'm not sure how feasible it is to develop a global criteria. It
probably needs to be determined through consensus whether it needs to
go into 0.8 or not. How about for now: by default we will not
incorporate fixes into 0.8 unless there is a compelling argument
(e.g., regression/clear bug with no good workaround) to do so.

Joel


On Tue, Jul 2, 2013 at 5:11 PM, Jay Kreps  wrote:
> What should the criteria for fixes on 0.8 be? This seems like a reasonable
> candidate but I don't think we discussed what we would be taking so I
> thought I would ask...
>
> https://issues.apache.org/jira/browse/KAFKA-946
>
> -Jay

criteria for fixes on 0.8 beta

[Jay Kreps]

What should the criteria for fixes on 0.8 be? This seems like a reasonable
candidate but I don't think we discussed what we would be taking so I
thought I would ask...

https://issues.apache.org/jira/browse/KAFKA-946

-Jay

[jira] [Commented] (KAFKA-946) Kafka Hadoop Consumer fails when verifying message checksum

[Jay Kreps (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13698427#comment-13698427
 ] 

Jay Kreps commented on KAFKA-946:
-

Ack, well that's broken.

Richard--is this a reasonable thing to do?
Jun--any concern with this going in 0.8?

A meta question is how to handle the poor state of maintenance of this Hadoop 
code. Started a discussion on that on the mailing list...

> Kafka Hadoop Consumer fails when verifying message checksum
> ---
>
> Key: KAFKA-946
> URL: https://issues.apache.org/jira/browse/KAFKA-946
> Project: Kafka
>  Issue Type: Bug
>  Components: contrib
>Affects Versions: 0.8
>Reporter: Sam Meder
>Priority: Critical
> Attachments: hadoop_consumer.patch
>
>
> The code tries to verify the checksum, but fails because the data available 
> isn't the same. In KafkaETLContext:
> protected boolean get(KafkaETLKey key, BytesWritable value) throws 
> IOException {
>   if (_messageIt != null && _messageIt.hasNext()) {
> MessageAndOffset messageAndOffset = _messageIt.next();
> ByteBuffer buf = messageAndOffset.message().payload();
> int origSize = buf.remaining();
> byte[] bytes = new byte[origSize];
>   buf.get(bytes, buf.position(), origSize);
> value.set(bytes, 0, origSize);
> key.set(_index, _offset, messageAndOffset.message().checksum());
> _offset = messageAndOffset.nextOffset();  //increase offset   
>   
>  
> _count ++;  //increase count  
>   
>  
> return true;
> }
> else return false;
> }
> Note that the message payload is used and the message checksum is included in 
> the key. The in SimpleKafkaETLMapper:
> @Override
> public void map(KafkaETLKey key, BytesWritable val,
> OutputCollector collector,
> Reporter reporter) throws IOException {
>   byte[] bytes = KafkaETLUtils.getBytes(val);
> //check the checksum of message   
>   
>  
> Message message = new Message(bytes);
> long checksum = key.getChecksum();
>   if (checksum != message.checksum())
> throw new IOException ("Invalid message checksum "
> + message.checksum() + ". 
> Expected " + key + ".");
> the Message object is initialized with the payload bytes and a new checksum 
> is calculated. The problem is that the original message checksum also 
> contains the key so checksum verification fails...

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Kafka/Hadoop consumers and producers

[Jay Kreps]

We currently have a contrib package for consuming and producing messages
from mapreduce (
https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tree;f=contrib;h=e53e1fb34893e733b10ff27e79e6a1dcbb8d7ab0;hb=HEAD
).

We keep running into problems (e.g. KAFKA-946) that are basically due to
the fact that the Kafka committers don't seem to mostly be Hadoop
developers and aren't doing a good job of maintaining this code (keeping it
tested, improving it, documenting it, writing tutorials, getting it moved
over to the more modern apis, getting it working with newer Hadoop
versions, etc).

A couple of options:
1. We could try to get someone in the Kafka community (either a current
committer or not) who would adopt this as their baby (it's not much code).
2. We could just let Camus take over this functionality. They already have
a more sophisticated consumer and the producer is pretty minimal.

So are there any people who would like to adopt the current Hadoop contrib
code?

Conversely would it be possible to provide the same or similar
functionality in Camus and just delete these?

-Jay

[jira] [Commented] (KAFKA-943) Move all configuration key string to constants

[Jay Kreps (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13698401#comment-13698401
 ] 

Jay Kreps commented on KAFKA-943:
-

Hey Sam, thanks for the patch it is great to get clean-up patches.

I have sort of a philosophical disagreement, though. I am likely the cause of 
the direct use of string names. This was done intentionally, here is the 
rationale:

Programmers have kind of a knee-jerk belief that using strings directly is bad 
and having variables that point at the string is good. However I don't agree. 
Here is what I think. A global variable is a layer of indirection. By having a 
layer of indirection you get the following trade-off: it is harder to figure 
out which configuration is pointed to by the variable BUT it is easier to 
change the configuration string later and have all uses updated. That is 
normally a good trade-off to make--we don't want string constants floating 
around all over the place. However configuration names are different--these are 
a sort of public api to everyone who uses our system. There are hundreds or 
thousands of people with these strings in their configuration files or code or 
whatever. Adding a layer of indirection doesn't help these people at all. So 
the fact is that configurations are meant to be a public api of sorts and only 
change as part of a major refactoring. So in other words we don't want to make 
it easy. At the same time if we use a variable name instead of the 
configuration name people think less about the name and we end up with bad 
names, which is really irritating.

Our approach to this was to get all the committers to call out and ask for 
review any configuration names in their JIRA and try to get extra review on 
these and then not change them. 

Let me know if you buy that argument. I don't think it is a huge deal either 
way, but I am a little on the side of using the name directly.

> Move all configuration key string to constants
> --
>
> Key: KAFKA-943
> URL: https://issues.apache.org/jira/browse/KAFKA-943
> Project: Kafka
>  Issue Type: Improvement
>  Components: config
>Affects Versions: 0.8
>Reporter: Sam Meder
> Attachments: configConstants.patch
>
>
> The current code base has configuration key strings duplicated all over the 
> place. They show up in the actual *Config classes, a lot of tests, command 
> line utilities and other examples. This makes changes hard and error prone. 
> DRY...
> The attached patch moves these configuration keys to constants and replaces 
> their usage with a reference to the constant. It also cleans up a few old 
> properties and a few misconfigured tests. I've admittedly not written a whole 
> lot of Scala, so there may be some improvements that can be made, in 
> particular I am not sure I chose the best strategy for keys needed by the 
> SyncProducerConfigShared trait (or traits in general).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (KAFKA-559) Garbage collect old consumer metadata entries

[Joel Koshy (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13698307#comment-13698307
 ] 

Joel Koshy commented on KAFKA-559:
--

One additional recommendation: support a --dry-run option.



> Garbage collect old consumer metadata entries
> -
>
> Key: KAFKA-559
> URL: https://issues.apache.org/jira/browse/KAFKA-559
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Jay Kreps
>Assignee: Tejas Patil
>  Labels: project
>
> Many use cases involve tranient consumers. These consumers create entries 
> under their consumer group in zk and maintain offsets there as well. There is 
> currently no way to delete these entries. It would be good to have a tool 
> that did something like
>   bin/delete-obsolete-consumer-groups.sh [--topic t1] --since [date] 
> --zookeeper [zk_connect]
> This would scan through consumer group entries and delete any that had no 
> offset update since the given date.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (KAFKA-559) Garbage collect old consumer metadata entries

[Joel Koshy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Koshy reassigned KAFKA-559:


Assignee: Tejas Patil

Assigning to Tejas, since he has done some work on this recently.



> Garbage collect old consumer metadata entries
> -
>
> Key: KAFKA-559
> URL: https://issues.apache.org/jira/browse/KAFKA-559
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Jay Kreps
>Assignee: Tejas Patil
>  Labels: project
>
> Many use cases involve tranient consumers. These consumers create entries 
> under their consumer group in zk and maintain offsets there as well. There is 
> currently no way to delete these entries. It would be good to have a tool 
> that did something like
>   bin/delete-obsolete-consumer-groups.sh [--topic t1] --since [date] 
> --zookeeper [zk_connect]
> This would scan through consumer group entries and delete any that had no 
> offset update since the given date.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Kafka User Group Meeting Jun. 27

[Jay Kreps]

Erp, actually I don't see a way to get the recording there, nm.

-Jay


On Tue, Jul 2, 2013 at 12:47 PM, Jay Kreps  wrote:

> The recording for the user group talks is available here:
> http://www.ustream.tv/linkedin-events
>
> -Jay
>
>
> On Wed, Jun 26, 2013 at 8:22 AM, Jun Rao  wrote:
>
>> Hi, Everyone,
>>
>> We have finalized our agenda of the meetup Thursday evening, with Speakers
>> from LinkedIn, RichRelevance, Netflix and Square. Please RSVP to the
>> meetup
>> link below. For remote people, we will post the streaming link at meetup
>> before the meeting and we will use our IRC for remote questions.
>>
>>
>> http://www.meetup.com/http-kafka-apache-org/events/125887332/?_af_eid=125887332&a=uc1_vm&_af=event
>>
>> See you Thursday evening.
>>
>> Thanks,
>>
>> Jun
>>
>
>

Re: Kafka User Group Meeting Jun. 27

[Jay Kreps]

The recording for the user group talks is available here:
http://www.ustream.tv/linkedin-events

-Jay


On Wed, Jun 26, 2013 at 8:22 AM, Jun Rao  wrote:

> Hi, Everyone,
>
> We have finalized our agenda of the meetup Thursday evening, with Speakers
> from LinkedIn, RichRelevance, Netflix and Square. Please RSVP to the meetup
> link below. For remote people, we will post the streaming link at meetup
> before the meeting and we will use our IRC for remote questions.
>
>
> http://www.meetup.com/http-kafka-apache-org/events/125887332/?_af_eid=125887332&a=uc1_vm&_af=event
>
> See you Thursday evening.
>
> Thanks,
>
> Jun
>

[jira] [Commented] (KAFKA-883) System Test - update migration tool testsuite after 0.7 ProducerPerformance sends seq MessageID

[John Fung (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13698140#comment-13698140
 ] 

John Fung commented on KAFKA-883:
-

This patch is OK to check in ONLY after the following :
1. Build a new kafka-perf-0.7.0.jar from KAFKA-882
2. Check in the above new kafka-perf-0.7.0.jar into 
system_test/migration_tool_testsuite/0.7/lib to replace the existing one.

> System Test - update migration tool testsuite after 0.7 ProducerPerformance 
> sends seq MessageID
> ---
>
> Key: KAFKA-883
> URL: https://issues.apache.org/jira/browse/KAFKA-883
> Project: Kafka
>  Issue Type: Task
>Reporter: John Fung
>Assignee: John Fung
>  Labels: kafka-0.8, replication-testing
> Attachments: kafka-883-v1.patch
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (KAFKA-883) System Test - update migration tool testsuite after 0.7 ProducerPerformance sends seq MessageID

[John Fung (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Fung updated KAFKA-883:


Status: Patch Available  (was: Open)

> System Test - update migration tool testsuite after 0.7 ProducerPerformance 
> sends seq MessageID
> ---
>
> Key: KAFKA-883
> URL: https://issues.apache.org/jira/browse/KAFKA-883
> Project: Kafka
>  Issue Type: Task
>Reporter: John Fung
>Assignee: John Fung
>  Labels: kafka-0.8, replication-testing
> Attachments: kafka-883-v1.patch
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (KAFKA-882) Enhance 0.7 ProducerPerformance to send sequential MessageID as in 0.8

[John Fung (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13698128#comment-13698128
 ] 

John Fung commented on KAFKA-882:
-

ProducerPerformance in 0.7 has quite a few nested if-else conditional logics in 
addition to while & for loops. In order to keep it simple, this patch contains 
the change to send sequential MessageID only when the following arguments are 
specified:

--initial-message-id 
--vary-message-size
--async

Otherwise, it will preserve the original 0.7 behavior.

> Enhance 0.7 ProducerPerformance to send sequential MessageID as in 0.8
> --
>
> Key: KAFKA-882
> URL: https://issues.apache.org/jira/browse/KAFKA-882
> Project: Kafka
>  Issue Type: Task
>Reporter: John Fung
>Assignee: John Fung
>  Labels: kafka
> Attachments: kafka-882-v1.patch
>
>
> Also update related files in the following in Kafka 0.8:
> 1. system_test/migration_tool_testsuite/config
> 2. system_test/migration_tool_testsuite/0.7/lib/kafka-perf-0.7.0.jar (after 
> this JIRA is completed)
> 3. 
> system_test/migration_tool_testsuite/testcase_900x/testcase_900x_properties.json

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (KAFKA-883) System Test - update migration tool testsuite after 0.7 ProducerPerformance sends seq MessageID

[John Fung (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Fung updated KAFKA-883:


Attachment: kafka-883-v1.patch

uploaded kafka-883-v1.patch

> System Test - update migration tool testsuite after 0.7 ProducerPerformance 
> sends seq MessageID
> ---
>
> Key: KAFKA-883
> URL: https://issues.apache.org/jira/browse/KAFKA-883
> Project: Kafka
>  Issue Type: Task
>Reporter: John Fung
>Assignee: John Fung
>  Labels: kafka-0.8, replication-testing
> Attachments: kafka-883-v1.patch
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (KAFKA-882) Enhance 0.7 ProducerPerformance to send sequential MessageID as in 0.8

[John Fung (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Fung updated KAFKA-882:


Status: Patch Available  (was: Open)

> Enhance 0.7 ProducerPerformance to send sequential MessageID as in 0.8
> --
>
> Key: KAFKA-882
> URL: https://issues.apache.org/jira/browse/KAFKA-882
> Project: Kafka
>  Issue Type: Task
>Reporter: John Fung
>Assignee: John Fung
>  Labels: kafka
> Attachments: kafka-882-v1.patch
>
>
> Also update related files in the following in Kafka 0.8:
> 1. system_test/migration_tool_testsuite/config
> 2. system_test/migration_tool_testsuite/0.7/lib/kafka-perf-0.7.0.jar (after 
> this JIRA is completed)
> 3. 
> system_test/migration_tool_testsuite/testcase_900x/testcase_900x_properties.json

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: site updates

[Jay Kreps]

The introduction and quickstart are all as of 0.8. The design document has
not yet been updated (working on it).

-Jay


On Tue, Jul 2, 2013 at 11:22 AM, S Ahmed  wrote:

> On the main page http://kafka.apache.org/<
> http://kafka.apache.org/design.html>,
> is the introduction/quickstart/design reflecting 0.8?
>
>
> On Mon, Jul 1, 2013 at 3:04 PM, Jay Kreps  wrote:
>
> > 1. I think it is fine to do a one-off since it won't impact the APIs. It
> > would be *awesome* to get this working.
> > 2. Let's sync up since I think we may be both working on the same page.
> >
> > -Jay
> >
> >
> > On Mon, Jul 1, 2013 at 9:50 AM, Sriram Subramanian <
> > srsubraman...@linkedin.com> wrote:
> >
> > > Also,
> > >
> > > 1. I am trying to get the api stuff working but it is little but of
> work.
> > > I need to make Kafka compile with Scala 2.10 first.
> > > 2. I have started a design page for kafka replication. The idea is that
> > it
> > > goes as a separate section under the current design page. I will update
> > > the page today and we can continue editing it. Sounds good?
> > >
> > > On 7/1/13 9:42 AM, "Jay Kreps"  wrote:
> > >
> > > >Yeah thanks for the feedback, that's helpful. Here was my thinking:
> > > >1. I think it just makes sense to have one design and implementation
> > page
> > > >which describe the most recent release and live at the top level. You
> > > >could
> > > >imagine wanting to read older design pages but that seems a bit
> unlikely
> > > >mostly, and it will be really duplicative since the design generally
> > won't
> > > >change a ton, so I think it is just confusing. Currently the design
> and
> > > >implementation page only cover 0.7 but that's just because I haven't
> > > >gotten
> > > >there yet--I hope to get to them in the next week.
> > > >2. Oops that's a typo will fix. I wanted to kind of walk people
> through
> > > >things step by step. I like to do tutorials like that where you just
> > kind
> > > >of cut and paste commands and watch what happens, that was the
> rationale
> > > >for repeating the command.
> > > >3. I guess I felt that although we do document that tool, migration is
> > > >important and a person interested in 0.8 would be more likely to look
> > > >under
> > > >"migration" than tools. I like the idea of having a tools page but
> right
> > > >now it is very incomplete as it doesn't cover most of the tools.
> Anyhow
> > I
> > > >thought migration was important enough to get its own link.
> > > >4. I agree. The old link structure was insane though as all the menus
> > > >disappeared when you clicked on a link and we had cut and pasted all
> the
> > > >shared files into the release dirs. Here was my plan. For now I think
> > 0.7
> > > >is the only stable release and 0.8 is beta so it makes sense to have
> > them
> > > >both though that does take up a lot of space. When we think 0.7 is no
> > > >longer relevant I will make an expandable nav with the title "older
> > > >releases" and shove that in there so when you click "older releases"
> it
> > > >will unhide all the old releases (which at first will just be 0.7).
> That
> > > >way we don't keep taking up space.
> > > >
> > > >I was going to put another day of work into the docs. My plan was to
> > add a
> > > >"use cases" page that covers the basics of tracking, messaging, etc,
> and
> > > >update the design page. If anyone else has ideas for other
> improvements
> > > >let
> > > >me know?
> > > >
> > > >Question: do you have any feedback on the intro page? The goal of that
> > was
> > > >to be something someone who just wants the basics of what Kafka is to
> > > >read.
> > > >It is a bit hard to write something like this because you have to put
> > > >yourself in the shoes of someone totally new to Kafka and potentially
> > new
> > > >to messaging and log aggregation and still explain things coherently.
> > > >Previously the only explanatory thing we had was the design page which
> > was
> > > >extremely detailed so pulling out the essentials hopefully gives a
> kind
> > of
> > > >executive summary.
> > > >
> > > >-Jay
> > > >
> > > >
> > > >
> > > >On Sun, Jun 30, 2013 at 3:32 PM, Jun Rao  wrote:
> > > >
> > > >> Thanks for cleaning up the site. Overall, it looks cleaner. A few
> > > >>comments:
> > > >>
> > > >> 1. implementation: This is mostly about the 0.7 implementation. So
> it
> > > >> probably should be added under 0.7.
> > > >>
> > > >> 2. 0.8 quickstart: Step 3, when the text says list topic, the
> command
> > is
> > > >> actually create topic. Step 4, not sure if we need to show the
> console
> > > >> producer command twice.
> > > >>
> > > >> 3. 0.8 migration: Since we have a separate bullet for migration,
> there
> > > >>is
> > > >> no need to describe the migration tool under the tools bullet.
> > > >>
> > > >> 4. We have the second level bullets for each release expanded in the
> > > >>left
> > > >> panel. This doesn't leave enough room for adding future releases.
> > > >>
> > > >> Jun
> > > >>
> > > >>
> > 

[jira] [Work stopped] (KAFKA-882) Enhance 0.7 ProducerPerformance to send sequential MessageID as in 0.8

[John Fung (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on KAFKA-882 stopped by John Fung.

> Enhance 0.7 ProducerPerformance to send sequential MessageID as in 0.8
> --
>
> Key: KAFKA-882
> URL: https://issues.apache.org/jira/browse/KAFKA-882
> Project: Kafka
>  Issue Type: Task
>Reporter: John Fung
>Assignee: John Fung
>  Labels: kafka
> Attachments: kafka-882-v1.patch
>
>
> Also update related files in the following in Kafka 0.8:
> 1. system_test/migration_tool_testsuite/config
> 2. system_test/migration_tool_testsuite/0.7/lib/kafka-perf-0.7.0.jar (after 
> this JIRA is completed)
> 3. 
> system_test/migration_tool_testsuite/testcase_900x/testcase_900x_properties.json

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (KAFKA-882) Enhance 0.7 ProducerPerformance to send sequential MessageID as in 0.8

[John Fung (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Fung updated KAFKA-882:


Attachment: kafka-882-v1.patch

uploaded kafka-882-v1.patch

> Enhance 0.7 ProducerPerformance to send sequential MessageID as in 0.8
> --
>
> Key: KAFKA-882
> URL: https://issues.apache.org/jira/browse/KAFKA-882
> Project: Kafka
>  Issue Type: Task
>Reporter: John Fung
>Assignee: John Fung
>  Labels: kafka
> Attachments: kafka-882-v1.patch
>
>
> Also update related files in the following in Kafka 0.8:
> 1. system_test/migration_tool_testsuite/config
> 2. system_test/migration_tool_testsuite/0.7/lib/kafka-perf-0.7.0.jar (after 
> this JIRA is completed)
> 3. 
> system_test/migration_tool_testsuite/testcase_900x/testcase_900x_properties.json

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: site updates

[S Ahmed]

On the main page http://kafka.apache.org/,
is the introduction/quickstart/design reflecting 0.8?


On Mon, Jul 1, 2013 at 3:04 PM, Jay Kreps  wrote:

> 1. I think it is fine to do a one-off since it won't impact the APIs. It
> would be *awesome* to get this working.
> 2. Let's sync up since I think we may be both working on the same page.
>
> -Jay
>
>
> On Mon, Jul 1, 2013 at 9:50 AM, Sriram Subramanian <
> srsubraman...@linkedin.com> wrote:
>
> > Also,
> >
> > 1. I am trying to get the api stuff working but it is little but of work.
> > I need to make Kafka compile with Scala 2.10 first.
> > 2. I have started a design page for kafka replication. The idea is that
> it
> > goes as a separate section under the current design page. I will update
> > the page today and we can continue editing it. Sounds good?
> >
> > On 7/1/13 9:42 AM, "Jay Kreps"  wrote:
> >
> > >Yeah thanks for the feedback, that's helpful. Here was my thinking:
> > >1. I think it just makes sense to have one design and implementation
> page
> > >which describe the most recent release and live at the top level. You
> > >could
> > >imagine wanting to read older design pages but that seems a bit unlikely
> > >mostly, and it will be really duplicative since the design generally
> won't
> > >change a ton, so I think it is just confusing. Currently the design and
> > >implementation page only cover 0.7 but that's just because I haven't
> > >gotten
> > >there yet--I hope to get to them in the next week.
> > >2. Oops that's a typo will fix. I wanted to kind of walk people through
> > >things step by step. I like to do tutorials like that where you just
> kind
> > >of cut and paste commands and watch what happens, that was the rationale
> > >for repeating the command.
> > >3. I guess I felt that although we do document that tool, migration is
> > >important and a person interested in 0.8 would be more likely to look
> > >under
> > >"migration" than tools. I like the idea of having a tools page but right
> > >now it is very incomplete as it doesn't cover most of the tools. Anyhow
> I
> > >thought migration was important enough to get its own link.
> > >4. I agree. The old link structure was insane though as all the menus
> > >disappeared when you clicked on a link and we had cut and pasted all the
> > >shared files into the release dirs. Here was my plan. For now I think
> 0.7
> > >is the only stable release and 0.8 is beta so it makes sense to have
> them
> > >both though that does take up a lot of space. When we think 0.7 is no
> > >longer relevant I will make an expandable nav with the title "older
> > >releases" and shove that in there so when you click "older releases" it
> > >will unhide all the old releases (which at first will just be 0.7). That
> > >way we don't keep taking up space.
> > >
> > >I was going to put another day of work into the docs. My plan was to
> add a
> > >"use cases" page that covers the basics of tracking, messaging, etc, and
> > >update the design page. If anyone else has ideas for other improvements
> > >let
> > >me know?
> > >
> > >Question: do you have any feedback on the intro page? The goal of that
> was
> > >to be something someone who just wants the basics of what Kafka is to
> > >read.
> > >It is a bit hard to write something like this because you have to put
> > >yourself in the shoes of someone totally new to Kafka and potentially
> new
> > >to messaging and log aggregation and still explain things coherently.
> > >Previously the only explanatory thing we had was the design page which
> was
> > >extremely detailed so pulling out the essentials hopefully gives a kind
> of
> > >executive summary.
> > >
> > >-Jay
> > >
> > >
> > >
> > >On Sun, Jun 30, 2013 at 3:32 PM, Jun Rao  wrote:
> > >
> > >> Thanks for cleaning up the site. Overall, it looks cleaner. A few
> > >>comments:
> > >>
> > >> 1. implementation: This is mostly about the 0.7 implementation. So it
> > >> probably should be added under 0.7.
> > >>
> > >> 2. 0.8 quickstart: Step 3, when the text says list topic, the command
> is
> > >> actually create topic. Step 4, not sure if we need to show the console
> > >> producer command twice.
> > >>
> > >> 3. 0.8 migration: Since we have a separate bullet for migration, there
> > >>is
> > >> no need to describe the migration tool under the tools bullet.
> > >>
> > >> 4. We have the second level bullets for each release expanded in the
> > >>left
> > >> panel. This doesn't leave enough room for adding future releases.
> > >>
> > >> Jun
> > >>
> > >>
> > >>
> > >> On Sat, Jun 29, 2013 at 3:09 PM, Jay Kreps 
> wrote:
> > >>
> > >> > Ack, nice catch--that migration tool thing was due to bad html, I
> > >> > forgot to close the link.
> > >> >
> > >> > For the configuration I actually think that is not right. We need to
> > >> > thoroughly document our configuration. Having the code/scaladoc be
> the
> > >> > documentation is fine for kafka developers but not where we want to
> >

client rewrite in 0.9?

[Jay Kreps]

Hey guys,

We have had a bunch of prior discussions and some prototyping on
generalizing and improving the jvm clients.

I have tried to consolidate all these thoughts into one larger plan:
https://cwiki.apache.org/confluence/display/KAFKA/Client+Rewrite

It's not nearly baked right now, but I thought I would throw this out there
for discussion.

-Jay

[jira] [Work started] (KAFKA-882) Enhance 0.7 ProducerPerformance to send sequential MessageID as in 0.8

[John Fung (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on KAFKA-882 started by John Fung.

> Enhance 0.7 ProducerPerformance to send sequential MessageID as in 0.8
> --
>
> Key: KAFKA-882
> URL: https://issues.apache.org/jira/browse/KAFKA-882
> Project: Kafka
>  Issue Type: Task
>Reporter: John Fung
>Assignee: John Fung
>  Labels: kafka
>
> Also update related files in the following in Kafka 0.8:
> 1. system_test/migration_tool_testsuite/config
> 2. system_test/migration_tool_testsuite/0.7/lib/kafka-perf-0.7.0.jar (after 
> this JIRA is completed)
> 3. 
> system_test/migration_tool_testsuite/testcase_900x/testcase_900x_properties.json

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira