Make documentation part of new features acceptance criteria?
I'm not sure if there's already a guideline like this, but I wouldn't it make sense to have it in order to keep documentation in sync with the code? Also, having this type of documentation as part of the codebase to allow proper versioning might be a good idea as well. Cosmin
Re: Make documentation part of new features acceptance criteria?
Cosmin, That's a good idea. In the past, for major new features, we tend to create a wiki page to outline the design. The wiki pages can be organized better. Is this what you are looking for? Thanks, Jun On Wed, Jul 10, 2013 at 1:17 AM, Cosmin Lehene cleh...@adobe.com wrote: I'm not sure if there's already a guideline like this, but I wouldn't it make sense to have it in order to keep documentation in sync with the code? Also, having this type of documentation as part of the codebase to allow proper versioning might be a good idea as well. Cosmin
Re: Make documentation part of new features acceptance criteria?
I like the idea of improving our documentation. Help is very much appreciated in this area (but of course the problem is that the people who experience the holes almost by definition can't fill them in). So even just pointing out areas that aren't covered is really helpful. We are in a sort of awkward stage this week because we have a 0.8 beta release but no detailed docs on its internals. WRT your specific proposals. I don't think we should do the documentation with each feature because I think that tends to lead to a bunch of little documents one for each change. I think we effectively get this out of JIRA+wiki today. This usually serves as a fairly complete design doc + commentary be others. It is pretty hard to get information out of this format for a new user, though. We do version control documentation but we can't physically version control it with the code because the code is in git and Apache only allows SVN as a mechanism for publishing to xxx.apache.org. :-( Instead what about this: we add a new release criteria for documentation completeness. It would be good to formalize the release criteria anyway. Informally they are something like 1. Developers think it is feature complete 2. Unit tests pass 3. Integration/stress tests pass 4. Some production usage It would be good to add to this list (5) documentation up-to-date and not do a release without this. It is debatable whether this should apply to beta releases, but probably it should. We can certainly apply it to the final 0.8 release if people are on board. -Jay On Wed, Jul 10, 2013 at 1:17 AM, Cosmin Lehene cleh...@adobe.com wrote: I'm not sure if there's already a guideline like this, but I wouldn't it make sense to have it in order to keep documentation in sync with the code? Also, having this type of documentation as part of the codebase to allow proper versioning might be a good idea as well. Cosmin
Re: having problem with 0.8 gzip compression
Jun,
I did a test this morning and got a very interesting result with you
command. I started by wipe all the log files and clean up all zookeeper
data files.
Once I restarted both server, producer and consumer then execute your
command, what I got is a empty log as following:
Dumping /Users/scott/Temp/kafka/test-topic-0/.log
Starting offset: 0
One observation, the .index file was getting huge but
there was nothing in .log file.
Thanks,
Scott
On Tue, Jul 9, 2013 at 8:40 PM, Jun Rao jun...@gmail.com wrote:
Could you run the following command on one of the log files of your topic
and attach the output?
bin/kafka-run-class.sh kafka.tools.DumpLogSegments --files
/tmp/kafka-logs/testtopic-0/.log
Thanks,
Jun
On Tue, Jul 9, 2013 at 3:23 PM, Scott Wang
scott.w...@rumbleentertainment.com wrote:
Another piece of information, the snappy compression also does not work.
Thanks,
Scott
On Tue, Jul 9, 2013 at 11:07 AM, Scott Wang
scott.w...@rumbleentertainment.com wrote:
I just try it and it still not showing up, thanks for looking into
this.
Thanks,
Scott
On Tue, Jul 9, 2013 at 8:06 AM, Jun Rao jun...@gmail.com wrote:
Could you try starting the consumer first (and enable gzip in the
producer)?
Thanks,
Jun
On Mon, Jul 8, 2013 at 9:37 PM, Scott Wang
scott.w...@rumbleentertainment.com wrote:
No, I did not start the consumer before the producer. I actually
started
the producer first and nothing showed up in the consumer unless I
commented
out this line -- props.put(compression.codec, gzip).If I
commented
out the compression codec, everything just works.
On Mon, Jul 8, 2013 at 9:07 PM, Jun Rao jun...@gmail.com wrote:
Did you start the consumer before the producer? Be default, the
consumer
gets only the new data?
Thanks,
Jun
On Mon, Jul 8, 2013 at 2:53 PM, Scott Wang
scott.w...@rumbleentertainment.com wrote:
I am testing with Kafka 0.8 beta and having problem of receiving
message
in
consumer. There is no error so does anyone have any insights.
When I
commented out the compression.code everything works fine.
My producer:
public class TestKafka08Prod {
public static void main(String [] args) {
ProducerInteger, String producer = null;
try {
Properties props = new Properties();
props.put(metadata.broker.list, localhost:9092);
props.put(serializer.class,
kafka.serializer.StringEncoder);
props.put(producer.type, sync);
props.put(request.required.acks,1);
props.put(compression.codec, gzip);
ProducerConfig config = new ProducerConfig(props);
producer = new ProducerInteger, String(config);
int j=0;
for(int i=0; i10; i++) {
KeyedMessageInteger, String data = new
KeyedMessageInteger, String(test-topic, test-message: +i+
+System.currentTimeMillis());
producer.send(data);
}
} catch (Exception e) {
System.out.println(Error happened: );
e.printStackTrace();
} finally {
if(null != null) {
producer.close();
}
System.out.println(Ened of Sending);
}
System.exit(0);
}
}
My consumer:
public class TestKafka08Consumer {
public static void main(String [] args) throws
UnknownHostException,
SocketException {
Properties props = new Properties();
props.put(zookeeper.connect,
localhost:2181/kafka_0_8);
props.put(group.id, test08ConsumerId);
props.put(zk.sessiontimeout.ms, 4000);
props.put(zk.synctime.ms, 2000);
props.put(autocommit.interval.ms, 1000);
ConsumerConfig consumerConfig = new
ConsumerConfig(props);
ConsumerConnector consumerConnector =
kafka.consumer.Consumer.createJavaConsumerConnector(consumerConfig);
String topic = test-topic;
MapString, Integer topicCountMap = new HashMapString,
Integer();
topicCountMap.put(topic, new Integer(1));
MapString, ListKafkaStreambyte[], byte[]
consumerMap =
consumerConnector.createMessageStreams(topicCountMap);
KafkaStreambyte[], byte[] stream =
consumerMap.get(topic).get(0);
ConsumerIteratorbyte[], byte[] it = stream.iterator();
int counter=0;
[jira] [Created] (KAFKA-968) Typographical Errors in Output
Rebecca Sealfon created KAFKA-968: - Summary: Typographical Errors in Output Key: KAFKA-968 URL: https://issues.apache.org/jira/browse/KAFKA-968 Project: Kafka Issue Type: Bug Components: core, replication Affects Versions: 0.8 Environment: Kafka was run on GNU/Linux x86_64 but this is relevant to all environments Reporter: Rebecca Sealfon Assignee: Neha Narkhede Priority: Trivial Fix For: 0.8 The word partition is referred to as partion in system_test/replication_testsuite/testcase_0106/testcase_0106_properties.json line 2 and core/src/main/scala/kafka/server/AbstractFetcherManager.scala line 49. This typo may interfere with text-based searching of output. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-959) DefaultEventHandler can send more produce requests than necesary
[ https://issues.apache.org/jira/browse/KAFKA-959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-959: Assignee: Guozhang Wang DefaultEventHandler can send more produce requests than necesary Key: KAFKA-959 URL: https://issues.apache.org/jira/browse/KAFKA-959 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8 Reporter: Jun Rao Assignee: Guozhang Wang In DefaultEventHandler, for a batch of messages, it picks a random partition per message (when there is no key specified). This means that it can send up to P produce requests where P is the number of partitions in a topic. A better way is probably to pick a single random partition for the whole batch of messages. This will reduce the number of produce requests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (KAFKA-969) Need to prevent failure of rebalance when there are no brokers available when consumer comes up
Sriram Subramanian created KAFKA-969: Summary: Need to prevent failure of rebalance when there are no brokers available when consumer comes up Key: KAFKA-969 URL: https://issues.apache.org/jira/browse/KAFKA-969 Project: Kafka Issue Type: Bug Reporter: Sriram Subramanian Assignee: Sriram Subramanian Attachments: emptybrokeronrebalance-1.patch There are some rare instances when a consumer would be up before bringing up the Kafka brokers. This would usually happen in a test scenario. In such conditions, during rebalance instead of failing the rebalance we just log the error and subscribe to broker changes. When the broker comes back up, we trigger the rebalance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-969) Need to prevent failure of rebalance when there are no brokers available when consumer comes up
[ https://issues.apache.org/jira/browse/KAFKA-969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sriram Subramanian updated KAFKA-969: - Attachment: emptybrokeronrebalance-1.patch Need to prevent failure of rebalance when there are no brokers available when consumer comes up --- Key: KAFKA-969 URL: https://issues.apache.org/jira/browse/KAFKA-969 Project: Kafka Issue Type: Bug Reporter: Sriram Subramanian Assignee: Sriram Subramanian Attachments: emptybrokeronrebalance-1.patch There are some rare instances when a consumer would be up before bringing up the Kafka brokers. This would usually happen in a test scenario. In such conditions, during rebalance instead of failing the rebalance we just log the error and subscribe to broker changes. When the broker comes back up, we trigger the rebalance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-969) Need to prevent failure of rebalance when there are no brokers available when consumer comes up
[ https://issues.apache.org/jira/browse/KAFKA-969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sriram Subramanian updated KAFKA-969: - Status: Patch Available (was: Open) Need to prevent failure of rebalance when there are no brokers available when consumer comes up --- Key: KAFKA-969 URL: https://issues.apache.org/jira/browse/KAFKA-969 Project: Kafka Issue Type: Bug Reporter: Sriram Subramanian Assignee: Sriram Subramanian Attachments: emptybrokeronrebalance-1.patch There are some rare instances when a consumer would be up before bringing up the Kafka brokers. This would usually happen in a test scenario. In such conditions, during rebalance instead of failing the rebalance we just log the error and subscribe to broker changes. When the broker comes back up, we trigger the rebalance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
formatting if statements
Guys,
I am seeing this:
if (condition) {
// something
}
else {
// something else
}
This is not the style we are using. Please don't do this or accept code
that looks like this. It should be
if (condition) {
// something
} else {
//something else
}
Thanks!
-Jay
Re: having problem with 0.8 gzip compression
Weird - I tried your exact code and it worked for me (although I was
using 0.8 head and not the beta). Can you re-run with trace logs
enabled in your producer and paste that output? Broker logs also if
you can?
Thanks,
Joel
On Wed, Jul 10, 2013 at 10:23 AM, Scott Wang
scott.w...@rumbleentertainment.com wrote:
Jun,
I did a test this morning and got a very interesting result with you
command. I started by wipe all the log files and clean up all zookeeper
data files.
Once I restarted both server, producer and consumer then execute your
command, what I got is a empty log as following:
Dumping /Users/scott/Temp/kafka/test-topic-0/.log
Starting offset: 0
One observation, the .index file was getting huge but
there was nothing in .log file.
Thanks,
Scott
On Tue, Jul 9, 2013 at 8:40 PM, Jun Rao jun...@gmail.com wrote:
Could you run the following command on one of the log files of your topic
and attach the output?
bin/kafka-run-class.sh kafka.tools.DumpLogSegments --files
/tmp/kafka-logs/testtopic-0/.log
Thanks,
Jun
On Tue, Jul 9, 2013 at 3:23 PM, Scott Wang
scott.w...@rumbleentertainment.com wrote:
Another piece of information, the snappy compression also does not work.
Thanks,
Scott
On Tue, Jul 9, 2013 at 11:07 AM, Scott Wang
scott.w...@rumbleentertainment.com wrote:
I just try it and it still not showing up, thanks for looking into
this.
Thanks,
Scott
On Tue, Jul 9, 2013 at 8:06 AM, Jun Rao jun...@gmail.com wrote:
Could you try starting the consumer first (and enable gzip in the
producer)?
Thanks,
Jun
On Mon, Jul 8, 2013 at 9:37 PM, Scott Wang
scott.w...@rumbleentertainment.com wrote:
No, I did not start the consumer before the producer. I actually
started
the producer first and nothing showed up in the consumer unless I
commented
out this line -- props.put(compression.codec, gzip).If I
commented
out the compression codec, everything just works.
On Mon, Jul 8, 2013 at 9:07 PM, Jun Rao jun...@gmail.com wrote:
Did you start the consumer before the producer? Be default, the
consumer
gets only the new data?
Thanks,
Jun
On Mon, Jul 8, 2013 at 2:53 PM, Scott Wang
scott.w...@rumbleentertainment.com wrote:
I am testing with Kafka 0.8 beta and having problem of receiving
message
in
consumer. There is no error so does anyone have any insights.
When I
commented out the compression.code everything works fine.
My producer:
public class TestKafka08Prod {
public static void main(String [] args) {
ProducerInteger, String producer = null;
try {
Properties props = new Properties();
props.put(metadata.broker.list, localhost:9092);
props.put(serializer.class,
kafka.serializer.StringEncoder);
props.put(producer.type, sync);
props.put(request.required.acks,1);
props.put(compression.codec, gzip);
ProducerConfig config = new ProducerConfig(props);
producer = new ProducerInteger, String(config);
int j=0;
for(int i=0; i10; i++) {
KeyedMessageInteger, String data = new
KeyedMessageInteger, String(test-topic, test-message: +i+
+System.currentTimeMillis());
producer.send(data);
}
} catch (Exception e) {
System.out.println(Error happened: );
e.printStackTrace();
} finally {
if(null != null) {
producer.close();
}
System.out.println(Ened of Sending);
}
System.exit(0);
}
}
My consumer:
public class TestKafka08Consumer {
public static void main(String [] args) throws
UnknownHostException,
SocketException {
Properties props = new Properties();
props.put(zookeeper.connect,
localhost:2181/kafka_0_8);
props.put(group.id, test08ConsumerId);
props.put(zk.sessiontimeout.ms, 4000);
props.put(zk.synctime.ms, 2000);
props.put(autocommit.interval.ms, 1000);
ConsumerConfig consumerConfig = new
ConsumerConfig(props);
ConsumerConnector consumerConnector =
kafka.consumer.Consumer.createJavaConsumerConnector(consumerConfig);
String topic = test-topic;
MapString, Integer topicCountMap = new HashMapString,
Integer();
topicCountMap.put(topic, new Integer(1));
[jira] [Commented] (KAFKA-969) Need to prevent failure of rebalance when there are no brokers available when consumer comes up
[ https://issues.apache.org/jira/browse/KAFKA-969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13705340#comment-13705340 ] Joel Koshy commented on KAFKA-969: -- This seems reasonable, but I'm not fully convinced about it. E.g., a test framework should ensure external dependencies are up before attempting to make service calls to those dependencies. That said, it is perhaps also reasonable from a consumer's perspective to expect that returned streams be empty at first, and whenever brokers and topics show up, then events should just show up. I'm +1 on this patch except for the if-else formatting issue. Also, I think this patch alone would be insufficient to meet the above. There are two other issues: - We should register a watcher under the topics path (currently done only if a wildcard is specified) - KAFKA-956 is also related. I need to give that one some thought. Need to prevent failure of rebalance when there are no brokers available when consumer comes up --- Key: KAFKA-969 URL: https://issues.apache.org/jira/browse/KAFKA-969 Project: Kafka Issue Type: Bug Reporter: Sriram Subramanian Assignee: Sriram Subramanian Attachments: emptybrokeronrebalance-1.patch There are some rare instances when a consumer would be up before bringing up the Kafka brokers. This would usually happen in a test scenario. In such conditions, during rebalance instead of failing the rebalance we just log the error and subscribe to broker changes. When the broker comes back up, we trigger the rebalance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-969) Need to prevent failure of rebalance when there are no brokers available when consumer comes up
[ https://issues.apache.org/jira/browse/KAFKA-969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13705461#comment-13705461 ] Sriram Subramanian commented on KAFKA-969: -- The other issues you mention are separate from this. You should file JIRAs for those. Need to prevent failure of rebalance when there are no brokers available when consumer comes up --- Key: KAFKA-969 URL: https://issues.apache.org/jira/browse/KAFKA-969 Project: Kafka Issue Type: Bug Reporter: Sriram Subramanian Assignee: Sriram Subramanian Attachments: emptybrokeronrebalance-1.patch There are some rare instances when a consumer would be up before bringing up the Kafka brokers. This would usually happen in a test scenario. In such conditions, during rebalance instead of failing the rebalance we just log the error and subscribe to broker changes. When the broker comes back up, we trigger the rebalance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-969) Need to prevent failure of rebalance when there are no brokers available when consumer comes up
[ https://issues.apache.org/jira/browse/KAFKA-969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13705484#comment-13705484 ] Guozhang Wang commented on KAFKA-969: - I think KAFKA-956 is orthogonal to this JIRA. It is about broker metadata not able to propagate to the brokers yet. Need to prevent failure of rebalance when there are no brokers available when consumer comes up --- Key: KAFKA-969 URL: https://issues.apache.org/jira/browse/KAFKA-969 Project: Kafka Issue Type: Bug Reporter: Sriram Subramanian Assignee: Sriram Subramanian Attachments: emptybrokeronrebalance-1.patch There are some rare instances when a consumer would be up before bringing up the Kafka brokers. This would usually happen in a test scenario. In such conditions, during rebalance instead of failing the rebalance we just log the error and subscribe to broker changes. When the broker comes back up, we trigger the rebalance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-969) Need to prevent failure of rebalance when there are no brokers available when consumer comes up
[ https://issues.apache.org/jira/browse/KAFKA-969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13705490#comment-13705490 ] Joel Koshy commented on KAFKA-969: -- As I already said, I'm +1 on this patch for what it intends to address. Those two issues I mentioned are orthogonal. By above in my comment I was referring to the possible expectation from consumers: .. from a consumer's perspective to expect that returned streams be empty at first, and whenever brokers and topics show up, then events should just show up. - not the failed to rebalance issue. Need to prevent failure of rebalance when there are no brokers available when consumer comes up --- Key: KAFKA-969 URL: https://issues.apache.org/jira/browse/KAFKA-969 Project: Kafka Issue Type: Bug Reporter: Sriram Subramanian Assignee: Sriram Subramanian Fix For: 0.8 Attachments: emptybrokeronrebalance-1.patch There are some rare instances when a consumer would be up before bringing up the Kafka brokers. This would usually happen in a test scenario. In such conditions, during rebalance instead of failing the rebalance we just log the error and subscribe to broker changes. When the broker comes back up, we trigger the rebalance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
