[GitHub] kafka pull request: MINOR: Add property to configure showing of st...

GitHub user ijuma opened a pull request: https://github.com/apache/kafka/pull/739 MINOR: Add property to configure showing of standard streams in Gradle This is handy when debugging certain kinds of Jenkins failures. You can merge this pull request into a Git repository by running:

[jira] [Work started] (KAFKA-3063) Transient exist(1) in unknown test cases

[ https://issues.apache.org/jira/browse/KAFKA-3063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-3063 started by Ismael Juma. -- > Transient exist(1) in unknown test cases >

[jira] [Updated] (KAFKA-3063) LogRecoveryTest exits with -1 occasionally

[ https://issues.apache.org/jira/browse/KAFKA-3063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-3063: --- Summary: LogRecoveryTest exits with -1 occasionally (was: Transient exist(1) in unknown test cases)

[jira] [Commented] (KAFKA-3068) NetworkClient may connect to a different Kafka cluster than originally configured

[ https://issues.apache.org/jira/browse/KAFKA-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15087309#comment-15087309 ] Eno Thereska commented on KAFKA-3068: - [~junrao], [~hachikuji] I understand the concer

[jira] [Commented] (KAFKA-2988) Change default configuration of the log cleaner

[ https://issues.apache.org/jira/browse/KAFKA-2988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1508#comment-1508 ] ASF GitHub Bot commented on KAFKA-2988: --- Github user asfgit closed the pull request

[GitHub] kafka pull request: KAFKA-2988: Change default configuration of th...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/686 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabl

Jenkins build is back to normal : kafka-trunk-jdk7 #943

See

Connecting with new consumers and existing group appears to cause existing group to rebalance

Hi, I'm testing out some changes with the 0.9.0.0 new KafkaConsumer API. We re-use the same consumer group ID across different components of the application (which consume from different topics). One topic is always being consumed from, the rest are turned on and off. If I only run the always-o

Re: Connecting with new consumers and existing group appears to cause existing group to rebalance

Hey Cliff, Are you using the 0.9.0.0 release? We've fixed a few problems in the 0.9.0 branch, some of which might explain the behavior you're seeing. There was one bug in particular which resulted in the consumer not fetching data for a set of partitions after a rebalance. -Jason On Thu, Jan 7,

Re: Connecting with new consumers and existing group appears to cause existing group to rebalance

Hi Jason, I'm just on the 0.9.0.0 release. Are the fixes in the client, the kafka service, or both? I'll give it a try. Is there a timeline for when 0.9.0.1 would be released? Thanks, Cliff On Thu, Jan 7, 2016 at 3:14 PM, Jason Gustafson wrote: > Hey Cliff, > > Are you using the 0.9.0.0 rel

[jira] [Created] (KAFKA-3076) BrokerChangeListener should log the brokers in order

Jun Rao created KAFKA-3076: -- Summary: BrokerChangeListener should log the brokers in order Key: KAFKA-3076 URL: https://issues.apache.org/jira/browse/KAFKA-3076 Project: Kafka Issue Type: Improvemen

[jira] [Assigned] (KAFKA-3068) NetworkClient may connect to a different Kafka cluster than originally configured

[ https://issues.apache.org/jira/browse/KAFKA-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eno Thereska reassigned KAFKA-3068: --- Assignee: Eno Thereska > NetworkClient may connect to a different Kafka cluster than original

Re: Connecting with new consumers and existing group appears to cause existing group to rebalance

There have been bugs affecting both the client and the server. The one I mentioned above only affected the client, so you could try updating it alone if that's easier, but it would be better to do both. I'll leave it to others to comment on the release timeline. I haven't seen any major consumer-r

[jira] [Commented] (KAFKA-3068) NetworkClient may connect to a different Kafka cluster than originally configured

[ https://issues.apache.org/jira/browse/KAFKA-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088145#comment-15088145 ] Ewen Cheslack-Postava commented on KAFKA-3068: -- You can get into bad states w

Re: Connecting with new consumers and existing group appears to cause existing group to rebalance

I'll give the 0.9.0 trunk a try. By the way, it looks to me that this might be a separate issue. I just setup unique group IDs for the stop/start topics from the always-on topic and my test passed. I think the issue is that the GroupCoordinator.doJoinGroup() only tracks the group and not the top

Re: Connecting with new consumers and existing group appears to cause existing group to rebalance

> > If a new consumer joins an existing group, it triggers a rebalance even > if it's > consuming from a new topic (which matches my symptoms). > Not sure I understand the issue. Rebalances are triggered when either 1) group membership changes, 2) a consumer's subscription changes, or 3) when the

Re: TMR should nopt create topic if not existing.

+ dev Hi There has been discussion on the ticket : https://issues.apache.org/jira/browse/KAFKA-2887, that we are going to deprecate auto-creation of topic or at least turn it off by default once we have the CreateTopics API. It also says the patch is available for this. The only other ticket tha

Re: Connecting with new consumers and existing group appears to cause existing group to rebalance

I'll explain. Say there are two topics, foo and bar. Foo has two partitions (foo-0 and foo-1). Bar has one partition (bar-0). The application uses one consumer group for all KafkaConsumers called "app". The application has two consumers always consuming from the two foo partitions, but creates

[jira] [Commented] (KAFKA-3068) NetworkClient may connect to a different Kafka cluster than originally configured

[ https://issues.apache.org/jira/browse/KAFKA-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088200#comment-15088200 ] Ewen Cheslack-Postava commented on KAFKA-3068: -- [~junrao] It's definitely a r

[jira] [Commented] (KAFKA-3068) NetworkClient may connect to a different Kafka cluster than originally configured

[ https://issues.apache.org/jira/browse/KAFKA-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088209#comment-15088209 ] Jason Gustafson commented on KAFKA-3068: [~ewencp] Is it not correct that the topi

Re: Connecting with new consumers and existing group appears to cause existing group to rebalance

Thanks for the explanation. Unfortunately, the consumer doesn't work that way. Rebalances affect all members of the group regardless of subscriptions. Partial rebalances could be an interesting idea to consider for the future, but we haven't had any cases where a group had differing subscriptions i

[jira] [Commented] (KAFKA-3068) NetworkClient may connect to a different Kafka cluster than originally configured

[ https://issues.apache.org/jira/browse/KAFKA-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088222#comment-15088222 ] Ewen Cheslack-Postava commented on KAFKA-3068: -- [~hachikuji] Sorry, yeah, tha

[jira] [Updated] (KAFKA-2856) add KTable

[ https://issues.apache.org/jira/browse/KAFKA-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-2856: - Assignee: Yasuhiro Matsuda > add KTable > -- > > Key: KAFKA-2856 >

Re: Connecting with new consumers and existing group appears to cause existing group to rebalance

Using separate groups would be only a slight inconvenience but is entirely doable. It sounds like that's the preferred model. Thanks for the explanation. On Thu, Jan 7, 2016 at 4:08 PM, Jason Gustafson wrote: > Thanks for the explanation. Unfortunately, the consumer doesn't work that > way. Re

[jira] [Created] (KAFKA-3077) Enable KafkaLog4jAppender to work with SASL enabled brokers.

Ashish K Singh created KAFKA-3077: - Summary: Enable KafkaLog4jAppender to work with SASL enabled brokers. Key: KAFKA-3077 URL: https://issues.apache.org/jira/browse/KAFKA-3077 Project: Kafka

[jira] [Created] (KAFKA-3078) Add ducktape tests for KafkaLog4jAppender producing to SASL enabled Kafka cluster

Ashish K Singh created KAFKA-3078: - Summary: Add ducktape tests for KafkaLog4jAppender producing to SASL enabled Kafka cluster Key: KAFKA-3078 URL: https://issues.apache.org/jira/browse/KAFKA-3078 Pro

[jira] [Commented] (KAFKA-3077) Enable KafkaLog4jAppender to work with SASL enabled brokers.

[ https://issues.apache.org/jira/browse/KAFKA-3077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088284#comment-15088284 ] ASF GitHub Bot commented on KAFKA-3077: --- GitHub user SinghAsDev opened a pull reques

[GitHub] kafka pull request: KAFKA-3077: Enable KafkaLog4jAppender to work ...

GitHub user SinghAsDev opened a pull request: https://github.com/apache/kafka/pull/740 KAFKA-3077: Enable KafkaLog4jAppender to work with SASL enabled brokers You can merge this pull request into a Git repository by running: $ git pull https://github.com/SinghAsDev/kafka KAFKA

Re: KIP-41: KafkaConsumer Max Records

Thanks Jason. I think it is a good feature to add, +1. As suggested in KIP-32, we'd better to keep end state of the KIP wiki with finalized implementation details rather than leaving a list of options. I agree that for both fairness and pre-fetching the simpler approach would be sufficient for mos

[jira] [Commented] (KAFKA-2649) Add support for custom partitioner in sink nodes

[ https://issues.apache.org/jira/browse/KAFKA-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088312#comment-15088312 ] ASF GitHub Bot commented on KAFKA-2649: --- Github user asfgit closed the pull request

[GitHub] kafka pull request: KAFKA-2649 Add support for custom partitioning...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/309 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabl

[jira] [Updated] (KAFKA-2649) Add support for custom partitioner in sink nodes

[ https://issues.apache.org/jira/browse/KAFKA-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-2649: - Resolution: Fixed Status: Resolved (was: Patch Available) Issue resolved by pull request

Re: TMR should nopt create topic if not existing.

Hey Mayuresh, The ticket that Grant Henke has been working on is here: https://issues.apache.org/jira/browse/KAFKA-2945. Is that what you were looking for? -Jason On Thu, Jan 7, 2016 at 1:50 PM, Mayuresh Gharat wrote: > + dev > > Hi > > There has been discussion on the ticket : > https://issue

Build failed in Jenkins: kafka-trunk-jdk8 #274

See Changes: [wangguoz] KAFKA-2649: Add support for custom partitioning in topology sinks -- Started by an SCM change [EnvInject] - Loading node environment variables. Building remotely on H11 (U

[GitHub] kafka pull request: KAFKA-3021: Centralize dependency version mana...

GitHub user granthenke opened a pull request: https://github.com/apache/kafka/pull/741 KAFKA-3021: Centralize dependency version management You can merge this pull request into a Git repository by running: $ git pull https://github.com/granthenke/kafka central-deps Alternativ

[jira] [Commented] (KAFKA-3021) Centralize dependency version managment

[ https://issues.apache.org/jira/browse/KAFKA-3021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088367#comment-15088367 ] ASF GitHub Bot commented on KAFKA-3021: --- GitHub user granthenke opened a pull reques

[jira] [Updated] (KAFKA-3021) Centralize dependency version managment

[ https://issues.apache.org/jira/browse/KAFKA-3021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated KAFKA-3021: --- Status: Patch Available (was: Open) > Centralize dependency version managment > -

Re: TMR should nopt create topic if not existing.

Hi Jason, Thanks. I had found this ticket before but the KAFKA-1507 also has some context about this and I was confused basically exactly which patch is going to go in. Thanks a lot for confirming. Thanks, Mayuresh On Thu, Jan 7, 2016 at 3:08 PM, Jason Gustafson wrote: > Hey Mayuresh, > > The

[jira] [Assigned] (KAFKA-2886) WorkerSinkTask doesn't catch exceptions from rebalance callbacks

[ https://issues.apache.org/jira/browse/KAFKA-2886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson reassigned KAFKA-2886: -- Assignee: Jason Gustafson (was: Ewen Cheslack-Postava) > WorkerSinkTask doesn't catch

[jira] [Resolved] (KAFKA-2655) Consumer.poll(0)'s overhead too large

[ https://issues.apache.org/jira/browse/KAFKA-2655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson resolved KAFKA-2655. Resolution: Not A Problem Closing this issue as not a problem (any longer). I think the caus

[jira] [Assigned] (KAFKA-3073) KafkaConnect should support regular expression for topics

[ https://issues.apache.org/jira/browse/KAFKA-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei reassigned KAFKA-3073: - Assignee: Liquan Pei > KafkaConnect should support regular expression for topics > --

[jira] [Created] (KAFKA-3079) org.apache.kafka.common.KafkaException: java.lang.SecurityException: Configuration Error:

Mohit Anchlia created KAFKA-3079: Summary: org.apache.kafka.common.KafkaException: java.lang.SecurityException: Configuration Error: Key: KAFKA-3079 URL: https://issues.apache.org/jira/browse/KAFKA-3079

[jira] [Created] (KAFKA-3080) ConsoleConsumerTest.test_version system test fails consistently

Ewen Cheslack-Postava created KAFKA-3080: Summary: ConsoleConsumerTest.test_version system test fails consistently Key: KAFKA-3080 URL: https://issues.apache.org/jira/browse/KAFKA-3080 Project

Re: TMR should nopt create topic if not existing.

Hi, I might have missed out the discussions on this :( I had some more questions : 1) We need a config in KafkaProducer for this. So when the KafkaProducer issues a TMR for a topic and receives a response that the topic does not exist, depending on the value of this config it should use the Crea

[jira] [Commented] (KAFKA-3079) org.apache.kafka.common.KafkaException: java.lang.SecurityException: Configuration Error:

[ https://issues.apache.org/jira/browse/KAFKA-3079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088504#comment-15088504 ] Ismael Juma commented on KAFKA-3079: C > org.apache.kafka.common.KafkaException: java

[jira] [Comment Edited] (KAFKA-3079) org.apache.kafka.common.KafkaException: java.lang.SecurityException: Configuration Error:

[ https://issues.apache.org/jira/browse/KAFKA-3079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088504#comment-15088504 ] Ismael Juma edited comment on KAFKA-3079 at 1/8/16 12:52 AM: -

Re: KIP-41: KafkaConsumer Max Records

Hi Json, I am concerned about how many records can be prefetched into consumer memory. Currently we control the maximum number of bytes per topic and partition by setting fetch.message.max.bytes The max.partition.fetch.bytes = #no of partitions * fetch.message.max.bytes However, partitions can be

[GitHub] kafka pull request: KAFKA-2653: Alternative Kafka Streams Stateful...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/730 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabl

[jira] [Resolved] (KAFKA-2653) Stateful operations in the KStream DSL layer

[ https://issues.apache.org/jira/browse/KAFKA-2653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-2653. -- Resolution: Fixed Fix Version/s: 0.9.1.0 Issue resolved by pull request 730 [https://gith

[jira] [Commented] (KAFKA-2653) Stateful operations in the KStream DSL layer

[ https://issues.apache.org/jira/browse/KAFKA-2653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088533#comment-15088533 ] ASF GitHub Bot commented on KAFKA-2653: --- Github user asfgit closed the pull request

[jira] [Created] (KAFKA-3081) KTable Aggregation Implementation

Guozhang Wang created KAFKA-3081: Summary: KTable Aggregation Implementation Key: KAFKA-3081 URL: https://issues.apache.org/jira/browse/KAFKA-3081 Project: Kafka Issue Type: Sub-task

Re: KIP-41: KafkaConsumer Max Records

@Jason, (apologies, got your name wrong the first time round) On Thu, Jan 7, 2016 at 5:15 PM, Aarti Gupta wrote: > Hi Json, > > I am concerned about how many records can be prefetched into consumer > memory. > Currently we control the maximum number of bytes per topic and partition > by setting

Re: KIP-41: KafkaConsumer Max Records

I think it is a general issue to bound the memory footprint on the Java consumer, no matter whether we do the prefetching in this KIP as even today we do not have anyway to manage memory usage on the consumer. Today on the producer side we bound the memory usage, and we may need to do the same on

Jenkins build is back to normal : kafka-trunk-jdk8 #275

See

[GitHub] kafka pull request: KAFKA-2979: Enable authorizer and ACLs in duck...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/683 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabl

[jira] [Commented] (KAFKA-2979) Enable authorizer and ACLs in ducktape tests

[ https://issues.apache.org/jira/browse/KAFKA-2979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088663#comment-15088663 ] ASF GitHub Bot commented on KAFKA-2979: --- Github user asfgit closed the pull request

[jira] [Resolved] (KAFKA-2979) Enable authorizer and ACLs in ducktape tests

[ https://issues.apache.org/jira/browse/KAFKA-2979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava resolved KAFKA-2979. -- Resolution: Fixed Issue resolved by pull request 683 [https://github.com/apache/

Build failed in Jenkins: kafka-trunk-jdk8 #276

See Changes: [me] KAFKA-2979: Enable authorizer and ACLs in ducktape tests -- Started by an SCM change [EnvInject] - Loading node environment variables. Building remotely on H11 (Ubuntu ubuntu) i

Build failed in Jenkins: kafka-trunk-jdk7 #946

See Changes: [me] KAFKA-2979: Enable authorizer and ACLs in ducktape tests -- [...truncated 47 lines...] Building project 'core' with Scala version 2.10.6 :kafka-trunk-jdk7:clients:compileJavaNot

[jira] [Created] (KAFKA-3082) Make LogManager.InitialTaskDelayMs configurable

Rado Buransky created KAFKA-3082: Summary: Make LogManager.InitialTaskDelayMs configurable Key: KAFKA-3082 URL: https://issues.apache.org/jira/browse/KAFKA-3082 Project: Kafka Issue Type: Imp

Upcoming Kafka Summit this April

Hi, Everyone, I wanted to remind you of a couple of Kafka Summit related deadlines coming up. On Monday Jan 11 the Call for Proposals closes at 11:59pm Pacific Time. We encourage you to participate - share your stories and best practices. Submit your proposal at www.kafka-summit.org On Friday Ja