Re: Consumer Offsets Compaction

Thanks for the background context Jay. Do we have any context on what size is small (but still effect for small deployments) for the compaction buffer? and what is large? what factors help you choose the correct (or a safe) size? Currently the default "log.cleaner.dedupe.buffer.size" is 500 MiB.

Re: [DISCUSS] KIP-32 - Add CreateTime and LogAppendTime to Kafka message

Jun, Jiangjie, I am confused about 3) here, if we use "the timestamp of the latest message" then doesn't this mean we will roll the log whenever a message delayed by rolling time is received as well? Just to clarify, my understanding of "the timestamp of the latest message", for example in the

[GitHub] kafka pull request: MINOR: Improve README

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/677 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

Re: [DISCUSS] KIP-32 - Add CreateTime and LogAppendTime to Kafka message

Jun, 1. I agree it would be nice to have the timestamps used in a unified way. My concern is that if we let server change timestamp of the inner message for LogAppendTime, that will enforce the user who are using LogAppendTime to always pay the recompression penalty. So using LogAppendTime makes

Re: Consumer Offsets Compaction

The buffer determines the maximum number of unique keys in the new writes that can be processed in one cleaning. Each key requires 24 bytes of space iirc, so 500 MB = ~21,845,333 unique keys (this is actually adjusted for some load factor and divided by the number of cleaner threads). If it is too

[jira] [Commented] (KAFKA-2547) Make DynamicConfigManager to use the ZkNodeChangeNotificationListener introduced as part of KAFKA-2211

[ https://issues.apache.org/jira/browse/KAFKA-2547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15058846#comment-15058846 ] ASF GitHub Bot commented on KAFKA-2547: --- GitHub user Parth-Brahmbhatt opened a pull request:

[jira] [Work started] (KAFKA-2547) Make DynamicConfigManager to use the ZkNodeChangeNotificationListener introduced as part of KAFKA-2211

[ https://issues.apache.org/jira/browse/KAFKA-2547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-2547 started by Parth Brahmbhatt. --- > Make DynamicConfigManager to use the ZkNodeChangeNotificationListener >

[GitHub] kafka pull request: KAFKA-2547: Make DynamicConfigManager to use t...

GitHub user Parth-Brahmbhatt opened a pull request: https://github.com/apache/kafka/pull/679 KAFKA-2547: Make DynamicConfigManager to use the ZkNodeChangeNotifica… …tionListener introduced as part of KAFKA-2211 You can merge this pull request into a Git repository by running:

[GitHub] kafka pull request: KAFKA-2509: Replace LeaderAndIsr{Request,Respo...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/647 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] kafka pull request: MINOR: StreamThread performance optimization

GitHub user ymatsuda opened a pull request: https://github.com/apache/kafka/pull/680 MINOR: StreamThread performance optimization @guozhangwang You can merge this pull request into a Git repository by running: $ git pull https://github.com/ymatsuda/kafka perf Alternatively

Build failed in Jenkins: kafka-trunk-jdk7 #907

See Changes: [harsha] MINOR: Improve README -- [...truncated 2801 lines...] kafka.integration.SaslSslTopicMetadataTest > testAutoCreateTopic PASSED kafka.integration.SaslSslTopicMetadataTest >

Re: Consumer Offsets Compaction

I'm thinking that anyone who actually uses compaction has non-standard configuration (at the very least, they had to enable the cleaner, and probably few other configurations too... Compaction is a bit fiddly from what I've seen). So, I'm in favor of minimal default buffer just for offsets and

[jira] [Updated] (KAFKA-2509) Replace LeaderAndIsr{Request,Response} with org.apache.kafka.common.network.requests equivalent

[ https://issues.apache.org/jira/browse/KAFKA-2509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-2509: Resolution: Fixed Fix Version/s: 0.9.1.0 Status: Resolved (was: Patch Available)

[jira] [Commented] (KAFKA-2509) Replace LeaderAndIsr{Request,Response} with org.apache.kafka.common.network.requests equivalent

[ https://issues.apache.org/jira/browse/KAFKA-2509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15058897#comment-15058897 ] ASF GitHub Bot commented on KAFKA-2509: --- Github user asfgit closed the pull request at:

Build failed in Jenkins: kafka-trunk-jdk8 #238

See Changes: [harsha] MINOR: Improve README -- [...truncated 2827 lines...] kafka.log.LogTest > testTruncateTo PASSED kafka.log.LogTest > testCleanShutdownFile PASSED

Re: Consumer Offsets Compaction

Following up based on some digging. There are some upper and lower bounds on the buffer size: log.cleaner.dedupe.buffer.size has a: - Minimum of 1 MiB per cleaner thread - https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/KafkaConfig.scala#L950 -

[jira] [Commented] (KAFKA-2991) Improve collection type consistency in KafkaConsumer API

[ https://issues.apache.org/jira/browse/KAFKA-2991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059130#comment-15059130 ] Jay Kreps commented on KAFKA-2991: -- Sucks we didn't catch this in our reviews, changing it now will break

[GitHub] kafka pull request: Minor: Guard trace statements in the inner loo...

GitHub user ckolbeck opened a pull request: https://github.com/apache/kafka/pull/682 Minor: Guard trace statements in the inner loop of the replica fetcher I'm not sure whether this warrants a ticket based on the contribution guidelines. If the maintainers feel it does, I'm happy

[jira] [Commented] (KAFKA-2991) Improve collection type consistency in KafkaConsumer API

[ https://issues.apache.org/jira/browse/KAFKA-2991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059145#comment-15059145 ] Jason Gustafson commented on KAFKA-2991: [~jkreps] It's a little annoying to do so, but we won't

[GitHub] kafka pull request: Minor: updating comment that fell out of sync ...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/681 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] kafka pull request: Minor: updating comment that fell out of sync ...

GitHub user gwenshap opened a pull request: https://github.com/apache/kafka/pull/681 Minor: updating comment that fell out of sync with code You can merge this pull request into a Git repository by running: $ git pull https://github.com/gwenshap/kafka minor-herder-comment

[jira] [Created] (KAFKA-2992) Trace log statements in the replica fetcher inner loop create large amounts of garbage

Cory Kolbeck created KAFKA-2992: --- Summary: Trace log statements in the replica fetcher inner loop create large amounts of garbage Key: KAFKA-2992 URL: https://issues.apache.org/jira/browse/KAFKA-2992

Jenkins build is back to normal : kafka-trunk-jdk7 #908

See

[jira] [Commented] (KAFKA-2992) Trace log statements in the replica fetcher inner loop create large amounts of garbage

[ https://issues.apache.org/jira/browse/KAFKA-2992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059225#comment-15059225 ] Gwen Shapira commented on KAFKA-2992: - [~ijuma] I'm planning on merging the PR, since it is small and

[jira] [Commented] (KAFKA-2985) Consumer group stuck in rebalancing state

[ https://issues.apache.org/jira/browse/KAFKA-2985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059141#comment-15059141 ] Jens Rantil commented on KAFKA-2985: > Could you attach the source for your consumer? Sure:

Jenkins build is back to normal : kafka-trunk-jdk8 #239

See

[jira] [Commented] (KAFKA-2992) Trace log statements in the replica fetcher inner loop create large amounts of garbage

[ https://issues.apache.org/jira/browse/KAFKA-2992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059256#comment-15059256 ] Cory Kolbeck commented on KAFKA-2992: - I haven't, deploying hand patched outside software is difficult

[jira] [Commented] (KAFKA-2992) Trace log statements in the replica fetcher inner loop create large amounts of garbage

[ https://issues.apache.org/jira/browse/KAFKA-2992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059219#comment-15059219 ] Ismael Juma commented on KAFKA-2992: Thanks for reporting. I think it would be interesting to fix the

[jira] [Commented] (KAFKA-2992) Trace log statements in the replica fetcher inner loop create large amounts of garbage

[ https://issues.apache.org/jira/browse/KAFKA-2992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059241#comment-15059241 ] Ismael Juma commented on KAFKA-2992: Cory, my suggestion is to add annotations to the log method, not

[jira] [Created] (KAFKA-2991) Improve collection type consistency in KafkaConsumer API

Jason Gustafson created KAFKA-2991: -- Summary: Improve collection type consistency in KafkaConsumer API Key: KAFKA-2991 URL: https://issues.apache.org/jira/browse/KAFKA-2991 Project: Kafka

[jira] [Commented] (KAFKA-2992) Trace log statements in the replica fetcher inner loop create large amounts of garbage

[ https://issues.apache.org/jira/browse/KAFKA-2992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059227#comment-15059227 ] Cory Kolbeck commented on KAFKA-2992: - In looking around the codebase, this seemed to be the only

[jira] [Commented] (KAFKA-2992) Trace log statements in the replica fetcher inner loop create large amounts of garbage

[ https://issues.apache.org/jira/browse/KAFKA-2992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059245#comment-15059245 ] Ismael Juma commented on KAFKA-2992: Cory, did you verify that this change fixed the issue? > Trace

[jira] [Commented] (KAFKA-2992) Trace log statements in the replica fetcher inner loop create large amounts of garbage

[ https://issues.apache.org/jira/browse/KAFKA-2992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059246#comment-15059246 ] Cory Kolbeck commented on KAFKA-2992: - Ahh, that makes sense, apologies. > Trace log statements in

[GitHub] kafka pull request: Kafka-2992: Guard trace statements in the inne...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/682 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[jira] [Resolved] (KAFKA-2992) Trace log statements in the replica fetcher inner loop create large amounts of garbage

[ https://issues.apache.org/jira/browse/KAFKA-2992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira resolved KAFKA-2992. - Resolution: Fixed Fix Version/s: (was: 0.8.1.2) (was: 0.10.0.0)

[jira] [Commented] (KAFKA-2992) Trace log statements in the replica fetcher inner loop create large amounts of garbage

[ https://issues.apache.org/jira/browse/KAFKA-2992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059267#comment-15059267 ] Gwen Shapira commented on KAFKA-2992: - Merged, since I can see how this can lead to large number of

[jira] [Commented] (KAFKA-2992) Trace log statements in the replica fetcher inner loop create large amounts of garbage

[ https://issues.apache.org/jira/browse/KAFKA-2992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059402#comment-15059402 ] Grant Henke commented on KAFKA-2992: [~ijuma] [~gwenshap] This is related to KAFKA-2285 and the

[jira] [Updated] (KAFKA-2993) compression-rate-avg always returns 0 even with compression.type being set

[ https://issues.apache.org/jira/browse/KAFKA-2993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] TAO XIAO updated KAFKA-2993: Summary: compression-rate-avg always returns 0 even with compression.type being set (was:

[jira] [Commented] (KAFKA-2992) Trace log statements in the replica fetcher inner loop create large amounts of garbage

[ https://issues.apache.org/jira/browse/KAFKA-2992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059436#comment-15059436 ] Ismael Juma commented on KAFKA-2992: Thanks for the reference Grant, I was planning to link to

[GitHub] kafka pull request: MINOR: Add information to upgrade notes

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/678 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

Build failed in Jenkins: kafka-trunk-jdk8 #241

See Changes: [me] MINOR: Add information to upgrade notes -- Started by an SCM change [EnvInject] - Loading node environment variables. Building remotely on ubuntu-2 (docker Ubuntu ubuntu) in

Build failed in Jenkins: kafka-trunk-jdk7 #911

See Changes: [me] MINOR: Add information to upgrade notes -- Started by an SCM change [EnvInject] - Loading node environment variables. Building remotely on H11 (Ubuntu ubuntu) in workspace

Re: KAFKA Connect - Source Connector for Mainframe REST Services

Also please share some example of JDBC Source Connector running in distributed mode and assigning tasks across different workers. RegardsSaravanan On Tuesday, 15 December 2015 1:23 PM, saravanan tirugnanum wrote: Hi  I am working on designing and building a

Kafka Connect - JDBC - Entries and Libraries - missing

Hi , I just found that the pom.xml of Kafka-Connect-jdbc is missing these entriesAlso , both the libraries Common-Config and Common-Utils are not found in the confluent maven repo. Please upload and update the pom.xml for the below entry. https://github.com/confluentinc/kafka-connect-jdbc     

KAFKA Connect - Source Connector for Mainframe REST Services

Hi  I am working on designing and building a SourceConnector to run in a distributed mode to transfer data from Mainframe data sources which are exposed as RESTful services. So , planning to spin multiple workers handling and processing subset of data coordinating with  all workers in

[GitHub] kafka pull request: MINOR: Update upgrade notes to include additio...

GitHub user ijuma opened a pull request: https://github.com/apache/kafka/pull/678 MINOR: Update upgrade notes to include additional information Credit to Gwen for some of the text. You can merge this pull request into a Git repository by running: $ git pull

[GitHub] kafka pull request: MINOR: Improve README

GitHub user ijuma opened a pull request: https://github.com/apache/kafka/pull/677 MINOR: Improve README Mention Java 7 requirement, fix and improve formatting and remove obsolete reference to SVN website. You can merge this pull request into a Git repository by running: $ git

[GitHub] kafka pull request: MINOR: Fix typos in code comments

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/673 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

Build failed in Jenkins: kafka-trunk-jdk7 #910

See Changes: [cshapi] Kafka-2992: Guard trace statements in the inner loop of the replica -- [...truncated 1425 lines...] kafka.log.LogCleanerIntegrationTest > cleanerTest[3] PASSED

[jira] [Created] (KAFKA-2993) compression-rate-avg always returns 0 event with compression.type being set

TAO XIAO created KAFKA-2993: --- Summary: compression-rate-avg always returns 0 event with compression.type being set Key: KAFKA-2993 URL: https://issues.apache.org/jira/browse/KAFKA-2993 Project: Kafka

[jira] [Created] (KAFKA-2994) Transient Failure due to BindException: Address already in use

Guozhang Wang created KAFKA-2994: Summary: Transient Failure due to BindException: Address already in use Key: KAFKA-2994 URL: https://issues.apache.org/jira/browse/KAFKA-2994 Project: Kafka

[GitHub] kafka pull request: MINOR: StreamThread performance optimization

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/680 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is