Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

In general max.message.delivery.wait.ms is a cleaner approach. That would make the guarantee clearer. That said, there seem subtleties in some scenarios: 1. I agree with Sumante that it is a little weird that a message could be expired immediately if it happens to enter a batch that is about to

[GitHub] kafka pull request #3188: KAFKA-5358: Consumer perf tool should count rebala...

Github user huxihx closed the pull request at: https://github.com/apache/kafka/pull/3188 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] kafka pull request #3723: KAFKA-5358: Consumer perf tool should count rebala...

GitHub user huxihx opened a pull request: https://github.com/apache/kafka/pull/3723 KAFKA-5358: Consumer perf tool should count rebalance time. You can merge this pull request into a Git repository by running: $ git pull https://github.com/huxihx/kafka KAFKA-5358

Re: [DISCUSS] KIP-188 - Add new metrics to support health checks

Hi, Rajini, Yes, if those error metrics are registered dynamically, we could worry about expiration later. Thanks, Jun On Fri, Aug 18, 2017 at 1:55 AM, Rajini Sivaram wrote: > Perhaps we could register dynamically for now. If we find that the cost of > retaining

[GitHub] kafka pull request #3717: KAFKA-5731 Corrected how the sink task worker upda...

Github user rhauch closed the pull request at: https://github.com/apache/kafka/pull/3717 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] kafka pull request #3722: KAFKA-5603: Don't abort TX for zombie tasks

GitHub user mjsax opened a pull request: https://github.com/apache/kafka/pull/3722 KAFKA-5603: Don't abort TX for zombie tasks You can merge this pull request into a Git repository by running: $ git pull https://github.com/mjsax/kafka

[GitHub] kafka pull request #3721: Upgrade to ducktape 0.7.0

GitHub user cmccabe opened a pull request: https://github.com/apache/kafka/pull/3721 Upgrade to ducktape 0.7.0 You can merge this pull request into a Git repository by running: $ git pull https://github.com/cmccabe/kafka KAFKA-5768 Alternatively you can review and apply

[GitHub] kafka pull request #3720: [DO NOT MERGE] KAFKA-3705: non-key joins

GitHub user guozhangwang opened a pull request: https://github.com/apache/kafka/pull/3720 [DO NOT MERGE] KAFKA-3705: non-key joins This is just for reviewing the diff easily to see how it is done by @jfillipiak. You can merge this pull request into a Git repository by running:

[jira] [Resolved] (KAFKA-2590) KIP-28: Kafka Streams Checklist

[ https://issues.apache.org/jira/browse/KAFKA-2590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-2590. -- Resolution: Fixed Assignee: Guozhang Wang > KIP-28: Kafka Streams Checklist >

[jira] [Created] (KAFKA-5768) Upgrade ducktape version to 0.7.0, and use new kill_java_processes

Colin P. McCabe created KAFKA-5768: -- Summary: Upgrade ducktape version to 0.7.0, and use new kill_java_processes Key: KAFKA-5768 URL: https://issues.apache.org/jira/browse/KAFKA-5768 Project: Kafka

[GitHub] kafka pull request #3719: KAFKA-5603: Don't abort TX for zombie tasks

GitHub user mjsax opened a pull request: https://github.com/apache/kafka/pull/3719 KAFKA-5603: Don't abort TX for zombie tasks You can merge this pull request into a Git repository by running: $ git pull https://github.com/mjsax/kafka

[GitHub] kafka pull request #3718: KAFKA-5767; Kafka server should halt if IBP < 1.0....

GitHub user lindong28 opened a pull request: https://github.com/apache/kafka/pull/3718 KAFKA-5767; Kafka server should halt if IBP < 1.0.0 and there is log directory failure You can merge this pull request into a Git repository by running: $ git pull

[GitHub] kafka pull request #3717: KAFKA-5731 Corrected how the sink task worker upda...

GitHub user rhauch opened a pull request: https://github.com/apache/kafka/pull/3717 KAFKA-5731 Corrected how the sink task worker updates the last committed offsets (0.10.2) Prior to this change, it was possible for the synchronous consumer commit request to be handled before

[jira] [Created] (KAFKA-5767) Kafka server should halt if IBP < 1.0.0 and there is log directory failure

Dong Lin created KAFKA-5767: --- Summary: Kafka server should halt if IBP < 1.0.0 and there is log directory failure Key: KAFKA-5767 URL: https://issues.apache.org/jira/browse/KAFKA-5767 Project: Kafka

Re: [DISCUSS] KIP-179: Change ReassignPartitionsCommand to use AdminClient

On Wed, Aug 9, 2017, at 07:17, Tom Bentley wrote: > Hi Dong and Jun, > > Thanks again for your input in this discussion and on KIP-113. It's > difficult that discussion is split between this thread and the one for > KIP-113, but I'll try to respond on this thread to questions asked on > this >

Re: [DISCUSS] KIP-185: Make exactly once in order delivery per partition the default producer setting

Thanks Apurva, On compatibility: I think the proposal makes sense. It's a pity that we can't support idempotence for 0.11.0.0 brokers in the "safe" mode even if it is supported by the broker. I can already imagine users complaining about this, but I guess it's the consequence of missing the

Re: [DISCUSS] KIP-184 Rename LogCleaner and related classes to LogCompactor

Hi Pranav, Yeah, I'd recommend closing it since the benefit is unclear and since no one has jumped in to offer stronger support for the change. Were you planning to do a KIP to deprecate `log.cleaner.enable`? I still think that makes sense. Thanks, Jason On Tue, Aug 22, 2017 at 1:47 PM, Colin

Re: [DISCUSS] KIP-189: Improve principal builder interface and add support for SASL

Bump. I'll open a vote in a few days if there are no comments. Thanks, Jason On Sat, Aug 19, 2017 at 12:28 AM, Ismael Juma wrote: > Thanks for the KIP Jason. It seems reasonable and cleans up some > inconsistencies in that area. It would be great to get some feedback from >

[GitHub] kafka pull request #3716: KAFKA-5342: Clarify fatal/abortable exceptions use...

GitHub user hachikuji opened a pull request: https://github.com/apache/kafka/pull/3716 KAFKA-5342: Clarify fatal/abortable exceptions used in producer You can merge this pull request into a Git repository by running: $ git pull https://github.com/hachikuji/kafka KAFKA-5342

[GitHub] kafka pull request #3715: KafkaService: print node hostname on failure

GitHub user cmccabe opened a pull request: https://github.com/apache/kafka/pull/3715 KafkaService: print node hostname on failure Print node hostname on failure. Check for both "kafka.Kafka" and "metrics.SupportedKafka" process names. You can merge this pull request into a Git

[GitHub] kafka pull request #3714: close iterator on doc example

GitHub user abbccdda opened a pull request: https://github.com/apache/kafka/pull/3714 close iterator on doc example The iterator interface usage has some examples missing explicit close operation after usage. We should remind the user to do so because un-closed iterator will leave

Re: [DISCUSS] KIP-184 Rename LogCleaner and related classes to LogCompactor

Hmm. There are a lot of configuration keys that involve "log cleaner." It seems like if we rename this component, logically we'd have to rename all of them and support the old versions as deprecated config keys: val LogCleanupPolicyProp = "log.cleanup.policy" val LogCleanerThreadsProp =

Re: [DISCUSS] KIP-171: Extend Consumer Group Reset Offset for Stream Application

Thanks for the update Jorge. I don't have any further comments. -Matthias On 8/12/17 6:43 PM, Jorge Esteban Quilcate Otoya wrote: > I have updated the KIP: > > - Change execution parameters, using `--dry-run` > - Reference KAFKA-4327 > - And advise about changes on `StreamResetter` > > Also

Re: [DISCUSS] KIP-183 - Change PreferredReplicaLeaderElectionCommand to use AdminClient

Hi, Thanks for the KIP. It looks good overall. On Tue, Aug 22, 2017, at 08:54, Tom Bentley wrote: > Hi Jun, > > Thanks for the feedback. > > I've updated the KIP to mention this new algorithm, which I agree will be > much better from the AdminClient PoV. > > I've also reverted the

Re: [DISCUSS] KIP-182: Reduce Streams DSL overloads and allow easier use of custom storage engines

Thanks to the update Damian! Couple of comments: KStream: leftJoin and outerJoin for KStream/KTable join should not have `JoinWindows` parameter Nit: TopologyBuilder -> Topology Nit: new class Serialized list static method #with twice WindowedKStream -> for consistency we should either

[jira] [Resolved] (KAFKA-3139) JMX metric ProducerRequestPurgatory doesn't exist, docs seem wrong.

[ https://issues.apache.org/jira/browse/KAFKA-3139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-3139. -- Resolution: Fixed Fixed in KAFKA-4252 > JMX metric ProducerRequestPurgatory doesn't exist, docs seem

[GitHub] kafka pull request #3713: MINOR: simplify state transition for Kafka Streams...

GitHub user guozhangwang opened a pull request: https://github.com/apache/kafka/pull/3713 MINOR: simplify state transition for Kafka Streams and stream threads You can merge this pull request into a Git repository by running: $ git pull https://github.com/guozhangwang/kafka

Re: Fault Injection

Hi Colin, The Kibosh code is just a README for now, is it going to be published soon? Tim On Tue, Aug 22, 2017 at 11:44 AM, Colin McCabe wrote: > Hi all, > > I've been working on a fault injector for Apache Kafka. The general > idea is to create faults such as network

[jira] [Resolved] (KAFKA-3653) expose the queue size in ControllerChannelManager

[ https://issues.apache.org/jira/browse/KAFKA-3653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-3653. -- Resolution: Fixed Fixed in KAFKA-5135/KIP-143 > expose the queue size in ControllerChannelManager >

[jira] [Resolved] (KAFKA-3800) java client can`t poll msg

[ https://issues.apache.org/jira/browse/KAFKA-3800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-3800. -- Resolution: Cannot Reproduce Please reopen if the issue still exists. > java client can`t poll msg

[jira] [Resolved] (KAFKA-3927) kafka broker config docs issue

[ https://issues.apache.org/jira/browse/KAFKA-3927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-3927. -- Resolution: Later Yes, These changes are done in KAFKA-615. Please reopen if the issue still exists.

Build failed in Jenkins: kafka-trunk-jdk8 #1933

See Changes: [wangguoz] KAFKA-5152: perform state restoration in poll loop -- [...truncated 920.92 KB...] kafka.integration.TopicMetadataTest > testGetAllTopicMetadata

Build failed in Jenkins: kafka-trunk-jdk7 #2664

See Changes: [wangguoz] KAFKA-5152: perform state restoration in poll loop -- [...truncated 915.54 KB...] kafka.server.ControlledShutdownLeaderSelectorTest >

[jira] [Created] (KAFKA-5766) Very high CPU-load of consumer when broker is down

Sebastian Bernauer created KAFKA-5766: - Summary: Very high CPU-load of consumer when broker is down Key: KAFKA-5766 URL: https://issues.apache.org/jira/browse/KAFKA-5766 Project: Kafka

[jira] [Resolved] (KAFKA-4823) Creating Kafka Producer on application running on Java older version

[ https://issues.apache.org/jira/browse/KAFKA-4823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-4823. -- Resolution: Won't Fix Kafka Dropped support for Java 1.6 from 0.9 release. You can try Rest

Fault Injection

Hi all, I've been working on a fault injector for Apache Kafka. The general idea is to create faults such as network partitions or disk failures, and see what happens in the cluster. The fault injector can run as part of a ducktape system test, or standalone. The fault injector has two

[jira] [Resolved] (KAFKA-5239) Producer buffer pool allocates memory inside a lock.

[ https://issues.apache.org/jira/browse/KAFKA-5239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-5239. -- Resolution: Fixed Assignee: Sean McCauliff Fix Version/s: 1.0.0 > Producer buffer pool

Build failed in Jenkins: kafka-trunk-jdk8 #1932

See Changes: [damian.guy] HOTFIX: Cherrypicking state deadlock fix from 0.11 -- [...truncated 4.33 MB...] kafka.integration.AutoOffsetResetTest >

Re: Confluence access rights

Attila, I have added you to the wiki page list. Cheers. Guozhang On Tue, Aug 22, 2017 at 4:29 AM, Attila Kreiner wrote: > Hey All, > > I need to create a KIP for this: > https://issues.apache.org/jira/browse/KAFKA-5726 > https://github.com/apache/kafka/pull/3669 > >

[GitHub] kafka pull request #3675: KAFKA-5152: perform state restoration in poll loop

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3675 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[jira] [Resolved] (KAFKA-5401) Kafka Over TLS Error - Failed to send SSL Close message - Broken Pipe

[ https://issues.apache.org/jira/browse/KAFKA-5401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-5401. -- Resolution: Duplicate > Kafka Over TLS Error - Failed to send SSL Close message - Broken Pipe >

Build failed in Jenkins: kafka-trunk-jdk8 #1931

See Changes: [ismael] KAFKA-4856; Synchronize registerAppInfo and unregisterAppInfo [ismael] KAFKA-5752; Update index files correctly during async delete --

Re: The last batch of data is never qualified for processing by the punctuate method. - Kafka Streams

The call to punctuate depend on progress on "stream time" (internally tracked progress of time, depending whatever is returned by TimestampExtractor). Thus, if you don't receive any more date, "stream time" does not make progress anymore and thus, punctuate is not called anymore. Does this

Jenkins build is back to normal : kafka-trunk-jdk7 #2663

See

Re: [DISCUSS] KIP-183 - Change PreferredReplicaLeaderElectionCommand to use AdminClient

Hi Jun, Thanks for the feedback. I've updated the KIP to mention this new algorithm, which I agree will be much better from the AdminClient PoV. I've also reverted the authorization to be against the cluster resource, rather than the topic, since, on re-reading, Ewen wasn't particularly

each file of the kafka data will be opened 70+ times

hi all, When i use kafka cluster in our project, we found a problem, each file(log, index, timeidex) of the kafka will be open many times(about 70+), was this normal? or was a bug? how to fix this. Thanks for your help！ -- Best Regards! ___

Build failed in Jenkins: kafka-trunk-jdk7 #2662

See Changes: [ismael] KAFKA-4856; Synchronize registerAppInfo and unregisterAppInfo -- [...truncated 919.94 KB...] kafka.server.ControlledShutdownLeaderSelectorTest >

[jira] [Created] (KAFKA-5764) KafkaShortnamer should allow for case inensitive matches

Ryan P created KAFKA-5764: - Summary: KafkaShortnamer should allow for case inensitive matches Key: KAFKA-5764 URL: https://issues.apache.org/jira/browse/KAFKA-5764 Project: Kafka Issue Type:

[GitHub] kafka pull request #3712: MINOR: Fix transient failure in SocketServerTest.t...

GitHub user rajinisivaram opened a pull request: https://github.com/apache/kafka/pull/3712 MINOR: Fix transient failure in SocketServerTest.testConnectionIdReuse You can merge this pull request into a Git repository by running: $ git pull

[GitHub] kafka pull request #3691: HOTFIX: Cherrypicking state deadlock fix from 0.11

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3691 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

Build failed in Jenkins: kafka-trunk-jdk8 #1930

See Changes: [damian.guy] MINOR: Document how to create source streams and tables -- [...truncated 926.34 KB...] kafka.controller.ControllerIntegrationTest >

[GitHub] kafka pull request #3700: KAFKA-5752: Update timeIndex, txnIndex file pointe...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3700 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

Kafka monitoring - Facing issues

Hello All, We are moving into production for a Kafka project next week and we do not have a working monitoring tool that'll help us with it. We've tried 2 tools till now and none of them are working. 1) Datadog: There's a bug at their end and that's why it is not working. 2) Burrow: I tried

[jira] [Resolved] (KAFKA-4856) Calling KafkaProducer.close() from multiple threads may cause spurious error

[ https://issues.apache.org/jira/browse/KAFKA-4856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-4856. Resolution: Fixed Fix Version/s: 1.0.0 0.11.0.1 > Calling

[GitHub] kafka pull request #3696: KAFKA-4856: make unregisterAppInfo method to synch...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3696 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

Build failed in Jenkins: kafka-trunk-jdk7 #2661

See Changes: [damian.guy] MINOR: Document how to create source streams and tables -- [...truncated 917.13 KB...] kafka.controller.ControllerEventManagerTest >

Re: [DISCUSS] KIP-182: Reduce Streams DSL overloads and allow easier use of custom storage engines

I've just updated the KIP with some additional changes targeted at StreamsBuilder Thanks, Damian On Thu, 10 Aug 2017 at 12:59 Damian Guy wrote: > >> Got it, thanks. >> >> Does it still make sense to have one static constructors for each spec, >> with one constructor

[GitHub] kafka pull request #3701: MINOR: Document how to create source streams and t...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3701 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[jira] [Created] (KAFKA-5763) Refactor NetworkClient to use LogContext

Ismael Juma created KAFKA-5763: -- Summary: Refactor NetworkClient to use LogContext Key: KAFKA-5763 URL: https://issues.apache.org/jira/browse/KAFKA-5763 Project: Kafka Issue Type: Improvement

Re: [DISCUSS] KIP-187 - Add cumulative count metric for all Kafka rate metrics

Hi all, If there are no more comments, I will start vote on this KIP tomorrow. Please let me know if there are any other concerns. Thanks, Rajini On Fri, Aug 18, 2017 at 3:38 AM, Rajini Sivaram wrote: > Hi Jun, > > I have added constructors for all the variations.

[jira] [Created] (KAFKA-5762) Refactor AdminClient to use LogContext

Ismael Juma created KAFKA-5762: -- Summary: Refactor AdminClient to use LogContext Key: KAFKA-5762 URL: https://issues.apache.org/jira/browse/KAFKA-5762 Project: Kafka Issue Type: Improvement

Re: [DISCUSS] KIP-188 - Add new metrics to support health checks

Hi all, We can discuss the expiry implementation as part of the PR review. If there are no further comments, I will start vote on this KIP tomorrow. Please let me know if there are any other concerns. On Fri, Aug 18, 2017 at 4:55 AM, Rajini Sivaram wrote: > Perhaps we

Re: [DISCUSS] KIP-152 - Improve diagnostics for SASL authentication failures

Hi all, If there are no more comments, I will start vote on this KIP tomorrow. Please let me know if there are any concerns. Thanks, Rajini On Mon, Aug 21, 2017 at 9:24 AM, Rajini Sivaram wrote: > Hi Jun, > > Thank you for the review. > > 1. Each token response will

Build failed in Jenkins: kafka-trunk-jdk8 #1929

See Changes: [damian.guy] KAFKA-5689; Add MeteredWindowStore and refactor store hierarchy -- [...truncated 4.32 MB...] kafka.integration.AutoOffsetResetTest >

Confluence access rights

Hey All, I need to create a KIP for this: https://issues.apache.org/jira/browse/KAFKA-5726 https://github.com/apache/kafka/pull/3669 However, I can't edit the page https://cwiki.apache.org/ confluence/display/KAFKA/Kafka+Improvement+Proposals and when I click Create button I do not see the KIP

Build failed in Jenkins: kafka-trunk-jdk7 #2660

See Changes: [damian.guy] KAFKA-5689; Add MeteredWindowStore and refactor store hierarchy -- [...truncated 915.41 KB...]

[jira] [Resolved] (KAFKA-5751) Kafka cannot start; corrupted index file(s)

[ https://issues.apache.org/jira/browse/KAFKA-5751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-5751. -- Resolution: Duplicate > Kafka cannot start; corrupted index file(s) >

[GitHub] kafka pull request #3692: KAFKA-5689:Add MeteredWindowStore and refactor sto...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3692 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[jira] [Resolved] (KAFKA-5689) Refactor WindowStore hierarchy so that Metered Store is the outermost store

[ https://issues.apache.org/jira/browse/KAFKA-5689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damian Guy resolved KAFKA-5689. --- Resolution: Fixed Fix Version/s: 1.0.0 Issue resolved by pull request 3692

Re: [DISCUSS] KIP-91 Provide Intuitive User Timeouts in The Producer

Hi all, The discussion has been going on for a while, would it help to have a call to discuss this? I'd like to start a vote soonish so that we can include this in 1.0.0. I personally prefer max.message.delivery.wait.ms. It seems like Jun, Apurva and Jason also prefer that. Sumant, it seems like

Re: [DISCUSS] 0.11.0.1 bug fix release

An update on the 0.11.0.1 release. We currently have 5 outstanding issues: https://issues.apache.org/jira/projects/KAFKA/versions/12340632 3 with patches available that we can hopefully get merged pretty soon (1 is actually already on 0.11.0.1) 2 issues that are Open, 1 is unassigned. Hopefully we

[jira] [Created] (KAFKA-5761) Serializer API should support ByteBuffer

Bhaskar Gollapudi created KAFKA-5761: Summary: Serializer API should support ByteBuffer Key: KAFKA-5761 URL: https://issues.apache.org/jira/browse/KAFKA-5761 Project: Kafka Issue Type:

[GitHub] kafka pull request #3711: MINOR: Stateless transformation documentation

GitHub user enothereska opened a pull request: https://github.com/apache/kafka/pull/3711 MINOR: Stateless transformation documentation Needs to come after https://github.com/apache/kafka/pull/3701 You can merge this pull request into a Git repository by running: $ git pull

[GitHub] kafka pull request #3710: [MINOR] Fix comment on how to consume __consumer_o...

GitHub user Elyahou opened a pull request: https://github.com/apache/kafka/pull/3710 [MINOR] Fix comment on how to consume __consumer_offsets You can merge this pull request into a Git repository by running: $ git pull https://github.com/Elyahou/kafka offset_consuming_comment