Jenkins problem with JDK 8 and Scala 2.12 ?

Hi devs, is there any problems with Jenkins for building with JDK 8 and Scala 2.12. I see the following error in this console log (https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/6618/consoleFull) ERROR: Error fetching remote repo 'origin' hudson.plugins.git.GitException: Failed to fetch

Re: Jenkins problem with JDK 8 and Scala 2.12 ?

I think the JDK8 Jenkins server had disk space issues yesterday and failed all builds, maybe this is still the same / caused by that as well. On Tue, Aug 8, 2017 at 9:52 AM, Paolo Patierno wrote: > Hi devs, > > is there any problems with Jenkins for building with JDK 8 and Scala 2.12. > > I see

[jira] [Created] (KAFKA-5710) KafkaAdminClient should remove inflight call correctly after response is received

Dong Lin created KAFKA-5710: --- Summary: KafkaAdminClient should remove inflight call correctly after response is received Key: KAFKA-5710 URL: https://issues.apache.org/jira/browse/KAFKA-5710 Project: Kafka

[GitHub] kafka pull request #3642: KafkaAdminClient should remove inflight call corre...

GitHub user lindong28 opened a pull request: https://github.com/apache/kafka/pull/3642 KafkaAdminClient should remove inflight call correctly after response is received You can merge this pull request into a Git repository by running: $ git pull https://github.com/lindong28/k

[jira] [Resolved] (KAFKA-5710) KafkaAdminClient should remove inflight call correctly after response is received

[ https://issues.apache.org/jira/browse/KAFKA-5710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-5710. Resolution: Duplicate Duplicate of KAFKA-5658. > KafkaAdminClient should remove inflight call corre

[GitHub] kafka pull request #3642: KafkaAdminClient should remove inflight call corre...

Github user lindong28 closed the pull request at: https://github.com/apache/kafka/pull/3642 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is e

[GitHub] kafka pull request #3584: KAFKA-5658. Fix AdminClient request timeout handli...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3584 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

Re: [DISCUSS] KIP-174 - Deprecate and remove internal converter configs in WorkerConfig

Hi Ewen, Sorry, I am bit late in responding this. Thanks for your inputs and I've updated the KIP by adding more details to it. Regards, Umesh On Mon, 31 Jul 2017 at 21:51 Ewen Cheslack-Postava wrote: > On Sun, Jul 30, 2017 at 10:21 PM, UMESH CHAUDHARY > wrote: > >> Hi Ewen, >> Thanks for you

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

> > > > Also, how do you think things would work in the context of KIP-179? Would > > the tool still invoke these requests or would it be done by the broker > > receiving the alterTopics/reassignPartitions protocol call? > > > > My gut feel is that the tool will still invoke these requests. But I h

[VOTE] KIP-176 : Remove deprecated new-consumer option for tools

Hi devs, I didn't see any more comments about this KIP. The JIRAs related to the first step (so making --new-consumer as deprecated with warning messages) are merged. I'd like to start a vote for this KIP. Thanks, Paolo Patierno Senior Software Engineer (IoT) @ Red Hat Microsoft MVP on Wind

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hi Dong, Thanks for your reply. Yeah I agree with you that the total disk capacity can be useful > particularly if it is different across brokers but it is probably of > limited use in most cases. I also expect that most users would have their > own customized tool across to determine the new par

Re: [VOTE] KIP-176 : Remove deprecated new-consumer option for tools

The KIP is here for any one, like me, who hasn't seen it yet: https://cwiki.apache.org/confluence/display/KAFKA/KIP-176:+Remove+deprecated+new-consumer+option+for+tools Paolo, the KIP says "On the next release cycle we could totally remove the option." Exactly which release are you proposing that

Re: [VOTE] KIP-176 : Remove deprecated new-consumer option for tools

Hi Tom, good question. Due to the new version policies, the deprecation will be in the coming 1.0.0 release but then, due to a breaking change on the API, the removal should be on 2.0.0 I guess. Thanks, Paolo Patierno Senior Software Engineer (IoT) @ Red Hat Microsoft MVP on Windows Embedd

Build failed in Jenkins: kafka-0.11.0-jdk7 #262

See Changes: [ismael] KAFKA-5658; Fix AdminClient request timeout handling bug resulting in -- [...truncated 859.11 KB...] kafka.api.AdminClientWithPoliciesIntegrationTes

[jira] [Created] (KAFKA-5711) Bulk Restore Should Handle Deletes

Bill Bejeck created KAFKA-5711: -- Summary: Bulk Restore Should Handle Deletes Key: KAFKA-5711 URL: https://issues.apache.org/jira/browse/KAFKA-5711 Project: Kafka Issue Type: Bug Compon

[GitHub] kafka pull request #3643: MINOR: Error when incompatible console producer co...

GitHub user aishraj opened a pull request: https://github.com/apache/kafka/pull/3643 MINOR: Error when incompatible console producer configs. Spinned off from [KAFKA-2526](https://issues.apache.org/jira/browse/KAFKA-2526), to throw an exception if the console producer is passed a k

Re: Kafka Client for Swift 4

You should be able to edit the wiki now. Thanks for the contribution! -Jason On Mon, Aug 7, 2017 at 1:54 PM, Kellan Burket Cummings < kellan.bur...@gmail.com> wrote: > Thanks, Jason, > It's kellan.burket > - Kellan > > On Thu, Aug 3, 2017 at 6:39 PM, Jason Gustafson > wrote: > > > Hi Kellan, >

[jira] [Reopened] (KAFKA-5705) Kafka Server start failed and reports "unsafe memory access operation"

[ https://issues.apache.org/jira/browse/KAFKA-5705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson reopened KAFKA-5705: > Kafka Server start failed and reports "unsafe memory access operation" > -

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Thanks for your reply. Yes, my original idea is that user can continue to collect the static information for reassignment as they are doing now. It is the status quo. I agree it can be beneficial to have a tool in Kafka to collect other information that may be needed for reassignment so that user

[jira] [Created] (KAFKA-5712) Expose ProducerConfig in the KafkaProducer clients

Leo Xuzhang Lin created KAFKA-5712: -- Summary: Expose ProducerConfig in the KafkaProducer clients Key: KAFKA-5712 URL: https://issues.apache.org/jira/browse/KAFKA-5712 Project: Kafka Issue Ty

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hi Dong, Replies inline, as usual > As I originally envisaged it, KIP-179's support for reassigning partitions > > would have more-or-less taken the logic currently in the > > ReassignPartitionsCommand (that is, writing JSON to the > > ZkUtils.ReassignPartitionsPath) > > and put it behind a suita

[jira] [Created] (KAFKA-5713) Improve '--group' option to understand strings with wildcards

Alla Tumarkin created KAFKA-5713: Summary: Improve '--group' option to understand strings with wildcards Key: KAFKA-5713 URL: https://issues.apache.org/jira/browse/KAFKA-5713 Project: Kafka

[GitHub] kafka pull request #3644: KAFKA-5711: batch restore should handle deletes

GitHub user bbejeck opened a pull request: https://github.com/apache/kafka/pull/3644 KAFKA-5711: batch restore should handle deletes You can merge this pull request into a Git repository by running: $ git pull https://github.com/bbejeck/kafka KAFKA-5711_bulk_restore_should_ha

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hey Tom, Thanks for the quick reply. Please see my comment inline. On Tue, Aug 8, 2017 at 11:06 AM, Tom Bentley wrote: > Hi Dong, > > Replies inline, as usual > > > As I originally envisaged it, KIP-179's support for reassigning > partitions > > > > would have more-or-less taken the logic curre

[jira] [Created] (KAFKA-5714) Allow whitespaces in the principal name

Alla Tumarkin created KAFKA-5714: Summary: Allow whitespaces in the principal name Key: KAFKA-5714 URL: https://issues.apache.org/jira/browse/KAFKA-5714 Project: Kafka Issue Type: Improvement

Re: [DISCUSS] KIP-182: Reduce Streams DSL overloads and allow easier use of custom storage engines

Damian, Thanks for the proposal, I had a few comments on the APIs: 1. Printed#withFile seems not needed, as users should always spec if it is to sysOut or to File at the beginning. In addition as a second thought, I think serdes are not useful for prints anyways since we assume `toString` is prov

Way to check if custom SMT has been added to the classpath or even if it i working.

Hi All, i have created a custom SMT and have deployed. I would like to know if there is a way to check if the transform is working or not.(def not working as the messages are not getting transformed) I am also trying to remote debug using intellij and nothing seam working, as i do not see any con

[GitHub] kafka pull request #3572: MINOR: Remove unused GroupState.state field

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3572 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] kafka pull request #3613: KAFKA-2360: Extract producer-specific configs out ...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3613 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] kafka pull request #3645: MINOR: support retrieving cluster_id in system tes...

GitHub user xvrl opened a pull request: https://github.com/apache/kafka/pull/3645 MINOR: support retrieving cluster_id in system tests @ewencp You can merge this pull request into a Git repository by running: $ git pull https://github.com/xvrl/kafka system-test-cluster-id Alt

[GitHub] kafka pull request #3646: MINOR: Remove unneeded error handlers in deprecate...

GitHub user hachikuji opened a pull request: https://github.com/apache/kafka/pull/3646 MINOR: Remove unneeded error handlers in deprecated request objects These handlers were previously used on the broker to handle uncaught exceptions, but now the broker users the new Java request o

[GitHub] kafka pull request #3619: MINOR: Update dependencies for 1.0.0 release

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3619 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[DISCUSS] KIP-185: Make exactly once in order delivery per partition the default producer setting

Hi, I've put together a new KIP which proposes to ship Kafka with its strongest delivery guarantees by default. We currently ship with at most once semantics and don't provide any ordering guarantees per partition. The proposal is is to provide exactly once in order delivery per partition by defa

[GitHub] kafka pull request #3647: KAFKA-4501: Java 9 compilation fixes

GitHub user ijuma opened a pull request: https://github.com/apache/kafka/pull/3647 KAFKA-4501: Java 9 compilation fixes Compilation fixes: - Avoid ambiguity error when appending to Properties in Scala code - Use position() and limit() to fix ambiguity issue - Disable find

[DISCUSS] KIP-186: Increase offsets retention default to 7 days

Hi all, I posted a simple new KIP for a problem we see with a lot of users: KIP-186: Increase offsets retention default to 7 days https://cwiki.apache.org/confluence/display/KAFKA/KIP-186%3A+Increase+offsets+retention+default+to+7+days Note that in addition to the KIP text itself, the linked JIR

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hi, Dong, I think Tom was suggesting to have the AlterTopicsRequest sent to any broker, which just writes the reassignment json to ZK. The controller will pick up the reassignment and act on it as usual. This should work, right? Having a separate AlterTopicsRequest and AlterReplicaDirRequest seem

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hey Jun, Thanks for the comment! Yes, it should work. The tool can send request to any broker and broker can just write the reassignment znode. My previous intuition is that it may be better to only send this request to controller. But I don't have good reasons for this restriction. My intuition

Re: [DISCUSS] KIP-179: Change ReassignPartitionsCommand to use AdminClient

Hi, Tom, Thanks for the KIP. A few minor comments below. 1. Implementation wise, the broker handles adding partitions differently from changing replica assignment. For the former, we directly update the topic path in ZK with the new partitions. For the latter, we write the new partition reassignm

答复: [DISCUSS] KIP-178: Size-based log directory selection strategy

Hi Lin, Yes, it is a problem since it's not easy for us to predict the possible disk spaces each partition occupies in the future. How about the algorithm below: 1. Introduce a data structure to maintain the current free spaces for each log directory in `log.dirs`. Say, a ConcurrentHashMap

[GitHub] kafka pull request #3645: MINOR: support retrieving cluster_id in system tes...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3645 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[jira] [Created] (KAFKA-5715) ConsumerGroupCommand failed to show in ascending order for partitions without consumers

huxihx created KAFKA-5715: - Summary: ConsumerGroupCommand failed to show in ascending order for partitions without consumers Key: KAFKA-5715 URL: https://issues.apache.org/jira/browse/KAFKA-5715 Project: Kaf

[GitHub] kafka pull request #3648: KAFKA-5715: ConsumerGroupCommand should always sho...

GitHub user huxihx opened a pull request: https://github.com/apache/kafka/pull/3648 KAFKA-5715: ConsumerGroupCommand should always show partitions in ascending order Currently, ConsumerGroupCommand shows in ascending order for partitions with active consumer assigned, but failed to

[jira] [Resolved] (KAFKA-5704) Auto topic creation causes failure with older clusters

[ https://issues.apache.org/jira/browse/KAFKA-5704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava resolved KAFKA-5704. -- Resolution: Fixed Fix Version/s: 1.0.0 Issue resolved by pull request 364

[GitHub] kafka pull request #3641: KAFKA-5704 Corrected Connect distributed startup b...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3641 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

Re: 答复: [VOTE] KIP-177: Consumer perf tool should count rebalance time

Hu, The vote period is supposed to give people who might still want to vote -1 a chance to do so, so we'd leave it open until at least 72 hours after the vote was initiated. However, now that enough time has passed feel free to close the vote and move the KIP to Adopted. I'll give it a +1 as well

Re: [VOTE] KIP-163: Lower the Minimum Required ACL Permission of OffsetFetch

+1, thanks for the KIP! On Sun, Aug 6, 2017 at 5:22 AM, Mickael Maison wrote: > +1 > While it's not fixing the Read/Write inconsistency it's a step in the > right direction > > On Thu, Aug 3, 2017 at 4:46 PM, Jason Gustafson > wrote: > > +1 > > > > On Wed, Aug 2, 2017 at 2:40 PM, Vahid S Hashe

答复: 答复: [VOTE] KIP-177: Consumer perf tool should count rebalance time

Ewen, Thanks for the information. Will do that asap. 发件人: Ewen Cheslack-Postava 发送时间: 2017年8月9日 11:33 收件人: dev@kafka.apache.org 主题: Re: 答复: [VOTE] KIP-177: Consumer perf tool should count rebalance time Hu, The vote period is supposed to give people who might

Build failed in Jenkins: kafka-trunk-jdk8 #1891

See Changes: [me] MINOR: support retrieving cluster_id in system tests -- [...truncated 878.30 KB...] kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTo

Re: [DISCUSS] KIP-183 - Change PreferredReplicaLeaderElectionCommand to use AdminClient

Thanks for the KIP. Generally the move away from ZK and to native Kafka requests is good, so I'm generally +1 on this. A couple of comments/questions. * You gave the signature electPreferredReplicaLeader(Collection partitions) for the admin client. The old command allows not specifying the topic p

Build failed in Jenkins: kafka-0.11.0-jdk7 #263

See Changes: [me] MINOR: support retrieving cluster_id in system tests -- [...truncated 2.43 MB...] org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStor

Re: [DISCUSS] KIP-184 Rename LogCleaner and related classes to LogCompactor

Thanks Guozhang for the suggestion. For now, I have updated KIP incorporating your suggestion. Personally I think implicitly enabling compaction whenever policy is set to compact is more appropriate. Because new users like me will always assume that setting policy to compact will enable compaction

Re: [DISCUSS] KIP-174 - Deprecate and remove internal converter configs in WorkerConfig

Great, looking good. I'd probably be a bit more concrete about the Proposed Changes (e.g., "will log an warning if the config is specified" and "since the JsonConverter is the default, the configs will be removed immediately from the example worker configuration files"). Other than that this LGTM

Re: [DISCUSS] KIP-184 Rename LogCleaner and related classes to LogCompactor

A simple log message is standard, but the KIP should probably specify what happens when the deprecated config is encountered. Other than that, the change LGTM. Other things that might be worth addressing * log.compactor.min.compaction.lag.ms seems a bit redundant with compactor and compaction. No

Jenkins build is back to normal : kafka-trunk-jdk8 #1892

See

Re: [DISCUSS] KIP-185: Make exactly once in order delivery per partition the default producer setting

Apurva, For the benchmarking, I have a couple of questions: 1. Re: the mention of exactly once, this is within a producer session, right? And so really only idempotent. Applications still need to take extra steps for exactly once if they, e.g., are producing data from some other log like a DB txn