[jira] [Created] (KAFKA-7082) Concurrent createTopics calls may throw NodeExistsException

[Ismael Juma (JIRA)]

Ismael Juma created KAFKA-7082:
--

 Summary: Concurrent createTopics calls may throw 
NodeExistsException
 Key: KAFKA-7082
 URL: https://issues.apache.org/jira/browse/KAFKA-7082
 Project: Kafka
  Issue Type: Bug
Affects Versions: 1.1.0
Reporter: Ismael Juma
Assignee: Ismael Juma
 Fix For: 2.0.1, 1.1.2


This exception is unexpected causing an `UnknownServerException` to be thrown 
back to the client. Example below:

{code}
org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = 
NodeExists for /config/topics/connect-configs
at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at kafka.zookeeper.AsyncResponse.maybeThrow(ZooKeeperClient.scala:472)
at kafka.zk.KafkaZkClient.createRecursive(KafkaZkClient.scala:1400)
at kafka.zk.KafkaZkClient.create$1(KafkaZkClient.scala:262)
at 
kafka.zk.KafkaZkClient.setOrCreateEntityConfigs(KafkaZkClient.scala:269)
at 
kafka.zk.AdminZkClient.createOrUpdateTopicPartitionAssignmentPathInZK(AdminZkClient.scala:99)
at kafka.server.AdminManager$$anonfun$2.apply(AdminManager.scala:126)
at kafka.server.AdminManager$$anonfun$2.apply(AdminManager.scala:81)
at 
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at 
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-6933) Broker reports Corrupted index warnings apparently infinitely

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-6933.

Resolution: Not A Bug

> Broker reports Corrupted index warnings apparently infinitely
> -
>
> Key: KAFKA-6933
> URL: https://issues.apache.org/jira/browse/KAFKA-6933
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 1.0.1
>Reporter: Franco Bonazza
>Priority: Major
>
> I'm running into a situation where the server logs show continuously the 
> following snippet:
> {noformat}
> [2018-05-23 10:58:56,590] INFO Loading producer state from offset 20601420 
> for partition transaction_r10_updates-6 with message format version 2 
> (kafka.log.Log)
> [2018-05-23 10:58:56,592] INFO Loading producer state from snapshot file 
> '/data/0/kafka-logs/transaction_r10_updates-6/20601420.snapshot' 
> for partition transaction_r10_u
> pdates-6 (kafka.log.ProducerStateManager)
> [2018-05-23 10:58:56,593] INFO Completed load of log 
> transaction_r10_updates-6 with 74 log segments, log start offset 0 and log 
> end offset 20601420 in 5823 ms (kafka.log.Log)
> [2018-05-23 10:58:58,761] WARN Found a corrupted index file due to 
> requirement failed: Corrupt index found, index file 
> (/data/0/kafka-logs/transaction_r10_updates-15/20544956.index) 
> has non-zero size but the last offset is 20544956 which is no larger than the 
> base offset 20544956.}. deleting 
> /data/0/kafka-logs/transaction_r10_updates-15/20544956.timeindex, 
> /data/0/kafka-logs/transaction_r10_updates-15/20544956.index, and 
> /data/0/kafka-logs/transaction_r10_updates-15/20544956.txnindex 
> and rebuilding index... (kafka.log.Log)
> [2018-05-23 10:58:58,763] INFO Loading producer state from snapshot file 
> '/data/0/kafka-logs/transaction_r10_updates-15/20544956.snapshot' 
> for partition transaction_r10_updates-15 (kafka.log.ProducerStateManager)
> [2018-05-23 10:59:02,202] INFO Recovering unflushed segment 20544956 in log 
> transaction_r10_updates-15. (kafka.log.Log){noformat}
> The set up is the following,
> Broker is 1.0.1
> There are mirrors from another cluster using client 0.10.2.1 
> There are kafka streams and other custom consumer / producers using 1.0.0 
> client.
>  
> While is doing this the JVM of the broker is up but it doesn't respond so 
> it's impossible to produce, consume or run any commands.
> If I delete all the index files the WARN turns into an ERROR, which takes a 
> long time (1 day last time I tried) but eventually it goes into a healthy 
> state, then I start the producers and things are still healthy, but when I 
> start the consumers it quickly goes into the original WARN loop, which seems 
> infinite.
>  
> I couldn't find any references to the problem, it seems to be at least 
> missreporting the issue, and perhaps it's not infinite? I let it loop over 
> the WARN for over a day and it never moved past that, and if there was 
> something really wrong with the state maybe it should be reported.
> The log cleaner log showed a few "too many files open" when it originally 
> happened but ulimit has always been set to unlimited so I'm not sure what 
> that error means.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-7063) Update documentation to remove references to old producers and consumers

[Ismael Juma (JIRA)]

Ismael Juma created KAFKA-7063:
--

 Summary: Update documentation to remove references to old 
producers and consumers
 Key: KAFKA-7063
 URL: https://issues.apache.org/jira/browse/KAFKA-7063
 Project: Kafka
  Issue Type: Improvement
Reporter: Ismael Juma


We should also remove any mention of "new consumer" or "new producer". They 
should just be "producer" and "consumer".

Finally, any mention of "Scala producer/consumer/client" should also be removed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-7062) Simplify MirrorMaker loop after removal of old consumer support

[Ismael Juma (JIRA)]

Ismael Juma created KAFKA-7062:
--

 Summary: Simplify MirrorMaker loop after removal of old consumer 
support
 Key: KAFKA-7062
 URL: https://issues.apache.org/jira/browse/KAFKA-7062
 Project: Kafka
  Issue Type: Improvement
Reporter: Ismael Juma


Once KAFKA-2983 is merged, we can simplify the MirrorMaker loop to be a single 
loop instead of two loops. In the old consumer, even if there is no message 
offsets would still be committed so receive() could block. The new consumer 
doesn't have this issue.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-6445) Remove deprecated metrics in 2.0

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-6445.

   Resolution: Fixed
 Assignee: Dong Lin  (was: Charly Molter)
Fix Version/s: (was: 2.1.0)
   2.0.0

> Remove deprecated metrics in 2.0
> 
>
> Key: KAFKA-6445
> URL: https://issues.apache.org/jira/browse/KAFKA-6445
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Charly Molter
>Assignee: Dong Lin
>Priority: Trivial
> Fix For: 2.0.0
>
>
> As part of KIP-225 we've replaced a metric and deprecated the old one.
> We should remove these metrics in 2.0.0 this Jira is to track all of the 
> metrics to remove in 2.0.0.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-2837) FAILING TEST: kafka.api.ProducerBounceTest > testBrokerFailure

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-2837.

   Resolution: Fixed
Fix Version/s: (was: 0.10.0.0)
   2.0.0

This test has been removed.

> FAILING TEST: kafka.api.ProducerBounceTest > testBrokerFailure 
> ---
>
> Key: KAFKA-2837
> URL: https://issues.apache.org/jira/browse/KAFKA-2837
> Project: Kafka
>  Issue Type: Sub-task
>Affects Versions: 0.9.0.0
>Reporter: Gwen Shapira
>Assignee: jin xing
>Priority: Major
>  Labels: newbie
> Fix For: 2.0.0
>
>
> {code}
> java.lang.AssertionError
>   at org.junit.Assert.fail(Assert.java:86)
>   at org.junit.Assert.assertTrue(Assert.java:41)
>   at org.junit.Assert.assertTrue(Assert.java:52)
>   at 
> kafka.api.ProducerBounceTest.testBrokerFailure(ProducerBounceTest.scala:117)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:105)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:56)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:64)
>   at 
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:50)
>   at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
>   at 
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
>   at 
> org.gradle.messaging.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
>   at 
> org.gradle.messaging.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
>   at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
>   at 
> org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:106)
>   at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
>   at 
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
>   at 
> org.gradle.messaging.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:360)
>   at 
> org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:54)
>   at 
> org.gradl

[jira] [Resolved] (KAFKA-4237) Avoid long request timeout for the consumer

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-4237.

Resolution: Duplicate

Duplicate of KAFKA-7050.

> Avoid long request timeout for the consumer
> ---
>
> Key: KAFKA-4237
> URL: https://issues.apache.org/jira/browse/KAFKA-4237
> Project: Kafka
>  Issue Type: Improvement
>  Components: consumer
>Reporter: Jason Gustafson
>Priority: Major
>
> In the consumer rebalance protocol, the JoinGroup can stay in purgatory on 
> the server for as long as the rebalance timeout. For the Java client, that 
> means that the request timeout must be at least as large as the rebalance 
> timeout (which is governed by {{max.poll.interval.ms}} since KIP-62 and 
> {{session.timeout.ms}} before then). By default, since 0.10.1, this is 5 
> minutes plus some change, which makes the clients slow to detect some hard 
> failures.
> To fix this, two options come to mind:
> 1. Right now, all request APIs are limited by the same request timeout in 
> {{NetworkClient}}, but there's not really any reason why this must be so. We 
> could use a separate timeout for the JoinGroup request (the implementations 
> of this is straightforward: 
> https://github.com/confluentinc/kafka/pull/108/files).
> 2. Alternatively, we could prevent the server from holding the JoinGroup in 
> purgatory for such a long time. Instead, it could return early from the 
> JoinGroup (say before the session timeout has expired) with an error code 
> (e.g. REBALANCE_IN_PROGRESS), which tells the client that it should just 
> resend the JoinGroup.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-3355) GetOffsetShell command doesn't work with SASL enabled Kafka

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-3355.

   Resolution: Duplicate
Fix Version/s: (was: 2.1.0)

This will be done as part of KAFKA-5235, so marking as duplicate.

> GetOffsetShell command doesn't work with SASL enabled Kafka
> ---
>
> Key: KAFKA-3355
> URL: https://issues.apache.org/jira/browse/KAFKA-3355
> Project: Kafka
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 0.9.0.1
> Environment: Kafka 0.9.0.1
>Reporter: TAO XIAO
>Assignee: Ashish Singh
>Priority: Major
>
> I found that GetOffsetShell doesn't work with SASL enabled Kafka. I believe 
> this is due to old producer being used in GetOffsetShell.
> Kafka version 0.9.0.1
> Exception
> % bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 
> localhost:9092 --topic test --time -1
> [2016-03-04 21:43:56,597] INFO Verifying properties 
> (kafka.utils.VerifiableProperties)
> [2016-03-04 21:43:56,613] INFO Property client.id is overridden to 
> GetOffsetShell (kafka.utils.VerifiableProperties)
> [2016-03-04 21:43:56,613] INFO Property metadata.broker.list is overridden to 
> localhost:9092 (kafka.utils.VerifiableProperties)
> [2016-03-04 21:43:56,613] INFO Property request.timeout.ms is overridden to 
> 1000 (kafka.utils.VerifiableProperties)
> [2016-03-04 21:43:56,674] INFO Fetching metadata from broker 
> BrokerEndPoint(0,localhost,9092) with correlation id 0 for 1 topic(s) 
> Set(test) (kafka.client.ClientUtils$)
> [2016-03-04 21:43:56,689] INFO Connected to localhost:9092 for producing 
> (kafka.producer.SyncProducer)
> [2016-03-04 21:43:56,705] WARN Fetching topic metadata with correlation id 0 
> for topics [Set(test)] from broker [BrokerEndPoint(0,localhost,9092)] failed 
> (kafka.client.ClientUtils$)
> java.nio.BufferUnderflowException
>   at java.nio.Buffer.nextGetIndex(Buffer.java:498)
>   at java.nio.HeapByteBuffer.getShort(HeapByteBuffer.java:304)
>   at kafka.api.ApiUtils$.readShortString(ApiUtils.scala:36)
>   at kafka.cluster.BrokerEndPoint$.readFrom(BrokerEndPoint.scala:52)
>   at 
> kafka.api.TopicMetadataResponse$$anonfun$1.apply(TopicMetadataResponse.scala:28)
>   at 
> kafka.api.TopicMetadataResponse$$anonfun$1.apply(TopicMetadataResponse.scala:28)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
>   at scala.collection.immutable.Range.foreach(Range.scala:166)
>   at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
>   at scala.collection.AbstractTraversable.map(Traversable.scala:104)
>   at 
> kafka.api.TopicMetadataResponse$.readFrom(TopicMetadataResponse.scala:28)
>   at kafka.producer.SyncProducer.send(SyncProducer.scala:120)
>   at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:59)
>   at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:94)
>   at kafka.tools.GetOffsetShell$.main(GetOffsetShell.scala:78)
>   at kafka.tools.GetOffsetShell.main(GetOffsetShell.scala)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-5234) GetOffsetShell: retrieve offsets for multiple topics with single request

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-5234.

Resolution: Duplicate

Let's consolidate GetOffsetShell improvements under KAFKA-5235.

> GetOffsetShell: retrieve offsets for multiple topics with single request
> 
>
> Key: KAFKA-5234
> URL: https://issues.apache.org/jira/browse/KAFKA-5234
> Project: Kafka
>  Issue Type: Improvement
>  Components: tools
>Reporter: Arseniy Tashoyan
>Priority: Major
>  Labels: tool
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> At present, GetOffsetShell is able to retrieve offsets for one topic only:
> --topic  REQUIRED: The topic to get offsets from.
> If user wants to get offsets for several topics, he has to call 
> GetOffsetShell as many times as the number of topics to explore. Some 
> solutions may have dozens of topics. Monitoring of a large Kafka cluster with 
> GetOffsetShell requires additional scripting efforts and produces visible 
> performance drawback due to multiple requests to the broker.
> Instead, GetOffsetShell should support multiple topics, for example:
> --topics topic1,topic2,topic3
> Moreover, GetOffsetShell should be able to retrieve offsets for _all_ topics, 
> when user specified none topics in command line.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-110: Add Codec for ZStandard Compression (Updated)

[Ismael Juma]

Sorry for the delay Dongjin. Everyone is busy finalising 2.0.0. This KIP
seems like a great candidate for 2.1.0 and hopefully there will be more of
a discussion next week. :)

Ismael

On Wed, 13 Jun 2018, 05:17 Dongjin Lee,  wrote:

> Hello. I just updated my draft implementation:
>
> 1. Rebased to latest trunk (commit 5145d6b)
> 2. Apply ZStd 1.3.4
>
> You can check out the implementation from here
> . If you experience any problem
> running it, don't hesitate to give me a mention.
>
> Best,
> Dongjin
>
> On Tue, Jun 12, 2018 at 6:50 PM Dongjin Lee  wrote:
>
> > Here is the short conclusion about the license problem: *We can use zstd
> > and zstd-jni without any problem, but we need to include their license,
> > e.g., BSD license.*
> >
> > Both of BSD 2 Clause License & 3 Clause License requires to include the
> > license used, and BSD 3 Clause License requires that the name of the
> > contributor can't be used to endorse or promote the product. That's it
> > <
> http://www.mikestratton.net/2011/12/is-bsd-license-compatible-with-apache-2-0-license/
> >
> > - They are not listed in the list of prohibited licenses
> >  also.
> >
> > Here is how Spark did for it
> > :
> >
> > - They made a directory dedicated to the dependency license files
> >  and added
> licenses
> > for Zstd
> > 
> &
> > Zstd-jni
> > <
> https://github.com/apache/spark/blob/master/licenses/LICENSE-zstd-jni.txt>
> > .
> > - Added a link to the original license files in LICENSE.
> > 
> >
> > If needed, I can make a similar update.
> >
> > Thanks for pointing out this problem, Viktor! Nice catch!
> >
> > Best,
> > Dongjin
> >
> >
> >
> > On Mon, Jun 11, 2018 at 11:50 PM Dongjin Lee  wrote:
> >
> >> I greatly appreciate your comprehensive reasoning. so: +1 for b until
> now.
> >>
> >> For the license issues, I will have a check on how the over projects are
> >> doing and share the results.
> >>
> >> Best,
> >> Dongjin
> >>
> >> On Mon, Jun 11, 2018 at 10:08 PM Viktor Somogyi <
> viktorsomo...@gmail.com>
> >> wrote:
> >>
> >>> Hi Dongjin,
> >>>
> >>> A couple of comments:
> >>> I would vote for option b. in the "backward compatibility" section. My
> >>> reasoning for this is that users upgrading to a zstd compatible version
> >>> won't start to use it automatically, so manual reconfiguration is
> >>> required.
> >>> Therefore an upgrade won't mess up the cluster. If not all the clients
> >>> are
> >>> upgraded but just some of them and they'd start to use zstd then it
> would
> >>> cause errors in the cluster. I'd like to presume though that this is a
> >>> very
> >>> obvious failure case and nobody should be surprised if it didn't work.
> >>> I wouldn't choose a. as I think we should bump the fetch and produce
> >>> requests if it's a change in the message format. Moreover if some of
> the
> >>> producers and the brokers are upgraded but some of the consumers are
> not,
> >>> then we wouldn't prevent the error when the old consumer tries to
> consume
> >>> the zstd compressed messages.
> >>> I wouldn't choose c. either as I think binding the compression type to
> an
> >>> API is not so obvious from the developer's perspective.
> >>>
> >>> I would also prefer to use the existing binding, however we must
> respect
> >>> the licenses:
> >>> "The code for these JNI bindings is licenced under 2-clause BSD
> license.
> >>> The native Zstd library is licensed under 3-clause BSD license and
> GPL2"
> >>> Based on the FAQ page
> >>> https://www.apache.org/legal/resolved.html#category-a
> >>> we may use 2- and 3-clause BSD licenses but the Apache license is not
> >>> compatible with GPL2. I'm hoping that the "3-clause BSD license and
> GPL2"
> >>> is really not an AND but an OR in this case, but I'm no lawyer, just
> >>> wanted
> >>> to make the point that we should watch out for licenses. :)
> >>>
> >>> Regards,
> >>> Viktor
> >>>
> >>>
> >>> On Sun, Jun 10, 2018 at 3:02 AM Ivan Babrou  wrote:
> >>>
> >>> > Hello,
> >>> >
> >>> > This is Ivan and I still very much support the fact that zstd
> >>> compression
> >>> > should be included out of the box.
> >>> >
> >>> > Please think about the environment, you can save quite a lot of
> >>> hardware
> >>> > with it.
> >>> >
> >>> > Thank you.
> >>> >
> >>> > On Sat, Jun 9, 2018 at 14:14 Dongjin Lee  wrote:
> >>> >
> >>> > > Since there are no responses for a week, I decided to reinitiate
> the
> >>> > > discussion thread.
> >>> > >
> >>> > >
> >>> > >
> >>> >
> >>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-110%3A+Add+Codec+for+ZStandard+Compression
> >>> > >
> >>> > > This KIP is about to introduce ZStandard Compression into Apache
> >>> Kafka.
> >>> > > The reason why it is

[jira] [Resolved] (KAFKA-7029) ReplicaVerificationTool should not use the deprecated SimpleConsumer

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-7029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-7029.

Resolution: Fixed

> ReplicaVerificationTool should not use the deprecated SimpleConsumer
> 
>
> Key: KAFKA-7029
> URL: https://issues.apache.org/jira/browse/KAFKA-7029
> Project: Kafka
>  Issue Type: Bug
>    Reporter: Ismael Juma
>Assignee: Manikumar
>Priority: Major
> Fix For: 2.0.0
>
>
> Unless there's a reason not to, the simplest would be to use KafkaConsumer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] KIP-266: Add TimeoutException for KafkaConsumer#position

[Ismael Juma]

; > > >
> > > > > > > > > > > > +1 for using `default.block.ms`.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > Dhruvil
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Jun 5, 2018 at 11:48 AM, Bill Bejeck <
> > > > > bbej...@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Jason,
> > > > > > > > > > > > >
> > > > > > > > > > > > > At first, I thought the same name between the
> > producer
> > > > and
> > > > > the
> > > > > > > > > consumer
> > > > > > > > > > > > was
> > > > > > > > > > > > > ideal.
> > > > > > > > > > > > >
> > > > > > > > > > > > > But your comment makes me realize consistent names
> > with
> > > > > > > different
> > > > > > > > > > > > semantics
> > > > > > > > > > > > > is even more confusing.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I'm +1 for not using `max.block.ms`.  I like
> > Guozhang's
> > > > > > > > > suggestion of
> > > > > > > > > > > `
> > > > > > > > > > > > > default.block.ms` for the name.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > Bill
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Jun 5, 2018 at 1:33 PM, Guozhang Wang <
> > > > > > > wangg...@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Jason,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Yeah I agree that "max.block.ms" makes people
> > thinking
> > > > > of
> > > > > > > the
> > > > > > > > > > > > producer's
> > > > > > > > > > > > > > config with the same name, but their semantics
> are
> > > > > different.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On the other hand, I'm a bit concerned with the
> > reusing
> > > > > of
> > > > > > > the
> > > > > > > > > term
> > > > > > > > > > > > > > `timeout` as we already have `session.timeout.ms
> `
> > and
> > > > `
> > > > > > > > > > > > > request.timeout.ms`
> > > > > > > > > > > > > > in ConsumerConfig.. How about using the name `
> > > > > > > > > default.api.block.ms`
> > > > > > > > > > > or
> > > > > > > > > > > > > > simply `default.block.ms`?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Guozhang
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, Jun 5, 2018 at 8:57 AM, Jason Gustafson <
> > > > > > > > > ja...@confluent.io>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hey All,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > &g

[jira] [Created] (KAFKA-7029) ReplicaVerificationTool should not use the deprecated SimpleConsumer

[Ismael Juma (JIRA)]

Ismael Juma created KAFKA-7029:
--

 Summary: ReplicaVerificationTool should not use the deprecated 
SimpleConsumer
 Key: KAFKA-7029
 URL: https://issues.apache.org/jira/browse/KAFKA-7029
 Project: Kafka
  Issue Type: Bug
Reporter: Ismael Juma


Unless there's a reason not to, the simplest would be to use KafkaConsumer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-7028) super.users doesn't work with custom principals

[Ismael Juma (JIRA)]

Ismael Juma created KAFKA-7028:
--

 Summary: super.users doesn't work with custom principals
 Key: KAFKA-7028
 URL: https://issues.apache.org/jira/browse/KAFKA-7028
 Project: Kafka
  Issue Type: Bug
Reporter: Ismael Juma
 Fix For: 2.0.0


SimpleAclAuthorizer creates a KafkaPrincipal for the users defined in the 
super.users broker config. However, it should use the configured 
KafkaPrincipalBuilder so that it works with a custom defined one.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: kafka ack=all and min-isr

[Ismael Juma]

The key point is:

if (leaderReplica.highWatermark.messageOffset >= requiredOffset) {

The high watermark only moves when all the replicas in ISR have that
particular offset. Does that clarify it?

Ismael

On Thu, Jun 7, 2018 at 10:31 PM Carl Samuelson 
wrote:

> Hi
>
> Hopefully this is the correct email address and forum for this.
> I asked this question on stack overflow, but did not get an answer:
> https://stackoverflow.com/questions/50689177/kafka-ack-all-and-min-isr
>
> *Summary*
>
> The docs and code comments for Kafka suggest that when the producer setting
> acks is set to allthen an ack will only be sent to the producer when *all
> in-sync replicas have caught up*, but the code (Partition.Scala,
> checkEnoughReplicasReachOffset) seems to suggest that the ack is sent as
> soon as *min in-sync replicas have caught up*.
>
> *Details*
>
> The kafka docs have this:
>
> acks=all This means the leader will wait for the full set of in-sync
> replicas to acknowledge the record. source
> 
>
> Also, looking at the Kafka source code - partition.scala
> checkEnoughReplicasReachOffset() has the following comment (emphasis mine):
>
> Note that this method will only be called if requiredAcks = -1 and we are
> waiting for *all replicas*in ISR to be fully caught up to the (local)
> leader's offset corresponding to this produce request before we acknowledge
> the produce request.
>
> Finally, this answer  on
> Stack
> Overflow (emphasis mine again)
>
> Also the min in-sync replica setting specifies the minimum number of
> replicas that need to be in-sync for the partition to remain available for
> writes. When a producer specifies ack (-1 / all config) it will still wait
> for acks from *all in sync replicas* at that moment (independent of the
> setting for min in-sync replicas).
>
> But when I look at the code in Partition.Scala (note minIsr <
> curInSyncReplicas.size):
>
> def checkEnoughReplicasReachOffset(requiredOffset: Long): (Boolean,
> Errors) = {
>   ...
>   val minIsr = leaderReplica.log.get.config.minInSyncReplicas
>   if (leaderReplica.highWatermark.messageOffset >= requiredOffset) {
> if (minIsr <= curInSyncReplicas.size)
>   (true, Errors.NONE)
>
> The code that calls this returns the ack:
>
> if (error != Errors.NONE || hasEnough) {
>   status.acksPending = false
>   status.responseStatus.error = error
> }
>
> So, the code looks like it returns an ack as soon as the in-sync replica
> set are greater than min in-sync replicas. However, the documentation and
> comments suggest that the ack is only sent once all in-sync replicas have
> caught up. What am I missing? At the very least, the comment above
> checkEnoughReplicasReachOffset looks like it should be changed.
> Regards,
>
> Carl
>

Re: [DISCUSS] 0.10.2.2 bug fix release

[Ismael Juma]

One more thing, I suggest we go with what we have  for all 3 releases you
are doing. We should aim to make the big fix release process as smooth as
possible and we should strive to avoid last minute additions to older
release branches. We can be a bit more flexible for 1.0.2 since it's more
recent.

Ismael

On 7 Jun 2018 4:34 pm, "Ismael Juma"  wrote:

Thanks for doing this Matthias, +1.

Ismael

On Thu, Jun 7, 2018 at 1:50 PM Matthias J. Sax 
wrote:

> Dear all,
>
> I want to propose a 0.10.2.2 bug fix release. 0.10.2.1 is over a year
> old and a couple of critical fixes are available for 0.10.2.2.
>
> Please find a list of all 24 resolved tickets here:
>
>
> https://issues.apache.org/jira/browse/KAFKA-6566?jql=project%20%3D%20KAFKA%20AND%20fixVersion%20%3D%200.10.2.2%20AND%20resolution%20!%3D%20Unresolved%20ORDER%20BY%20priority%20DESC%2C%20status%20DESC%2C%20updated%20DESC%20%20
>
> There are no open tickets with target version 0.10.2.2 at the moment. If
> there are any tickets you want to get included in 0.10.2.2 please let us
> know as soon as possible.
>
>
> If nobody objects, I plan to create the first RC for 0.10.2.2 next
> Thursday. Please find a summary in the wiki:
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+0.10.2.2
>
>
> Thanks a lot,
>   Matthias
>
>
>
>

Re: [DISCUSS] 0.11.0.3 bug fix release

[Ismael Juma]

+1, thanks!

On Thu, 7 Jun 2018, 11:16 Matthias J. Sax,  wrote:

> Dear all,
>
> I want to propose a 0.11.0.3 bug fix release. 0.11.0.2 is 6 months old
> and a couple of critical fixes are available for 0.11.0.3.
>
> Please find a list of all 16 resolved tickets here:
>
>
> https://issues.apache.org/jira/browse/KAFKA-6925?jql=project%20%3D%20KAFKA%20AND%20status%20in%20(Resolved%2C%20Closed)%20AND%20fixVersion%20%3D%200.11.0.3
>
> There are no open tickets with target version 0.11.0.3 at the moment. If
> there are any tickets you want to get included in 0.11.0.3 please let us
> know as soon as possible.
>
>
> If nobody objects, I plan to create the first RC for 0.11.0.3 next
> Thursday. Please find a summary in the wiki:
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+0.11.0.3
>
>
> Thanks a lot,
>   Matthias
>
>
>

Re: [DISCUSS] 1.0.2 bug fix release

[Ismael Juma]

+1, thanks!

On Thu, 7 Jun 2018, 11:16 Matthias J. Sax,  wrote:

> Dear all,
>
> I want to propose a 1.0.2 bug fix release. 1.0.1 is 3 months old and a
> couple of critical fixes are available for 1.0.2.
>
> Please find a list of all 14 resolved tickets here:
>
>
> https://issues.apache.org/jira/browse/KAFKA-6937?jql=project%20%3D%20KAFKA%20AND%20status%20in%20(Resolved%2C%20Closed)%20AND%20fixVersion%20%3D%201.0.2
>
> There are 7 open tickets with target version 1.0.2.
>
>
> https://issues.apache.org/jira/browse/KAFKA-6083?jql=project%20%3D%20KAFKA%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened%2C%20%22Patch%20Available%22)%20AND%20fixVersion%20%3D%201.0.2
>
> If you own one of these tickets, please let us know if you plan to
> resolve them soon. Otherwise, please change the target version to a
> future release.
>
> If there are any other tickets you want to get included in 1.0.2 please
> let us know as soon as possible.
>
>
> If nobody objects, I plan to create the first RC for 1.0.2 next
> Thursday. Please find a summary in the wiki:
>
> https://cwiki.apache.org/confluence/display/KAFKA/Copy+of+Release+Plan+1.0.2
>
>
> Thanks a lot,
>   Matthias
>
>
>

Re: [DISCUSS] 0.10.2.2 bug fix release

[Ismael Juma]

Thanks for doing this Matthias, +1.

Ismael

On Thu, Jun 7, 2018 at 1:50 PM Matthias J. Sax 
wrote:

> Dear all,
>
> I want to propose a 0.10.2.2 bug fix release. 0.10.2.1 is over a year
> old and a couple of critical fixes are available for 0.10.2.2.
>
> Please find a list of all 24 resolved tickets here:
>
>
> https://issues.apache.org/jira/browse/KAFKA-6566?jql=project%20%3D%20KAFKA%20AND%20fixVersion%20%3D%200.10.2.2%20AND%20resolution%20!%3D%20Unresolved%20ORDER%20BY%20priority%20DESC%2C%20status%20DESC%2C%20updated%20DESC%20%20
>
> There are no open tickets with target version 0.10.2.2 at the moment. If
> there are any tickets you want to get included in 0.10.2.2 please let us
> know as soon as possible.
>
>
> If nobody objects, I plan to create the first RC for 0.10.2.2 next
> Thursday. Please find a summary in the wiki:
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+0.10.2.2
>
>
> Thanks a lot,
>   Matthias
>
>
>
>

[jira] [Resolved] (KAFKA-6919) Trogdor documentation points to wrong location for trogdor.sh

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-6919.

   Resolution: Fixed
Fix Version/s: 2.0.0

> Trogdor documentation points to wrong location for trogdor.sh
> -
>
> Key: KAFKA-6919
> URL: https://issues.apache.org/jira/browse/KAFKA-6919
> Project: Kafka
>  Issue Type: Bug
>Reporter: Koen De Groote
>Priority: Minor
> Fix For: 2.0.0
>
>
> On this page: https://github.com/apache/kafka/blob/trunk/TROGDOR.md
> To get test results, the command shown is: ./tests/bin/trogdor.sh client 
> --show-tasks localhost:8889
> However, trogdor.sh isn't in ./tests/bin, it's in ./bin/
> Documentation should be updated accordingly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-6996) ConsumerGroupCommand.scala uses deprecated components

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-6996.

Resolution: Duplicate

Duplicate of KAFKA-6884,

> ConsumerGroupCommand.scala uses deprecated components
> -
>
> Key: KAFKA-6996
> URL: https://issues.apache.org/jira/browse/KAFKA-6996
> Project: Kafka
>  Issue Type: Bug
>  Components: admin
>Affects Versions: 1.1.0
>Reporter: Zeynep Arikoglu
>Priority: Minor
>
> ConsumerGroupCommand.scala which is the only component to get the topic lags 
> still uses the deprecated components, such as SimpleConsumer and 
> ZkGroupTopicDirs have been deprecated with Kafka 0.11.0.0.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-6762) log-cleaner thread terminates due to java.lang.IllegalStateException

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-6762.

Resolution: Duplicate

Duplicate of KAFKA-6854.

> log-cleaner thread terminates due to java.lang.IllegalStateException
> 
>
> Key: KAFKA-6762
> URL: https://issues.apache.org/jira/browse/KAFKA-6762
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.0.0
> Environment: os: GNU/Linux 
> arch: x86_64 
> Kernel: 4.9.77 
> jvm: OpenJDK 1.8.0
>Reporter: Ricardo Bartolome
>Priority: Major
> Attachments: __consumer_offsets-9_.tar.xz
>
>
> We are experiencing some problems with kafka log-cleaner thread on Kafka 
> 1.0.0. We have planned to update this cluster to 1.1.0 by next week in order 
> to fix KAFKA-6683, but until then we can only confirm that it happens in 
> 1.0.0.
> log-cleaner thread crashes after a while with the following error:
> {code:java}
> [2018-03-28 11:14:40,199] INFO Cleaner 0: Beginning cleaning of log 
> __consumer_offsets-31. (kafka.log.LogCleaner)
> [2018-03-28 11:14:40,199] INFO Cleaner 0: Building offset map for 
> __consumer_offsets-31... (kafka.log.LogCleaner)
> [2018-03-28 11:14:40,218] INFO Cleaner 0: Building offset map for log 
> __consumer_offsets-31 for 16 segments in offset range [1612869, 14282934). 
> (kafka.log.LogCleaner)
> [2018-03-28 11:14:58,566] INFO Cleaner 0: Offset map for log 
> __consumer_offsets-31 complete. (kafka.log.LogCleaner)
> [2018-03-28 11:14:58,566] INFO Cleaner 0: Cleaning log __consumer_offsets-31 
> (cleaning prior to Tue Mar 27 09:25:09 GMT 2018, discarding tombstones prior 
> to Sat Feb 24 11:04:21 GMT 2018
> )... (kafka.log.LogCleaner)
> [2018-03-28 11:14:58,567] INFO Cleaner 0: Cleaning segment 0 in log 
> __consumer_offsets-31 (largest timestamp Fri Feb 23 11:40:54 GMT 2018) into 
> 0, discarding deletes. (kafka.log.LogClea
> ner)
> [2018-03-28 11:14:58,570] INFO Cleaner 0: Growing cleaner I/O buffers from 
> 262144bytes to 524288 bytes. (kafka.log.LogCleaner)
> [2018-03-28 11:14:58,576] INFO Cleaner 0: Growing cleaner I/O buffers from 
> 524288bytes to 112 bytes. (kafka.log.LogCleaner)
> [2018-03-28 11:14:58,593] ERROR [kafka-log-cleaner-thread-0]: Error due to 
> (kafka.log.LogCleaner)
> java.lang.IllegalStateException: This log contains a message larger than 
> maximum allowable size of 112.
> at kafka.log.Cleaner.growBuffers(LogCleaner.scala:622)
> at kafka.log.Cleaner.cleanInto(LogCleaner.scala:574)
> at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:459)
> at kafka.log.Cleaner.$anonfun$doClean$6(LogCleaner.scala:396)
> at kafka.log.Cleaner.$anonfun$doClean$6$adapted(LogCleaner.scala:395)
> at scala.collection.immutable.List.foreach(List.scala:389)
> at kafka.log.Cleaner.doClean(LogCleaner.scala:395)
> at kafka.log.Cleaner.clean(LogCleaner.scala:372)
> at 
> kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:263)
> at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:243)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:64)
> [2018-03-28 11:14:58,601] INFO [kafka-log-cleaner-thread-0]: Stopped 
> (kafka.log.LogCleaner)
> [2018-04-04 14:25:12,773] INFO The cleaning for partition 
> __broker-11-health-check-0 is aborted and paused (kafka.log.LogCleaner)
> [2018-04-04 14:25:12,773] INFO Compaction for partition 
> __broker-11-health-check-0 is resumed (kafka.log.LogCleaner)
> [2018-04-04 14:25:12,774] INFO The cleaning for partition 
> __broker-11-health-check-0 is aborted (kafka.log.LogCleaner)
> [2018-04-04 14:25:22,850] INFO Shutting down the log cleaner. 
> (kafka.log.LogCleaner)
> [2018-04-04 14:25:22,850] INFO [kafka-log-cleaner-thread-0]: Shutting down 
> (kafka.log.LogCleaner)
> [2018-04-04 14:25:22,850] INFO [kafka-log-cleaner-thread-0]: Shutdown 
> completed (kafka.log.LogCleaner)
> {code}
> What we know so far is:
>  * We are unable to reproduce it yet in a consistent manner.
>  * It only happens in the PRO cluster and not in the PRE cluster for the same 
> customer (which message payloads are very similar)
>  * Checking our Kafka logs, it only happened on the internal topics 
> *__consumer_offsets-**
>  * When we restart the broker process the log-cleaner starts working again 
> but it can take between 3 minutes and some hours to die again.
>  * We workaround it by temporary increasing the message.max.bytes and 
> replica.fetch.max.bytes 

[jira] [Resolved] (KAFKA-6965) log4j:WARN log messages printed when running kafka-console-producer OOB

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-6965.

Resolution: Not A Bug

> log4j:WARN log messages printed when running kafka-console-producer OOB
> ---
>
> Key: KAFKA-6965
> URL: https://issues.apache.org/jira/browse/KAFKA-6965
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 1.1.0
>Reporter: Yeva Byzek
>Priority: Major
>  Labels: newbie
>
> This error message is presented running `bin/kafka-console-producer` out of 
> the box.  
> {noformat}
> log4j:WARN No appenders could be found for logger 
> (kafka.utils.Log4jControllerRegistration$).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
> info.
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-6760) responses not logged properly in controller

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-6760.

   Resolution: Fixed
Fix Version/s: 1.1.1
   2.0.0

> responses not logged properly in controller
> ---
>
> Key: KAFKA-6760
> URL: https://issues.apache.org/jira/browse/KAFKA-6760
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 1.1.0
>Reporter: Jun Rao
>Assignee: Mickael Maison
>Priority: Major
>  Labels: newbie
> Fix For: 2.0.0, 1.1.1
>
>
> Saw the following logging in controller.log. We need to log the 
> StopReplicaResponse properly in KafkaController.
> [2018-04-05 14:38:41,878] DEBUG [Controller id=0] Delete topic callback 
> invoked for org.apache.kafka.common.requests.StopReplicaResponse@263d40c 
> (kafka.controller.K
> afkaController)
> It seems that the same issue exists for LeaderAndIsrResponse as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-6973) setting invalid timestamp causes Kafka broker restart to fail

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-6973.

   Resolution: Fixed
Fix Version/s: 2.0.0

> setting invalid timestamp causes Kafka broker restart to fail
> -
>
> Key: KAFKA-6973
> URL: https://issues.apache.org/jira/browse/KAFKA-6973
> Project: Kafka
>  Issue Type: Bug
>  Components: admin
>Affects Versions: 1.1.0
>Reporter: Paul Brebner
>Assignee: huxihx
>Priority: Critical
> Fix For: 2.0.0
>
>
> Setting timestamp to invalid value causes Kafka broker to fail upon startup. 
> E.g.
> ./kafka-topics.sh --create --zookeeper localhost --topic duck3 --partitions 1 
> --replication-factor 1 --config message.timestamp.type=boom
>  
> Also note that the docs says the parameter name is 
> log.message.timestamp.type, but this is silently ignored.
> This works with no error for the invalid timestamp value. But next time you 
> restart Kafka:
>  
> [2018-05-29 13:09:05,806] FATAL [KafkaServer id=0] Fatal error during 
> KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
> java.util.NoSuchElementException: Invalid timestamp type boom
> at org.apache.kafka.common.record.TimestampType.forName(TimestampType.java:39)
> at kafka.log.LogConfig.(LogConfig.scala:94)
> at kafka.log.LogConfig$.fromProps(LogConfig.scala:279)
> at kafka.log.LogManager$$anonfun$17.apply(LogManager.scala:786)
> at kafka.log.LogManager$$anonfun$17.apply(LogManager.scala:785)
> at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:221)
> at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:428)
> at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:428)
> at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
> at scala.collection.AbstractTraversable.map(Traversable.scala:104)
> at kafka.log.LogManager$.apply(LogManager.scala:785)
> at kafka.server.KafkaServer.startup(KafkaServer.scala:222)
> at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38)
> at kafka.Kafka$.main(Kafka.scala:92)
> at kafka.Kafka.main(Kafka.scala)
> [2018-05-29 13:09:05,811] INFO [KafkaServer id=0] shutting down 
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: referencing OffsetCheckpoint in downstream project

[Ismael Juma]

Hi Ted,

There are two such classes. The example you have is for the broker class,
not the Streams one.

Ismael

On Thu, 31 May 2018, 09:03 Ted Yu,  wrote:

> Hi,
> OffsetCheckpoint has been relocated
> to org.apache.kafka.streams.state.internals package.
>
> Does this mean that downstream project should no longer reference this
> class ?
>
> This is how the class is used (against Kafka 0.10.0.1 release) :
>
> // ensure that topic is removed from all cleaner offsets
> assert(servers.forall(server => topicAndPartitions.forall { tp =>
>   val checkpoints = server.getLogManager().logDirs.map { logDir =>
> new OffsetCheckpoint(new File(logDir,
> "cleaner-offset-checkpoint")).read()
>   }
>   checkpoints.forall(checkpointsPerLogDir =>
> !checkpointsPerLogDir.contains(tp))
> }), s"checkpoint for topic $topic still exists")
>
> Cheers
>

Re: [VOTE] KIP-266: Add TimeoutException for KafkaConsumer#position

[Ismael Juma]

Sounds good to me,

On Wed, May 30, 2018 at 12:40 PM Jason Gustafson  wrote:

> Perhaps one minute? That is the default used by the producer.
>
> -Jason
>
> On Wed, May 30, 2018 at 9:50 AM, Ismael Juma  wrote:
>
> > Option 1 sounds good to me provided that we can come up with a good
> > default. What would you suggest?
> >
> > Ismael
> >
> > On Wed, May 30, 2018 at 9:41 AM Jason Gustafson 
> > wrote:
> >
> > > Hi Everyone,
> > >
> > > There remains some inconsistency in the timeout behavior of the
> consumer
> > > APIs which do not accept a timeout. Some of them block forever (e.g.
> > > position()) and some of them use request.timeout.ms (e.g.
> > > parititonsFor()).
> > > I think we'd probably all agree that blocking forever is not useful
> > > behavior and using request.timeout.ms has always been a hack since it
> > > controls a separate concern. I think there are basically two options to
> > > address this:
> > >
> > > 1. We can add max.block.ms to match the producer and use it as the
> > default
> > > timeout when a timeout is not explicitly provided. This will fix the
> > > indefinite blocking behavior and avoid conflating request.timeout.ms.
> > > 2. We can deprecate the methods which don't accept a timeout.
> > >
> > > I'm leaning toward the first solution because I think we want to push
> > users
> > > to specifying timeouts through configuration rather than in code (Jay's
> > > original argument). I think the overloads are still useful for advanced
> > > usage (e.g. in kafka streams), but we should give users an easy option
> > with
> > > reasonable default behavior.
> > >
> > > If that sounds ok, I'd propose we add it to this KIP and fix it now.
> This
> > > gives users an easy way to get the benefit of the improvements from
> this
> > > KIP without changing any code.
> > >
> > > Thanks,
> > > Jason
> > >
> > >
> > >
> > >
> > > On Sun, May 13, 2018 at 7:58 PM, Richard Yu <
> yohan.richard...@gmail.com>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > With 3 binding votes and 6 non-binding, this KIP would be accepted.
> > > >
> > > > Thanks for participating.
> > > >
> > > > On Thu, May 10, 2018 at 2:35 AM, Edoardo Comar 
> > > wrote:
> > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > On 10 May 2018 at 10:29, zhenya Sun  wrote:
> > > > >
> > > > > > +1 non-binding
> > > > > >
> > > > > > > 在 2018年5月10日，下午5:19，Manikumar  写道：
> > > > > > >
> > > > > > > +1 (non-binding).
> > > > > > > Thanks.
> > > > > > >
> > > > > > > On Thu, May 10, 2018 at 2:33 PM, Mickael Maison <
> > > > > > mickael.mai...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > >> +1 (non binding)
> > > > > > >> Thanks
> > > > > > >>
> > > > > > >> On Thu, May 10, 2018 at 9:39 AM, Rajini Sivaram <
> > > > > > rajinisiva...@gmail.com>
> > > > > > >> wrote:
> > > > > > >>> Hi Richard, Thanks for the KIP.
> > > > > > >>>
> > > > > > >>> +1 (binding)
> > > > > > >>>
> > > > > > >>> Regards,
> > > > > > >>>
> > > > > > >>> Rajini
> > > > > > >>>
> > > > > > >>> On Wed, May 9, 2018 at 10:54 PM, Guozhang Wang <
> > > wangg...@gmail.com
> > > > >
> > > > > > >> wrote:
> > > > > > >>>
> > > > > > >>>> +1 from me, thanks!
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>> Guozhang
> > > > > > >>>>
> > > > > > >>>> On Wed, May 9, 2018 at 10:46 AM, Jason Gustafson <
> > > > > ja...@confluent.io>
> > > > > > >>>> wrote:
> > > > > > >>>>
> > > > > > >>>>> Thanks for 

Re: [VOTE] KIP-266: Add TimeoutException for KafkaConsumer#position

[Ismael Juma]

Option 1 sounds good to me provided that we can come up with a good
default. What would you suggest?

Ismael

On Wed, May 30, 2018 at 9:41 AM Jason Gustafson  wrote:

> Hi Everyone,
>
> There remains some inconsistency in the timeout behavior of the consumer
> APIs which do not accept a timeout. Some of them block forever (e.g.
> position()) and some of them use request.timeout.ms (e.g.
> parititonsFor()).
> I think we'd probably all agree that blocking forever is not useful
> behavior and using request.timeout.ms has always been a hack since it
> controls a separate concern. I think there are basically two options to
> address this:
>
> 1. We can add max.block.ms to match the producer and use it as the default
> timeout when a timeout is not explicitly provided. This will fix the
> indefinite blocking behavior and avoid conflating request.timeout.ms.
> 2. We can deprecate the methods which don't accept a timeout.
>
> I'm leaning toward the first solution because I think we want to push users
> to specifying timeouts through configuration rather than in code (Jay's
> original argument). I think the overloads are still useful for advanced
> usage (e.g. in kafka streams), but we should give users an easy option with
> reasonable default behavior.
>
> If that sounds ok, I'd propose we add it to this KIP and fix it now. This
> gives users an easy way to get the benefit of the improvements from this
> KIP without changing any code.
>
> Thanks,
> Jason
>
>
>
>
> On Sun, May 13, 2018 at 7:58 PM, Richard Yu 
> wrote:
>
> > Hi,
> >
> > With 3 binding votes and 6 non-binding, this KIP would be accepted.
> >
> > Thanks for participating.
> >
> > On Thu, May 10, 2018 at 2:35 AM, Edoardo Comar 
> wrote:
> >
> > > +1 (non-binding)
> > >
> > > On 10 May 2018 at 10:29, zhenya Sun  wrote:
> > >
> > > > +1 non-binding
> > > >
> > > > > 在 2018年5月10日，下午5:19，Manikumar  写道：
> > > > >
> > > > > +1 (non-binding).
> > > > > Thanks.
> > > > >
> > > > > On Thu, May 10, 2018 at 2:33 PM, Mickael Maison <
> > > > mickael.mai...@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> +1 (non binding)
> > > > >> Thanks
> > > > >>
> > > > >> On Thu, May 10, 2018 at 9:39 AM, Rajini Sivaram <
> > > > rajinisiva...@gmail.com>
> > > > >> wrote:
> > > > >>> Hi Richard, Thanks for the KIP.
> > > > >>>
> > > > >>> +1 (binding)
> > > > >>>
> > > > >>> Regards,
> > > > >>>
> > > > >>> Rajini
> > > > >>>
> > > > >>> On Wed, May 9, 2018 at 10:54 PM, Guozhang Wang <
> wangg...@gmail.com
> > >
> > > > >> wrote:
> > > > >>>
> > > >  +1 from me, thanks!
> > > > 
> > > > 
> > > >  Guozhang
> > > > 
> > > >  On Wed, May 9, 2018 at 10:46 AM, Jason Gustafson <
> > > ja...@confluent.io>
> > > >  wrote:
> > > > 
> > > > > Thanks for the KIP, +1 (binding).
> > > > >
> > > > > One small correction: the KIP mentions that close() will be
> > > > >> deprecated,
> > > >  but
> > > > > we do not want to do this because it is needed by the Closeable
> > > >  interface.
> > > > > We only want to deprecate close(long, TimeUnit) in favor of
> > > > > close(Duration).
> > > > >
> > > > > -Jason
> > > > >
> > > > > On Tue, May 8, 2018 at 12:43 AM, khaireddine Rezgui <
> > > > > khaireddine...@gmail.com> wrote:
> > > > >
> > > > >> +1
> > > > >>
> > > > >> 2018-05-07 20:35 GMT+01:00 Bill Bejeck :
> > > > >>
> > > > >>> +1
> > > > >>>
> > > > >>> Thanks,
> > > > >>> Bill
> > > > >>>
> > > > >>> On Fri, May 4, 2018 at 7:21 PM, Richard Yu <
> > > >  yohan.richard...@gmail.com
> > > > >>
> > > > >>> wrote:
> > > > >>>
> > > >  Hi all, I would like to bump this thread since discussion in
> > the
> > > >  KIP
> > > >  appears to be reaching its conclusion.
> > > > 
> > > > 
> > > > 
> > > >  On Thu, Mar 15, 2018 at 3:30 PM, Richard Yu <
> > > > >> yohan.richard...@gmail.com>
> > > >  wrote:
> > > > 
> > > > > Hi all,
> > > > >
> > > > > Since there does not seem to be too much discussion in
> > > > >> KIP-266, I
> > > > >> will
> > > > >>> be
> > > > > starting a voting thread.
> > > > > Here is the link to KIP-266 for reference:
> > > > >
> > > > > https://cwiki.apache.org/confluence/pages/viewpage.
> > > >  action?pageId=75974886
> > > > >
> > > > > Recently, I have made some updates to the KIP. To
> reiterate,
> > I
> > > >  have
> > > > > included KafkaConsumer's commitSync,
> > > > > poll, and committed in the KIP. (we will be adding to a
> > > > >>> TimeoutException
> > > > > to them as well, in a similar manner
> > > > > to what we will be doing for position())
> > > > >
> > > > > Thanks,
> > > > > Richard Yu
> > > > >
> > > > >
> > > > 
> > > > >>>
> > > > >>
> > > > >>
> > > > >>>

[jira] [Created] (KAFKA-6956) Use Java AdminClient in BrokerApiVersionsCommand

[Ismael Juma (JIRA)]

Ismael Juma created KAFKA-6956:
--

 Summary: Use Java AdminClient in BrokerApiVersionsCommand
 Key: KAFKA-6956
 URL: https://issues.apache.org/jira/browse/KAFKA-6956
 Project: Kafka
  Issue Type: Improvement
Reporter: Ismael Juma


The Scala AdminClient was introduced as a stop gap until we had an officially 
supported API. The Java AdminClient is the supported API so we should migrate 
all usages to it and remove the Scala AdminClient. This JIRA is for using the 
Java AdminClient in BrokerApiVersionsCommand. We would need to verify that the 
necessary APIs are available via the Java AdminClient.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-6955) Use Java AdminClient in DeleteRecordsCommand

[Ismael Juma (JIRA)]

Ismael Juma created KAFKA-6955:
--

 Summary: Use Java AdminClient in DeleteRecordsCommand
 Key: KAFKA-6955
 URL: https://issues.apache.org/jira/browse/KAFKA-6955
 Project: Kafka
  Issue Type: Improvement
Reporter: Ismael Juma


The Scala AdminClient was introduced as a stop gap until we had an officially 
supported API. The Java AdminClient is the supported API so we should migrate 
all usages to it and remove the Scala AdminClient. This JIRA is for using the 
Java AdminClient in DeleteRecordsCommand.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-6930) Update KafkaZkClient debug log

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-6930.

Resolution: Fixed

> Update KafkaZkClient debug log
> --
>
> Key: KAFKA-6930
> URL: https://issues.apache.org/jira/browse/KAFKA-6930
> Project: Kafka
>  Issue Type: Improvement
>  Components: core, zkclient
>Affects Versions: 1.1.0
>Reporter: darion yaphet
>Priority: Trivial
> Attachments: [KAFKA-6930]_Update_KafkaZkClient_debug_log.patch, 
> snapshot.png
>
>
> Currently , KafkaZkClient could print data: Array[Byte] in debug log , we 
> should print data as String . 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] KIP-176: Remove deprecated new-consumer option for tools

[Ismael Juma]

+1 (binding).

Ismael

On Wed, 23 May 2018, 09:04 Paolo Patierno,  wrote:

> Sorry ... I hope it's not too late but I created the KIP-176 on September
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-176%3A+Remove+deprecated+new-consumer+option+for+tools
>
> but due to be a breaking change, I needed to wait for a major release ...
> and the right time is now.
> Can you vote for that adding to the release plan, please ?
>
> Thanks,
>
> Paolo Patierno
> Principal Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Azure & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno
> Linkedin : paolopatierno
> Blog : DevExperience
>

Re: [VOTE] KIP-235 Add DNS alias support for secured connection

[Ismael Juma]

Thanks for the KIP. I think this is a good and low risk change. It would be
good to ensure that it works well with KIP-302 if we think that makes sense
too. In any case, +1 (binding).

Ismael

On Fri, Mar 23, 2018 at 12:05 PM Skrzypek, Jonathan <
jonathan.skrzy...@gs.com> wrote:

> Hi,
>
> I would like to start a vote for KIP-235
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-235%3A+Add+DNS+alias+support+for+secured+connection
>
> This is a proposition to add an option for reverse dns lookup of
> bootstrap.servers hosts, allowing the use of dns aliases on clusters using
> SASL authentication.
>
>
>
>

Java 8 switch in trunk

[Ismael Juma]

Hi all,

We have switched to Java 8 in trunk. We can now use Java 8 for new code,
but I suggest we don't refactor existing code to use Java 8 features like
lambdas just yet. Let's focus on getting the 2.0.0 release out and after
the first bug fix, we can consider refactors that will make backports
harder.

Ismael

Re: [VOTE] KIP-303: Add Dynamic Routing Support in Kafka Streams' Topology Sink

[Ismael Juma]

Breaking API without a deprecation cycle doesn't seem good. Are we sure
about this?

Ismael

On Mon, 21 May 2018, 15:04 Guozhang Wang,  wrote:

> Hello Matthias,
>
> I've tried it out on the PR, the implementation should be fine but one
> concern I had is that, as you may also realize users of
> DynamicStreamPartitioner needs to implement two interface functions, with
> and without the topic name if it is extending from StreamPartitioner; we
> could also let it to not extend from StreamPartition so it has one function
> only but then we'd need Produced to have two functions allowing
> StreamPartitioner and DynamicStreamPartitioner. Thinking about the pros and
> cons I'm think it may be better to just change the interface of
> StreamPartitioner itself, since even without dynamic routing, allowing the
> topic name could let users to give one partitioner implementation that
> branch on the topic names other than having one partitioner per topic.
>
>
> Guozhang
>
>
> On Mon, May 21, 2018 at 11:56 AM, Matthias J. Sax 
> wrote:
>
> > I think that the risk of the change is moderate as I expect most people
> > to use the DefaultStreamPartitioner.
> >
> > However, there would still be possibility to define a new interface
> > instead of changing the old:
> >
> > > public interface DynamicStreamPartitioner {
> > > Integer partition(String topic, K key, V value, int numPartitions);
> > > }
> >
> > The newly added methods `Topology#addSink` and `KStream#to` would take
> > this new interface instead of the old.
> >
> > Maybe `DynamicStreamPartitioner` must extend `StreamPartitioner` to make
> > runtime code work though...
> >
> > WDYT?
> >
> > -Matthias
> >
> > On 5/21/18 11:47 AM, Guozhang Wang wrote:
> > > Hello everyone,
> > >
> > > While implementing the PR for this KIP I realized there is once place
> > which
> > > we should consider modifying on public APIs as well:
> > > StreamPartitioner#partition, to add the topic name string. Note it will
> > be
> > > a incompatible change that requires users who have customized
> > > StreamPartitioner implementations.
> > >
> > > I've updated the wiki page of KIP-303, please recast your vote on this
> > > thread. Thanks!!!
> > >
> > >
> > > Guozhang
> > >
> > >
> > > On Thu, May 17, 2018 at 3:15 PM, John Roesler 
> wrote:
> > >
> > >> +1 non-binding
> > >>
> > >> On Thu, May 17, 2018 at 4:44 PM, Matthias J. Sax <
> matth...@confluent.io
> > >
> > >> wrote:
> > >>
> > >>> +1 (binding)
> > >>>
> > >>>
> > >>> On 5/17/18 12:18 PM, Ted Yu wrote:
> >  +1
> >   Original message From: Gwen Shapira <
> > >> g...@confluent.io>
> > >>> Date: 5/17/18  11:53 AM  (GMT-08:00) To: dev 
> > >>> Subject: Re: [VOTE] KIP-303: Add Dynamic Routing Support in Kafka
> > >> Streams'
> > >>> Topology Sink
> >  Yay, its about time :)
> > 
> >  +1
> > 
> >  On Thu, May 17, 2018 at 12:38 PM, Guozhang Wang  >
> > >>> wrote:
> > 
> > > Hello folks,
> > >
> > > I'd like to start a voting thread on adding dynamic routing
> > >>> functionality
> > > in Streams sink node. Please find a KIP here:
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 303%3A+Add+Dynamic+Routing+in+Streams+Sink
> > >
> > >
> > > And the PR itself ready for review as well under KAFKA-4936:
> > >
> > > https://github.com/apache/kafka/pull/5018
> > >
> > >
> > >
> > > Thanks!
> > > -- Guozhang
> > >
> > 
> > 
> > 
> > >>>
> > >>>
> > >>
> > >
> > >
> > >
> >
> >
>
>
> --
> -- Guozhang
>

Re: [DISCUSS] KIP-290: Support for wildcard suffixed ACLs

[Ismael Juma]

It got 3 binding votes already and the deadline is only tomorrow. :)

Ismael

On Mon, 21 May 2018, 21:50 Colin McCabe,  wrote:

> On Mon, May 21, 2018, at 04:53, Andy Coates wrote:
> > Hey Piyush,
> >
> > Thanks for the updated KIP! Couple of minor points from me:
> >
> > When storing wildcard-suffixed Acls in ZK, drop the asterisk of the end
> for
> > the path, e.g. change "*/kafka-wildcard-acl/Topic/teamA*" * to "*/*
> > *kafka-wildcard-acl**/Topic/**teamA"*. This reduces error conditions,
> i.e.
> > this is a place for storing wildcard-suffixed Acls, so it implicitly ends
> > in an asterisk. If you include the asterisk in the path then you need to
> > validate each entry, when reading, ends with an asterisk, and do
> something
> > if they don't. If you don't include the training asterisk then there is
> > nothing to validate and you can treat the prefix as a literal, (i.e. no
> > escaping needed).  TBH I'd probably drop the asterisk from the in-memory
> > representation as well, for the same reason.
>
> Hi Andy,
>
> I agree.  If everything in ZK under /kafka-wildcard-acl/ is a prefix ACL,
> there is no need to include the star at the end.  And really, it should be
> called something like /kafka-prefix-acl/, since it's only vaguely related
> to the idea of wildcards.
>
> >
> > Rather than creating an enum to indicate the type of a resource, you
> could
> > instead use polymorphism, e.g. make Resource an interface and have two
> > implementations: LiteralResource and WildcardSuffixedResource.  This is
> > also extensible, but may also allow for a cleaner implementation.
>
> Since Resource is a concrete class now, we can't make it an interface
> without breaking API compatibility.
>
> Even if it were possible to do compatibly, I would argue it's a bad idea.
> If we need to add another bit of state like case insensitivity, we don't
> want to have LiteralCaseInsensistiveResource,
> WildcardSuffixedCaseInsensitiveResource, etc. etc.  You need 2^n subclasses
> classes to represent N bits of state.
>
> I would argue that there should be a field in Resource like NameType which
> can be LITERAL or PREFIX.  That leaves us in a good position when someone
> inevitably comes up with a new NameType.
>
> Does this still have a chance to get in, or has the KIP window closed?  I
> would argue with one or two minor changes it's ready to go.  Pretty much
> all of the compatibility problems are solved with the separate ZK hierarchy.
>
> best,
> Colin
>
> >
> > Andy
> >
> > On 21 May 2018 at 01:58, Rajini Sivaram  wrote:
> >
> > > Hi Piyush, Thanks for the KIP!
> > >
> > > +1 (binding)
> > >
> > > Regards,
> > >
> > > Rajini
> > >
> > > On Sun, May 20, 2018 at 2:53 PM, Andy Coates 
> wrote:
> > >
> > > > Awesome last minute effort Piyush.
> > > >
> > > > Really appreciate your time and input,
> > > >
> > > > Andy
> > > >
> > > > Sent from my iPhone
> > > >
> > > > > On 19 May 2018, at 03:43, Piyush Vijay 
> wrote:
> > > > >
> > > > > Updated the KIP.
> > > > >
> > > > > 1. New enum field 'ResourceNameType' in Resource and ResourceFilter
> > > > classes.
> > > > > 2. modify getAcls() and rely on ResourceNameType' field in
> Resource to
> > > > > return either exact matches or all matches based on
> wildcard-suffix.
> > > > > 3. CLI changes to identify if resource name is literal or
> > > wildcard-suffix
> > > > > 4. Escaping doesn't work and isn't required if we're keeping a
> separate
> > > > > path on ZK (kafka-wildcard-acl) to store wildcard-suffix ACLs.
> > > > > 5. New API keys for Create / Delete / Describe Acls request with a
> new
> > > > > field in schemas for 'ResourceNameType'.
> > > > >
> > > > > Looks ready to me for the vote, will start voting thread now.
> Thanks
> > > > > everyone for the valuable feedback.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Piyush Vijay
> > > > >
> > > > >
> > > > > Piyush Vijay
> > > > >
> > > > >> On Fri, May 18, 2018 at 6:07 PM, Andy Coates 
> > > wrote:
> > > > >>
> > > > >> Hi Piyush,
> > > > >>
> > > > >> We're fast approaching the KIP deadline. Are you actively working
> on
> > > > this?
> > > > >> If you're not I can take over.
> > > > >>
> > > > >> Thanks,
> > > > >>
> > > > >> Andy
> > > > >>
> > > > >>> On 18 May 2018 at 14:25, Andy Coates  wrote:
> > > > >>>
> > > > >>> OK I've read it now.
> > > > >>>
> > > > >>> 1. I see you have an example:
> > > >  For example: If I want to fetch all ACLs that match ’topicA*’,
> it’s
> > > > not
> > > > >>> possible without introducing new API AND maintaining backwards
> > > > >>> compatibility.
> > > > >>> getAcls takes a Resource, right, which would be either a full
> > > resource
> > > > >>> name or 'ALL', i.e. '*', right?  The point of the call is to get
> all
> > > > ACLs
> > > > >>> relating to a specific resource, not a partial resource like
> > > 'topicA*'.
> > > > >>> Currently, I'm guessing / half-remembering that if you ask it for
> > > ACLs
> > > > >> for
> > > > >>> topic 'foo' it doesn't include global 'ALL' AC

Re: KAFKA-6733 feedback

[Ismael Juma]

Hi Mateusz,

This is a good change, but it requires a KIP as mentioned in the PR. You
gave an example of a similar PR without a KIP, but there was a KIP for it:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-265%3A+Make+Windowed+Serde+to+public+APIs

Ismael

On Mon, May 21, 2018 at 4:00 PM Mateusz Zakarczemny 
wrote:

> Hi,
> Could I ask for some feedback regarding
> https://github.com/apache/kafka/pull/4807 ?
> It's waiting 1,5 month. I had to resolve conflicts with trunk couple of
> times.
> I would be grateful if someone could take a look it.
>
> Regards,
> Mateusz Zakarczemny
>

Re: [VOTE] KIP-306: Configuration for Delaying Response to Failed Client Authentication

[Ismael Juma]

Thanks for the KIP, +1 (binding).

Ismael

On Mon, May 21, 2018 at 7:52 AM Dhruvil Shah  wrote:

> Hi,
>
> I would like to start a vote on KIP-306 which proposes to add a
> configuration to delay responses to failed authentication.
>
> Link to the KIP:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-306%3A+Configuration+for+Delaying+Response+to+Failed+Client+Authentication
>
> Thanks,
> Dhruvil
>

[jira] [Reopened] (KAFKA-3665) Default ssl.endpoint.identification.algorithm should be https

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma reopened KAFKA-3665:


> Default ssl.endpoint.identification.algorithm should be https
> -
>
> Key: KAFKA-3665
> URL: https://issues.apache.org/jira/browse/KAFKA-3665
> Project: Kafka
>  Issue Type: Bug
>  Components: security
>Affects Versions: 0.9.0.1, 0.10.0.0
>    Reporter: Ismael Juma
>Assignee: Rajini Sivaram
>Priority: Major
> Fix For: 2.0.0
>
>
> The default `ssl.endpoint.identification.algorithm` is `null` which is not a 
> secure default (man in the middle attacks are possible).
> We should probably use `https` instead. A more conservative alternative would 
> be to update the documentation instead of changing the default.
> A paper on the topic (thanks to Ryan Pridgeon for the reference): 
> http://www.cs.utexas.edu/~shmat/shmat_ccs12.pdf



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-3665) Default ssl.endpoint.identification.algorithm should be https

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-3665.

Resolution: Fixed

> Default ssl.endpoint.identification.algorithm should be https
> -
>
> Key: KAFKA-3665
> URL: https://issues.apache.org/jira/browse/KAFKA-3665
> Project: Kafka
>  Issue Type: Bug
>  Components: security
>Affects Versions: 0.9.0.1, 0.10.0.0
>    Reporter: Ismael Juma
>Assignee: Rajini Sivaram
>Priority: Major
> Fix For: 2.0.0
>
>
> The default `ssl.endpoint.identification.algorithm` is `null` which is not a 
> secure default (man in the middle attacks are possible).
> We should probably use `https` instead. A more conservative alternative would 
> be to update the documentation instead of changing the default.
> A paper on the topic (thanks to Ryan Pridgeon for the reference): 
> http://www.cs.utexas.edu/~shmat/shmat_ccs12.pdf



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] KIP-277 - Fine Grained ACL for CreateTopics API

[Ismael Juma]

Thanks for the KIP, +1 (binding). Can you also please describe the
compatibility impact of changing the error code from
CLUSTER_AUTHORIZATION_FAILED to TOPIC_AUTHORIZATION_FAILED?

Ismael

On Wed, Apr 25, 2018 at 2:45 AM Edoardo Comar  wrote:

> Hi,
>
> The discuss thread on KIP-277 (
> https://www.mail-archive.com/dev@kafka.apache.org/msg86540.html )
> seems to have been fruitful and concerns have been addressed, please allow
> me start a vote on it:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-277+-+Fine+Grained+ACL+for+CreateTopics+API
>
> I will update the small PR to the latest KIP semantics if the vote passes
> (as I hope :-).
>
> cheers
> Edo
> --
>
> Edoardo Comar
>
> IBM Message Hub
>
> IBM UK Ltd, Hursley Park, SO21 2JN
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>

[jira] [Resolved] (KAFKA-6873) Broker is not returning data including requested offset

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-6873.

Resolution: Not A Problem

> Broker is not returning data including requested offset
> ---
>
> Key: KAFKA-6873
> URL: https://issues.apache.org/jira/browse/KAFKA-6873
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 1.1.0
> Environment: ubuntu
>Reporter: Adam Dratwinski
>Priority: Minor
>
> After upgrading Kafka to 1.1.0 from 0.9.x I experience issues related with 
> broker returning incomplete responses. This happens for my all log compacted 
> topics. I am using Golang client (Sarama).
> I debugged the issue and found that for some requests brokers return 
> FetchResponse with all messages having offsets lower then requested. For 
> example, I request for offset 1078831, I get FetchResponse with only one 
> message having offset 1078830, which produces missing blocks error. If I 
> request the next offset (1078832), then I get a block with many messages, 
> starting with much higher offset (e.g 1083813). There is a big gap in offsets 
> between these records, probably because I am using log compacted topics, but 
> all expected messages are there.
> Sarama client treats this as consumer error:
> {quote}kafka: response did not contain all the expected topic/partition blocks
> {quote}
> For build-in java client this issue is not happening. Looks like it is less 
> restrict regarding the data order, and when the offset is missing in the 
> returned block, it just simply request the next offset.
> I reported this issue at Shopify/sarama Github project (see 
> [https://github.com/Shopify/sarama/issues/1087)], where I got response, that 
> this seems to be Kafka bug, as according to the documentation, in this 
> situation broker should never return only messages having lower offsets.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-6923) Deprecate ExtendedSerializer/Serializer and ExtendedDeserializer/Deserializer

[Ismael Juma (JIRA)]

Ismael Juma created KAFKA-6923:
--

 Summary: Deprecate ExtendedSerializer/Serializer and 
ExtendedDeserializer/Deserializer
 Key: KAFKA-6923
 URL: https://issues.apache.org/jira/browse/KAFKA-6923
 Project: Kafka
  Issue Type: Improvement
  Components: clients
Reporter: Ismael Juma
 Fix For: 2.1.0


The Javadoc of ExtendedDeserializer states:

{code}
 * Prefer {@link Deserializer} if access to the headers is not required. Once 
Kafka drops support for Java 7, the
 * {@code deserialize()} method introduced by this interface will be added to 
Deserializer with a default implementation
 * so that backwards compatibility is maintained. This interface may be 
deprecated once that happens.
{code}

Since we have dropped Java 7 support, we should figure out how to do this. 
There are compatibility implications, so a KIP is needed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-6921) Remove old Scala producer and all related code, tests, and tools

[Ismael Juma (JIRA)]

Ismael Juma created KAFKA-6921:
--

 Summary: Remove old Scala producer and all related code, tests, 
and tools
 Key: KAFKA-6921
 URL: https://issues.apache.org/jira/browse/KAFKA-6921
 Project: Kafka
  Issue Type: Improvement
Reporter: Ismael Juma
Assignee: Ismael Juma
 Fix For: 2.0.0






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] KIP-282: Add the listener name to the authentication context

[Ismael Juma]

Thanks for the KIP, +1 (binding).

Ismael

On Wed, Apr 25, 2018 at 1:52 AM Mickael Maison 
wrote:

> Hi,
>
> There has been no objections in the DISCUSS thread so I'd like to
> start a vote on KIP-282:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-282%3A+Add+the+listener+name+to+the+authentication+context
>
> Thanks
>

Re: [VOTE] KIP-283: Efficient Memory Usage for Down-Conversion

[Ismael Juma]

Thanks for the KIP Dhruvil, this is a welcome improvement! My understanding
is that you have done some work to validate that the change has the desired
effect, it would be good to include that information in the "Testing
Strategy" section.

+1 (binding)

Ismael

On Wed, May 2, 2018 at 9:27 AM Dhruvil Shah  wrote:

> Hi all,
>
> I would like to start the vote on KIP-238: Efficient Memory Usage for
> Down-Conversion.
>
> For reference, the link to the KIP is here:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-283%3A+Efficient+Memory+Usage+for+Down-Conversion
>
> and the discussion thread is here:
> https://www.mail-archive.com/dev@kafka.apache.org/msg86799.html
>
> Thanks,
> Dhruvil
>

[jira] [Resolved] (KAFKA-5907) Support aggregatedJavadoc in Java 9

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-5907.

Resolution: Fixed

[~omkreddy] It does seem to work now, so will mark it as resolved.

> Support aggregatedJavadoc in Java 9
> ---
>
> Key: KAFKA-5907
> URL: https://issues.apache.org/jira/browse/KAFKA-5907
> Project: Kafka
>  Issue Type: Improvement
>    Reporter: Ismael Juma
>Priority: Major
> Fix For: 2.0.0
>
>
> The Java 9 Javadoc tool has some improvements including a search bar. 
> However, it currently fails with a number of errors like:
> {code}
> > Task :aggregatedJavadoc
> /Users/ijuma/src/kafka/streams/src/main/java/org/apache/kafka/streams/Topology.java:29:
>  error: package org.apache.kafka.streams.processor.internals does not exist
> import org.apache.kafka.streams.processor.internals.InternalTopologyBuilder;
>^
> /Users/ijuma/src/kafka/streams/src/main/java/org/apache/kafka/streams/Topology.java:30:
>  error: package org.apache.kafka.streams.processor.internals does not exist
> import org.apache.kafka.streams.processor.internals.ProcessorNode;
>^
> /Users/ijuma/src/kafka/streams/src/main/java/org/apache/kafka/streams/Topology.java:31:
>  error: package org.apache.kafka.streams.processor.internals does not exist
> import org.apache.kafka.streams.processor.internals.ProcessorTopology;
>^
> /Users/ijuma/src/kafka/streams/src/main/java/org/apache/kafka/streams/Topology.java:32:
>  error: package org.apache.kafka.streams.processor.internals does not exist
> import org.apache.kafka.streams.processor.internals.SinkNode;
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] KIP-235 Add DNS alias support for secured connection

[Ismael Juma]

Thanks for the KIP, Jonathan. It would be helpful to have more detail on
how SSL authentication could be broken if the new behaviour is the default.
I know this was discussed in the mailing list thread, but it's important to
include it in the KIP since it's the main reason why a new config is needed
(and configs should be avoided whenever we can just do the right thing).

Ismael

On Fri, Mar 23, 2018 at 12:05 PM Skrzypek, Jonathan <
jonathan.skrzy...@gs.com> wrote:

> Hi,
>
> I would like to start a vote for KIP-235
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-235%3A+Add+DNS+alias+support+for+secured+connection
>
> This is a proposition to add an option for reverse dns lookup of
> bootstrap.servers hosts, allowing the use of dns aliases on clusters using
> SASL authentication.
>
>
>
>

Re: [VOTE] KIP-281: ConsumerPerformance: Increase Polling Loop Timeout and Make It Reachable by the End User

[Ismael Juma]

Thanks for the KIP, +1 (binding). A few suggestions:

1. We normally include the time unit in configs. Not sure if we do it for
command line parameters though, so can we please verify and make it
consistent?
2. The KIP mentions --polling-loop-timeout and --timeout. Which is it?
3. Can we include the description of the new parameter in the KIP? In the
PR it says "Consumer polling loop timeout", but I think this is a bit
unclear. What are we actually measuring here?

Ismael

On Mon, Apr 16, 2018 at 2:25 PM Alex Dunayevsky 
wrote:

> Hello friends,
>
> Let's start the vote for KIP-281: ConsumerPerformance: Increase Polling
> Loop Timeout and Make It Reachable by the End User:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-281%3A+ConsumerPerformance%3A+Increase+Polling+Loop+Timeout+and+Make+It+Reachable+by+the+End+User
>
> Thank you,
> Alexander Dunayevsky
>

Re: [VOTE] KIP-294 - Enable TLS hostname verification by default

[Ismael Juma]

Thanks for the KIP, +1 (binding) from me.

Ismael

On Wed, May 9, 2018 at 8:29 AM Rajini Sivaram 
wrote:

> Hi all,
>
> Since there have been no objections on this straightforward KIP, I would
> like to initiate the voting process. KIP-294 proposes to use a secure
> default value for endpoint identification when using SSL as the security
> protocol. The KIP Is here:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-294+-+Enable+TLS+hostname+verification+by+default
>
> If there are any concerns, please add them to this thread or the discussion
> thread (https://www.mail-archive.com/dev@kafka.apache.org/msg87549.html)
>
> Regards,
>
> Rajini
>

[jira] [Resolved] (KAFKA-6828) Index files are no longer sparse in Java 9/10 due to OpenJDK regression

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-6828.

Resolution: Fixed

Great! Let's close this then.

> Index files are no longer sparse in Java 9/10 due to OpenJDK regression
> ---
>
> Key: KAFKA-6828
> URL: https://issues.apache.org/jira/browse/KAFKA-6828
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 1.0.0
> Environment: CentosOS 7 on EXT4 FS
>Reporter: Enrico Olivelli
>Priority: Critical
>
> This is a very strage case. I have a Kafka broker (part of a cluster of 3 
> brokers) which cannot start upgrading Java from Oracle JDK8 to Oracle JDK 
> 9.0.4 (the same with JDK 10.0.0)
> There are a lot of .index and .timeindex files taking 10MB, they are for 
> empty partiions.
> Running with Java 9 the server seems to rebuild these files and each file 
> takes "really" 10MB.The sum of all the files (calculated using du -sh) is 
> 22GB and the broker crashes during startup, disk becomes full and no log more 
> is written. (I can send an extraction of the logs, but the tell only  about 
> 'rebuilding index', the same as on Java 8)
> Reverting the same broker to Java 8 and removing the index files, the broker 
> rebuilds such files, each files take 10MB, but the full sum of sizes 
> (calculated using du -sh) is 38 MB !
> I am running this broker on CentosOS 7 on EXT4 FS.
> I have upgraded the broker to latest and greatest Kafka 1.0.0 (from 0.10.2) 
> without any success.
>   
>  After checking on JDK nio-dev list it appears a regresion in the behaviour 
> of RandomAccessFile
>   Just for reference see this discussion  on nio-dev list on OpenJDK
>  [http://mail.openjdk.java.net/pipermail/nio-dev/2018-April/005008.html]
> see
>  [https://bugs.openjdk.java.net/browse/JDK-8168628]
>   
>   
>   



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Use of a formatter like Scalafmt

[Ismael Juma]

Sounds good about doing this for Kafka streams scala first. Core is a bit
more complicated so may require more discussion.

Ismael

On Wed, 9 May 2018, 16:59 Matthias J. Sax,  wrote:

> Joan,
>
> thanks for starting this initiative. I am overall +1
>
> However, I am worried about applying it to `core` module as the change
> is massive. For the Kafka Streams Scala module, that is new and was not
> released yet, I am +1.
>
> A personal thing about the style: the 2-space indention for JavaDocs is
> a little weird.
>
> /**
>  *
>  */
>
> is changed to
>
> /**
>   *
>   */
>
> Not sure if this can be fixed easily in the style file? If not, I am
> also fine with the change.
>
> This change also affect the license headers of many files and exposing
> that those use the wrong comment format anyway. They should use regular
> comments
>
> /*
>  *
>  */
>
> but not JavaDoc comments
>
> /**
>  *
>  */
>
> (We fixed this for Java source code in the past already -- maybe it's
> time to fix it for Scala code base, too.
>
>
>
> -Matthias
>
> On 5/9/18 4:45 PM, Joan Goyeau wrote:
> > Hi Ted,
> >
> > As far as I understand this is an issue for PRs and back porting the
> > changes to other branches.
> >
> > Applying the tool to the other branches should also resolve the conflicts
> > as the formattings will match, leaving only the actual changes in the
> diffs.
> > That's what we did sometime ago at my work and it went quiet smoothly.
> >
> > If we don't want to do a big bang commit then I'm thinking we might want
> to
> > make it gradually by applying it module by module?
> > This is one idea do you have any other?
> >
> > I know formatting sounds like the useless thing that doesn't matter and I
> > totally agree with this, that's why I don't want to care about it while
> > coding.
> >
> > Thanks
> >
> > On Thu, 10 May 2018 at 00:15 Ted Yu  wrote:
> >
> >> Applying the tool across code base would result in massive changes.
> >> How would this be handled ?
> >>  Original message From: Joan Goyeau 
> >> Date: 5/9/18  3:31 PM  (GMT-08:00) To: dev@kafka.apache.org Subject:
> Use
> >> of a formatter like Scalafmt
> >> Hi,
> >>
> >> Contributing to Kafka Streams' Scala API, I've been kinda lost on how
> >> should I format my code.
> >> I know formatting is the start of religion wars but personally I have no
> >> preference at all. I just want consistency across the codebase, no
> >> unnecessary formatting diffs in PRs and offload the formatting to a tool
> >> that will do it for me and concentrate on what matters (not formatting).
> >>
> >> So I opened the following PR where I put arbitrary rules in
> .scalafmt.conf
> >> <
> >>
> https://github.com/apache/kafka/pull/4965/files#diff-8af3e1355c23c331ee2b848e12c5219f
> >>>
> >> :
> >> https://github.com/apache/kafka/pull/4965
> >>
> >> Please let me know what do you think and if we can move this forward and
> >> settle something.
> >>
> >> Thanks
> >>
> >
>
>

Re: [VOTE] KIP-283: Efficient Memory Usage for Down-Conversion

[Ismael Juma]

Maybe it should message instead of record to be consistent with
message.format.version.

On Wed, 9 May 2018, 09:04 Jason Gustafson,  wrote:

> Hi Dhruvil,
>
> Thanks for the KIP. +1 from me. Just a minor nitpick on the name of the new
> config. I would suggest "record.downconversion.enable". The "record" prefix
> emphasizes what is being down-converted and similar existing configs use
> "enable" rather than "enabled."
>
> -Jason
>
> On Wed, May 2, 2018 at 9:35 AM, Ted Yu  wrote:
>
> > +1
> >
> > On Wed, May 2, 2018 at 9:27 AM, Dhruvil Shah 
> wrote:
> >
> > > Hi all,
> > >
> > > I would like to start the vote on KIP-238: Efficient Memory Usage for
> > > Down-Conversion.
> > >
> > > For reference, the link to the KIP is here:
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 283%3A+Efficient+Memory+Usage+for+Down-Conversion
> > >
> > > and the discussion thread is here:
> > > https://www.mail-archive.com/dev@kafka.apache.org/msg86799.html
> > >
> > > Thanks,
> > > Dhruvil
> > >
> >
>

[jira] [Resolved] (KAFKA-6879) Controller deadlock following session expiration

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-6879.

Resolution: Fixed

> Controller deadlock following session expiration
> 
>
> Key: KAFKA-6879
> URL: https://issues.apache.org/jira/browse/KAFKA-6879
> Project: Kafka
>  Issue Type: Bug
>  Components: controller
>Affects Versions: 1.1.0
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Critical
> Fix For: 2.0.0, 1.1.1
>
>
> We have observed an apparent deadlock situation which occurs following a 
> session expiration. The suspected deadlock occurs between the zookeeper 
> "initializationLock" and the latch inside the Expire event which we use to 
> ensure all events have been handled.
> In the logs, we see the "Session expired" message following acquisition of 
> the initialization lock: 
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/zookeeper/ZooKeeperClient.scala#L358
> But we never see any logs indicating that the new session is being 
> initialized. In fact, the controller logs are basically empty from that point 
> on. The problem we suspect is that completion of the 
> {{beforeInitializingSession}} callback requires that all events have finished 
> processing in order to count down the latch: 
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/controller/KafkaController.scala#L1525.
> But an event which was dequeued just prior to the acquisition of the write 
> lock may be unable to complete because it is awaiting acquisition of the 
> initialization lock: 
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/zookeeper/ZooKeeperClient.scala#L137.
> The impact is that the broker continues in a zombie state. It continues 
> fetching and is periodically added to ISRs, but it never receives any further 
> requests from the controller since it is not registered.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-6390) Update ZooKeeper to 3.4.11, Gradle and other minor updates

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-6390.

Resolution: Fixed

> Update ZooKeeper to 3.4.11, Gradle and other minor updates
> --
>
> Key: KAFKA-6390
> URL: https://issues.apache.org/jira/browse/KAFKA-6390
> Project: Kafka
>  Issue Type: Bug
>    Reporter: Ismael Juma
>    Assignee: Ismael Juma
>Priority: Major
> Fix For: 2.0.0
>
>
> https://issues.apache.org/jira/browse/ZOOKEEPER-2614 is a helpful fix.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Reopened] (KAFKA-4041) kafka unable to reconnect to zookeeper behind an ELB

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma reopened KAFKA-4041:


> kafka unable to reconnect to zookeeper behind an ELB
> 
>
> Key: KAFKA-4041
> URL: https://issues.apache.org/jira/browse/KAFKA-4041
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.1, 0.10.0.1
> Environment: RHEL EC2 instances
>Reporter: prabhakar
>Priority: Blocker
>
> Kafka brokers are unable to connect to  zookeeper which is behind an ELB.
> Kafka is using zkClient which is caching the IP address of zookeeper  and 
> even when there is a change in the IP for zookeeper it is using the Old 
> zookeeper IP.
> The server.properties has a DNS name. Ideally kafka should resolve the IP 
> using the DNS in case of any failures connecting to the broker.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Java 10 replacing Java 9 in Jenkins for trunk

[Ismael Juma]

Hi Ted,

We are in the process of updating the system tests infrastructure so that
it works with Java 8. Once that happens, we will switch the build to use
Java 8.

Ismael

On Sat, 5 May 2018, 19:57 Ted Yu,  wrote:

> In PR build, I noticed the following (
> https://builds.apache.org/job/kafka-pr-jdk10-scala2.12/622/console) :
>
> *02:32:11*
> :clients:compileJava/home/jenkins/jenkins-slave/workspace/kafka-pr-jdk10-scala2.12/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java:1262:
> error: lambda expressions are not supported in -source 7*02:32:12*
> client.poll(pollTimeout, nowMs, () -> {*02:32:12*
>   ^*02:32:12*   (use -source 8 or higher to
> enable lambda expressions)*02:32:12* 1 error
>
>
> Could the above be due to the following in build.gradle :
>
> if (JavaVersion.current().isJava9Compatible())
>   options.compilerArgs << "--release" << "7"
>
> If so, we need to adjust it.
>
> Cheers
>
> On Mon, Apr 9, 2018 at 9:46 AM, Ismael Juma  wrote:
>
> > Hi all,
> >
> > Java 10 was recently released and support for Java 9 has ended since it's
> > not a LTS release. I've added a kafka-trunk Jenkins job for Java 10 and
> > disabled the Java 9 job. I also added a PR Jenkins job for Java 10 and
> will
> > soon disable the Java 9 PR job.
> >
> > The general idea is to have a separate Jenkins job for the latest non LTS
> > release (Java 10) and all supported LTS releases (Java 8 and Java 7
> > currently, soon to become Java 8 only).
> >
> > Let me know if you have any questions or concerns.
> >
> > Ismael
> >
>

[jira] [Resolved] (KAFKA-6853) ResponseMetadata calculates latency incorrectly (and therefore ZooKeeperRequestLatencyMs is incorrect)

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-6853.

Resolution: Fixed

> ResponseMetadata calculates latency incorrectly (and therefore 
> ZooKeeperRequestLatencyMs is incorrect)
> --
>
> Key: KAFKA-6853
> URL: https://issues.apache.org/jira/browse/KAFKA-6853
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.1.0
>Reporter: Fuud
>Priority: Minor
> Fix For: 2.0.0, 1.1.1
>
>
> responseTimeMs always negative.
> Currently:
> {code}
> case class ResponseMetadata(sendTimeMs: Long, receivedTimeMs: Long) {
>   def responseTimeMs: Long = sendTimeMs - receivedTimeMs
> }
> {code}
> Should be:
> {code}
> case class ResponseMetadata(sendTimeMs: Long, receivedTimeMs: Long) {
>   def responseTimeMs: Long = receivedTimeMs - sendTimeMs
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-6855) Kafka fails to start due to faulty Java version detection

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-6855.

   Resolution: Fixed
Fix Version/s: 1.1.1
   2.0.0

Thanks for the report. This has already been fixed:

[https://github.com/apache/kafka/commit/e9f86c3085fa8b65e77072389e0dd147b744f117]

Since we had no JIRA for it, I will use this one.

> Kafka fails to start due to faulty Java version detection
> -
>
> Key: KAFKA-6855
> URL: https://issues.apache.org/jira/browse/KAFKA-6855
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.1.0
> Environment: Ubuntu 18.04
> Java 10
>Reporter: Anders Tornblad
>Assignee: Ismael Juma
>Priority: Major
> Fix For: 2.0.0, 1.1.1
>
>
> After downloading fresh installations of ZooKeeper and Kafka, and then 
> starting ZooKeeper and Kafka the way that is recommended on 
> [http://kafka.apache.org/documentation/#quickstart] the following error 
> message is shown:
> {{Unrecognized VM option 'PrintGCDateStamps'}}
> I found the error in the kafka-run-class.sh file, where the Java version is 
> determined and put in the JAVA_MAJOR_VERSION variable. My Java runtime 
> reports the version as openjdk version "10.0.1" 2018-04-17, which makes the 
> JAVA_MAJOR_VERSION value be "10 2018-04-17" instead of just "10". That makes 
> the subsequent if statement fail.
> I found the following line to fix the problem:
> {{JAVA_MAJOR_VERSION=$($JAVA -version 2>&1 | sed -E -n 's/.* version 
> "([^.-]*).*/\1p')}}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] KIP-282: Add the listener name to the authentication context

[Ismael Juma]

ListenerName is in an internal package, so we would have to move it before
exposing it in a public API.

Ismael

On Wed, 25 Apr 2018, 01:52 Mickael Maison,  wrote:

> Hi,
>
> There has been no objections in the DISCUSS thread so I'd like to
> start a vote on KIP-282:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 282%3A+Add+the+listener+name+to+the+authentication+context
>
> Thanks
>

Re: [DISCUSS] KIP-219 - Improve Quota Communication

[Ismael Juma]

Hi Jon,

Not sure about this approach. It's worth mentioning this in the vote thread
as well so that the people who voted originally have a chance to comment.
Also, we should really get input from developers of Kafka clients
(librdkafka, kafka-python, etc.) for this KIP.

Ismael

On Thu, Apr 5, 2018 at 2:50 PM, Jonghyun Lee  wrote:

> Hi,
>
> I have been implementing KIP-219. I discussed the interface changes with
> Becket Qin and Dong Lin, and we decided to bump up the protocol version of
> ApiVersionsRequest and ApiVersionsResponse only, instead of bumping up all
> requests/responses that may be throttled, to indicate clients whether or
> not brokers perform throttling before sending out responses. We think this
> is sufficient given that network client exchanges
> ApiVersionsRequest/Response with each broker when establishing connection
> to it and thus it can detect the broker's throttling behavior just by
> examining the ApiVersionsResponse version.
>
> Please respond to this e-mail if you have any questions or concerns.
>
> Thanks,
> Jon Lee
>
>
> On Thu, Apr 5, 2018 at 2:29 PM, Becket Qin  wrote:
>
> >
> >
> > On Thu, Nov 16, 2017 at 3:49 PM, Becket Qin 
> wrote:
> >
> >> Thanks Rajini,
> >>
> >> I updated the KIP wiki to clarify the scope of the KIP. To summarize,
> the
> >> current quota has a few caveats:
> >> 1. The brokers are only throttling the NEXT request even if the current
> >> request is already violating quota. So the broker will always process at
> >> least one request from the client.
> >> 2. The brokers are not able to know the client id until they fully read
> >> the request out of the sockets even if that client is being throttled.
> >> 3. The brokers are not communicating to the clients promptly, so the
> >> clients have to blindly wait and sometimes times out unnecessarily.
> >>
> >> This KIP only tries to address 3 but not 1 and 2 because A) those two
> >> issues are sort of orthogonal to 3 and B) the solution to 1 and 2 are
> much
> >> more complicated and worth a separate discussion.
> >>
> >> I'll wait till tomorrow and start a voting thread if there are further
> >> concerns raised about the KIP.
> >>
> >> Thanks,
> >>
> >> Jiangjie (Becket) Qin
> >>
> >> On Thu, Nov 16, 2017 at 11:47 AM, Rajini Sivaram <
> rajinisiva...@gmail.com
> >> > wrote:
> >>
> >>> Hi Becket,
> >>>
> >>> The current user quota doesn't solve the problem. But I was thinking
> that
> >>> if we could ensure we don't read more from the network than the quota
> >>> allows, we may be able to fix the issue in a different way (throttling
> >>> all
> >>> connections, each for a limited time prior to reading large requests).
> >>> But
> >>> it would be more complex (and even more messy for client-id quotas),
> so I
> >>> can understand why the solution you proposed in the KIP makes sense for
> >>> the
> >>> scenario that you described.
> >>>
> >>> Regards,
> >>>
> >>> Rajini
> >>>
> >>> On Tue, Nov 14, 2017 at 11:30 PM, Becket Qin 
> >>> wrote:
> >>>
> >>> > Hi Rajini,
> >>> >
> >>> > We are using SSL so we could use user quota. But I am not sure if
> that
> >>> > would solve the problem. The key issue in our case is that each
> broker
> >>> can
> >>> > only handle ~300 MB/s of incoming bytes, but the MapReduce job is
> >>> trying to
> >>> > push 1-2 GB/s, unless we can throttle the clients to 300 MB/s, the
> >>> broker
> >>> > cannot sustain. In order to do that, we need to be able to throttle
> >>> > requests for more than request timeout, potentially a few minutes. It
> >>> seems
> >>> > neither user quota nor limited throttle time can achieve this.
> >>> >
> >>> > Thanks,
> >>> >
> >>> > Jiangjie (Becket) Qin
> >>> >
> >>> > On Tue, Nov 14, 2017 at 7:44 AM, Rajini Sivaram <
> >>> rajinisiva...@gmail.com>
> >>> > wrote:
> >>> >
> >>> > > Hi Becket,
> >>> > >
> >>> > > For the specific scenario that you described, would it be possible
> >>> to use
> >>> > > user quotas rather than client-id quotas? With user quotas, perhaps
> >>> we
> >>> > can
> >>> > > throttle more easily before reading requests as well (as you
> >>> mentioned,
> >>> > the
> >>> > > difficulty with client-id quota is that we have to read partial
> >>> requests
> >>> > > and handle client-ids at network layer making that a much bigger
> >>> change).
> >>> > > If your clients are using SASL/SSL, I was wondering whether a
> >>> solution
> >>> > that
> >>> > > improves user quotas and limits throttle time would work for you.
> >>> > >
> >>> > > Regards,
> >>> > >
> >>> > > Rajini
> >>> > >
> >>> > >
> >>> > >
> >>> > > On Thu, Nov 9, 2017 at 12:59 AM, Becket Qin 
> >>> > wrote:
> >>> > >
> >>> > > > Since we will bump up the wire request version, another option is
> >>> for
> >>> > > > clients that are sending old request versions the broker can just
> >>> keep
> >>> > > the
> >>> > > > current behavior. For clients sending the new request versions,
> the
> >>> > > broker
> >>> > > > can respond then mute the channel as described in the KIP wik

[jira] [Resolved] (KAFKA-6796) Surprising UNKNOWN_TOPIC error for produce/fetch requests to non-replicas

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-6796.

   Resolution: Fixed
Fix Version/s: 1.2.0

> Surprising UNKNOWN_TOPIC error for produce/fetch requests to non-replicas
> -
>
> Key: KAFKA-6796
> URL: https://issues.apache.org/jira/browse/KAFKA-6796
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 1.1.0, 1.0.1
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Major
> Fix For: 1.2.0
>
>
> Currently if the client sends a produce request or a fetch request to a 
> broker which isn't a replica, we return UNKNOWN_TOPIC_OR_PARTITION. This is a 
> bit surprising to see when the topic actually exists. It would be better to 
> return NOT_LEADER to avoid confusion. Clients typically handle both errors by 
> refreshing metadata and retrying, so changing this should not cause any 
> change in behavior on the client. This case can be hit following a partition 
> reassignment after the leader is moved and the local replica is deleted.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] Kafka 2.0.0 in June 2018

[Ismael Juma]

Thanks to everyone who voted. The vote passed with the following votes:

* 6 binding +1s: Rajini, Gwen, Jason, Matthias, Guozhang, myself
* 16 non-binding +1s: Ted, Edoardo, Jakub, Bill, Rahul, Mickael, Stephane,
Manikumar, Jorge, Vahid, Tao, Matt, Sandor, Zhenya, Viktor, Thomas
* No +0, -0 or -1 votes

On Wed, Apr 18, 2018 at 11:35 AM, Ismael Juma  wrote:

> Hi all,
>
> I started a discussion last year about bumping the version of the June
> 2018 release to 2.0.0[1]. To reiterate the reasons in the original post:
>
> 1. Adopt KIP-118 (Drop Support for Java 7), which requires a major version
> bump due to semantic versioning.
>
> 2. Take the chance to remove deprecated code that was deprecated prior to
> 1.0.0, but not removed in 1.0.0 (e.g. old Scala clients) so that we can
> move faster.
>
> One concern that was raised is that we still do not have a rolling upgrade
> path for the old ZK-based consumers. Since the Scala clients haven't been
> updated in a long time (they don't support security or the latest message
> format), users who need them can continue to use 1.1.0 with no loss of
> functionality.
>
> Since it's already mid-April and people seemed receptive during the
> discussion last year, I'm going straight to a vote, but we can discuss more
> if needed (of course).
>
> Ismael
>
> [1] https://lists.apache.org/thread.html/dd9d3e31d7e9590c1f727ef5560c93
> 3281bad0de3134469b7b3c4257@%3Cdev.kafka.apache.org%3E
>

Re: [DISCUSS] KIP-278: Add version option to Kafka's commands

[Ismael Juma]

FYI, the injection via the build process that is mentioned here already
happens. See AppInfoParser.

Ismael

On Mon, Apr 23, 2018 at 9:39 AM, Colin McCabe  wrote:

> Hi Sasaki,
>
> Thanks for the KIP.  I think a version flag is a good idea.
>
> Can you give a little more detail about what would be displayed when the
> version command was used?
>
> We clearly want the version number, but we probably also want to know if
> this is an official release, or a random SNAPSHOT from a branch.  If this
> is a release candidate, we probably want the RC number as well, like
> "1.1-rc3"  We also want a git hash.  This can be injected by the build
> process.  In the case of an official release, where the source code is not
> under git, we can pull it from a file.
>
> For example, hadoop's version output looks like this:
>
>  > cmccabe@aurora:~/Downloads/hadoop-2.8.3> ./bin/hadoop version
>  > Hadoop 2.8.3
>  > Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r
> b3fe56402d908019d99af1f1f4fc65cb1d1436a2
>  > Compiled by jdu on 2017-12-05T03:43Z
>  > Compiled with protoc 2.5.0
>  > From source with checksum 9ff4856d824e983fa510d3f843e3f19d
>  > This command was run using /home/cmccabe/Downloads/
> hadoop-2.8.3/share/hadoop/common/hadoop-common-2.8.3.jar
>
> (The "subversion" line here is a little weird -- it now refers to git, not
> svn)
>
> On Wed, Apr 11, 2018, at 13:58, Jason Gustafson wrote:
> > Hey Sasaki,
> >
> > Yeah, I don't feel too strongly about only supporting --version. I agree
> it
> > may help discoverability given the current approach. On the other hand,
> if
> > we refactored all of the tools so that we could use a common set of base
> > options, it might be a little annoying to have to continue supporting
> both
> > variations. For example, tool standardization was proposed in KIP-14 and
> > I'm still holding out hope that someone will have time to pick this work
> > back up. It's always easier to add an option than remove one, so I'm
> > slightly inclined to have only --version for now. What do you think?
>
> The double dash version is more consistent with how our other flags work.
>
> In general, I feel that if --version is supported, --help should say so.
>
> best,
> Colin
>
>
> >
> > Thanks,
> > Jason
> >
> > On Tue, Apr 10, 2018 at 12:00 AM, Sasaki Toru  >
> > wrote:
> >
> > > Hi Jason
> > >
> > > Thank you for helpful comments. I updated wiki based on your advice.
> > >
> > > I thought this option was relatively common and making maintenance easy
> > > was also important.
> > > However, as you said, it is not good that version option won't be
> shown up
> > > in help description.
> > >
> > > I thought accepting both single-dash and double-dash will help to find
> > > this option.
> > > In my approach this option won't be showed, but most of software which
> has
> > > this option accepts either single-dash or double-dash.
> > > I guess it doesn't need to support both if we take another way.
> > >
> > >
> > > Thanks
> > >
> > > @Ted Yeah, you're right. Sorry about the confusion.
> > >>
> > >> Since we're here, I think this KIP is a nice improvement. It's
> definitely
> > >> nice to have an easy way to check the version. That said, do we really
> > >> need
> > >> to support both `-version` and `--version`? The latter is consistent
> with
> > >> our current tools.
> > >>
> > >> Also, I think the approach we've taken is basically to build the
> --version
> > >> functionality into the bash script. This is nice because it saves a
> lot of
> > >> work to update the commands individually and we don't need to do
> anything
> > >> when we add new tools. The downside is that `--version` won't show up
> as
> > >> an
> > >> option in any of the --help output. Not sure if that is too big of a
> > >> problem, but maybe worth mentioning this in the rejected alternatives
> > >> section.
> > >>
> > >>
> > >> -Jason
> > >>
> > >> On Wed, Apr 4, 2018 at 9:42 AM, Ted Yu  wrote:
> > >>
> > >> Jason:
> > >>> Maybe your reply was intended for another KIP ?
> > >>>
> > >>> KIP-278 is about adding version option, not timeout.
> > >>>
> > >>> Cheers
> > >>>
> > >>> On Wed, Apr 4, 2018 at 9:36 AM, Jason Gustafson 
> > >>> wrote:
> > >>>
> > >>> Hi Sasaki,
> > 
> >  Thanks for the KIP. I think the timeout controls the maximum allowed
> > 
> > >>> time
> > >>
> > >>> that the consumer will block for the next record. Maybe the meaning
> > 
> > >>> would
> > >>
> > >>> be clearer with the more concise name `--timeout`? That also fits
> with
> > 
> > >>> the
> > >>>
> >  old consumer which overrides the `consumer.timeout.ms` property.
> > 
> >  By the way, it seems like the default value was intentionally set
> low
> > 
> > >>> for
> > >>
> > >>> both the old and new consumers, but I'm not sure of the reason. We
> could
> >  leave the default as it is if we want to be safe, but increasing it
> > 
> > >>> seems
> > >>
> > >>> ok to me. Perhaps we could start a little lower, 

[jira] [Reopened] (KAFKA-2334) Prevent HW from going back during leader failover

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma reopened KAFKA-2334:


> Prevent HW from going back during leader failover 
> --
>
> Key: KAFKA-2334
> URL: https://issues.apache.org/jira/browse/KAFKA-2334
> Project: Kafka
>  Issue Type: Bug
>  Components: replication
>Affects Versions: 0.8.2.1
>Reporter: Guozhang Wang
>Priority: Major
>  Labels: reliability
>
> Consider the following scenario:
> 0. Kafka use replication factor of 2, with broker B1 as the leader, and B2 as 
> the follower. 
> 1. A producer keep sending to Kafka with ack=-1.
> 2. A consumer repeat issuing ListOffset request to Kafka.
> And the following sequence:
> 0. B1 current log-end-offset (LEO) 0, HW-offset 0; and same with B2.
> 1. B1 receive a ProduceRequest of 100 messages, append to local log (LEO 
> becomes 100) and hold the request in purgatory.
> 2. B1 receive a FetchRequest starting at offset 0 from follower B2, and 
> returns the 100 messages.
> 3. B2 append its received message to local log (LEO becomes 100).
> 4. B1 receive another FetchRequest starting at offset 100 from B2, knowing 
> that B2's LEO has caught up to 100, and hence update its own HW, and 
> satisfying the ProduceRequest in purgatory, and sending the FetchResponse 
> with HW 100 back to B2 ASYNCHRONOUSLY.
> 5. B1 successfully sends the ProduceResponse to the producer, and then fails, 
> hence the FetchResponse did not reach B2, whose HW remains 0.
> From the consumer's point of view, it could first see the latest offset of 
> 100 (from B1), and then see the latest offset of 0 (from B2), and then the 
> latest offset gradually catch up to 100.
> This is because we use HW to guard the ListOffset and 
> Fetch-from-ordinary-consumer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[VOTE] Kafka 2.0.0 in June 2018

[Ismael Juma]

Hi all,

I started a discussion last year about bumping the version of the June 2018
release to 2.0.0[1]. To reiterate the reasons in the original post:

1. Adopt KIP-118 (Drop Support for Java 7), which requires a major version
bump due to semantic versioning.

2. Take the chance to remove deprecated code that was deprecated prior to
1.0.0, but not removed in 1.0.0 (e.g. old Scala clients) so that we can
move faster.

One concern that was raised is that we still do not have a rolling upgrade
path for the old ZK-based consumers. Since the Scala clients haven't been
updated in a long time (they don't support security or the latest message
format), users who need them can continue to use 1.1.0 with no loss of
functionality.

Since it's already mid-April and people seemed receptive during the
discussion last year, I'm going straight to a vote, but we can discuss more
if needed (of course).

Ismael

[1]
https://lists.apache.org/thread.html/dd9d3e31d7e9590c1f727ef5560c933281bad0de3134469b7b3c4257@%3Cdev.kafka.apache.org%3E

Re: [VOTE] KIP-279: Fix log divergence between leader and follower after fast leader fail over

[Ismael Juma]

Thanks for the detailed KIP. +1 (binding)

Ismael

On Sat, Apr 14, 2018 at 3:54 PM, Anna Povzner  wrote:

> Hi All,
>
>
> I would like to start the vote on KIP-279: Fix log divergence between
> leader and follower after fast leader fail over.
>
>
> For reference, here's the KIP wiki:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 279%3A+Fix+log+divergence+between+leader+and+follower+
> after+fast+leader+fail+over
>
>
>
> and discussion thread:
> https://www.mail-archive.com/dev@kafka.apache.org/msg86753.html
>
>
> Thanks,
>
> Anna
>

Re: [VOTE] KIP-270 A Scala wrapper library for Kafka Streams

[Ismael Juma]

Thanks for the contribution. I haven't reviewed all the new APIs in detail,
but the general approach sounds good to me. +1 (binding).

Ismael

On Wed, Apr 11, 2018 at 3:09 AM, Debasish Ghosh <
debasish.gh...@lightbend.com> wrote:

> Hello everyone -
>
> This is in continuation to the discussion regarding
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 270+-+A+Scala+Wrapper+Library+for+Kafka+Streams,
> which is a KIP for implementing a Scala wrapper library for Kafka Streams.
>
> We have had a PR (https://github.com/apache/kafka/pull/4756) for quite
> some
> time now and we have had lots of discussions and suggested improvements.
> Thanks to all who participated in the discussion and came up with all the
> suggestions for improvements.
>
> The purpose of this thread is to get an agreement on the implementation and
> have it included as part of Kafka.
>
> Looking forward ..
>
> regards.
>
> --
> Debasish Ghosh
> Principal Engineer
>
> Twitter: @debasishg
> Blog: http://debasishg.blogspot.com
> Code: https://github.com/debasishg
>

[jira] [Created] (KAFKA-6772) Broker should load credentials from ZK before requests are allowed

[Ismael Juma (JIRA)]

Ismael Juma created KAFKA-6772:
--

 Summary: Broker should load credentials from ZK before requests 
are allowed
 Key: KAFKA-6772
 URL: https://issues.apache.org/jira/browse/KAFKA-6772
 Project: Kafka
  Issue Type: Improvement
Affects Versions: 1.1.0, 1.0.0
Reporter: Ismael Juma


It is currently possible for clients to get an AuthenticationException during 
start-up if the brokers have not yet loaded credentials from ZK. This 
definitely affects SCRAM, but it may also affect delegation tokens.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Java 10 replacing Java 9 in Jenkins for trunk

[Ismael Juma]

Hi all,

Java 10 was recently released and support for Java 9 has ended since it's
not a LTS release. I've added a kafka-trunk Jenkins job for Java 10 and
disabled the Java 9 job. I also added a PR Jenkins job for Java 10 and will
soon disable the Java 9 PR job.

The general idea is to have a separate Jenkins job for the latest non LTS
release (Java 10) and all supported LTS releases (Java 8 and Java 7
currently, soon to become Java 8 only).

Let me know if you have any questions or concerns.

Ismael

[jira] [Created] (KAFKA-6763) Consider using direct byte buffers in SslTransportLayer

[Ismael Juma (JIRA)]

Ismael Juma created KAFKA-6763:
--

 Summary: Consider using direct byte buffers in SslTransportLayer
 Key: KAFKA-6763
 URL: https://issues.apache.org/jira/browse/KAFKA-6763
 Project: Kafka
  Issue Type: Improvement
Reporter: Ismael Juma


We use heap byte buffers in SslTransportLayer. For netReadBuffer and 
netWriteBuffer, it means that the NIO layer has to copy to/from a native buffer 
before it can write/read to the socket. It would be good to test if switching 
to direct byte buffers improves performance. We can't be sure as the benefit of 
avoiding the copy could be offset by the specifics of the operations we perform 
on netReadBuffer, netWriteBuffer and appReadBuffer.

We should benchmark produce and consume performance and try a few combinations 
of direct/heap byte buffers for netReadBuffer, netWriteBuffer and appReadBuffer 
(the latter should probably remain as a heap byte buffer, but no harm in 
testing it too).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-257 - Configurable Quota Management

[Ismael Juma]

Hi Rajini,

Can you share the motivation for the change? Not sure if the new approach
is better.

Ismael

On Thu, Apr 5, 2018 at 4:06 PM, Rajini Sivaram 
wrote:

> The quota callback interface was updated based on Jun's suggestion during
> the PR review. I have updated the KIP to reflect this.
>
> Let me know if there are any concerns.
>
> Thanks,
>
> Rajini
>
> On Thu, Apr 5, 2018 at 1:04 PM, Rajini Sivaram 
> wrote:
>
> > Thanks, Ismael.
> >
> > I have updated the KIP and the PR.
> >
> > On Wed, Apr 4, 2018 at 7:37 PM, Ismael Juma  wrote:
> >
> >> Sounds good to me Rajini. Good catch spotting this before it's included
> in
> >> a release. :)
> >>
> >> Ismael
> >>
> >> On Wed, Apr 4, 2018 at 11:13 AM, Rajini Sivaram <
> rajinisiva...@gmail.com>
> >> wrote:
> >>
> >> > For compatibility reasons, we are now using Java rather than Scala for
> >> all
> >> > pluggable interfaces including those on the broker. There is already a
> >> KIP
> >> > to move Authorizer to Java as well. As we will be removing support for
> >> Java
> >> > 7 in the next release, we can also use default methods in Java when we
> >> need
> >> > to update pluggable Java interfaces. So the plan is to use Java for
> all
> >> new
> >> > pluggable interfaces.
> >> >
> >> > We already have the package org.apache.kafka.server, under which we
> have
> >> > the sub-package for policies, so it makes sense to define quota
> >> callback as
> >> > a Java interface here too.
> >> >
> >> > If there are any concerns, please let me know. Otherwise I will update
> >> the
> >> > KIP and the associated PR.
> >> >
> >> > Thank you,
> >> >
> >> > Rajini
> >> >
> >> > On Thu, Mar 22, 2018 at 9:52 PM, Rajini Sivaram <
> >> rajinisiva...@gmail.com>
> >> > wrote:
> >> >
> >> > > Since there all the comments so far have been addressed, I will
> start
> >> > vote
> >> > > for this KIP.
> >> > >
> >> > > Regards,
> >> > >
> >> > > Rajini
> >> > >
> >> > > On Thu, Mar 15, 2018 at 6:37 PM, Rajini Sivaram <
> >> rajinisiva...@gmail.com
> >> > >
> >> > > wrote:
> >> > >
> >> > >> Thanks, Jun.
> >> > >>
> >> > >> 11. updatePartitionMetadata() provides all partitions with their
> >> leaders
> >> > >> so that callbacks that scale down quotas based on fraction of
> >> partitions
> >> > >> hosted on each broker may compute the scaling factor. Callbacks
> that
> >> > scale
> >> > >> up quotas based on the partition count hosted on each broker can
> >> simply
> >> > >> filter out the others. I have updated the Javadoc in the KIP.
> >> > >>
> >> > >> On Thu, Mar 15, 2018 at 5:24 PM, Jun Rao  wrote:
> >> > >>
> >> > >>> Hi, Rajini,
> >> > >>>
> >> > >>> Thanks for the explanation. It makes sense.
> >> > >>>
> >> > >>> 11. We probably want to clarify in the interface that every time
> >> when
> >> > >>> updatePartitionMetadata() is called, the full list of partitions
> >> whose
> >> > >>> leader is on this broker will be passed in?
> >> > >>>
> >> > >>> Jun
> >> > >>>
> >> > >>> On Thu, Mar 15, 2018 at 4:42 AM, Rajini Sivaram <
> >> > rajinisiva...@gmail.com
> >> > >>> >
> >> > >>> wrote:
> >> > >>>
> >> > >>> > Hi Jun,
> >> > >>> >
> >> > >>> > 12. Sorry, I had to revert the change that removed `
> >> > >>> > ClientQuotaCallback.quotaLimit()`. We are allowing quota
> >> callbacks
> >> > to
> >> > >>> use
> >> > >>> > custom metric tags. For each request, quota manager uses `
> >> > >>> > ClientQuotaCallback.quota()` to map (user-principal, client-id)
> to
> >> > the
> >> > >>> > metric tags that determine which clients share the quot

Re: [DISCUSS] KIP-257 - Configurable Quota Management

[Ismael Juma]

Sounds good to me Rajini. Good catch spotting this before it's included in
a release. :)

Ismael

On Wed, Apr 4, 2018 at 11:13 AM, Rajini Sivaram 
wrote:

> For compatibility reasons, we are now using Java rather than Scala for all
> pluggable interfaces including those on the broker. There is already a KIP
> to move Authorizer to Java as well. As we will be removing support for Java
> 7 in the next release, we can also use default methods in Java when we need
> to update pluggable Java interfaces. So the plan is to use Java for all new
> pluggable interfaces.
>
> We already have the package org.apache.kafka.server, under which we have
> the sub-package for policies, so it makes sense to define quota callback as
> a Java interface here too.
>
> If there are any concerns, please let me know. Otherwise I will update the
> KIP and the associated PR.
>
> Thank you,
>
> Rajini
>
> On Thu, Mar 22, 2018 at 9:52 PM, Rajini Sivaram 
> wrote:
>
> > Since there all the comments so far have been addressed, I will start
> vote
> > for this KIP.
> >
> > Regards,
> >
> > Rajini
> >
> > On Thu, Mar 15, 2018 at 6:37 PM, Rajini Sivaram  >
> > wrote:
> >
> >> Thanks, Jun.
> >>
> >> 11. updatePartitionMetadata() provides all partitions with their leaders
> >> so that callbacks that scale down quotas based on fraction of partitions
> >> hosted on each broker may compute the scaling factor. Callbacks that
> scale
> >> up quotas based on the partition count hosted on each broker can simply
> >> filter out the others. I have updated the Javadoc in the KIP.
> >>
> >> On Thu, Mar 15, 2018 at 5:24 PM, Jun Rao  wrote:
> >>
> >>> Hi, Rajini,
> >>>
> >>> Thanks for the explanation. It makes sense.
> >>>
> >>> 11. We probably want to clarify in the interface that every time when
> >>> updatePartitionMetadata() is called, the full list of partitions whose
> >>> leader is on this broker will be passed in?
> >>>
> >>> Jun
> >>>
> >>> On Thu, Mar 15, 2018 at 4:42 AM, Rajini Sivaram <
> rajinisiva...@gmail.com
> >>> >
> >>> wrote:
> >>>
> >>> > Hi Jun,
> >>> >
> >>> > 12. Sorry, I had to revert the change that removed `
> >>> > ClientQuotaCallback.quotaLimit()`. We are allowing quota callbacks
> to
> >>> use
> >>> > custom metric tags. For each request, quota manager uses `
> >>> > ClientQuotaCallback.quota()` to map (user-principal, client-id) to
> the
> >>> > metric tags that determine which clients share the quota. When quotas
> >>> are
> >>> > updated using  `updateQuota` or `updatePartitionMetadata`, existing
> >>> metrics
> >>> > need to updated, but quota managers don't have a reverse mapping of
> >>> metric
> >>> > tags to (user-principal, client-id) for invoking`ClientQuotaCallback.
> >>> > quota()
> >>> > ` . Callbacks cannot return all updated metrics since they don't have
> >>> > access to the metrics object and we don't want to require callbacks
> to
> >>> > track all the entities for which metrics have been created (since
> they
> >>> may
> >>> > contain client-ids and hence need expiring). With the extra method,
> >>> quota
> >>> > manager traverses the metric list after `updateQuota` or `
> >>> > updatePartitionMetadata` and obtains the latest value corresponding
> to
> >>> each
> >>> > metric based on the tags using `ClientQuotaCallback.quotaLimit()`.
> >>> >
> >>> > An alternative may be to delay quota metrics updates until the next
> >>> request
> >>> > that uses the metric. When we get sensors, we can check if the quota
> >>> > configured in the metric matches the value returned by `
> >>> > ClientQuotaCallback.quota()`. This will be slightly more expensive
> >>> since we
> >>> > need to check on every request, but the callback API as well as the
> >>> quota
> >>> > manager update code path would be simpler. What do you think?
> >>> >
> >>> > Thanks,
> >>> >
> >>> > Rajini
> >>> >
> >>> >
> >>> >
> >>> > On Wed, Mar 14, 2018 at 11:21 PM, Rajini Sivaram <
> >>> rajinisiva...@gmail.com>
> >>> > wrote:
> >>> >
> >>> > > Hi Jun,
> >>> > >
> >>> > > Thank you for reviewing the KIP.
> >>> > >
> >>> > > 10. This is the current behaviour (this KIP retains the same
> >>> behaviour
> >>> > for
> >>> > > the default quota callback). We include 'user' and 'client-id' tags
> >>> in
> >>> > > all the quota metrics, rather than omit tags at the moment.
> >>> > >
> >>> > > 11. Ah, I hadn't realised that. I wasn't expecting to include
> deleted
> >>> > > partitions in updatePartitionMetadata. I have updated the Javadoc
> in
> >>> the
> >>> > > KIP to reflect that.
> >>> > >
> >>> > > 12. When quotas are updated as a result of `updateQuota` or `
> >>> > > updatePartitionMetadata`, we may need to update quota bound for one
> >>> or
> >>> > > more existing metrics. I didn't want to expose metrics to the
> >>> callback.
> >>> > So `
> >>> > > quotaLimit` was providing the new quotas corresponding to existing
> >>> > > metrics. But perhaps a neater way to do this is to return updated
> >>> quotas
> >>> > as
> >>> > > the return value of `updat

Re: Permissions to edit KIP page / add pages

[Ismael Juma]

Done.

On Wed, Apr 4, 2018 at 3:41 AM, Moshe Blumberg  wrote:

> https://cwiki.apache.org/confluence/display/KAFKA/
> Kafka+Improvement+Proposals
>
> Can you please add permissions to the following users:
>
> mine:
> https://cwiki.apache.org/confluence/users/viewuserprofile.action?
> username=mblumberg
> a customer:
> https://cwiki.apache.org/confluence/users/viewuserprofile.action?
> username=alex.dunayevsky
>
> --
> Thanks,
> Moshe
> Customer Operations Engineer
>
> Check us out: Twitter  | LinkedIn
>  | Blog
>  
>
> Download Apache Kafka and Confluent Platform*: *
> *https://www.confluent.io/download/* 
>

Re: Permission request

[Ismael Juma]

Done.

On Wed, Apr 4, 2018 at 2:04 AM, Alex Dunayevsky 
wrote:

> Hello,
>
> I would like to add a KIP for kafka-consumer-perf-test according to this
> PR:
> https://github.com/apache/kafka/pull/4818
> (please check the details in comments) but do not have a permission to
> create a page.
>
> Thank you
>
> https://cwiki.apache.org/confluence/display/KAFKA/
> Kafka+Improvement+Proposals
>
> Best Regards,
> Alex Dunayevsky
>

[jira] [Reopened] (KAFKA-6390) Update ZooKeeper to 3.4.11, Gradle and other minor updates

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma reopened KAFKA-6390:


> Update ZooKeeper to 3.4.11, Gradle and other minor updates
> --
>
> Key: KAFKA-6390
> URL: https://issues.apache.org/jira/browse/KAFKA-6390
> Project: Kafka
>  Issue Type: Bug
>    Reporter: Ismael Juma
>    Assignee: Ismael Juma
>Priority: Major
> Fix For: 1.2.0
>
>
> https://issues.apache.org/jira/browse/ZOOKEEPER-2614 is a helpful fix.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-277 - Fine Grained ACL for CreateTopics API

[Ismael Juma]

Thanks for the KIP. I think this is going in the right direction, but we
need a better compatibility story. Also, it's worth considering whether we
want to tackle better wildcard support at the same time.

Ismael

On Thu, Mar 29, 2018 at 6:51 AM, Edoardo Comar  wrote:

> Hi all,
>
> We have submitted KIP-277 to give users permission to manage the lifecycle
> of a defined set of topics;
> the current ACL checks are for permission to create *any* topic and on
> delete for permission against the *named* topics.
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 277+-+Fine+Grained+ACL+for+CreateTopics+API
>
> Feedback and suggestions are welcome, thanks.
>
> Edo & Mickael
> --
>
> Edoardo Comar
>
> IBM Message Hub
>
> IBM UK Ltd, Hursley Park, SO21 2JN
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>

Re: [ANNOUNCE] Apache Kafka 1.1.0 Released

[Ismael Juma]

Thanks to Damian and Rajini for running the release and thanks to everyone
who helped make it happen!

Ismael

On Thu, Mar 29, 2018 at 2:27 AM, Rajini Sivaram  wrote:

> The Apache Kafka community is pleased to announce the release for
>
> Apache Kafka 1.1.0.
>
>
> Kafka 1.1.0 includes a number of significant new features.
>
> Here is a summary of some notable changes:
>
>
> ** Kafka 1.1.0 includes significant improvements to the Kafka Controller
>
>that speed up controlled shutdown. ZooKeeper session expiration edge
> cases
>
>have also been fixed as part of this effort.
>
>
> ** Controller improvements also enable more partitions to be supported on a
>
>single cluster. KIP-227 introduced incremental fetch requests, providing
>
>more efficient replication when the number of partitions is large.
>
>
> ** KIP-113 added support for replica movement between log directories to
>
>enable data balancing with JBOD.
>
>
> ** Some of the broker configuration options like SSL keystores can now be
>
>updated dynamically without restarting the broker. See KIP-226 for
> details
>
>and the full list of dynamic configs.
>
>
> ** Delegation token based authentication (KIP-48) has been added to Kafka
>
>brokers to support large number of clients without overloading Kerberos
>
>KDCs or other authentication servers.
>
>
> ** Several new features have been added to Kafka Connect, including header
>
>support (KIP-145), SSL and Kafka cluster identifiers in the Connect REST
>
>interface (KIP-208 and KIP-238), validation of connector names (KIP-212)
>
>and support for topic regex in sink connectors (KIP-215). Additionally,
>
>the default maximum heap size for Connect workers was increased to 2GB.
>
>
> ** Several improvements have been added to the Kafka Streams API, including
>
>reducing repartition topic partitions footprint, customizable error
>
>handling for produce failures and enhanced resilience to broker
>
>unavailability.  See KIPs 205, 210, 220, 224 and 239 for details.
>
>
> All of the changes in this release can be found in the release notes:
>
>
>
> https://dist.apache.org/repos/dist/release/kafka/1.1.0/RELEASE_NOTES.html
>
>
>
>
> You can download the source release from:
>
>
>
> https://www.apache.org/dyn/closer.cgi?path=/kafka/1.1.0/
> kafka-1.1.0-src.tgz
>
>
>
> and binary releases from:
>
>
>
> https://www.apache.org/dyn/closer.cgi?path=/kafka/1.1.0/
> kafka_2.11-1.1.0.tgz
>
> (Scala 2.11)
>
>
> https://www.apache.org/dyn/closer.cgi?path=/kafka/1.1.0/
> kafka_2.12-1.1.0.tgz
>
> (Scala 2.12)
>
>
> 
> --
>
>
>
> Apache Kafka is a distributed streaming platform with four core APIs:
>
>
>
> ** The Producer API allows an application to publish a stream records to
>
> one or more Kafka topics.
>
>
>
> ** The Consumer API allows an application to subscribe to one or more
>
> topics and process the stream of records produced to them.
>
>
>
> ** The Streams API allows an application to act as a stream processor,
>
> consuming an input stream from one or more topics and producing an output
>
> stream to one or more output topics, effectively transforming the input
>
> streams to output streams.
>
>
>
> ** The Connector API allows building and running reusable producers or
>
> consumers that connect Kafka topics to existing applications or data
>
> systems. For example, a connector to a relational database might capture
>
> every change to a table.three key capabilities:
>
>
>
>
> With these APIs, Kafka can be used for two broad classes of application:
>
> ** Building real-time streaming data pipelines that reliably get data
>
> between systems or applications.
>
>
>
> ** Building real-time streaming applications that transform or react to the
>
> streams of data.
>
>
>
>
> Apache Kafka is in use at large and small companies worldwide, including
>
> Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
>
> Target, The New York Times, Uber, Yelp, and Zalando, among others.
>
>
>
>
> A big thank you for the following 120 contributors to this release!
>
>
> Adem Efe Gencer, Alex Good, Andras Beni, Andy Bryant, Antony Stubbs,
>
> Apurva Mehta, Arjun Satish, bartdevylder, Bill Bejeck, Charly Molter,
>
> Chris Egerton, Clemens Valiente, cmolter, Colin P. Mccabe,
>
> Colin Patrick McCabe, ConcurrencyPractitioner, Damian Guy, dan norwood,
>
> Daniel

Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

[Ismael Juma]

Here's a link to the previous discussion:

https://lists.apache.org/thread.html/2bc187040051008452b40b313db06b476c248ef7a5ed7529afe7b118@1448997154@%3Cdev.kafka.apache.org%3E

Ismael

On Wed, Mar 28, 2018 at 10:40 AM, Ismael Juma  wrote:

> Hi Gwen,
>
> I don't think the reasons why a pluggable metastore is not desirable have
> changed. My suggestion is that the KIP should try to address the concerns
> raised previously as part of the proposal.
>
> Ismael
>
> On Wed, Mar 28, 2018 at 10:24 AM, Gwen Shapira  wrote:
>
>> While I'm not in favor of the proposal, I want to point out that the
>> ecosystem changed quite a bit since KIP-30 was first proposed. Kubernetes
>> deployments are far more common now and are growing in popularity, and the
>> problem in deployment, discovery and management that ZK poses is therefore
>> more relevant now than it was at the time. There are reasons for the
>> community to change its collective mind even if the objections are still
>> valid.
>>
>> Since the KIP doesn't include the etcd implementation, the proposal looks
>> like very simple refactoring. Of course, the big change is a new public
>> API. But it's difficult to judge from the KIP if the API is a good one
>> because it is built to 100% match the one implementation we have. I'm
>> curious if the plan includes contributing the Etcd module to Apache Kafka?
>>
>>
>> On Wed, Mar 28, 2018 at 9:54 AM, Ismael Juma  wrote:
>>
>> > Thanks for the KIP. This was proposed previously via "KIP-30 Allow for
>> > brokers to have plug-able consensus and meta data storage sub systems"
>> and
>> > the community was not in favour. Have you considered the points
>> discussed
>> > then?
>> >
>> > Ismael
>> >
>> > On Wed, Mar 28, 2018 at 9:18 AM, Molnár Bálint 
>> > wrote:
>> >
>> > > Hi all,
>> > >
>> > > I have created KIP-273: Kafka to support using ETCD beside Zookeeper
>> > >
>> > > Here is the link to the KIP:
>> > >
>> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> > > 273+-+Kafka+to+support+using+ETCD+beside+Zookeeper
>> > >
>> > > Looking forward to the discussion.
>> > >
>> > > Thanks,
>> > > Balint
>> > >
>> >
>>
>>
>>
>> --
>> *Gwen Shapira*
>> Product Manager | Confluent
>> 650.450.2760 | @gwenshap
>> Follow us: Twitter <https://twitter.com/ConfluentInc> | blog
>> <http://www.confluent.io/blog>
>>
>
>

Re: [ANNOUNCE] New Committer: Dong Lin

[Ismael Juma]

Congratulations Dong! Thanks for your contributions so far and looking
forward to future ones.

Ismael

On Wed, 28 Mar 2018, 10:58 Becket Qin,  wrote:

> Hello everyone,
>
> The PMC of Apache Kafka is pleased to announce that Dong Lin has accepted
> our invitation to be a new Kafka committer.
>
> Dong started working on Kafka about four years ago, since which he has
> contributed numerous features and patches. His work on Kafka core has been
> consistent and important. Among his contributions, most noticeably, Dong
> developed JBOD (KIP-112, KIP-113) to handle disk failures and to reduce
> overall cost, added deleteDataBefore() API (KIP-107) to allow users
> actively remove old messages. Dong has also been active in the community,
> participating in KIP discussions and doing code reviews.
>
> Congratulations and looking forward to your future contribution, Dong!
>
> Jiangjie (Becket) Qin, on behalf of Apache Kafka PMC
>

Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

[Ismael Juma]

Hi Gwen,

I don't think the reasons why a pluggable metastore is not desirable have
changed. My suggestion is that the KIP should try to address the concerns
raised previously as part of the proposal.

Ismael

On Wed, Mar 28, 2018 at 10:24 AM, Gwen Shapira  wrote:

> While I'm not in favor of the proposal, I want to point out that the
> ecosystem changed quite a bit since KIP-30 was first proposed. Kubernetes
> deployments are far more common now and are growing in popularity, and the
> problem in deployment, discovery and management that ZK poses is therefore
> more relevant now than it was at the time. There are reasons for the
> community to change its collective mind even if the objections are still
> valid.
>
> Since the KIP doesn't include the etcd implementation, the proposal looks
> like very simple refactoring. Of course, the big change is a new public
> API. But it's difficult to judge from the KIP if the API is a good one
> because it is built to 100% match the one implementation we have. I'm
> curious if the plan includes contributing the Etcd module to Apache Kafka?
>
>
> On Wed, Mar 28, 2018 at 9:54 AM, Ismael Juma  wrote:
>
> > Thanks for the KIP. This was proposed previously via "KIP-30 Allow for
> > brokers to have plug-able consensus and meta data storage sub systems"
> and
> > the community was not in favour. Have you considered the points discussed
> > then?
> >
> > Ismael
> >
> > On Wed, Mar 28, 2018 at 9:18 AM, Molnár Bálint 
> > wrote:
> >
> > > Hi all,
> > >
> > > I have created KIP-273: Kafka to support using ETCD beside Zookeeper
> > >
> > > Here is the link to the KIP:
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 273+-+Kafka+to+support+using+ETCD+beside+Zookeeper
> > >
> > > Looking forward to the discussion.
> > >
> > > Thanks,
> > > Balint
> > >
> >
>
>
>
> --
> *Gwen Shapira*
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter <https://twitter.com/ConfluentInc> | blog
> <http://www.confluent.io/blog>
>

Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

[Ismael Juma]

Thanks for the KIP. This was proposed previously via "KIP-30 Allow for
brokers to have plug-able consensus and meta data storage sub systems" and
the community was not in favour. Have you considered the points discussed
then?

Ismael

On Wed, Mar 28, 2018 at 9:18 AM, Molnár Bálint 
wrote:

> Hi all,
>
> I have created KIP-273: Kafka to support using ETCD beside Zookeeper
>
> Here is the link to the KIP:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 273+-+Kafka+to+support+using+ETCD+beside+Zookeeper
>
> Looking forward to the discussion.
>
> Thanks,
> Balint
>

Re: [DISCUSSION] KIP-266: Add TimeoutException to KafkaConsumer#position()

[Ismael Juma]

Hi Ewen,

Yeah, I mentioned KAFKA-2391 where some of this was discussed. Jay was
against having timeouts in the methods at the time. However, as Jason said
offline, we did end up with a timeout parameter in `poll`.

Ismael

On Fri, Mar 23, 2018 at 4:26 PM, Ewen Cheslack-Postava 
wrote:

> Regarding the flexibility question, has someone tried to dig up the
> discussion of the new consumer APIs when they were being written? I vaguely
> recall these exact questions about using APIs vs configs and flexibility vs
> bloating the API surface area having already been discussed. (Not that we
> shouldn't revisit, just that it might also be a faster way to get to a full
> understanding of the options, concerns, and tradeoffs).
>
> -Ewen
>
> On Thu, Mar 22, 2018 at 7:19 AM, Richard Yu 
> wrote:
>
> > I do have one question though: in the current KIP, throwing
> > TimeoutException to mark
> > that time limit is exceeded is applied to all new methods introduced in
> > this proposal.
> > However, how would users respond when a TimeoutException (since it is
> > considered
> > a RuntimeException)?
> >
> > Thanks,
> > Richard
> >
> >
> >
> > On Mon, Mar 19, 2018 at 6:10 PM, Richard Yu 
> > wrote:
> >
> > > Hi Ismael,
> > >
> > > You have a great point. Since most of the methods in this KIP have
> > similar
> > > callbacks (position() and committed() both use fetchCommittedOffsets(),
> > > and
> > > commitSync() is similar to position(), except just updating offsets),
> the
> > > amount of time
> > > they block should be also about equal.
> > >
> > > However, I think that we need to take into account a couple of things.
> > For
> > > starters,
> > > if the new methods were all reliant on one config, there is likelihood
> > > that the
> > > shortcomings for this approach would be similar to what we faced if we
> > let
> > > request.timeout.ms control all method timeouts. In comparison, adding
> > > overloads
> > > does not have this problem.
> > >
> > > If you have further thoughts, please let me know.
> > >
> > > Richard
> > >
> > >
> > > On Mon, Mar 19, 2018 at 5:12 PM, Ismael Juma 
> wrote:
> > >
> > >> Hi,
> > >>
> > >> An option that is not currently covered in the KIP is to have a
> separate
> > >> config max.block.ms, which is similar to the producer config with the
> > >> same
> > >> name. This came up during the KAFKA-2391 discussion. I think it's
> clear
> > >> that we can't rely on request.timeout.ms, so the decision is between
> > >> adding
> > >> overloads or adding a new config. People seemed to be leaning towards
> > the
> > >> latter in KAFKA-2391, but Jason makes a good point that the overloads
> > are
> > >> more flexible. A couple of questions from me:
> > >>
> > >> 1. Do we need the additional flexibility?
> > >> 2. If we do, do we need it for every blocking method?
> > >>
> > >> Ismael
> > >>
> > >> On Mon, Mar 19, 2018 at 5:03 PM, Richard Yu <
> yohan.richard...@gmail.com
> > >
> > >> wrote:
> > >>
> > >> > Hi Guozhang,
> > >> >
> > >> > I made some clarifications to KIP-266, namely:
> > >> > 1. Stated more specifically that commitSync will accept user input.
> > >> > 2. fetchCommittedOffsets(): Made its role in blocking more clear to
> > the
> > >> > reader.
> > >> > 3. Sketched what would happen when time limit is exceeded.
> > >> >
> > >> > These changes should make the KIP easier to understand.
> > >> >
> > >> > Cheers,
> > >> > Richard
> > >> >
> > >> > On Mon, Mar 19, 2018 at 9:33 AM, Guozhang Wang 
> > >> wrote:
> > >> >
> > >> > > Hi Richard,
> > >> > >
> > >> > > I made a pass over the KIP again, some more clarifications /
> > comments:
> > >> > >
> > >> > > 1. seek() call itself is not blocking, only the following poll()
> > call
> > >> may
> > >> > > be blocking as the actually metadata rq will happen.
> > >> > >
> > >> > > 2. I saw you did not include Consumer.partitionFor(),
> > >> > > Consumer.OffsetAndTimesta

[jira] [Resolved] (KAFKA-6683) ReplicaFetcher crashes with "Attempted to complete a transaction which was not started"

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-6683.

   Resolution: Fixed
Fix Version/s: 1.1.0

> ReplicaFetcher crashes with "Attempted to complete a transaction which was 
> not started" 
> 
>
> Key: KAFKA-6683
> URL: https://issues.apache.org/jira/browse/KAFKA-6683
> Project: Kafka
>  Issue Type: Bug
>  Components: replication
>Affects Versions: 1.0.0
> Environment: os: GNU/Linux 
> arch: x86_64
> Kernel: 4.9.77
> jvm: OpenJDK 1.8.0
>Reporter: Chema Sanchez
>Assignee: Jason Gustafson
>Priority: Critical
> Fix For: 1.1.0
>
> Attachments: server.properties
>
>
> We have been experiencing this issue lately when restarting or replacing 
> brokers of our Kafka clusters during maintenance operations.
> Having restarted or replaced a broker, after some minutes performing normally 
> it may suddenly throw the following exception and stop replicating some 
> partitions:
> {code:none}
> 2018-03-15 17:23:01,482] ERROR [ReplicaFetcher replicaId=12, leaderId=10, 
> fetcherId=0] Error due to (kafka.server.ReplicaFetcherThread)
> java.lang.IllegalArgumentException: Attempted to complete a transaction which 
> was not started
>     at 
> kafka.log.ProducerStateManager.completeTxn(ProducerStateManager.scala:720)
>     at kafka.log.Log.$anonfun$loadProducersFromLog$4(Log.scala:540)
>     at 
> kafka.log.Log.$anonfun$loadProducersFromLog$4$adapted(Log.scala:540)
>     at scala.collection.immutable.List.foreach(List.scala:389)
>     at 
> scala.collection.generic.TraversableForwarder.foreach(TraversableForwarder.scala:35)
>     at 
> scala.collection.generic.TraversableForwarder.foreach$(TraversableForwarder.scala:35)
>     at scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:44)
>     at kafka.log.Log.loadProducersFromLog(Log.scala:540)
>     at kafka.log.Log.$anonfun$loadProducerState$5(Log.scala:521)
>     at kafka.log.Log.$anonfun$loadProducerState$5$adapted(Log.scala:514)
>     at scala.collection.Iterator.foreach(Iterator.scala:929)
>     at scala.collection.Iterator.foreach$(Iterator.scala:929)
>     at scala.collection.AbstractIterator.foreach(Iterator.scala:1417)
>     at scala.collection.IterableLike.foreach(IterableLike.scala:71)
>     at scala.collection.IterableLike.foreach$(IterableLike.scala:70)
>     at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
>     at kafka.log.Log.loadProducerState(Log.scala:514)
>     at kafka.log.Log.$anonfun$truncateTo$2(Log.scala:1487)
>     at 
> scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:12)
>     at kafka.log.Log.maybeHandleIOException(Log.scala:1669)
>     at kafka.log.Log.truncateTo(Log.scala:1467)
>     at kafka.log.LogManager.$anonfun$truncateTo$2(LogManager.scala:454)
>     at 
> kafka.log.LogManager.$anonfun$truncateTo$2$adapted(LogManager.scala:445)
>     at 
> scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:789)
>     at scala.collection.immutable.Map$Map1.foreach(Map.scala:120)
>     at 
> scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:788)
>     at kafka.log.LogManager.truncateTo(LogManager.scala:445)
>     at 
> kafka.server.ReplicaFetcherThread.$anonfun$maybeTruncate$1(ReplicaFetcherThread.scala:281)
>     at scala.collection.Iterator.foreach(Iterator.scala:929)
>     at scala.collection.Iterator.foreach$(Iterator.scala:929)
>     at scala.collection.AbstractIterator.foreach(Iterator.scala:1417)
>     at scala.collection.IterableLike.foreach(IterableLike.scala:71)
>     at scala.collection.IterableLike.foreach$(IterableLike.scala:70)
>     at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
>     at 
> kafka.server.ReplicaFetcherThread.maybeTruncate(ReplicaFetcherThread.scala:265)
>     at 
> kafka.server.AbstractFetcherThread.$anonfun$maybeTruncate$2(AbstractFetcherThread.scala:135)
>     at 
> scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
>     at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:217)
>     at 
> kafka.server.AbstractFetcherThread.maybeTruncate(AbstractFetcherThread.scala:132)
>     at 
> kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:102)
>     at kafka.utils.ShutdownableThread.run(Shutdown

Re: Kafka version and JDK version

[Ismael Juma]

Hi Stephane,

I don't see why we would increment Kafka versions as quick as Java
versions. The way I think it should work is that we support LTS versions
for a long time and only support the most recent non LTS version. The
latter is to ensure that we catch any issues with newer Java releases
quickly, but people are encouraged to use Java LTS versions in production.
Given that, I don't think major version bumps in Kafka will happen often.
The bump to 2.0 also gives us an opportunity to drop the old Scala clients.
This will be a huge win in tech debt reduction so the project will be able
to move faster after that.

Ismael

On Tue, 20 Mar 2018, 20:24 Stephane Maarek, 
wrote:

> Hi
>
> If I remember correctly, Kafka 2.0 is targeted this summer as it'll drop
> support for java 7 and dropping a java version is supposed to imply a major
> version bump in Kafka.
>
> Now that Java has a very quick release cycle for JDK (version 10 today), my
> question is: how fast will Kafka versioning go ?
>
> My point of view is that we shouldn't increment the Kafka version as fast
> as Java, but that's currently the way it seems it'll go
>
> My perspective is that from someone who teaches Kafka, students expect
> major version bumps to have major effects on how they program. But it's a
> tough sell to explain that Kafka 2.0 has nothing major in the functioning
> or programming style except the underlying Java version.
>
> I just want to hear thoughts and opinions and start a discussion.
>
> Thanks !
> Stéphane
>

Re: [DISCUSS] KIP-263: Allow broker to skip sanity check of inactive segments on broker startup

[Ismael Juma]

Clarificarion, it's a by name parameter so nothing happens unless the first
parameter evaluates to false.

Ismael

On Mon, 19 Mar 2018, 19:21 Jay Kreps,  wrote:

> Optimizing startup seems really valuable but I'm a little confused by this.
>
> There are two different things:
> 1. Recovery
> 2. Sanity check
>
> The terminology we're using is a bit mixed here.
>
> Recovery means checksumming the log segments and rebuilding the index on a
> hard crash. This only happens on unflushed segments, which is generally
> just the last segment. Recovery is essential for the correctness guarantees
> of the log and you shouldn't disable it. It only happens on hard crash and
> is not a factor in graceful restart. We can likely optimize it but that
> would make most sense to do in a data driven fashion off some profiling.
>
> However there is also a ton of disk activity that happens during
> initialization (lots of checks on the file size, absolute path, etc). I
> think these have crept in over time with people not really realizing this
> code is perf sensitive and java hiding a lot of what is and isn't a file
> operation. One part of this is the sanityCheck() call for the two indexes.
> I don't think this call reads the full index, just the last entry in the
> index, right?. There should be no need to read the full index except during
> recovery (and then only for the segments being recovered). I think it would
> make a ton of sense to optimize this but I don't think that optimization
> needs to be configurable as this is just a helpful sanity check to detect
> common non-sensical things in the index files, but it isn't part of the
> core guarantees, in general you aren't supposed to lose committed data from
> disk, and if you do we may be able to fail faster but we fundamentally
> can't really help you. Again I think this would make the most sense to do
> in a data driven way, if you look at that code I think it is doing crazy
> amounts of file operations (e.g. getAbsolutePath, file sizes, etc). I think
> it'd make most sense to profile startup with a cold cash on a large log
> directory and do the same with an strace to see how many redundant system
> calls we do per segment and what is costing us and then cut some of this
> out. I suspect we could speed up our startup time quite a lot if we did
> that.
>
> For example we have a bunch of calls like this:
>
> require(len % entrySize == 0,
>
> "Index file " + file.getAbsolutePath + " is corrupt, found " +
> len +
>
> " bytes which is not positive or not a multiple of 8.")
> I'm pretty such file.getAbsolutePath is a system call and I assume that
> happens whether or not you fail the in-memory check?
>
> -Jay
>
>
> On Sun, Feb 25, 2018 at 10:27 PM, Dong Lin  wrote:
>
> > Hi all,
> >
> > I have created KIP-263: Allow broker to skip sanity check of inactive
> > segments on broker startup. See
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 263%3A+Allow+broker+to+skip+sanity+check+of+inactive+
> > segments+on+broker+startup
> > .
> >
> > This KIP provides a way to significantly reduce time to rolling bounce a
> > Kafka cluster.
> >
> > Comments are welcome!
> >
> > Thanks,
> > Dong
> >
>

Re: [DISCUSSION] KIP-266: Add TimeoutException to KafkaConsumer#position()

[Ismael Juma]

Hi,

An option that is not currently covered in the KIP is to have a separate
config max.block.ms, which is similar to the producer config with the same
name. This came up during the KAFKA-2391 discussion. I think it's clear
that we can't rely on request.timeout.ms, so the decision is between adding
overloads or adding a new config. People seemed to be leaning towards the
latter in KAFKA-2391, but Jason makes a good point that the overloads are
more flexible. A couple of questions from me:

1. Do we need the additional flexibility?
2. If we do, do we need it for every blocking method?

Ismael

On Mon, Mar 19, 2018 at 5:03 PM, Richard Yu 
wrote:

> Hi Guozhang,
>
> I made some clarifications to KIP-266, namely:
> 1. Stated more specifically that commitSync will accept user input.
> 2. fetchCommittedOffsets(): Made its role in blocking more clear to the
> reader.
> 3. Sketched what would happen when time limit is exceeded.
>
> These changes should make the KIP easier to understand.
>
> Cheers,
> Richard
>
> On Mon, Mar 19, 2018 at 9:33 AM, Guozhang Wang  wrote:
>
> > Hi Richard,
> >
> > I made a pass over the KIP again, some more clarifications / comments:
> >
> > 1. seek() call itself is not blocking, only the following poll() call may
> > be blocking as the actually metadata rq will happen.
> >
> > 2. I saw you did not include Consumer.partitionFor(),
> > Consumer.OffsetAndTimestamp() and Consumer.listTopics() in your KIP.
> After
> > a second thought, I think this may be a better idea to not tackle them in
> > the same KIP, and probably we should consider whether we would change the
> > behavior or not in another discussion. So I agree to not include them.
> >
> > 3. In your wiki you mentioned "Another change shall be made to
> > KafkaConsumer#poll(), due to its call to updateFetchPositions() which
> > blocks indefinitely." This part may a bit obscure to most readers who's
> not
> > familiar with the KafkaConsumer internals, could you please add more
> > elaborations. More specifically, I think the root causes of the public
> APIs
> > mentioned are a bit different while the KIP's explanation sounds like
> they
> > are due to the same reason:
> >
> > 3.1 fetchCommittedOffsets(): this internal call will block forever if the
> > committed offsets cannot be fetched successfully and affect position()
> and
> > committed(). We need to break out of its internal while loop.
> > 3.2 position() itself will while loop when offsets cannot be retrieved in
> > the underlying async call. We need to break out this while loop.
> > 3.3 commitSync() passed Long.MAX_VALUE as the timeout value, we should
> take
> > the user specified timeouts when applicable.
> >
> >
> >
> > Guozhang
> >
> > On Sat, Mar 17, 2018 at 4:44 PM, Richard Yu 
> > wrote:
> >
> > > Actually, what I said above is inaccurate. In
> > > testSeekAndCommitWithBrokerFailures, TestUtils.waitUntilTrue blocks,
> not
> > > seek.
> > > My assumption is that seek did not update correctly. I will be digging
> > > further into this.
> > >
> > >
> > >
> > > On Sat, Mar 17, 2018 at 4:16 PM, Richard Yu <
> yohan.richard...@gmail.com>
> > > wrote:
> > >
> > > > One more thing: when looking through tests, I have realized that
> seek()
> > > > methods can potentially block indefinitely. As you well know, seek()
> is
> > > > called when pollOnce() or position() is active. Thus, if position()
> > > blocks
> > > > indefinitely, then so would seek(). Should bounding seek() also be
> > > included
> > > > in this KIP?
> > > >
> > > > Thanks, Richard
> > > >
> > > > On Sat, Mar 17, 2018 at 1:16 PM, Richard Yu <
> > yohan.richard...@gmail.com>
> > > > wrote:
> > > >
> > > >> Thanks for the advice, Jason
> > > >>
> > > >> I have modified KIP-266 to include the java doc for committed() and
> > > other
> > > >> blocking methods, and I also
> > > >> mentioned poll() which will also be bounded. Let me know if there is
> > > >> anything else. :)
> > > >>
> > > >> Sincerely, Richard
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> On Sat, Mar 17, 2018 at 12:00 PM, Jason Gustafson <
> ja...@confluent.io
> > >
> > > >> wrote:
> > > >>
> > > >>> Hi Richard,
> > > >>>
> > > >>> Thanks for the updates. I'm really glad you picked this up. A
> couple
> > > >>> minor
> > > >>> comments:
> > > >>>
> > > >>> 1. Can you list the full set of new APIs explicitly in the KIP?
> > > >>> Currently I
> > > >>> only see the javadoc for `position()`.
> > > >>>
> > > >>> 2. We should consider adding `TimeUnit` to the new methods to avoid
> > > unit
> > > >>> confusion. I know it's inconsistent with the poll() API, but I
> think
> > it
> > > >>> was
> > > >>> probably a mistake not to include it there, so better not to double
> > > down
> > > >>> on
> > > >>> that mistake. And note that we do already have `close(long,
> > TimeUnit)`.
> > > >>>
> > > >>> Other than that, I think the current KIP seems reasonable.
> > > >>>
> > > >>> Thanks,
> > > >>> Jason
> > > >>>
> > > >>> On Wed, Mar 14, 2018 at 5:00 

Re: Subject: [VOTE] 1.1.0 RC3

[Ismael Juma]

Vahid,

The Java 9 Connect issue is similar to the one being fixed for Trogdor in
the following PR:

https://github.com/apache/kafka/pull/4725

We need to do something similar for Connect.

Ismael

On Fri, Mar 16, 2018 at 3:10 PM, Vahid S Hashemian <
vahidhashem...@us.ibm.com> wrote:

> Hi Damian,
>
> Thanks for running the release.
>
> I tried building from source and running the quick start on Linux &
> Windows with both Java 8 & 9.
> Here's the result:
>
> +-+-+-+
> | |  Linux  | Windows |
> + +-+-+
> | | J8 | J9 | J8 | J9 |
> +-+++++
> |  Build  |  + |  + |  + |  + |
> +-+++++
> |  Single broker  |  + |  + |  + |  + |
> | produce/consume |||||
> +-+++++
> | Connect |  + |  ? |  - |  - |
> +-+++++
> | Streams |  + |  + |  + |  + |
> +-+++++
>
> ?: Connect quickstart on Linux with Java 9 runs but the connect tool
> throws a bunch of exceptions (https://www.codepile.net/pile/yVg8XJB8)
> -: Connect quickstart on Windows fails (Java 8:
> https://www.codepile.net/pile/xJGra6BP, Java 9:
> https://www.codepile.net/pile/oREYeORK)
>
> Given that Windows is not an officially supported platform, and the
> exceptions with Linux/Java 9 are not breaking the functionality, my vote
> is a +1 (non-binding).
>
> Thanks.
> --Vahid
>
>
>
>
> From:   Damian Guy 
> To: dev@kafka.apache.org, us...@kafka.apache.org,
> kafka-clie...@googlegroups.com
> Date:   03/15/2018 07:55 AM
> Subject:Subject: [VOTE] 1.1.0 RC3
>
>
>
> Hello Kafka users, developers and client-developers,
>
> This is the fourth candidate for release of Apache Kafka 1.1.0.
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.
> apache.org_confluence_pages_viewpage.action-3FpageId-
> 3D75957546&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_
> itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc&m=Qn2GySTKcOV5MFr3WDl63BDv7pTd2s
> gX46mjZPws01U&s=cKgJtQXXRauZ3HSAoSbsC9SLVTAkO-pbLdPrOCBuJzE&e=
>
>
> A few highlights:
>
> * Significant Controller improvements (much faster and session expiration
> edge cases fixed)
> * Data balancing across log directories (JBOD)
> * More efficient replication when the number of partitions is large
> * Dynamic Broker Configs
> * Delegation tokens (KIP-48)
> * Kafka Streams API improvements (KIP-205 / 210 / 220 / 224 / 239)
>
>
> Release notes for the 1.1.0 release:
> https://urldefense.proofpoint.com/v2/url?u=http-3A__home.
> apache.org_-7Edamianguy_kafka-2D1.1.0-2Drc3_RELEASE-5FNOTES.
> html&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-
> kjJc7uSVcviKUc&m=Qn2GySTKcOV5MFr3WDl63BDv7pTd2sgX46mjZPws01U&s=
> 26FgbzRhKImhxyEkB4KzDPG-l8W_Y99xU6LykOAgpFI&e=
>
>
> *** Please download, test and vote by Monday, March 19, 9am PDT
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://urldefense.proofpoint.com/v2/url?u=http-3A__kafka.
> apache.org_KEYS&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_
> itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc&m=Qn2GySTKcOV5MFr3WDl63BDv7pTd2s
> gX46mjZPws01U&s=xlnrfgxVFMRCKk8pTOhujyC-Um4ogtsxK6Xwks6mc3U&e=
>
>
> * Release artifacts to be voted upon (source and binary):
> https://urldefense.proofpoint.com/v2/url?u=http-3A__home.
> apache.org_-7Edamianguy_kafka-2D1.1.0-2Drc3_&d=DwIBaQ&c=jf_
> iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc&m=
> Qn2GySTKcOV5MFr3WDl63BDv7pTd2sgX46mjZPws01U&s=
> ulHUeYnWIp28Gsn4VV1NK3FrGV4Jn5rUpuU6tvgekME&e=
>
>
> * Maven artifacts to be voted upon:
> https://urldefense.proofpoint.com/v2/url?u=https-3A__
> repository.apache.org_content_groups_staging_&d=DwIBaQ&c=jf_
> iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc&m=
> Qn2GySTKcOV5MFr3WDl63BDv7pTd2sgX46mjZPws01U&s=G9o4hXVXF0bjL_
> a3Wocod9GUEfy9WBBgoGa2u6yFKQw&e=
>
>
> * Javadoc:
> https://urldefense.proofpoint.com/v2/url?u=http-3A__home.
> apache.org_-7Edamianguy_kafka-2D1.1.0-2Drc3_javadoc_&d=
> DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-
> kjJc7uSVcviKUc&m=Qn2GySTKcOV5MFr3WDl63BDv7pTd2sgX46mjZPws01U&s=
> 2auaI4IIJhEORGYm1Kdpxt5TDHh0PzSvtK77lC3SJVY&e=
>
>
> * Tag to be voted upon (off 1.1 branch) is the 1.1.0 tag:
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.
> com_apache_kafka_tree_1.1.0-2Drc3&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_
> itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc&m=Qn2GySTKcOV5MFr3WDl63BDv7pTd2s
> gX46mjZPws01U&s=h7G8XPD8vAWl_gqySi2Iocag5NnP32IT_PyirPC3Lss&e=
>
>
>
> * Documentation:
> https://urldefense.proofpoint.com/v2/url?u=http-3A__kafka.
> apache.org_11_documentation.html&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_
> itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc&m=Qn2GySTKcOV5MFr3WDl63BDv7pTd2s
> gX46mjZPws01U&s=KcPsL867-tPQxKPC4ufl5tqg9RSL3lxwsgKhOxeA7t0&e=
>
> <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__kaf

Re: [DISCUSS] KIP-270 A Scala wrapper library for Kafka Streams

[Ismael Juma]

 it
> >>>>>> need to be "org.apache.kafka"; for the version, it need to be Kafka
> >>>>> release
> >>>>>> versions, e.g. 1.2.0.
> >>>>>>
> >>>>>> 4. In "New or Changed Public Interfaces" section, it does not
> >> mention
> >>>>> the
> >>>>>> serde classes like "DefaultSerdes", but I think they should also be
> >>>>>> considered public APIs as users would most likely import these in
> >> her
> >>>>>> implementation.
> >>>>>>
> >>>>>> 5. Could you also list the changes you'd propose to made to
> >>> build.gradle
> >>>>>> in kafka for adding this artifact? More details will help readers to
> >>>>> better
> >>>>>> understand your proposal.
> >>>>>>
> >>>>>> 6. I think it'd will be good to have a WordCount example code as
> >> part
> >>> of
> >>>>>> this KIP, to illustrate how to code in this Scala wrapper, e.g. in
> >>>>>> o.a.k.scala.examples. But for this class we probably do not need to
> >>>>> have a
> >>>>>> separate artifact for it as we did in kafka-streams-examples.
> >>>>>>
> >>>>>>
> >>>>>> Guozhang
> >>>>>>
> >>>>>>
> >>>>>> On Mon, Mar 19, 2018 at 9:16 AM, Ted Yu 
> >> wrote:
> >>>>>>
> >>>>>>> I agree with Sean on name unification.
> >>>>>>>
> >>>>>>> +1 to option 2.
> >>>>>>>
> >>>>>>> On Mon, Mar 19, 2018 at 7:23 AM, Sean Glover <
> >>>>> sean.glo...@lightbend.com>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Type names: I vote for option 2.  The user must explicitly add a
> >>>>>>> dependency
> >>>>>>>> to this library and the wrapper types are in a different package.
> >>> It
> >>>>>>> seems
> >>>>>>>> reasonable to expect them to do an import rename if there's a
> >> need
> >>> to
> >>>>>>> drop
> >>>>>>>> down to the Java API.
> >>>>>>>>
> >>>>>>>> Test Utils: The test utils in kafka-streams-scala are nice and
> >>> lean,
> >>>>> but
> >>>>>>>> I'm not sure if it provides much more value than other options
> >> that
> >>>>>>> exist
> >>>>>>>> in the community.  There's an embedded Kafka/ZK project
> >>>>> implementation
> >>>>>>> for
> >>>>>>>> ScalaTest that's popular and active: manub/scalatest-embedded-
> >>> kakfa.
> >>>>>>> It
> >>>>>>>> implies you must also use ScalaTest, which I acknowledge isn't
> >>>>>>> everyone's
> >>>>>>>> first choice for Scala test framework, but it probably is one of,
> >>> if
> >>>>> not
> >>>>>>>> the most, popular library.  It includes a DSL for Kafka Streams
> >>>>> too.  If
> >>>>>>>> this KIP is accepted then perhaps a PR to that project could be
> >>> made
> >>>>> to
> >>>>>>>> support the new wrapper implementations.
> >>>>>>>>
> >>>>>>>> https://github.com/manub/scalatest-embedded-kafka#
> >>>>>>>> scalatest-embedded-kafka-streams
> >>>>>>>>
> >>>>>>>> Sean
> >>>>>>>>
> >>>>>>>> On Sun, Mar 18, 2018 at 4:05 AM, Debasish Ghosh <
> >>>>>>>> debasish.gh...@lightbend.com> wrote:
> >>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> Should this be 1.2  (maybe it's even better to not put any
> >>>>>>> version at
> >>>>>>>>>> all)
> >>>>>>>>>
> >>>>>>>>>
> >>>

Re: [DISCUSS] KIP-270 A Scala wrapper library for Kafka Streams

[Ismael Juma]

If this is rare (as it sounds), relying on import renames seems fine to me.
Let's see what others think.

Ismael

On Fri, Mar 16, 2018 at 10:51 AM, Debasish Ghosh <
debasish.gh...@lightbend.com> wrote:

> I am not sure if this is practical or not. But theoretically a user may
> want to extract the unsafe Java abstraction from the Scala ones and use
> Java APIs on them .. e.g.
>
> val userClicksStream: KStreamS[String, Long] =
> builder.stream(userClicksTopic) // Scala abstraction
>
> val jStream: KStream[String, Long] = userClicksStream.inner // publishes
> the underlying Java abstraction
>
> //.. work with Java, may be pass to some function written in Java
>
> I do realize this is somewhat of a convoluted use case and may not be
> practically useful ..
>
> Otherwise we can very well work on the suggested approach of unifying the
> names ..
>
> regards.
>
>
>
> On Fri, Mar 16, 2018 at 10:28 PM, Ismael Juma  wrote:
>
> > What does "mixed mode application" mean? What are the cases where a user
> > would want to use both APIs? I think that would help understand the
> > reasoning.
> >
> > Thanks,
> > Ismael
> >
> > On Fri, Mar 16, 2018 at 8:48 AM, Debasish Ghosh <
> > debasish.gh...@lightbend.com> wrote:
> >
> > > Hi Damian -
> > >
> > > We could. But in case the user wants to use both Scala and Java APIs
> (may
> > > be for some mixed mode application), won't that be confusing ? She will
> > > have to do something like ..
> > >
> > > import o.a.k.s.scala.{KStream => KStreamS}
> > >
> > > to rename Scala imports or the other way round for imported Java
> classes.
> > >
> > > regards.
> > >
> > >
> > >
> > > On Fri, Mar 16, 2018 at 9:07 PM, Damian Guy 
> > wrote:
> > >
> > > > Hi Debasish,
> > > >
> > > > Thanks for the KIP - will be a great addition to streams. I've only
> > had a
> > > > quick scan, but seeing as the Scala classes are going to be in their
> > own
> > > > package could we drop the S at the end of the class names?
> > > >
> > > > Thanks,
> > > > Damian
> > > >
> > > >
> > > > On Fri, 16 Mar 2018 at 15:25 Debasish Ghosh <
> > > debasish.gh...@lightbend.com>
> > > > wrote:
> > > >
> > > > > Hi -
> > > > >
> > > > > A new KIP, KIP-270 is up for discussion:
> > > > >
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > 270+-+A+Scala+Wrapper+Library+for+Kafka+Streams
> > > > >
> > > > > The relevant JIRA issue: https://issues.apache.org/
> > > > jira/browse/KAFKA-6670
> > > > >
> > > > > The library as proposed in the KIP has been implemented at
> > > > > https://github.com/lightbend/kafka-streams-scala and the current
> > > release
> > > > > is
> > > > > 0.2.0 (
> > > > > https://github.com/lightbend/kafka-streams-scala/releases/
> tag/v0.2.0
> > ).
> > > > > We at Lightbend has been using it since quite some time now.
> > > > >
> > > > > regards.
> > > > >
> > > > > --
> > > > > Debasish Ghosh
> > > > > Principal Engineer
> > > > >
> > > > > Twitter: @debasishg
> > > > > Blog: http://debasishg.blogspot.com
> > > > > Code: https://github.com/debasishg
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Debasish Ghosh
> > > Principal Engineer
> > >
> > > Twitter: @debasishg
> > > Blog: http://debasishg.blogspot.com
> > > Code: https://github.com/debasishg
> > >
> >
>
>
>
> --
> Debasish Ghosh
> Principal Engineer
>
> Twitter: @debasishg
> Blog: http://debasishg.blogspot.com
> Code: https://github.com/debasishg
>

Re: [DISCUSS] KIP-270 A Scala wrapper library for Kafka Streams

[Ismael Juma]

What does "mixed mode application" mean? What are the cases where a user
would want to use both APIs? I think that would help understand the
reasoning.

Thanks,
Ismael

On Fri, Mar 16, 2018 at 8:48 AM, Debasish Ghosh <
debasish.gh...@lightbend.com> wrote:

> Hi Damian -
>
> We could. But in case the user wants to use both Scala and Java APIs (may
> be for some mixed mode application), won't that be confusing ? She will
> have to do something like ..
>
> import o.a.k.s.scala.{KStream => KStreamS}
>
> to rename Scala imports or the other way round for imported Java classes.
>
> regards.
>
>
>
> On Fri, Mar 16, 2018 at 9:07 PM, Damian Guy  wrote:
>
> > Hi Debasish,
> >
> > Thanks for the KIP - will be a great addition to streams. I've only had a
> > quick scan, but seeing as the Scala classes are going to be in their own
> > package could we drop the S at the end of the class names?
> >
> > Thanks,
> > Damian
> >
> >
> > On Fri, 16 Mar 2018 at 15:25 Debasish Ghosh <
> debasish.gh...@lightbend.com>
> > wrote:
> >
> > > Hi -
> > >
> > > A new KIP, KIP-270 is up for discussion:
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 270+-+A+Scala+Wrapper+Library+for+Kafka+Streams
> > >
> > > The relevant JIRA issue: https://issues.apache.org/
> > jira/browse/KAFKA-6670
> > >
> > > The library as proposed in the KIP has been implemented at
> > > https://github.com/lightbend/kafka-streams-scala and the current
> release
> > > is
> > > 0.2.0 (
> > > https://github.com/lightbend/kafka-streams-scala/releases/tag/v0.2.0).
> > > We at Lightbend has been using it since quite some time now.
> > >
> > > regards.
> > >
> > > --
> > > Debasish Ghosh
> > > Principal Engineer
> > >
> > > Twitter: @debasishg
> > > Blog: http://debasishg.blogspot.com
> > > Code: https://github.com/debasishg
> > >
> >
>
>
>
> --
> Debasish Ghosh
> Principal Engineer
>
> Twitter: @debasishg
> Blog: http://debasishg.blogspot.com
> Code: https://github.com/debasishg
>

Re: [DISCUSS] KIP-270 A Scala wrapper library for Kafka Streams

[Ismael Juma]

Thanks for the KIP! It will be nice to have an API for Kafka Streams that
is more concise _and_ safer.

Like Damian, I would prefer if we didn't have a suffix and relied on
package names. However, if there's a good reason why this won't work well,
we should explain it in the "Rejected Alternatives" section.

Ismael

On Fri, Mar 16, 2018 at 8:37 AM, Damian Guy  wrote:

> Hi Debasish,
>
> Thanks for the KIP - will be a great addition to streams. I've only had a
> quick scan, but seeing as the Scala classes are going to be in their own
> package could we drop the S at the end of the class names?
>
> Thanks,
> Damian
>
>
> On Fri, 16 Mar 2018 at 15:25 Debasish Ghosh 
> wrote:
>
> > Hi -
> >
> > A new KIP, KIP-270 is up for discussion:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 270+-+A+Scala+Wrapper+Library+for+Kafka+Streams
> >
> > The relevant JIRA issue: https://issues.apache.org/
> jira/browse/KAFKA-6670
> >
> > The library as proposed in the KIP has been implemented at
> > https://github.com/lightbend/kafka-streams-scala and the current release
> > is
> > 0.2.0 (
> > https://github.com/lightbend/kafka-streams-scala/releases/tag/v0.2.0).
> > We at Lightbend has been using it since quite some time now.
> >
> > regards.
> >
> > --
> > Debasish Ghosh
> > Principal Engineer
> >
> > Twitter: @debasishg
> > Blog: http://debasishg.blogspot.com
> > Code: https://github.com/debasishg
> >
>

Re: [VOTE] 1.1.0 RC1

[Ismael Juma]

Thanks Jeff:

https://github.com/apache/kafka/pull/4678

Ismael

On Fri, Mar 9, 2018 at 1:56 AM, Damian Guy  wrote:

> Hi Jeff,
>
> Thanks, we will look into this.
>
> Regards,
> Damian
>
> On Thu, 8 Mar 2018 at 18:27 Jeff Chao  wrote:
>
> > Hello,
> >
> > We at Heroku have run 1.1.0 RC1 through our normal performance and
> > regression test suite and have found performance to be comparable to
> 1.0.0.
> >
> > That said, we're however -1 (non-binding) since this release includes
> > Zookeeper 3.4.11 
> which
> > is affected by the critical regression ZOOKEEPER-2960
> > . As 3.4.12 isn't
> > released yet, it might be better to have 3.4.10 included instead.
> >
> > Jeff
> > Heroku
> >
> >
> > On Tue, Mar 6, 2018 at 1:19 PM, Ted Yu  wrote:
> >
> > > +1
> > >
> > > Checked signature
> > > Ran test suite - apart from flaky testMetricsLeak, other tests passed.
> > >
> > > On Tue, Mar 6, 2018 at 2:45 AM, Damian Guy 
> wrote:
> > >
> > > > Hello Kafka users, developers and client-developers,
> > > >
> > > > This is the second candidate for release of Apache Kafka 1.1.0.
> > > >
> > > > This is minor version release of Apache Kakfa. It Includes 29 new
> KIPs.
> > > > Please see the release plan for more details:
> > > >
> > > > https://cwiki.apache.org/confluence/pages/viewpage.
> > > action?pageId=71764913
> > > >
> > > > A few highlights:
> > > >
> > > > * Significant Controller improvements (much faster and session
> > expiration
> > > > edge cases fixed)
> > > > * Data balancing across log directories (JBOD)
> > > > * More efficient replication when the number of partitions is large
> > > > * Dynamic Broker Configs
> > > > * Delegation tokens (KIP-48)
> > > > * Kafka Streams API improvements (KIP-205 / 210 / 220 / 224 / 239)
> > > >
> > > > Release notes for the 1.1.0 release:
> > > > http://home.apache.org/~damianguy/kafka-1.1.0-rc1/RELEASE_NOTES.html
> > > >
> > > > *** Please download, test and vote by Friday, March 9th, 5pm PT
> > > >
> > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > http://kafka.apache.org/KEYS
> > > >
> > > > * Release artifacts to be voted upon (source and binary):
> > > > http://home.apache.org/~damianguy/kafka-1.1.0-rc1/
> > > >
> > > > * Maven artifacts to be voted upon:
> > > > https://repository.apache.org/content/groups/staging/
> > > >
> > > > * Javadoc:
> > > > http://home.apache.org/~damianguy/kafka-1.1.0-rc1/javadoc/
> > > >
> > > > * Tag to be voted upon (off 1.1 branch) is the 1.1.0 tag:
> > > > https://github.com/apache/kafka/tree/1.1.0-rc1
> > > >
> > > >
> > > > * Documentation:
> > > > http://kafka.apache.org/11/documentation.html
> > > >
> > > > * Protocol:
> > > > http://kafka.apache.org/11/protocol.html
> > > >
> > > > * Successful Jenkins builds for the 1.1 branch:
> > > > Unit/integration tests:
> > https://builds.apache.org/job/kafka-1.1-jdk7/68
> > > > System tests: https://jenkins.confluent.io/
> > > job/system-test-kafka/job/1.1/
> > > > 30/
> > > >
> > > > /**
> > > >
> > > > Thanks,
> > > > Damian Guy
> > > >
> > >
> >
>

Re: [ANNOUNCE] Apache Kafka 1.0.1 Released

[Ismael Juma]

Thanks for running the release Ewen and great work everyone!

Ismael

On Tue, Mar 6, 2018 at 1:14 PM, Ewen Cheslack-Postava  wrote:

> The Apache Kafka community is pleased to announce the release for Apache
> Kafka
> 1.0.1.
>
> This is a bugfix release for the 1.0 branch that was first released with
> 1.0.0 about 4 months ago. We've fixed 49 issues since that release. Most of
> these are non-critical, but in aggregate these fixes will have significant
> impact. A few of the more significant fixes include:
>
> * KAFKA-6277: Make loadClass thread-safe for class loaders of Connect
> plugins
> * KAFKA-6185: Selector memory leak with high likelihood of OOM in case of
> down conversion
> * KAFKA-6269: KTable state restore fails after rebalance
> * KAFKA-6190: GlobalKTable never finishes restoring when consuming
> transactional messages
> * KAFKA-6529: Stop file descriptor leak when client disconnects with
> staged receives
> * KAFKA-6238: Issues with protocol version when applying a rolling upgrade
> to 1.0.0
>
>
> All of the changes in this release can be found in the release notes:
>
>
> https://dist.apache.org/repos/dist/release/kafka/1.0.1/RELEASE_NOTES.html
>
>
>
> You can download the source release from:
>
>
> https://www.apache.org/dyn/closer.cgi?path=/kafka/1.0.1/
> kafka-1.0.1-src.tgz
>
>
> and binary releases from:
>
>
> https://www.apache.org/dyn/closer.cgi?path=/kafka/1.0.1/
> kafka_2.11-1.0.1.tgz
> (Scala 2.11)
>
> https://www.apache.org/dyn/closer.cgi?path=/kafka/1.0.1/
> kafka_2.12-1.0.1.tgz
> (Scala 2.12)
>
> 
> ---
>
>
> Apache Kafka is a distributed streaming platform with four core APIs:
>
>
> ** The Producer API allows an application to publish a stream records to
> one or more Kafka topics.
>
>
> ** The Consumer API allows an application to subscribe to one or more
> topics and process the stream of records produced to them.
>
>
> ** The Streams API allows an application to act as a stream processor,
> consuming an input stream from one or more topics and producing an output
> stream to one or more output topics, effectively transforming the input
> streams to output streams.
>
>
> ** The Connector API allows building and running reusable producers or
> consumers that connect Kafka topics to existing applications or data
> systems. For example, a connector to a relational database might capture
> every change to a table.three key capabilities:
>
>
>
> With these APIs, Kafka can be used for two broad classes of application:
>
>
> ** Building real-time streaming data pipelines that reliably get data
> between systems or applications.
>
>
> ** Building real-time streaming applications that transform or react to the
> streams of data.
>
>
>
> Apache Kafka is in use at large and small companies worldwide, including
> Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
> Target, The New York Times, Uber, Yelp, and Zalando, among others.
>
>
>
> A big thank you for the following 36 contributors to this release!
>
> Alex Good, Andras Beni, Andy Bryant, Arjun Satish, Bill Bejeck, Colin P.
> Mccabe, Colin Patrick McCabe, ConcurrencyPractitioner, Damian Guy, Daniel
> Wojda, Dong Lin, Edoardo Comar, Ewen Cheslack-Postava, Filipe Agapito,
> fredfp, Guozhang Wang, huxihx, Ismael Juma, Jason Gustafson, Jeremy
> Custenborder, Jiangjie (Becket) Qin, Joel Hamill, Konstantine Karantasis,
> lisa2lisa, Logan Buckley, Manjula K, Matthias J. Sax, Nick Chiu, parafiend,
> Rajini Sivaram, Randall Hauch, Robert Yokota, Ron Dagostino, tedyu,
> Yaswanth Kumar, Yu.
>
>
> We welcome your help and feedback. For more information on how to
> report problems,
> and to get involved, visit the project website at http://kafka.apache.org/
>
>
> Thank you!
> Ewen
>

[jira] [Resolved] (KAFKA-4974) System test failure in 0.8.2.2 upgrade tests

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-4974.

Resolution: Fixed

> System test failure in 0.8.2.2 upgrade tests
> 
>
> Key: KAFKA-4974
> URL: https://issues.apache.org/jira/browse/KAFKA-4974
> Project: Kafka
>  Issue Type: Bug
>  Components: system tests
>Reporter: Magnus Edenhill
>Priority: Major
>
> The 0.10.2 system test failed in one of the upgrade tests from 0.8.2.2:
> http://testing.confluent.io/confluent-kafka-0-10-2-system-test-results/?prefix=2017-03-21--001.1490092219--apache--0.10.2--4a019bd/TestUpgrade/test_upgrade/from_kafka_version=0.8.2.2.to_message_format_version=None.compression_types=.snappy.new_consumer=False/
> {noformat}
> [INFO  - 2017-03-21 07:35:48,802 - runner_client - log - lineno:221]: 
> RunnerClient: 
> kafkatest.tests.core.upgrade_test.TestUpgrade.test_upgrade.from_kafka_version=0.8.2.2.to_message_format_version=None.compression_types=.snappy.new_consumer=False:
>  FAIL: Kafka server didn't finish startup
> Traceback (most recent call last):
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka-0.10.2/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.6.0-py2.7.egg/ducktape/tests/runner_client.py",
>  line 123, in run
> data = self.run_test()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka-0.10.2/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.6.0-py2.7.egg/ducktape/tests/runner_client.py",
>  line 176, in run_test
> return self.test_context.function(self.test)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka-0.10.2/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.6.0-py2.7.egg/ducktape/mark/_mark.py",
>  line 321, in wrapper
> return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka-0.10.2/kafka/tests/kafkatest/tests/core/upgrade_test.py",
>  line 125, in test_upgrade
> self.run_produce_consume_validate(core_test_action=lambda: 
> self.perform_upgrade(from_kafka_version,
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka-0.10.2/kafka/tests/kafkatest/tests/produce_consume_validate.py",
>  line 114, in run_produce_consume_validate
> core_test_action(*args)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka-0.10.2/kafka/tests/kafkatest/tests/core/upgrade_test.py",
>  line 126, in 
> to_message_format_version))
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka-0.10.2/kafka/tests/kafkatest/tests/core/upgrade_test.py",
>  line 52, in perform_upgrade
> self.kafka.start_node(node)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka-0.10.2/kafka/tests/kafkatest/services/kafka/kafka.py",
>  line 222, in start_node
> monitor.wait_until("Kafka Server.*started", timeout_sec=30, 
> backoff_sec=.25, err_msg="Kafka server didn't finish startup")
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka-0.10.2/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.6.0-py2.7.egg/ducktape/cluster/remoteaccount.py",
>  line 642, in wait_until
> return wait_until(lambda: self.acct.ssh("tail -c +%d %s | grep '%s'" % 
> (self.offset+1, self.log, pattern), allow_fail=True) == 0, **kwargs)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka-0.10.2/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.6.0-py2.7.egg/ducktape/utils/util.py",
>  line 36, in wait_until
> {noformat}
> Logs:
> {noformat}
> ==> ./KafkaService-0-140646398705744/worker9/server-start-stdout-stderr.log 
> <==
> [2017-03-21 07:35:18,250] DEBUG Leaving process event 
> (org.I0Itec.zkclient.ZkClient)
> [2017-03-21 07:35:18,250] INFO EventThread shut down 
> (org.apache.zookeeper.ClientCnxn)
> [2017-03-21 07:35:18,250] INFO [Kafka Server 2], shut down completed 
> (kafka.server.KafkaServer)
> Error: Exception thrown by the agent : java.rmi.server.ExportException: Port 
> already in use: 9192; nested exception is: 
>   java.net.BindException: Address already in use
> {noformat}
> That's from starting the upgraded broker, which seems to indicate that 
> 0.8.2.2 was not properly shut down or has its RMI port in the close-wait 
> state.
> Since there probably isn't much to do about 0.8.2.2 the test should probably 
> be hardened to either select a random port, or wait for lingering port to 
> become available (can use netstat for that).
> This earlier failrue from th

[jira] [Created] (KAFKA-6616) kafka-merge-pr.py should use GitHub's REST API to merge

[Ismael Juma (JIRA)]

Ismael Juma created KAFKA-6616:
--

 Summary: kafka-merge-pr.py should use GitHub's REST API to merge
 Key: KAFKA-6616
 URL: https://issues.apache.org/jira/browse/KAFKA-6616
 Project: Kafka
  Issue Type: Improvement
Reporter: Ismael Juma


The merge script currently squashes the commits in the pull request locally and 
then merges it to the target branch. It can also cherry-pick it to other 
branches. The downside is that GitHub doesn't know that the pull request has 
been merged. As a workaround, the script includes a keyword in the commit 
message to close the pull request. Since the merged commit is different to the 
pull request branch, GitHub assumes that the PR was not merged.

[~hachikuji] suggested that an API may be available that mimics what the GitHub 
merge button does. And he is correct. Given our recent transition to GitBox, 
committers have write access to GitHub, so it's feasible to update the merge 
script to do this. Rough steps:
 # Replace local squashing and merging with GitHub REST API for merging 
([https://developer.github.com/v3/pulls/#merge-a-pull-request-merge-button)]
 # After the merge, pull changes from target branch and offer the option to 
cherry-pick to other branches (i.e. the code may have to be updated a little 
for the rest of the workflow to work).
 # Update wiki documentation and code to state that GITHUB_OAUTH_KEY must be 
set (it's currently optional since we don't rely on any operations that require 
authentication).
 # Update wiki documentation to remove the main downside for using the merge 
script and perhaps to recommend it.

Documentation: 
https://cwiki.apache.org/confluence/display/KAFKA/Merging+Github+Pull+Requests



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] KIP-186: Increase offsets retention default to 7 days

[Ismael Juma]

+1 (binding)

Ismael

On Mon, Mar 5, 2018 at 11:35 AM, Ewen Cheslack-Postava 
wrote:

> I'd like to kick off voting for KIP-186:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 186%3A+Increase+offsets+retention+default+to+7+days
>
> This is the trivial fix that people in the DISCUSS thread were in favor of.
> There are some ideas for further refinements, but I think we can follow up
> with those in subsequent KIPs, see the discussion thread for details. Also
> note this is related, but complementary, to
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 211%3A+Revise+Expiration+Semantics+of+Consumer+Group+Offsets
> .
>
> And of course +1 (binding) from me.
>
> Thanks,
> Ewen
>

Re: [VOTE] 1.0.1 RC2

[Ismael Juma]

Thanks for running the release Ewen!

Ismael

On Fri, Mar 2, 2018 at 10:10 AM, Ewen Cheslack-Postava 
wrote:

> Thanks everyone for voting. This passes with 3 binding +1, 5 non-binding
> +1, and no dissenting votes.
>
> I'll work on getting the release finalized and send out an announcement
> when it is ready.
>
> -Ewen
>
> On Tue, Feb 27, 2018 at 11:18 PM, Jason Gustafson 
> wrote:
>
> > +1. Verified artifacts and ran the basic quickstart.
> >
> > -Jason
> >
> > On Mon, Feb 26, 2018 at 1:08 AM, Manikumar 
> > wrote:
> >
> > > +1 (non-binding)
> > > Built src and ran tests
> > > Ran core quick start
> > >
> > > On Sat, Feb 24, 2018 at 8:44 PM, Jakub Scholz  wrote:
> > >
> > > > +1 (non-binding) ... I used the Scala 2.12 binaries and run my tests
> > with
> > > > producers / consumers.
> > > >
> > > > On Thu, Feb 22, 2018 at 1:06 AM, Ewen Cheslack-Postava <
> > > e...@confluent.io>
> > > > wrote:
> > > >
> > > > > Hello Kafka users, developers and client-developers,
> > > > >
> > > > > This is the third candidate for release of Apache Kafka 1.0.1.
> > > > >
> > > > > This is a bugfix release for the 1.0 branch that was first released
> > > with
> > > > > 1.0.0 about 3 months ago. We've fixed 49 issues since that release.
> > > Most
> > > > of
> > > > > these are non-critical, but in aggregate these fixes will have
> > > > significant
> > > > > impact. A few of the more significant fixes include:
> > > > >
> > > > > * KAFKA-6277: Make loadClass thread-safe for class loaders of
> Connect
> > > > > plugins
> > > > > * KAFKA-6185: Selector memory leak with high likelihood of OOM in
> > case
> > > of
> > > > > down conversion
> > > > > * KAFKA-6269: KTable state restore fails after rebalance
> > > > > * KAFKA-6190: GlobalKTable never finishes restoring when consuming
> > > > > transactional messages
> > > > > * KAFKA-6529: Stop file descriptor leak when client disconnects
> with
> > > > staged
> > > > > receives
> > > > > * KAFKA-6238: Issues with protocol version when applying a rolling
> > > > upgrade
> > > > > to 1.0.0
> > > > >
> > > > > Release notes for the 1.0.1 release:
> > > > > http://home.apache.org/~ewencp/kafka-1.0.1-rc2/RELEASE_NOTES.html
> > > > >
> > > > > *** Please download, test and vote by Saturday Feb 24, 9pm PT ***
> > > > >
> > > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > > http://kafka.apache.org/KEYS
> > > > >
> > > > > * Release artifacts to be voted upon (source and binary):
> > > > > http://home.apache.org/~ewencp/kafka-1.0.1-rc2/
> > > > >
> > > > > * Maven artifacts to be voted upon:
> > > > > https://repository.apache.org/content/groups/staging/
> > > > >
> > > > > * Javadoc:
> > > > > http://home.apache.org/~ewencp/kafka-1.0.1-rc2/javadoc/
> > > > >
> > > > > * Tag to be voted upon (off 1.0 branch) is the 1.0.1 tag:
> > > > > https://github.com/apache/kafka/tree/1.0.1-rc2
> > > > >
> > > > > * Documentation:
> > > > > http://kafka.apache.org/10/documentation.html
> > > > >
> > > > > * Protocol:
> > > > > http://kafka.apache.org/10/protocol.html
> > > > >
> > > > > /**
> > > > >
> > > > > Thanks,
> > > > > Ewen Cheslack-Postava
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] KIP-171 - Extend Consumer Group Reset Offset for Stream Application

[Ismael Juma]

Hi Guozhang,

To clarify my comment: any change with a backwards compatibility impact
should be mentioned in the "Compatibility, Deprecation, and Migration Plan"
section (in addition to the deprecation period and only happening in a
major release as you said).

Ismael

On Thu, Feb 22, 2018 at 11:10 AM, Guozhang Wang  wrote:

> Just to clarify, the KIP itself has mentioned about the change so the PR
> was not un-intentional:
>
> "
>
> 3. Keep execution parameters uniform between both tools: It will execute by
> default, and have a `dry-run` parameter just show the results. This will
> involve change current `ConsumerGroupCommand` to change execution options.
>
> "
>
> We were agreed that the proposed change is better than the current status,
> since may people not using "--execute" on consumer reset tool were actually
> surprised that nothing gets executed. What we were concerning as a
> hind-sight is that instead of doing such change in a minor release like
> 1.1, we should consider only doing that in the next major release as it
> breaks compatibility. In the past when we are going to remove / replace
> certain option we would first add a going-to-be-deprecated warning in the
> previous releases until it was finally removed. So Jason's suggestion is to
> do the same: we are not reverting this change forever, but trying to delay
> it after 1.1.
>
>
> Guozhang
>
>
> On Thu, Feb 22, 2018 at 10:56 AM, Colin McCabe  wrote:
>
> > Perhaps, if the user doesn't pass the --execute flag, the tool should
> > print a prompt like "would you like to perform this reset?" and wait for
> a
> > Y / N (or yes or no) input from the command-line.  Then, if the --execute
> > flag is passed, we skip this.  That seems 99% compatible, and also
> > accomplishes the goal of making the tool less confusing.
> >
> > best,
> > Colin
> >
> >
> > On Thu, Feb 22, 2018, at 10:23, Ismael Juma wrote:
> > > Yes, let's revert the incompatible changes. There was no mention of
> > > compatibility impact on the KIP and we should ensure that is the case
> for
> > > 1.1.0.
> > >
> > > Ismael
> > >
> > > On Thu, Feb 22, 2018 at 9:55 AM, Jason Gustafson 
> > wrote:
> > >
> > > > I know it's a been a while since this vote passed, but I think we
> need
> > to
> > > > reconsider the incompatible changes to the consumer reset tool.
> > > > Specifically, we have removed the --execute option without
> deprecating
> > it
> > > > first, and we have changed the default behavior to execute rather
> than
> > do a
> > > > dry run. The latter in particular seems dangerous since users who
> were
> > > > previously using the default behavior to view offsets will now
> suddenly
> > > > find the offsets already committed. As far as I can tell, this change
> > was
> > > > done mostly for cosmetic reasons. Without a compelling reason, I
> think
> > we
> > > > should err on the side of maintaining compatibility. At a minimum, if
> > we
> > > > really want to break compatibility, we should wait for the next major
> > > > release.
> > > >
> > > > Note that I have submitted a patch to revert this change here:
> > > > https://github.com/apache/kafka/pull/4611.
> > > >
> > > > Thoughts?
> > > >
> > > > Thanks,
> > > > Jason
> > > >
> > > >
> > > >
> > > > On Tue, Nov 14, 2017 at 3:26 AM, Jorge Esteban Quilcate Otoya <
> > > > quilcate.jo...@gmail.com> wrote:
> > > >
> > > > > Thanks to everyone for your feedback.
> > > > >
> > > > > KIP has been accepted and discussion is moved to PR.
> > > > >
> > > > > Cheers,
> > > > > Jorge.
> > > > >
> > > > > El lun., 6 nov. 2017 a las 17:31, Rajini Sivaram (<
> > > > rajinisiva...@gmail.com
> > > > > >)
> > > > > escribió:
> > > > >
> > > > > > +1 (binding)
> > > > > > Thanks for the KIP,  Jorge.
> > > > > >
> > > > > > Regards,
> > > > > >
> > > > > > Rajini
> > > > > >
> > > > > > On Tue, Oct 31, 2017 at 9:58 AM, Damian Guy <
> damian@gmail.com>
> > > > > wrote:
> > > > > >
> > > 

Re: [VOTE] KIP-171 - Extend Consumer Group Reset Offset for Stream Application

[Ismael Juma]

Yes, let's revert the incompatible changes. There was no mention of
compatibility impact on the KIP and we should ensure that is the case for
1.1.0.

Ismael

On Thu, Feb 22, 2018 at 9:55 AM, Jason Gustafson  wrote:

> I know it's a been a while since this vote passed, but I think we need to
> reconsider the incompatible changes to the consumer reset tool.
> Specifically, we have removed the --execute option without deprecating it
> first, and we have changed the default behavior to execute rather than do a
> dry run. The latter in particular seems dangerous since users who were
> previously using the default behavior to view offsets will now suddenly
> find the offsets already committed. As far as I can tell, this change was
> done mostly for cosmetic reasons. Without a compelling reason, I think we
> should err on the side of maintaining compatibility. At a minimum, if we
> really want to break compatibility, we should wait for the next major
> release.
>
> Note that I have submitted a patch to revert this change here:
> https://github.com/apache/kafka/pull/4611.
>
> Thoughts?
>
> Thanks,
> Jason
>
>
>
> On Tue, Nov 14, 2017 at 3:26 AM, Jorge Esteban Quilcate Otoya <
> quilcate.jo...@gmail.com> wrote:
>
> > Thanks to everyone for your feedback.
> >
> > KIP has been accepted and discussion is moved to PR.
> >
> > Cheers,
> > Jorge.
> >
> > El lun., 6 nov. 2017 a las 17:31, Rajini Sivaram (<
> rajinisiva...@gmail.com
> > >)
> > escribió:
> >
> > > +1 (binding)
> > > Thanks for the KIP,  Jorge.
> > >
> > > Regards,
> > >
> > > Rajini
> > >
> > > On Tue, Oct 31, 2017 at 9:58 AM, Damian Guy 
> > wrote:
> > >
> > > > Thanks for the KIP - +1 (binding)
> > > >
> > > > On Mon, 23 Oct 2017 at 18:39 Guozhang Wang 
> wrote:
> > > >
> > > > > Thanks Jorge for driving this KIP! +1 (binding).
> > > > >
> > > > >
> > > > > Guozhang
> > > > >
> > > > > On Mon, Oct 16, 2017 at 2:11 PM, Bill Bejeck 
> > > wrote:
> > > > >
> > > > > > +1
> > > > > >
> > > > > > Thanks,
> > > > > > Bill
> > > > > >
> > > > > > On Fri, Oct 13, 2017 at 6:36 PM, Ted Yu 
> > wrote:
> > > > > >
> > > > > > > +1
> > > > > > >
> > > > > > > On Fri, Oct 13, 2017 at 3:32 PM, Matthias J. Sax <
> > > > > matth...@confluent.io>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > +1
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On 9/11/17 3:04 PM, Jorge Esteban Quilcate Otoya wrote:
> > > > > > > > > Hi All,
> > > > > > > > >
> > > > > > > > > It seems that there is no further concern with the KIP-171.
> > > > > > > > > At this point we would like to start the voting process.
> > > > > > > > >
> > > > > > > > > The KIP can be found here:
> > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > > > > 171+-+Extend+Consumer+Group+Reset+Offset+for+Stream+
> > Application
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Thanks!
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > -- Guozhang
> > > > >
> > > >
> > >
> >
>