[jira] [Resolved] (KAFKA-6134) High memory usage on controller during partition reassignment

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-6134.

Resolution: Fixed

> High memory usage on controller during partition reassignment
> -
>
> Key: KAFKA-6134
> URL: https://issues.apache.org/jira/browse/KAFKA-6134
> Project: Kafka
>  Issue Type: Bug
>  Components: controller
>Affects Versions: 0.11.0.0, 0.11.0.1
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Critical
>  Labels: regression
> Fix For: 1.0.0, 0.11.0.2
>
> Attachments: Screen Shot 2017-10-26 at 3.05.40 PM.png
>
>
> We've had a couple users reporting spikes in memory usage when the controller 
> is performing partition reassignment in 0.11. After investigation, we found 
> that the controller event queue was using most of the retained memory. In 
> particular, we found several thousand {{PartitionReassignment}} objects, each 
> one containing one fewer partition than the previous one (see the attached 
> image).
> From the code, it seems clear why this is happening. We have a watch on the 
> partition reassignment path which adds the {{PartitionReassignment}} object 
> to the event queue:
> {code}
>   override def handleDataChange(dataPath: String, data: Any): Unit = {
> val partitionReassignment = 
> ZkUtils.parsePartitionReassignmentData(data.toString)
> eventManager.put(controller.PartitionReassignment(partitionReassignment))
>   }
> {code}
> In the {{PartitionReassignment}} event handler, we iterate through all of the 
> partitions in the reassignment. After we complete reassignment for each 
> partition, we remove that partition and update the node in zookeeper. 
> {code}
> // remove this partition from that list
> val updatedPartitionsBeingReassigned = partitionsBeingReassigned - 
> topicAndPartition
> // write the new list to zookeeper
>   
> zkUtils.updatePartitionReassignmentData(updatedPartitionsBeingReassigned.mapValues(_.newReplicas))
> {code}
> This triggers the handler above which adds a new event in the queue. So what 
> you get is an n^2 increase in memory where n is the number of partitions.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] kafka pull request #4141: KAFKA-6134: Read partition reassignment lazily on ...

[hachikuji]

Github user hachikuji closed the pull request at:

https://github.com/apache/kafka/pull/4141


---

Re: [DISCUSS] KIP-213 Support non-key joining in KTable

[Ted Yu]

Do you mind addressing my previous comments ?

http://search-hadoop.com/m/Kafka/uyzND1hzF8SRzUqb?subj=Re+DISCUSS+KIP+213+Support+non+key+joining+in+KTable

On Thu, Oct 26, 2017 at 9:38 PM, Jan Filipiak 
wrote:

> Hello everyone,
>
> this is the new discussion thread after the ID-clash.
>
> Best
> Jan
>
> __
>
>
> Hello Kafka-users,
>
> I want to continue with the development of KAFKA-3705, which allows the
> Streams DSL to perform KTableKTable-Joins when the KTables have a
> one-to-many relationship.
> To make sure we cover the requirements of as many users as possible and
> have a good solution afterwards I invite everyone to read through the KIP I
> put together and discuss it here in this Thread.
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-213+
> Support+non-key+joining+in+KTable
> https://issues.apache.org/jira/browse/KAFKA-3705
> https://github.com/apache/kafka/pull/3720
>
> I think a public discussion and vote on a solution is exactly what is
> needed to bring this feauture into kafka-streams. I am looking forward to
> everyones opinion!
>
> Please keep the discussion on the mailing list rather than commenting on
> the wiki (wiki discussions get unwieldy fast).
>
> Best
> Jan
>
>
>

[DISCUSS] KIP-213 Support non-key joining in KTable

[Jan Filipiak]

Hello everyone,

this is the new discussion thread after the ID-clash.

Best
Jan

__


Hello Kafka-users,

I want to continue with the development of KAFKA-3705, which allows the 
Streams DSL to perform KTableKTable-Joins when the KTables have a 
one-to-many relationship.
To make sure we cover the requirements of as many users as possible and 
have a good solution afterwards I invite everyone to read through the 
KIP I put together and discuss it here in this Thread.


https://cwiki.apache.org/confluence/display/KAFKA/KIP-213+Support+non-key+joining+in+KTable 


https://issues.apache.org/jira/browse/KAFKA-3705
https://github.com/apache/kafka/pull/3720

I think a public discussion and vote on a solution is exactly what is 
needed to bring this feauture into kafka-streams. I am looking forward 
to everyones opinion!


Please keep the discussion on the mailing list rather than commenting on 
the wiki (wiki discussions get unwieldy fast).


Best
Jan

Jenkins build is back to normal : kafka-trunk-jdk8 #2171

[Apache Jenkins Server]

See 

Jenkins build is back to normal : kafka-trunk-jdk9 #154

[Apache Jenkins Server]

See 

Jenkins build is back to normal : kafka-0.11.0-jdk7 #328

[Apache Jenkins Server]

See 

Jenkins build is back to normal : kafka-trunk-jdk7 #2926

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-215: Add topic regex support for Connect sinks

[Ewen Cheslack-Postava]

It's fine to be more detailed, but ConfigException is already implied for
all other config issues as well.

Default could be either null or just empty string. re: alternatives, if you
wanted to be slightly more detailed (though still a bit vague) re:
supported syntax, you could just say that while Pattern is used, we only
guarantee support for common regular expression syntax. Not sure if there's
a good way of defining what "common" syntax is.

Otherwise LGTM, and thanks for helping fill in a longstanding gap!

-Ewen

On Thu, Oct 26, 2017 at 7:56 PM, Ted Yu  wrote:

> bq. Users may specify only one of 'topics' or 'topics.pattern'.
>
> Can you fill in which exception would be thrown if both of them are
> specified
> ?
>
> Cheers
>
> On Thu, Oct 26, 2017 at 6:27 PM, Jeff Klukas  wrote:
>
> > Looking for feedback on
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 215%3A+Add+topic+regex+support+for+Connect+sinks
> >
>

Build failed in Jenkins: kafka-1.0-jdk7 #59

[Apache Jenkins Server]

See 


Changes:

[ismael] KAFKA-6131; Use atomic putIfAbsent to create txn marker queues

--
[...truncated 370.96 KB...]

kafka.server.ServerGenerateBrokerIdTest > testUserConfigAndGeneratedBrokerId 
STARTED

kafka.server.ServerGenerateBrokerIdTest > testUserConfigAndGeneratedBrokerId 
PASSED

kafka.server.ServerGenerateBrokerIdTest > 
testConsistentBrokerIdFromUserConfigAndMetaProps STARTED

kafka.server.ServerGenerateBrokerIdTest > 
testConsistentBrokerIdFromUserConfigAndMetaProps PASSED

kafka.server.DelayedOperationTest > testRequestPurge STARTED

kafka.server.DelayedOperationTest > testRequestPurge PASSED

kafka.server.DelayedOperationTest > testRequestExpiry STARTED

kafka.server.DelayedOperationTest > testRequestExpiry PASSED

kafka.server.DelayedOperationTest > 
shouldReturnNilOperationsOnCancelForKeyWhenKeyDoesntExist STARTED

kafka.server.DelayedOperationTest > 
shouldReturnNilOperationsOnCancelForKeyWhenKeyDoesntExist PASSED

kafka.server.DelayedOperationTest > testDelayedOperationLockOverride STARTED

kafka.server.DelayedOperationTest > testDelayedOperationLockOverride PASSED

kafka.server.DelayedOperationTest > 
shouldCancelForKeyReturningCancelledOperations STARTED

kafka.server.DelayedOperationTest > 
shouldCancelForKeyReturningCancelledOperations PASSED

kafka.server.DelayedOperationTest > testRequestSatisfaction STARTED

kafka.server.DelayedOperationTest > testRequestSatisfaction PASSED

kafka.server.DelayedOperationTest > testDelayedOperationLock STARTED

kafka.server.DelayedOperationTest > testDelayedOperationLock PASSED

kafka.server.MultipleListenersWithDefaultJaasContextTest > testProduceConsume 
STARTED

kafka.server.MultipleListenersWithDefaultJaasContextTest > testProduceConsume 
PASSED

kafka.server.ThrottledResponseExpirationTest > testThrottledRequest STARTED

kafka.server.ThrottledResponseExpirationTest > testThrottledRequest PASSED

kafka.server.ThrottledResponseExpirationTest > testExpire STARTED

kafka.server.ThrottledResponseExpirationTest > testExpire PASSED

kafka.server.ReplicaManagerQuotasTest > shouldGetBothMessagesIfQuotasAllow 
STARTED

kafka.server.ReplicaManagerQuotasTest > shouldGetBothMessagesIfQuotasAllow 
PASSED

kafka.server.ReplicaManagerQuotasTest > 
shouldExcludeSubsequentThrottledPartitions STARTED

kafka.server.ReplicaManagerQuotasTest > 
shouldExcludeSubsequentThrottledPartitions PASSED

kafka.server.ReplicaManagerQuotasTest > 
shouldGetNoMessagesIfQuotasExceededOnSubsequentPartitions STARTED

kafka.server.ReplicaManagerQuotasTest > 
shouldGetNoMessagesIfQuotasExceededOnSubsequentPartitions PASSED

kafka.server.ReplicaManagerQuotasTest > shouldIncludeInSyncThrottledReplicas 
STARTED

kafka.server.ReplicaManagerQuotasTest > shouldIncludeInSyncThrottledReplicas 
PASSED

kafka.server.SaslApiVersionsRequestTest > 
testApiVersionsRequestWithUnsupportedVersion STARTED

kafka.server.SaslApiVersionsRequestTest > 
testApiVersionsRequestWithUnsupportedVersion PASSED

kafka.server.SaslApiVersionsRequestTest > 
testApiVersionsRequestBeforeSaslHandshakeRequest STARTED

kafka.server.SaslApiVersionsRequestTest > 
testApiVersionsRequestBeforeSaslHandshakeRequest PASSED

kafka.server.SaslApiVersionsRequestTest > 
testApiVersionsRequestAfterSaslHandshakeRequest STARTED

kafka.server.SaslApiVersionsRequestTest > 
testApiVersionsRequestAfterSaslHandshakeRequest PASSED

kafka.server.ControlledShutdownLeaderSelectorTest > testSelectLeader STARTED

kafka.server.ControlledShutdownLeaderSelectorTest > testSelectLeader PASSED

kafka.server.MetadataCacheTest > 
getTopicMetadataWithNonSupportedSecurityProtocol STARTED

kafka.server.MetadataCacheTest > 
getTopicMetadataWithNonSupportedSecurityProtocol PASSED

kafka.server.MetadataCacheTest > getTopicMetadataIsrNotAvailable STARTED

kafka.server.MetadataCacheTest > getTopicMetadataIsrNotAvailable PASSED

kafka.server.MetadataCacheTest > getTopicMetadata STARTED

kafka.server.MetadataCacheTest > getTopicMetadata PASSED

kafka.server.MetadataCacheTest > getTopicMetadataReplicaNotAvailable STARTED

kafka.server.MetadataCacheTest > getTopicMetadataReplicaNotAvailable PASSED

kafka.server.MetadataCacheTest > getTopicMetadataPartitionLeaderNotAvailable 
STARTED

kafka.server.MetadataCacheTest > getTopicMetadataPartitionLeaderNotAvailable 
PASSED

kafka.server.MetadataCacheTest > getAliveBrokersShouldNotBeMutatedByUpdateCache 
STARTED

kafka.server.MetadataCacheTest > getAliveBrokersShouldNotBeMutatedByUpdateCache 
PASSED

kafka.server.MetadataCacheTest > getTopicMetadataNonExistingTopics STARTED

kafka.server.MetadataCacheTest > getTopicMetadataNonExistingTopics PASSED

kafka.server.FetchRequestTest > testBrokerRespectsPartitionsOrderAndSizeLimits 
STARTED

kafka.server.FetchRequestTest > testBrokerRespectsPartitionsOrderAndSizeLimits 
PASSED

kafka.server.FetchRequestTest > 

Re: [DISCUSS] KIP-215: Add topic regex support for Connect sinks

[Ted Yu]

bq. Users may specify only one of 'topics' or 'topics.pattern'.

Can you fill in which exception would be thrown if both of them are specified
?

Cheers

On Thu, Oct 26, 2017 at 6:27 PM, Jeff Klukas  wrote:

> Looking for feedback on
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 215%3A+Add+topic+regex+support+for+Connect+sinks
>

Re: Failed to start Apache Kafka

[Ted Yu]

Hmm. You ran into KAFKA-5077.

Can you try 1.0.0 RC3 ?

http://search-hadoop.com/m/Kafka/uyzND18Q3Vh1Q5jUV1?subj=+VOTE+1+0+0+RC3



On Thu, Oct 26, 2017 at 10:08 AM, Naren gs  wrote:

> Hello,
> I am  starter in Apache Kafka, I downloaded the latest version and on
> running the config of zookeeper I get an error as below:
>
> [0.001s][warning][gc] -Xloggc is deprecated. Will use
> -Xlog:gc:/Users/user-mac/Documents/js/kafka_2.11-0.11.
> 0.1/bin/../logs/zookeeper-gc.log instead.
> Unrecognized VM option 'PrintGCDateStamps'
> Error: Could not create the Java Virtual Machine.
> Error: A fatal exception has occurred. Program will exit.
>
> I am using Java 9.
>
> Can you help me to solve this issue.
>
> Regards,
> Naren G S

Failed to start Apache Kafka

[Naren gs]

Hello,
I am  starter in Apache Kafka, I downloaded the latest version and on running 
the config of zookeeper I get an error as below:

[0.001s][warning][gc] -Xloggc is deprecated. Will use 
-Xlog:gc:/Users/user-mac/Documents/js/kafka_2.11-0.11.0.1/bin/../logs/zookeeper-gc.log
 instead.
Unrecognized VM option 'PrintGCDateStamps'
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

I am using Java 9.

Can you help me to solve this issue.

Regards,
Naren G S

[GitHub] kafka pull request #4142: MINOR: Use Scala Future in CoreUtils test

[ijuma]

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/4142

MINOR: Use Scala Future in CoreUtils test

Also rename UtilsTest to CoreUtilsTest and note
that `getOrElseUpdate` has the right behaviour
in Scala 2.12.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka 
use-scala-futures-in-core-utils-test

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4142.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4142


commit 5a1b360f29dbecca0e66e74e0516b51f686baaef
Author: Ismael Juma 
Date:   2017-10-27T02:46:46Z

MINOR: Use Scala Future in CoreUtils test

Also rename UtilsTest to CoreUtilsTest.




---

[DISCUSS] KIP-215: Add topic regex support for Connect sinks

[Jeff Klukas]

Looking for feedback on

https://cwiki.apache.org/confluence/display/KAFKA/KIP-215%3A+Add+topic+regex+support+for+Connect+sinks

[GitHub] kafka pull request #4141: KAFKA-6134: Read partition reassignment lazily on ...

[hachikuji]

GitHub user hachikuji opened a pull request:

https://github.com/apache/kafka/pull/4141

KAFKA-6134: Read partition reassignment lazily on event handling

This patch prevents an O(n^2) increase in memory utilization during 
partition reassignment. Instead of storing the reassigned partitions in the 
`PartitionReassignment` object (which is added after ever partition 
reassignment), we read the data fresh from ZK when processing the event.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hachikuji/kafka KAFKA-6134

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4141.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4141


commit 5131bb19f6fe7fc1939035c48ead052a0ac967a4
Author: Jason Gustafson 
Date:   2017-10-27T02:01:05Z

KAFKA-6134: Read partition reassignment lazily on event handling




---

[jira] [Resolved] (KAFKA-6131) Transaction markers are sometimes discarded if txns complete concurrently

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-6131.

   Resolution: Fixed
Fix Version/s: 1.1.0
   0.11.0.2
   1.0.0

> Transaction markers are sometimes discarded if txns complete concurrently
> -
>
> Key: KAFKA-6131
> URL: https://issues.apache.org/jira/browse/KAFKA-6131
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.11.0.1, 1.0.0
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
> Fix For: 1.0.0, 0.11.0.2, 1.1.0
>
>
> Concurrent tests being added under KAFKA-6096 for transaction coordinator 
> fail to complete some transactions when multiple transactions are completed 
> concurrently.
> The problem is with the following code snippet - there are two very similar 
> uses of concurrent map in {{TransactionMarkerChannelManager}} and the test 
> fails because some transaction markers are discarded. {{getOrElseUpdate}} in 
> scala maps are not atomic. The test passes consistently with one thread.
> {quote}
> val markersQueuePerBroker: concurrent.Map[Int, TxnMarkerQueue] = new 
> ConcurrentHashMap[Int, TxnMarkerQueue]().asScala
> val brokerRequestQueue = markersQueuePerBroker.getOrElseUpdate(brokerId, new 
> TxnMarkerQueue(broker))
> {quote}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] kafka pull request #4140: KAFKA-6131: Use atomic putIfAbsent to create txn m...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/4140


---

[jira] [Created] (KAFKA-6135) TransactionsTest#testFencingOnCommit may fail due to unexpected KafkaException

[Ted Yu (JIRA)]

Ted Yu created KAFKA-6135:
-

 Summary: TransactionsTest#testFencingOnCommit may fail due to 
unexpected KafkaException
 Key: KAFKA-6135
 URL: https://issues.apache.org/jira/browse/KAFKA-6135
 Project: Kafka
  Issue Type: Bug
Reporter: Ted Yu


>From 
>https://builds.apache.org/job/kafka-pr-jdk9-scala2.12/2293/testReport/junit/kafka.api/TransactionsTest/testFencingOnCommit/
> :
{code}
org.scalatest.junit.JUnitTestFailedError: Got an unexpected exception from a 
fenced producer.
at 
org.scalatest.junit.AssertionsForJUnit.newAssertionFailedException(AssertionsForJUnit.scala:100)
at 
org.scalatest.junit.AssertionsForJUnit.newAssertionFailedException$(AssertionsForJUnit.scala:99)
at 
org.scalatest.junit.JUnitSuite.newAssertionFailedException(JUnitSuite.scala:71)
at org.scalatest.Assertions.fail(Assertions.scala:1105)
at org.scalatest.Assertions.fail$(Assertions.scala:1101)
at org.scalatest.junit.JUnitSuite.fail(JUnitSuite.scala:71)
at 
kafka.api.TransactionsTest.testFencingOnCommit(TransactionsTest.scala:319)
...
Caused by: org.apache.kafka.common.KafkaException: Cannot execute transactional 
method because we are in an error state
at 
org.apache.kafka.clients.producer.internals.TransactionManager.maybeFailWithError(TransactionManager.java:782)
at 
org.apache.kafka.clients.producer.internals.TransactionManager.beginCommit(TransactionManager.java:220)
at 
org.apache.kafka.clients.producer.KafkaProducer.commitTransaction(KafkaProducer.java:617)
at 
kafka.api.TransactionsTest.testFencingOnCommit(TransactionsTest.scala:313)
... 48 more
Caused by: org.apache.kafka.common.errors.ProducerFencedException: Producer 
attempted an operation with an old epoch. Either there is a newer producer with 
the same transactionalId, or the producer's transaction has been expired by the 
broker.
{code}
Confirmed with [~apurva] that the above would not be covered by his fix for 
KAFKA-6119

Temporarily marking this as bug.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: kafka-trunk-jdk8 #2170

[Apache Jenkins Server]

See 


Changes:

[wangguoz] MINOR: Improve a Windows quickstart instruction

--
[...truncated 3.31 MB...]

kafka.security.auth.ResourceTypeTest > testJavaConversions STARTED

kafka.security.auth.ResourceTypeTest > testJavaConversions PASSED

kafka.security.auth.ResourceTypeTest > testFromString STARTED

kafka.security.auth.ResourceTypeTest > testFromString PASSED

kafka.security.auth.OperationTest > testJavaConversions STARTED

kafka.security.auth.OperationTest > testJavaConversions PASSED

kafka.security.auth.AclTest > testAclJsonConversion STARTED

kafka.security.auth.AclTest > testAclJsonConversion PASSED

kafka.security.auth.ZkAuthorizationTest > testIsZkSecurityEnabled STARTED

kafka.security.auth.ZkAuthorizationTest > testIsZkSecurityEnabled PASSED

kafka.security.auth.ZkAuthorizationTest > testZkUtils STARTED

kafka.security.auth.ZkAuthorizationTest > testZkUtils PASSED

kafka.security.auth.ZkAuthorizationTest > testZkAntiMigration STARTED

kafka.security.auth.ZkAuthorizationTest > testZkAntiMigration PASSED

kafka.security.auth.ZkAuthorizationTest > testZkMigration STARTED

kafka.security.auth.ZkAuthorizationTest > testZkMigration PASSED

kafka.security.auth.ZkAuthorizationTest > testChroot STARTED

kafka.security.auth.ZkAuthorizationTest > testChroot PASSED

kafka.security.auth.ZkAuthorizationTest > testDelete STARTED

kafka.security.auth.ZkAuthorizationTest > testDelete PASSED

kafka.security.auth.ZkAuthorizationTest > testDeleteRecursive STARTED

kafka.security.auth.ZkAuthorizationTest > testDeleteRecursive PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testAllowAllAccess STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testAllowAllAccess PASSED

kafka.security.auth.SimpleAclAuthorizerTest > 
testLocalConcurrentModificationOfResourceAcls STARTED

kafka.security.auth.SimpleAclAuthorizerTest > 
testLocalConcurrentModificationOfResourceAcls PASSED

kafka.security.auth.SimpleAclAuthorizerTest > 
testHighConcurrencyDeletionOfResourceAcls STARTED

kafka.security.auth.SimpleAclAuthorizerTest > 
testHighConcurrencyDeletionOfResourceAcls PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testNoAclFound STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testNoAclFound PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testAclInheritance STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testAclInheritance PASSED

kafka.security.auth.SimpleAclAuthorizerTest > 
testDistributedConcurrentModificationOfResourceAcls STARTED

kafka.security.auth.SimpleAclAuthorizerTest > 
testDistributedConcurrentModificationOfResourceAcls PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testAclManagementAPIs STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testAclManagementAPIs PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testWildCardAcls STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testWildCardAcls PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testTopicAcl STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testTopicAcl PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testSuperUserHasAccess STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testSuperUserHasAccess PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testDenyTakesPrecedence STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testDenyTakesPrecedence PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testNoAclFoundOverride STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testNoAclFoundOverride PASSED

kafka.security.auth.SimpleAclAuthorizerTest > 
testHighConcurrencyModificationOfResourceAcls STARTED

kafka.security.auth.SimpleAclAuthorizerTest > 
testHighConcurrencyModificationOfResourceAcls PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testLoadCache STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testLoadCache PASSED

kafka.integration.PrimitiveApiTest > testMultiProduce STARTED

kafka.integration.PrimitiveApiTest > testMultiProduce PASSED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch STARTED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch PASSED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize 
STARTED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize PASSED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests STARTED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests PASSED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch STARTED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch PASSED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression STARTED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression PASSED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic STARTED

kafka.integration.PrimitiveApiTest 

Build failed in Jenkins: kafka-trunk-jdk7 #2925

[Apache Jenkins Server]

See 


Changes:

[wangguoz] MINOR: Improve a Windows quickstart instruction

--
[...truncated 1.83 MB...]
java.lang.NullPointerException
at 
hudson.plugins.toolenv.ToolEnvBuildWrapper$1.buildEnvVars(ToolEnvBuildWrapper.java:46)
at hudson.model.AbstractBuild.getEnvironment(AbstractBuild.java:887)
at hudson.plugins.git.GitSCM.getParamExpandedRepos(GitSCM.java:419)
at 
hudson.plugins.git.GitSCM.compareRemoteRevisionWithImpl(GitSCM.java:627)
at hudson.plugins.git.GitSCM.compareRemoteRevisionWith(GitSCM.java:592)
at hudson.scm.SCM.compareRemoteRevisionWith(SCM.java:391)
at hudson.scm.SCM.poll(SCM.java:408)
at hudson.model.AbstractProject._poll(AbstractProject.java:1394)
at hudson.model.AbstractProject.poll(AbstractProject.java:1297)
at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:594)
at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:640)
at 
hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:119)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

org.apache.kafka.streams.KafkaStreamsTest > shouldReturnThreadMetadata PASSED

org.apache.kafka.streams.KafkaStreamsTest > testCloseIsIdempotent STARTED

org.apache.kafka.streams.KafkaStreamsTest > testCloseIsIdempotent PASSED

org.apache.kafka.streams.KafkaStreamsTest > testCannotCleanupWhileRunning 
STARTED

org.apache.kafka.streams.KafkaStreamsTest > testCannotCleanupWhileRunning PASSED

org.apache.kafka.streams.KafkaStreamsTest > testStateThreadClose STARTED

org.apache.kafka.streams.KafkaStreamsTest > testStateThreadClose PASSED

org.apache.kafka.streams.KafkaStreamsTest > testStateChanges STARTED

org.apache.kafka.streams.KafkaStreamsTest > testStateChanges PASSED

org.apache.kafka.streams.KafkaStreamsTest > testCannotStartTwice STARTED

org.apache.kafka.streams.KafkaStreamsTest > testCannotStartTwice PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldResetToDefaultIfProducerEnableIdempotenceIsOverriddenIfEosEnabled STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldResetToDefaultIfProducerEnableIdempotenceIsOverriddenIfEosEnabled PASSED

org.apache.kafka.streams.StreamsConfigTest > shouldUseNewConfigsWhenPresent 
STARTED

org.apache.kafka.streams.StreamsConfigTest > shouldUseNewConfigsWhenPresent 
PASSED

org.apache.kafka.streams.StreamsConfigTest > testGetConsumerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > testGetConsumerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > shouldAcceptAtLeastOnce STARTED

org.apache.kafka.streams.StreamsConfigTest > shouldAcceptAtLeastOnce PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldUseCorrectDefaultsWhenNoneSpecified STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldUseCorrectDefaultsWhenNoneSpecified PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldAllowSettingProducerEnableIdempotenceIfEosDisabled STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldAllowSettingProducerEnableIdempotenceIfEosDisabled PASSED

org.apache.kafka.streams.StreamsConfigTest > defaultSerdeShouldBeConfigured 
STARTED

org.apache.kafka.streams.StreamsConfigTest > defaultSerdeShouldBeConfigured 
PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSetDifferentDefaultsIfEosEnabled STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSetDifferentDefaultsIfEosEnabled PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldNotOverrideUserConfigRetriesIfExactlyOnceEnabled STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldNotOverrideUserConfigRetriesIfExactlyOnceEnabled PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedConsumerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedConsumerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldOverrideStreamsDefaultConsumerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldOverrideStreamsDefaultConsumerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > testGetProducerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > testGetProducerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldAllowSettingProducerMaxInFlightRequestPerConnectionsWhenEosDisabled 
STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldAllowSettingProducerMaxInFlightRequestPerConnectionsWhenEosDisabled PASSED

org.apache.kafka.streams.StreamsConfigTest > 

Jenkins build is back to normal : kafka-trunk-jdk8 #2169

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-6134) High memory usage on controller during partition reassignment

[Jason Gustafson (JIRA)]

Jason Gustafson created KAFKA-6134:
--

 Summary: High memory usage on controller during partition 
reassignment
 Key: KAFKA-6134
 URL: https://issues.apache.org/jira/browse/KAFKA-6134
 Project: Kafka
  Issue Type: Bug
  Components: controller
Affects Versions: 0.11.0.0, 0.11.0.1
Reporter: Jason Gustafson
 Attachments: Screen Shot 2017-10-26 at 3.05.40 PM.png

We've had a couple users reporting spikes in memory usage when the controller 
is performing partition reassignment in 0.11. After investigation, we found 
that the controller event queue was using most of the retained memory. In 
particular, we found several thousand {{PartitionReassignment}} objects, each 
one containing one fewer partition than the previous one:

!Screen Shot 2017-10-26 at 3.05.40 PM.png|thumbnail!.

>From the code, it seems clear why this is happening. We have a watch on the 
>partition reassignment path which adds the {{PartitionReassignment}} object to 
>the event queue:

{code}
  override def handleDataChange(dataPath: String, data: Any): Unit = {
val partitionReassignment = 
ZkUtils.parsePartitionReassignmentData(data.toString)
eventManager.put(controller.PartitionReassignment(partitionReassignment))
  }
{code}

In the {{PartitionReassignment}} event handler, we iterate through all of the 
partitions in the reassignment. After we complete reassignment for each 
partition, we remove that partition and update the node in zookeeper. 

{code}
// remove this partition from that list
val updatedPartitionsBeingReassigned = partitionsBeingReassigned - 
topicAndPartition
// write the new list to zookeeper
  
zkUtils.updatePartitionReassignmentData(updatedPartitionsBeingReassigned.mapValues(_.newReplicas))
{code}

This triggers the handler above which adds a new event in the queue. So what 
you get is an n^2 increase in memory where n is the number of partitions.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is back to normal : kafka-1.0-jdk7 #58

[Apache Jenkins Server]

See 

Build failed in Jenkins: kafka-trunk-jdk7 #2924

[Apache Jenkins Server]

See 


Changes:

[wangguoz] KAFKA-6100: Down-grade RocksDB to 5.7.3

--
[...truncated 1.83 MB...]

org.apache.kafka.streams.kstream.internals.KTableAggregateTest > 
testAggRepartition STARTED

org.apache.kafka.streams.kstream.internals.KTableAggregateTest > 
testAggRepartition PASSED

org.apache.kafka.streams.kstream.internals.KTableAggregateTest > 
testRemoveOldBeforeAddNew STARTED

org.apache.kafka.streams.kstream.internals.KTableAggregateTest > 
testRemoveOldBeforeAddNew PASSED

org.apache.kafka.streams.kstream.internals.KTableAggregateTest > 
testCountCoalesced STARTED

org.apache.kafka.streams.kstream.internals.KTableAggregateTest > 
testCountCoalesced PASSED

org.apache.kafka.streams.kstream.internals.KTableForeachTest > testForeach 
STARTED

org.apache.kafka.streams.kstream.internals.KTableForeachTest > testForeach 
PASSED

org.apache.kafka.streams.kstream.internals.KTableForeachTest > testTypeVariance 
STARTED

org.apache.kafka.streams.kstream.internals.KTableForeachTest > testTypeVariance 
PASSED

org.apache.kafka.streams.kstream.SessionWindowsTest > 
retentionTimeShouldBeGapIfGapIsLargerThanDefaultRetentionTime STARTED

org.apache.kafka.streams.kstream.SessionWindowsTest > 
retentionTimeShouldBeGapIfGapIsLargerThanDefaultRetentionTime PASSED

org.apache.kafka.streams.kstream.SessionWindowsTest > shouldSetWindowGap STARTED

org.apache.kafka.streams.kstream.SessionWindowsTest > shouldSetWindowGap PASSED

org.apache.kafka.streams.kstream.SessionWindowsTest > 
shouldBeEqualWhenGapAndMaintainMsAreTheSame STARTED

org.apache.kafka.streams.kstream.SessionWindowsTest > 
shouldBeEqualWhenGapAndMaintainMsAreTheSame PASSED

org.apache.kafka.streams.kstream.SessionWindowsTest > 
retentionTimeMustNotBeNegative STARTED

org.apache.kafka.streams.kstream.SessionWindowsTest > 
retentionTimeMustNotBeNegative PASSED

org.apache.kafka.streams.kstream.SessionWindowsTest > 
shouldNotBeEqualWhenMaintainMsDifferent STARTED

org.apache.kafka.streams.kstream.SessionWindowsTest > 
shouldNotBeEqualWhenMaintainMsDifferent PASSED

org.apache.kafka.streams.kstream.SessionWindowsTest > windowSizeMustNotBeZero 
STARTED

org.apache.kafka.streams.kstream.SessionWindowsTest > windowSizeMustNotBeZero 
PASSED

org.apache.kafka.streams.kstream.SessionWindowsTest > 
windowSizeMustNotBeNegative STARTED

org.apache.kafka.streams.kstream.SessionWindowsTest > 
windowSizeMustNotBeNegative PASSED

org.apache.kafka.streams.kstream.SessionWindowsTest > 
shouldSetWindowRetentionTime STARTED

org.apache.kafka.streams.kstream.SessionWindowsTest > 
shouldSetWindowRetentionTime PASSED

org.apache.kafka.streams.kstream.SessionWindowsTest > 
shouldNotBeEqualWhenGapIsDifferent STARTED

org.apache.kafka.streams.kstream.SessionWindowsTest > 
shouldNotBeEqualWhenGapIsDifferent PASSED

org.apache.kafka.streams.kstream.WindowTest > 
shouldThrowIfEndIsSmallerThanStart STARTED

org.apache.kafka.streams.kstream.WindowTest > 
shouldThrowIfEndIsSmallerThanStart PASSED

org.apache.kafka.streams.kstream.WindowTest > 
shouldNotBeEqualIfDifferentWindowType STARTED

org.apache.kafka.streams.kstream.WindowTest > 
shouldNotBeEqualIfDifferentWindowType PASSED

org.apache.kafka.streams.kstream.WindowTest > shouldBeEqualIfStartAndEndSame 
STARTED

org.apache.kafka.streams.kstream.WindowTest > shouldBeEqualIfStartAndEndSame 
PASSED

org.apache.kafka.streams.kstream.WindowTest > shouldNotBeEqualIfNull STARTED

org.apache.kafka.streams.kstream.WindowTest > shouldNotBeEqualIfNull PASSED

org.apache.kafka.streams.kstream.WindowTest > shouldThrowIfStartIsNegative 
STARTED

org.apache.kafka.streams.kstream.WindowTest > shouldThrowIfStartIsNegative 
PASSED

org.apache.kafka.streams.kstream.WindowTest > 
shouldNotBeEqualIfStartOrEndIsDifferent STARTED

org.apache.kafka.streams.kstream.WindowTest > 
shouldNotBeEqualIfStartOrEndIsDifferent PASSED

org.apache.kafka.streams.kstream.KStreamBuilderTest > 
shouldAddRegexTopicToLatestAutoOffsetResetList STARTED

org.apache.kafka.streams.kstream.KStreamBuilderTest > 
shouldAddRegexTopicToLatestAutoOffsetResetList PASSED

org.apache.kafka.streams.kstream.KStreamBuilderTest > 
shouldMapStateStoresToCorrectSourceTopics STARTED

org.apache.kafka.streams.kstream.KStreamBuilderTest > 
shouldMapStateStoresToCorrectSourceTopics PASSED

org.apache.kafka.streams.kstream.KStreamBuilderTest > 
shouldAddTableToEarliestAutoOffsetResetList STARTED

org.apache.kafka.streams.kstream.KStreamBuilderTest > 
shouldAddTableToEarliestAutoOffsetResetList PASSED

org.apache.kafka.streams.kstream.KStreamBuilderTest > 
shouldAddTimestampExtractorToTablePerSource STARTED

org.apache.kafka.streams.kstream.KStreamBuilderTest > 
shouldAddTimestampExtractorToTablePerSource PASSED

org.apache.kafka.streams.kstream.KStreamBuilderTest > 
shouldBuildGlobalTopologyWithAllGlobalTables STARTED

org.apache.kafka.streams.kstream.KStreamBuilderTest > 

Build failed in Jenkins: kafka-trunk-jdk9 #153

[Apache Jenkins Server]

See 


Changes:

[wangguoz] KAFKA-6100: Down-grade RocksDB to 5.7.3

[wangguoz] MINOR: Improve a Windows quickstart instruction

--
[...truncated 1.41 MB...]

kafka.log.ProducerStateManagerTest > testBasicIdMapping PASSED

kafka.log.ProducerStateManagerTest > updateProducerTransactionState STARTED

kafka.log.ProducerStateManagerTest > updateProducerTransactionState PASSED

kafka.log.ProducerStateManagerTest > testRecoverFromSnapshot STARTED

kafka.log.ProducerStateManagerTest > testRecoverFromSnapshot PASSED

kafka.log.ProducerStateManagerTest > 
testSequenceNotValidatedForGroupMetadataTopic STARTED

kafka.log.ProducerStateManagerTest > 
testSequenceNotValidatedForGroupMetadataTopic PASSED

kafka.log.ProducerStateManagerTest > 
testLoadFromSnapshotRemovesNonRetainedProducers STARTED

kafka.log.ProducerStateManagerTest > 
testLoadFromSnapshotRemovesNonRetainedProducers PASSED

kafka.log.ProducerStateManagerTest > testFirstUnstableOffset STARTED

kafka.log.ProducerStateManagerTest > testFirstUnstableOffset PASSED

kafka.log.ProducerStateManagerTest > testTxnFirstOffsetMetadataCached STARTED

kafka.log.ProducerStateManagerTest > testTxnFirstOffsetMetadataCached PASSED

kafka.log.ProducerStateManagerTest > testCoordinatorFencedAfterReload STARTED

kafka.log.ProducerStateManagerTest > testCoordinatorFencedAfterReload PASSED

kafka.log.ProducerStateManagerTest > testControlRecordBumpsEpoch STARTED

kafka.log.ProducerStateManagerTest > testControlRecordBumpsEpoch PASSED

kafka.log.ProducerStateManagerTest > 
testAcceptAppendWithoutProducerStateOnReplica STARTED

kafka.log.ProducerStateManagerTest > 
testAcceptAppendWithoutProducerStateOnReplica PASSED

kafka.log.ProducerStateManagerTest > testLoadFromCorruptSnapshotFile STARTED

kafka.log.ProducerStateManagerTest > testLoadFromCorruptSnapshotFile PASSED

kafka.log.ProducerStateManagerTest > testProducerSequenceWrapAround STARTED

kafka.log.ProducerStateManagerTest > testProducerSequenceWrapAround PASSED

kafka.log.ProducerStateManagerTest > testPidExpirationTimeout STARTED

kafka.log.ProducerStateManagerTest > testPidExpirationTimeout PASSED

kafka.log.ProducerStateManagerTest > testAcceptAppendWithSequenceGapsOnReplica 
STARTED

kafka.log.ProducerStateManagerTest > testAcceptAppendWithSequenceGapsOnReplica 
PASSED

kafka.log.ProducerStateManagerTest > testOldEpochForControlRecord STARTED

kafka.log.ProducerStateManagerTest > testOldEpochForControlRecord PASSED

kafka.log.ProducerStateManagerTest > 
testTruncateAndReloadRemovesOutOfRangeSnapshots STARTED

kafka.log.ProducerStateManagerTest > 
testTruncateAndReloadRemovesOutOfRangeSnapshots PASSED

kafka.log.ProducerStateManagerTest > testStartOffset STARTED

kafka.log.ProducerStateManagerTest > testStartOffset PASSED

kafka.log.ProducerStateManagerTest > testProducerSequenceInvalidWrapAround 
STARTED

kafka.log.ProducerStateManagerTest > testProducerSequenceInvalidWrapAround 
PASSED

kafka.log.ProducerStateManagerTest > testTruncateHead STARTED

kafka.log.ProducerStateManagerTest > testTruncateHead PASSED

kafka.log.ProducerStateManagerTest > 
testNonTransactionalAppendWithOngoingTransaction STARTED

kafka.log.ProducerStateManagerTest > 
testNonTransactionalAppendWithOngoingTransaction PASSED

kafka.log.ProducerStateManagerTest > testSkipSnapshotIfOffsetUnchanged STARTED

kafka.log.ProducerStateManagerTest > testSkipSnapshotIfOffsetUnchanged PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[0] STARTED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[0] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[1] STARTED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[1] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[2] STARTED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[2] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[3] STARTED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[3] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[4] STARTED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[4] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[5] STARTED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[5] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[6] STARTED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[6] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[7] STARTED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[7] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[8] STARTED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[8] PASSED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[9] STARTED

kafka.log.BrokerCompressionTest > testBrokerSideCompression[9] PASSED

kafka.log.BrokerCompressionTest > 

Build failed in Jenkins: kafka-1.0-jdk7 #57

[Apache Jenkins Server]

See 


Changes:

[ismael] MINOR: Add missing semicolon to example jaas configuration

--
[...truncated 1.82 MB...]

org.apache.kafka.streams.integration.JoinIntegrationTest > 
testInnerKStreamKStream PASSED

org.apache.kafka.streams.integration.JoinIntegrationTest > 
testOuterKStreamKStream STARTED

org.apache.kafka.streams.integration.JoinIntegrationTest > 
testOuterKStreamKStream PASSED

org.apache.kafka.streams.integration.JoinIntegrationTest > 
testInnerKStreamKTable STARTED

org.apache.kafka.streams.integration.JoinIntegrationTest > 
testInnerKStreamKTable PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftOuterJoinQueryable STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftOuterJoinQueryable PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerLeftJoin STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerLeftJoin PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterOuterJoin STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterOuterJoin PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerOuterJoin STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerOuterJoin PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftInnerJoin STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftInnerJoin PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerInnerJoinQueryable STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerInnerJoinQueryable PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerOuterJoinQueryable STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerOuterJoinQueryable PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterLeftJoin STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterLeftJoin PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterInnerJoin STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterInnerJoin PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerInnerJoin STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerInnerJoin PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftOuterJoin STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftOuterJoin PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterLeftJoinQueryable STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterLeftJoinQueryable PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftLeftJoin STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftLeftJoin PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterInnerJoinQueryable STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterInnerJoinQueryable PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftInnerJoinQueryable STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftInnerJoinQueryable PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerLeftJoinQueryable STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerLeftJoinQueryable PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftLeftJoinQueryable STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftLeftJoinQueryable PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterOuterJoinQueryable STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterOuterJoinQueryable PASSED

org.apache.kafka.streams.integration.KStreamRepartitionJoinTest > 
shouldCorrectlyRepartitionOnJoinOperationsWithZeroSizedCache STARTED

org.apache.kafka.streams.integration.KStreamRepartitionJoinTest > 
shouldCorrectlyRepartitionOnJoinOperationsWithZeroSizedCache PASSED

org.apache.kafka.streams.integration.KStreamRepartitionJoinTest > 
shouldCorrectlyRepartitionOnJoinOperationsWithNonZeroSizedCache STARTED

org.apache.kafka.streams.integration.KStreamRepartitionJoinTest > 

[GitHub] kafka pull request #4083: MINOR: Improve a Windows quickstart instruction

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/4083


---

[GitHub] kafka-site pull request #104: Replace link

[manjuapu]

Github user manjuapu commented on a diff in the pull request:

https://github.com/apache/kafka-site/pull/104#discussion_r147262245
  
--- Diff: powered-by.html ---
@@ -2,453 +2,452 @@
 

[GitHub] kafka-site pull request #104: Replace link

[manjuapu]

Github user manjuapu commented on a diff in the pull request:

https://github.com/apache/kafka-site/pull/104#discussion_r147262007
  
--- Diff: powered-by.html ---
@@ -2,453 +2,452 @@
 

[GitHub] kafka-site pull request #104: Replace link

[guozhangwang]

Github user guozhangwang commented on a diff in the pull request:

https://github.com/apache/kafka-site/pull/104#discussion_r147261460
  
--- Diff: powered-by.html ---
@@ -2,453 +2,452 @@
 

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

[Jeyhun Karimov]

Hi,

Thanks for your suggestions.

I have some comments, to make sure that there is no misunderstanding.


1. Maybe we can deprecate the `commit()` in ProcessorContext, to enforce
> user to consolidate this call as
> "processorContext.recordContext().commit()". And internal implementation
> of
> `ProcessorContext.commit()` in `ProcessorContextImpl` is also changed to
> this call.


- I think we should not deprecate `ProcessorContext.commit()`. The main
intuition that we introduce `commit()` in `RecordContext` is that,
`RecordContext` is the one which is provided in Rich interfaces. So if user
wants to commit, then there should be some method inside `RecordContext` to
do so. Internally, `RecordContext.commit()` calls
`ProcessorContext.commit()`  (see the last code snippet in KIP-159):

@Override
public void process(final K1 key, final V1 value) {

recordContext = new RecordContext() {   //
recordContext initialization is added in this KIP
@Override
public void commit() {
context().commit();
}

@Override
public long offset() {
return context().recordContext().offset();
}

@Override
public long timestamp() {
return context().recordContext().timestamp();
}

@Override
public String topic() {
return context().recordContext().topic();
}

@Override
public int partition() {
return context().recordContext().partition();
}
  };


So, we cannot deprecate `ProcessorContext.commit()` in this case IMO.


2. Add the `task` reference to the impl class, `ProcessorRecordContext`, so
> that it can implement the commit call itself.


- Actually, I don't think that we need `commit()` in
`ProcessorRecordContext`. The main intuition is to "transfer"
`ProcessorContext.commit()` call to Rich interfaces, to support
user-specific committing.
 To do so, we introduce `commit()` method in `RecordContext()` just only to
call ProcessorContext.commit() inside. (see the above code snippet)
So, in Rich interfaces, we are not dealing with  `ProcessorRecordContext`
at all, and we leave all its methods as it is.
In this KIP, we made `RecordContext` to be the parent class of
`ProcessorRecordContext`, just because of they share quite amount of
methods and it is logical to enable inheritance between those two.

3. In the wiki page, the statement that "However, call to a commit() method,
> is valid only within RecordContext interface (at least for now), we throw
> an exception in ProcessorRecordContext.commit()." and the code snippet
> below would need to be updated as well.


- I think above explanation covers this as well.


I want to gain some speed to this KIP, as it has gone though many changes
based on user/developer needs, both in documentation-/implementation-wise.


Cheers,
Jeyhun



On Tue, Oct 24, 2017 at 1:41 AM Guozhang Wang  wrote:

> Thanks for the information Jeyhun. I had also forgot about KAFKA-3907 with
> this KIP..
>
> Thinking a bit more, I'm now inclined to go with what we agreed before, to
> add the commit() call to `RecordContext`. A few minor tweaks on its
> implementation:
>
> 1. Maybe we can deprecate the `commit()` in ProcessorContext, to enforce
> user to consolidate this call as
> "processorContext.recordContext().commit()". And internal implementation
> of
> `ProcessorContext.commit()` in `ProcessorContextImpl` is also changed to
> this call.
>
> 2. Add the `task` reference to the impl class, `ProcessorRecordContext`, so
> that it can implement the commit call itself.
>
> 3. In the wiki page, the statement that "However, call to a commit()
> method,
> is valid only within RecordContext interface (at least for now), we throw
> an exception in ProcessorRecordContext.commit()." and the code snippet
> below would need to be updated as well.
>
>
> Guozhang
>
>
>
> On Mon, Oct 23, 2017 at 1:40 PM, Matthias J. Sax 
> wrote:
>
> > Fair point. This is a long discussion and I totally forgot that we
> > discussed this.
> >
> > Seems I changed my opinion about including KAFKA-3907...
> >
> > Happy to hear what others think.
> >
> >
> > -Matthias
> >
> > On 10/23/17 1:20 PM, Jeyhun Karimov wrote:
> > > Hi Matthias,
> > >
> > > It is probably my bad, the discussion was a bit long in this thread. I
> > > proposed the related issue in the related KIP discuss thread [1] and
> got
> > an
> > > approval [2,3].
> > > Maybe I misunderstood.
> > >
> > > [1]
> > > http://search-hadoop.com/m/Kafka/uyzND19Asmg1GKKXT1?subj=
> > Re+DISCUSS+KIP+159+Introducing+Rich+functions+to+Streams
> > > [2]
> > > http://search-hadoop.com/m/Kafka/uyzND1kpct22GKKXT1?subj=
> > Re+DISCUSS+KIP+159+Introducing+Rich+functions+to+Streams
> > > [3]
> > > http://search-hadoop.com/m/Kafka/uyzND1G6TGIGKKXT1?subj=
> > 

Re: [VOTE] KIP-205: Add all() and range() API to ReadOnlyWindowStore

[Guozhang Wang]

BTW, Xavier has an argument to rename "range" as it does not take a key or
key range at all. I'd propose we rename this function to "fetchAll".

Guozhang

On Wed, Oct 25, 2017 at 7:27 AM, Damian Guy  wrote:

> +1
>
> On Tue, 24 Oct 2017 at 16:46 Guozhang Wang  wrote:
>
> > +1. Thanks.
> >
> > On Mon, Oct 23, 2017 at 8:11 PM, Richard Yu 
> > wrote:
> >
> > > Hi all,
> > >
> > > I want to propose KIP-205 for the addition of new API. It is about
> adding
> > > methods similar to those found in ReadOnlyKeyValueStore to the
> > > ReadOnlyWindowStore class. As it appears the discussion has reached a
> > > conclusion, I would like to start the voting process.
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 205%3A+Add+all%28%29+and+range%28%29+API+to+ReadOnlyWindowStore
> > >
> > > Thanks for your patience!
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>



-- 
-- Guozhang

Re: [DISCUSS] KIP-205: Add getAllKeys() API to ReadOnlyWindowStore

[Guozhang Wang]

Hello Richard, Xavier:

I think I am convinced my your arguments. How about keeping all() as is and
rename "range" to "fetchAll" then?


Guozhang



On Thu, Oct 26, 2017 at 9:21 AM, Xavier Léauté  wrote:

> I don't feel this worth holding up the vote for, if no one else shares my
> concerns.
> On Wed, Oct 25, 2017 at 15:59 Richard Yu 
> wrote:
>
> > Xavier: There has been two pluses on the voting thread. Are you fine with
> > the current formation?
> >
> > On Tue, Oct 24, 2017 at 4:26 PM, Richard Yu 
> > wrote:
> >
> > > I think we can come up with this compromise: range(long timeFrom, long
> > > timeTo) will be changed to getKeys(long timeFrom, long timeTo). Sounds
> > fair?
> > >
> > >
> > > On Tue, Oct 24, 2017 at 10:44 AM, Xavier Léauté 
> > > wrote:
> > >
> > >> >
> > >> > Generally I think having `all / range` is better in terms of
> > consistency
> > >> > with key-value windows. I.e. queries with key are named as `get /
> > fetch`
> > >> > for kv / window stores, and queries without key are named as `range
> /
> > >> all`.
> > >> >
> > >>
> > >> For kv stores, range takes a range of keys, and with this proposal
> range
> > >> on
> > >> window stores would take a range of time, that does not sound
> consistent
> > >> to
> > >> me at all.
> > >>
> > >> We also already have fetch which take both a range of time and keys.
> > >>
> > >
> > >
> >
>



-- 
-- Guozhang

[GitHub] kafka pull request #4136: KAFKA-6100: Down-grade RocksDB to 5.7.3

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/4136


---

[jira] [Resolved] (KAFKA-6100) Streams quick start crashes Java on Windows

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-6100.
--
   Resolution: Fixed
Fix Version/s: 1.0.0

Issue resolved by pull request 4136
[https://github.com/apache/kafka/pull/4136]

> Streams quick start crashes Java on Windows 
> 
>
> Key: KAFKA-6100
> URL: https://issues.apache.org/jira/browse/KAFKA-6100
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
> Environment: Windows 10 VM
>Reporter: Vahid Hashemian
> Fix For: 1.0.0
>
> Attachments: Screen Shot 2017-10-20 at 11.53.14 AM.png, 
> java.exe_171023_115335.dmp.zip
>
>
> *This issue was detected in 1.0.0 RC2.*
> The following step in streams quick start crashes Java on Windows 10:
> {{bin/kafka-run-class.sh 
> org.apache.kafka.streams.examples.wordcount.WordCountDemo}}
> I tracked this down to [this 
> change|https://github.com/apache/kafka/commit/196bcfca0c56420793f85514d1602bde564b0651#diff-6512f838e273b79676cac5f72456127fR67],
>  and it seems to new version of RocksDB is to blame.  I tried the quick start 
> with the previous version of RocksDB (5.7.3) and did not run into this issue.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] kafka-site pull request #104: Replace link

[joel-hamill]

Github user joel-hamill commented on a diff in the pull request:

https://github.com/apache/kafka-site/pull/104#discussion_r147256782
  
--- Diff: 0110/streams/index.html ---
@@ -17,20 +17,20 @@
 
 

[GitHub] kafka-site pull request #104: Replace link

[joel-hamill]

Github user joel-hamill commented on a diff in the pull request:

https://github.com/apache/kafka-site/pull/104#discussion_r147256754
  
--- Diff: 0110/streams/developer-guide.html ---
@@ -19,7 +19,17 @@
 
 

[GitHub] kafka-site pull request #104: Replace link

[joel-hamill]

Github user joel-hamill commented on a diff in the pull request:

https://github.com/apache/kafka-site/pull/104#discussion_r147256824
  
--- Diff: 0110/streams/quickstart.html ---
@@ -18,11 +18,21 @@
 
 

[GitHub] kafka-site pull request #104: Replace link

[joel-hamill]

Github user joel-hamill commented on a diff in the pull request:

https://github.com/apache/kafka-site/pull/104#discussion_r147256597
  
--- Diff: 0110/streams/core-concepts.html ---
@@ -19,7 +19,17 @@
 
 

[GitHub] kafka-site pull request #104: Replace link

[joel-hamill]

Github user joel-hamill commented on a diff in the pull request:

https://github.com/apache/kafka-site/pull/104#discussion_r147256868
  
--- Diff: 0110/streams/tutorial.html ---
@@ -18,7 +18,17 @@
 
 

[GitHub] kafka-site pull request #104: Replace link

[manjuapu]

GitHub user manjuapu opened a pull request:

https://github.com/apache/kafka-site/pull/104

Replace link

@guozhangwang Please review.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/manjuapu/kafka-site replace-link

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka-site/pull/104.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #104


commit 15f317907127b35eaaeb4b58c3984d5345eaf1e5
Author: Manjula Kumar 
Date:   2017-10-18T15:52:31Z

MINOR: Pinterest link correction

Social proof logo column swap

Add zalando blog link to streams page

commit 21802dd1ff0c0ee6a076c1f2f43706830b45239c
Author: Manjula Kumar 
Date:   2017-10-26T20:01:05Z

Adding nav bar to streams sub page

commit c612e3be09c545c9a06137196fda815b074fd47b
Author: Manjula Kumar 
Date:   2017-10-26T20:08:02Z

Resolved conflicts




---

Re: [DISCUSS] KIP-210: Provide for custom error handling when Kafka Streams fails to produce

[Matt Farmer]

Hello all,

I've updated the KIP based on this conversation, and made it so that its
interface, config setting, and parameters line up more closely with the
interface in KIP-161 (deserialization handler).

I believe there are a few specific questions I need to reply to.

> The question I had about then handle parameters are around the record,
> should it be `ProducerRecord`, or be generics of
> `ProducerRecord` or `ProducerRecord Object, ? extends Object>`?

At this point in the code we're guaranteed that this is a
ProducerRecord, so the generics would just make it harder
to work with the key and value.

> Also, should the handle function include the `RecordMetadata` as well in
> case it is not null?

Please correct me if I'm wrong, but my understanding is that the record
metadata is always null if an exception occurred while trying to produce.

> We may probably try to write down at least the following handling logic
and
> see if the given API is sufficient for it

I've added some examples to the KIP. Let me know what you think.

Cheers,
Matt

On Mon, Oct 23, 2017 at 9:00 PM Matt Farmer  wrote:

> Thanks for this feedback. I’m at a conference right now and am planning on
> updating the KIP again with details from this conversation later this week.
>
> I’ll shoot you a more detailed response then! :)
> On Mon, Oct 23, 2017 at 8:16 PM Guozhang Wang  wrote:
>
>> Thanks for the KIP Matt.
>>
>> Regarding the handle interface of ProductionExceptionHandlerResponse,
>> could
>> you write it on the wiki also, along with the actual added config names
>> (e.g. what
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-161%3A+streams+deserialization+exception+handlers
>> described).
>>
>> The question I had about then handle parameters are around the record,
>> should it be `ProducerRecord`, or be generics of
>> `ProducerRecord` or `ProducerRecord> Object, ? extends Object>`?
>>
>> Also, should the handle function include the `RecordMetadata` as well in
>> case it is not null?
>>
>> We may probably try to write down at least the following handling logic
>> and
>> see if the given API is sufficient for it: 1) throw exception immediately
>> to fail fast and stop the world, 2) log the error and drop record and
>> proceed silently, 3) send such errors to a specific "error" Kafka topic,
>> or
>> record it as an app-level metrics (
>> https://kafka.apache.org/documentation/#kafka_streams_monitoring) for
>> monitoring.
>>
>> Guozhang
>>
>>
>>
>> On Fri, Oct 20, 2017 at 5:47 PM, Matt Farmer  wrote:
>>
>> > I did some more digging tonight.
>> >
>> > @Ted: It looks like the deserialization handler uses
>> > "default.deserialization.exception.handler" for the config name. No
>> > ".class" on the end. I'm inclined to think this should use
>> > "default.production.exception.handler".
>> >
>> > On Fri, Oct 20, 2017 at 8:22 PM Matt Farmer  wrote:
>> >
>> > > Okay, I've dug into this a little bit.
>> > >
>> > > I think getting access to the serialized record is possible, and
>> changing
>> > > the naming and return type is certainly doable. However, because we're
>> > > hooking into the onCompletion callback we have no guarantee that the
>> > > ProcessorContext state hasn't changed by the time this particular
>> handler
>> > > runs. So I think the signature would change to something like:
>> > >
>> > > ProductionExceptionHandlerResponse handle(final ProducerRecord<..>
>> > record,
>> > > final Exception exception)
>> > >
>> > > Would this be acceptable?
>> > >
>> > > On Thu, Oct 19, 2017 at 7:33 PM Matt Farmer  wrote:
>> > >
>> > >> Ah good idea. Hmmm. I can line up the naming and return type but I’m
>> not
>> > >> sure if I can get my hands on the context and the record itself
>> without
>> > >> other changes.
>> > >>
>> > >> Let me dig in and follow up here tomorrow.
>> > >> On Thu, Oct 19, 2017 at 7:14 PM Matthias J. Sax <
>> matth...@confluent.io>
>> > >> wrote:
>> > >>
>> > >>> Thanks for the KIP.
>> > >>>
>> > >>> Are you familiar with KIP-161?
>> > >>>
>> > >>>
>> > >>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-161%3A+streams+
>> > deserialization+exception+handlers
>> > >>>
>> > >>> I thinks, we should align the design (parameter naming, return
>> types,
>> > >>> class names etc) of KIP-210 to KIP-161 to get a unified user
>> > experience.
>> > >>>
>> > >>>
>> > >>>
>> > >>> -Matthias
>> > >>>
>> > >>>
>> > >>> On 10/18/17 4:20 PM, Matt Farmer wrote:
>> > >>> > I’ll create the JIRA ticket.
>> > >>> >
>> > >>> > I think that config name will work. I’ll update the KIP
>> accordingly.
>> > >>> > On Wed, Oct 18, 2017 at 6:09 PM Ted Yu 
>> wrote:
>> > >>> >
>> > >>> >> Can you create JIRA that corresponds to the KIP ?
>> > >>> >>
>> > >>> >> For the new config, how about naming it
>> > >>> >> production.exception.processor.class
>> > >>> >> ? This way it is 

[jira] [Created] (KAFKA-6133) NullPointerException in S3 Connector when using rotate.interval.ms

[Elizabeth Bennett (JIRA)]

Elizabeth Bennett created KAFKA-6133:


 Summary: NullPointerException in S3 Connector when using 
rotate.interval.ms
 Key: KAFKA-6133
 URL: https://issues.apache.org/jira/browse/KAFKA-6133
 Project: Kafka
  Issue Type: Bug
  Components: KafkaConnect
Affects Versions: 0.11.0.0
Reporter: Elizabeth Bennett


I just tried out the new rotate.interval.ms feature in the S3 connector to do 
time based flushing. I am getting this NPE on every event:

{{[2017-10-20 23:21:35,233] ERROR Task foo-to-S3-0 threw an uncaught and 
unrecoverable exception (org.apache.kafka.connect.runtime.WorkerSinkTask)
java.lang.NullPointerException
at 
io.confluent.connect.s3.TopicPartitionWriter.rotateOnTime(TopicPartitionWriter.java:288)
at 
io.confluent.connect.s3.TopicPartitionWriter.write(TopicPartitionWriter.java:234)
at io.confluent.connect.s3.S3SinkTask.put(S3SinkTask.java:180)
at 
org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:464)
at 
org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:265)
at 
org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:182)
at 
org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:150)
at 
org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:146)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:190)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
[2017-10-20 23:21:35,233] ERROR Task is being killed and will not recover until 
manually restarted (org.apache.kafka.connect.runtime.WorkerSinkTask)
[2017-10-20 23:21:35,233] ERROR Task foo-to-S3-0 threw an uncaught and 
unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask)
org.apache.kafka.connect.errors.ConnectException: Exiting WorkerSinkTask due to 
unrecoverable exception.
at 
org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:484)
at 
org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:265)
at 
org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:182)
at 
org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:150)
at 
org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:146)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:190)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)}}

I dug into the S3 connect code a bit and it looks like the 
{{rotate.interval.ms}} feature only works if you are using the 
TimeBasedPartitioner. It will get the TimestampExtractor class from the 
TimeBasedPartitioner to determine the timestamp of the event, and will use this 
for the time based flushing.

I'm using a custom partitioner, but I'd still really like to use the 
{{rotate.interval.ms}} feature, using wall clock time to determine the flushing 
behavior.

I'd be willing to work on fixing this issue, but I want to confirm it is 
actually bug, and not that it was specifically designed to only work with the 
TimeBasedPartitioner. Even if it is the later, it should probably not crash 
with an NPE.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: kafka-trunk-jdk7 #2923

[Apache Jenkins Server]

See 


Changes:

[ismael] MINOR: Revert EmbeddedZooKeeper rename

[ismael] MINOR: Add missing semicolon to example jaas configuration

--
[...truncated 1.83 MB...]
org.apache.kafka.streams.kstream.internals.KTableAggregateTest > 
testAggCoalesced PASSED

org.apache.kafka.streams.kstream.internals.KTableAggregateTest > 
testAggRepartition STARTED

org.apache.kafka.streams.kstream.internals.KTableAggregateTest > 
testAggRepartition PASSED

org.apache.kafka.streams.kstream.internals.KTableAggregateTest > 
testRemoveOldBeforeAddNew STARTED

org.apache.kafka.streams.kstream.internals.KTableAggregateTest > 
testRemoveOldBeforeAddNew PASSED

org.apache.kafka.streams.kstream.internals.KTableAggregateTest > 
testCountCoalesced STARTED

org.apache.kafka.streams.kstream.internals.KTableAggregateTest > 
testCountCoalesced PASSED

org.apache.kafka.streams.kstream.internals.KTableForeachTest > testForeach 
STARTED

org.apache.kafka.streams.kstream.internals.KTableForeachTest > testForeach 
PASSED

org.apache.kafka.streams.kstream.internals.KTableForeachTest > testTypeVariance 
STARTED

org.apache.kafka.streams.kstream.internals.KTableForeachTest > testTypeVariance 
PASSED

org.apache.kafka.streams.kstream.SessionWindowsTest > 
retentionTimeShouldBeGapIfGapIsLargerThanDefaultRetentionTime STARTED

org.apache.kafka.streams.kstream.SessionWindowsTest > 
retentionTimeShouldBeGapIfGapIsLargerThanDefaultRetentionTime PASSED

org.apache.kafka.streams.kstream.SessionWindowsTest > shouldSetWindowGap STARTED

org.apache.kafka.streams.kstream.SessionWindowsTest > shouldSetWindowGap PASSED

org.apache.kafka.streams.kstream.SessionWindowsTest > 
shouldBeEqualWhenGapAndMaintainMsAreTheSame STARTED

org.apache.kafka.streams.kstream.SessionWindowsTest > 
shouldBeEqualWhenGapAndMaintainMsAreTheSame PASSED

org.apache.kafka.streams.kstream.SessionWindowsTest > 
retentionTimeMustNotBeNegative STARTED

org.apache.kafka.streams.kstream.SessionWindowsTest > 
retentionTimeMustNotBeNegative PASSED

org.apache.kafka.streams.kstream.SessionWindowsTest > 
shouldNotBeEqualWhenMaintainMsDifferent STARTED

org.apache.kafka.streams.kstream.SessionWindowsTest > 
shouldNotBeEqualWhenMaintainMsDifferent PASSED

org.apache.kafka.streams.kstream.SessionWindowsTest > windowSizeMustNotBeZero 
STARTED

org.apache.kafka.streams.kstream.SessionWindowsTest > windowSizeMustNotBeZero 
PASSED

org.apache.kafka.streams.kstream.SessionWindowsTest > 
windowSizeMustNotBeNegative STARTED

org.apache.kafka.streams.kstream.SessionWindowsTest > 
windowSizeMustNotBeNegative PASSED

org.apache.kafka.streams.kstream.SessionWindowsTest > 
shouldSetWindowRetentionTime STARTED

org.apache.kafka.streams.kstream.SessionWindowsTest > 
shouldSetWindowRetentionTime PASSED

org.apache.kafka.streams.kstream.SessionWindowsTest > 
shouldNotBeEqualWhenGapIsDifferent STARTED

org.apache.kafka.streams.kstream.SessionWindowsTest > 
shouldNotBeEqualWhenGapIsDifferent PASSED

org.apache.kafka.streams.kstream.WindowTest > 
shouldThrowIfEndIsSmallerThanStart STARTED

org.apache.kafka.streams.kstream.WindowTest > 
shouldThrowIfEndIsSmallerThanStart PASSED

org.apache.kafka.streams.kstream.WindowTest > 
shouldNotBeEqualIfDifferentWindowType STARTED

org.apache.kafka.streams.kstream.WindowTest > 
shouldNotBeEqualIfDifferentWindowType PASSED

org.apache.kafka.streams.kstream.WindowTest > shouldBeEqualIfStartAndEndSame 
STARTED

org.apache.kafka.streams.kstream.WindowTest > shouldBeEqualIfStartAndEndSame 
PASSED

org.apache.kafka.streams.kstream.WindowTest > shouldNotBeEqualIfNull STARTED

org.apache.kafka.streams.kstream.WindowTest > shouldNotBeEqualIfNull PASSED

org.apache.kafka.streams.kstream.WindowTest > shouldThrowIfStartIsNegative 
STARTED

org.apache.kafka.streams.kstream.WindowTest > shouldThrowIfStartIsNegative 
PASSED

org.apache.kafka.streams.kstream.WindowTest > 
shouldNotBeEqualIfStartOrEndIsDifferent STARTED

org.apache.kafka.streams.kstream.WindowTest > 
shouldNotBeEqualIfStartOrEndIsDifferent PASSED

org.apache.kafka.streams.kstream.KStreamBuilderTest > 
shouldAddRegexTopicToLatestAutoOffsetResetList STARTED

org.apache.kafka.streams.kstream.KStreamBuilderTest > 
shouldAddRegexTopicToLatestAutoOffsetResetList PASSED

org.apache.kafka.streams.kstream.KStreamBuilderTest > 
shouldMapStateStoresToCorrectSourceTopics STARTED

org.apache.kafka.streams.kstream.KStreamBuilderTest > 
shouldMapStateStoresToCorrectSourceTopics PASSED

org.apache.kafka.streams.kstream.KStreamBuilderTest > 
shouldAddTableToEarliestAutoOffsetResetList STARTED

org.apache.kafka.streams.kstream.KStreamBuilderTest > 
shouldAddTableToEarliestAutoOffsetResetList PASSED

org.apache.kafka.streams.kstream.KStreamBuilderTest > 
shouldAddTimestampExtractorToTablePerSource STARTED

org.apache.kafka.streams.kstream.KStreamBuilderTest > 
shouldAddTimestampExtractorToTablePerSource PASSED

[GitHub] kafka pull request #4140: KAFKA-6131: Use atomic putIfAbsent to create txn m...

[rajinisivaram]

GitHub user rajinisivaram opened a pull request:

https://github.com/apache/kafka/pull/4140

KAFKA-6131: Use atomic putIfAbsent to create txn marker queues



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rajinisivaram/kafka 
KAFKA-6131-txn-concurrentmap

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4140.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4140


commit b80fcb5bfc8cb70f270eb558ed6267dca99d1e08
Author: Rajini Sivaram 
Date:   2017-10-26T17:45:47Z

KAFKA-6131: Use atomic putIfAbsent to create txn marker queues




---

Build failed in Jenkins: kafka-trunk-jdk8 #2168

[Apache Jenkins Server]

See 


Changes:

[ismael] MINOR: Revert EmbeddedZooKeeper rename

[ismael] MINOR: Add missing semicolon to example jaas configuration

--
[...truncated 381.98 KB...]

kafka.message.MessageCompressionTest > testCompressSize STARTED

kafka.message.MessageCompressionTest > testCompressSize PASSED

kafka.message.MessageCompressionTest > testSimpleCompressDecompress STARTED

kafka.message.MessageCompressionTest > testSimpleCompressDecompress PASSED

kafka.message.ByteBufferMessageSetTest > testMessageWithProvidedOffsetSeq 
STARTED

kafka.message.ByteBufferMessageSetTest > testMessageWithProvidedOffsetSeq PASSED

kafka.message.ByteBufferMessageSetTest > testValidBytes STARTED

kafka.message.ByteBufferMessageSetTest > testValidBytes PASSED

kafka.message.ByteBufferMessageSetTest > testValidBytesWithCompression STARTED

kafka.message.ByteBufferMessageSetTest > testValidBytesWithCompression PASSED

kafka.message.ByteBufferMessageSetTest > 
testWriteToChannelThatConsumesPartially STARTED

kafka.message.ByteBufferMessageSetTest > 
testWriteToChannelThatConsumesPartially PASSED

kafka.message.ByteBufferMessageSetTest > testIteratorIsConsistent STARTED

kafka.message.ByteBufferMessageSetTest > testIteratorIsConsistent PASSED

kafka.message.ByteBufferMessageSetTest > testWrittenEqualsRead STARTED

kafka.message.ByteBufferMessageSetTest > testWrittenEqualsRead PASSED

kafka.message.ByteBufferMessageSetTest > testWriteTo STARTED

kafka.message.ByteBufferMessageSetTest > testWriteTo PASSED

kafka.message.ByteBufferMessageSetTest > testEquals STARTED

kafka.message.ByteBufferMessageSetTest > testEquals PASSED

kafka.message.ByteBufferMessageSetTest > testSizeInBytes STARTED

kafka.message.ByteBufferMessageSetTest > testSizeInBytes PASSED

kafka.message.ByteBufferMessageSetTest > testIterator STARTED

kafka.message.ByteBufferMessageSetTest > testIterator PASSED

kafka.metrics.KafkaTimerTest > testKafkaTimer STARTED

kafka.metrics.KafkaTimerTest > testKafkaTimer PASSED

kafka.metrics.MetricsTest > testMetricsReporterAfterDeletingTopic STARTED

kafka.metrics.MetricsTest > testMetricsReporterAfterDeletingTopic PASSED

kafka.metrics.MetricsTest > 
testBrokerTopicMetricsUnregisteredAfterDeletingTopic STARTED

kafka.metrics.MetricsTest > 
testBrokerTopicMetricsUnregisteredAfterDeletingTopic PASSED

kafka.metrics.MetricsTest > testClusterIdMetric STARTED

kafka.metrics.MetricsTest > testClusterIdMetric PASSED

kafka.metrics.MetricsTest > testControllerMetrics STARTED

kafka.metrics.MetricsTest > testControllerMetrics PASSED

kafka.metrics.MetricsTest > testMetricsLeak STARTED

kafka.metrics.MetricsTest > testMetricsLeak PASSED

kafka.metrics.MetricsTest > testBrokerTopicMetricsBytesInOut STARTED

kafka.metrics.MetricsTest > testBrokerTopicMetricsBytesInOut PASSED

kafka.javaapi.consumer.ZookeeperConsumerConnectorTest > testBasic STARTED

kafka.javaapi.consumer.ZookeeperConsumerConnectorTest > testBasic PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testSizeInBytesWithCompression 
STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testSizeInBytesWithCompression 
PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > 
testIteratorIsConsistentWithCompression STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > 
testIteratorIsConsistentWithCompression PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testIteratorIsConsistent 
STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testIteratorIsConsistent PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testEqualsWithCompression 
STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testEqualsWithCompression 
PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testWrittenEqualsRead STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testWrittenEqualsRead PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testEquals STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testEquals PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testSizeInBytes STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testSizeInBytes PASSED

kafka.security.auth.ResourceTypeTest > testJavaConversions STARTED

kafka.security.auth.ResourceTypeTest > testJavaConversions PASSED

kafka.security.auth.ResourceTypeTest > testFromString STARTED

kafka.security.auth.ResourceTypeTest > testFromString PASSED

kafka.security.auth.OperationTest > testJavaConversions STARTED

kafka.security.auth.OperationTest > testJavaConversions PASSED

kafka.security.auth.PermissionTypeTest > testJavaConversions STARTED

kafka.security.auth.PermissionTypeTest > testJavaConversions PASSED

kafka.security.auth.PermissionTypeTest > testFromString STARTED

kafka.security.auth.PermissionTypeTest > testFromString PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testAllowAllAccess STARTED

kafka.security.auth.SimpleAclAuthorizerTest > 

[jira] [Created] (KAFKA-6132) KafkaProducer.initTransactions dead locks

[Piotr Nowojski (JIRA)]

Piotr Nowojski created KAFKA-6132:
-

 Summary: KafkaProducer.initTransactions dead locks
 Key: KAFKA-6132
 URL: https://issues.apache.org/jira/browse/KAFKA-6132
 Project: Kafka
  Issue Type: Bug
  Components: producer 
Affects Versions: 0.11.0.0, 0.11.0.1
 Environment: Travis
Reporter: Piotr Nowojski
Priority: Critical


I have found some intermittent failures on travis when using Kafka 0.11 
transactions for writing. One of them is a apparent deadlock with the following 
stack trace:

{code:java}
"KafkaTestThread[19, false]" #166 prio=5 os_prio=0 tid=0x7f4b64c29800 
nid=0x1260 waiting on condition [0x7f4b10fa4000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x947048a8> (a 
java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
at 
org.apache.kafka.clients.producer.internals.TransactionalRequestResult.await(TransactionalRequestResult.java:50)
at 
org.apache.kafka.clients.producer.KafkaProducer.initTransactions(KafkaProducer.java:537)
{code}

I was unsuccessful to reproduce it locally, however I think I can semi reliably 
reproduce it on Travis. Scenario includes simultaneous sequence of 
instantiating new producers, calling {{KafkaProducer.initTransactions}}, 
closing them interleaved with writing. I have created a stripped down version 
of this scenario as a github project:
https://github.com/pnowojski/kafka-init-deadlock
The code for test scenario is here:
https://github.com/pnowojski/kafka-init-deadlock/blob/master/src/test/java/pl/nowojski/KafkaInitDeadLockTest.java

I have defined 30 build profiles that run this test and in case of detecting a 
dead lock (5 minutes period of inactivity) stack trace of all threads is being 
printed out. Example travis run:
https://travis-ci.org/pnowojski/kafka-init-deadlock/builds/293256284
as you can see deadlock occurred in 7 out of 30 builds. It seems like in this 
scenario all of them are failing/dead locking in exactly same way. 

I have observed this issue both on 0.11.0.0 and 0.11.0.1 





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-6131) Transaction markers are sometimes discarded if txns complete concurrently

[Rajini Sivaram (JIRA)]

Rajini Sivaram created KAFKA-6131:
-

 Summary: Transaction markers are sometimes discarded if txns 
complete concurrently
 Key: KAFKA-6131
 URL: https://issues.apache.org/jira/browse/KAFKA-6131
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 1.0.0
Reporter: Rajini Sivaram
Assignee: Rajini Sivaram


Concurrent tests being added under KAFKA-6096 for transaction coordinator fail 
to complete some transactions when multiple transactions are completed 
concurrently.

The problem is with the following code snippet - there are two very similar 
uses of concurrent map in {{TransactionMarkerChannelManager}} and the test 
fails because some transaction markers are discarded. {{getOrElseUpdate}} in 
scala maps are not atomic. The test passes consistently with one thread.
{quote}
val markersQueuePerBroker: concurrent.Map[Int, TxnMarkerQueue] = new 
ConcurrentHashMap[Int, TxnMarkerQueue]().asScala
val brokerRequestQueue = markersQueuePerBroker.getOrElseUpdate(brokerId, new 
TxnMarkerQueue(broker))
{quote}





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: kafka-0.11.0-jdk7 #327

[Apache Jenkins Server]

See 


Changes:

[ismael] MINOR: Add missing semicolon to example jaas configuration

--
[...truncated 976.88 KB...]

kafka.utils.UtilsTest > testUrlSafeBase64EncodeUUID STARTED

kafka.utils.UtilsTest > testUrlSafeBase64EncodeUUID PASSED

kafka.utils.UtilsTest > testCsvMap STARTED

kafka.utils.UtilsTest > testCsvMap PASSED

kafka.utils.UtilsTest > testInLock STARTED

kafka.utils.UtilsTest > testInLock PASSED

kafka.utils.UtilsTest > testSwallow STARTED

kafka.utils.UtilsTest > testSwallow PASSED

kafka.utils.timer.TimerTest > testAlreadyExpiredTask STARTED

kafka.utils.timer.TimerTest > testAlreadyExpiredTask PASSED

kafka.utils.timer.TimerTest > testTaskExpiration STARTED

kafka.utils.timer.TimerTest > testTaskExpiration PASSED

kafka.utils.timer.TimerTaskListTest > testAll STARTED

kafka.utils.timer.TimerTaskListTest > testAll PASSED

kafka.utils.IteratorTemplateTest > testIterator STARTED

kafka.utils.IteratorTemplateTest > testIterator PASSED

kafka.controller.ControllerIntegrationTest > 
testPartitionReassignmentWithOfflineReplicaHaltingProgress STARTED

kafka.controller.ControllerIntegrationTest > 
testPartitionReassignmentWithOfflineReplicaHaltingProgress PASSED

kafka.controller.ControllerIntegrationTest > 
testControllerEpochPersistsWhenAllBrokersDown STARTED

kafka.controller.ControllerIntegrationTest > 
testControllerEpochPersistsWhenAllBrokersDown PASSED

kafka.controller.ControllerIntegrationTest > 
testTopicCreationWithOfflineReplica STARTED

kafka.controller.ControllerIntegrationTest > 
testTopicCreationWithOfflineReplica PASSED

kafka.controller.ControllerIntegrationTest > 
testPartitionReassignmentResumesAfterReplicaComesOnline STARTED

kafka.controller.ControllerIntegrationTest > 
testPartitionReassignmentResumesAfterReplicaComesOnline PASSED

kafka.controller.ControllerIntegrationTest > 
testLeaderAndIsrWhenEntireIsrOfflineAndUncleanLeaderElectionDisabled STARTED

kafka.controller.ControllerIntegrationTest > 
testLeaderAndIsrWhenEntireIsrOfflineAndUncleanLeaderElectionDisabled PASSED

kafka.controller.ControllerIntegrationTest > 
testTopicPartitionExpansionWithOfflineReplica STARTED

kafka.controller.ControllerIntegrationTest > 
testTopicPartitionExpansionWithOfflineReplica PASSED

kafka.controller.ControllerIntegrationTest > 
testPreferredReplicaLeaderElectionWithOfflinePreferredReplica STARTED

kafka.controller.ControllerIntegrationTest > 
testPreferredReplicaLeaderElectionWithOfflinePreferredReplica PASSED

kafka.controller.ControllerIntegrationTest > 
testAutoPreferredReplicaLeaderElection STARTED

kafka.controller.ControllerIntegrationTest > 
testAutoPreferredReplicaLeaderElection PASSED

kafka.controller.ControllerIntegrationTest > testTopicCreation STARTED

kafka.controller.ControllerIntegrationTest > testTopicCreation PASSED

kafka.controller.ControllerIntegrationTest > testPartitionReassignment STARTED

kafka.controller.ControllerIntegrationTest > testPartitionReassignment PASSED

kafka.controller.ControllerIntegrationTest > testTopicPartitionExpansion STARTED

kafka.controller.ControllerIntegrationTest > testTopicPartitionExpansion PASSED

kafka.controller.ControllerIntegrationTest > 
testControllerMoveIncrementsControllerEpoch STARTED

kafka.controller.ControllerIntegrationTest > 
testControllerMoveIncrementsControllerEpoch PASSED

kafka.controller.ControllerIntegrationTest > 
testLeaderAndIsrWhenEntireIsrOfflineAndUncleanLeaderElectionEnabled STARTED

kafka.controller.ControllerIntegrationTest > 
testLeaderAndIsrWhenEntireIsrOfflineAndUncleanLeaderElectionEnabled PASSED

kafka.controller.ControllerIntegrationTest > testEmptyCluster STARTED

kafka.controller.ControllerIntegrationTest > testEmptyCluster PASSED

kafka.controller.ControllerIntegrationTest > testPreferredReplicaLeaderElection 
STARTED

kafka.controller.ControllerIntegrationTest > testPreferredReplicaLeaderElection 
PASSED

kafka.controller.ControllerEventManagerTest > testEventThatThrowsException 
STARTED

kafka.controller.ControllerEventManagerTest > testEventThatThrowsException 
PASSED

kafka.controller.ControllerEventManagerTest > testSuccessfulEvent STARTED

kafka.controller.ControllerEventManagerTest > testSuccessfulEvent PASSED

kafka.controller.ControllerFailoverTest > testHandleIllegalStateException 
STARTED

kafka.controller.ControllerFailoverTest > testHandleIllegalStateException PASSED

kafka.network.SocketServerTest > testGracefulClose STARTED

kafka.network.SocketServerTest > testGracefulClose PASSED

kafka.network.SocketServerTest > testClientDisconnectionUpdatesRequestMetrics 
STARTED

kafka.network.SocketServerTest > testClientDisconnectionUpdatesRequestMetrics 
PASSED

kafka.network.SocketServerTest > testProcessorMetricsTags STARTED

kafka.network.SocketServerTest > testProcessorMetricsTags PASSED

kafka.network.SocketServerTest > testMaxConnectionsPerIp STARTED

Build failed in Jenkins: kafka-trunk-jdk9 #152

[Apache Jenkins Server]

See 


Changes:

[me] MINOR: reset state in cleanup, fixes jmx mixin flakiness

[junrao] MINOR: Rename and change package of async ZooKeeper classes

[ismael] MINOR: Revert EmbeddedZooKeeper rename

[ismael] MINOR: Add missing semicolon to example jaas configuration

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H26 (couchdbtest ubuntu xenial) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/trunk^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/trunk^{commit} # timeout=10
Checking out Revision adb9d5ae7671c5befa615e369c97a701d7dfb58b 
(refs/remotes/origin/trunk)
Commit message: "MINOR: Add missing semicolon to example jaas configuration"
 > git config core.sparsecheckout # timeout=10
 > git checkout -f adb9d5ae7671c5befa615e369c97a701d7dfb58b
 > git rev-list 5f779ca3f38a1149aea1219ed4a91c677955101d # timeout=10
Setting 
GRADLE_3_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_3.4-rc-2
[kafka-trunk-jdk9] $ /bin/bash -xe /tmp/jenkins820876939951886897.sh
+ rm -rf 
+ 
/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_3.4-rc-2/bin/gradle

ERROR: JAVA_HOME is set to an invalid directory: 
/home/jenkins/tools/java/latest1.9

Please set the JAVA_HOME variable in your environment to match the
location of your Java installation.

Build step 'Execute shell' marked build as failure
Recording test results
Setting 
GRADLE_3_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_3.4-rc-2
ERROR: Step ?Publish JUnit test result report? failed: Test reports were found 
but none of them are new. Did tests run? 
For example, 

 is 1 mo 4 days old

Setting 
GRADLE_3_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_3.4-rc-2
Not sending mail to unregistered user ism...@juma.me.uk

Re: [DISCUSS] KIP-205: Add getAllKeys() API to ReadOnlyWindowStore

[Xavier Léauté]

I don't feel this worth holding up the vote for, if no one else shares my
concerns.
On Wed, Oct 25, 2017 at 15:59 Richard Yu  wrote:

> Xavier: There has been two pluses on the voting thread. Are you fine with
> the current formation?
>
> On Tue, Oct 24, 2017 at 4:26 PM, Richard Yu 
> wrote:
>
> > I think we can come up with this compromise: range(long timeFrom, long
> > timeTo) will be changed to getKeys(long timeFrom, long timeTo). Sounds
> fair?
> >
> >
> > On Tue, Oct 24, 2017 at 10:44 AM, Xavier Léauté 
> > wrote:
> >
> >> >
> >> > Generally I think having `all / range` is better in terms of
> consistency
> >> > with key-value windows. I.e. queries with key are named as `get /
> fetch`
> >> > for kv / window stores, and queries without key are named as `range /
> >> all`.
> >> >
> >>
> >> For kv stores, range takes a range of keys, and with this proposal range
> >> on
> >> window stores would take a range of time, that does not sound consistent
> >> to
> >> me at all.
> >>
> >> We also already have fetch which take both a range of time and keys.
> >>
> >
> >
>

[GitHub] kafka pull request #4139: MINOR: Indentation Fix

[vahidhashemian]

GitHub user vahidhashemian opened a pull request:

https://github.com/apache/kafka/pull/4139

MINOR: Indentation Fix



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vahidhashemian/kafka 
minor/indentation_fix_1710

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4139.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4139


commit 73223d4567ec88c6be634a236af189ca2602941f
Author: Vahid Hashemian 
Date:   2017-10-26T15:55:07Z

MINOR: Indentation Fix




---

Permission to create a KIP

[Elizabeth Bennett]

Hi Kafka Devs,
I'd like permission to create a KIP. This is following up on a discussion
from the #connect slack channel regarding overrides for producer/consumer
configs on a per-connector level.

My confluence id is: elizabeth.bennett


Thanks,
Liz

Re: [VOTE] KIP-207:The Offsets which ListOffsetsResponse returns should monotonically increase even during a partition leader change

[Manikumar]

Thanks for the KIP.
+1 (non-binding)


On Thu, Oct 26, 2017 at 5:58 AM, Jason Gustafson  wrote:

> +1. Thanks for the KIP.
>
> On Mon, Oct 23, 2017 at 11:30 AM, Colin McCabe  wrote:
>
> > On Mon, Oct 23, 2017, at 10:29, Jason Gustafson wrote:
> > > Thanks for the KIP. I'm assuming the new behavior only affects
> > > ListOffsets requests from the consumer.
> >
> > That's a very good point.  I will add a caveat that we only apply the
> > KIP-207 behavior to requests from clients, not requests from other
> > brokers (such as the ones made by ReplicaFetcherThread).
> >
> > > Might be worth mentioning that in the KIP.
> > > Also, does it affect all ListOffsets requests, or only those that
> specify
> > > the latest offset?
> >
> > I don't feel great about allowing someone to ask for the offset at time
> > T, get back X, and then ask again for the offset at T the next second
> > and get back InvalidOffsetException.  So it's probably best just to
> > apply the KIP-207 behavior to all ListOffsets requests from consumers.
> >
> > Thinking about it a bit more, we should disable the KIP-207 behavior
> > when unclean leader elections are enabled on the broker.  When unclean
> > leader elections are enabled, data loss is possible.  So we cannot
> > guarantee that offsets will always go forwards, even in theory, in this
> > mode.
> >
> > I update the kip-- check it out.
> >
> > best,
> > Colin
> >
> >
> > >
> > > -Jason
> > >
> > > On Wed, Oct 18, 2017 at 9:15 AM, Colin McCabe 
> > wrote:
> > >
> > > > On Wed, Oct 18, 2017, at 04:09, Ismael Juma wrote:
> > > > > Thanks for the KIP, +1 (binding). A few comments:
> > > > >
> > > > > 1. I agree with Jun about LEADER_NOT_AVAILABLE for the error code
> for
> > > > > older
> > > > > versions.
> > > > > 2. OffsetNotAvailableException seems clear enough (i.e. we don't
> > need the
> > > > > "ForPartition" part)
> > > >
> > > > Yeah, that is shorter and probably clearer.  Changed.
> > > >
> > > > > 3. The KIP seems to be missing the compatibility section.
> > > >
> > > > Added.
> > > >
> > > > > 4. It would be good to mention that it's now possible for a fetch
> to
> > > > > succeed while list offsets will not for a period of time. And for
> > older
> > > > > versions, the latter will return LeaderNotAvailable while the
> former
> > > > > would
> > > > > work fine, which is a bit unexpected. Not much we can do about it,
> > but
> > > > > worth mentioning it in my opinion.
> > > >
> > > > Fair enough
> > > >
> > > > cheers,
> > > > Colin
> > > >
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Tue, Oct 17, 2017 at 9:26 PM, Jun Rao  wrote:
> > > > >
> > > > > > Hi, Colin,
> > > > > >
> > > > > > Thanks for the KIP. +1. Just a minor comment. For the old client
> > > > requests,
> > > > > > would it be better to return a LEADER_NOT_AVAILABLE error
> instead?
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > > On Tue, Oct 17, 2017 at 11:11 AM, Colin McCabe <
> cmcc...@apache.org
> > >
> > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I'd like to start the voting process for KIP-207:The  Offsets
> > which
> > > > > > > ListOffsetsResponse returns should monotonically increase even
> > > > during a
> > > > > > > partition leader change.
> > > > > > >
> > > > > > > See
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > > > 207%3A+Offsets+returned+by+ListOffsetsResponse+should+be+
> > > > > > > monotonically+increasing+even+during+a+partition+leader+change
> > > > > > > for details.
> > > > > > >
> > > > > > > The voting process will run for at least 72 hours.
> > > > > > >
> > > > > > > regards,
> > > > > > > Colin
> > > > > > >
> > > > > >
> > > >
> >
>

[GitHub] kafka pull request #4101: MINOR: Add missing semicolon to example jaas confi...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/4101


---

[GitHub] kafka pull request #4138: MINOR: Revert EmbeddedZooKeeper rename

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/4138


---

[jira] [Resolved] (KAFKA-6128) Shutdown script does not do a clean shutdown

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-6128.

Resolution: Not A Problem

> Shutdown script does not do a clean shutdown
> 
>
> Key: KAFKA-6128
> URL: https://issues.apache.org/jira/browse/KAFKA-6128
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.11.0.1
>Reporter: Alastair Munro
>Priority: Minor
>
> Shutdown script (sending term signal) does not do a clean shutdown.
> We are running kafka in kubernetes/openshift 0.11.0.0. The statefulset kafka 
> runs the shutdown script prior to stopping the pod kafka is running on:
> {code}
> lifecycle:
>   preStop:
> exec:
>   command:
>   - ./bin/kafka-server-stop.sh
> {code}
> This worked perfectly in 0.11.0.0 but doesn't in 0.11.0.1. Also we see the 
> same behaviour if we send a TERM signal to the kafka process (same as the 
> shutdown script).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Reopened] (KAFKA-6128) Shutdown script does not do a clean shutdown

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma reopened KAFKA-6128:


> Shutdown script does not do a clean shutdown
> 
>
> Key: KAFKA-6128
> URL: https://issues.apache.org/jira/browse/KAFKA-6128
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.11.0.1
>Reporter: Alastair Munro
>Priority: Minor
>
> Shutdown script (sending term signal) does not do a clean shutdown.
> We are running kafka in kubernetes/openshift 0.11.0.0. The statefulset kafka 
> runs the shutdown script prior to stopping the pod kafka is running on:
> {code}
> lifecycle:
>   preStop:
> exec:
>   command:
>   - ./bin/kafka-server-stop.sh
> {code}
> This worked perfectly in 0.11.0.0 but doesn't in 0.11.0.1. Also we see the 
> same behaviour if we send a TERM signal to the kafka process (same as the 
> shutdown script).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Use self contained tokens instead of ACL

[Sönke Liebau]

Hi Peter,

I am not entirely sure that I understand what you mean when you say "at
application level" to be honest, but I do understand that you want to
forego 3rd party tools. However, this would mean that you have to implement
some form of trust between your portal which issues the tokens and Kafka
which checks the tokens. I am not sure that I'd recommend rolling your own
solution here, authentication has a lot of pitfalls that can turn around to
bite you.
Same thing as what I proposed with Kerberos could be done via certificates
of course. If your user requests access to a topic in the portal he uploads
a certificate signing request with the topic name encoded in some field
(lots of extensions to pick from). The portal signs the request and returns
it to the user. When the user now connects to Kafka he does so using this
certificate and it should be fairly easy for you to extend
the KafkaPrincipalBuilder class and extract the list of these topics from
the certificate. Then you'd need to also extend SimpleAclAuthorizer to
check for these topics in the principal name and allow access if the topic
is present in the certificate.

I am unsure of the benefits of this system over for example having the
portal create ACLs in Kafka, when the user is granted access to a topic.
One scenario that comes to mind is if you dynamically spin up new clusters,
if these new clusters all trust the same ca they would also accept these
tokens right away, no need to create ACLs in fresh clusters. But this could
also be solved by having a central repository for ACLs that are applied to
all clusters regularly..
A drawback of this system is that you'd need to revoke certificates if you
want to withdraw access to a topic, which is significantly harder that just
deleting an ACL.

Anyway, not sure if this helps you at all, just some random musings - if
you explain your specific needs a bit more we can discuss further.

Kind regards,
Sönke

On Wed, Oct 25, 2017 at 1:10 PM, Postmann, P. (Peter) <
peter.postm...@ing.com.invalid> wrote:

> Hi Sönke,
>
> Thanks for the fast replay. We don’t want to use Kerberos since we want to
> do the authorization on Application level and without involvement of a 3rd
> party during runtime.
>
> -Original Message-
> From: Sönke Liebau [mailto:soenke.lie...@opencore.com.INVALID]
> Sent: Mittwoch, 25. Oktober 2017 12:37
> To: dev@kafka.apache.org
> Subject: Re: Use self contained tokens instead of ACL
>
> The concept you describe sounds similar to what Microsoft calls "claims
> based authorization".
>
> At a high level I should think that using Kerberos as a vehicle to
> transport the information would be the way to go, as it is established and
> already supported by Kafka. I believe tickets have a field that can be used
> for authorization information, so if information about the topics that a
> user has access to were to be encoded in this field you could probably
> extend Kafka to extract that information and use it instead of ACLs.
>
> I am not well versed in what exactly Microsoft does and how you can
> control the granting side of things, but I do believe that AD server has
> support for something along those lines already.
>
> The upside of this would be that you don't have to implement anything
> around security, trust, encryption, etc. because everything is provided by
> Kerberos.
>
> Not much information in here I am afraid, but maybe a useful direction for
> future research.
>
> Kind regards,
> Sönke
>
> On Wed, Oct 25, 2017 at 11:55 AM, Postmann, P. (Peter) <
> peter.postm...@ing.com.invalid> wrote:
>
> > Hi everyone,
> >
> > I´m working on a concept to use Kafka with self-contained tokens
> > (instead of ACL).
> >
> > The idea:
> >
> > -  A client requests access to a certain topic (in some kind of
> > portal)
> >
> > -  The owner of the topic approves the request (in some kind of
> > portal)
> >
> > -  The client receives a signed tokens which contains the topic
> > (in some kind of portal)
> >
> > -  The client sends the token when he connects to Kafka
> >
> > -  Kafka validates the token and grants access
> >
> > Token Format:
> >
> > -  List of Topics and methods
> >
> > o   E.g. read /topic1
> >
> > -  Expire Date
> >
> > -  Signature
> >
> > Implementation Idea:
> >
> > -  Create a custom Authorization Class which checks the signature
> >
> > -  Implement the possibility to send arbitrary data (key->value)
> > along with the request when the client connects to the cluster
> >
> > I´m looking forward for feedback on this approach and would be happy
> > if you could give me a starting where to start with the implementation
> > (or if there already is a way to send arbitrary data to a custom
> Authorizer).
> >
> > Kind Regards,
> > Peter
> >
> > -
> > ATTENTION:
> > The information in this e-mail is confidential and only meant for the
> > 

[jira] [Resolved] (KAFKA-6128) Shutdown script does not do a clean shutdown

[Alastair Munro (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alastair Munro resolved KAFKA-6128.
---
   Resolution: Fixed
Fix Version/s: 0.11.0.0
   0.11.0.1

Broken zookeeper replication for /brokers/ids/.

> Shutdown script does not do a clean shutdown
> 
>
> Key: KAFKA-6128
> URL: https://issues.apache.org/jira/browse/KAFKA-6128
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.11.0.1
>Reporter: Alastair Munro
>Priority: Minor
> Fix For: 0.11.0.1, 0.11.0.0
>
>
> Shutdown script (sending term signal) does not do a clean shutdown.
> We are running kafka in kubernetes/openshift 0.11.0.0. The statefulset kafka 
> runs the shutdown script prior to stopping the pod kafka is running on:
> {code}
> lifecycle:
>   preStop:
> exec:
>   command:
>   - ./bin/kafka-server-stop.sh
> {code}
> This worked perfectly in 0.11.0.0 but doesn't in 0.11.0.1. Also we see the 
> same behaviour if we send a TERM signal to the kafka process (same as the 
> shutdown script).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] kafka pull request #4138: MINOR: Revert EmbeddedZooKeeper rename

[ijuma]

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/4138

MINOR: Revert EmbeddedZooKeeper rename

Even though this class is internal, it's widely
used by other projects and it's better to avoid
breaking them until we have a publicly supported
test library.

Also made some fields private since users should
not need access to them.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka revert-embedded-zookeeper-rename

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/4138.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4138


commit 0f2a1e3559c0760c4337b875eb426f0b361bb2dc
Author: Ismael Juma 
Date:   2017-10-26T10:52:01Z

MINOR: Revert EmbeddedZooKeeper rename

Even though this class is internal, it's widely
used by other projects and it's better to avoid
breaking them until we have a publicly supported
test library.

Also made some fields private since users should
not need access to them.




---

[jira] [Created] (KAFKA-6130) VerifiableConsume with --max-messages

[Tom Bentley (JIRA)]

Tom Bentley created KAFKA-6130:
--

 Summary: VerifiableConsume with --max-messages 
 Key: KAFKA-6130
 URL: https://issues.apache.org/jira/browse/KAFKA-6130
 Project: Kafka
  Issue Type: Bug
Reporter: Tom Bentley
Assignee: Tom Bentley
Priority: Minor


If I run {{kafka-verifiable-consumer.sh --max-messages=N}} I expect the tool to 
consume N messages and then exit. It will actually consume as many messages as 
are in the topic and then block.

The problem is that although  the max messages will cause the loop in 
onRecordsReceived() to break, the loop in run() will just call 
onRecordsReceived() again.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-6129) kafka issue when exposing through nodeport in kubernetes

[Francesco vigotti (JIRA)]

Francesco vigotti created KAFKA-6129:


 Summary: kafka issue when exposing through nodeport in kubernetes
 Key: KAFKA-6129
 URL: https://issues.apache.org/jira/browse/KAFKA-6129
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.10.2.1
 Environment: kubernetes
Reporter: Francesco vigotti
Priority: Critical


I've started writing in this issue: 
https://issues.apache.org/jira/browse/KAFKA-2729
but then I'm going to open this new issue because I've probably found the cause 
in my kubernetes setup, but In my opinion kubernetes did nothing wrong in his 
setup ( and all other application works using the same nodeport redirection , 
ie: zookeeper )
kafka brokers fails , silently (randomly in multiple brokers setup)  and with a 
misleading error from producer so I think that Kafka should be improved, 
providing more robust pre-startup flight-checks and identifying/reporting the 
current issue 

After further investigation from my reply here 
https://issues.apache.org/jira/browse/KAFKA-2729  with a minimum size cluster ( 
1 zk + 1 kafka-broker ) I've found the problem, 
the problem is with kubernetes, ( I don't know why this issue appeared only now 
to me , if something changed in recent kube-proxy versions or in kafka 0.10+ , 
or ... ) 
anyway my old kafka cluster started being underreplicated and return various 
problem , 

the problem happens when in kubernetes pods are created and redirected using a 
nodeport-service ( over a static ip in my case ) to expose kafka brokers from 
the host, when using hostNetwork  ( so no redirection ) everything works, what 
is strange is that zookeeper instead works fine with nodeport ( which create a 
redirection rule in iptables->nat->prerouting ) the only application I've found 
problems with this kubernetes configuration is kafka,
what is weird is that kafka starts correctly without errors, but on multiple 
broker clusters there are random issues, on single broker cluster instead the 
console-producer fails with infinite looop of :

```
[2017-10-26 09:38:23,281] WARN Error while fetching metadata with correlation 
id 5 : {test6=UNKNOWN_TOPIC_OR_PARTITION} 
(org.apache.kafka.clients.NetworkClient)
[2017-10-26 09:38:23,383] WARN Error while fetching metadata with correlation 
id 6 : {test6=UNKNOWN_TOPIC_OR_PARTITION} 
(org.apache.kafka.clients.NetworkClient)
[2017-10-26 09:38:23,485] WARN Error while fetching metadata with correlation 
id 7 : {test6=UNKNOWN_TOPIC_OR_PARTITION} 
(org.apache.kafka.clients.NetworkClient)
```
, still no errors reported from broker or zookeeper,
Also I want to say that I've come across this discussion : 
 
https://stackoverflow.com/questions/35788697/leader-not-available-kafka-in-console-producer
 
but the proposed solution for the host pod ( to allow self-resolving of 
advertised hostname) didn't worked 
``` 
hostAliases:
  - ip: "127.0.0.1"
hostnames:
- "---myhosthostname---"







--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: kafka-1.0-jdk7 #56

[Apache Jenkins Server]

See 


Changes:

[me] MINOR: reset state in cleanup, fixes jmx mixin flakiness

--
[...truncated 1.82 MB...]

org.apache.kafka.streams.TopologyTest > 
singleSourcePatternShouldHaveSingleSubtopology PASSED

org.apache.kafka.streams.TopologyTest > singleSourceShouldHaveSingleSubtopology 
STARTED

org.apache.kafka.streams.TopologyTest > singleSourceShouldHaveSingleSubtopology 
PASSED

org.apache.kafka.streams.TopologyTest > 
processorsWithSameSinkShouldHaveSameSubtopology STARTED

org.apache.kafka.streams.TopologyTest > 
processorsWithSameSinkShouldHaveSameSubtopology PASSED

org.apache.kafka.streams.TopologyTest > 
shouldNotAllowNullTopicsWhenAddingSoureWithTopic STARTED

org.apache.kafka.streams.TopologyTest > 
shouldNotAllowNullTopicsWhenAddingSoureWithTopic PASSED

org.apache.kafka.streams.TopologyTest > shouldFailWithUnknownParent STARTED

org.apache.kafka.streams.TopologyTest > shouldFailWithUnknownParent PASSED

org.apache.kafka.streams.TopologyTest > shouldFailIfNodeIsItsOwnParent STARTED

org.apache.kafka.streams.TopologyTest > shouldFailIfNodeIsItsOwnParent PASSED

org.apache.kafka.streams.TopologyTest > shouldNotAllowToAddStateStoreToSource 
STARTED

org.apache.kafka.streams.TopologyTest > shouldNotAllowToAddStateStoreToSource 
PASSED

org.apache.kafka.streams.TopologyTest > 
shouldNotAllowNullNameWhenAddingProcessor STARTED

org.apache.kafka.streams.TopologyTest > 
shouldNotAllowNullNameWhenAddingProcessor PASSED

org.apache.kafka.streams.TopologyTest > shouldNotAllowToAddStoreWithSameName 
STARTED

org.apache.kafka.streams.TopologyTest > shouldNotAllowToAddStoreWithSameName 
PASSED

org.apache.kafka.streams.TopologyTest > 
shouldNotAllowNullNameWhenAddingSourceWithPattern STARTED

org.apache.kafka.streams.TopologyTest > 
shouldNotAllowNullNameWhenAddingSourceWithPattern PASSED

org.apache.kafka.streams.TopologyTest > 
multipleSourcesWithSinksShouldHaveDistinctSubtopologies STARTED

org.apache.kafka.streams.TopologyTest > 
multipleSourcesWithSinksShouldHaveDistinctSubtopologies PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldResetToDefaultIfProducerEnableIdempotenceIsOverriddenIfEosEnabled STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldResetToDefaultIfProducerEnableIdempotenceIsOverriddenIfEosEnabled PASSED

org.apache.kafka.streams.StreamsConfigTest > shouldUseNewConfigsWhenPresent 
STARTED

org.apache.kafka.streams.StreamsConfigTest > shouldUseNewConfigsWhenPresent 
PASSED

org.apache.kafka.streams.StreamsConfigTest > testGetConsumerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > testGetConsumerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > shouldAcceptAtLeastOnce STARTED

org.apache.kafka.streams.StreamsConfigTest > shouldAcceptAtLeastOnce PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldUseCorrectDefaultsWhenNoneSpecified STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldUseCorrectDefaultsWhenNoneSpecified PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldAllowSettingProducerEnableIdempotenceIfEosDisabled STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldAllowSettingProducerEnableIdempotenceIfEosDisabled PASSED

org.apache.kafka.streams.StreamsConfigTest > defaultSerdeShouldBeConfigured 
STARTED

org.apache.kafka.streams.StreamsConfigTest > defaultSerdeShouldBeConfigured 
PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSetDifferentDefaultsIfEosEnabled STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSetDifferentDefaultsIfEosEnabled PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldNotOverrideUserConfigRetriesIfExactlyOnceEnabled STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldNotOverrideUserConfigRetriesIfExactlyOnceEnabled PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedConsumerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedConsumerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldOverrideStreamsDefaultConsumerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldOverrideStreamsDefaultConsumerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > testGetProducerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > testGetProducerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldAllowSettingProducerMaxInFlightRequestPerConnectionsWhenEosDisabled 
STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldAllowSettingProducerMaxInFlightRequestPerConnectionsWhenEosDisabled PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldThrowStreamsExceptionIfValueSerdeConfigFails STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldThrowStreamsExceptionIfValueSerdeConfigFails PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldResetToDefaultIfConsumerAutoCommitIsOverridden STARTED

[jira] [Created] (KAFKA-6128) Shutdown script does not do a clean shutdown

[Alastair Munro (JIRA)]

Alastair Munro created KAFKA-6128:
-

 Summary: Shutdown script does not do a clean shutdown
 Key: KAFKA-6128
 URL: https://issues.apache.org/jira/browse/KAFKA-6128
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.11.0.1
Reporter: Alastair Munro
Priority: Minor


Shutdown script (sending term signal) does not do a clean shutdown.

We are running kafka in kubernetes/openshift 0.11.0.0. The statefulset kafka 
runs the shutdown script prior to stopping the pod kafka is running on:

{code}
lifecycle:
  preStop:
exec:
  command:
  - ./bin/kafka-server-stop.sh
{code}

This worked perfectly in 0.11.0.0 but doesn't in 0.11.0.1. Also we see the same 
behaviour if we send a TERM signal to the kafka process (same as the shutdown 
script).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSS] KIP-214: Add zookeeper.max.in.flight.requests config to the broker

[Ted Yu]

Can you fill out JIRA number ?
Thanks
 Original message From: Onur Karaman 
 Date: 10/25/17  10:37 PM  (GMT-08:00) To: 
dev@kafka.apache.org Subject: [DISCUSS] KIP-214: Add 
zookeeper.max.in.flight.requests config to the broker 
Hey everyone.

Giving this another shot since it looks like there was a KIP number
collision on the wiki page.

I made a config kip, KIP-214: Add zookeeper.max.in.flight.requests config
to the broker:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-214%3A+Add+zookeeper.max.in.flight.requests+config+to+the+broker

Comments are welcome.

- Onur