Build failed in Jenkins: kafka-trunk-jdk7 #1501

[Apache Jenkins Server]

See 

Changes:

[ismael] MINOR: Move a few methods from the `ZKUtils` class to the companion

[cshapi] MINOR: Update Kafka configuration documentation to use kafka-configs.…

[cshapi] KAFKA-4052: Allow passing properties file to ProducerPerformance

[cshapi] KAFKA-4070: implement Connect Struct.toString()

[cshapi] MINOR: Improve log message in `ReplicaManager.becomeLeaderOrFollower`

--
[...truncated 3468 lines...]

kafka.producer.AsyncProducerTest > testIncompatibleEncoder STARTED

kafka.producer.AsyncProducerTest > testIncompatibleEncoder PASSED

kafka.producer.ProducerTest > testSendToNewTopic STARTED

kafka.producer.ProducerTest > testSendToNewTopic PASSED

kafka.producer.ProducerTest > testAsyncSendCanCorrectlyFailWithTimeout STARTED

kafka.producer.ProducerTest > testAsyncSendCanCorrectlyFailWithTimeout PASSED

kafka.producer.ProducerTest > testSendNullMessage STARTED

kafka.producer.ProducerTest > testSendNullMessage PASSED

kafka.producer.ProducerTest > testUpdateBrokerPartitionInfo STARTED

kafka.producer.ProducerTest > testUpdateBrokerPartitionInfo PASSED

kafka.producer.ProducerTest > testSendWithDeadBroker STARTED

kafka.producer.ProducerTest > testSendWithDeadBroker PASSED

kafka.tools.ConsoleConsumerTest > shouldLimitReadsToMaxMessageLimit STARTED

kafka.tools.ConsoleConsumerTest > shouldLimitReadsToMaxMessageLimit PASSED

kafka.tools.ConsoleConsumerTest > shouldParseValidNewConsumerValidConfig STARTED

kafka.tools.ConsoleConsumerTest > shouldParseValidNewConsumerValidConfig PASSED

kafka.tools.ConsoleConsumerTest > shouldStopWhenOutputCheckErrorFails STARTED

kafka.tools.ConsoleConsumerTest > shouldStopWhenOutputCheckErrorFails PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewSimpleConsumerValidConfigWithStringOffset STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewSimpleConsumerValidConfigWithStringOffset PASSED

kafka.tools.ConsoleConsumerTest > shouldParseConfigsFromFile STARTED

kafka.tools.ConsoleConsumerTest > shouldParseConfigsFromFile PASSED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewSimpleConsumerValidConfigWithNumericOffset STARTED

kafka.tools.ConsoleConsumerTest > 
shouldParseValidNewSimpleConsumerValidConfigWithNumericOffset PASSED

kafka.tools.ConsoleConsumerTest > shouldParseValidOldConsumerValidConfig STARTED

kafka.tools.ConsoleConsumerTest > shouldParseValidOldConsumerValidConfig PASSED

kafka.tools.ConsoleProducerTest > testParseKeyProp STARTED

kafka.tools.ConsoleProducerTest > testParseKeyProp PASSED

kafka.tools.ConsoleProducerTest > testValidConfigsOldProducer STARTED

kafka.tools.ConsoleProducerTest > testValidConfigsOldProducer PASSED

kafka.tools.ConsoleProducerTest > testInvalidConfigs STARTED

kafka.tools.ConsoleProducerTest > testInvalidConfigs PASSED

kafka.tools.ConsoleProducerTest > testValidConfigsNewProducer STARTED

kafka.tools.ConsoleProducerTest > testValidConfigsNewProducer PASSED

kafka.tools.MirrorMakerTest > 
testDefaultMirrorMakerMessageHandlerWithNoTimestampInSourceMessage STARTED

kafka.tools.MirrorMakerTest > 
testDefaultMirrorMakerMessageHandlerWithNoTimestampInSourceMessage PASSED

kafka.tools.MirrorMakerTest > testDefaultMirrorMakerMessageHandler STARTED

kafka.tools.MirrorMakerTest > testDefaultMirrorMakerMessageHandler PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testSizeInBytesWithCompression 
STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testSizeInBytesWithCompression 
PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > 
testIteratorIsConsistentWithCompression STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > 
testIteratorIsConsistentWithCompression PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testIteratorIsConsistent 
STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testIteratorIsConsistent PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testEqualsWithCompression 
STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testEqualsWithCompression 
PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testWrittenEqualsRead STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testWrittenEqualsRead PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testEquals STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testEquals PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testSizeInBytes STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testSizeInBytes PASSED

kafka.javaapi.consumer.ZookeeperConsumerConnectorTest > testBasic STARTED

kafka.javaapi.consumer.ZookeeperConsumerConnectorTest > testBasic PASSED

kafka.common.ConfigTest > testInvalidGroupIds STARTED

kafka.common.ConfigTest > testInvalidGroupIds PASSED

kafka.common.ConfigTest > testInvalidClientIds STARTED

kafka.common.ConfigTest > testInvalidClientIds PASSED

kafka.common.ZkNodeChangeNotificationListenerTest > testProcessNotification 
STARTED

Build failed in Jenkins: kafka-trunk-jdk8 #845

[Apache Jenkins Server]

See 

Changes:

[ismael] MINOR: Move a few methods from the `ZKUtils` class to the companion

--
[...truncated 12257 lines...]
org.apache.kafka.streams.processor.internals.MinTimestampTrackerTest > 
testTracking PASSED

org.apache.kafka.streams.processor.internals.StreamTaskTest > testProcessOrder 
STARTED

org.apache.kafka.streams.processor.internals.StreamTaskTest > testProcessOrder 
PASSED

org.apache.kafka.streams.processor.internals.StreamTaskTest > testPauseResume 
STARTED

org.apache.kafka.streams.processor.internals.StreamTaskTest > testPauseResume 
PASSED

org.apache.kafka.streams.processor.internals.StreamTaskTest > 
testMaybePunctuate STARTED

org.apache.kafka.streams.processor.internals.StreamTaskTest > 
testMaybePunctuate PASSED

org.apache.kafka.streams.processor.internals.assignment.AssignmentInfoTest > 
testEncodeDecode STARTED

org.apache.kafka.streams.processor.internals.assignment.AssignmentInfoTest > 
testEncodeDecode PASSED

org.apache.kafka.streams.processor.internals.assignment.AssignmentInfoTest > 
shouldDecodePreviousVersion STARTED

org.apache.kafka.streams.processor.internals.assignment.AssignmentInfoTest > 
shouldDecodePreviousVersion PASSED

org.apache.kafka.streams.processor.internals.assignment.SubscriptionInfoTest > 
testEncodeDecode STARTED

org.apache.kafka.streams.processor.internals.assignment.SubscriptionInfoTest > 
testEncodeDecode PASSED

org.apache.kafka.streams.processor.internals.assignment.SubscriptionInfoTest > 
shouldEncodeDecodeWithUserEndPoint STARTED

org.apache.kafka.streams.processor.internals.assignment.SubscriptionInfoTest > 
shouldEncodeDecodeWithUserEndPoint PASSED

org.apache.kafka.streams.processor.internals.assignment.SubscriptionInfoTest > 
shouldBeBackwardCompatible STARTED

org.apache.kafka.streams.processor.internals.assignment.SubscriptionInfoTest > 
shouldBeBackwardCompatible PASSED

org.apache.kafka.streams.processor.internals.assignment.TaskAssignorTest > 
testStickiness STARTED

org.apache.kafka.streams.processor.internals.assignment.TaskAssignorTest > 
testStickiness PASSED

org.apache.kafka.streams.processor.internals.assignment.TaskAssignorTest > 
testAssignWithStandby STARTED

org.apache.kafka.streams.processor.internals.assignment.TaskAssignorTest > 
testAssignWithStandby PASSED

org.apache.kafka.streams.processor.internals.assignment.TaskAssignorTest > 
testAssignWithoutStandby STARTED

org.apache.kafka.streams.processor.internals.assignment.TaskAssignorTest > 
testAssignWithoutStandby PASSED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingMultiplexingTopology STARTED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingMultiplexingTopology PASSED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingStatefulTopology STARTED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingStatefulTopology PASSED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingSimpleTopology STARTED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingSimpleTopology PASSED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testTopologyMetadata STARTED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testTopologyMetadata PASSED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingMultiplexByNameTopology STARTED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingMultiplexByNameTopology PASSED

org.apache.kafka.streams.processor.internals.RecordCollectorTest > 
testSpecificPartition STARTED

org.apache.kafka.streams.processor.internals.RecordCollectorTest > 
testSpecificPartition PASSED

org.apache.kafka.streams.processor.internals.RecordCollectorTest > 
testStreamPartitioner STARTED

org.apache.kafka.streams.processor.internals.RecordCollectorTest > 
testStreamPartitioner PASSED

org.apache.kafka.streams.processor.internals.PunctuationQueueTest > 
testPunctuationInterval STARTED

org.apache.kafka.streams.processor.internals.PunctuationQueueTest > 
testPunctuationInterval PASSED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
shouldThrowExceptionIfApplicationServerConfigIsNotHostPortPair STARTED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
shouldThrowExceptionIfApplicationServerConfigIsNotHostPortPair PASSED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
shouldMapUserEndPointToTopicPartitions STARTED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
shouldMapUserEndPointToTopicPartitions PASSED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
shouldAddUserDefinedEndPointToSubscription STARTED

org.apache.kafka.streams.processor.internals.StreamPar

[jira] [Commented] (KAFKA-1650) Mirror Maker could lose data on unclean shutdown.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15438411#comment-15438411
 ] 

ASF GitHub Bot commented on KAFKA-1650:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1654


> Mirror Maker could lose data on unclean shutdown.
> -
>
> Key: KAFKA-1650
> URL: https://issues.apache.org/jira/browse/KAFKA-1650
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Fix For: 0.9.0.0
>
> Attachments: KAFKA-1650.patch, KAFKA-1650_2014-10-06_10:17:46.patch, 
> KAFKA-1650_2014-11-12_09:51:30.patch, KAFKA-1650_2014-11-17_18:44:37.patch, 
> KAFKA-1650_2014-11-20_12:00:16.patch, KAFKA-1650_2014-11-24_08:15:17.patch, 
> KAFKA-1650_2014-12-03_15:02:31.patch, KAFKA-1650_2014-12-03_19:02:13.patch, 
> KAFKA-1650_2014-12-04_11:59:07.patch, KAFKA-1650_2014-12-06_18:58:57.patch, 
> KAFKA-1650_2014-12-08_01:36:01.patch, KAFKA-1650_2014-12-16_08:03:45.patch, 
> KAFKA-1650_2014-12-17_12:29:23.patch, KAFKA-1650_2014-12-18_18:48:18.patch, 
> KAFKA-1650_2014-12-18_22:17:08.patch, KAFKA-1650_2014-12-18_22:53:26.patch, 
> KAFKA-1650_2014-12-18_23:41:16.patch, KAFKA-1650_2014-12-22_19:07:24.patch, 
> KAFKA-1650_2014-12-23_07:04:28.patch, KAFKA-1650_2014-12-23_16:44:06.patch
>
>
> Currently if mirror maker got shutdown uncleanly, the data in the data 
> channel and buffer could potentially be lost. With the new producer's 
> callback, this issue could be solved.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1654: MINOR: Update MirrorMaker docs to remove multiple ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1654


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #1706: MINOR: doc changes for QueueTimeMs JMX metrics.

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1706


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (KAFKA-3742) Can't run connect-distributed.sh with -daemon flag

[Gwen Shapira (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gwen Shapira resolved KAFKA-3742.
-
   Resolution: Fixed
Fix Version/s: 0.10.1.0

Issue resolved by pull request 1717
[https://github.com/apache/kafka/pull/1717]

> Can't run connect-distributed.sh with -daemon flag
> --
>
> Key: KAFKA-3742
> URL: https://issues.apache.org/jira/browse/KAFKA-3742
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 0.10.0.0
>Reporter: Geoff Anderson
>Assignee: Liquan Pei
>Priority: Minor
> Fix For: 0.10.1.0
>
>
> Running on ubuntu 14.04. Discovered while experimenting various different 
> kafka components. 
> This error probably applies to other scripts as well.
> Running connect-distributed.sh thusly
> {code}connect-distributed.sh -daemon /tmp/connect-distributed.properties{code}
> gives errors like this 
> {code}
> root@worker1:/home/vagrant# connect-distributed.sh -daemon 
> /tmp/connect-distributed.properties
> Exception in thread "main" java.io.FileNotFoundException: -daemon (No such 
> file or directory)
>   at java.io.FileInputStream.open(Native Method)
>   at java.io.FileInputStream.(FileInputStream.java:146)
>   at java.io.FileInputStream.(FileInputStream.java:101)
>   at org.apache.kafka.common.utils.Utils.loadProps(Utils.java:446)
>   at 
> org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:61)
> {code}
> Note that this runs:
> connect-distributed.sh /tmp/connect-distributed.properties -daemon
> However, the daemon flag is not activated in this case
> Underlying cause:
> kafka-run-class.sh assumes -daemon comes before the classpath
> The scripts for which -daemon works use something like
> {code}
> EXTRA_ARGS="-name kafkaServer -loggc"
> COMMAND=$1
> case $COMMAND in
>   -daemon)
> EXTRA_ARGS="-daemon "$EXTRA_ARGS
> shift
> ;;
>   *)
> ;;
> esac
> exec $base_dir/kafka-run-class.sh $EXTRA_ARGS 
> io.confluent.support.metrics.SupportedKafka "$@"
> {code}
> but connect-distributed does this:
> {code}
> exec $(dirname $0)/kafka-run-class.sh 
> org.apache.kafka.connect.cli.ConnectDistributed "$@"
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3742) Can't run connect-distributed.sh with -daemon flag

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15438399#comment-15438399
 ] 

ASF GitHub Bot commented on KAFKA-3742:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1717


> Can't run connect-distributed.sh with -daemon flag
> --
>
> Key: KAFKA-3742
> URL: https://issues.apache.org/jira/browse/KAFKA-3742
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 0.10.0.0
>Reporter: Geoff Anderson
>Assignee: Liquan Pei
>Priority: Minor
> Fix For: 0.10.1.0
>
>
> Running on ubuntu 14.04. Discovered while experimenting various different 
> kafka components. 
> This error probably applies to other scripts as well.
> Running connect-distributed.sh thusly
> {code}connect-distributed.sh -daemon /tmp/connect-distributed.properties{code}
> gives errors like this 
> {code}
> root@worker1:/home/vagrant# connect-distributed.sh -daemon 
> /tmp/connect-distributed.properties
> Exception in thread "main" java.io.FileNotFoundException: -daemon (No such 
> file or directory)
>   at java.io.FileInputStream.open(Native Method)
>   at java.io.FileInputStream.(FileInputStream.java:146)
>   at java.io.FileInputStream.(FileInputStream.java:101)
>   at org.apache.kafka.common.utils.Utils.loadProps(Utils.java:446)
>   at 
> org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:61)
> {code}
> Note that this runs:
> connect-distributed.sh /tmp/connect-distributed.properties -daemon
> However, the daemon flag is not activated in this case
> Underlying cause:
> kafka-run-class.sh assumes -daemon comes before the classpath
> The scripts for which -daemon works use something like
> {code}
> EXTRA_ARGS="-name kafkaServer -loggc"
> COMMAND=$1
> case $COMMAND in
>   -daemon)
> EXTRA_ARGS="-daemon "$EXTRA_ARGS
> shift
> ;;
>   *)
> ;;
> esac
> exec $base_dir/kafka-run-class.sh $EXTRA_ARGS 
> io.confluent.support.metrics.SupportedKafka "$@"
> {code}
> but connect-distributed does this:
> {code}
> exec $(dirname $0)/kafka-run-class.sh 
> org.apache.kafka.connect.cli.ConnectDistributed "$@"
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1717: KAFKA-3742: (FIX) Can't run bin/connect-*.sh with ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1717


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #1789: MINOR: Improve log message in `ReplicaManager.beco...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1789


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-4070) Implement a useful Struct.toString()

[Gwen Shapira (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gwen Shapira updated KAFKA-4070:

Assignee: Shikhar Bhushan

> Implement a useful Struct.toString()
> 
>
> Key: KAFKA-4070
> URL: https://issues.apache.org/jira/browse/KAFKA-4070
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Reporter: Shikhar Bhushan
>Assignee: Shikhar Bhushan
>Priority: Minor
> Fix For: 0.10.1.0
>
>
> Logging of {{Struct}}'s does not currently provide any useful output, and 
> users also find it unhelpful e.g. when hooking up a Kafka topic with Avro 
> data with the {{FileSinkConnector}} which simply {{toString()}}'s the values 
> to the file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4070) Implement a useful Struct.toString()

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15438336#comment-15438336
 ] 

ASF GitHub Bot commented on KAFKA-4070:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1790


> Implement a useful Struct.toString()
> 
>
> Key: KAFKA-4070
> URL: https://issues.apache.org/jira/browse/KAFKA-4070
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Reporter: Shikhar Bhushan
>Priority: Minor
> Fix For: 0.10.1.0
>
>
> Logging of {{Struct}}'s does not currently provide any useful output, and 
> users also find it unhelpful e.g. when hooking up a Kafka topic with Avro 
> data with the {{FileSinkConnector}} which simply {{toString()}}'s the values 
> to the file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1790: KAFKA-4070: implement Connect Struct.toString()

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1790


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (KAFKA-4070) Implement a useful Struct.toString()

[Gwen Shapira (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gwen Shapira resolved KAFKA-4070.
-
   Resolution: Fixed
Fix Version/s: 0.10.1.0

Issue resolved by pull request 1790
[https://github.com/apache/kafka/pull/1790]

> Implement a useful Struct.toString()
> 
>
> Key: KAFKA-4070
> URL: https://issues.apache.org/jira/browse/KAFKA-4070
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Reporter: Shikhar Bhushan
>Priority: Minor
> Fix For: 0.10.1.0
>
>
> Logging of {{Struct}}'s does not currently provide any useful output, and 
> users also find it unhelpful e.g. when hooking up a Kafka topic with Avro 
> data with the {{FileSinkConnector}} which simply {{toString()}}'s the values 
> to the file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4052) Allow passing properties file to ProducerPerformance

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15438327#comment-15438327
 ] 

ASF GitHub Bot commented on KAFKA-4052:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1749


> Allow passing properties file to ProducerPerformance
> 
>
> Key: KAFKA-4052
> URL: https://issues.apache.org/jira/browse/KAFKA-4052
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Ashish K Singh
>Assignee: Ashish K Singh
> Fix For: 0.10.1.0
>
>
> Allow passing properties file to ProducerPerformance, to enable using the 
> tool against secure Kafka installs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1749: KAFKA-4052: Allow passing properties file to Produ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1749


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-4052) Allow passing properties file to ProducerPerformance

[Gwen Shapira (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gwen Shapira updated KAFKA-4052:

   Resolution: Fixed
Fix Version/s: 0.10.1.0
   Status: Resolved  (was: Patch Available)

Issue resolved by pull request 1749
[https://github.com/apache/kafka/pull/1749]

> Allow passing properties file to ProducerPerformance
> 
>
> Key: KAFKA-4052
> URL: https://issues.apache.org/jira/browse/KAFKA-4052
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Ashish K Singh
>Assignee: Ashish K Singh
> Fix For: 0.10.1.0
>
>
> Allow passing properties file to ProducerPerformance, to enable using the 
> tool against secure Kafka installs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1772: MINOR: Update Kafka configuration documentation to...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1772


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: kafka-trunk-jdk7 #1500

[Apache Jenkins Server]

See 

Changes:

[ismael] HOTFIX: disabled application-reset-tool integration test

[ismael] HOTFIX: Fix verbose logging in ControllerChannelManager.brokerReady

--
[...truncated 12215 lines...]
org.apache.kafka.streams.processor.internals.MinTimestampTrackerTest > 
testTracking PASSED

org.apache.kafka.streams.processor.internals.StreamTaskTest > testProcessOrder 
STARTED

org.apache.kafka.streams.processor.internals.StreamTaskTest > testProcessOrder 
PASSED

org.apache.kafka.streams.processor.internals.StreamTaskTest > testPauseResume 
STARTED

org.apache.kafka.streams.processor.internals.StreamTaskTest > testPauseResume 
PASSED

org.apache.kafka.streams.processor.internals.StreamTaskTest > 
testMaybePunctuate STARTED

org.apache.kafka.streams.processor.internals.StreamTaskTest > 
testMaybePunctuate PASSED

org.apache.kafka.streams.processor.internals.assignment.AssignmentInfoTest > 
testEncodeDecode STARTED

org.apache.kafka.streams.processor.internals.assignment.AssignmentInfoTest > 
testEncodeDecode PASSED

org.apache.kafka.streams.processor.internals.assignment.AssignmentInfoTest > 
shouldDecodePreviousVersion STARTED

org.apache.kafka.streams.processor.internals.assignment.AssignmentInfoTest > 
shouldDecodePreviousVersion PASSED

org.apache.kafka.streams.processor.internals.assignment.SubscriptionInfoTest > 
testEncodeDecode STARTED

org.apache.kafka.streams.processor.internals.assignment.SubscriptionInfoTest > 
testEncodeDecode PASSED

org.apache.kafka.streams.processor.internals.assignment.SubscriptionInfoTest > 
shouldEncodeDecodeWithUserEndPoint STARTED

org.apache.kafka.streams.processor.internals.assignment.SubscriptionInfoTest > 
shouldEncodeDecodeWithUserEndPoint PASSED

org.apache.kafka.streams.processor.internals.assignment.SubscriptionInfoTest > 
shouldBeBackwardCompatible STARTED

org.apache.kafka.streams.processor.internals.assignment.SubscriptionInfoTest > 
shouldBeBackwardCompatible PASSED

org.apache.kafka.streams.processor.internals.assignment.TaskAssignorTest > 
testStickiness STARTED

org.apache.kafka.streams.processor.internals.assignment.TaskAssignorTest > 
testStickiness PASSED

org.apache.kafka.streams.processor.internals.assignment.TaskAssignorTest > 
testAssignWithStandby STARTED

org.apache.kafka.streams.processor.internals.assignment.TaskAssignorTest > 
testAssignWithStandby PASSED

org.apache.kafka.streams.processor.internals.assignment.TaskAssignorTest > 
testAssignWithoutStandby STARTED

org.apache.kafka.streams.processor.internals.assignment.TaskAssignorTest > 
testAssignWithoutStandby PASSED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingMultiplexingTopology STARTED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingMultiplexingTopology PASSED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingStatefulTopology STARTED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingStatefulTopology PASSED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingSimpleTopology STARTED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingSimpleTopology PASSED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testTopologyMetadata STARTED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testTopologyMetadata PASSED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingMultiplexByNameTopology STARTED

org.apache.kafka.streams.processor.internals.ProcessorTopologyTest > 
testDrivingMultiplexByNameTopology PASSED

org.apache.kafka.streams.processor.internals.RecordCollectorTest > 
testSpecificPartition STARTED

org.apache.kafka.streams.processor.internals.RecordCollectorTest > 
testSpecificPartition PASSED

org.apache.kafka.streams.processor.internals.RecordCollectorTest > 
testStreamPartitioner STARTED

org.apache.kafka.streams.processor.internals.RecordCollectorTest > 
testStreamPartitioner PASSED

org.apache.kafka.streams.processor.internals.PunctuationQueueTest > 
testPunctuationInterval STARTED

org.apache.kafka.streams.processor.internals.PunctuationQueueTest > 
testPunctuationInterval PASSED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
shouldThrowExceptionIfApplicationServerConfigIsNotHostPortPair STARTED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
shouldThrowExceptionIfApplicationServerConfigIsNotHostPortPair PASSED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
shouldMapUserEndPointToTopicPartitions STARTED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
shouldMapUserEndPointToTopicPartitions PASSED

org.apache.kafka.streams.processor.internals.StreamPartitionAssignorTest > 
shouldAddUserDefinedEndPointToSubscrip

[GitHub] kafka pull request #1775: MINOR: Move a few methods from the `ZKUtils` class...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1775


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-4089) KafkaProducer raises Batch Expired exception

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15438269#comment-15438269
 ] 

ASF GitHub Bot commented on KAFKA-4089:
---

GitHub user sutambe opened a pull request:

https://github.com/apache/kafka/pull/1791

KAFKA-4089: KafkaProducer expires batch when metadata is stale



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sutambe/kafka batch-expired

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1791.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1791


commit aeab247716506ed226bf3bb118e7aedf9981e9a6
Author: Sumant Tambe 
Date:   2016-08-26T01:24:06Z

KAFKA-4089: fixing batch expired when stale metadata

commit 53f2a6e628c715efe14afebf3725d1a38f9632b9
Author: Sumant Tambe 
Date:   2016-08-26T01:24:06Z

KAFKA-4089: fixing batch expired when stale metadata

commit 8200d8b4822440fefa1a58a962bc746171b260eb
Author: Sumant Tambe 
Date:   2016-08-26T01:30:41Z

Merge branch 'batch-expired' of https://github.com/sutambe/kafka into 
batch-expired




> KafkaProducer raises Batch Expired exception 
> -
>
> Key: KAFKA-4089
> URL: https://issues.apache.org/jira/browse/KAFKA-4089
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.10.0.1
>Reporter: Sumant Tambe
>Assignee: Dong Lin
>
> The basic idea of batch expiration is that we don't expire batches when 
> producer thinks "it can make progress". Currently the notion of "making 
> progress" involves only in-flight requests (muted partitions). That's not 
> sufficient. The other half of the "making progress" is that if we have stale 
> metadata, we cannot trust it and therefore can't say we can't make progress. 
> Therefore, we don't expire batched when metadata is stale. This also implies 
> we don't want to expire batches when we can still make progress even if the 
> batch remains in the queue longer than the batch expiration time. 
> The current condition in {{abortExpiredBatches}} that bypasses muted 
> partitions is necessary but not sufficient. It should additionally restrict 
> ejection when metadata is stale. 
> Conversely, it should expire batches only when the following is true
> # !muted AND
> # meta-data is fresh AND
> # batch remained in the queue longer than request timeout.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1791: KAFKA-4089: KafkaProducer expires batch when metad...

[sutambe]

GitHub user sutambe opened a pull request:

https://github.com/apache/kafka/pull/1791

KAFKA-4089: KafkaProducer expires batch when metadata is stale



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sutambe/kafka batch-expired

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1791.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1791


commit aeab247716506ed226bf3bb118e7aedf9981e9a6
Author: Sumant Tambe 
Date:   2016-08-26T01:24:06Z

KAFKA-4089: fixing batch expired when stale metadata

commit 53f2a6e628c715efe14afebf3725d1a38f9632b9
Author: Sumant Tambe 
Date:   2016-08-26T01:24:06Z

KAFKA-4089: fixing batch expired when stale metadata

commit 8200d8b4822440fefa1a58a962bc746171b260eb
Author: Sumant Tambe 
Date:   2016-08-26T01:30:41Z

Merge branch 'batch-expired' of https://github.com/sutambe/kafka into 
batch-expired




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to normal : kafka-trunk-jdk8 #843

[Apache Jenkins Server]

See 

[jira] [Updated] (KAFKA-4089) KafkaProducer raises Batch Expired exception

[Sumant Tambe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sumant Tambe updated KAFKA-4089:

Description: 
The basic idea of batch expiration is that we don't expire batches when 
producer thinks "it can make progress". Currently the notion of "making 
progress" involves only in-flight requests (muted partitions). That's not 
sufficient. The other half of the "making progress" is that if we have stale 
metadata, we cannot trust it and therefore can't say we can't make progress. 
Therefore, we don't expire batched when metadata is stale. This also implies we 
don't want to expire batches when we can still make progress even if the batch 
remains in the queue longer than the batch expiration time. 

The current condition in {{abortExpiredBatches}} that bypasses muted partitions 
is necessary but not sufficient. It should additionally restrict ejection when 
metadata is stale. 

Conversely, it should expire batches only when the following is true
# !muted AND
# meta-data is fresh AND
# batch remained in the queue longer than request timeout.

  was:
The basic idea of batch expiration is that we don't expire batches when 
producer thinks "it can make progress". Currently the notion of "making 
progress" involves only in-flight requests (muted partitions). That's not 
sufficient. The other half of the "making progress" is that if we have stale 
metadata, we cannot trust it and therefore can't say we can't make progress. 
Therefore, we don't expire batched when metadata is stale. This also implies we 
don't want to expire batches when we can still make progress even if the batch 
remains in the queue longer than the batch expiration time. 

The current condition in {{abortExpiredBatches}} that bypasses muted partitions 
is necessary but not sufficient. It should additionally restrict ejection when 
metadata is stale. 

Conversely, it should expire batches only when the following is true
# meta-data is fresh AND
# batch remained in the queue longer than request timeout.


> KafkaProducer raises Batch Expired exception 
> -
>
> Key: KAFKA-4089
> URL: https://issues.apache.org/jira/browse/KAFKA-4089
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.10.0.1
>Reporter: Sumant Tambe
>Assignee: Dong Lin
>
> The basic idea of batch expiration is that we don't expire batches when 
> producer thinks "it can make progress". Currently the notion of "making 
> progress" involves only in-flight requests (muted partitions). That's not 
> sufficient. The other half of the "making progress" is that if we have stale 
> metadata, we cannot trust it and therefore can't say we can't make progress. 
> Therefore, we don't expire batched when metadata is stale. This also implies 
> we don't want to expire batches when we can still make progress even if the 
> batch remains in the queue longer than the batch expiration time. 
> The current condition in {{abortExpiredBatches}} that bypasses muted 
> partitions is necessary but not sufficient. It should additionally restrict 
> ejection when metadata is stale. 
> Conversely, it should expire batches only when the following is true
> # !muted AND
> # meta-data is fresh AND
> # batch remained in the queue longer than request timeout.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4089) KafkaProducer raises Batch Expired exception

[Sumant Tambe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sumant Tambe updated KAFKA-4089:

Description: 
The basic idea of batch expiration is that we don't expire batches when 
producer thinks "it can make progress". Currently the notion of "making 
progress" involves only in-flight requests (muted partitions). That's not 
sufficient. The other half of the "making progress" is that if we have stale 
metadata, we cannot trust it and therefore can't say we can't make progress. 
Therefore, we don't expire batched when metadata is stale. This also implies we 
don't want to expire batches when we can still make progress even if the batch 
remains in the queue longer than the batch expiration time. 

The current condition in {{abortExpiredBatches}} that bypasses muted partitions 
is necessary but not sufficient. It should additionally restrict ejection when 
metadata is stale. 

Conversely, it should expire batches only when the following is true
# meta-data is fresh AND
# batch remained in the queue longer than request timeout.

  was:
The basic idea of batch expiration is that we don't expire batches when 
producer thinks "it can make progress". Currently the notion of "making 
progress" involves only in-flight requests (muted partitions). That's not 
sufficient. The other half of the "making progress" is that if we have stale 
metadata, we cannot trust it and therefore can't say we can't make progress. 
Therefore, we don't expire batched when metadata is stale. This also implies we 
don't want to expire batches when we can still make progress even if the batch 
remains in the queue longer than the batch expiration time. 

More concretely, the batch expiration logic 
({{RecordAccumualator.abortExpiredBatches}})  ejects batches out when the 
cluster metadata needs an update ({{Metadata.timeToNextUpdate==0}}). In this 
case, no nodes are "ready" to send data to ({{result.readyNodes}} is empty). As 
a consequence, {{Sender.drain}} does not drain any batch at all and therefore 
no new topic-partitions are muted. 

The batch expiration logic ({{RecordAccumualator.abortExpiredBatches}})  
bypasses muted partitions only. As there are no new muted partitions, 
everything that was not sent in previous drains is subject to expiration. As a 
result, a group of batches expire if they linger in the queue for longer than 
{{requestTimeout}}.

The current condition in {{abortExpiredBatches}} that bypasses muted partitions 
is necessary but not sufficient. It should additionally restrict ejection when 
metadata is stale. 

Conversely, it should expire batches only when the following is true
# !muted AND
# meta-data is fresh but leader not available 
# batch remained in the queue longer than request timeout.


> KafkaProducer raises Batch Expired exception 
> -
>
> Key: KAFKA-4089
> URL: https://issues.apache.org/jira/browse/KAFKA-4089
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.10.0.1
>Reporter: Sumant Tambe
>Assignee: Dong Lin
>
> The basic idea of batch expiration is that we don't expire batches when 
> producer thinks "it can make progress". Currently the notion of "making 
> progress" involves only in-flight requests (muted partitions). That's not 
> sufficient. The other half of the "making progress" is that if we have stale 
> metadata, we cannot trust it and therefore can't say we can't make progress. 
> Therefore, we don't expire batched when metadata is stale. This also implies 
> we don't want to expire batches when we can still make progress even if the 
> batch remains in the queue longer than the batch expiration time. 
> The current condition in {{abortExpiredBatches}} that bypasses muted 
> partitions is necessary but not sufficient. It should additionally restrict 
> ejection when metadata is stale. 
> Conversely, it should expire batches only when the following is true
> # meta-data is fresh AND
> # batch remained in the queue longer than request timeout.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4089) KafkaProducer raises Batch Expired exception

[Sumant Tambe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sumant Tambe updated KAFKA-4089:

Description: 
The basic idea of batch expiration is that we don't expire batches when 
producer thinks "it can make progress". Currently the notion of "making 
progress" involves only in-flight requests (muted partitions). That's not 
sufficient. The other half of the "making progress" is that if we have stale 
metadata, we cannot trust it and therefore can't say we can't make progress. 
Therefore, we don't expire batched when metadata is stale. This also implies we 
don't want to expire batches when we can still make progress even if the batch 
remains in the queue longer than the batch expiration time. 

More concretely, the batch expiration logic 
({{RecordAccumualator.abortExpiredBatches}})  ejects batches out when the 
cluster metadata needs an update ({{Metadata.timeToNextUpdate==0}}). In this 
case, no nodes are "ready" to send data to ({{result.readyNodes}} is empty). As 
a consequence, {{Sender.drain}} does not drain any batch at all and therefore 
no new topic-partitions are muted. 

The batch expiration logic ({{RecordAccumualator.abortExpiredBatches}})  
bypasses muted partitions only. As there are no new muted partitions, 
everything that was not sent in previous drains is subject to expiration. As a 
result, a group of batches expire if they linger in the queue for longer than 
{{requestTimeout}}.

The current condition in {{abortExpiredBatches}} that bypasses muted partitions 
is necessary but not sufficient. It should additionally restrict ejection when 
metadata is stale. 

Conversely, it should expire batches only when the following is true
# !muted AND
# meta-data is fresh but leader not available 
# batch remained in the queue longer than request timeout.

  was:
The basic idea of batch expiration is that we don't expire batches when 
producer thinks "it can make progress". Currently the notion of "making 
progress" involves only in-flight requests (muted partitions). That's not 
sufficient. The other half of the "making progress" is that if we have stale 
metadata, we cannot trust it and therefore can't say we can't make progress. 
Therefore, we don't expire batched when metadata is stale. This also implies we 
don't want to expire batches when we can still make progress even if the batch 
remains in the queue longer than the batch expiration time. 

More concretely, the batch expiration logic 
({{RecordAccumualator.abortExpiredBatches}})  ejects batches out when the 
cluster metadata needs an update ({{Metadata.timeToNextUpdate==0}}). In this 
case, no nodes are "ready" to send data to ({{result.readyNodes}} is empty). As 
a consequence, {{Sender.drain}} does not drain any batch at all and therefore 
no new topic-partitions are muted. 

The batch expiration logic ({{RecordAccumualator.abortExpiredBatches}})  
bypasses muted partitions only. As there are no new muted partitions, 
everything that was not sent in the last drain is subject to expiration. As a 
result, a group of batches expire if they linger in the queue for longer than 
{{requestTimeout}}.

The current condition in {{abortExpiredBatches}} that bypasses muted partitions 
is necessary but not sufficient. It should additionally restrict ejection when 
metadata is stale. 

Conversely, it should expire batches only when the following is true
# !muted AND
# meta-data is fresh but leader not available 
# batch remained in the queue longer than request timeout.


> KafkaProducer raises Batch Expired exception 
> -
>
> Key: KAFKA-4089
> URL: https://issues.apache.org/jira/browse/KAFKA-4089
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.10.0.1
>Reporter: Sumant Tambe
>Assignee: Dong Lin
>
> The basic idea of batch expiration is that we don't expire batches when 
> producer thinks "it can make progress". Currently the notion of "making 
> progress" involves only in-flight requests (muted partitions). That's not 
> sufficient. The other half of the "making progress" is that if we have stale 
> metadata, we cannot trust it and therefore can't say we can't make progress. 
> Therefore, we don't expire batched when metadata is stale. This also implies 
> we don't want to expire batches when we can still make progress even if the 
> batch remains in the queue longer than the batch expiration time. 
> More concretely, the batch expiration logic 
> ({{RecordAccumualator.abortExpiredBatches}})  ejects batches out when the 
> cluster metadata needs an update ({{Metadata.timeToNextUpdate==0}}). In this 
> case, no nodes are "ready" to send data to ({{result.readyNodes}} is empty). 
> As a consequence, {{Sender.drain}} does not drain any batch at all and 
> therefore no new topic-partiti

[jira] [Updated] (KAFKA-4089) KafkaProducer raises Batch Expired exception

[Sumant Tambe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sumant Tambe updated KAFKA-4089:

Description: 
The basic idea of batch expiration is that we don't expire batches when 
producer thinks "it can make progress". Currently the notion of "making 
progress" involves only in-flight requests (muted partitions). That's not 
sufficient. The other half of the "making progress" is that if we have stale 
metadata, we cannot trust it and therefore can't say we can't make progress. 
Therefore, we don't expire batched when metadata is stale. This also implies we 
don't want to expire batches when we can still make progress even if the batch 
remains in the queue longer than the batch expiration time. 

More concretely, the batch expiration logic 
({{RecordAccumualator.abortExpiredBatches}})  ejects batches out when the 
cluster metadata needs an update ({{Metadata.timeToNextUpdate==0}}). In this 
case, no nodes are "ready" to send data to ({{result.readyNodes}} is empty). As 
a consequence, {{Sender.drain}} does not drain any batch at all and therefore 
no new topic-partitions are muted. 

The batch expiration logic ({{RecordAccumualator.abortExpiredBatches}})  
bypasses muted partitions only. As there are no new muted partitions, 
everything that was not sent in the last drain is subject to expiration. As a 
result, a group of batches expire if they linger in the queue for longer than 
{{requestTimeout}}.

The current condition in {{abortExpiredBatches}} that bypasses muted partitions 
is necessary but not sufficient. It should additionally restrict ejection when 
metadata is stale. 

Conversely, it should expire batches only when the following is true
# !muted AND
# meta-data is fresh but leader not available 
# batch remained in the queue longer than request timeout.

  was:
The batch expiration logic ({{RecordAccumualator.abortExpiredBatches}})  ejects 
batches out when the cluster metadata needs an update 
({{Metadata.timeToNextUpdate==0}}). In this case, no nodes are "ready" to send 
data to ({{result.readyNodes}} is empty). As a consequence, {{Sender.drain}} 
does not drain any batch at all and therefore no new topic-partitions are 
muted. 

The batch expiration logic ({{RecordAccumualator.abortExpiredBatches}})  
bypasses muted partitions only. As there are no new muted partitions, all 
batches, regardless of topic-partition, are subject to expiration. As a result, 
a group of batches expire if they linger in the queue for longer than 
{{requestTimeout}}.

Expiring batches unconditionally is a bug. It's too greedy. 

The current condition in {{abortExpiredBatches}} that bypasses muted partitions 
is necessary but not sufficient. It should additionally bypass partitions for 
which leader information is known and fresh. 

Conversely, it should expire batches only when the following is true
# !muted AND
# meta-data is fresh but leader not available 


> KafkaProducer raises Batch Expired exception 
> -
>
> Key: KAFKA-4089
> URL: https://issues.apache.org/jira/browse/KAFKA-4089
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.10.0.1
>Reporter: Sumant Tambe
>Assignee: Dong Lin
>
> The basic idea of batch expiration is that we don't expire batches when 
> producer thinks "it can make progress". Currently the notion of "making 
> progress" involves only in-flight requests (muted partitions). That's not 
> sufficient. The other half of the "making progress" is that if we have stale 
> metadata, we cannot trust it and therefore can't say we can't make progress. 
> Therefore, we don't expire batched when metadata is stale. This also implies 
> we don't want to expire batches when we can still make progress even if the 
> batch remains in the queue longer than the batch expiration time. 
> More concretely, the batch expiration logic 
> ({{RecordAccumualator.abortExpiredBatches}})  ejects batches out when the 
> cluster metadata needs an update ({{Metadata.timeToNextUpdate==0}}). In this 
> case, no nodes are "ready" to send data to ({{result.readyNodes}} is empty). 
> As a consequence, {{Sender.drain}} does not drain any batch at all and 
> therefore no new topic-partitions are muted. 
> The batch expiration logic ({{RecordAccumualator.abortExpiredBatches}})  
> bypasses muted partitions only. As there are no new muted partitions, 
> everything that was not sent in the last drain is subject to expiration. As a 
> result, a group of batches expire if they linger in the queue for longer than 
> {{requestTimeout}}.
> The current condition in {{abortExpiredBatches}} that bypasses muted 
> partitions is necessary but not sufficient. It should additionally restrict 
> ejection when metadata is stale. 
> Conversely, it should expire batches only when the fol

[jira] [Commented] (KAFKA-4070) Implement a useful Struct.toString()

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15438139#comment-15438139
 ] 

ASF GitHub Bot commented on KAFKA-4070:
---

GitHub user shikhar opened a pull request:

https://github.com/apache/kafka/pull/1790

KAFKA-4070: implement Connect Struct.toString()



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/shikhar/kafka add-struct-tostring

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1790.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1790


commit 00a47cca8f18f9de8f69718fc41c02a2162e07c6
Author: Shikhar Bhushan 
Date:   2016-08-25T23:38:27Z

KAFKA-4070: implement Connect Struct.toString()




> Implement a useful Struct.toString()
> 
>
> Key: KAFKA-4070
> URL: https://issues.apache.org/jira/browse/KAFKA-4070
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Reporter: Shikhar Bhushan
>Priority: Minor
>
> Logging of {{Struct}}'s does not currently provide any useful output, and 
> users also find it unhelpful e.g. when hooking up a Kafka topic with Avro 
> data with the {{FileSinkConnector}} which simply {{toString()}}'s the values 
> to the file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1790: KAFKA-4070: implement Connect Struct.toString()

[shikhar]

GitHub user shikhar opened a pull request:

https://github.com/apache/kafka/pull/1790

KAFKA-4070: implement Connect Struct.toString()



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/shikhar/kafka add-struct-tostring

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1790.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1790


commit 00a47cca8f18f9de8f69718fc41c02a2162e07c6
Author: Shikhar Bhushan 
Date:   2016-08-25T23:38:27Z

KAFKA-4070: implement Connect Struct.toString()




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #1786: HOTFIX: Fix verbose logging in ControllerChannelMa...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1786


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-4089) KafkaProducer raises Batch Expired exception

[Sumant Tambe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sumant Tambe updated KAFKA-4089:

Description: 
The batch expiration logic ({{RecordAccumualator.abortExpiredBatches}})  ejects 
batches out when the cluster metadata needs an update 
({{Metadata.timeToNextUpdate==0}}). In this case, no nodes are "ready" to send 
data to ({{result.readyNodes}} is empty). As a consequence, {{Sender.drain}} 
does not drain any batch at all and therefore no new topic-partitions are 
muted. 

The batch expiration logic ({{RecordAccumualator.abortExpiredBatches}})  
bypasses muted partitions only. As there are no new muted partitions, all 
batches, regardless of topic-partition, are subject to expiration. As a result, 
a group of batches expire if they linger in the queue for longer than 
{{requestTimeout}}.

Expiring batches unconditionally is a bug. It's too greedy. 

The current condition in {{abortExpiredBatches}} that bypasses muted partitions 
is necessary but not sufficient. It should additionally bypass partitions for 
which leader information is known and fresh. 

Conversely, it should expire batches only when the following is true
# !muted AND
# meta-data is fresh but leader not available 

  was:
The batch expiration logic ({{RecordAccumualator.abortExpiredBatches}})  ejects 
batches out the cluster metadata needed an update 
({{Metadata.timeToNextUpdate==0}}). In this case, no nodes are "ready" to send 
data to ({{result.readyNodes}} is empty). As a consequence, {{Sender.drain}} 
does not drain any batch at all and therefore no new topic-partitions are 
muted. 

The batch expiration logic ({{RecordAccumualator.abortExpiredBatches}})  
bypasses muted partitions only. As there are no new muted partitions, all 
batches, regardless of topic-partition, are subject to expiration. As a result, 
a group of batches expire if they linger in the queue for longer than 
{{requestTimeout}}.

Expiring batches unconditionally is a bug. It's too greedy. 

The current condition in {{abortExpiredBatches}} that bypasses muted partitions 
is necessary but not sufficient. It should additionally bypass partitions for 
which leader information is known and fresh. 

Conversely, it should expire batches only when the following is true
# !muted AND
# meta-data is fresh but leader not available 


> KafkaProducer raises Batch Expired exception 
> -
>
> Key: KAFKA-4089
> URL: https://issues.apache.org/jira/browse/KAFKA-4089
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.10.0.1
>Reporter: Sumant Tambe
>Assignee: Dong Lin
>
> The batch expiration logic ({{RecordAccumualator.abortExpiredBatches}})  
> ejects batches out when the cluster metadata needs an update 
> ({{Metadata.timeToNextUpdate==0}}). In this case, no nodes are "ready" to 
> send data to ({{result.readyNodes}} is empty). As a consequence, 
> {{Sender.drain}} does not drain any batch at all and therefore no new 
> topic-partitions are muted. 
> The batch expiration logic ({{RecordAccumualator.abortExpiredBatches}})  
> bypasses muted partitions only. As there are no new muted partitions, all 
> batches, regardless of topic-partition, are subject to expiration. As a 
> result, a group of batches expire if they linger in the queue for longer than 
> {{requestTimeout}}.
> Expiring batches unconditionally is a bug. It's too greedy. 
> The current condition in {{abortExpiredBatches}} that bypasses muted 
> partitions is necessary but not sufficient. It should additionally bypass 
> partitions for which leader information is known and fresh. 
> Conversely, it should expire batches only when the following is true
> # !muted AND
> # meta-data is fresh but leader not available 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (KAFKA-4089) KafkaProducer raises Batch Expired exception

[Dong Lin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dong Lin reassigned KAFKA-4089:
---

Assignee: Dong Lin

> KafkaProducer raises Batch Expired exception 
> -
>
> Key: KAFKA-4089
> URL: https://issues.apache.org/jira/browse/KAFKA-4089
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.10.0.1
>Reporter: Sumant Tambe
>Assignee: Dong Lin
>
> The batch expiration logic ({{RecordAccumualator.abortExpiredBatches}})  
> ejects batches out the cluster metadata needed an update 
> ({{Metadata.timeToNextUpdate==0}}). In this case, no nodes are "ready" to 
> send data to ({{result.readyNodes}} is empty). As a consequence, 
> {{Sender.drain}} does not drain any batch at all and therefore no new 
> topic-partitions are muted. 
> The batch expiration logic ({{RecordAccumualator.abortExpiredBatches}})  
> bypasses muted partitions only. As there are no new muted partitions, all 
> batches, regardless of topic-partition, are subject to expiration. As a 
> result, a group of batches expire if they linger in the queue for longer than 
> {{requestTimeout}}.
> Expiring batches unconditionally is a bug. It's too greedy. 
> The current condition in {{abortExpiredBatches}} that bypasses muted 
> partitions is necessary but not sufficient. It should additionally bypass 
> partitions for which leader information is known and fresh. 
> Conversely, it should expire batches only when the following is true
> # !muted AND
> # meta-data is fresh but leader not available 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-4089) KafkaProducer raises Batch Expired exception

[Sumant Tambe (JIRA)]

Sumant Tambe created KAFKA-4089:
---

 Summary: KafkaProducer raises Batch Expired exception 
 Key: KAFKA-4089
 URL: https://issues.apache.org/jira/browse/KAFKA-4089
 Project: Kafka
  Issue Type: Bug
  Components: clients
Affects Versions: 0.10.0.1
Reporter: Sumant Tambe


The batch expiration logic ({{RecordAccumualator.abortExpiredBatches}})  ejects 
batches out the cluster metadata needed an update 
({{Metadata.timeToNextUpdate==0}}). In this case, no nodes are "ready" to send 
data to ({{result.readyNodes}} is empty). As a consequence, {{Sender.drain}} 
does not drain any batch at all and therefore no new topic-partitions are 
muted. 

The batch expiration logic ({{RecordAccumualator.abortExpiredBatches}})  
bypasses muted partitions only. As there are no new muted partitions, all 
batches, regardless of topic-partition, are subject to expiration. As a result, 
a group of batches expire if they linger in the queue for longer than 
{{requestTimeout}}.

Expiring batches unconditionally is a bug. It's too greedy. 

The current condition in {{abortExpiredBatches}} that bypasses muted partitions 
is necessary but not sufficient. It should additionally bypass partitions for 
which leader information is known and fresh. 

Conversely, it should expire batches only when the following is true
# !muted AND
# meta-data is fresh but leader not available 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1789: MINOR: Improve log message in `ReplicaManager.beco...

[ijuma]

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/1789

MINOR: Improve log message in `ReplicaManager.becomeLeaderOrFollower`



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka 
improve-log-message-in-replica-manager

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1789.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1789


commit b43759ac85d59561684bdf0b3a56d6bf822d67c8
Author: Ismael Juma 
Date:   2016-08-25T23:00:15Z

Improve log message in `ReplicaManager.becomeLeaderOrFollower`




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #1785: HOTFIX: disabled application-reset-tool integratio...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1785


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-4088) Add regression check for KAFKA-4073

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15437963#comment-15437963
 ] 

Jun Rao commented on KAFKA-4088:


It may be possible to cover this in a unit test. 
ByteBufferMessageSetTest.getMessages() allows us to generate message sets in 
0.9 format. So, we can start a broker, use the log.append() api to insert some 
0.9 message sets. Then, run MirrorMaker to mirror the data to another cluster.

> Add regression check for KAFKA-4073
> ---
>
> Key: KAFKA-4088
> URL: https://issues.apache.org/jira/browse/KAFKA-4088
> Project: Kafka
>  Issue Type: Test
>Reporter: Geoff Anderson
>
> The patch for KAFKA-4073 fixed an issue introduced in 0.10.0.1, it may be 
> worth adding a regression test
> Ideally this would be a unit test, but it's not immediately clear how to do 
> so since it's hard to produce a pre-0.10.0 message through the producer api 
> in 0.10.0
> If we were to write a system test, it would look something like:
> Setup: 
> - Two small kafka clusters, 0.9.0.X (source), and 0.10.0.X (destination)
> - 0.9.0.X mirror maker from source to destination
> Produce messages with 0.9.X producer to source brokers.
> upgrade source brokers to 0.10.0.1
> upgrade mirror maker to 0.10.0.1 (don't upgrade 0.9 producer)
> This should reveal the problem if run without the fix introduced in KAFKA-4073



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4088) Add regression check for KAFKA-4073

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15437930#comment-15437930
 ] 

Ismael Juma commented on KAFKA-4088:


Thanks for filing this. I think what we should do is basically extend all of 
our upgrade tests to include MirrorMaker. Would that make sense?

> Add regression check for KAFKA-4073
> ---
>
> Key: KAFKA-4088
> URL: https://issues.apache.org/jira/browse/KAFKA-4088
> Project: Kafka
>  Issue Type: Test
>Reporter: Geoff Anderson
>
> The patch for KAFKA-4073 fixed an issue introduced in 0.10.0.1, it may be 
> worth adding a regression test
> Ideally this would be a unit test, but it's not immediately clear how to do 
> so since it's hard to produce a pre-0.10.0 message through the producer api 
> in 0.10.0
> If we were to write a system test, it would look something like:
> Setup: 
> - Two small kafka clusters, 0.9.0.X (source), and 0.10.0.X (destination)
> - 0.9.0.X mirror maker from source to destination
> Produce messages with 0.9.X producer to source brokers.
> upgrade source brokers to 0.10.0.1
> upgrade mirror maker to 0.10.0.1 (don't upgrade 0.9 producer)
> This should reveal the problem if run without the fix introduced in KAFKA-4073



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-4088) Add regression check for KAFKA-4073

[Geoff Anderson (JIRA)]

Geoff Anderson created KAFKA-4088:
-

 Summary: Add regression check for KAFKA-4073
 Key: KAFKA-4088
 URL: https://issues.apache.org/jira/browse/KAFKA-4088
 Project: Kafka
  Issue Type: Test
Reporter: Geoff Anderson


The patch for KAFKA-4073 fixed an issue introduced in 0.10.0.1, it may be worth 
adding a regression test

Ideally this would be a unit test, but it's not immediately clear how to do so 
since it's hard to produce a pre-0.10.0 message through the producer api in 
0.10.0

If we were to write a system test, it would look something like:

Setup: 
- Two small kafka clusters, 0.9.0.X (source), and 0.10.0.X (destination)
- 0.9.0.X mirror maker from source to destination

Produce messages with 0.9.X producer to source brokers.
upgrade source brokers to 0.10.0.1
upgrade mirror maker to 0.10.0.1 (don't upgrade 0.9 producer)

This should reveal the problem if run without the fix introduced in KAFKA-4073



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3008) Connect should parallelize task start/stop

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15437761#comment-15437761
 ] 

ASF GitHub Bot commented on KAFKA-3008:
---

GitHub user kkonstantine opened a pull request:

https://github.com/apache/kafka/pull/1788

KAFKA-3008: Parallel start and stop of connectors and tasks in Connect 



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kkonstantine/kafka 
KAFKA-3008-Parallel-start-and-stop-of-connectors-and-tasks

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1788.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1788


commit d1f876d4be7394688f6070169cded7748f8492cf
Author: Konstantine Karantasis 
Date:   2016-08-25T05:38:45Z

KAFKA-3008: Parallel start and stop of connectors and tasks in Connect

Refactoring Worker to make start and stop methods thread safe.

commit 938b8ad2cba770dad71d3749b26ae59a3c4c3425
Author: Konstantine Karantasis 
Date:   2016-08-25T21:31:57Z

Parallelizing start and stop of connectors and tasks in Distributed Herder




> Connect should parallelize task start/stop
> --
>
> Key: KAFKA-3008
> URL: https://issues.apache.org/jira/browse/KAFKA-3008
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Affects Versions: 0.9.0.0
>Reporter: Ewen Cheslack-Postava
>Assignee: Liquan Pei
>Priority: Minor
>
> The Herder implementations currently iterate over all connectors/tasks and 
> sequentially start/stop them. We should parallelize this. This is less 
> critical for {{StandaloneHerder}}, but pretty important for 
> {{DistributedHerder}} since it will generally be managing more tasks and any 
> delay starting/stopping a single task will impact every other task on the 
> node (and can ultimately result in incorrect behavior in the case of a single 
> offset commit in one connector taking too long preventing all of the rest 
> from committing offsets).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1788: KAFKA-3008: Parallel start and stop of connectors ...

[kkonstantine]

GitHub user kkonstantine opened a pull request:

https://github.com/apache/kafka/pull/1788

KAFKA-3008: Parallel start and stop of connectors and tasks in Connect 



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kkonstantine/kafka 
KAFKA-3008-Parallel-start-and-stop-of-connectors-and-tasks

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1788.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1788


commit d1f876d4be7394688f6070169cded7748f8492cf
Author: Konstantine Karantasis 
Date:   2016-08-25T05:38:45Z

KAFKA-3008: Parallel start and stop of connectors and tasks in Connect

Refactoring Worker to make start and stop methods thread safe.

commit 938b8ad2cba770dad71d3749b26ae59a3c4c3425
Author: Konstantine Karantasis 
Date:   2016-08-25T21:31:57Z

Parallelizing start and stop of connectors and tasks in Distributed Herder




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] KIP-76: Enable getting password from executable rather than passing as plaintext in config files

[Matthias J. Sax]

I guess this should be KIP-77 ?

KIP-76 is "Improve Kafka Streams Join Semantics"

See http://search-hadoop.com/m/uyzND19SmQJ1yfCQ42/v=plain

-Matthias

On 08/25/2016 10:13 PM, Ashish Singh wrote:
> Hey Gwen,
> 
> You’re right that if someone can alter the executable then they can do
> things in the context of the thing running the script, like kafka. But if
> you were kafka admin user (or root), you could also do lots of things to
> lots of other different files owned by the user, so it’s not really that
> much different than the current state of things.
> 
> You’re right to wonder about the real security gains here. In some sense,
> they aren’t many, because if you know where to look and what to do, you can
> coax the password out of that executable. What this approach really does is
> make it *nontrivial* for an attacker to get the password. And people tend
> to flip out when they see passwords sitting in the clear on a disk, because
> we’ve all been rightly trained that cleartext passwords are bad.
> 
> This approach when combined with some strong security practices, like the
> ones mentioned below makes the system reasonably secure. This approach is
> probably the simplest way for folks to strengthen their Kafka security.
> There are other more complicated ways, like Hadoop’s credential store,
> which depends on external systems. If the community feels that this does
> not help, we can definitely move towards more complicated mechanisms.
> However, this has sufficed for our needs so far and others have expressed
> their satisfaction on the JIRA.
> 
>- Executable decrypts a file that stores encrypted passwords.
>- The secret to decrypt that file is passed in via environment, which is
>generally a bit harder to find than files on disk.
>- The perms also protect the executable.
>- The file sits on an ephemeral disk that’s mounted to memory, so
>stealing a physical disk won’t result in getting even the encrypted
>password.
> 
> On Thu, Aug 25, 2016 at 9:07 AM, Gwen Shapira  wrote:
> 
> Hi Ashish,
>>
>> I appreciate the need to integrate our authentication with other
>> systems that store passwords.
>> I am not sure that doing so by running a binary is the best solution.
>>
>> First, it does not add security: As you said, a file is just "sitting
>> there" the same way an executable is just "sitting there" - we still
>> rely on file system privileges for security.
>> Second, the idea that Kafka will run arbitrary filesystem executables
>> is pretty terrifying. Reading a string from a file is harmless, but an
>> incorrectly privileged executable can be replaced with "rm -rf /" or
>> anything really. Kafka sometimes runs from privileged account, so this
>> is a serious risk.
>>
>> I looked at the Hadoop credential store you helpfully linked to in the
>> KIP, and it seems like the Hadoop proposal includes a well thought out
>> API to integrate with external systems. Since we took this approach in
>> the past, I'm wondering why not follow the same and use an API to
>> integrate with credential stores rather than arbitrary executables.
>>
>> Gwen
>>
>> On Wed, Aug 24, 2016 at 12:03 PM, Ashish Singh 
>> wrote:
>>> Hey Guys,
>>>
>>> I’ve just posted KIP-76: Enable getting password from executable rather
>>> than passing as plaintext in config files
>>> > able+getting+password+from+executable+rather+than+passing+
>> as+plaintext+in+config+files>
>>> .
>>>
>>> The proposal is to enable getting passwords from executable. This is an
>> ask
>>> from very security conscious users.
>>>
>>> Full details are here:
>>>
>>> KIP:
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-76+Ena
>> ble+getting+password+from+executable+rather+than+passing+as+
>> plaintext+in+config+files
>>> JIRA: https://issues.apache.org/jira/browse/KAFKA-2629
>>> POC: https://github.com/apache/kafka/pull/1770
>>>
>>> Thanks
>>>
>>> --
>>>
>>> Regards,
>>> Ashish
>>
>>
>>
>> --
>> Gwen Shapira
>> Product Manager | Confluent
>> 650.450.2760 | @gwenshap
>> Follow us: Twitter | blog
>>
> ​
> 



signature.asc
Description: OpenPGP digital signature

[jira] [Work started] (KAFKA-3940) Log should check the return value of dir.mkdirs()

[Ishita Mandhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on KAFKA-3940 started by Ishita Mandhan.
-
> Log should check the return value of dir.mkdirs()
> -
>
> Key: KAFKA-3940
> URL: https://issues.apache.org/jira/browse/KAFKA-3940
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.10.0.0
>Reporter: Jun Rao
>Assignee: Ishita Mandhan
>  Labels: newbie
>
> In Log.loadSegments(), we call dir.mkdirs() w/o checking the return value and 
> just assume the directory will exist after the call. However, if the 
> directory can't be created (e.g. due to no space), we will hit 
> NullPointerException in the next statement, which will be confusing.
>for(file <- dir.listFiles if file.isFile) {



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1787: KAFKA-3940 Log should check the return value of di...

[imandhan]

GitHub user imandhan opened a pull request:

https://github.com/apache/kafka/pull/1787

KAFKA-3940 Log should check the return value of dir.mkdirs()



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/imandhan/kafka KAFKA-3940

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1787.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1787


commit e3f5f293104013da3aa765c9ba7cbdb2553f6485
Author: Ishita Mandhan 
Date:   2016-08-25T21:38:16Z

KAFKA-3940 Log should check the return value of dir.mkdirs()




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3940) Log should check the return value of dir.mkdirs()

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15437729#comment-15437729
 ] 

ASF GitHub Bot commented on KAFKA-3940:
---

GitHub user imandhan opened a pull request:

https://github.com/apache/kafka/pull/1787

KAFKA-3940 Log should check the return value of dir.mkdirs()



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/imandhan/kafka KAFKA-3940

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1787.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1787


commit e3f5f293104013da3aa765c9ba7cbdb2553f6485
Author: Ishita Mandhan 
Date:   2016-08-25T21:38:16Z

KAFKA-3940 Log should check the return value of dir.mkdirs()




> Log should check the return value of dir.mkdirs()
> -
>
> Key: KAFKA-3940
> URL: https://issues.apache.org/jira/browse/KAFKA-3940
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.10.0.0
>Reporter: Jun Rao
>Assignee: Ishita Mandhan
>  Labels: newbie
>
> In Log.loadSegments(), we call dir.mkdirs() w/o checking the return value and 
> just assume the directory will exist after the call. However, if the 
> directory can't be created (e.g. due to no space), we will hit 
> NullPointerException in the next statement, which will be confusing.
>for(file <- dir.listFiles if file.isFile) {



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] Remove beta label from the new Java consumer

[Jason Gustafson]

I'm +1 also. I feel a lot more confident about this with all of the system
testing we now have in place (including the tests covering Streams and
Connect).

-Jason

On Thu, Aug 25, 2016 at 9:57 AM, Gwen Shapira  wrote:

> Makes sense :)
>
> On Thu, Aug 25, 2016 at 9:40 AM, Neha Narkhede  wrote:
> > Yeah, I'm supportive of this.
> >
> > On Thu, Aug 25, 2016 at 9:26 AM Ismael Juma  wrote:
> >
> >> Hi Gwen,
> >>
> >> We have a few recent stories of people using Connect and Streams in
> >> production. That means the new Java Consumer too. :)
> >>
> >> Ismael
> >>
> >> On Thu, Aug 25, 2016 at 5:09 PM, Gwen Shapira 
> wrote:
> >>
> >> > Originally, we suggested keeping the beta label until we know someone
> >> > successfully uses the new consumer in production.
> >> >
> >> > We can consider the recent KIPs enough, but IMO it will be better if
> >> > someone with production deployment hanging out on our mailing list
> >> > will confirm good experience with the new consumer.
> >> >
> >> > Gwen
> >> >
> >> > On Wed, Aug 24, 2016 at 8:45 PM, Ismael Juma 
> wrote:
> >> > > Hi all,
> >> > >
> >> > > We currently say the following in our documentation:
> >> > >
> >> > > "As of the 0.9.0 release we have added a new Java consumer to
> replace
> >> our
> >> > > existing high-level ZooKeeper-based consumer and low-level consumer
> >> APIs.
> >> > > This client is considered beta quality."[1]
> >> > >
> >> > > Since then, Jason and the community have done a lot of work to
> improve
> >> it
> >> > > (including KIP-41 and KIP-62), we declared it API stable in 0.10.0.0
> >> and
> >> > > it's the only option for those that need security support. Yes, it
> >> still
> >> > > has bugs, but so does the old consumer and all development is
> currently
> >> > > focused on the new consumer.
> >> > >
> >> > > As such, I propose we remove the beta label for the next release and
> >> > switch
> >> > > our tools to use the new consumer by default unless the zookeeper
> >> > > command-line option is present (for compatibility). This is similar
> to
> >> > what
> >> > > we did it for the new producer in 0.9.0.0, but backwards compatible.
> >> > >
> >> > > Thoughts?
> >> > >
> >> > > Ismael
> >> > >
> >> > > [1] http://kafka.apache.org/documentation.html#consumerapi
> >> >
> >> >
> >> >
> >> > --
> >> > Gwen Shapira
> >> > Product Manager | Confluent
> >> > 650.450.2760 | @gwenshap
> >> > Follow us: Twitter | blog
> >> >
> >>
> > --
> > Thanks,
> > Neha
>
>
>
> --
> Gwen Shapira
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter | blog
>

Re: [DISCUSS] Time-based releases for Apache Kafka

[Ofir Manor]

I happily agree that Kafka is a solid and the community is great :)
But I think there is a gap in perception here.
For me, LTS means that someone is actively taking care of a release -
actively backporting critical fixes (security, stability, data loss,
corruption, hangs etc) from trunk to that LTS version periodically for an
extended period of time, for example 18-36 months... So people can really
rely on the same Kafka version for a long time.
Is someone doing it today for 0.9.0? When is 0.9.0.2 expected? When is
0.8.2.3 expected? Will they cover all known critical issues for whoever
relies on them in production?
In other words, what is the scope of support that the community want to
commit for older versions? (upgrade compatibility? investigating bug
reports? proactively backporting fixes?)
BTW, another legit option is that the Apache Kafka project won't commit to
LTS releases. It could let commercial vendors compete on supporting very
old versions. I find that actually quite reasonable as well.

Ofir Manor

Co-Founder & CTO | Equalum

Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io

On Thu, Aug 25, 2016 at 8:19 PM, Andrew Schofield <
andrew_schofield_j...@outlook.com> wrote:

> I agree that the Kafka community has managed to maintain a very high
> quality level, so I'm not concerned
> about the quality of non-LTS releases. If the principle is that every
> release is supported for 2 years, that
> would be good. I suppose that if the burden of having that many in-support
> releases proves too heavy,
> as you say we could reconsider.
>
> Andrew Schofield
>
> 
> > From: g...@confluent.io
> > Date: Thu, 25 Aug 2016 09:57:30 -0700
> > Subject: Re: [DISCUSS] Time-based releases for Apache Kafka
> > To: dev@kafka.apache.org
> >
> > I prefer Ismael's suggestion for supporting 2-years (6 releases)
> > rather than have designated LTS releases.
> >
> > The LTS model seems to work well when some releases are high quality
> > (LTS) and the rest are a bit more questionable. It is great for
> > companies like Redhat, where they have to invest less to support few
> > releases and let the community deal with everything else.
> >
> > Until now the Kafka community has managed to maintain very high
> > quality level. Not just for releases, our trunk is often of better
> > quality than other project's releases - we don't think of stability as
> > something you tuck into a release (and just some releases) but rather
> > as an on-going concern. There are costs to doing things that way, but
> > in general, I think it has served us well - allowing even conservative
> > companies to run on the latest released version.
> >
> > I hope we can agree to at least try maintaining last 6 releases as LTS
> > (i.e. every single release is supported for 2 years) rather than
> > designate some releases as better than others. Of course, if this
> > totally fails, we can reconsider.
> >
> > Gwen
> >
> > On Thu, Aug 25, 2016 at 9:51 AM, Andrew Schofield
> >  wrote:
> >> The proposal sounds pretty good, but the main thing currently missing
> is a proper long-term support release.
> >>
> >> Having 3 releases a year sounds OK, but if they're all equivalent and
> bugfix releases are produced for the most
> >> recent 2 or 3 releases, anyone wanting to run on an "in support"
> release of Kafka has to upgrade every 8-12 months.
> >> If you don't actually want anything specific from the newer releases,
> it's just unnecessary churn.
> >>
> >> Wouldn't it be better to designate one release every 12-18 months as a
> long-term support release with bugfix releases
> >> produced for those for a longer period of say 24 months. That halves
> the upgrade work for people just wanting to keep
> >> "in support". Now that adoption is increasing, there are plenty of
> users that just want a dependable messaging system
> >> without having to be deeply knowledgeable about its innards.
> >>
> >> LTS works nicely for plenty of open-source projects. I think it would
> work well for Kafka too.
> >>
> >> Andrew Schofield
> >>
> >> 
> >>> From: ofir.ma...@equalum.io
> >>> Date: Thu, 25 Aug 2016 16:07:07 +0300
> >>> Subject: Re: [DISCUSS] Time-based releases for Apache Kafka
> >>> To: dev@kafka.apache.org
> >>>
> >>> Regarding bug fixes, you may want to consider to have an LTS release
> once a
> >>> year - designating it for longer-term support / better for the masses.
> >>> If you like that - then fix bugs in trunk, backport important ones to
> >>> latest release + the last two LTS releases.
> >>> Even if you don't - if a downstream distribution picks a Kafka version
> and
> >>> plans to support it over a few years, it could be nice of them to "own"
> >>> that older release - volunteer to be a release manager for bug
> backports to
> >>> that version over a longer period...
> >>> Just my two cents :)
> >>>
> >>> Ofir Manor
> >>>
> >>> Co-Founder & CTO | Equalum
> >>>
> >>> Mobile: +972-

[GitHub] kafka pull request #1786: HOTFIX: Fix verbose logging in ControllerChannelMa...

[hachikuji]

GitHub user hachikuji opened a pull request:

https://github.com/apache/kafka/pull/1786

HOTFIX: Fix verbose logging in ControllerChannelManager.brokerReady



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hachikuji/kafka 
hotfix-ctrlchannelmgr-verbose-logging

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1786.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1786


commit f62ffaf473d00b3636ccb59764ebdee3caafd4dc
Author: Jason Gustafson 
Date:   2016-08-25T20:34:01Z

HOTFIX: Fix verbose logging in ControllerChannelManager.brokerReady




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] KIP-76: Enable getting password from executable rather than passing as plaintext in config files

[Ashish Singh]

Hey Gwen,

You’re right that if someone can alter the executable then they can do
things in the context of the thing running the script, like kafka. But if
you were kafka admin user (or root), you could also do lots of things to
lots of other different files owned by the user, so it’s not really that
much different than the current state of things.

You’re right to wonder about the real security gains here. In some sense,
they aren’t many, because if you know where to look and what to do, you can
coax the password out of that executable. What this approach really does is
make it *nontrivial* for an attacker to get the password. And people tend
to flip out when they see passwords sitting in the clear on a disk, because
we’ve all been rightly trained that cleartext passwords are bad.

This approach when combined with some strong security practices, like the
ones mentioned below makes the system reasonably secure. This approach is
probably the simplest way for folks to strengthen their Kafka security.
There are other more complicated ways, like Hadoop’s credential store,
which depends on external systems. If the community feels that this does
not help, we can definitely move towards more complicated mechanisms.
However, this has sufficed for our needs so far and others have expressed
their satisfaction on the JIRA.

   - Executable decrypts a file that stores encrypted passwords.
   - The secret to decrypt that file is passed in via environment, which is
   generally a bit harder to find than files on disk.
   - The perms also protect the executable.
   - The file sits on an ephemeral disk that’s mounted to memory, so
   stealing a physical disk won’t result in getting even the encrypted
   password.

On Thu, Aug 25, 2016 at 9:07 AM, Gwen Shapira  wrote:

Hi Ashish,
>
> I appreciate the need to integrate our authentication with other
> systems that store passwords.
> I am not sure that doing so by running a binary is the best solution.
>
> First, it does not add security: As you said, a file is just "sitting
> there" the same way an executable is just "sitting there" - we still
> rely on file system privileges for security.
> Second, the idea that Kafka will run arbitrary filesystem executables
> is pretty terrifying. Reading a string from a file is harmless, but an
> incorrectly privileged executable can be replaced with "rm -rf /" or
> anything really. Kafka sometimes runs from privileged account, so this
> is a serious risk.
>
> I looked at the Hadoop credential store you helpfully linked to in the
> KIP, and it seems like the Hadoop proposal includes a well thought out
> API to integrate with external systems. Since we took this approach in
> the past, I'm wondering why not follow the same and use an API to
> integrate with credential stores rather than arbitrary executables.
>
> Gwen
>
> On Wed, Aug 24, 2016 at 12:03 PM, Ashish Singh 
> wrote:
> > Hey Guys,
> >
> > I’ve just posted KIP-76: Enable getting password from executable rather
> > than passing as plaintext in config files
> >  able+getting+password+from+executable+rather+than+passing+
> as+plaintext+in+config+files>
> > .
> >
> > The proposal is to enable getting passwords from executable. This is an
> ask
> > from very security conscious users.
> >
> > Full details are here:
> >
> > KIP:
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-76+Ena
> ble+getting+password+from+executable+rather+than+passing+as+
> plaintext+in+config+files
> > JIRA: https://issues.apache.org/jira/browse/KAFKA-2629
> > POC: https://github.com/apache/kafka/pull/1770
> >
> > Thanks
> >
> > --
> >
> > Regards,
> > Ashish
>
>
>
> --
> Gwen Shapira
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter | blog
>
​
-- 

Regards,
Ashish

[jira] [Commented] (KAFKA-3940) Log should check the return value of dir.mkdirs()

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15437515#comment-15437515
 ] 

ASF GitHub Bot commented on KAFKA-3940:
---

Github user imandhan closed the pull request at:

https://github.com/apache/kafka/pull/1748


> Log should check the return value of dir.mkdirs()
> -
>
> Key: KAFKA-3940
> URL: https://issues.apache.org/jira/browse/KAFKA-3940
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.10.0.0
>Reporter: Jun Rao
>Assignee: Ishita Mandhan
>  Labels: newbie
>
> In Log.loadSegments(), we call dir.mkdirs() w/o checking the return value and 
> just assume the directory will exist after the call. However, if the 
> directory can't be created (e.g. due to no space), we will hit 
> NullPointerException in the next statement, which will be confusing.
>for(file <- dir.listFiles if file.isFile) {



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1748: KAFKA-3940 Log should check the return value of di...

[imandhan]

Github user imandhan closed the pull request at:

https://github.com/apache/kafka/pull/1748


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-4087) DefaultParitioner Implementation Issue

[Bharat Viswanadham (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bharat Viswanadham updated KAFKA-4087:
--
Affects Version/s: 0.10.0.1

> DefaultParitioner Implementation Issue
> --
>
> Key: KAFKA-4087
> URL: https://issues.apache.org/jira/browse/KAFKA-4087
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Affects Versions: 0.9.0.1, 0.10.0.1
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>  Labels: partitioners, producer
>
> In DefaultPartitioner implementation, when key is null
>  if (availablePartitions.size() > 0) {
> int part = Utils.toPositive(nextValue) % 
> availablePartitions.size();
> return availablePartitions.get(part).partition();
> }
> Where as when key is not null
>  return Utils.toPositive(Utils.murmur2(keyBytes)) % numPartitions;
> We are returning partition by using total number of partitions.
> Should n't we do the same as by considering only available partitions?
> https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/producer/internals/DefaultPartitioner.java#L67
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4087) DefaultParitioner Implementation Issue

[Bharat Viswanadham (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bharat Viswanadham updated KAFKA-4087:
--
Description: 
In DefaultPartitioner implementation, when key is null
 if (availablePartitions.size() > 0) {
int part = Utils.toPositive(nextValue) % 
availablePartitions.size();
return availablePartitions.get(part).partition();
}

Where as when key is not null
 return Utils.toPositive(Utils.murmur2(keyBytes)) % numPartitions;

We are returning partition by using total number of partitions.
Should n't we do the same as by considering only available partitions?

https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/producer/internals/DefaultPartitioner.java#L67
 


  was:
In DefaultPartitioner implementation, when key is null
 if (availablePartitions.size() > 0) {
int part = Utils.toPositive(nextValue) % 
availablePartitions.size();
return availablePartitions.get(part).partition();
}

Where as when key is not null
 return Utils.toPositive(Utils.murmur2(keyBytes)) % numPartitions;

We are returning partition by using total number of partitions.
Should n't we do the same as by considering only available partitions?



> DefaultParitioner Implementation Issue
> --
>
> Key: KAFKA-4087
> URL: https://issues.apache.org/jira/browse/KAFKA-4087
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Affects Versions: 0.9.0.1, 0.10.0.1
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>  Labels: partitioners, producer
>
> In DefaultPartitioner implementation, when key is null
>  if (availablePartitions.size() > 0) {
> int part = Utils.toPositive(nextValue) % 
> availablePartitions.size();
> return availablePartitions.get(part).partition();
> }
> Where as when key is not null
>  return Utils.toPositive(Utils.murmur2(keyBytes)) % numPartitions;
> We are returning partition by using total number of partitions.
> Should n't we do the same as by considering only available partitions?
> https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/producer/internals/DefaultPartitioner.java#L67
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-4087) DefaultParitioner Implementation Issue

[Bharat Viswanadham (JIRA)]

Bharat Viswanadham created KAFKA-4087:
-

 Summary: DefaultParitioner Implementation Issue
 Key: KAFKA-4087
 URL: https://issues.apache.org/jira/browse/KAFKA-4087
 Project: Kafka
  Issue Type: Bug
  Components: producer 
Affects Versions: 0.9.0.1
Reporter: Bharat Viswanadham
Assignee: Bharat Viswanadham


In DefaultPartitioner implementation, when key is null
 if (availablePartitions.size() > 0) {
int part = Utils.toPositive(nextValue) % 
availablePartitions.size();
return availablePartitions.get(part).partition();
}

Where as when key is not null
 return Utils.toPositive(Utils.murmur2(keyBytes)) % numPartitions;

We are returning partition by using total number of partitions.
Should n't we do the same as by considering only available partitions?




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-trunk-jdk7 #1499

[Apache Jenkins Server]

See 

Changes:

[junrao] KAFKA-4015; Change cleanup.policy config to accept a list of valid

--
[...truncated 1 lines...]
org.apache.kafka.streams.processor.internals.assignment.AssignmentInfoTest > 
testEncodeDecode PASSED

org.apache.kafka.streams.processor.internals.assignment.AssignmentInfoTest > 
shouldDecodePreviousVersion STARTED

org.apache.kafka.streams.processor.internals.assignment.AssignmentInfoTest > 
shouldDecodePreviousVersion PASSED

org.apache.kafka.streams.processor.internals.assignment.SubscriptionInfoTest > 
testEncodeDecode STARTED

org.apache.kafka.streams.processor.internals.assignment.SubscriptionInfoTest > 
testEncodeDecode PASSED

org.apache.kafka.streams.processor.internals.assignment.SubscriptionInfoTest > 
shouldEncodeDecodeWithUserEndPoint STARTED

org.apache.kafka.streams.processor.internals.assignment.SubscriptionInfoTest > 
shouldEncodeDecodeWithUserEndPoint PASSED

org.apache.kafka.streams.processor.internals.assignment.SubscriptionInfoTest > 
shouldBeBackwardCompatible STARTED

org.apache.kafka.streams.processor.internals.assignment.SubscriptionInfoTest > 
shouldBeBackwardCompatible PASSED

org.apache.kafka.streams.processor.internals.StreamThreadTest > 
testInjectClients STARTED

org.apache.kafka.streams.processor.internals.StreamThreadTest > 
testInjectClients PASSED

org.apache.kafka.streams.processor.internals.StreamThreadTest > testMaybeCommit 
STARTED

org.apache.kafka.streams.processor.internals.StreamThreadTest > testMaybeCommit 
PASSED

org.apache.kafka.streams.processor.internals.StreamThreadTest > 
testPartitionAssignmentChange STARTED

org.apache.kafka.streams.processor.internals.StreamThreadTest > 
testPartitionAssignmentChange PASSED

org.apache.kafka.streams.processor.internals.StreamThreadTest > testMaybeClean 
STARTED

org.apache.kafka.streams.processor.internals.StreamThreadTest > testMaybeClean 
PASSED

org.apache.kafka.streams.processor.internals.PunctuationQueueTest > 
testPunctuationInterval STARTED

org.apache.kafka.streams.processor.internals.PunctuationQueueTest > 
testPunctuationInterval PASSED

org.apache.kafka.streams.processor.internals.RecordQueueTest > testTimeTracking 
STARTED

org.apache.kafka.streams.processor.internals.RecordQueueTest > testTimeTracking 
PASSED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
testRegisterNonPersistentStore STARTED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
testRegisterNonPersistentStore PASSED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
testGetStore STARTED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
testGetStore PASSED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
testClose STARTED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
testClose PASSED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
testChangeLogOffsets STARTED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
testChangeLogOffsets PASSED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
testRegisterPersistentStore STARTED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
testRegisterPersistentStore PASSED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
testNoTopic STARTED

org.apache.kafka.streams.processor.internals.ProcessorStateManagerTest > 
testNoTopic PASSED

org.apache.kafka.streams.processor.internals.StreamTaskTest > testProcessOrder 
STARTED

org.apache.kafka.streams.processor.internals.StreamTaskTest > testProcessOrder 
PASSED

org.apache.kafka.streams.processor.internals.StreamTaskTest > testPauseResume 
STARTED

org.apache.kafka.streams.processor.internals.StreamTaskTest > testPauseResume 
PASSED

org.apache.kafka.streams.processor.internals.StreamTaskTest > 
testMaybePunctuate STARTED

org.apache.kafka.streams.processor.internals.StreamTaskTest > 
testMaybePunctuate PASSED

org.apache.kafka.streams.processor.internals.QuickUnionTest > testUnite STARTED

org.apache.kafka.streams.processor.internals.QuickUnionTest > testUnite PASSED

org.apache.kafka.streams.processor.internals.QuickUnionTest > testUniteMany 
STARTED

org.apache.kafka.streams.processor.internals.QuickUnionTest > testUniteMany 
PASSED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
shouldNotAllowNullProcessorSupplier STARTED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
shouldNotAllowNullProcessorSupplier PASSED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
shouldNotSetApplicationIdToNull STARTED

org.apache.kafka.streams.processor.TopologyBuilderTest > 
shouldNotSetApplicationIdToNull PASSED

org.apache.kafka.streams.processor.TopologyBuilderTest > testSourceTopics 
STARTED

org.apache.kafka.streams.processor.TopologyBuilderTe

Build failed in Jenkins: kafka-trunk-jdk8 #842

[Apache Jenkins Server]

See 

Changes:

[junrao] KAFKA-4015; Change cleanup.policy config to accept a list of valid

--
[...truncated 6875 lines...]

kafka.api.AuthorizerIntegrationTest > 
testCreatePermissionNeededForWritingToNonExistentTopic STARTED

kafka.api.AuthorizerIntegrationTest > 
testCreatePermissionNeededForWritingToNonExistentTopic PASSED

kafka.api.AuthorizerIntegrationTest > 
testSimpleConsumeWithOffsetLookupAndNoGroupAccess STARTED

kafka.api.AuthorizerIntegrationTest > 
testSimpleConsumeWithOffsetLookupAndNoGroupAccess PASSED

kafka.api.AuthorizerIntegrationTest > testConsumeWithTopicDescribe STARTED

kafka.api.AuthorizerIntegrationTest > testConsumeWithTopicDescribe PASSED

kafka.api.AuthorizerIntegrationTest > testOffsetFetchWithNoTopicAccess STARTED

kafka.api.AuthorizerIntegrationTest > testOffsetFetchWithNoTopicAccess PASSED

kafka.api.AuthorizerIntegrationTest > testCommitWithNoTopicAccess STARTED

kafka.api.AuthorizerIntegrationTest > testCommitWithNoTopicAccess PASSED

kafka.api.AuthorizerIntegrationTest > testProduceWithNoTopicAccess STARTED

kafka.api.AuthorizerIntegrationTest > testProduceWithNoTopicAccess PASSED

kafka.api.AuthorizerIntegrationTest > testProduceWithTopicWrite STARTED

kafka.api.AuthorizerIntegrationTest > testProduceWithTopicWrite PASSED

kafka.api.AuthorizerIntegrationTest > testAuthorization STARTED

kafka.api.AuthorizerIntegrationTest > testAuthorization PASSED

kafka.api.AuthorizerIntegrationTest > testUnauthorizedDeleteWithDescribe STARTED

kafka.api.AuthorizerIntegrationTest > testUnauthorizedDeleteWithDescribe PASSED

kafka.api.AuthorizerIntegrationTest > testConsumeWithTopicAndGroupRead STARTED

kafka.api.AuthorizerIntegrationTest > testConsumeWithTopicAndGroupRead PASSED

kafka.api.AuthorizerIntegrationTest > testUnauthorizedDeleteWithoutDescribe 
STARTED

kafka.api.AuthorizerIntegrationTest > testUnauthorizedDeleteWithoutDescribe 
PASSED

kafka.api.AuthorizerIntegrationTest > testConsumeWithTopicWrite STARTED

kafka.api.AuthorizerIntegrationTest > testConsumeWithTopicWrite PASSED

kafka.api.AuthorizerIntegrationTest > testOffsetFetchWithNoGroupAccess STARTED

kafka.api.AuthorizerIntegrationTest > testOffsetFetchWithNoGroupAccess PASSED

kafka.api.AuthorizerIntegrationTest > testCommitWithNoAccess STARTED

kafka.api.AuthorizerIntegrationTest > testCommitWithNoAccess PASSED

kafka.api.AuthorizerIntegrationTest > testCommitWithNoGroupAccess STARTED

kafka.api.AuthorizerIntegrationTest > testCommitWithNoGroupAccess PASSED

kafka.api.AuthorizerIntegrationTest > testConsumeWithNoAccess STARTED

kafka.api.AuthorizerIntegrationTest > testConsumeWithNoAccess PASSED

kafka.api.AuthorizerIntegrationTest > testOffsetFetchWithTopicAndGroupRead 
STARTED

kafka.api.AuthorizerIntegrationTest > testOffsetFetchWithTopicAndGroupRead 
PASSED

kafka.api.AuthorizerIntegrationTest > testCommitWithTopicDescribe STARTED

kafka.api.AuthorizerIntegrationTest > testCommitWithTopicDescribe PASSED

kafka.api.AuthorizerIntegrationTest > testProduceWithTopicDescribe STARTED

kafka.api.AuthorizerIntegrationTest > testProduceWithTopicDescribe PASSED

kafka.api.AuthorizerIntegrationTest > testOffsetFetchTopicDescribe STARTED

kafka.api.AuthorizerIntegrationTest > testOffsetFetchTopicDescribe PASSED

kafka.api.AuthorizerIntegrationTest > testCommitWithTopicAndGroupRead STARTED

kafka.api.AuthorizerIntegrationTest > testCommitWithTopicAndGroupRead PASSED

kafka.api.AuthorizerIntegrationTest > 
testSimpleConsumeWithExplicitSeekAndNoGroupAccess STARTED

kafka.api.AuthorizerIntegrationTest > 
testSimpleConsumeWithExplicitSeekAndNoGroupAccess PASSED

kafka.api.SaslPlainPlaintextConsumerTest > testCoordinatorFailover STARTED

kafka.api.SaslPlainPlaintextConsumerTest > testCoordinatorFailover PASSED

kafka.api.SaslPlainPlaintextConsumerTest > testSimpleConsumption STARTED

kafka.api.SaslPlainPlaintextConsumerTest > testSimpleConsumption PASSED

kafka.api.RequestResponseSerializationTest > 
testSerializationAndDeserialization STARTED

kafka.api.RequestResponseSerializationTest > 
testSerializationAndDeserialization PASSED

kafka.api.RequestResponseSerializationTest > testFetchResponseVersion STARTED

kafka.api.RequestResponseSerializationTest > testFetchResponseVersion PASSED

kafka.api.RequestResponseSerializationTest > testProduceResponseVersion STARTED

kafka.api.RequestResponseSerializationTest > testProduceResponseVersion PASSED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > testNoConsumeAcl STARTED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > testNoConsumeAcl PASSED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > testProduceConsumeViaAssign 
STARTED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > testProduceConsumeViaAssign 
PASSED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > testNoProduceAcl STARTED

kafka.api.SaslPlainSslEndToEndAuthorizationTest > testNoProduceAcl PASSED

kafka.api.SaslPla

[jira] [Commented] (KAFKA-3083) a soft failure in controller may leave a topic partition in an inconsistent state

[Mayuresh Gharat (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15437288#comment-15437288
 ] 

Mayuresh Gharat commented on KAFKA-3083:


[~fpj] do we have an umbrella jira where this issue is been tracked with the 
changes required to be made that are mentioned in this patch?

> a soft failure in controller may leave a topic partition in an inconsistent 
> state
> -
>
> Key: KAFKA-3083
> URL: https://issues.apache.org/jira/browse/KAFKA-3083
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.9.0.0
>Reporter: Jun Rao
>Assignee: Mayuresh Gharat
>
> The following sequence can happen.
> 1. Broker A is the controller and is in the middle of processing a broker 
> change event. As part of this process, let's say it's about to shrink the isr 
> of a partition.
> 2. Then broker A's session expires and broker B takes over as the new 
> controller. Broker B sends the initial leaderAndIsr request to all brokers.
> 3. Broker A continues by shrinking the isr of the partition in ZK and sends 
> the new leaderAndIsr request to the broker (say C) that leads the partition. 
> Broker C will reject this leaderAndIsr since the request comes from a 
> controller with an older epoch. Now we could be in a situation that Broker C 
> thinks the isr has all replicas, but the isr stored in ZK is different.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

RE: [DISCUSS] Time-based releases for Apache Kafka

[Andrew Schofield]

I agree that the Kafka community has managed to maintain a very high quality 
level, so I'm not concerned
about the quality of non-LTS releases. If the principle is that every release 
is supported for 2 years, that
would be good. I suppose that if the burden of having that many in-support 
releases proves too heavy,
as you say we could reconsider.

Andrew Schofield


> From: g...@confluent.io
> Date: Thu, 25 Aug 2016 09:57:30 -0700
> Subject: Re: [DISCUSS] Time-based releases for Apache Kafka
> To: dev@kafka.apache.org
>
> I prefer Ismael's suggestion for supporting 2-years (6 releases)
> rather than have designated LTS releases.
>
> The LTS model seems to work well when some releases are high quality
> (LTS) and the rest are a bit more questionable. It is great for
> companies like Redhat, where they have to invest less to support few
> releases and let the community deal with everything else.
>
> Until now the Kafka community has managed to maintain very high
> quality level. Not just for releases, our trunk is often of better
> quality than other project's releases - we don't think of stability as
> something you tuck into a release (and just some releases) but rather
> as an on-going concern. There are costs to doing things that way, but
> in general, I think it has served us well - allowing even conservative
> companies to run on the latest released version.
>
> I hope we can agree to at least try maintaining last 6 releases as LTS
> (i.e. every single release is supported for 2 years) rather than
> designate some releases as better than others. Of course, if this
> totally fails, we can reconsider.
>
> Gwen
>
> On Thu, Aug 25, 2016 at 9:51 AM, Andrew Schofield
>  wrote:
>> The proposal sounds pretty good, but the main thing currently missing is a 
>> proper long-term support release.
>>
>> Having 3 releases a year sounds OK, but if they're all equivalent and bugfix 
>> releases are produced for the most
>> recent 2 or 3 releases, anyone wanting to run on an "in support" release of 
>> Kafka has to upgrade every 8-12 months.
>> If you don't actually want anything specific from the newer releases, it's 
>> just unnecessary churn.
>>
>> Wouldn't it be better to designate one release every 12-18 months as a 
>> long-term support release with bugfix releases
>> produced for those for a longer period of say 24 months. That halves the 
>> upgrade work for people just wanting to keep
>> "in support". Now that adoption is increasing, there are plenty of users 
>> that just want a dependable messaging system
>> without having to be deeply knowledgeable about its innards.
>>
>> LTS works nicely for plenty of open-source projects. I think it would work 
>> well for Kafka too.
>>
>> Andrew Schofield
>>
>> 
>>> From: ofir.ma...@equalum.io
>>> Date: Thu, 25 Aug 2016 16:07:07 +0300
>>> Subject: Re: [DISCUSS] Time-based releases for Apache Kafka
>>> To: dev@kafka.apache.org
>>>
>>> Regarding bug fixes, you may want to consider to have an LTS release once a
>>> year - designating it for longer-term support / better for the masses.
>>> If you like that - then fix bugs in trunk, backport important ones to
>>> latest release + the last two LTS releases.
>>> Even if you don't - if a downstream distribution picks a Kafka version and
>>> plans to support it over a few years, it could be nice of them to "own"
>>> that older release - volunteer to be a release manager for bug backports to
>>> that version over a longer period...
>>> Just my two cents :)
>>>
>>> Ofir Manor
>>>
>>> Co-Founder & CTO | Equalum
>>>
>>> Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io
>>>
>>> On Thu, Aug 25, 2016 at 12:32 PM, Ismael Juma  wrote:
>>>
 Thanks for putting this together Gwen. I think it sounds reasonable and
 instead of trying to optimise every aspect of it ahead of time (which is
 hard, subjective and time-consuming), I am happy to try what is being
 proposed and tweak based on experience. One thing we should pay particular
 attention to is how the stabilisation process works out in practice.

 A couple of comments:

 "Given 3 releases a year and the fact that no one upgrades three times a
 year, we propose making sure (by testing!) that rolling upgrade can be done
 from each release in the past year (i.e. last 3 releases) to the latest
 version."

 Because the cost of doing this for a larger number of releases is
 relatively low, I still think we should go for 6 here (our code currently
 supports 5 versions as I said in a previous message, so we're close to that
 target already). I'm generally very keen to make upgrades as easy as
 possible so that people have no reason not to upgrade. :)

 "We will also attempt, as a community to do bugfix releases as needed for
 the last 3 releases."

 I would suggest 2, personally, but since this is a bit fuzzy, I am OK w

[GitHub] kafka-site pull request #18: Implementation: Clean-up invalid HTML

[epeay]

GitHub user epeay opened a pull request:

https://github.com/apache/kafka-site/pull/18

Implementation: Clean-up invalid HTML

Small typo I found in `implementation.html`

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/epeay/kafka-site patch-1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka-site/pull/18.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #18


commit b5098566f8ba84fff8c877508aa217c3eafddbf9
Author: Elliott Peay 
Date:   2016-08-25T17:06:08Z

Implementation: Clean-up invalid HTML

Small typo I found in `implementation.html`




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] Remove beta label from the new Java consumer

[Gwen Shapira]

Makes sense :)

On Thu, Aug 25, 2016 at 9:40 AM, Neha Narkhede  wrote:
> Yeah, I'm supportive of this.
>
> On Thu, Aug 25, 2016 at 9:26 AM Ismael Juma  wrote:
>
>> Hi Gwen,
>>
>> We have a few recent stories of people using Connect and Streams in
>> production. That means the new Java Consumer too. :)
>>
>> Ismael
>>
>> On Thu, Aug 25, 2016 at 5:09 PM, Gwen Shapira  wrote:
>>
>> > Originally, we suggested keeping the beta label until we know someone
>> > successfully uses the new consumer in production.
>> >
>> > We can consider the recent KIPs enough, but IMO it will be better if
>> > someone with production deployment hanging out on our mailing list
>> > will confirm good experience with the new consumer.
>> >
>> > Gwen
>> >
>> > On Wed, Aug 24, 2016 at 8:45 PM, Ismael Juma  wrote:
>> > > Hi all,
>> > >
>> > > We currently say the following in our documentation:
>> > >
>> > > "As of the 0.9.0 release we have added a new Java consumer to replace
>> our
>> > > existing high-level ZooKeeper-based consumer and low-level consumer
>> APIs.
>> > > This client is considered beta quality."[1]
>> > >
>> > > Since then, Jason and the community have done a lot of work to improve
>> it
>> > > (including KIP-41 and KIP-62), we declared it API stable in 0.10.0.0
>> and
>> > > it's the only option for those that need security support. Yes, it
>> still
>> > > has bugs, but so does the old consumer and all development is currently
>> > > focused on the new consumer.
>> > >
>> > > As such, I propose we remove the beta label for the next release and
>> > switch
>> > > our tools to use the new consumer by default unless the zookeeper
>> > > command-line option is present (for compatibility). This is similar to
>> > what
>> > > we did it for the new producer in 0.9.0.0, but backwards compatible.
>> > >
>> > > Thoughts?
>> > >
>> > > Ismael
>> > >
>> > > [1] http://kafka.apache.org/documentation.html#consumerapi
>> >
>> >
>> >
>> > --
>> > Gwen Shapira
>> > Product Manager | Confluent
>> > 650.450.2760 | @gwenshap
>> > Follow us: Twitter | blog
>> >
>>
> --
> Thanks,
> Neha



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [DISCUSS] Time-based releases for Apache Kafka

[Gwen Shapira]

I prefer Ismael's suggestion for supporting 2-years (6 releases)
rather than have designated LTS releases.

The LTS model seems to work well when some releases are high quality
(LTS) and the rest are a bit more questionable. It is great for
companies like Redhat, where they have to invest less to support few
releases and let the community deal with everything else.

Until now the Kafka community has managed to maintain very high
quality level. Not just for releases, our trunk is often of better
quality than other project's releases - we don't think of stability as
something you tuck into a release (and just some releases) but rather
as an on-going concern. There are costs to doing things that way, but
in general, I think it has served us well - allowing even conservative
companies to run on the latest released version.

I hope we can agree to at least try maintaining last 6 releases as LTS
(i.e. every single release is supported for 2 years) rather than
designate some releases as better than others. Of course, if this
totally fails, we can reconsider.

Gwen

On Thu, Aug 25, 2016 at 9:51 AM, Andrew Schofield
 wrote:
> The proposal sounds pretty good, but the main thing currently missing is a 
> proper long-term support release.
>
> Having 3 releases a year sounds OK, but if they're all equivalent and bugfix 
> releases are produced for the most
> recent 2 or 3 releases, anyone wanting to run on an "in support" release of 
> Kafka has to upgrade every 8-12 months.
> If you don't actually want anything specific from the newer releases, it's 
> just unnecessary churn.
>
> Wouldn't it be better to designate one release every 12-18 months as a 
> long-term support release with bugfix releases
> produced for those for a longer period of say 24 months. That halves the 
> upgrade work for people just wanting to keep
> "in support". Now that adoption is increasing, there are plenty of users that 
> just want a dependable messaging system
> without having to be deeply knowledgeable about its innards.
>
> LTS works nicely for plenty of open-source projects. I think it would work 
> well for Kafka too.
>
> Andrew Schofield
>
> 
>> From: ofir.ma...@equalum.io
>> Date: Thu, 25 Aug 2016 16:07:07 +0300
>> Subject: Re: [DISCUSS] Time-based releases for Apache Kafka
>> To: dev@kafka.apache.org
>>
>> Regarding bug fixes, you may want to consider to have an LTS release once a
>> year - designating it for longer-term support / better for the masses.
>> If you like that - then fix bugs in trunk, backport important ones to
>> latest release + the last two LTS releases.
>> Even if you don't - if a downstream distribution picks a Kafka version and
>> plans to support it over a few years, it could be nice of them to "own"
>> that older release - volunteer to be a release manager for bug backports to
>> that version over a longer period...
>> Just my two cents :)
>>
>> Ofir Manor
>>
>> Co-Founder & CTO | Equalum
>>
>> Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io
>>
>> On Thu, Aug 25, 2016 at 12:32 PM, Ismael Juma  wrote:
>>
>>> Thanks for putting this together Gwen. I think it sounds reasonable and
>>> instead of trying to optimise every aspect of it ahead of time (which is
>>> hard, subjective and time-consuming), I am happy to try what is being
>>> proposed and tweak based on experience. One thing we should pay particular
>>> attention to is how the stabilisation process works out in practice.
>>>
>>> A couple of comments:
>>>
>>> "Given 3 releases a year and the fact that no one upgrades three times a
>>> year, we propose making sure (by testing!) that rolling upgrade can be done
>>> from each release in the past year (i.e. last 3 releases) to the latest
>>> version."
>>>
>>> Because the cost of doing this for a larger number of releases is
>>> relatively low, I still think we should go for 6 here (our code currently
>>> supports 5 versions as I said in a previous message, so we're close to that
>>> target already). I'm generally very keen to make upgrades as easy as
>>> possible so that people have no reason not to upgrade. :)
>>>
>>> "We will also attempt, as a community to do bugfix releases as needed for
>>> the last 3 releases."
>>>
>>> I would suggest 2, personally, but since this is a bit fuzzy, I am OK with
>>> 3 if people prefer that.
>>>
>>> Ismael
>>>
>>> On Thu, Aug 25, 2016 at 6:22 AM, Gwen Shapira  wrote:
>>>
 Hi Team Kafka,

 As per the KIP meeting, I created a wiki:
 https://cwiki.apache.org/confluence/display/KAFKA/Time+
>>> Based+Release+Plan
 Summarizing most of the discussion so far.

 Comments and additional discussion is welcome :)

 Gwen

 On Wed, Aug 17, 2016 at 12:31 PM, Vahid S Hashemian
  wrote:
> Time-based releases is a good idea and something that has proved to be
> working in a number of open source projects. One successful example is
> Node.js, that goes through two major releases a year. Th

RE: [DISCUSS] Time-based releases for Apache Kafka

[Andrew Schofield]

The proposal sounds pretty good, but the main thing currently missing is a 
proper long-term support release.

Having 3 releases a year sounds OK, but if they're all equivalent and bugfix 
releases are produced for the most
recent 2 or 3 releases, anyone wanting to run on an "in support" release of 
Kafka has to upgrade every 8-12 months.
If you don't actually want anything specific from the newer releases, it's just 
unnecessary churn.

Wouldn't it be better to designate one release every 12-18 months as a 
long-term support release with bugfix releases
produced for those for a longer period of say 24 months. That halves the 
upgrade work for people just wanting to keep
"in support". Now that adoption is increasing, there are plenty of users that 
just want a dependable messaging system
without having to be deeply knowledgeable about its innards.

LTS works nicely for plenty of open-source projects. I think it would work well 
for Kafka too.

Andrew Schofield


> From: ofir.ma...@equalum.io
> Date: Thu, 25 Aug 2016 16:07:07 +0300
> Subject: Re: [DISCUSS] Time-based releases for Apache Kafka
> To: dev@kafka.apache.org
>
> Regarding bug fixes, you may want to consider to have an LTS release once a
> year - designating it for longer-term support / better for the masses.
> If you like that - then fix bugs in trunk, backport important ones to
> latest release + the last two LTS releases.
> Even if you don't - if a downstream distribution picks a Kafka version and
> plans to support it over a few years, it could be nice of them to "own"
> that older release - volunteer to be a release manager for bug backports to
> that version over a longer period...
> Just my two cents :)
>
> Ofir Manor
>
> Co-Founder & CTO | Equalum
>
> Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io
>
> On Thu, Aug 25, 2016 at 12:32 PM, Ismael Juma  wrote:
>
>> Thanks for putting this together Gwen. I think it sounds reasonable and
>> instead of trying to optimise every aspect of it ahead of time (which is
>> hard, subjective and time-consuming), I am happy to try what is being
>> proposed and tweak based on experience. One thing we should pay particular
>> attention to is how the stabilisation process works out in practice.
>>
>> A couple of comments:
>>
>> "Given 3 releases a year and the fact that no one upgrades three times a
>> year, we propose making sure (by testing!) that rolling upgrade can be done
>> from each release in the past year (i.e. last 3 releases) to the latest
>> version."
>>
>> Because the cost of doing this for a larger number of releases is
>> relatively low, I still think we should go for 6 here (our code currently
>> supports 5 versions as I said in a previous message, so we're close to that
>> target already). I'm generally very keen to make upgrades as easy as
>> possible so that people have no reason not to upgrade. :)
>>
>> "We will also attempt, as a community to do bugfix releases as needed for
>> the last 3 releases."
>>
>> I would suggest 2, personally, but since this is a bit fuzzy, I am OK with
>> 3 if people prefer that.
>>
>> Ismael
>>
>> On Thu, Aug 25, 2016 at 6:22 AM, Gwen Shapira  wrote:
>>
>>> Hi Team Kafka,
>>>
>>> As per the KIP meeting, I created a wiki:
>>> https://cwiki.apache.org/confluence/display/KAFKA/Time+
>> Based+Release+Plan
>>> Summarizing most of the discussion so far.
>>>
>>> Comments and additional discussion is welcome :)
>>>
>>> Gwen
>>>
>>> On Wed, Aug 17, 2016 at 12:31 PM, Vahid S Hashemian
>>>  wrote:
 Time-based releases is a good idea and something that has proved to be
 working in a number of open source projects. One successful example is
 Node.js, that goes through two major releases a year. The interesting
>>> fact
 about the two releases is that only one (the even-number release) comes
 with a long term support (LTS) plan (30 months). More can be read here:
 https://github.com/nodejs/LTS. The odd-number releases still come with
 major changes and help build the ecosystem, but as far as LTS goes,
>> there
 is only one per year. This LTS plan makes most enterprises want to
>> stick
 to even-number releases, which is okay since frequent upgrades is not
 something they are normally interested in anyway.

 There could be several minor releases (non-breaking) in between major
 releases. A major release contains all the features / bug fixes in the
 master branch a month before the release date, with the potential
>>> addition
 of (non-breaking) bug fixes until the release day. The deprecation
>> cycle
 is one major release: any functionality that is decided to be removed
>> is
 deprecated in the next major release, and removed in the major release
 after that.

 Because of the success of LTS models in this and other open source
 projects, I would suggest implementing a formal LTS plan for Kafka too.

 Regards,
 --Vahid



>>

[GitHub] kafka pull request #1785: HOTFIX: disabled application-reset-tool integratio...

[mjsax]

GitHub user mjsax opened a pull request:

https://github.com/apache/kafka/pull/1785

HOTFIX: disabled application-reset-tool integration test



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mjsax/kafka disableTest

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1785.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1785


commit 179ba5877d2dec862f4f51597ea65cc1aacc64ec
Author: Matthias J. Sax 
Date:   2016-08-25T16:33:51Z

HOTFIX: disabled application-reset-tool integration test




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] Remove beta label from the new Java consumer

[Neha Narkhede]

Yeah, I'm supportive of this.

On Thu, Aug 25, 2016 at 9:26 AM Ismael Juma  wrote:

> Hi Gwen,
>
> We have a few recent stories of people using Connect and Streams in
> production. That means the new Java Consumer too. :)
>
> Ismael
>
> On Thu, Aug 25, 2016 at 5:09 PM, Gwen Shapira  wrote:
>
> > Originally, we suggested keeping the beta label until we know someone
> > successfully uses the new consumer in production.
> >
> > We can consider the recent KIPs enough, but IMO it will be better if
> > someone with production deployment hanging out on our mailing list
> > will confirm good experience with the new consumer.
> >
> > Gwen
> >
> > On Wed, Aug 24, 2016 at 8:45 PM, Ismael Juma  wrote:
> > > Hi all,
> > >
> > > We currently say the following in our documentation:
> > >
> > > "As of the 0.9.0 release we have added a new Java consumer to replace
> our
> > > existing high-level ZooKeeper-based consumer and low-level consumer
> APIs.
> > > This client is considered beta quality."[1]
> > >
> > > Since then, Jason and the community have done a lot of work to improve
> it
> > > (including KIP-41 and KIP-62), we declared it API stable in 0.10.0.0
> and
> > > it's the only option for those that need security support. Yes, it
> still
> > > has bugs, but so does the old consumer and all development is currently
> > > focused on the new consumer.
> > >
> > > As such, I propose we remove the beta label for the next release and
> > switch
> > > our tools to use the new consumer by default unless the zookeeper
> > > command-line option is present (for compatibility). This is similar to
> > what
> > > we did it for the new producer in 0.9.0.0, but backwards compatible.
> > >
> > > Thoughts?
> > >
> > > Ismael
> > >
> > > [1] http://kafka.apache.org/documentation.html#consumerapi
> >
> >
> >
> > --
> > Gwen Shapira
> > Product Manager | Confluent
> > 650.450.2760 | @gwenshap
> > Follow us: Twitter | blog
> >
>
-- 
Thanks,
Neha

Re: [DISCUSS] Remove beta label from the new Java consumer

[Ismael Juma]

Hi Gwen,

We have a few recent stories of people using Connect and Streams in
production. That means the new Java Consumer too. :)

Ismael

On Thu, Aug 25, 2016 at 5:09 PM, Gwen Shapira  wrote:

> Originally, we suggested keeping the beta label until we know someone
> successfully uses the new consumer in production.
>
> We can consider the recent KIPs enough, but IMO it will be better if
> someone with production deployment hanging out on our mailing list
> will confirm good experience with the new consumer.
>
> Gwen
>
> On Wed, Aug 24, 2016 at 8:45 PM, Ismael Juma  wrote:
> > Hi all,
> >
> > We currently say the following in our documentation:
> >
> > "As of the 0.9.0 release we have added a new Java consumer to replace our
> > existing high-level ZooKeeper-based consumer and low-level consumer APIs.
> > This client is considered beta quality."[1]
> >
> > Since then, Jason and the community have done a lot of work to improve it
> > (including KIP-41 and KIP-62), we declared it API stable in 0.10.0.0 and
> > it's the only option for those that need security support. Yes, it still
> > has bugs, but so does the old consumer and all development is currently
> > focused on the new consumer.
> >
> > As such, I propose we remove the beta label for the next release and
> switch
> > our tools to use the new consumer by default unless the zookeeper
> > command-line option is present (for compatibility). This is similar to
> what
> > we did it for the new producer in 0.9.0.0, but backwards compatible.
> >
> > Thoughts?
> >
> > Ismael
> >
> > [1] http://kafka.apache.org/documentation.html#consumerapi
>
>
>
> --
> Gwen Shapira
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter | blog
>

Re: [DISCUSS] Remove beta label from the new Java consumer

[Gwen Shapira]

Originally, we suggested keeping the beta label until we know someone
successfully uses the new consumer in production.

We can consider the recent KIPs enough, but IMO it will be better if
someone with production deployment hanging out on our mailing list
will confirm good experience with the new consumer.

Gwen

On Wed, Aug 24, 2016 at 8:45 PM, Ismael Juma  wrote:
> Hi all,
>
> We currently say the following in our documentation:
>
> "As of the 0.9.0 release we have added a new Java consumer to replace our
> existing high-level ZooKeeper-based consumer and low-level consumer APIs.
> This client is considered beta quality."[1]
>
> Since then, Jason and the community have done a lot of work to improve it
> (including KIP-41 and KIP-62), we declared it API stable in 0.10.0.0 and
> it's the only option for those that need security support. Yes, it still
> has bugs, but so does the old consumer and all development is currently
> focused on the new consumer.
>
> As such, I propose we remove the beta label for the next release and switch
> our tools to use the new consumer by default unless the zookeeper
> command-line option is present (for compatibility). This is similar to what
> we did it for the new producer in 0.9.0.0, but backwards compatible.
>
> Thoughts?
>
> Ismael
>
> [1] http://kafka.apache.org/documentation.html#consumerapi



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [DISCUSS] KIP-76: Enable getting password from executable rather than passing as plaintext in config files

[Gwen Shapira]

Hi Ashish,

I appreciate the need to integrate our authentication with other
systems that store passwords.
I am not sure that doing so by running a binary is the best solution.

First, it does not add security: As you said, a file is just "sitting
there" the same way an executable is just "sitting there" - we still
rely on file system privileges for security.
Second, the idea that Kafka will run arbitrary filesystem executables
is pretty terrifying. Reading a string from a file is harmless, but an
incorrectly privileged executable can be replaced with "rm -rf /" or
anything really. Kafka sometimes runs from privileged account, so this
is a serious risk.

I looked at the Hadoop credential store you helpfully linked to in the
KIP, and it seems like the Hadoop proposal includes a well thought out
API to integrate with external systems. Since we took this approach in
the past, I'm wondering why not follow the same and use an API to
integrate with credential stores rather than arbitrary executables.

Gwen

On Wed, Aug 24, 2016 at 12:03 PM, Ashish Singh  wrote:
> Hey Guys,
>
> I’ve just posted KIP-76: Enable getting password from executable rather
> than passing as plaintext in config files
> 
> .
>
> The proposal is to enable getting passwords from executable. This is an ask
> from very security conscious users.
>
> Full details are here:
>
> KIP:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-76+Enable+getting+password+from+executable+rather+than+passing+as+plaintext+in+config+files
> JIRA: https://issues.apache.org/jira/browse/KAFKA-2629
> POC: https://github.com/apache/kafka/pull/1770
>
> Thanks
>
> --
>
> Regards,
> Ashish



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [VOTE] KIP-63: Unify store and downstream caching in streams

[Matthias J. Sax]

+1

On 08/25/2016 04:22 PM, Damian Guy wrote:
> +1
> 
> On Thu, 25 Aug 2016 at 11:57 Eno Thereska  wrote:
> 
>> Hi folks,
>>
>> We'd like to start the vote for KIP-63. At this point the Wiki addresses
>> all previous questions and we believe the PoC is feature-complete.
>>
>> Thanks
>> Eno
>>
> 



signature.asc
Description: OpenPGP digital signature

[jira] [Resolved] (KAFKA-4015) Change cleanup.policy config to accept a list of valid policies

[Jun Rao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao resolved KAFKA-4015.

Resolution: Fixed

Issue resolved by pull request 1742
[https://github.com/apache/kafka/pull/1742]

> Change cleanup.policy config to accept a list of valid policies
> ---
>
> Key: KAFKA-4015
> URL: https://issues.apache.org/jira/browse/KAFKA-4015
> Project: Kafka
>  Issue Type: New Feature
>  Components: core
>Affects Versions: 0.10.1.0
>Reporter: Damian Guy
>Assignee: Damian Guy
> Fix For: 0.10.1.0
>
>
> There are some use cases where it is desirable to have a topic that supports 
> both compact and delete policies, i.e., any topic that wants to be compacted 
> by key, but also wants keys that haven't been updated for some time to be 
> automatically expired.
> Add a new compact_and_delete option to cleanup.policy. When set, both compact 
> and delete cleanup strategies should run. This change needs to guarantee 
> thread-safety.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4015) Change cleanup.policy config to accept a list of valid policies

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15437062#comment-15437062
 ] 

ASF GitHub Bot commented on KAFKA-4015:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1742


> Change cleanup.policy config to accept a list of valid policies
> ---
>
> Key: KAFKA-4015
> URL: https://issues.apache.org/jira/browse/KAFKA-4015
> Project: Kafka
>  Issue Type: New Feature
>  Components: core
>Affects Versions: 0.10.1.0
>Reporter: Damian Guy
>Assignee: Damian Guy
> Fix For: 0.10.1.0
>
>
> There are some use cases where it is desirable to have a topic that supports 
> both compact and delete policies, i.e., any topic that wants to be compacted 
> by key, but also wants keys that haven't been updated for some time to be 
> automatically expired.
> Add a new compact_and_delete option to cleanup.policy. When set, both compact 
> and delete cleanup strategies should run. This change needs to guarantee 
> thread-safety.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1742: KAFKA-4015: Change cleanup.policy config to accept...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1742


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-955) After a leader change, messages sent with ack=0 are lost

[Mazhar Shaikh (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15436999#comment-15436999
 ] 

Mazhar Shaikh commented on KAFKA-955:
-

Hi All,

affected version & fixed version is same, just want to know if this fix is 
available in "0.8.2"

I'm facing similar issue in "0.8.2".

Thanks.

> After a leader change, messages sent with ack=0 are lost
> 
>
> Key: KAFKA-955
> URL: https://issues.apache.org/jira/browse/KAFKA-955
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.0
>Reporter: Jason Rosenberg
>Assignee: Guozhang Wang
> Fix For: 0.8.0
>
> Attachments: KAFKA-955-followup.v1.patch, KAFKA-955.v1.patch, 
> KAFKA-955.v1.patch, KAFKA-955.v2.patch, KAFKA-955.v3.patch, 
> KAFKA-955.v4.patch, KAFKA-955.v5.patch, KAFKA-955.v6.patch, KAFKA-955.v7.patch
>
>
> If the leader changes for a partition, and a producer is sending messages 
> with ack=0, then messages will be lost, since the producer has no active way 
> of knowing that the leader has changed, until it's next metadata refresh 
> update.
> The broker receiving the message, which is no longer the leader, logs a 
> message like this:
> Produce request with correlation id 7136261 from client  on partition 
> [mytopic,0] failed due to Leader not local for partition [mytopic,0] on 
> broker 508818741
> This is exacerbated by the controlled shutdown mechanism, which forces an 
> immediate leader change.
> A possible solution to this would be for a broker which receives a message, 
> for a topic that it is no longer the leader for (and if the ack level is 0), 
> then the broker could just silently forward the message over to the current 
> leader.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] KIP-63: Unify store and downstream caching in streams

[Damian Guy]

+1

On Thu, 25 Aug 2016 at 11:57 Eno Thereska  wrote:

> Hi folks,
>
> We'd like to start the vote for KIP-63. At this point the Wiki addresses
> all previous questions and we believe the PoC is feature-complete.
>
> Thanks
> Eno
>

Re: [DISCUSS] Time-based releases for Apache Kafka

[Ofir Manor]

Regarding bug fixes, you may want to consider to have an LTS release once a
year - designating it for longer-term support / better for the masses.
If you like that - then fix bugs in trunk, backport important ones to
latest release + the last two LTS releases.
Even if you don't - if a downstream distribution picks a Kafka version and
plans to support it over a few years, it could be nice of them to "own"
that older release - volunteer to be a release manager for bug backports to
that version over a longer period...
Just my two cents :)

Ofir Manor

Co-Founder & CTO | Equalum

Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io

On Thu, Aug 25, 2016 at 12:32 PM, Ismael Juma  wrote:

> Thanks for putting this together Gwen. I think it sounds reasonable and
> instead of trying to optimise every aspect of it ahead of time (which is
> hard, subjective and time-consuming), I am happy to try what is being
> proposed and tweak based on experience. One thing we should pay particular
> attention to is how the stabilisation process works out in practice.
>
> A couple of comments:
>
> "Given 3 releases a year and the fact that no one upgrades three times a
> year, we propose making sure (by testing!) that rolling upgrade can be done
> from each release in the past year (i.e. last 3 releases) to the latest
> version."
>
> Because the cost of doing this for a larger number of releases is
> relatively low, I still think we should go for 6 here (our code currently
> supports 5 versions as I said in a previous message, so we're close to that
> target already). I'm generally very keen to make upgrades as easy as
> possible so that people have no reason not to upgrade. :)
>
> "We will also attempt, as a community to do bugfix releases as needed for
> the last 3 releases."
>
> I would suggest 2, personally, but since this is a bit fuzzy, I am OK with
> 3 if people prefer that.
>
> Ismael
>
> On Thu, Aug 25, 2016 at 6:22 AM, Gwen Shapira  wrote:
>
> > Hi Team Kafka,
> >
> > As per the KIP meeting, I created a wiki:
> > https://cwiki.apache.org/confluence/display/KAFKA/Time+
> Based+Release+Plan
> > Summarizing most of the discussion so far.
> >
> > Comments and additional discussion is welcome :)
> >
> > Gwen
> >
> > On Wed, Aug 17, 2016 at 12:31 PM, Vahid S Hashemian
> >  wrote:
> > > Time-based releases is a good idea and something that has proved to be
> > > working in a number of open source projects. One successful example is
> > > Node.js, that goes through two major releases a year. The interesting
> > fact
> > > about the two releases is that only one (the even-number release) comes
> > > with a long term support (LTS) plan (30 months). More can be read here:
> > > https://github.com/nodejs/LTS. The odd-number releases still come with
> > > major changes and help build the ecosystem, but as far as LTS goes,
> there
> > > is only one per year. This LTS plan makes most enterprises want to
> stick
> > > to even-number releases, which is okay since frequent upgrades is not
> > > something they are normally interested in anyway.
> > >
> > > There could be several minor releases (non-breaking) in between major
> > > releases. A major release contains all the features / bug fixes in the
> > > master branch a month before the release date, with the potential
> > addition
> > > of (non-breaking) bug fixes until the release day. The deprecation
> cycle
> > > is one major release: any functionality that is decided to be removed
> is
> > > deprecated in the next major release, and removed in the major release
> > > after that.
> > >
> > > Because of the success of LTS models in this and other open source
> > > projects, I would suggest implementing a formal LTS plan for Kafka too.
> > >
> > > Regards,
> > > --Vahid
> > >
> > >
> > >
> > > From:   Gwen Shapira 
> > > To: dev@kafka.apache.org
> > > Date:   08/09/2016 04:49 PM
> > > Subject:[DISCUSS] Time-based releases for Apache Kafka
> > >
> > >
> > >
> > > Dear Kafka Developers and Users,
> > >
> > > In the past, our releases have been quite unpredictable. We'll notice
> > > that a large number of nice features made it in (or are close),
> > > someone would suggest a release and we'd do it. This is fun, but makes
> > > planning really hard - we saw it during the last release which we
> > > decided to delay by a few weeks to allow more features to "land".
> > >
> > > Many other communities have adopted time-based releases successfully
> > > (Cassandra, GCC, LLVM, Fedora, Gnome, Ubuntu, etc.). And I thought it
> > > will make sense for the Apache Kafka community to try doing the same.
> > >
> > > The benefits of this approach are:
> > >
> > > 1. A quicker feedback cycle and users can benefit from features
> > > quicker (assuming for reasonably short time between releases - I was
> > > thinking 4 months)
> > >
> > > 2. Predictability for contributors and users:
> > > * Developers and reviewers can decide in advance what release they are
> > > aiming for with specific 

[VOTE] KIP-63: Unify store and downstream caching in streams

[Eno Thereska]

Hi folks,

We'd like to start the vote for KIP-63. At this point the Wiki addresses
all previous questions and we believe the PoC is feature-complete.

Thanks
Eno

[jira] [Commented] (KAFKA-3172) Consumer threads stay in 'Watiting' status and are blocked at consumer poll method

[Oleg Gorobets (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15436624#comment-15436624
 ] 

Oleg Gorobets commented on KAFKA-3172:
--

Looks like JDK bug: https://bugs.openjdk.java.net/browse/JDK-8153192

> Consumer threads stay in 'Watiting' status and are blocked at consumer poll 
> method
> --
>
> Key: KAFKA-3172
> URL: https://issues.apache.org/jira/browse/KAFKA-3172
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.9.0.0
> Environment: linux
>Reporter: Dany Benjamin
>Assignee: Neha Narkhede
>Priority: Critical
> Fix For: 0.9.0.0
>
> Attachments: jmx_info.png, jstack.png, lagSample.png
>
>
> When running multiple consumers on same group (400 - for a 400 partitioned 
> topic), the application for all threads blocks at consumer.poll() method. The 
> timeout parameter sent in is 1.
> Stack dump:
> "pool-1-thread-198" #424 prio=5 os_prio=0 tid=0x7f6bb6d53800 nid=0xc349 
> waiting on condition [0x7f63df8f7000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x000605812710> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at 
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "kafka-producer-network-thread | producer-198" #423 daemon prio=5 os_prio=0 
> tid=0x7f6bb6d52000 nid=0xc348 runnable [0x7f63df9f8000]
>java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> - locked <0x0006058283e8> (a sun.nio.ch.Util$2)
> - locked <0x0006058283d8> (a 
> java.util.Collections$UnmodifiableSet)
> - locked <0x000605828390> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> at org.apache.kafka.common.network.Selector.select(Selector.java:425)
> at org.apache.kafka.common.network.Selector.poll(Selector.java:254)
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:270)
> at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:216)
> at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:128)
> at java.lang.Thread.run(Thread.java:745)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4065) Property missing in ProcuderConfig.java - KafkaProducer API 0.9.0.0

[Manikumar Reddy (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15436585#comment-15436585
 ] 

Manikumar Reddy commented on KAFKA-4065:


Agree, currently new producer don't support "compressed.topics" config 
property, This support can be added. Let us wait for other opinions.

> Property missing in ProcuderConfig.java - KafkaProducer API 0.9.0.0
> ---
>
> Key: KAFKA-4065
> URL: https://issues.apache.org/jira/browse/KAFKA-4065
> Project: Kafka
>  Issue Type: Bug
>Reporter: manzar
>
> 1 ) "compressed.topics" property is missing in ProducerConfig.java in 
> KafkaProducer API 0.9.0.0. due to that we can't enable some specific topic 
> for compression.
> 2) "compression.type" property is there in ProducerConfig.java that was 
> expected to be "compression.codec" according to official document.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4074) Deleting a topic can make it unavailable even if delete.topic.enable is false

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15436571#comment-15436571
 ] 

ASF GitHub Bot commented on KAFKA-4074:
---

GitHub user omkreddy opened a pull request:

https://github.com/apache/kafka/pull/1784

KAFKA-4074: Deleting a topic can make it unavailable even if 
delete.topic.enable is false



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/omkreddy/kafka KAFKA-4074-DELETE-TOPIC

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1784.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1784


commit 0dddedfb6b131908ed22cb1571cfb8149458ca7f
Author: Manikumar Reddy O 
Date:   2016-08-25T09:33:13Z

KAFKA-4074: Deleting a topic can make it unavailable even if 
delete.topic.enable is false




> Deleting a topic can make it unavailable even if delete.topic.enable is false
> -
>
> Key: KAFKA-4074
> URL: https://issues.apache.org/jira/browse/KAFKA-4074
> Project: Kafka
>  Issue Type: Bug
>  Components: controller
>Reporter: Joel Koshy
>Assignee: Manikumar Reddy
> Fix For: 0.10.1.0
>
>
> The {{delete.topic.enable}} configuration does not completely block the 
> effects of delete topic since the controller may (indirectly) query the list 
> of topics under the delete-topic znode.
> To reproduce:
> * Delete topic X
> * Force a controller move (either by bouncing or removing the controller 
> znode)
> * The new controller will send out UpdateMetadataRequests with leader=-2 for 
> the partitions of X
> * Producers eventually stop producing to that topic
> The reason for this is that when ControllerChannelManager adds 
> UpdateMetadataRequests for brokers, we directly use the partitionsToBeDeleted 
> field of the DeleteTopicManager (which is set to the partitions of the topics 
> under the delete-topic znode on controller startup).
> In order to get out of the situation you have to remove X from the znode and 
> then force another controller move.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1784: KAFKA-4074: Deleting a topic can make it unavailab...

[omkreddy]

GitHub user omkreddy opened a pull request:

https://github.com/apache/kafka/pull/1784

KAFKA-4074: Deleting a topic can make it unavailable even if 
delete.topic.enable is false



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/omkreddy/kafka KAFKA-4074-DELETE-TOPIC

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1784.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1784


commit 0dddedfb6b131908ed22cb1571cfb8149458ca7f
Author: Manikumar Reddy O 
Date:   2016-08-25T09:33:13Z

KAFKA-4074: Deleting a topic can make it unavailable even if 
delete.topic.enable is false




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] Time-based releases for Apache Kafka

[Ismael Juma]

Thanks for putting this together Gwen. I think it sounds reasonable and
instead of trying to optimise every aspect of it ahead of time (which is
hard, subjective and time-consuming), I am happy to try what is being
proposed and tweak based on experience. One thing we should pay particular
attention to is how the stabilisation process works out in practice.

A couple of comments:

"Given 3 releases a year and the fact that no one upgrades three times a
year, we propose making sure (by testing!) that rolling upgrade can be done
from each release in the past year (i.e. last 3 releases) to the latest
version."

Because the cost of doing this for a larger number of releases is
relatively low, I still think we should go for 6 here (our code currently
supports 5 versions as I said in a previous message, so we're close to that
target already). I'm generally very keen to make upgrades as easy as
possible so that people have no reason not to upgrade. :)

"We will also attempt, as a community to do bugfix releases as needed for
the last 3 releases."

I would suggest 2, personally, but since this is a bit fuzzy, I am OK with
3 if people prefer that.

Ismael

On Thu, Aug 25, 2016 at 6:22 AM, Gwen Shapira  wrote:

> Hi Team Kafka,
>
> As per the KIP meeting, I created a wiki:
> https://cwiki.apache.org/confluence/display/KAFKA/Time+Based+Release+Plan
> Summarizing most of the discussion so far.
>
> Comments and additional discussion is welcome :)
>
> Gwen
>
> On Wed, Aug 17, 2016 at 12:31 PM, Vahid S Hashemian
>  wrote:
> > Time-based releases is a good idea and something that has proved to be
> > working in a number of open source projects. One successful example is
> > Node.js, that goes through two major releases a year. The interesting
> fact
> > about the two releases is that only one (the even-number release) comes
> > with a long term support (LTS) plan (30 months). More can be read here:
> > https://github.com/nodejs/LTS. The odd-number releases still come with
> > major changes and help build the ecosystem, but as far as LTS goes, there
> > is only one per year. This LTS plan makes most enterprises want to stick
> > to even-number releases, which is okay since frequent upgrades is not
> > something they are normally interested in anyway.
> >
> > There could be several minor releases (non-breaking) in between major
> > releases. A major release contains all the features / bug fixes in the
> > master branch a month before the release date, with the potential
> addition
> > of (non-breaking) bug fixes until the release day. The deprecation cycle
> > is one major release: any functionality that is decided to be removed is
> > deprecated in the next major release, and removed in the major release
> > after that.
> >
> > Because of the success of LTS models in this and other open source
> > projects, I would suggest implementing a formal LTS plan for Kafka too.
> >
> > Regards,
> > --Vahid
> >
> >
> >
> > From:   Gwen Shapira 
> > To: dev@kafka.apache.org
> > Date:   08/09/2016 04:49 PM
> > Subject:[DISCUSS] Time-based releases for Apache Kafka
> >
> >
> >
> > Dear Kafka Developers and Users,
> >
> > In the past, our releases have been quite unpredictable. We'll notice
> > that a large number of nice features made it in (or are close),
> > someone would suggest a release and we'd do it. This is fun, but makes
> > planning really hard - we saw it during the last release which we
> > decided to delay by a few weeks to allow more features to "land".
> >
> > Many other communities have adopted time-based releases successfully
> > (Cassandra, GCC, LLVM, Fedora, Gnome, Ubuntu, etc.). And I thought it
> > will make sense for the Apache Kafka community to try doing the same.
> >
> > The benefits of this approach are:
> >
> > 1. A quicker feedback cycle and users can benefit from features
> > quicker (assuming for reasonably short time between releases - I was
> > thinking 4 months)
> >
> > 2. Predictability for contributors and users:
> > * Developers and reviewers can decide in advance what release they are
> > aiming for with specific features.
> > * If a feature misses a release we have a good idea of when it will show
> > up.
> > * Users know when to expect their features
> >
> > 3. Transparency - There will be a published cut-off date (AKA feature
> > freeze) for the release and people will know about it in advance.
> > Hopefully this will remove the contention around which features make
> > it.
> >
> > 4. Quality - we've seen issues pop up in release candidates due to
> > last-minute features that didn't have proper time to bake in. More
> > time between feature freeze and release will let us test more,
> > document more and resolve more issues.
> >
> > Since nothing is ever perfect, there will be some downsides:
> >
> > 1. Most notably, features that miss the feature-freeze date for a
> > release will have to wait few month for the next release. Features
> > will reach users faster overall as per benefit #

[jira] [Updated] (KAFKA-3777) Extract the existing LRU cache out of RocksDBStore

[Eno Thereska (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eno Thereska updated KAFKA-3777:

Assignee: Damian Guy  (was: Eno Thereska)

> Extract the existing LRU cache out of RocksDBStore
> --
>
> Key: KAFKA-3777
> URL: https://issues.apache.org/jira/browse/KAFKA-3777
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams
>Affects Versions: 0.10.1.0
>Reporter: Eno Thereska
>Assignee: Damian Guy
> Fix For: 0.10.1.0
>
>
> The LRU cache that is currently inside the RocksDbStore class. As part of 
> KAFKA-3776 it needs to come outside of RocksDbStore and be a separate 
> component used in:
> 1. KGroupedStream.aggregate() / reduce(), 
> 2. KStream.aggregateByKey() / reduceByKey(),
> 3. KTable.to() (this will be done in KAFKA-3779).
> As all of the above operators can have a cache on top to deduplicate the 
> materialized state store in RocksDB.
> The scope of this JIRA is to extract out the cache of RocksDBStore, and keep 
> them as item 1) and 2) above; and it should be done together / after 
> KAFKA-3780.
> Note it is NOT in the scope of this JIRA to re-write the cache, so this will 
> basically stay the same record-based cache we currently have.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-3961) broker sends malformed response when switching from no compression to snappy/gzip

[Dieter Plaetinck (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dieter Plaetinck resolved KAFKA-3961.
-
Resolution: Invalid

This is most likely due to an issue with the sarama client library , not kafka 
itself. see https://github.com/Shopify/sarama/issues/720 

> broker sends malformed response when switching from no compression to 
> snappy/gzip
> -
>
> Key: KAFKA-3961
> URL: https://issues.apache.org/jira/browse/KAFKA-3961
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.0.0
> Environment: docker container java:openjdk-8-jre on arch linux 
> 4.5.4-1-ARCH
>Reporter: Dieter Plaetinck
>
> Hi this is my first time using this tracker, so please bear with me (priority 
> seems to be major by default?)
> I should be allowed to switch back and forth between none/gzip/snappy 
> compression to the same topic/partition, right?
> (I couldn't find this explicitly anywhere but seems implied through the docs 
> and also from https://issues.apache.org/jira/browse/KAFKA-1499)
> when I try this, first i use no compression, than kill my producer, restart 
> it with snappy or gzip compression, send data to the same topic/partition 
> again, it seems the broker is sending a malformed response to my consumer.  
> At least that's what was suggested when i was reporting this problem in the 
> tracker for the client library I use 
> (https://github.com/Shopify/sarama/issues/698). Also noteworthy is that the 
> broker doesn't log anything when this happens.
> thanks!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)