Jenkins build is back to normal : kafka-trunk-jdk7 #1895

[Apache Jenkins Server]

See 

Build failed in Jenkins: kafka-trunk-jdk7 #1894

[Apache Jenkins Server]

See 

Changes:

[wangguoz] MINOR: update JavaDocs for Kafka Streams DSL helpers

[ismael] KAFKA-4635; Client Compatibility follow-ups

[ismael] KAFKA-4450; Add upgrade tests for 0.10.1 and rename TRUNK to DEV_BRANCH

--
[...truncated 4084 lines...]

kafka.coordinator.GroupMetadataManagerTest > testLoadGroupWithTombstone STARTED

kafka.coordinator.GroupMetadataManagerTest > testLoadGroupWithTombstone PASSED

kafka.coordinator.GroupMetadataManagerTest > testLoadOffsetsAndGroup STARTED

kafka.coordinator.GroupMetadataManagerTest > testLoadOffsetsAndGroup PASSED

kafka.coordinator.GroupMetadataManagerTest > testStoreGroupErrorMapping STARTED

kafka.coordinator.GroupMetadataManagerTest > testStoreGroupErrorMapping PASSED

kafka.coordinator.GroupMetadataManagerTest > testCommitOffsetFailure STARTED

kafka.coordinator.GroupMetadataManagerTest > testCommitOffsetFailure PASSED

kafka.coordinator.GroupMetadataManagerTest > testExpireOffset STARTED

kafka.coordinator.GroupMetadataManagerTest > testExpireOffset PASSED

kafka.coordinator.GroupMetadataManagerTest > testExpireOffsetsWithActiveGroup 
STARTED

kafka.coordinator.GroupMetadataManagerTest > testExpireOffsetsWithActiveGroup 
PASSED

kafka.coordinator.GroupMetadataManagerTest > testExpireGroupWithOffsetsOnly 
STARTED

kafka.coordinator.GroupMetadataManagerTest > testExpireGroupWithOffsetsOnly 
PASSED

kafka.coordinator.GroupMetadataManagerTest > testStoreEmptyGroup STARTED

kafka.coordinator.GroupMetadataManagerTest > testStoreEmptyGroup PASSED

kafka.coordinator.MemberMetadataTest > testMatchesSupportedProtocols STARTED

kafka.coordinator.MemberMetadataTest > testMatchesSupportedProtocols PASSED

kafka.coordinator.MemberMetadataTest > testMetadata STARTED

kafka.coordinator.MemberMetadataTest > testMetadata PASSED

kafka.coordinator.MemberMetadataTest > testMetadataRaisesOnUnsupportedProtocol 
STARTED

kafka.coordinator.MemberMetadataTest > testMetadataRaisesOnUnsupportedProtocol 
PASSED

kafka.coordinator.MemberMetadataTest > testVoteForPreferredProtocol STARTED

kafka.coordinator.MemberMetadataTest > testVoteForPreferredProtocol PASSED

kafka.coordinator.MemberMetadataTest > testVoteRaisesOnNoSupportedProtocols 
STARTED

kafka.coordinator.MemberMetadataTest > testVoteRaisesOnNoSupportedProtocols 
PASSED

kafka.coordinator.GroupMetadataTest > testDeadToAwaitingSyncIllegalTransition 
STARTED

kafka.coordinator.GroupMetadataTest > testDeadToAwaitingSyncIllegalTransition 
PASSED

kafka.coordinator.GroupMetadataTest > testOffsetCommitFailure STARTED

kafka.coordinator.GroupMetadataTest > testOffsetCommitFailure PASSED

kafka.coordinator.GroupMetadataTest > 
testPreparingRebalanceToStableIllegalTransition STARTED

kafka.coordinator.GroupMetadataTest > 
testPreparingRebalanceToStableIllegalTransition PASSED

kafka.coordinator.GroupMetadataTest > testStableToDeadTransition STARTED

kafka.coordinator.GroupMetadataTest > testStableToDeadTransition PASSED

kafka.coordinator.GroupMetadataTest > testInitNextGenerationEmptyGroup STARTED

kafka.coordinator.GroupMetadataTest > testInitNextGenerationEmptyGroup PASSED

kafka.coordinator.GroupMetadataTest > testCannotRebalanceWhenDead STARTED

kafka.coordinator.GroupMetadataTest > testCannotRebalanceWhenDead PASSED

kafka.coordinator.GroupMetadataTest > testInitNextGeneration STARTED

kafka.coordinator.GroupMetadataTest > testInitNextGeneration PASSED

kafka.coordinator.GroupMetadataTest > testPreparingRebalanceToEmptyTransition 
STARTED

kafka.coordinator.GroupMetadataTest > testPreparingRebalanceToEmptyTransition 
PASSED

kafka.coordinator.GroupMetadataTest > testSelectProtocol STARTED

kafka.coordinator.GroupMetadataTest > testSelectProtocol PASSED

kafka.coordinator.GroupMetadataTest > testCannotRebalanceWhenPreparingRebalance 
STARTED

kafka.coordinator.GroupMetadataTest > testCannotRebalanceWhenPreparingRebalance 
PASSED

kafka.coordinator.GroupMetadataTest > 
testDeadToPreparingRebalanceIllegalTransition STARTED

kafka.coordinator.GroupMetadataTest > 
testDeadToPreparingRebalanceIllegalTransition PASSED

kafka.coordinator.GroupMetadataTest > testCanRebalanceWhenAwaitingSync STARTED

kafka.coordinator.GroupMetadataTest > testCanRebalanceWhenAwaitingSync PASSED

kafka.coordinator.GroupMetadataTest > 
testAwaitingSyncToPreparingRebalanceTransition STARTED

kafka.coordinator.GroupMetadataTest > 
testAwaitingSyncToPreparingRebalanceTransition PASSED

kafka.coordinator.GroupMetadataTest > testStableToAwaitingSyncIllegalTransition 
STARTED

kafka.coordinator.GroupMetadataTest > testStableToAwaitingSyncIllegalTransition 
PASSED

kafka.coordinator.GroupMetadataTest > testEmptyToDeadTransition STARTED

kafka.coordinator.GroupMetadataTest > testEmptyToDeadTransition PASSED

kafka.coordinator.GroupMetadataTest > testSelectProtocolRaisesIfNoMembers 
STARTED

kafka.coordinator.GroupMetadataTest > testSelectProtocolRaisesIfNoMembers 

Build failed in Jenkins: kafka-trunk-jdk8 #1232

[Apache Jenkins Server]

See 

Changes:

[me] KAFKA-4679: Remove unstable markers from Connect APIs

--
[...truncated 95 lines...]
  if (partitionData.timestamp == 
OffsetCommitRequest.DEFAULT_TIMESTAMP)
^
:326:
 value timestamp in class PartitionData is deprecated: see corresponding 
Javadoc for more information.
offsetRetention + partitionData.timestamp
^
:578:
 method offsetData in class ListOffsetRequest is deprecated: see corresponding 
Javadoc for more information.
val (authorizedRequestInfo, unauthorizedRequestInfo) = 
offsetRequest.offsetData.asScala.partition {
 ^
:578:
 class PartitionData in object ListOffsetRequest is deprecated: see 
corresponding Javadoc for more information.
val (authorizedRequestInfo, unauthorizedRequestInfo) = 
offsetRequest.offsetData.asScala.partition {

  ^
:583:
 constructor PartitionData in class PartitionData is deprecated: see 
corresponding Javadoc for more information.
  new 
ListOffsetResponse.PartitionData(Errors.UNKNOWN_TOPIC_OR_PARTITION.code, 
List[JLong]().asJava)
  ^
:608:
 constructor PartitionData in class PartitionData is deprecated: see 
corresponding Javadoc for more information.
(topicPartition, new ListOffsetResponse.PartitionData(Errors.NONE.code, 
offsets.map(new JLong(_)).asJava))
 ^
:615:
 constructor PartitionData in class PartitionData is deprecated: see 
corresponding Javadoc for more information.
  (topicPartition, new 
ListOffsetResponse.PartitionData(Errors.forException(e).code, 
List[JLong]().asJava))
   ^
:618:
 constructor PartitionData in class PartitionData is deprecated: see 
corresponding Javadoc for more information.
  (topicPartition, new 
ListOffsetResponse.PartitionData(Errors.forException(e).code, 
List[JLong]().asJava))
   ^
:269:
 class PartitionData in object ListOffsetRequest is deprecated: see 
corresponding Javadoc for more information.
val partitions = Map(topicPartition -> new 
ListOffsetRequest.PartitionData(earliestOrLatest, 1))
 ^
:282:
 value offsets in class PartitionData is deprecated: see corresponding Javadoc 
for more information.
  partitionData.offsets.get(0)
^
:45:
 class OldProducer in package producer is deprecated: This class has been 
deprecated and will be removed in a future release. Please use 
org.apache.kafka.clients.producer.KafkaProducer instead.
new OldProducer(getOldProducerProps(config))
^
:47:
 class NewShinyProducer in package producer is deprecated: This class has been 
deprecated and will be removed in a future release. Please use 
org.apache.kafka.clients.producer.KafkaProducer instead.
new NewShinyProducer(getNewProducerProps(config))
^
21 warnings found
warning: [options] bootstrap class path not set in conjunction with -source 1.7
1 warning
:kafka-trunk-jdk8:core:processResources UP-TO-DATE
:kafka-trunk-jdk8:core:classes
:kafka-trunk-jdk8:core:checkstyleMain
:kafka-trunk-jdk8:core:compileTestJava UP-TO-DATE
:kafka-trunk-jdk8:core:compileTestScala
:88:
 method createAndShutdownStep in class MetricsTest is deprecated: This test has 
been deprecated and it will be removed in a future release
createAndShutdownStep("group0", "consumer0", "producer0")
  

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

[radai]

a few extra points:

1. broker per disk might also incur more client <--> broker sockets:
suppose every producer / consumer "talks" to >1 partition, there's a very
good chance that partitions that were co-located on a single 10-disk broker
would now be split between several single-disk broker processes on the same
machine. hard to put a multiplier on this, but likely >x1. sockets are a
limited resource at the OS level and incur some memory cost (kernel buffers)

2. there's a memory overhead to spinning up a JVM (compiled code and byte
code objects etc). if we assume this overhead is ~300 MB (order of
magnitude, specifics vary) than spinning up 10 JVMs would lose you 3 GB of
RAM. not a ton, but non negligible.

3. there would also be some overhead downstream of kafka in any management
/ monitoring / log aggregation system. likely less than x10 though.

4. (related to above) - added complexity of administration with more
running instances.

is anyone running kafka with anywhere near 100GB heaps? i thought the point
was to rely on kernel page cache to do the disk buffering 

On Thu, Jan 26, 2017 at 11:00 AM, Dong Lin  wrote:

> Hey Colin,
>
> Thanks much for the comment. Please see me comment inline.
>
> On Thu, Jan 26, 2017 at 10:15 AM, Colin McCabe  wrote:
>
> > On Wed, Jan 25, 2017, at 13:50, Dong Lin wrote:
> > > Hey Colin,
> > >
> > > Good point! Yeah we have actually considered and tested this solution,
> > > which we call one-broker-per-disk. It would work and should require no
> > > major change in Kafka as compared to this JBOD KIP. So it would be a
> good
> > > short term solution.
> > >
> > > But it has a few drawbacks which makes it less desirable in the long
> > > term.
> > > Assume we have 10 disks on a machine. Here are the problems:
> >
> > Hi Dong,
> >
> > Thanks for the thoughtful reply.
> >
> > >
> > > 1) Our stress test result shows that one-broker-per-disk has 15% lower
> > > throughput
> > >
> > > 2) Controller would need to send 10X as many LeaderAndIsrRequest,
> > > MetadataUpdateRequest and StopReplicaRequest. This increases the burden
> > > on
> > > controller which can be the performance bottleneck.
> >
> > Maybe I'm misunderstanding something, but there would not be 10x as many
> > StopReplicaRequest RPCs, would there?  The other requests would increase
> > 10x, but from a pretty low base, right?  We are not reassigning
> > partitions all the time, I hope (or else we have bigger problems...)
> >
>
> I think the controller will group StopReplicaRequest per broker and send
> only one StopReplicaRequest to a broker during controlled shutdown. Anyway,
> we don't have to worry about this if we agree that other requests will
> increase by 10X. One MetadataRequest to send to each broker in the cluster
> every time there is leadership change. I am not sure this is a real
> problem. But in theory this makes the overhead complexity O(number of
> broker) and may be a concern in the future. Ideally we should avoid it.
>
>
> >
> > >
> > > 3) Less efficient use of physical resource on the machine. The number
> of
> > > socket on each machine will increase by 10X. The number of connection
> > > between any two machine will increase by 100X.
> > >
> > > 4) Less efficient way to management memory and quota.
> > >
> > > 5) Rebalance between disks/brokers on the same machine will less
> > > efficient
> > > and less flexible. Broker has to read data from another broker on the
> > > same
> > > machine via socket. It is also harder to do automatic load balance
> > > between
> > > disks on the same machine in the future.
> > >
> > > I will put this and the explanation in the rejected alternative
> section.
> > > I
> > > have a few questions:
> > >
> > > - Can you explain why this solution can help avoid scalability
> > > bottleneck?
> > > I actually think it will exacerbate the scalability problem due the 2)
> > > above.
> > > - Why can we push more RPC with this solution?
> >
> > To really answer this question we'd have to take a deep dive into the
> > locking of the broker and figure out how effectively it can parallelize
> > truly independent requests.  Almost every multithreaded process is going
> > to have shared state, like shared queues or shared sockets, that is
> > going to make scaling less than linear when you add disks or processors.
> >  (And clearly, another option is to improve that scalability, rather
> > than going multi-process!)
> >
>
> Yeah I also think it is better to improve scalability inside kafka code if
> possible. I am not sure we currently have any scalability issue inside
> Kafka that can not be removed without using multi-process.
>
>
> >
> > > - It is true that a garbage collection in one broker would not affect
> > > others. But that is after every broker only uses 1/10 of the memory.
> Can
> > > we be sure that this will actually help performance?
> >
> > The big question is, how much memory do Kafka brokers use now, and how
> 

[jira] [Commented] (KAFKA-4679) Remove unstable markers from Connect APIs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15843816#comment-15843816
 ] 

ASF GitHub Bot commented on KAFKA-4679:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2423


> Remove unstable markers from Connect APIs
> -
>
> Key: KAFKA-4679
> URL: https://issues.apache.org/jira/browse/KAFKA-4679
> Project: Kafka
>  Issue Type: Task
>  Components: KafkaConnect
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
>Priority: Blocker
> Fix For: 0.10.2.0, 0.10.3.0
>
>
> Connect has had a stable API for awhile now and we are careful about 
> compatibility. It's safe to remove the unstable markers now.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #2423: KAFKA-4679 Remove unstable markers from Connect AP...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2423


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-4679) Remove unstable markers from Connect APIs

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava updated KAFKA-4679:
-
   Resolution: Fixed
Fix Version/s: 0.10.3.0
   Status: Resolved  (was: Patch Available)

Issue resolved by pull request 2423
[https://github.com/apache/kafka/pull/2423]

> Remove unstable markers from Connect APIs
> -
>
> Key: KAFKA-4679
> URL: https://issues.apache.org/jira/browse/KAFKA-4679
> Project: Kafka
>  Issue Type: Task
>  Components: KafkaConnect
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
>Priority: Blocker
> Fix For: 0.10.3.0, 0.10.2.0
>
>
> Connect has had a stable API for awhile now and we are careful about 
> compatibility. It's safe to remove the unstable markers now.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-trunk-jdk8 #1231

[Apache Jenkins Server]

See 

Changes:

[wangguoz] MINOR: update JavaDocs for Kafka Streams DSL helpers

[ismael] KAFKA-4635; Client Compatibility follow-ups

[ismael] KAFKA-4450; Add upgrade tests for 0.10.1 and rename TRUNK to DEV_BRANCH

--
[...truncated 8347 lines...]

kafka.utils.timer.TimerTaskListTest > testAll STARTED

kafka.utils.timer.TimerTaskListTest > testAll PASSED

kafka.utils.CommandLineUtilsTest > testParseEmptyArg STARTED

kafka.utils.CommandLineUtilsTest > testParseEmptyArg PASSED

kafka.utils.CommandLineUtilsTest > testParseSingleArg STARTED

kafka.utils.CommandLineUtilsTest > testParseSingleArg PASSED

kafka.utils.CommandLineUtilsTest > testParseArgs STARTED

kafka.utils.CommandLineUtilsTest > testParseArgs PASSED

kafka.utils.CommandLineUtilsTest > testParseEmptyArgAsValid STARTED

kafka.utils.CommandLineUtilsTest > testParseEmptyArgAsValid PASSED

kafka.utils.ReplicationUtilsTest > testUpdateLeaderAndIsr STARTED

kafka.utils.ReplicationUtilsTest > testUpdateLeaderAndIsr PASSED

kafka.utils.ReplicationUtilsTest > testGetLeaderIsrAndEpochForPartition STARTED

kafka.utils.ReplicationUtilsTest > testGetLeaderIsrAndEpochForPartition PASSED

kafka.utils.JsonTest > testJsonEncoding STARTED

kafka.utils.JsonTest > testJsonEncoding PASSED

kafka.utils.SchedulerTest > testMockSchedulerNonPeriodicTask STARTED

kafka.utils.SchedulerTest > testMockSchedulerNonPeriodicTask PASSED

kafka.utils.SchedulerTest > testMockSchedulerPeriodicTask STARTED

kafka.utils.SchedulerTest > testMockSchedulerPeriodicTask PASSED

kafka.utils.SchedulerTest > testNonPeriodicTask STARTED

kafka.utils.SchedulerTest > testNonPeriodicTask PASSED

kafka.utils.SchedulerTest > testRestart STARTED

kafka.utils.SchedulerTest > testRestart PASSED

kafka.utils.SchedulerTest > testReentrantTaskInMockScheduler STARTED

kafka.utils.SchedulerTest > testReentrantTaskInMockScheduler PASSED

kafka.utils.SchedulerTest > testPeriodicTask STARTED

kafka.utils.SchedulerTest > testPeriodicTask PASSED

kafka.utils.ZkUtilsTest > testAbortedConditionalDeletePath STARTED

kafka.utils.ZkUtilsTest > testAbortedConditionalDeletePath PASSED

kafka.utils.ZkUtilsTest > testSuccessfulConditionalDeletePath STARTED

kafka.utils.ZkUtilsTest > testSuccessfulConditionalDeletePath PASSED

kafka.utils.ZkUtilsTest > testClusterIdentifierJsonParsing STARTED

kafka.utils.ZkUtilsTest > testClusterIdentifierJsonParsing PASSED

kafka.utils.IteratorTemplateTest > testIterator STARTED

kafka.utils.IteratorTemplateTest > testIterator PASSED

kafka.utils.UtilsTest > testGenerateUuidAsBase64 STARTED

kafka.utils.UtilsTest > testGenerateUuidAsBase64 PASSED

kafka.utils.UtilsTest > testAbs STARTED

kafka.utils.UtilsTest > testAbs PASSED

kafka.utils.UtilsTest > testReplaceSuffix STARTED

kafka.utils.UtilsTest > testReplaceSuffix PASSED

kafka.utils.UtilsTest > testCircularIterator STARTED

kafka.utils.UtilsTest > testCircularIterator PASSED

kafka.utils.UtilsTest > testReadBytes STARTED

kafka.utils.UtilsTest > testReadBytes PASSED

kafka.utils.UtilsTest > testCsvList STARTED

kafka.utils.UtilsTest > testCsvList PASSED

kafka.utils.UtilsTest > testReadInt STARTED

kafka.utils.UtilsTest > testReadInt PASSED

kafka.utils.UtilsTest > testUrlSafeBase64EncodeUUID STARTED

kafka.utils.UtilsTest > testUrlSafeBase64EncodeUUID PASSED

kafka.utils.UtilsTest > testCsvMap STARTED

kafka.utils.UtilsTest > testCsvMap PASSED

kafka.utils.UtilsTest > testInLock STARTED

kafka.utils.UtilsTest > testInLock PASSED

kafka.utils.UtilsTest > testSwallow STARTED

kafka.utils.UtilsTest > testSwallow PASSED

kafka.producer.AsyncProducerTest > testFailedSendRetryLogic STARTED

kafka.producer.AsyncProducerTest > testFailedSendRetryLogic PASSED

kafka.producer.AsyncProducerTest > testQueueTimeExpired STARTED

kafka.producer.AsyncProducerTest > testQueueTimeExpired PASSED

kafka.producer.AsyncProducerTest > testPartitionAndCollateEvents STARTED

kafka.producer.AsyncProducerTest > testPartitionAndCollateEvents PASSED

kafka.producer.AsyncProducerTest > testBatchSize STARTED

kafka.producer.AsyncProducerTest > testBatchSize PASSED

kafka.producer.AsyncProducerTest > testSerializeEvents STARTED

kafka.producer.AsyncProducerTest > testSerializeEvents PASSED

kafka.producer.AsyncProducerTest > testProducerQueueSize STARTED

kafka.producer.AsyncProducerTest > testProducerQueueSize PASSED

kafka.producer.AsyncProducerTest > testRandomPartitioner STARTED

kafka.producer.AsyncProducerTest > testRandomPartitioner PASSED

kafka.producer.AsyncProducerTest > testInvalidConfiguration STARTED

kafka.producer.AsyncProducerTest > testInvalidConfiguration PASSED

kafka.producer.AsyncProducerTest > testInvalidPartition STARTED

kafka.producer.AsyncProducerTest > testInvalidPartition PASSED

kafka.producer.AsyncProducerTest > testNoBroker STARTED

kafka.producer.AsyncProducerTest > testNoBroker PASSED

kafka.producer.AsyncProducerTest > 

[jira] [Updated] (KAFKA-3733) Avoid long command lines by setting CLASSPATH in environment

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava updated KAFKA-3733:
-
Assignee: Adrian Muraru

> Avoid long command lines by setting CLASSPATH in environment
> 
>
> Key: KAFKA-3733
> URL: https://issues.apache.org/jira/browse/KAFKA-3733
> Project: Kafka
>  Issue Type: Improvement
>  Components: tools
>Reporter: Adrian Muraru
>Assignee: Adrian Muraru
>Priority: Minor
> Fix For: 0.10.2.0
>
>
> {{kafka-run-class.sh}} sets the JVM classpath in the command line via {{-cp}}.
> This generates long command lines that gets trimmed by the shell in commands 
> like ps, pgrep,etc.
> An alternative is to set the CLASSPATH in environment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to normal : kafka-trunk-jdk7 #1893

[Apache Jenkins Server]

See 

[jira] [Updated] (KAFKA-4450) Add missing 0.10.1.x upgrade tests and ensure ongoing compatibility checks

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-4450:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Add missing 0.10.1.x upgrade tests and ensure ongoing compatibility checks
> --
>
> Key: KAFKA-4450
> URL: https://issues.apache.org/jira/browse/KAFKA-4450
> Project: Kafka
>  Issue Type: Bug
>  Components: system tests
>Affects Versions: 0.10.1.0
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
>Priority: Blocker
> Fix For: 0.10.2.0
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> We have upgrade system tests, but we neglected to update them for the most 
> recent released versions (we only have LATEST_0_10_0 but not something from 
> 0_10_1).
> We should probably not only add these versions, but also a) make sure some 
> TRUNK version is always included since upgrade to trunk would always be 
> possible to avoid issues for anyone deploying off trunk (we want every commit 
> to trunk to be solid & compatible) and b) make sure there aren't gaps between 
> versions annotated on the test vs versions that are officially released 
> (which may not be easy statically with the decorators, but might be possible 
> by checking the kafkatest version against previous versions and checking for 
> gaps?).
> Perhaps we need to be able to get the most recent release/snapshot version 
> from the python code so we can always validate previous versions? Even if 
> that's possible, is there going to be a reliable way to get all the previous 
> released versions so we can make sure we have all upgrade tests in place?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4450) Add missing 0.10.1.x upgrade tests and ensure ongoing compatibility checks

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15843756#comment-15843756
 ] 

ASF GitHub Bot commented on KAFKA-4450:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2457


> Add missing 0.10.1.x upgrade tests and ensure ongoing compatibility checks
> --
>
> Key: KAFKA-4450
> URL: https://issues.apache.org/jira/browse/KAFKA-4450
> Project: Kafka
>  Issue Type: Bug
>  Components: system tests
>Affects Versions: 0.10.1.0
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
>Priority: Blocker
> Fix For: 0.10.2.0
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> We have upgrade system tests, but we neglected to update them for the most 
> recent released versions (we only have LATEST_0_10_0 but not something from 
> 0_10_1).
> We should probably not only add these versions, but also a) make sure some 
> TRUNK version is always included since upgrade to trunk would always be 
> possible to avoid issues for anyone deploying off trunk (we want every commit 
> to trunk to be solid & compatible) and b) make sure there aren't gaps between 
> versions annotated on the test vs versions that are officially released 
> (which may not be easy statically with the decorators, but might be possible 
> by checking the kafkatest version against previous versions and checking for 
> gaps?).
> Perhaps we need to be able to get the most recent release/snapshot version 
> from the python code so we can always validate previous versions? Even if 
> that's possible, is there going to be a reliable way to get all the previous 
> released versions so we can make sure we have all upgrade tests in place?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #2457: KAFKA-4450: Add upgrade tests for 0.10.1 releases ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2457


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #2462: MINOR: JavaDoc markup cleanup

[mjsax]

GitHub user mjsax opened a pull request:

https://github.com/apache/kafka/pull/2462

MINOR: JavaDoc markup cleanup



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mjsax/kafka javaDocImprovements9

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2462.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2462


commit 6a7d94ee032bae2d7f8c5f3d3471be3049b16b61
Author: Matthias J. Sax 
Date:   2017-01-28T01:20:36Z

MINOR: JavaDoc markup cleanup




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-4635) Client Compatibility follow-up

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-4635:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Issue resolved by pull request 2414
[https://github.com/apache/kafka/pull/2414]

> Client Compatibility follow-up
> --
>
> Key: KAFKA-4635
> URL: https://issues.apache.org/jira/browse/KAFKA-4635
> Project: Kafka
>  Issue Type: Sub-task
>  Components: clients
>Reporter: Ismael Juma
>Assignee: Colin P. McCabe
> Fix For: 0.10.2.0
>
>
> I collected a number of improvements that I think would be good to do before 
> the release. [~cmccabe], please correct if I got anything wrong and feel free 
> to move some items to separate JIRAs.
> 1. OffsetAndTimestamp is a public class and the javadoc should only include 
> the behaviour that users will see. The following (or part of it) should 
> probably be a non-javadoc comment as it only happens internally:
> "* The timestamp should never be negative, unless it is invalid.  This could 
> happen when handling a response from a broker that doesn't support KIP-79."
> 2. There was a bit of a discussion with regards to the name of the exception 
> that is thrown when a broker is too old. The current name is 
> ObsoleteBrokerException. We should decide on the name and then we should 
> update the relevant producer/consumer methods to mention it.
> 3. [~junrao] suggested that it would be a good idea log when downgrading 
> requests as the behaviour can be a little different. We should decide the 
> right logging level and add this.
> 4. We should have a system test against 0.9.0.1 brokers. We don't support it, 
> but we should ideally give a reasonable error message.
> 5. It seems like `Fetcher.listOffset` could use `retrieveOffsetsByTimes` 
> instead of calling `sendListOffsetRequests` directly. I think that would be a 
> little better, but not sure if others disagree.
> 6. [~hachikuji] suggested that a version mismatch in the `offsetsForTimes` 
> call should result in null entry in map instead of exception for consistency 
> with how we handle the unsupported message format case. I am adding this to 
> make sure we discuss it, but I am not actually sure that is what we should 
> do. Under normal circumstances, the brokers are either too old or not whereas 
> the message format is a topic level configuration and, strictly speaking, 
> independent of the broker version (there is a correlation in practice).
> 7. We log a warning in case of an error while doing an ApiVersions request. 
> Because it is the first request and we retry, the warning in the log is 
> useful. We have a similar warning for Metadata requests, but we only did it 
> for bootstrap brokers. Would it make sense to do the same for ApiVersions?
> 8. It would be good to add a few more tests for the usable versions 
> computation. We have a single simple one at the moment.
> 9. We should add a note to the upgrade notes specifying the change in 
> behaviour with regards to older broker versions.
> cc [~hachikuji].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4635) Client Compatibility follow-up

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15843741#comment-15843741
 ] 

ASF GitHub Bot commented on KAFKA-4635:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2414


> Client Compatibility follow-up
> --
>
> Key: KAFKA-4635
> URL: https://issues.apache.org/jira/browse/KAFKA-4635
> Project: Kafka
>  Issue Type: Sub-task
>  Components: clients
>Reporter: Ismael Juma
>Assignee: Colin P. McCabe
> Fix For: 0.10.2.0
>
>
> I collected a number of improvements that I think would be good to do before 
> the release. [~cmccabe], please correct if I got anything wrong and feel free 
> to move some items to separate JIRAs.
> 1. OffsetAndTimestamp is a public class and the javadoc should only include 
> the behaviour that users will see. The following (or part of it) should 
> probably be a non-javadoc comment as it only happens internally:
> "* The timestamp should never be negative, unless it is invalid.  This could 
> happen when handling a response from a broker that doesn't support KIP-79."
> 2. There was a bit of a discussion with regards to the name of the exception 
> that is thrown when a broker is too old. The current name is 
> ObsoleteBrokerException. We should decide on the name and then we should 
> update the relevant producer/consumer methods to mention it.
> 3. [~junrao] suggested that it would be a good idea log when downgrading 
> requests as the behaviour can be a little different. We should decide the 
> right logging level and add this.
> 4. We should have a system test against 0.9.0.1 brokers. We don't support it, 
> but we should ideally give a reasonable error message.
> 5. It seems like `Fetcher.listOffset` could use `retrieveOffsetsByTimes` 
> instead of calling `sendListOffsetRequests` directly. I think that would be a 
> little better, but not sure if others disagree.
> 6. [~hachikuji] suggested that a version mismatch in the `offsetsForTimes` 
> call should result in null entry in map instead of exception for consistency 
> with how we handle the unsupported message format case. I am adding this to 
> make sure we discuss it, but I am not actually sure that is what we should 
> do. Under normal circumstances, the brokers are either too old or not whereas 
> the message format is a topic level configuration and, strictly speaking, 
> independent of the broker version (there is a correlation in practice).
> 7. We log a warning in case of an error while doing an ApiVersions request. 
> Because it is the first request and we retry, the warning in the log is 
> useful. We have a similar warning for Metadata requests, but we only did it 
> for bootstrap brokers. Would it make sense to do the same for ApiVersions?
> 8. It would be good to add a few more tests for the usable versions 
> computation. We have a single simple one at the moment.
> 9. We should add a note to the upgrade notes specifying the change in 
> behaviour with regards to older broker versions.
> cc [~hachikuji].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #2414: KAFKA-4635: Client Compatibility follow-ups

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2414


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #2459: MINOR: update JavaDocs for Kafka Streams DSL helpe...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2459


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to normal : kafka-0.10.1-jdk7 #114

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

[Jason Gustafson]

A few more responses:


> 101. Compatibility during upgrade: Suppose that the brokers are upgraded to
> the new version, but the broker message format is still the old one. If a
> new producer uses the transaction feature, should the producer get an error
> in this case? A tricky case can be that the leader broker is on the new
> message format, but the follower broker is still on the old message format.
> In this case, the transactional info will be lost in the follower due to
> down conversion. Should we failed the transactional requests when the
> followers are still on the old message format?


We've added some more details to the document about migration. Please take
a look. Two points worth mentioning:

1. Replicas currently take the message format used by the leader. As long
as users do the usual procedure of two rolling bounces, it should be safe
to upgrade the message format.

2. There is no way to support idempotent or transactional features if we
downgrade the message format in the produce request handler. We've modified
the design document to only permit message downgrades if the producer has
disabled idempotence. Otherwise, we will return an
UNSUPPORTED_FOR_MESSAGE_FORMAT error.

110. Transaction log:
> 110.1 "Key => Version AppID Version" It seems that Version should really be
> Type?
> 110.2 "Value => Version Epoch Status ExpirationTime [Topic Partition]"
> Should we store [Topic [Partition]] instead?
> 110.3 To expire an AppId, do we need to insert a tombstone with the expired
> AppID as the key to physically remove the existing AppID entries in the
> transaction log?


Fixed in the document. For 110.3, yes, we need to insert a tombstone after
the AppID has expired. This will work in much the same way as the consumer
coordinator expires offsets using a periodic task.

116. ProducerRequest: The existing format doesn't have "MessageSetSize" at
> the partition level.


This was intentional, but it is easy to overlook. The idea is to modify the
ProduceRequest so that only one message set is included for each partition.
Since the message set contains its own length field, it seemed unnecessary
to have a separate field. The justification for this change was to make the
produce request atomic. With only a single message set for each partition,
either it will be written successfully or not, so an error in the response
will be unambiguous. We are uncertain whether there are legitimate use
cases that require producing smaller message sets in the ProduceRequest, so
we would love to hear feedback on this.

Thanks,
Jason

On Fri, Jan 27, 2017 at 4:21 PM, Apurva Mehta  wrote:

> Hi again Jun,
>
> I have update the document to address your comments below, but including
> the responses inline to make it easier for everyone to stay on top of the
> conversation.
>
>
>
> > 106. Compacted topics.
> > 106.1. When all messages in a transaction are removed, we could remove
> the
> > commit/abort marker for that transaction too. However, we have to be a
> bit
> > careful. If the marker is removed too quickly, it's possible for a
> consumer
> > to see a message in that transaction, but not to see the marker, and
> > therefore will be stuck in that transaction forever. We have a similar
> > issue when dealing with tombstones. The solution is to preserve the
> > tombstone for at least a preconfigured amount of time after the cleaning
> > has passed the tombstone. Then, as long as a consumer can finish reading
> to
> > the cleaning point within the configured amount of time, it's guaranteed
> > not to miss the tombstone after it has seen a non-tombstone message on
> the
> > same key. I am wondering if we should do something similar here.
> >
>
> This is a good point. As we discussed offline, the solution for the removal
> of control messages will be the same as the solution for problem of
> tombstone removal documented in
> https://issues.apache.org/jira/browse/KAFKA-4545.
>
> 106.2. "To address this problem, we propose to preserve the last epoch and
> > sequence number written by each producer for a fixed amount of time as an
> > empty message set. This is allowed by the new message format we are
> > proposing in this document. The time to preserve the sequence number will
> > be governed by the log retention settings. " Could you be a bit more
> > specific on what retention time will be used since by default, there is
> no
> > retention time for compacted (but not delete) topic?
> >
>
> We discussed this offline, and the consensus that it is reasonable to use
> brokers global log.retention.* settings for these messages.
>
>
> > 106.3 "As for control messages, if the broker does not have any
> > corresponding transaction cached with the PID when encountering a control
> > message, that message can be safely removed."
> > Do controlled messages have keys? If not, do we need to relax the
>
> constraint that messages in a compacted topic must have keys?
> >
>
> The key of a control messages is the 

[jira] [Updated] (KAFKA-4327) Move Reset Tool from core to streams

[Matthias J. Sax (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax updated KAFKA-4327:
---
Description: 
This is a follow up on https://issues.apache.org/jira/browse/KAFKA-4008

Currently, Kafka Streams Application Reset Tool is part of {{core}} module due 
to ZK dependency. After KIP-4 got merged, this dependency can be dropped and 
the Reset Tool can be moved to {{streams}} module.

This should also update {{InternalTopicManager#filterExistingTopics}} that 
revers to ResetTool in an exception message:
{{"Use 'kafka.tools.StreamsResetter' tool"}}
-> {{"Use '" + kafka.tools.StreamsResetter.getClass().getName() + "' tool"}}

Doing this JIRA also requires to update the docs with regard to broker backward 
compatibility -- not all broker support "topic delete request" and thus, the 
reset tool will not be backward compatible to all broker versions.

  was:
This is a follow up on https://issues.apache.org/jira/browse/KAFKA-4008

Currently, Kafka Streams Application Reset Tool is part of {{core}} module due 
to ZK dependency. After KIP-4 got merged, this dependency can be dropped and 
the Reset Tool can be moved to {{streams}} module.

This should also update {{InternalTopicManager#filterExistingTopics}} that 
revers to ResetTool in an exception message:
{{"Use 'kafka.tools.StreamsResetter' tool"}}
-> {{"Use '" + kafka.tools.StreamsResetter.getClass().getName() + "' tool"}}


> Move Reset Tool from core to streams
> 
>
> Key: KAFKA-4327
> URL: https://issues.apache.org/jira/browse/KAFKA-4327
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Matthias J. Sax
>Assignee: Matthias J. Sax
>Priority: Minor
>
> This is a follow up on https://issues.apache.org/jira/browse/KAFKA-4008
> Currently, Kafka Streams Application Reset Tool is part of {{core}} module 
> due to ZK dependency. After KIP-4 got merged, this dependency can be dropped 
> and the Reset Tool can be moved to {{streams}} module.
> This should also update {{InternalTopicManager#filterExistingTopics}} that 
> revers to ResetTool in an exception message:
> {{"Use 'kafka.tools.StreamsResetter' tool"}}
> -> {{"Use '" + kafka.tools.StreamsResetter.getClass().getName() + "' tool"}}
> Doing this JIRA also requires to update the docs with regard to broker 
> backward compatibility -- not all broker support "topic delete request" and 
> thus, the reset tool will not be backward compatible to all broker versions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

[Apurva Mehta]

Hi again Jun,

I have update the document to address your comments below, but including
the responses inline to make it easier for everyone to stay on top of the
conversation.



> 106. Compacted topics.
> 106.1. When all messages in a transaction are removed, we could remove the
> commit/abort marker for that transaction too. However, we have to be a bit
> careful. If the marker is removed too quickly, it's possible for a consumer
> to see a message in that transaction, but not to see the marker, and
> therefore will be stuck in that transaction forever. We have a similar
> issue when dealing with tombstones. The solution is to preserve the
> tombstone for at least a preconfigured amount of time after the cleaning
> has passed the tombstone. Then, as long as a consumer can finish reading to
> the cleaning point within the configured amount of time, it's guaranteed
> not to miss the tombstone after it has seen a non-tombstone message on the
> same key. I am wondering if we should do something similar here.
>

This is a good point. As we discussed offline, the solution for the removal
of control messages will be the same as the solution for problem of
tombstone removal documented in
https://issues.apache.org/jira/browse/KAFKA-4545.

106.2. "To address this problem, we propose to preserve the last epoch and
> sequence number written by each producer for a fixed amount of time as an
> empty message set. This is allowed by the new message format we are
> proposing in this document. The time to preserve the sequence number will
> be governed by the log retention settings. " Could you be a bit more
> specific on what retention time will be used since by default, there is no
> retention time for compacted (but not delete) topic?
>

We discussed this offline, and the consensus that it is reasonable to use
brokers global log.retention.* settings for these messages.


> 106.3 "As for control messages, if the broker does not have any
> corresponding transaction cached with the PID when encountering a control
> message, that message can be safely removed."
> Do controlled messages have keys? If not, do we need to relax the

constraint that messages in a compacted topic must have keys?
>

The key of a control messages is the control message type. As such, regular
compaction logic based on key will not apply to control messages. We will
have to update the log cleaner to ignore messages which have the control
message bit set.

Control messages can be removed at some point after the last messages of
the corresponding transaction are removed. As suggested in KAFKA-4545, we
can use the timestamp associated with the log segment to deduce the safe
expiration time for control messages in that segment.



> 112. Control message: Will control messages be used for timestamp indexing?
> If so, what timestamp will we use if the timestamp type is creation time?
>
>
Control messages will not be used for timestamp indexing. Each control
message will have the log append time for the timestamp, but these messages
will be ignored when building the timestamp index. Since control messages
are for system use only and will never be exposed to users, it doesn't make
sense to include them in the timestamp index.

Further, as you mentioned, when a topic uses creation time, it is
impossible to ensure that control messages will not skew the time based
index, since these messages are sent by the transaction coordinator which
has no notion of the application level message creation time.

Thanks,
Apurva

[GitHub] kafka pull request #2461: MINOR: added upgrade and API changes to docs

[mjsax]

GitHub user mjsax opened a pull request:

https://github.com/apache/kafka/pull/2461

MINOR: added upgrade and API changes to docs



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mjsax/kafka addStreamsUpdateSecton

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2461.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2461


commit 51855f6f48954bd5fec03228cf8b2dd0953a2e06
Author: Matthias J. Sax 
Date:   2017-01-27T23:37:52Z

MINOR: added upgrade and API changes to docs




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #2449: KAFKA-4557: Handle Producer.send correctly in expi...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2449


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-4557) ConcurrentModificationException in KafkaProducer event loop

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15843647#comment-15843647
 ] 

ASF GitHub Bot commented on KAFKA-4557:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2449


> ConcurrentModificationException in KafkaProducer event loop
> ---
>
> Key: KAFKA-4557
> URL: https://issues.apache.org/jira/browse/KAFKA-4557
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.10.1.0
>Reporter: Sergey Alaev
>Assignee: Rajini Sivaram
>Priority: Critical
>  Labels: reliability
> Fix For: 0.10.2.0
>
>
> Under heavy load, Kafka producer can stop publishing events. Logs below.
> [2016-12-19T15:01:28.779Z] [sgs] [kafka-producer-network-thread | producer-3] 
> [NetworkClient] [] [] [] [DEBUG]: Disconnecting from node 2 due to 
> request timeout.
> [2016-12-19T15:01:28.793Z] [sgs] [kafka-producer-network-thread | producer-3] 
> [KafkaProducerClient] [] [] [1B2M2Y8Asg] [WARN]: Error sending message 
> to Kafka
> org.apache.kafka.common.errors.NetworkException: The server disconnected 
> before a response was received.
> [2016-12-19T15:01:28.838Z] [sgs] [kafka-producer-network-thread | producer-3] 
> [KafkaProducerClient] [] [] [1B2M2Y8Asg] [WARN]: Error sending message 
> to Kafka
>   org.apache.kafka.common.errors.NetworkException: The server disconnected 
> before a response was received. (#2 from 2016-12-19T15:01:28.793Z)
> 
> [2016-12-19T15:01:28.956Z] [sgs] [kafka-producer-network-thread | producer-3] 
> [KafkaProducerClient] [] [] [1B2M2Y8Asg] [WARN]: Error sending message 
> to Kafka
>   org.apache.kafka.common.errors.TimeoutException: Expiring 46 record(s) for 
> events-deadletter-0 due to 30032 ms has passed since batch creation plus 
> linger time (#285 from 2016-12-19
> T15:01:28.793Z)
> [2016-12-19T15:01:28.956Z] [sgs] [kafka-producer-network-thread | producer-3] 
> [SgsService] [] [] [1B2M2Y8Asg] [WARN]: Error writing signal to Kafka 
> deadletter queue
>   org.apache.kafka.common.errors.TimeoutException: Expiring 46 record(s) for 
> events-deadletter-0 due to 30032 ms has passed since batch creation plus 
> linger time (#286 from 2016-12-19
> T15:01:28.793Z)
> [2016-12-19T15:01:28.960Z] [sgs] [kafka-producer-network-thread | producer-3] 
> [Sender] [] [] [1B2M2Y8Asg] [ERROR]: Uncaught error in kafka producer 
> I/O thread:
> java.util.ConcurrentModificationException: null
> at java.util.ArrayDeque$DeqIterator.next(ArrayDeque.java:643) 
> ~[na:1.8.0_45]
> at 
> org.apache.kafka.clients.producer.internals.RecordAccumulator.abortExpiredBatches(RecordAccumulator.java:242)
>  ~[kafka-clients-0.10.1.0.jar:na]
> at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:212) 
> ~[kafka-clients-0.10.1.0.jar:na]
> at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:135) 
> ~[kafka-clients-0.10.1.0.jar:na]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> [2016-12-19T15:01:28.981Z] [sgs] [kafka-producer-network-thread | producer-3] 
> [NetworkClient] [] [] [1B2M2Y8Asg] [WARN]: Error while fetching 
> metadata with correlation id 28711 : {events-deadletter=LEADER_NOT_AVAILABLE}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-4557) ConcurrentModificationException in KafkaProducer event loop

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-4557.

Resolution: Fixed

Issue resolved by pull request 2449
[https://github.com/apache/kafka/pull/2449]

> ConcurrentModificationException in KafkaProducer event loop
> ---
>
> Key: KAFKA-4557
> URL: https://issues.apache.org/jira/browse/KAFKA-4557
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.10.1.0
>Reporter: Sergey Alaev
>Assignee: Rajini Sivaram
>Priority: Critical
>  Labels: reliability
> Fix For: 0.10.2.0
>
>
> Under heavy load, Kafka producer can stop publishing events. Logs below.
> [2016-12-19T15:01:28.779Z] [sgs] [kafka-producer-network-thread | producer-3] 
> [NetworkClient] [] [] [] [DEBUG]: Disconnecting from node 2 due to 
> request timeout.
> [2016-12-19T15:01:28.793Z] [sgs] [kafka-producer-network-thread | producer-3] 
> [KafkaProducerClient] [] [] [1B2M2Y8Asg] [WARN]: Error sending message 
> to Kafka
> org.apache.kafka.common.errors.NetworkException: The server disconnected 
> before a response was received.
> [2016-12-19T15:01:28.838Z] [sgs] [kafka-producer-network-thread | producer-3] 
> [KafkaProducerClient] [] [] [1B2M2Y8Asg] [WARN]: Error sending message 
> to Kafka
>   org.apache.kafka.common.errors.NetworkException: The server disconnected 
> before a response was received. (#2 from 2016-12-19T15:01:28.793Z)
> 
> [2016-12-19T15:01:28.956Z] [sgs] [kafka-producer-network-thread | producer-3] 
> [KafkaProducerClient] [] [] [1B2M2Y8Asg] [WARN]: Error sending message 
> to Kafka
>   org.apache.kafka.common.errors.TimeoutException: Expiring 46 record(s) for 
> events-deadletter-0 due to 30032 ms has passed since batch creation plus 
> linger time (#285 from 2016-12-19
> T15:01:28.793Z)
> [2016-12-19T15:01:28.956Z] [sgs] [kafka-producer-network-thread | producer-3] 
> [SgsService] [] [] [1B2M2Y8Asg] [WARN]: Error writing signal to Kafka 
> deadletter queue
>   org.apache.kafka.common.errors.TimeoutException: Expiring 46 record(s) for 
> events-deadletter-0 due to 30032 ms has passed since batch creation plus 
> linger time (#286 from 2016-12-19
> T15:01:28.793Z)
> [2016-12-19T15:01:28.960Z] [sgs] [kafka-producer-network-thread | producer-3] 
> [Sender] [] [] [1B2M2Y8Asg] [ERROR]: Uncaught error in kafka producer 
> I/O thread:
> java.util.ConcurrentModificationException: null
> at java.util.ArrayDeque$DeqIterator.next(ArrayDeque.java:643) 
> ~[na:1.8.0_45]
> at 
> org.apache.kafka.clients.producer.internals.RecordAccumulator.abortExpiredBatches(RecordAccumulator.java:242)
>  ~[kafka-clients-0.10.1.0.jar:na]
> at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:212) 
> ~[kafka-clients-0.10.1.0.jar:na]
> at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:135) 
> ~[kafka-clients-0.10.1.0.jar:na]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> [2016-12-19T15:01:28.981Z] [sgs] [kafka-producer-network-thread | producer-3] 
> [NetworkClient] [] [] [1B2M2Y8Asg] [WARN]: Error while fetching 
> metadata with correlation id 28711 : {events-deadletter=LEADER_NOT_AVAILABLE}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to normal : kafka-0.10.2-jdk7 #50

[Apache Jenkins Server]

See 

[jira] [Commented] (KAFKA-4277) creating ephemeral node already exist

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15843636#comment-15843636
 ] 

Jun Rao commented on KAFKA-4277:


According to the following, it doesn't seem this can happen in ZK.

http://mail-archives.apache.org/mod_mbox/zookeeper-user/201701.mbox/%3CB512F6DE-C0BF-45CE-8102-6F242988268E%40apache.org%3E

> creating ephemeral node already exist
> -
>
> Key: KAFKA-4277
> URL: https://issues.apache.org/jira/browse/KAFKA-4277
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.0.0
>Reporter: Feixiang Yan
>
> I use zookeeper 3.4.6.
> Zookeeper session time out, zkClient try reconnect failed. Then re-establish 
> the session and re-registering broker info in ZK, throws NODEEXISTS Exception.
>  I think it is because the ephemeral node which created by old session has 
> not removed. 
> I read the 
> [ZkUtils.scala|https://github.com/apache/kafka/blob/0.8.1/core/src/main/scala/kafka/utils/ZkUtils.scala]
>  of 0.8.1, createEphemeralPathExpectConflictHandleZKBug try create node in a 
> while loop until create success. This can solve the issue. But in 
> [ZkUtils.scala|https://github.com/apache/kafka/blob/0.10.0.1/core/src/main/scala/kafka/utils/ZkUtils.scala]
>   0.10.1 the function removed.
> {noformat}
> [2016-10-07 19:00:32,562] INFO Socket connection established to 
> 10.191.155.238/10.191.155.238:21819, initiating session 
> (org.apache.zookeeper.ClientCnxn)
> [2016-10-07 19:00:32,563] INFO zookeeper state changed (Expired) 
> (org.I0Itec.zkclient.ZkClient)
> [2016-10-07 19:00:32,564] INFO Unable to reconnect to ZooKeeper service, 
> session 0x1576b11f9b201bd has expired, closing socket connection 
> (org.apache.zookeeper.ClientCnxn)
> [2016-10-07 19:00:32,564] INFO Initiating client connection, 
> connectString=10.191.155.237:21819,10.191.155.238:21819,10.191.155.239:21819/cluster2
>  sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@ae71be2 
> (org.apache.zookeeper.ZooKeeper)
> [2016-10-07 19:00:32,566] INFO Opening socket connection to server 
> 10.191.155.237/10.191.155.237:21819. Will not attempt to authenticate using 
> SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
> [2016-10-07 19:00:32,566] INFO Socket connection established to 
> 10.191.155.237/10.191.155.237:21819, initiating session 
> (org.apache.zookeeper.ClientCnxn)
> [2016-10-07 19:00:32,566] INFO EventThread shut down 
> (org.apache.zookeeper.ClientCnxn)
> [2016-10-07 19:00:32,567] INFO Session establishment complete on server 
> 10.191.155.237/10.191.155.237:21819, sessionid = 0x1579ecd39c20006, 
> negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
> [2016-10-07 19:00:32,567] INFO zookeeper state changed (SyncConnected) 
> (org.I0Itec.zkclient.ZkClient)
> [2016-10-07 19:00:32,608] INFO re-registering broker info in ZK for broker 3 
> (kafka.server.KafkaHealthcheck$SessionExpireListener)
> [2016-10-07 19:00:32,610] INFO Creating /brokers/ids/3 (is it secure? false) 
> (kafka.utils.ZKCheckedEphemeral)
> [2016-10-07 19:00:32,611] INFO Result of znode creation is: NODEEXISTS 
> (kafka.utils.ZKCheckedEphemeral)
> [2016-10-07 19:00:32,614] ERROR Error handling event ZkEvent[New session 
> event sent to kafka.server.KafkaHealthcheck$SessionExpireListener@324f1bc] 
> (org.I0Itec.zkclient.ZkEventThread)
> java.lang.RuntimeException: A broker is already registered on the path 
> /brokers/ids/3. This probably indicates that you either have configured a 
> brokerid that is already in use, or else you have shutdown this broker and 
> restarted it faster than the zookeeper timeout so it appears to be 
> re-registering.
> at kafka.utils.ZkUtils.registerBrokerInZk(ZkUtils.scala:305)
> at kafka.utils.ZkUtils.registerBrokerInZk(ZkUtils.scala:291)
> at kafka.server.KafkaHealthcheck.register(KafkaHealthcheck.scala:70)
> at 
> kafka.server.KafkaHealthcheck$SessionExpireListener.handleNewSession(KafkaHealthcheck.scala:104)
> at org.I0Itec.zkclient.ZkClient$6.run(ZkClient.java:735)
> at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-trunk-jdk7 #1892

[Apache Jenkins Server]

See 

Changes:

[ismael] KAFKA-4664; Update docs/protocol.html with KIP-97 information

[me] MINOR: Update copyright year in the NOTICE file.

--
[...truncated 63 lines...]
:kafka-trunk-jdk7:clients:processTestResources
:kafka-trunk-jdk7:clients:testClasses
:kafka-trunk-jdk7:core:compileJava UP-TO-DATE
:kafka-trunk-jdk7:core:compileScala
:79:
 value DEFAULT_TIMESTAMP in object OffsetCommitRequest is deprecated: see 
corresponding Javadoc for more information.

org.apache.kafka.common.requests.OffsetCommitRequest.DEFAULT_TIMESTAMP
 ^
:36:
 value DEFAULT_TIMESTAMP in object OffsetCommitRequest is deprecated: see 
corresponding Javadoc for more information.
 commitTimestamp: Long = 
org.apache.kafka.common.requests.OffsetCommitRequest.DEFAULT_TIMESTAMP,

  ^
:37:
 value DEFAULT_TIMESTAMP in object OffsetCommitRequest is deprecated: see 
corresponding Javadoc for more information.
 expireTimestamp: Long = 
org.apache.kafka.common.requests.OffsetCommitRequest.DEFAULT_TIMESTAMP) {

  ^
:501:
 value DEFAULT_TIMESTAMP in object OffsetCommitRequest is deprecated: see 
corresponding Javadoc for more information.
  if (offsetAndMetadata.expireTimestamp == 
org.apache.kafka.common.requests.OffsetCommitRequest.DEFAULT_TIMESTAMP)

^
:323:
 value DEFAULT_TIMESTAMP in object OffsetCommitRequest is deprecated: see 
corresponding Javadoc for more information.
  if (partitionData.timestamp == 
OffsetCommitRequest.DEFAULT_TIMESTAMP)
 ^
:94:
 class ProducerConfig in package producer is deprecated: This class has been 
deprecated and will be removed in a future release. Please use 
org.apache.kafka.clients.producer.ProducerConfig instead.
val producerConfig = new ProducerConfig(props)
 ^
:95:
 method fetchTopicMetadata in object ClientUtils is deprecated: This method has 
been deprecated and will be removed in a future release.
fetchTopicMetadata(topics, brokers, producerConfig, correlationId)
^
:187:
 object ProducerRequestStatsRegistry in package producer is deprecated: This 
object has been deprecated and will be removed in a future release.
ProducerRequestStatsRegistry.removeProducerRequestStats(clientId)
^
:129:
 method readFromReadableChannel in class NetworkReceive is deprecated: see 
corresponding Javadoc for more information.
  response.readFromReadableChannel(channel)
   ^
:323:
 value timestamp in class PartitionData is deprecated: see corresponding 
Javadoc for more information.
  if (partitionData.timestamp == 
OffsetCommitRequest.DEFAULT_TIMESTAMP)
^
:326:
 value timestamp in class PartitionData is deprecated: see corresponding 
Javadoc for more information.
offsetRetention + partitionData.timestamp
^
:578:
 method offsetData in class ListOffsetRequest is deprecated: see corresponding 
Javadoc for more information.
val (authorizedRequestInfo, unauthorizedRequestInfo) = 
offsetRequest.offsetData.asScala.partition {
 ^

[jira] [Assigned] (KAFKA-4220) Clean up & provide better error message when incorrect argument types are provided in the command line client

[Ishita Mandhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ishita Mandhan reassigned KAFKA-4220:
-

Assignee: Vahid Hashemian  (was: Ishita Mandhan)

> Clean up & provide better error message when incorrect argument types are 
> provided in the command line client
> -
>
> Key: KAFKA-4220
> URL: https://issues.apache.org/jira/browse/KAFKA-4220
> Project: Kafka
>  Issue Type: Bug
>Reporter: Ishita Mandhan
>Assignee: Vahid Hashemian
>Priority: Minor
>
> When the argument provided to a command line statement is not of the right 
> type, a stack trace is returned. This can be replaced by a cleaner error 
> message that is earlier to read & understand for the user.
> For example-
> bin/kafka-console-consumer.sh --new-consumer --bootstrap-server 
> localhost:9092 --topic foo --timeout-ms abc
> 'abc' is an incorrect type for the --timeout-ms parameter, which expects a 
> number.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (KAFKA-2857) ConsumerGroupCommand throws GroupCoordinatorNotAvailableException when describing a non-existent group before the offset topic is created

[Ishita Mandhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ishita Mandhan reassigned KAFKA-2857:
-

Assignee: Vahid Hashemian  (was: Ishita Mandhan)

> ConsumerGroupCommand throws GroupCoordinatorNotAvailableException when 
> describing a non-existent group before the offset topic is created
> -
>
> Key: KAFKA-2857
> URL: https://issues.apache.org/jira/browse/KAFKA-2857
> Project: Kafka
>  Issue Type: Bug
>  Components: tools
>Reporter: Ismael Juma
>Assignee: Vahid Hashemian
>Priority: Minor
>
> If we describe a non-existing group before the offset topic is created, like 
> the following:
> {code}
> bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --new-consumer 
> --describe --group 
> {code}
> We get the following error:
> {code}
> Error while executing consumer group command The group coordinator is not 
> available.
> org.apache.kafka.common.errors.GroupCoordinatorNotAvailableException: The 
> group coordinator is not available.
> {code}
> The exception is thrown in the `adminClient.describeConsumerGroup` call. We 
> can't interpret this exception as meaning that the group doesn't exist 
> because it could also be thrown f all replicas for a offset topic partition 
> are down (as explained by Jun).
> Jun also suggested that we should distinguish if a coordinator is not 
> available from the case where a coordinator doesn't exist.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #2460: MINOR: Update copyright year in the NOTICE file.

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2460


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #1787: KAFKA-3940 Log should check the return value of di...

[imandhan]

Github user imandhan closed the pull request at:

https://github.com/apache/kafka/pull/1787


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #2203: KAFKA-2857 ConsumerGroupCommand throws GroupCoordi...

[imandhan]

Github user imandhan closed the pull request at:

https://github.com/apache/kafka/pull/2203


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3940) Log should check the return value of dir.mkdirs()

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15843581#comment-15843581
 ] 

ASF GitHub Bot commented on KAFKA-3940:
---

Github user imandhan closed the pull request at:

https://github.com/apache/kafka/pull/1787


> Log should check the return value of dir.mkdirs()
> -
>
> Key: KAFKA-3940
> URL: https://issues.apache.org/jira/browse/KAFKA-3940
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.10.0.0
>Reporter: Jun Rao
>Assignee: Ishita Mandhan
>  Labels: newbie, reliability
>
> In Log.loadSegments(), we call dir.mkdirs() w/o checking the return value and 
> just assume the directory will exist after the call. However, if the 
> directory can't be created (e.g. due to no space), we will hit 
> NullPointerException in the next statement, which will be confusing.
>for(file <- dir.listFiles if file.isFile) {



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2857) ConsumerGroupCommand throws GroupCoordinatorNotAvailableException when describing a non-existent group before the offset topic is created

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15843579#comment-15843579
 ] 

ASF GitHub Bot commented on KAFKA-2857:
---

Github user imandhan closed the pull request at:

https://github.com/apache/kafka/pull/2203


> ConsumerGroupCommand throws GroupCoordinatorNotAvailableException when 
> describing a non-existent group before the offset topic is created
> -
>
> Key: KAFKA-2857
> URL: https://issues.apache.org/jira/browse/KAFKA-2857
> Project: Kafka
>  Issue Type: Bug
>  Components: tools
>Reporter: Ismael Juma
>Assignee: Ishita Mandhan
>Priority: Minor
>
> If we describe a non-existing group before the offset topic is created, like 
> the following:
> {code}
> bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --new-consumer 
> --describe --group 
> {code}
> We get the following error:
> {code}
> Error while executing consumer group command The group coordinator is not 
> available.
> org.apache.kafka.common.errors.GroupCoordinatorNotAvailableException: The 
> group coordinator is not available.
> {code}
> The exception is thrown in the `adminClient.describeConsumerGroup` call. We 
> can't interpret this exception as meaning that the group doesn't exist 
> because it could also be thrown f all replicas for a offset topic partition 
> are down (as explained by Jun).
> Jun also suggested that we should distinguish if a coordinator is not 
> available from the case where a coordinator doesn't exist.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #2288: MINOR: Create Hyperlinks in protocol api docs

[imandhan]

Github user imandhan closed the pull request at:

https://github.com/apache/kafka/pull/2288


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-4664) Update docs/protocol.html with KIP-97 information

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15843531#comment-15843531
 ] 

ASF GitHub Bot commented on KAFKA-4664:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2387


> Update docs/protocol.html with KIP-97 information
> -
>
> Key: KAFKA-4664
> URL: https://issues.apache.org/jira/browse/KAFKA-4664
> Project: Kafka
>  Issue Type: Sub-task
>  Components: documentation
>Reporter: Colin P. McCabe
>Assignee: Colin P. McCabe
>Priority: Minor
> Fix For: 0.10.2.0
>
>
> Update docs/protocol.html with KIP-97 information



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4664) Update docs/protocol.html with KIP-97 information

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-4664:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Issue resolved by pull request 2387
[https://github.com/apache/kafka/pull/2387]

> Update docs/protocol.html with KIP-97 information
> -
>
> Key: KAFKA-4664
> URL: https://issues.apache.org/jira/browse/KAFKA-4664
> Project: Kafka
>  Issue Type: Sub-task
>  Components: documentation
>Reporter: Colin P. McCabe
>Assignee: Colin P. McCabe
>Priority: Minor
> Fix For: 0.10.2.0
>
>
> Update docs/protocol.html with KIP-97 information



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #2387: KAFKA-4664: Update docs/protocol.html with KIP-97 ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2387


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: kafka-0.10.2-jdk7 #49

[Apache Jenkins Server]

See 

Changes:

[ismael] KAFKA-4704; Coordinator cache loading fails if groupId is reused for

--
[...truncated 4080 lines...]
kafka.server.KafkaConfigTest > testUncleanLeaderElectionDefault STARTED

kafka.server.KafkaConfigTest > testUncleanLeaderElectionDefault PASSED

kafka.server.KafkaConfigTest > testInvalidAdvertisedListenersProtocol STARTED

kafka.server.KafkaConfigTest > testInvalidAdvertisedListenersProtocol PASSED

kafka.server.KafkaConfigTest > testUncleanElectionEnabled STARTED

kafka.server.KafkaConfigTest > testUncleanElectionEnabled PASSED

kafka.server.KafkaConfigTest > testAdvertisePortDefault STARTED

kafka.server.KafkaConfigTest > testAdvertisePortDefault PASSED

kafka.server.KafkaConfigTest > testVersionConfiguration STARTED

kafka.server.KafkaConfigTest > testVersionConfiguration PASSED

kafka.server.KafkaConfigTest > testEqualAdvertisedListenersProtocol STARTED

kafka.server.KafkaConfigTest > testEqualAdvertisedListenersProtocol PASSED

kafka.server.SslReplicaFetchTest > testReplicaFetcherThread STARTED

kafka.server.SslReplicaFetchTest > testReplicaFetcherThread PASSED

kafka.server.IsrExpirationTest > testIsrExpirationForSlowFollowers STARTED

kafka.server.IsrExpirationTest > testIsrExpirationForSlowFollowers PASSED

kafka.server.IsrExpirationTest > testIsrExpirationForStuckFollowers STARTED

kafka.server.IsrExpirationTest > testIsrExpirationForStuckFollowers PASSED

kafka.server.IsrExpirationTest > testIsrExpirationIfNoFetchRequestMade STARTED

kafka.server.IsrExpirationTest > testIsrExpirationIfNoFetchRequestMade PASSED

kafka.server.SaslPlaintextReplicaFetchTest > testReplicaFetcherThread STARTED

kafka.server.SaslPlaintextReplicaFetchTest > testReplicaFetcherThread PASSED

kafka.server.ReplicationQuotasTest > 
shouldBootstrapTwoBrokersWithLeaderThrottle STARTED

kafka.server.ReplicationQuotasTest > 
shouldBootstrapTwoBrokersWithLeaderThrottle PASSED

kafka.server.ReplicationQuotasTest > shouldThrottleOldSegments STARTED

kafka.server.ReplicationQuotasTest > shouldThrottleOldSegments PASSED

kafka.server.ReplicationQuotasTest > 
shouldBootstrapTwoBrokersWithFollowerThrottle STARTED

kafka.server.ReplicationQuotasTest > 
shouldBootstrapTwoBrokersWithFollowerThrottle PASSED

kafka.server.ServerStartupTest > testBrokerStateRunningAfterZK STARTED

kafka.server.ServerStartupTest > testBrokerStateRunningAfterZK PASSED

kafka.server.ServerStartupTest > testBrokerCreatesZKChroot STARTED

kafka.server.ServerStartupTest > testBrokerCreatesZKChroot PASSED

kafka.server.ServerStartupTest > testConflictBrokerStartupWithSamePort STARTED

kafka.server.ServerStartupTest > testConflictBrokerStartupWithSamePort PASSED

kafka.server.ServerStartupTest > testConflictBrokerRegistration STARTED

kafka.server.ServerStartupTest > testConflictBrokerRegistration PASSED

kafka.server.ServerStartupTest > testBrokerSelfAware STARTED

kafka.server.ServerStartupTest > testBrokerSelfAware PASSED

kafka.server.ProduceRequestTest > testSimpleProduceRequest STARTED

kafka.server.ProduceRequestTest > testSimpleProduceRequest PASSED

kafka.server.ProduceRequestTest > testCorruptLz4ProduceRequest STARTED

kafka.server.ProduceRequestTest > testCorruptLz4ProduceRequest PASSED

kafka.server.ReplicaManagerTest > testHighWaterMarkDirectoryMapping STARTED

kafka.server.ReplicaManagerTest > testHighWaterMarkDirectoryMapping PASSED

kafka.server.ReplicaManagerTest > 
testFetchBeyondHighWatermarkReturnEmptyResponse STARTED

kafka.server.ReplicaManagerTest > 
testFetchBeyondHighWatermarkReturnEmptyResponse PASSED

kafka.server.ReplicaManagerTest > testIllegalRequiredAcks STARTED

kafka.server.ReplicaManagerTest > testIllegalRequiredAcks PASSED

kafka.server.ReplicaManagerTest > testClearPurgatoryOnBecomingFollower STARTED

kafka.server.ReplicaManagerTest > testClearPurgatoryOnBecomingFollower PASSED

kafka.server.ReplicaManagerTest > testHighwaterMarkRelativeDirectoryMapping 
STARTED

kafka.server.ReplicaManagerTest > testHighwaterMarkRelativeDirectoryMapping 
PASSED

kafka.server.KafkaMetricReporterClusterIdTest > testClusterIdPresent STARTED

kafka.server.KafkaMetricReporterClusterIdTest > testClusterIdPresent PASSED

kafka.server.CreateTopicsRequestWithPolicyTest > testValidCreateTopicsRequests 
STARTED

kafka.server.CreateTopicsRequestWithPolicyTest > testValidCreateTopicsRequests 
PASSED

kafka.server.CreateTopicsRequestWithPolicyTest > testErrorCreateTopicsRequests 
STARTED

kafka.server.CreateTopicsRequestWithPolicyTest > testErrorCreateTopicsRequests 
PASSED

kafka.server.OffsetCommitTest > testUpdateOffsets STARTED

kafka.server.OffsetCommitTest > testUpdateOffsets PASSED

kafka.server.OffsetCommitTest > testLargeMetadataPayload STARTED

kafka.server.OffsetCommitTest > testLargeMetadataPayload PASSED

kafka.server.OffsetCommitTest > testOffsetsDeleteAfterTopicDeletion STARTED

kafka.server.OffsetCommitTest > 

Jenkins build is back to normal : kafka-trunk-jdk8 #1228

[Apache Jenkins Server]

See 

[GitHub] kafka pull request #2460: MINOR: Update copyright year in the NOTICE file.

[ewencp]

GitHub user ewencp opened a pull request:

https://github.com/apache/kafka/pull/2460

MINOR: Update copyright year in the NOTICE file.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ewencp/kafka minor-update-notice-year

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2460.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2460


commit c69567fe83fabee7bc8db3eb8b0bef54172597b3
Author: Ewen Cheslack-Postava 
Date:   2017-01-27T21:10:03Z

MINOR: Update copyright year in the NOTICE file.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-4708) Transient failure in BrokerApiVersionsCommandTest.checkBrokerApiVersionCommandOutput

[Colin P. McCabe (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15843491#comment-15843491
 ] 

Colin P. McCabe commented on KAFKA-4708:


Hmm.  I ran this unit test locally in a loop for an hour and didn't see any 
failures.

I'm not sure how the extra blank line got in there in Jenkins.  I suppose one 
possibility is that the broker is not up when the command is run... although 
that's confusing too, since the command should wait until it can get the 
cluster metadata...

> Transient failure in 
> BrokerApiVersionsCommandTest.checkBrokerApiVersionCommandOutput
> 
>
> Key: KAFKA-4708
> URL: https://issues.apache.org/jira/browse/KAFKA-4708
> Project: Kafka
>  Issue Type: Bug
>  Components: unit tests
>Reporter: Jason Gustafson
>Assignee: Colin P. McCabe
> Fix For: 0.10.2.0
>
>
> {code}
> kafka.admin.BrokerApiVersionsCommandTest > checkBrokerApiVersionCommandOutput 
> FAILED
> org.junit.ComparisonFailure: expected:<[localhost:34091 (id: 0 rack: 
> null) -> (]> but was:<[]>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at 
> kafka.admin.BrokerApiVersionsCommandTest.checkBrokerApiVersionCommandOutput(BrokerApiVersionsCommandTest.scala:44)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-4704) Group coordinator cache loading fails if groupId is used first for consumer groups and then for simple consumer

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-4704.

Resolution: Fixed

Issue resolved by pull request 2455
[https://github.com/apache/kafka/pull/2455]

> Group coordinator cache loading fails if groupId is used first for consumer 
> groups and then for simple consumer
> ---
>
> Key: KAFKA-4704
> URL: https://issues.apache.org/jira/browse/KAFKA-4704
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.0.0, 0.10.0.1, 0.10.1.0, 0.10.1.1
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
> Fix For: 0.10.2.0
>
>
> When all the members in a consumer group have died and all of its offsets 
> have expired, we write a tombstone to __consumer_offsets so that its group 
> metadata is cleaned up. It is possible that after this happens, the same 
> groupId is then used only for offset storage (i.e. by "simple" consumers). 
> Our current cache loading logic, which is triggered when a coordinator first 
> takes over control of a partition, does not account for this scenario and 
> would currently fail.
> This is probably an unlikely scenario to hit in practice, but it reveals the 
> lack of test coverage around the cache loading logic. We should improve this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4704) Group coordinator cache loading fails if groupId is used first for consumer groups and then for simple consumer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15843451#comment-15843451
 ] 

ASF GitHub Bot commented on KAFKA-4704:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2455


> Group coordinator cache loading fails if groupId is used first for consumer 
> groups and then for simple consumer
> ---
>
> Key: KAFKA-4704
> URL: https://issues.apache.org/jira/browse/KAFKA-4704
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.0.0, 0.10.0.1, 0.10.1.0, 0.10.1.1
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
> Fix For: 0.10.2.0
>
>
> When all the members in a consumer group have died and all of its offsets 
> have expired, we write a tombstone to __consumer_offsets so that its group 
> metadata is cleaned up. It is possible that after this happens, the same 
> groupId is then used only for offset storage (i.e. by "simple" consumers). 
> Our current cache loading logic, which is triggered when a coordinator first 
> takes over control of a partition, does not account for this scenario and 
> would currently fail.
> This is probably an unlikely scenario to hit in practice, but it reveals the 
> lack of test coverage around the cache loading logic. We should improve this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #2455: KAFKA-4704: Coordinator cache loading fails if gro...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2455


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-4704) Group coordinator cache loading fails if groupId is used first for consumer groups and then for simple consumer

[Jason Gustafson (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15843389#comment-15843389
 ] 

Jason Gustafson commented on KAFKA-4704:


[~onurkaraman] Interesting point. Unfortunately, this patch won't fix that 
problem. The only thing I can think of would be to skip the generation check 
when the group is empty. Seems a little dangerous, but possibly justifiable. I 
suggest we address this as part of the consumer migration work itself.

> Group coordinator cache loading fails if groupId is used first for consumer 
> groups and then for simple consumer
> ---
>
> Key: KAFKA-4704
> URL: https://issues.apache.org/jira/browse/KAFKA-4704
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.0.0, 0.10.0.1, 0.10.1.0, 0.10.1.1
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
> Fix For: 0.10.2.0
>
>
> When all the members in a consumer group have died and all of its offsets 
> have expired, we write a tombstone to __consumer_offsets so that its group 
> metadata is cleaned up. It is possible that after this happens, the same 
> groupId is then used only for offset storage (i.e. by "simple" consumers). 
> Our current cache loading logic, which is triggered when a coordinator first 
> takes over control of a partition, does not account for this scenario and 
> would currently fail.
> This is probably an unlikely scenario to hit in practice, but it reveals the 
> lack of test coverage around the cache loading logic. We should improve this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-4708) Transient failure in BrokerApiVersionsCommandTest.checkBrokerApiVersionCommandOutput

[Jason Gustafson (JIRA)]

Jason Gustafson created KAFKA-4708:
--

 Summary: Transient failure in 
BrokerApiVersionsCommandTest.checkBrokerApiVersionCommandOutput
 Key: KAFKA-4708
 URL: https://issues.apache.org/jira/browse/KAFKA-4708
 Project: Kafka
  Issue Type: Bug
  Components: unit tests
Reporter: Jason Gustafson
Assignee: Colin P. McCabe
 Fix For: 0.10.2.0


{code}
kafka.admin.BrokerApiVersionsCommandTest > checkBrokerApiVersionCommandOutput 
FAILED
org.junit.ComparisonFailure: expected:<[localhost:34091 (id: 0 rack: null) 
-> (]> but was:<[]>
at org.junit.Assert.assertEquals(Assert.java:115)
at org.junit.Assert.assertEquals(Assert.java:144)
at 
kafka.admin.BrokerApiVersionsCommandTest.checkBrokerApiVersionCommandOutput(BrokerApiVersionsCommandTest.scala:44)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-4707) Transient failure FetchRequestTest.testBrokerRespectsPartitionsOrderAndSizeLimits

[Jason Gustafson (JIRA)]

Jason Gustafson created KAFKA-4707:
--

 Summary: Transient failure 
FetchRequestTest.testBrokerRespectsPartitionsOrderAndSizeLimits
 Key: KAFKA-4707
 URL: https://issues.apache.org/jira/browse/KAFKA-4707
 Project: Kafka
  Issue Type: Bug
  Components: unit tests
Affects Versions: 0.10.2.0
Reporter: Jason Gustafson


{code}
kafka.server.FetchRequestTest > testBrokerRespectsPartitionsOrderAndSizeLimits 
FAILED
java.util.concurrent.ExecutionException: 
org.apache.kafka.common.errors.NotEnoughReplicasException: Messages are 
rejected since there are fewer in-sync replicas than required.
at 
org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:70)
at 
org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:57)
at 
org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:25)
at 
kafka.server.FetchRequestTest$$anonfun$produceData$2.apply(FetchRequestTest.scala:231)
at 
kafka.server.FetchRequestTest$$anonfun$produceData$2.apply(FetchRequestTest.scala:231)
at 
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at kafka.server.FetchRequestTest.produceData(FetchRequestTest.scala:231)
at 
kafka.server.FetchRequestTest.testBrokerRespectsPartitionsOrderAndSizeLimits(FetchRequestTest.scala:88)

Caused by:
org.apache.kafka.common.errors.NotEnoughReplicasException: Messages are 
rejected since there are fewer in-sync replicas than required.
{code}

Here: https://builds.apache.org/job/kafka-trunk-jdk8/1224/console, and here: 
https://builds.apache.org/job/kafka-trunk-jdk7/1887/console



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-trunk-jdk8 #1227

[Apache Jenkins Server]

See 

Changes:

[wangguoz] MINOR: Update KTable JavaDoc

[wangguoz] KAFKA-4644: Improve test coverage of StreamsPartitionAssignor

--
[...truncated 8319 lines...]

kafka.integration.SaslPlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithCollision STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithCollision PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokerListWithNoTopics STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokerListWithNoTopics PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testAutoCreateTopic STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > testAutoCreateTopic PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testGetAllTopicMetadata 
STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > testGetAllTopicMetadata 
PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testBasicTopicMetadata 
STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > testBasicTopicMetadata PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED

kafka.integration.PrimitiveApiTest > testMultiProduce STARTED

kafka.integration.PrimitiveApiTest > testMultiProduce PASSED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch STARTED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch PASSED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize 
STARTED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize PASSED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests STARTED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests PASSED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch STARTED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch PASSED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression STARTED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression PASSED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic STARTED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic PASSED

kafka.integration.PrimitiveApiTest > testEmptyFetchRequest STARTED

kafka.integration.PrimitiveApiTest > testEmptyFetchRequest PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack PASSED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopicWithCollision 
STARTED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopicWithCollision 
PASSED

kafka.integration.PlaintextTopicMetadataTest > testAliveBrokerListWithNoTopics 
STARTED

kafka.integration.PlaintextTopicMetadataTest > testAliveBrokerListWithNoTopics 
PASSED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopic STARTED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopic PASSED

kafka.integration.PlaintextTopicMetadataTest > testGetAllTopicMetadata STARTED

kafka.integration.PlaintextTopicMetadataTest > testGetAllTopicMetadata PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.integration.PlaintextTopicMetadataTest > testBasicTopicMetadata STARTED

kafka.integration.PlaintextTopicMetadataTest > testBasicTopicMetadata PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooLow 
STARTED

kafka.integration.AutoOffsetResetTest > 

Re: [DISCUSS]- JIRA ISSUE KAFKA-4566 : Can't Symlink to Kafka bins

[Colin McCabe]

Thanks for the background.  This is a tough one, because all the
proposed solutions are very long, and we would have to put them in each
script file.  I think it's probably not worth it, since people can write
their own wrappers that suit their needs.  But I haven't thought about
it that much-- maybe there is a good solution out there.

best,
Colin


On Thu, Jan 26, 2017, at 22:37, Akhilesh Naidu wrote:
> Hi Colin,
> 
> 
> I had picked up this bug from when it was reported specifically for the
> script 'kafka-console-consumer.sh'.
> 
> Going through the bug description it seemed the reporter wanted to call
> the scripts form maybe another
> 
> location using symlinks. Hence the extra code to replicate the readlink
> behavior.
> 
> 
> Going ahead we thought that this could also end up being a generic case
> for all the other scripts in the bin file
> 
> and hence the confusion about having the same code duplicated in all the
> other files in bin directory.
> 
> 
> 
> 
> Regards
> 
> Akhilesh
> 
> 
> 
> 
> 
> From: Colin McCabe 
> Sent: Thursday, January 26, 2017 12:55:16 AM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS]- JIRA ISSUE KAFKA-4566 : Can't Symlink to Kafka
> bins
> 
> Thanks for looking at this, Akhilesh.  Can you be a little bit clearer
> about why keeping all the scripts or script symlinks in the same
> directory is not an option for you?
> 
> 
> best,
> 
> Colin
> 
> 
> 
> On Tue, Jan 24, 2017, at 06:01, Akhilesh Naidu wrote:
> 
> > Hi,
> 
> 
> 
> >
> 
> 
> 
> >
> 
> 
> 
> > The bug deals with the problem one encounters in case if the script
> > 'kafka-console-consumer.sh' is executed
> >  through a symlink which could be placed on a different location
> >  on disk
> >
> 
> >  The initial suggestion provided in the bug was to make changes in the
> >  below line
> >  exec $(dirname $0)/kafka-run-class.sh
> >  kafka.tools.ConsoleConsumer "$@"
> >
> 
> >  to replace it to
> 
> >  "$(dirname "$(readlink -e "$0")")"
> 
> >
> 
> >  But as commented in the bug earlier,
> 
> >  the above would be OS dependent as MacOS version of readlink does not
> >  have an -e option.
> >
> 
> >  1) One approach could be to simulate the working of the readlink
> > function, in a portable manner. I have a working patch for this.
> >  The details are available in the comment link
> 
> >  
> > https://issues.apache.org/jira/browse/KAFKA-4566?focusedCommentId=15831442=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15831442
> >
> 
> >
> 
> >  2) Now seeing that the other scripts in the 'kafka/bin/' location
> > also could have similar calls from symlink
> >  I tried moving the function snippet into a separate utilities file,
> >  in order to reuse,
> >  but if we intend to include the utilities file in all the scripts we
> >  need to have the exact base location to our utilities file,
> >  which is what we wrote our function in the first place for :(.
> 
> >  So the only option seems to duplicate the function code in all
> >  required scripts.
> >
> 
> >  Any suggestions on how to go about the above.
> 
> >
> 
> >
> 
> 
> 
> >
> 
> 
> 
> >
> 
> 
> 
> > Regards
> 
> 
> 
> > Akhilesh
> 
> 
> 
> >
> 
> 
> 
> >
> 
> 
> 
> > DISCLAIMER == This e-mail may contain privileged and
> > confidential information which is the property of Persistent Systems
> > Ltd. It is intended only for the use of the individual or entity to
> > which it is addressed. If you are not the intended recipient, you are
> > not authorized to read, retain, copy, print, distribute or use this
> > message. If you have received this communication in error, please
> > notify the sender and delete all copies of this message. Persistent
> > Systems Ltd. does not accept any liability for virus infected mails.
> 
> 
> DISCLAIMER
> ==
> This e-mail may contain privileged and confidential information which is
> the property of Persistent Systems Ltd. It is intended only for the use
> of the individual or entity to which it is addressed. If you are not the
> intended recipient, you are not authorized to read, retain, copy, print,
> distribute or use this message. If you have received this communication
> in error, please notify the sender and delete all copies of this message.
> Persistent Systems Ltd. does not accept any liability for virus infected
> mails.
> 

[GitHub] kafka pull request #2435: Replaced for with foreach and replace if ,else if,...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2435


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (KAFKA-4317) RocksDB checkpoint files lost on kill -9

[Damian Guy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Damian Guy reassigned KAFKA-4317:
-

Assignee: Damian Guy

> RocksDB checkpoint files lost on kill -9
> 
>
> Key: KAFKA-4317
> URL: https://issues.apache.org/jira/browse/KAFKA-4317
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.0.1
>Reporter: Greg Fodor
>Assignee: Damian Guy
>Priority: Critical
>  Labels: architecture, user-experience
>
> Right now, the checkpoint files for logged RocksDB stores are written during 
> a graceful shutdown, and removed upon restoration. Unfortunately this means 
> that in a scenario where the process is forcibly killed, the checkpoint files 
> are not there, so all RocksDB stores are rematerialized from scratch on the 
> next launch.
> In a way, this is good, because it simulates bootstrapping a new node (for 
> example, its a good way to see how much I/O is used to rematerialize the 
> stores) however it leads to longer recovery times when a non-graceful 
> shutdown occurs and we want to get the job up and running again.
> It seems that two possible things to consider:
> - Simply do not remove checkpoint files on restoring. This way a kill -9 will 
> result in only repeating the restoration of all the data generated in the 
> source topics since the last graceful shutdown.
> - Continually update the checkpoint files (perhaps on commit) -- this would 
> result in the least amount of overhead/latency in restarting, but the 
> additional complexity may not be worth it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4644) Improve test coverage of StreamsPartitionAssignor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15843104#comment-15843104
 ] 

ASF GitHub Bot commented on KAFKA-4644:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2448


> Improve test coverage of StreamsPartitionAssignor
> -
>
> Key: KAFKA-4644
> URL: https://issues.apache.org/jira/browse/KAFKA-4644
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams
>Affects Versions: 0.10.3.0
>Reporter: Damian Guy
>Assignee: Damian Guy
>Priority: Minor
> Fix For: 0.10.2.0, 0.10.3.0
>
>
> Exception paths not covered



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4644) Improve test coverage of StreamsPartitionAssignor

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-4644:
-
   Resolution: Fixed
Fix Version/s: 0.10.2.0
   Status: Resolved  (was: Patch Available)

Issue resolved by pull request 2448
[https://github.com/apache/kafka/pull/2448]

> Improve test coverage of StreamsPartitionAssignor
> -
>
> Key: KAFKA-4644
> URL: https://issues.apache.org/jira/browse/KAFKA-4644
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams
>Affects Versions: 0.10.3.0
>Reporter: Damian Guy
>Assignee: Damian Guy
>Priority: Minor
> Fix For: 0.10.3.0, 0.10.2.0
>
>
> Exception paths not covered



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #2448: KAFKA-4644: Improve test coverage of StreamsPartit...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2448


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #2438: MINOR: update KTable JavaDoc

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2438


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (KAFKA-4650) Improve test coverage org.apache.kafka.streams.kstream.internals

[Damian Guy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Damian Guy reassigned KAFKA-4650:
-

Assignee: Damian Guy

> Improve test coverage org.apache.kafka.streams.kstream.internals
> 
>
> Key: KAFKA-4650
> URL: https://issues.apache.org/jira/browse/KAFKA-4650
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams
>Reporter: Damian Guy
>Assignee: Damian Guy
>Priority: Minor
> Fix For: 0.10.3.0
>
>
> Lots of classes have little or no coverage at all, i.e., 
> {{KTableAggregate.KTableAggregateValueGetter}}
> {{KTableKTableRightJoin.KTableKTableRightJoinValueGetterSupplier}}
> {{KStreamAggregate.KStreamAggregateValueGetter}}
> {{KStreamReduce.KStreamReduceValueGetter}}
> {{KStreamWindowReduce.new KTableValueGetterSupplier}}
> {{KTableAggregate.new KTableValueGetterSupplier}}
> {{KTableRepartitionMap.new KTableValueGetterSupplier}}
> {{KTableKTableRightJoin.KTableKTableRightJoinValueGetter}}
> {{KTableKTableLeftJoinValueGetter}}
> {{KStreamWindowReduce.KStreamWindowReduceValueGetter}}
> {{TimeWindow}}
> {{ChangedSerializer}}
> {{UnlimitedWindow}}
> {{WindowedDeserializer}}
> {{KStreamSessionWindowAggregate.KTableSessionWindowValueGetter}}
> {{KTableRepartitionMap}}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Fwd: [DISCUSS] KIP-114: KTable materialization and improved semantics

[Damian Guy]

I think Jan is saying that they don't always need to be materialized, i.e.,
filter just needs to apply the ValueGetter, it doesn't need yet another
physical state store.

On Fri, 27 Jan 2017 at 15:49 Michael Noll  wrote:

> Like Damian, and for the same reasons, I am more in favor of overloading
> methods rather than introducing `materialize()`.
> FWIW, we already have a similar API setup for e.g.
> `KTable#through(topicName, stateStoreName)`.
>
> A related but slightly different question is what e.g. Jan Filipiak
> mentioned earlier in this thread:
> I think we need to explain more clearly why KIP-114 doesn't propose the
> seemingly simpler solution of always materializing tables/state stores.
>
>
>
> On Fri, Jan 27, 2017 at 4:38 PM, Jan Filipiak 
> wrote:
>
> > Hi,
> >
> > Yeah its confusing, Why shoudn't it be querable by IQ? If you uses the
> > ValueGetter of Filter it will apply the filter and should be completely
> > transparent as to if another processor or IQ is accessing it? How can
> this
> > new method help?
> >
> > I cannot see the reason for the additional materialize method being
> > required! Hence I suggest leave it alone.
> > regarding removing the others I dont have strong opinions and it seems to
> > be unrelated.
> >
> > Best Jan
> >
> >
> >
> >
> > On 26.01.2017 20:48, Eno Thereska wrote:
> >
> >> Forwarding this thread to the users list too in case people would like
> to
> >> comment. It is also on the dev list.
> >>
> >> Thanks
> >> Eno
> >>
> >> Begin forwarded message:
> >>>
> >>> From: "Matthias J. Sax" 
> >>> Subject: Re: [DISCUSS] KIP-114: KTable materialization and improved
> >>> semantics
> >>> Date: 24 January 2017 at 19:30:10 GMT
> >>> To: dev@kafka.apache.org
> >>> Reply-To: dev@kafka.apache.org
> >>>
> >>> That not what I meant by "huge impact".
> >>>
> >>> I refer to the actions related to materialize a KTable: creating a
> >>> RocksDB store and a changelog topic -- users should be aware about
> >>> runtime implication and this is better expressed by an explicit method
> >>> call, rather than implicitly triggered by using a different overload of
> >>> a method.
> >>>
> >>>
> >>> -Matthias
> >>>
> >>> On 1/24/17 1:35 AM, Damian Guy wrote:
> >>>
>  I think your definition of a huge impact and mine are rather different
>  ;-P
>  Overloading a few methods  is not really a huge impact IMO. It is
> also a
>  sacrifice worth making for readability, usability of the API.
> 
>  On Mon, 23 Jan 2017 at 17:55 Matthias J. Sax 
>  wrote:
> 
>  I understand your argument, but do not agree with it.
> >
> > Your first version (even if the "flow" is not as nice) is more
> explicit
> > than the second version. Adding a stateStoreName parameter is quite
> > implicit but has a huge impact -- thus, I prefer the rather more
> > verbose
> > but explicit version.
> >
> >
> > -Matthias
> >
> > On 1/23/17 1:39 AM, Damian Guy wrote:
> >
> >> I'm not a fan of materialize. I think it interrupts the flow, i.e,
> >>
> >> table.mapValue(..).materialize().join(..).materialize()
> >> compared to:
> >> table.mapValues(..).join(..)
> >>
> >> I know which one i prefer.
> >> My preference is stil to provide overloaded methods where people can
> >> specify the store names if they want, otherwise we just generate
> them.
> >>
> >> On Mon, 23 Jan 2017 at 05:30 Matthias J. Sax  >
> >>
> > wrote:
> >
> >> Hi,
> >>>
> >>> thanks for the KIP Eno! Here are my 2 cents:
> >>>
> >>> 1) I like Guozhang's proposal about removing store name from all
> >>> KTable
> >>> methods and generate internal names (however, I would do this as
> >>> overloads). Furthermore, I would not force users to call
> >>> .materialize()
> >>> if they want to query a store, but add one more method
> >>> .stateStoreName()
> >>> that returns the store name if the KTable is materialized. Thus,
> also
> >>> .materialize() must not necessarily have a parameter storeName (ie,
> >>> we
> >>> should have some overloads here).
> >>>
> >>> I would also not allow to provide a null store name (to indicate no
> >>> materialization if not necessary) but throw an exception.
> >>>
> >>> This yields some simplification (see below).
> >>>
> >>>
> >>> 2) I also like Guozhang's proposal about KStream#toTable()
> >>>
> >>>
> >>> 3)
> >>>
>    3. What will happen when you call materialize on KTable that is
> >
>  already
> >>>
>    materialized? Will it create another StateStore (providing the
> > name
> >
>  is
> >
> >>   different), throw an Exception?
> >
>  Currently an exception is thrown, but see below.
> 
> 

Re: Fwd: [DISCUSS] KIP-114: KTable materialization and improved semantics

[Michael Noll]

Like Damian, and for the same reasons, I am more in favor of overloading
methods rather than introducing `materialize()`.
FWIW, we already have a similar API setup for e.g.
`KTable#through(topicName, stateStoreName)`.

A related but slightly different question is what e.g. Jan Filipiak
mentioned earlier in this thread:
I think we need to explain more clearly why KIP-114 doesn't propose the
seemingly simpler solution of always materializing tables/state stores.



On Fri, Jan 27, 2017 at 4:38 PM, Jan Filipiak 
wrote:

> Hi,
>
> Yeah its confusing, Why shoudn't it be querable by IQ? If you uses the
> ValueGetter of Filter it will apply the filter and should be completely
> transparent as to if another processor or IQ is accessing it? How can this
> new method help?
>
> I cannot see the reason for the additional materialize method being
> required! Hence I suggest leave it alone.
> regarding removing the others I dont have strong opinions and it seems to
> be unrelated.
>
> Best Jan
>
>
>
>
> On 26.01.2017 20:48, Eno Thereska wrote:
>
>> Forwarding this thread to the users list too in case people would like to
>> comment. It is also on the dev list.
>>
>> Thanks
>> Eno
>>
>> Begin forwarded message:
>>>
>>> From: "Matthias J. Sax" 
>>> Subject: Re: [DISCUSS] KIP-114: KTable materialization and improved
>>> semantics
>>> Date: 24 January 2017 at 19:30:10 GMT
>>> To: dev@kafka.apache.org
>>> Reply-To: dev@kafka.apache.org
>>>
>>> That not what I meant by "huge impact".
>>>
>>> I refer to the actions related to materialize a KTable: creating a
>>> RocksDB store and a changelog topic -- users should be aware about
>>> runtime implication and this is better expressed by an explicit method
>>> call, rather than implicitly triggered by using a different overload of
>>> a method.
>>>
>>>
>>> -Matthias
>>>
>>> On 1/24/17 1:35 AM, Damian Guy wrote:
>>>
 I think your definition of a huge impact and mine are rather different
 ;-P
 Overloading a few methods  is not really a huge impact IMO. It is also a
 sacrifice worth making for readability, usability of the API.

 On Mon, 23 Jan 2017 at 17:55 Matthias J. Sax 
 wrote:

 I understand your argument, but do not agree with it.
>
> Your first version (even if the "flow" is not as nice) is more explicit
> than the second version. Adding a stateStoreName parameter is quite
> implicit but has a huge impact -- thus, I prefer the rather more
> verbose
> but explicit version.
>
>
> -Matthias
>
> On 1/23/17 1:39 AM, Damian Guy wrote:
>
>> I'm not a fan of materialize. I think it interrupts the flow, i.e,
>>
>> table.mapValue(..).materialize().join(..).materialize()
>> compared to:
>> table.mapValues(..).join(..)
>>
>> I know which one i prefer.
>> My preference is stil to provide overloaded methods where people can
>> specify the store names if they want, otherwise we just generate them.
>>
>> On Mon, 23 Jan 2017 at 05:30 Matthias J. Sax 
>>
> wrote:
>
>> Hi,
>>>
>>> thanks for the KIP Eno! Here are my 2 cents:
>>>
>>> 1) I like Guozhang's proposal about removing store name from all
>>> KTable
>>> methods and generate internal names (however, I would do this as
>>> overloads). Furthermore, I would not force users to call
>>> .materialize()
>>> if they want to query a store, but add one more method
>>> .stateStoreName()
>>> that returns the store name if the KTable is materialized. Thus, also
>>> .materialize() must not necessarily have a parameter storeName (ie,
>>> we
>>> should have some overloads here).
>>>
>>> I would also not allow to provide a null store name (to indicate no
>>> materialization if not necessary) but throw an exception.
>>>
>>> This yields some simplification (see below).
>>>
>>>
>>> 2) I also like Guozhang's proposal about KStream#toTable()
>>>
>>>
>>> 3)
>>>
   3. What will happen when you call materialize on KTable that is
>
 already
>>>
   materialized? Will it create another StateStore (providing the
> name
>
 is
>
>>   different), throw an Exception?
>
 Currently an exception is thrown, but see below.


 If we follow approach (1) from Guozhang, there is no need to worry
>>> about
>>> a second materialization and also no exception must be throws. A
>>> call to
>>> .materialize() basically sets a "materialized flag" (ie, idempotent
>>> operation) and sets a new name.
>>>
>>>
>>> 4)
>>>
 Rename toStream() to toKStream() for consistency.
>
 Not sure whether that is really required. We also use
 `KStreamBuilder#stream()` and 

Build failed in Jenkins: kafka-trunk-jdk8 #1226

[Apache Jenkins Server]

See 

Changes:

[ismael] MINOR: Include more detail in `ConfigDef.parseType` exception message

--
[...truncated 8314 lines...]

kafka.integration.SaslPlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithCollision STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithCollision PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokerListWithNoTopics STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokerListWithNoTopics PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testAutoCreateTopic STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > testAutoCreateTopic PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testGetAllTopicMetadata 
STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > testGetAllTopicMetadata 
PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testBasicTopicMetadata 
STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > testBasicTopicMetadata PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED

kafka.integration.PrimitiveApiTest > testMultiProduce STARTED

kafka.integration.PrimitiveApiTest > testMultiProduce PASSED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch STARTED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch PASSED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize 
STARTED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize PASSED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests STARTED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests PASSED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch STARTED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch PASSED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression STARTED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression PASSED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic STARTED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic PASSED

kafka.integration.PrimitiveApiTest > testEmptyFetchRequest STARTED

kafka.integration.PrimitiveApiTest > testEmptyFetchRequest PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack PASSED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopicWithCollision 
STARTED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopicWithCollision 
PASSED

kafka.integration.PlaintextTopicMetadataTest > testAliveBrokerListWithNoTopics 
STARTED

kafka.integration.PlaintextTopicMetadataTest > testAliveBrokerListWithNoTopics 
PASSED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopic STARTED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopic PASSED

kafka.integration.PlaintextTopicMetadataTest > testGetAllTopicMetadata STARTED

kafka.integration.PlaintextTopicMetadataTest > testGetAllTopicMetadata PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.integration.PlaintextTopicMetadataTest > testBasicTopicMetadata STARTED

kafka.integration.PlaintextTopicMetadataTest > testBasicTopicMetadata PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooLow 
STARTED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooLow PASSED

Jenkins build is back to normal : kafka-0.10.2-jdk7 #46

[Apache Jenkins Server]

See 

Jenkins build is back to normal : kafka-trunk-jdk8 #1225

[Apache Jenkins Server]

See 

Build failed in Jenkins: kafka-0.10.2-jdk7 #45

[Apache Jenkins Server]

See 

Changes:

[ismael] KAFKA-4578; Upgrade notes for 0.10.2.0

--
[...truncated 8302 lines...]

kafka.server.KafkaConfigTest > testUncleanLeaderElectionDefault STARTED

kafka.server.KafkaConfigTest > testUncleanLeaderElectionDefault PASSED

kafka.server.KafkaConfigTest > testInvalidAdvertisedListenersProtocol STARTED

kafka.server.KafkaConfigTest > testInvalidAdvertisedListenersProtocol PASSED

kafka.server.KafkaConfigTest > testUncleanElectionEnabled STARTED

kafka.server.KafkaConfigTest > testUncleanElectionEnabled PASSED

kafka.server.KafkaConfigTest > testAdvertisePortDefault STARTED

kafka.server.KafkaConfigTest > testAdvertisePortDefault PASSED

kafka.server.KafkaConfigTest > testVersionConfiguration STARTED

kafka.server.KafkaConfigTest > testVersionConfiguration PASSED

kafka.server.KafkaConfigTest > testEqualAdvertisedListenersProtocol STARTED

kafka.server.KafkaConfigTest > testEqualAdvertisedListenersProtocol PASSED

kafka.server.SslReplicaFetchTest > testReplicaFetcherThread STARTED

kafka.server.SslReplicaFetchTest > testReplicaFetcherThread PASSED

kafka.server.IsrExpirationTest > testIsrExpirationForSlowFollowers STARTED

kafka.server.IsrExpirationTest > testIsrExpirationForSlowFollowers PASSED

kafka.server.IsrExpirationTest > testIsrExpirationForStuckFollowers STARTED

kafka.server.IsrExpirationTest > testIsrExpirationForStuckFollowers PASSED

kafka.server.IsrExpirationTest > testIsrExpirationIfNoFetchRequestMade STARTED

kafka.server.IsrExpirationTest > testIsrExpirationIfNoFetchRequestMade PASSED

kafka.server.SaslPlaintextReplicaFetchTest > testReplicaFetcherThread STARTED

kafka.server.SaslPlaintextReplicaFetchTest > testReplicaFetcherThread PASSED

kafka.server.ReplicationQuotasTest > 
shouldBootstrapTwoBrokersWithLeaderThrottle STARTED

kafka.server.ReplicationQuotasTest > 
shouldBootstrapTwoBrokersWithLeaderThrottle PASSED

kafka.server.ReplicationQuotasTest > shouldThrottleOldSegments STARTED

kafka.server.ReplicationQuotasTest > shouldThrottleOldSegments PASSED

kafka.server.ReplicationQuotasTest > 
shouldBootstrapTwoBrokersWithFollowerThrottle STARTED

kafka.server.ReplicationQuotasTest > 
shouldBootstrapTwoBrokersWithFollowerThrottle PASSED

kafka.server.ServerStartupTest > testBrokerStateRunningAfterZK STARTED

kafka.server.ServerStartupTest > testBrokerStateRunningAfterZK PASSED

kafka.server.ServerStartupTest > testBrokerCreatesZKChroot STARTED

kafka.server.ServerStartupTest > testBrokerCreatesZKChroot PASSED

kafka.server.ServerStartupTest > testConflictBrokerStartupWithSamePort STARTED

kafka.server.ServerStartupTest > testConflictBrokerStartupWithSamePort PASSED

kafka.server.ServerStartupTest > testConflictBrokerRegistration STARTED

kafka.server.ServerStartupTest > testConflictBrokerRegistration PASSED

kafka.server.ServerStartupTest > testBrokerSelfAware STARTED

kafka.server.ServerStartupTest > testBrokerSelfAware PASSED

kafka.server.ProduceRequestTest > testSimpleProduceRequest STARTED

kafka.server.ProduceRequestTest > testSimpleProduceRequest PASSED

kafka.server.ProduceRequestTest > testCorruptLz4ProduceRequest STARTED

kafka.server.ProduceRequestTest > testCorruptLz4ProduceRequest PASSED

kafka.server.ReplicaManagerTest > testHighWaterMarkDirectoryMapping STARTED

kafka.server.ReplicaManagerTest > testHighWaterMarkDirectoryMapping PASSED

kafka.server.ReplicaManagerTest > 
testFetchBeyondHighWatermarkReturnEmptyResponse STARTED

kafka.server.ReplicaManagerTest > 
testFetchBeyondHighWatermarkReturnEmptyResponse PASSED

kafka.server.ReplicaManagerTest > testIllegalRequiredAcks STARTED

kafka.server.ReplicaManagerTest > testIllegalRequiredAcks PASSED

kafka.server.ReplicaManagerTest > testClearPurgatoryOnBecomingFollower STARTED

kafka.server.ReplicaManagerTest > testClearPurgatoryOnBecomingFollower PASSED

kafka.server.ReplicaManagerTest > testHighwaterMarkRelativeDirectoryMapping 
STARTED

kafka.server.ReplicaManagerTest > testHighwaterMarkRelativeDirectoryMapping 
PASSED

kafka.server.KafkaMetricReporterClusterIdTest > testClusterIdPresent STARTED

kafka.server.KafkaMetricReporterClusterIdTest > testClusterIdPresent PASSED

kafka.server.CreateTopicsRequestWithPolicyTest > testValidCreateTopicsRequests 
STARTED

kafka.server.CreateTopicsRequestWithPolicyTest > testValidCreateTopicsRequests 
PASSED

kafka.server.CreateTopicsRequestWithPolicyTest > testErrorCreateTopicsRequests 
STARTED

kafka.server.CreateTopicsRequestWithPolicyTest > testErrorCreateTopicsRequests 
PASSED

kafka.server.OffsetCommitTest > testUpdateOffsets STARTED

kafka.server.OffsetCommitTest > testUpdateOffsets PASSED

kafka.server.OffsetCommitTest > testLargeMetadataPayload STARTED

kafka.server.OffsetCommitTest > testLargeMetadataPayload PASSED

kafka.server.OffsetCommitTest > testOffsetsDeleteAfterTopicDeletion STARTED

kafka.server.OffsetCommitTest > testOffsetsDeleteAfterTopicDeletion 

Jenkins build is back to normal : kafka-trunk-jdk7 #1888

[Apache Jenkins Server]

See 

[jira] [Updated] (KAFKA-4385) producer is sending too many unnecessary meta data request if the meta data for a topic is not available and "auto.create.topics.enable" =false

[Stevo Slavic (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stevo Slavic updated KAFKA-4385:

Affects Version/s: 0.9.0.1

> producer is sending too many unnecessary meta data request if the meta data 
> for a topic is not available and "auto.create.topics.enable" =false
> ---
>
> Key: KAFKA-4385
> URL: https://issues.apache.org/jira/browse/KAFKA-4385
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.1
>Reporter: Jun Yao
>
> All current kafka-client producer implementation (<= 0.10.1.0),
> When sending a msg to a topic, it will first check if meta data for this 
> topic is available or not, 
> when not available, it will set "metadata.requestUpdate()" and wait for meta 
> data from brokers, 
> The thing is inside "org.apache.kafka.clients.Metadata.awaitUpdate()", it's 
> already doing a "while (this.version <= lastVersion)" loop waiting for new 
> version response, 
> So the loop inside 
> "org.apache.kafka.clients.producer.KafkaProducer.waitOnMetadata() is not 
> needed, 
> When "auto.create.topics.enable" is false, sending msgs to a non-exist topic 
> will trigger too many meta requests, everytime a metadata response is 
> returned, because it does not contain the metadata for the topic, it's going 
> to try again until TimeoutException is thrown; 
> This is a waste and sometimes causes too much overhead when unexpected msgs 
> are arrived. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4385) producer is sending too many unnecessary meta data request if the meta data for a topic is not available and "auto.create.topics.enable" =false

[Stevo Slavic (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15842831#comment-15842831
 ] 

Stevo Slavic commented on KAFKA-4385:
-

{{UnknownTopicOrPartitionException}} extends {{InvalidMetadataException}} which 
extends {{RetriableException}}.

This shows root cause of the problem - by current Kafka design 
{{UnknownTopicOrPartitionException}} is considered to be retriable exception at 
all times. IMO, when auto topic creation is disabled, 
{{UnknownTopicOrPartitionException}} should not be considered as retriable.

Besides of unnecessary metadata retrieval retries, I've found that 
{{KafkaProducer}}, in 0.9.0.1 and 0.10.1.1 with auto topic creation disabled, 
when one tries to send to non-existing topic, registered callback will not be 
completed with {{UnknownTopicOrPartitionException}}. Instead 
{noformat}
"org.apache.kafka.common.errors.TimeoutException: Failed to update metadata 
after X ms."
{noformat}
will be thrown.

> producer is sending too many unnecessary meta data request if the meta data 
> for a topic is not available and "auto.create.topics.enable" =false
> ---
>
> Key: KAFKA-4385
> URL: https://issues.apache.org/jira/browse/KAFKA-4385
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jun Yao
>
> All current kafka-client producer implementation (<= 0.10.1.0),
> When sending a msg to a topic, it will first check if meta data for this 
> topic is available or not, 
> when not available, it will set "metadata.requestUpdate()" and wait for meta 
> data from brokers, 
> The thing is inside "org.apache.kafka.clients.Metadata.awaitUpdate()", it's 
> already doing a "while (this.version <= lastVersion)" loop waiting for new 
> version response, 
> So the loop inside 
> "org.apache.kafka.clients.producer.KafkaProducer.waitOnMetadata() is not 
> needed, 
> When "auto.create.topics.enable" is false, sending msgs to a non-exist topic 
> will trigger too many meta requests, everytime a metadata response is 
> returned, because it does not contain the metadata for the topic, it's going 
> to try again until TimeoutException is thrown; 
> This is a waste and sometimes causes too much overhead when unexpected msgs 
> are arrived. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #2345: MINOR: ConfigDef `parseType` exception message upd...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2345


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #2459: MINOR: update JavaDocs for Kafka Streams DSL helpe...

[mjsax]

GitHub user mjsax opened a pull request:

https://github.com/apache/kafka/pull/2459

MINOR: update JavaDocs for Kafka Streams DSL helpers

 - also deprecate ZK config for Streams

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mjsax/kafka javaDocImprovements8

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2459.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2459


commit 8f90c94c971d96f996ca6130c41f0cbe84d6052d
Author: Matthias J. Sax 
Date:   2017-01-27T12:00:59Z

MINOR: update JavaDocs for Kafka Streams DSL helpers
 - also deprecate ZK config for Streams




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (KAFKA-4703) Test with two SASL_SSL listeners with different JAAS contexts

[Balint Molnar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Balint Molnar reassigned KAFKA-4703:


Assignee: Balint Molnar

> Test with two SASL_SSL listeners with different JAAS contexts
> -
>
> Key: KAFKA-4703
> URL: https://issues.apache.org/jira/browse/KAFKA-4703
> Project: Kafka
>  Issue Type: Test
>Reporter: Ismael Juma
>Assignee: Balint Molnar
>  Labels: newbie
>
> [~rsivaram] suggested the following in 
> https://github.com/apache/kafka/pull/2406:
> {quote}
> I think this feature allows two SASL_SSL listeners, one for external and one 
> for internal and the two can use different mechanisms and different JAAS 
> contexts. That makes the multi-mechanism configuration neater. I think it 
> will be useful to have an integration test for this, perhaps change 
> SaslMultiMechanismConsumerTest.
> {quote}
> And my reply:
> {quote}
> I think it's a bit tricky to support multiple listeners in 
> KafkaServerTestHarness. Maybe it's easier to do the test you suggest in 
> MultipleListenersWithSameSecurityProtocolTest.
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #2445: KAFKA-4578: Upgrade notes for 0.10.2.0

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2445


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-4578) Upgrade notes for 0.10.2.0

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-4578:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Issue resolved by pull request 2445
[https://github.com/apache/kafka/pull/2445]

> Upgrade notes for 0.10.2.0
> --
>
> Key: KAFKA-4578
> URL: https://issues.apache.org/jira/browse/KAFKA-4578
> Project: Kafka
>  Issue Type: Task
>Reporter: Ismael Juma
>Assignee: Ismael Juma
>Priority: Blocker
> Fix For: 0.10.2.0
>
>
> Some basic text has been added already, but it should be improved before the 
> release.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4578) Upgrade notes for 0.10.2.0

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15842576#comment-15842576
 ] 

ASF GitHub Bot commented on KAFKA-4578:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2445


> Upgrade notes for 0.10.2.0
> --
>
> Key: KAFKA-4578
> URL: https://issues.apache.org/jira/browse/KAFKA-4578
> Project: Kafka
>  Issue Type: Task
>Reporter: Ismael Juma
>Assignee: Ismael Juma
>Priority: Blocker
> Fix For: 0.10.2.0
>
>
> Some basic text has been added already, but it should be improved before the 
> release.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (KAFKA-4474) Poor kafka-streams throughput

[Andres Gomez Ferrer (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15842519#comment-15842519
 ] 

Andres Gomez Ferrer edited comment on KAFKA-4474 at 1/27/17 10:40 AM:
--

Thank you [~enothereska] and [~jjchorrobe]!! I know that there is no immediate 
fix. The problem occurs under special circumstances. I keep the JIRA issue 
opened, no problem! :) 


was (Author: agomez):
Thank you [~enothereska]!! I know that there is no immediate fix. The problem 
occurs under special circumstances. I keep the JIRA issue opened, no problem! 
:) 

> Poor kafka-streams throughput
> -
>
> Key: KAFKA-4474
> URL: https://issues.apache.org/jira/browse/KAFKA-4474
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.1.0
>Reporter: Juan Chorro
>Assignee: Eno Thereska
> Attachments: hctop sreenshot.png, kafka-streams-bug-1.png, 
> kafka-streams-bug-2.png, Performance test results.png
>
>
> Hi! 
> I'm writing because I have a worry about kafka-streams throughput.
> I have only a kafka-streams application instance that consumes from 'input' 
> topic, prints on the screen and produces in 'output' topic. All topics have 4 
> partitions. As can be observed the topology is very simple.
> I produce 120K messages/second to 'input' topic, when I measure the 'output' 
> topic I detect that I'm receiving ~4K messages/second. I had next 
> configuration (Remaining parameters by default):
> application.id: myApp
> bootstrap.servers: localhost:9092
> zookeeper.connect: localhost:2181
> num.stream.threads: 1
> I was doing proofs and tests without success, but when I created a new 
> 'input' topic with 1 partition (Maintain 'output' topic with 4 partitions) I 
> got in 'output' topic 120K messages/seconds.
> I have been doing some performance tests and proof with next cases (All 
> topics have 4 partitions in all cases):
> Case A - 1 Instance:
> - With num.stream.threads set to 1 I had ~3785 messages/second
> - With num.stream.threads set to 2 I had ~3938 messages/second
> - With num.stream.threads set to 4 I had ~120K messages/second
> Case B - 2 Instances:
> - With num.stream.threads set to 1 I had ~3930 messages/second for each 
> instance (And throughput ~8K messages/second)
> - With num.stream.threads set to 2 I had ~3945 messages/second for each 
> instance (And more or less same throughput that with num.stream.threads set 
> to 1)
> Case C - 4 Instances
> - With num.stream.threads set to 1 I had 3946 messages/seconds for each 
> instance (And throughput ~17K messages/second):
> As can be observed when num.stream.threads is set to #partitions I have best 
> results. Then I have next questions:
> - Why whether I have a topic with #partitions > 1 and with 
> num.streams.threads is set to 1 I have ~4K messages/second always?
> - In case C. 4 instances with num.stream.threads set to 1 should be better 
> that 1 instance with num.stream.threads set to 4. Is corrects this 
> supposition?
> This is the kafka-streams application that I use: 
> https://gist.github.com/Chorro/5522ec4acd1a005eb8c9663da86f5a18



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4685) All partitions offline, no controller znode in ZK

[JIRA]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sinóros-Szabó Péter updated KAFKA-4685:
---
Summary: All partitions offline, no controller znode in ZK  (was: All 
partitions offline, no conroller znode in ZK)

> All partitions offline, no controller znode in ZK
> -
>
> Key: KAFKA-4685
> URL: https://issues.apache.org/jira/browse/KAFKA-4685
> Project: Kafka
>  Issue Type: Bug
>Reporter: Sinóros-Szabó Péter
> Attachments: kafka-0-logs.zip, kafka-1-logs.zip, kafka-2-logs.zip, 
> zookeeper-logs.zip
>
>
> Setup: 3 Kafka 0.11.1.1 nodes on kubernetes (in AWS), and another 3 nodes of 
> Zookeeper 3.5.2-alpha also in kubernetes (in AWS).
> At 2017-01-23 06:51 ZK sessions expired. It seems from the logs that kafka-2 
> was elected as the new controller, but I am not sure how to read that logs.
> I've checked the ZK data and both the /controller is empty and also the 
> /brokers/ids is empty. Kafka reports that all partitions are offline, 
> although it seems to be working because messages are coming and going.
> We are using an alpha version, I know that it may be a problem, but I suppose 
> that Kafka should see that there is not any node registered as controller.
> I have attached the Kafka and ZK logs



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4474) Poor kafka-streams throughput

[Andres Gomez Ferrer (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15842519#comment-15842519
 ] 

Andres Gomez Ferrer commented on KAFKA-4474:


Thank you [~enothereska]!! I know that there is no immediate fix. The problem 
occurs under special circumstances. I keep the JIRA issue opened, no problem! 
:) 

> Poor kafka-streams throughput
> -
>
> Key: KAFKA-4474
> URL: https://issues.apache.org/jira/browse/KAFKA-4474
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.1.0
>Reporter: Juan Chorro
>Assignee: Eno Thereska
> Attachments: hctop sreenshot.png, kafka-streams-bug-1.png, 
> kafka-streams-bug-2.png, Performance test results.png
>
>
> Hi! 
> I'm writing because I have a worry about kafka-streams throughput.
> I have only a kafka-streams application instance that consumes from 'input' 
> topic, prints on the screen and produces in 'output' topic. All topics have 4 
> partitions. As can be observed the topology is very simple.
> I produce 120K messages/second to 'input' topic, when I measure the 'output' 
> topic I detect that I'm receiving ~4K messages/second. I had next 
> configuration (Remaining parameters by default):
> application.id: myApp
> bootstrap.servers: localhost:9092
> zookeeper.connect: localhost:2181
> num.stream.threads: 1
> I was doing proofs and tests without success, but when I created a new 
> 'input' topic with 1 partition (Maintain 'output' topic with 4 partitions) I 
> got in 'output' topic 120K messages/seconds.
> I have been doing some performance tests and proof with next cases (All 
> topics have 4 partitions in all cases):
> Case A - 1 Instance:
> - With num.stream.threads set to 1 I had ~3785 messages/second
> - With num.stream.threads set to 2 I had ~3938 messages/second
> - With num.stream.threads set to 4 I had ~120K messages/second
> Case B - 2 Instances:
> - With num.stream.threads set to 1 I had ~3930 messages/second for each 
> instance (And throughput ~8K messages/second)
> - With num.stream.threads set to 2 I had ~3945 messages/second for each 
> instance (And more or less same throughput that with num.stream.threads set 
> to 1)
> Case C - 4 Instances
> - With num.stream.threads set to 1 I had 3946 messages/seconds for each 
> instance (And throughput ~17K messages/second):
> As can be observed when num.stream.threads is set to #partitions I have best 
> results. Then I have next questions:
> - Why whether I have a topic with #partitions > 1 and with 
> num.streams.threads is set to 1 I have ~4K messages/second always?
> - In case C. 4 instances with num.stream.threads set to 1 should be better 
> that 1 instance with num.stream.threads set to 4. Is corrects this 
> supposition?
> This is the kafka-streams application that I use: 
> https://gist.github.com/Chorro/5522ec4acd1a005eb8c9663da86f5a18



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4474) Poor kafka-streams throughput

[Eno Thereska (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15842514#comment-15842514
 ] 

Eno Thereska commented on KAFKA-4474:
-

Thanks [~agomez] for your excellent investigation. I can confirm that indeed 
that is the case. There is no immediate fix to this problem. We need to make 
poll() cheaper, or call it on a separate thread, or call it less often (in this 
latter case we'd add latency to a request that might have just arrived). Let's 
keep this JIRA open until we fix the problem, but it's not straightforward, 
we'll think more about it.

> Poor kafka-streams throughput
> -
>
> Key: KAFKA-4474
> URL: https://issues.apache.org/jira/browse/KAFKA-4474
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.1.0
>Reporter: Juan Chorro
>Assignee: Eno Thereska
> Attachments: hctop sreenshot.png, kafka-streams-bug-1.png, 
> kafka-streams-bug-2.png, Performance test results.png
>
>
> Hi! 
> I'm writing because I have a worry about kafka-streams throughput.
> I have only a kafka-streams application instance that consumes from 'input' 
> topic, prints on the screen and produces in 'output' topic. All topics have 4 
> partitions. As can be observed the topology is very simple.
> I produce 120K messages/second to 'input' topic, when I measure the 'output' 
> topic I detect that I'm receiving ~4K messages/second. I had next 
> configuration (Remaining parameters by default):
> application.id: myApp
> bootstrap.servers: localhost:9092
> zookeeper.connect: localhost:2181
> num.stream.threads: 1
> I was doing proofs and tests without success, but when I created a new 
> 'input' topic with 1 partition (Maintain 'output' topic with 4 partitions) I 
> got in 'output' topic 120K messages/seconds.
> I have been doing some performance tests and proof with next cases (All 
> topics have 4 partitions in all cases):
> Case A - 1 Instance:
> - With num.stream.threads set to 1 I had ~3785 messages/second
> - With num.stream.threads set to 2 I had ~3938 messages/second
> - With num.stream.threads set to 4 I had ~120K messages/second
> Case B - 2 Instances:
> - With num.stream.threads set to 1 I had ~3930 messages/second for each 
> instance (And throughput ~8K messages/second)
> - With num.stream.threads set to 2 I had ~3945 messages/second for each 
> instance (And more or less same throughput that with num.stream.threads set 
> to 1)
> Case C - 4 Instances
> - With num.stream.threads set to 1 I had 3946 messages/seconds for each 
> instance (And throughput ~17K messages/second):
> As can be observed when num.stream.threads is set to #partitions I have best 
> results. Then I have next questions:
> - Why whether I have a topic with #partitions > 1 and with 
> num.streams.threads is set to 1 I have ~4K messages/second always?
> - In case C. 4 instances with num.stream.threads set to 1 should be better 
> that 1 instance with num.stream.threads set to 4. Is corrects this 
> supposition?
> This is the kafka-streams application that I use: 
> https://gist.github.com/Chorro/5522ec4acd1a005eb8c9663da86f5a18



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-trunk-jdk7 #1887

[Apache Jenkins Server]

See 

Changes:

[wangguoz] MINOR: Add Streams system test for broker backwards compatibility

[wangguoz] MINOR: Streams API JavaDoc improvements

--
[...truncated 8317 lines...]

kafka.producer.AsyncProducerTest > testInvalidPartition STARTED

kafka.producer.AsyncProducerTest > testInvalidPartition PASSED

kafka.producer.AsyncProducerTest > testNoBroker STARTED

kafka.producer.AsyncProducerTest > testNoBroker PASSED

kafka.producer.AsyncProducerTest > testProduceAfterClosed STARTED

kafka.producer.AsyncProducerTest > testProduceAfterClosed PASSED

kafka.producer.AsyncProducerTest > testJavaProducer STARTED

kafka.producer.AsyncProducerTest > testJavaProducer PASSED

kafka.producer.AsyncProducerTest > testIncompatibleEncoder STARTED

kafka.producer.AsyncProducerTest > testIncompatibleEncoder PASSED

kafka.metrics.KafkaTimerTest > testKafkaTimer STARTED

kafka.metrics.KafkaTimerTest > testKafkaTimer PASSED

kafka.metrics.MetricsTest > testMetricsReporterAfterDeletingTopic STARTED

kafka.metrics.MetricsTest > testMetricsReporterAfterDeletingTopic PASSED

kafka.metrics.MetricsTest > 
testBrokerTopicMetricsUnregisteredAfterDeletingTopic STARTED

kafka.metrics.MetricsTest > 
testBrokerTopicMetricsUnregisteredAfterDeletingTopic PASSED

kafka.metrics.MetricsTest > testClusterIdMetric STARTED

kafka.metrics.MetricsTest > testClusterIdMetric PASSED

kafka.metrics.MetricsTest > testMetricsLeak STARTED

kafka.metrics.MetricsTest > testMetricsLeak PASSED

kafka.zk.ZKPathTest > testCreatePersistentSequentialThrowsException STARTED

kafka.zk.ZKPathTest > testCreatePersistentSequentialThrowsException PASSED

kafka.zk.ZKPathTest > testCreatePersistentSequentialExists STARTED

kafka.zk.ZKPathTest > testCreatePersistentSequentialExists PASSED

kafka.zk.ZKPathTest > testCreateEphemeralPathExists STARTED

kafka.zk.ZKPathTest > testCreateEphemeralPathExists PASSED

kafka.zk.ZKPathTest > testCreatePersistentPath STARTED

kafka.zk.ZKPathTest > testCreatePersistentPath PASSED

kafka.zk.ZKPathTest > testMakeSurePersistsPathExistsThrowsException STARTED

kafka.zk.ZKPathTest > testMakeSurePersistsPathExistsThrowsException PASSED

kafka.zk.ZKPathTest > testCreateEphemeralPathThrowsException STARTED

kafka.zk.ZKPathTest > testCreateEphemeralPathThrowsException PASSED

kafka.zk.ZKPathTest > testCreatePersistentPathThrowsException STARTED

kafka.zk.ZKPathTest > testCreatePersistentPathThrowsException PASSED

kafka.zk.ZKPathTest > testMakeSurePersistsPathExists STARTED

kafka.zk.ZKPathTest > testMakeSurePersistsPathExists PASSED

kafka.zk.ZKEphemeralTest > testOverlappingSessions[0] STARTED

kafka.zk.ZKEphemeralTest > testOverlappingSessions[0] PASSED

kafka.zk.ZKEphemeralTest > testEphemeralNodeCleanup[0] STARTED

kafka.zk.ZKEphemeralTest > testEphemeralNodeCleanup[0] PASSED

kafka.zk.ZKEphemeralTest > testZkWatchedEphemeral[0] STARTED

kafka.zk.ZKEphemeralTest > testZkWatchedEphemeral[0] PASSED

kafka.zk.ZKEphemeralTest > testSameSession[0] STARTED

kafka.zk.ZKEphemeralTest > testSameSession[0] PASSED

kafka.zk.ZKEphemeralTest > testOverlappingSessions[1] STARTED

kafka.zk.ZKEphemeralTest > testOverlappingSessions[1] PASSED

kafka.zk.ZKEphemeralTest > testEphemeralNodeCleanup[1] STARTED

kafka.zk.ZKEphemeralTest > testEphemeralNodeCleanup[1] PASSED

kafka.zk.ZKEphemeralTest > testZkWatchedEphemeral[1] STARTED

kafka.zk.ZKEphemeralTest > testZkWatchedEphemeral[1] PASSED

kafka.zk.ZKEphemeralTest > testSameSession[1] STARTED

kafka.zk.ZKEphemeralTest > testSameSession[1] PASSED

kafka.security.auth.AclTest > testAclJsonConversion STARTED

kafka.security.auth.AclTest > testAclJsonConversion PASSED

kafka.security.auth.ResourceTypeTest > testFromString STARTED

kafka.security.auth.ResourceTypeTest > testFromString PASSED

kafka.security.auth.ZkAuthorizationTest > testIsZkSecurityEnabled STARTED

kafka.security.auth.ZkAuthorizationTest > testIsZkSecurityEnabled PASSED

kafka.security.auth.ZkAuthorizationTest > testZkUtils STARTED

kafka.security.auth.ZkAuthorizationTest > testZkUtils PASSED

kafka.security.auth.ZkAuthorizationTest > testZkAntiMigration STARTED

kafka.security.auth.ZkAuthorizationTest > testZkAntiMigration PASSED

kafka.security.auth.ZkAuthorizationTest > testZkMigration STARTED

kafka.security.auth.ZkAuthorizationTest > testZkMigration PASSED

kafka.security.auth.ZkAuthorizationTest > testChroot STARTED

kafka.security.auth.ZkAuthorizationTest > testChroot PASSED

kafka.security.auth.ZkAuthorizationTest > testDelete STARTED

kafka.security.auth.ZkAuthorizationTest > testDelete PASSED

kafka.security.auth.ZkAuthorizationTest > testDeleteRecursive STARTED

kafka.security.auth.ZkAuthorizationTest > testDeleteRecursive PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testAllowAllAccess STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testAllowAllAccess PASSED

kafka.security.auth.SimpleAclAuthorizerTest > 

Re: Trying to understand design decision about producer ack and min.insync.replicas

[James Cheng]

> On Jan 27, 2017, at 12:18 AM, Ewen Cheslack-Postava  wrote:
> 
> On Thu, Jan 26, 2017 at 4:23 PM, Luciano Afranllie > wrote:
> 
>> I was thinking about the situation where you have less brokers in the ISR
>> list than the number set in min.insync.replicas.
>> 
>> My idea was that if I, as an administrator, for a given topic, want to
>> favor durability over availability, then if that topic has less ISR than
>> the value set in min.insync.replicas I may want to stop producing to the
>> topic. In the way min.insync.replicas and ack work, I need to coordinate
>> with all producers in order to achieve this. There is no way (or I don't
>> know it) to globally enforce stop producing to a topic if it is under
>> replicated.
>> 
>> I don't see why, for the same topic, some producers might want get an error
>> when the number of ISR is below min.insync.replicas while other producers
>> don't. I think it could be more useful to be able to set that ALL producers
>> should get an error when a given topic is under replicated so they stop
>> producing, than for a single producer to get an error when ANY topic is
>> under replicated. I don't have a lot of experience with Kafka so I may be
>> missing some use cases.
>> 
> 
> It's also a matter of not having to do a ton of configuration on a
> per-topic basis. Putting some control in the producer apps hands means you
> can set reasonably global defaults which make sense for apps that require
> stronger durability while letting cases that have lower requirements still
> benefit from the durability before consumers see data but not block
> producers because the producer chooses lower requirements. WIthout
> requiring the ability to make config changes on the Kafka brokers (which
> may be locked down and restricted only to Kafka admins), the producer
> application can choose to accept weaker guarantees based on the tradeoffs
> it needs to make.
> 

I'm not sure I follow, Ewen.

I do agree that if I set min.insync.replicas at a broker level, then of course 
I would like individual producers to decide whether their topic (which inherits 
from the global setting) should reject writes if that topic has 
size(ISR) The ability to make this tradeoff in different places can seem more complex
> (and really by definition *is* more complex), but it also offers more
> flexibility.
> 
> -Ewen
> 
> 
>> But I understand your point, min.insync.replicas setting should be
>> understood as "if a producer wants to get an error when topics are under
>> replicated, then how many replicas are enough for not raising an error?"
>> 
>> 
>> On Thu, Jan 26, 2017 at 4:16 PM, Ewen Cheslack-Postava 
>> wrote:
>> 
>>> The acks setting for the producer doesn't affect the final durability
>>> guarantees. These are still enforced by the replication and min ISR
>>> settings. Instead, the ack setting just lets the producer control how
>>> durable the write is before *that producer* can consider the write
>>> "complete", i.e. before it gets an ack.
>>> 
>>> -Ewen
>>> 
>>> On Tue, Jan 24, 2017 at 12:46 PM, Luciano Afranllie <
>>> listas.luaf...@gmail.com> wrote:
>>> 
 Hi everybody
 
 I am trying to understand why Kafka let each individual producer, on a
 connection per connection basis, choose the tradeoff between
>> availability
 and durability, honoring min.insync.replicas value only if producer
>> uses
 ack=all.
 
 I mean, for a single topic, cluster administrators can't enforce
>> messages
 to be stores in a minimum number of replicas without coordinating with
>>> all
 producers to that topic so all of them use ack=all.
 
 Is there something that I am missing? Is there any other strategy to
 overcome this situation?
 
 Regards
 Luciano
 
>>> 
>> 

[jira] [Reopened] (KAFKA-4222) Transient failure in QueryableStateIntegrationTest.queryOnRebalance

[Damian Guy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Damian Guy reopened KAFKA-4222:
---

re-opening as there was one more case of this recently. However the build has 
been cleaned up. Just worth keeping an eye on for a bit

> Transient failure in QueryableStateIntegrationTest.queryOnRebalance
> ---
>
> Key: KAFKA-4222
> URL: https://issues.apache.org/jira/browse/KAFKA-4222
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams, unit tests
>Reporter: Jason Gustafson
>Assignee: Matthias J. Sax
> Fix For: 0.10.1.0
>
>
> Seen here: https://builds.apache.org/job/kafka-trunk-jdk8/915/console
> {code}
> org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
> queryOnRebalance[1] FAILED
> java.lang.AssertionError: Condition not met within timeout 3. waiting 
> for metadata, store and value to be non null
> at 
> org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:276)
> at 
> org.apache.kafka.streams.integration.QueryableStateIntegrationTest.verifyAllKVKeys(QueryableStateIntegrationTest.java:263)
> at 
> org.apache.kafka.streams.integration.QueryableStateIntegrationTest.queryOnRebalance(QueryableStateIntegrationTest.java:342)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4696) Streams standby task assignment should be state-store aware

[Damian Guy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Damian Guy updated KAFKA-4696:
--
Description: 
Task Assignment is currently not aware of which tasks have State Stores. This 
can result in uneven balance of standby task assignment as all tasks are 
assigned, but only those tasks with state-stores are ever created by 
{{StreamThread}}. So what seems like an optimal strategy during assignment time 
could be sub-optimal post-assignment.

For example, lets say we have 4 tasks (2 with state-stores), 2 clients, 
numStandbyReplicas = 1. Each client would get 2 active and 2 standby tasks.  
One of the clients may end up with both state-store tasks, while the other has 
none.

Further to this, standby task configuration is currently "all or nothing". It 
might make sense to allow more fine grained configuration, i.e., the ability to 
specify the number of standby replicas individually for each stateful operator.


  was:
Task Assignment is currently not aware of which tasks have State Stores. This 
can result in uneven balance of standby task assignment as all tasks are 
assigned, but only those tasks with state-stores are ever created by 
{{StreamThread}}. So what seems like an optimal strategy during assignment time 
could be sub-optimal post-assignment.
For example, lets say we have 4 tasks (2 with state-stores), 2 clients, 
numStandbyReplicas = 1. Each client would get 2 active and 2 standby tasks.  
One of the clients may end up with both state-store tasks, while the other has 
none.


> Streams standby task assignment should be state-store aware
> ---
>
> Key: KAFKA-4696
> URL: https://issues.apache.org/jira/browse/KAFKA-4696
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.0, 0.10.3.0
>Reporter: Damian Guy
> Fix For: 0.10.3.0
>
>
> Task Assignment is currently not aware of which tasks have State Stores. This 
> can result in uneven balance of standby task assignment as all tasks are 
> assigned, but only those tasks with state-stores are ever created by 
> {{StreamThread}}. So what seems like an optimal strategy during assignment 
> time could be sub-optimal post-assignment.
> For example, lets say we have 4 tasks (2 with state-stores), 2 clients, 
> numStandbyReplicas = 1. Each client would get 2 active and 2 standby tasks.  
> One of the clients may end up with both state-store tasks, while the other 
> has none.
> Further to this, standby task configuration is currently "all or nothing". It 
> might make sense to allow more fine grained configuration, i.e., the ability 
> to specify the number of standby replicas individually for each stateful 
> operator.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Trying to understand design decision about producer ack and min.insync.replicas

[Ewen Cheslack-Postava]

On Thu, Jan 26, 2017 at 4:23 PM, Luciano Afranllie  wrote:

> I was thinking about the situation where you have less brokers in the ISR
> list than the number set in min.insync.replicas.
>
> My idea was that if I, as an administrator, for a given topic, want to
> favor durability over availability, then if that topic has less ISR than
> the value set in min.insync.replicas I may want to stop producing to the
> topic. In the way min.insync.replicas and ack work, I need to coordinate
> with all producers in order to achieve this. There is no way (or I don't
> know it) to globally enforce stop producing to a topic if it is under
> replicated.
>
> I don't see why, for the same topic, some producers might want get an error
> when the number of ISR is below min.insync.replicas while other producers
> don't. I think it could be more useful to be able to set that ALL producers
> should get an error when a given topic is under replicated so they stop
> producing, than for a single producer to get an error when ANY topic is
> under replicated. I don't have a lot of experience with Kafka so I may be
> missing some use cases.
>

It's also a matter of not having to do a ton of configuration on a
per-topic basis. Putting some control in the producer apps hands means you
can set reasonably global defaults which make sense for apps that require
stronger durability while letting cases that have lower requirements still
benefit from the durability before consumers see data but not block
producers because the producer chooses lower requirements. WIthout
requiring the ability to make config changes on the Kafka brokers (which
may be locked down and restricted only to Kafka admins), the producer
application can choose to accept weaker guarantees based on the tradeoffs
it needs to make.

The ability to make this tradeoff in different places can seem more complex
(and really by definition *is* more complex), but it also offers more
flexibility.

-Ewen


> But I understand your point, min.insync.replicas setting should be
> understood as "if a producer wants to get an error when topics are under
> replicated, then how many replicas are enough for not raising an error?"
>
>
> On Thu, Jan 26, 2017 at 4:16 PM, Ewen Cheslack-Postava 
> wrote:
>
> > The acks setting for the producer doesn't affect the final durability
> > guarantees. These are still enforced by the replication and min ISR
> > settings. Instead, the ack setting just lets the producer control how
> > durable the write is before *that producer* can consider the write
> > "complete", i.e. before it gets an ack.
> >
> > -Ewen
> >
> > On Tue, Jan 24, 2017 at 12:46 PM, Luciano Afranllie <
> > listas.luaf...@gmail.com> wrote:
> >
> > > Hi everybody
> > >
> > > I am trying to understand why Kafka let each individual producer, on a
> > > connection per connection basis, choose the tradeoff between
> availability
> > > and durability, honoring min.insync.replicas value only if producer
> uses
> > > ack=all.
> > >
> > > I mean, for a single topic, cluster administrators can't enforce
> messages
> > > to be stores in a minimum number of replicas without coordinating with
> > all
> > > producers to that topic so all of them use ack=all.
> > >
> > > Is there something that I am missing? Is there any other strategy to
> > > overcome this situation?
> > >
> > > Regards
> > > Luciano
> > >
> >
>

Jenkins build is back to normal : kafka-trunk-jdk7 #1886

[Apache Jenkins Server]

See