Build failed in Jenkins: kafka-0.10.1-jdk7 #56

[Apache Jenkins Server]

See 

Changes:

[jason] KAFKA-4274; Consumer `offsetForTimes` times out on empty map

--
[...truncated 7141 lines...]
kafka.server.ServerGenerateBrokerIdTest > testMultipleLogDirsMetaProps STARTED

kafka.server.ServerGenerateBrokerIdTest > testMultipleLogDirsMetaProps PASSED

kafka.server.ServerGenerateBrokerIdTest > testDisableGeneratedBrokerId STARTED

kafka.server.ServerGenerateBrokerIdTest > testDisableGeneratedBrokerId PASSED

kafka.server.ServerGenerateBrokerIdTest > testUserConfigAndGeneratedBrokerId 
STARTED

kafka.server.ServerGenerateBrokerIdTest > testUserConfigAndGeneratedBrokerId 
PASSED

kafka.server.ServerGenerateBrokerIdTest > 
testConsistentBrokerIdFromUserConfigAndMetaProps STARTED

kafka.server.ServerGenerateBrokerIdTest > 
testConsistentBrokerIdFromUserConfigAndMetaProps PASSED

kafka.server.ThrottledResponseExpirationTest > testThrottledRequest STARTED

kafka.server.ThrottledResponseExpirationTest > testThrottledRequest PASSED

kafka.server.ThrottledResponseExpirationTest > testExpire STARTED

kafka.server.ThrottledResponseExpirationTest > testExpire PASSED

kafka.server.ServerGenerateClusterIdTest > 
testAutoGenerateClusterIdForKafkaClusterParallel STARTED

kafka.server.ServerGenerateClusterIdTest > 
testAutoGenerateClusterIdForKafkaClusterParallel PASSED

kafka.server.ServerGenerateClusterIdTest > testAutoGenerateClusterId STARTED

kafka.server.ServerGenerateClusterIdTest > testAutoGenerateClusterId PASSED

kafka.server.ServerGenerateClusterIdTest > 
testAutoGenerateClusterIdForKafkaClusterSequential STARTED

kafka.server.ServerGenerateClusterIdTest > 
testAutoGenerateClusterIdForKafkaClusterSequential PASSED

kafka.server.SslReplicaFetchTest > testReplicaFetcherThread STARTED

kafka.server.SslReplicaFetchTest > testReplicaFetcherThread PASSED

kafka.server.AbstractFetcherThreadTest > testConsumerLagRemovedWithPartition 
STARTED

kafka.server.AbstractFetcherThreadTest > testConsumerLagRemovedWithPartition 
PASSED

kafka.server.AbstractFetcherThreadTest > testMetricsRemovedOnShutdown STARTED

kafka.server.AbstractFetcherThreadTest > testMetricsRemovedOnShutdown PASSED

kafka.server.ServerShutdownTest > testCleanShutdownAfterFailedStartup STARTED

kafka.server.ServerShutdownTest > testCleanShutdownAfterFailedStartup PASSED

kafka.server.ServerShutdownTest > testConsecutiveShutdown STARTED

kafka.server.ServerShutdownTest > testConsecutiveShutdown PASSED

kafka.server.ServerShutdownTest > testCleanShutdown STARTED

kafka.server.ServerShutdownTest > testCleanShutdown PASSED

kafka.server.ServerShutdownTest > testCleanShutdownWithDeleteTopicEnabled 
STARTED

kafka.server.ServerShutdownTest > testCleanShutdownWithDeleteTopicEnabled PASSED

kafka.server.KafkaConfigTest > testAdvertiseConfigured STARTED

kafka.server.KafkaConfigTest > testAdvertiseConfigured PASSED

kafka.server.KafkaConfigTest > testLogRetentionTimeHoursProvided STARTED

kafka.server.KafkaConfigTest > testLogRetentionTimeHoursProvided PASSED

kafka.server.KafkaConfigTest > testLogRollTimeBothMsAndHoursProvided STARTED

kafka.server.KafkaConfigTest > testLogRollTimeBothMsAndHoursProvided PASSED

kafka.server.KafkaConfigTest > testLogRetentionValid STARTED

kafka.server.KafkaConfigTest > testLogRetentionValid PASSED

kafka.server.KafkaConfigTest > testSpecificProperties STARTED

kafka.server.KafkaConfigTest > testSpecificProperties PASSED

kafka.server.KafkaConfigTest > testDefaultCompressionType STARTED

kafka.server.KafkaConfigTest > testDefaultCompressionType PASSED

kafka.server.KafkaConfigTest > testDuplicateListeners STARTED

kafka.server.KafkaConfigTest > testDuplicateListeners PASSED

kafka.server.KafkaConfigTest > testLogRetentionUnlimited STARTED

kafka.server.KafkaConfigTest > testLogRetentionUnlimited PASSED

kafka.server.KafkaConfigTest > testLogRetentionTimeMsProvided STARTED

kafka.server.KafkaConfigTest > testLogRetentionTimeMsProvided PASSED

kafka.server.KafkaConfigTest > testLogRollTimeNoConfigProvided STARTED

kafka.server.KafkaConfigTest > testLogRollTimeNoConfigProvided PASSED

kafka.server.KafkaConfigTest > testInvalidInterBrokerSecurityProtocol STARTED

kafka.server.KafkaConfigTest > testInvalidInterBrokerSecurityProtocol PASSED

kafka.server.KafkaConfigTest > testAdvertiseDefaults STARTED

kafka.server.KafkaConfigTest > testAdvertiseDefaults PASSED

kafka.server.KafkaConfigTest > testBadListenerProtocol STARTED

kafka.server.KafkaConfigTest > testBadListenerProtocol PASSED

kafka.server.KafkaConfigTest > testListenerDefaults STARTED

kafka.server.KafkaConfigTest > testListenerDefaults PASSED

kafka.server.KafkaConfigTest > testLogRetentionTimeBothMinutesAndHoursProvided 
STARTED

kafka.server.KafkaConfigTest > testLogRetentionTimeBothMinutesAndHoursProvided 
PASSED

kafka.server.KafkaConfigTest > testUncleanElectionDisabled STARTED

kafka.server.KafkaConfigTest > testUncleanElectionDisabled PASSED

Build failed in Jenkins: kafka-trunk-jdk7 #1612

[Apache Jenkins Server]

See 

Changes:

[jason] KAFKA-4274; Consumer `offsetForTimes` times out on empty map

--
[...truncated 1093 lines...]
kafka.utils.ReplicationUtilsTest > testGetLeaderIsrAndEpochForPartition PASSED

kafka.utils.timer.TimerTest > testAlreadyExpiredTask STARTED

kafka.utils.timer.TimerTest > testAlreadyExpiredTask PASSED

kafka.utils.timer.TimerTest > testTaskExpiration STARTED

kafka.utils.timer.TimerTest > testTaskExpiration PASSED

kafka.utils.timer.TimerTaskListTest > testAll STARTED

kafka.utils.timer.TimerTaskListTest > testAll PASSED

kafka.utils.SchedulerTest > testMockSchedulerNonPeriodicTask STARTED

kafka.utils.SchedulerTest > testMockSchedulerNonPeriodicTask PASSED

kafka.utils.SchedulerTest > testMockSchedulerPeriodicTask STARTED

kafka.utils.SchedulerTest > testMockSchedulerPeriodicTask PASSED

kafka.utils.SchedulerTest > testNonPeriodicTask STARTED

kafka.utils.SchedulerTest > testNonPeriodicTask PASSED

kafka.utils.SchedulerTest > testRestart STARTED

kafka.utils.SchedulerTest > testRestart PASSED

kafka.utils.SchedulerTest > testReentrantTaskInMockScheduler STARTED

kafka.utils.SchedulerTest > testReentrantTaskInMockScheduler PASSED

kafka.utils.SchedulerTest > testPeriodicTask STARTED

kafka.utils.SchedulerTest > testPeriodicTask PASSED

kafka.utils.CommandLineUtilsTest > testParseEmptyArg STARTED

kafka.utils.CommandLineUtilsTest > testParseEmptyArg PASSED

kafka.utils.CommandLineUtilsTest > testParseSingleArg STARTED

kafka.utils.CommandLineUtilsTest > testParseSingleArg PASSED

kafka.utils.CommandLineUtilsTest > testParseArgs STARTED

kafka.utils.CommandLineUtilsTest > testParseArgs PASSED

kafka.utils.CommandLineUtilsTest > testParseEmptyArgAsValid STARTED

kafka.utils.CommandLineUtilsTest > testParseEmptyArgAsValid PASSED

kafka.utils.UtilsTest > testGenerateUuidAsBase64 STARTED

kafka.utils.UtilsTest > testGenerateUuidAsBase64 PASSED

kafka.utils.UtilsTest > testAbs STARTED

kafka.utils.UtilsTest > testAbs PASSED

kafka.utils.UtilsTest > testReplaceSuffix STARTED

kafka.utils.UtilsTest > testReplaceSuffix PASSED

kafka.utils.UtilsTest > testCircularIterator STARTED

kafka.utils.UtilsTest > testCircularIterator PASSED

kafka.utils.UtilsTest > testReadBytes STARTED

kafka.utils.UtilsTest > testReadBytes PASSED

kafka.utils.UtilsTest > testCsvList STARTED

kafka.utils.UtilsTest > testCsvList PASSED

kafka.utils.UtilsTest > testReadInt STARTED

kafka.utils.UtilsTest > testReadInt PASSED

kafka.utils.UtilsTest > testUrlSafeBase64EncodeUUID STARTED

kafka.utils.UtilsTest > testUrlSafeBase64EncodeUUID PASSED

kafka.utils.UtilsTest > testCsvMap STARTED

kafka.utils.UtilsTest > testCsvMap PASSED

kafka.utils.UtilsTest > testInLock STARTED

kafka.utils.UtilsTest > testInLock PASSED

kafka.utils.UtilsTest > testSwallow STARTED

kafka.utils.UtilsTest > testSwallow PASSED

kafka.utils.JsonTest > testJsonEncoding STARTED

kafka.utils.JsonTest > testJsonEncoding PASSED

kafka.utils.IteratorTemplateTest > testIterator STARTED

kafka.utils.IteratorTemplateTest > testIterator PASSED

kafka.metrics.KafkaTimerTest > testKafkaTimer STARTED

kafka.metrics.KafkaTimerTest > testKafkaTimer PASSED

kafka.metrics.MetricsTest > testMetricsReporterAfterDeletingTopic STARTED

kafka.metrics.MetricsTest > testMetricsReporterAfterDeletingTopic PASSED

kafka.metrics.MetricsTest > 
testBrokerTopicMetricsUnregisteredAfterDeletingTopic STARTED

kafka.metrics.MetricsTest > 
testBrokerTopicMetricsUnregisteredAfterDeletingTopic PASSED

kafka.metrics.MetricsTest > testClusterIdMetric STARTED

kafka.metrics.MetricsTest > testClusterIdMetric PASSED

kafka.metrics.MetricsTest > testMetricsLeak STARTED

kafka.metrics.MetricsTest > testMetricsLeak PASSED

kafka.consumer.TopicFilterTest > testWhitelists STARTED

kafka.consumer.TopicFilterTest > testWhitelists PASSED

kafka.consumer.TopicFilterTest > 
testWildcardTopicCountGetTopicCountMapEscapeJson STARTED

kafka.consumer.TopicFilterTest > 
testWildcardTopicCountGetTopicCountMapEscapeJson PASSED

kafka.consumer.TopicFilterTest > testBlacklists STARTED

kafka.consumer.TopicFilterTest > testBlacklists PASSED

kafka.consumer.PartitionAssignorTest > testRoundRobinPartitionAssignor STARTED

kafka.consumer.PartitionAssignorTest > testRoundRobinPartitionAssignor PASSED

kafka.consumer.PartitionAssignorTest > testRangePartitionAssignor STARTED

kafka.consumer.PartitionAssignorTest > testRangePartitionAssignor PASSED

kafka.consumer.ZookeeperConsumerConnectorTest > testBasic STARTED

kafka.consumer.ZookeeperConsumerConnectorTest > testBasic PASSED

kafka.consumer.ZookeeperConsumerConnectorTest > testCompressionSetConsumption 
STARTED

kafka.consumer.ZookeeperConsumerConnectorTest > testCompressionSetConsumption 
PASSED

kafka.consumer.ZookeeperConsumerConnectorTest > testLeaderSelectionForPartition 
STARTED

kafka.consumer.ZookeeperConsumerConnectorTest > testLeaderSelectionForPartition 

Build failed in Jenkins: kafka-trunk-jdk8 #957

[Apache Jenkins Server]

See 

Changes:

[jason] KAFKA-4274; Consumer `offsetForTimes` times out on empty map

--
[...truncated 1093 lines...]
kafka.utils.ReplicationUtilsTest > testUpdateLeaderAndIsr STARTED

kafka.utils.ReplicationUtilsTest > testUpdateLeaderAndIsr PASSED

kafka.utils.ReplicationUtilsTest > testGetLeaderIsrAndEpochForPartition STARTED

kafka.utils.ReplicationUtilsTest > testGetLeaderIsrAndEpochForPartition PASSED

kafka.utils.timer.TimerTest > testAlreadyExpiredTask STARTED

kafka.utils.timer.TimerTest > testAlreadyExpiredTask PASSED

kafka.utils.timer.TimerTest > testTaskExpiration STARTED

kafka.utils.timer.TimerTest > testTaskExpiration PASSED

kafka.utils.timer.TimerTaskListTest > testAll STARTED

kafka.utils.timer.TimerTaskListTest > testAll PASSED

kafka.utils.SchedulerTest > testMockSchedulerNonPeriodicTask STARTED

kafka.utils.SchedulerTest > testMockSchedulerNonPeriodicTask PASSED

kafka.utils.SchedulerTest > testMockSchedulerPeriodicTask STARTED

kafka.utils.SchedulerTest > testMockSchedulerPeriodicTask PASSED

kafka.utils.SchedulerTest > testNonPeriodicTask STARTED

kafka.utils.SchedulerTest > testNonPeriodicTask PASSED

kafka.utils.SchedulerTest > testRestart STARTED

kafka.utils.SchedulerTest > testRestart PASSED

kafka.utils.SchedulerTest > testReentrantTaskInMockScheduler STARTED

kafka.utils.SchedulerTest > testReentrantTaskInMockScheduler PASSED

kafka.utils.SchedulerTest > testPeriodicTask STARTED

kafka.utils.SchedulerTest > testPeriodicTask PASSED

kafka.utils.CommandLineUtilsTest > testParseEmptyArg STARTED

kafka.utils.CommandLineUtilsTest > testParseEmptyArg PASSED

kafka.utils.CommandLineUtilsTest > testParseSingleArg STARTED

kafka.utils.CommandLineUtilsTest > testParseSingleArg PASSED

kafka.utils.CommandLineUtilsTest > testParseArgs STARTED

kafka.utils.CommandLineUtilsTest > testParseArgs PASSED

kafka.utils.CommandLineUtilsTest > testParseEmptyArgAsValid STARTED

kafka.utils.CommandLineUtilsTest > testParseEmptyArgAsValid PASSED

kafka.utils.UtilsTest > testGenerateUuidAsBase64 STARTED

kafka.utils.UtilsTest > testGenerateUuidAsBase64 PASSED

kafka.utils.UtilsTest > testAbs STARTED

kafka.utils.UtilsTest > testAbs PASSED

kafka.utils.UtilsTest > testReplaceSuffix STARTED

kafka.utils.UtilsTest > testReplaceSuffix PASSED

kafka.utils.UtilsTest > testCircularIterator STARTED

kafka.utils.UtilsTest > testCircularIterator PASSED

kafka.utils.UtilsTest > testReadBytes STARTED

kafka.utils.UtilsTest > testReadBytes PASSED

kafka.utils.UtilsTest > testCsvList STARTED

kafka.utils.UtilsTest > testCsvList PASSED

kafka.utils.UtilsTest > testReadInt STARTED

kafka.utils.UtilsTest > testReadInt PASSED

kafka.utils.UtilsTest > testUrlSafeBase64EncodeUUID STARTED

kafka.utils.UtilsTest > testUrlSafeBase64EncodeUUID PASSED

kafka.utils.UtilsTest > testCsvMap STARTED

kafka.utils.UtilsTest > testCsvMap PASSED

kafka.utils.UtilsTest > testInLock STARTED

kafka.utils.UtilsTest > testInLock PASSED

kafka.utils.UtilsTest > testSwallow STARTED

kafka.utils.UtilsTest > testSwallow PASSED

kafka.utils.JsonTest > testJsonEncoding STARTED

kafka.utils.JsonTest > testJsonEncoding PASSED

kafka.utils.IteratorTemplateTest > testIterator STARTED

kafka.utils.IteratorTemplateTest > testIterator PASSED

kafka.metrics.KafkaTimerTest > testKafkaTimer STARTED

kafka.metrics.KafkaTimerTest > testKafkaTimer PASSED

kafka.metrics.MetricsTest > testMetricsReporterAfterDeletingTopic STARTED

kafka.metrics.MetricsTest > testMetricsReporterAfterDeletingTopic PASSED

kafka.metrics.MetricsTest > 
testBrokerTopicMetricsUnregisteredAfterDeletingTopic STARTED

kafka.metrics.MetricsTest > 
testBrokerTopicMetricsUnregisteredAfterDeletingTopic PASSED

kafka.metrics.MetricsTest > testClusterIdMetric STARTED

kafka.metrics.MetricsTest > testClusterIdMetric PASSED

kafka.metrics.MetricsTest > testMetricsLeak STARTED

kafka.metrics.MetricsTest > testMetricsLeak PASSED

kafka.consumer.TopicFilterTest > testWhitelists STARTED

kafka.consumer.TopicFilterTest > testWhitelists PASSED

kafka.consumer.TopicFilterTest > 
testWildcardTopicCountGetTopicCountMapEscapeJson STARTED

kafka.consumer.TopicFilterTest > 
testWildcardTopicCountGetTopicCountMapEscapeJson PASSED

kafka.consumer.TopicFilterTest > testBlacklists STARTED

kafka.consumer.TopicFilterTest > testBlacklists PASSED

kafka.consumer.PartitionAssignorTest > testRoundRobinPartitionAssignor STARTED

kafka.consumer.PartitionAssignorTest > testRoundRobinPartitionAssignor PASSED

kafka.consumer.PartitionAssignorTest > testRangePartitionAssignor STARTED

kafka.consumer.PartitionAssignorTest > testRangePartitionAssignor PASSED

kafka.consumer.ZookeeperConsumerConnectorTest > testBasic STARTED

kafka.consumer.ZookeeperConsumerConnectorTest > testBasic PASSED

kafka.consumer.ZookeeperConsumerConnectorTest > testCompressionSetConsumption 
STARTED

kafka.consumer.ZookeeperConsumerConnectorTest > 

Re: [DISCUSS] KIP-47 - Add timestamp-based log deletion policy

[Becket Qin]

Hi Bill,

I saw the voting thread and think it may be better to discuss this in the
discussion thread.

It would be good to have the KIP wiki to clarify the behavior when
log.retention.ms, log.retention.bytes and log.retention.min.timestamp are
all set.
For example, if the size of the partition has reached beyond
log.retention.bytes but the timestamp has not reached
log.retention.min.timestamp,
do we delete the segment?

Thanks,

Jiangjie (Becket) Qin

On Fri, Jun 3, 2016 at 11:02 AM, Magnus Edenhill  wrote:

> Bumping this thread so Wes can reply to it. Ignore this mail.
>
> 2016-02-24 0:36 GMT+01:00 Joel Koshy :
>
> > Great - thanks for clarifying.
> >
> > Joel
> >
> > On Tue, Feb 23, 2016 at 1:47 PM, Bill Warshaw 
> wrote:
> >
> > > Sorry that I didn't see this comment before the meeting Joel.  I'll try
> > to
> > > clarify what I said at the meeting:
> > >
> > > - The KIP currently states that timestamp-based log deletion will only
> > work
> > > with LogAppendTime.  I need to update the KIP to reflect that, after
> the
> > > work is done for KIP-33, it will work with both LogAppendTime and
> > > CreateTime.
> > > - To use the existing time-based retention mechanism to delete a
> precise
> > > range of messages, a client application would need to do the following:
> > >   - by default, turn off these retention mechanisms
> > >   - when the application wishes to delete a range of messages which
> were
> > > sent before a certain time, compute an approximate value to set
> > > "log.retention.minutes" to, to create a window of messages based on
> that
> > > timestamp that are ok to delete.  There is some degree of imprecision
> > > implied here.
> > >   - wait until we are confident that the log retention mechanism has
> been
> > > run and deleted any stale segments
> > >   - reset "log.retention.minutes" to turn off time-based log retention
> > > until the next time the client application wants to delete something
> > >
> > > - To use the proposed timestamp-based retention mechanism, there is
> only
> > > one step: the application just has to set "log.retention.min.timestamp"
> > to
> > > whatever time boundary it deems fit.  It doesn't need to compute any
> > fuzzy
> > > windows, try to wait until asynchronous processes have been completed
> or
> > > continually flip settings between enabled and disabled.
> > >
> > > I will update the KIP to reflect the discussion around LogAppendTime vs
> > > CreateTime and the work being done in KIP-33.
> > >
> > > Thanks,
> > > Bill
> > >
> > >
> > > On Tue, Feb 23, 2016 at 1:22 PM, Joel Koshy 
> wrote:
> > >
> > > > I'm having some trouble reconciling the current proposal with your
> > > original
> > > > requirement which was essentially being able to purge log data up to
> a
> > > > precise point (an offset). The KIP currently suggests that
> > > timestamp-based
> > > > deletion would only work with LogAppendTime, so it does not seem
> > > > significantly different from time-based retention (after KIP-32/33) -
> > IOW
> > > > to me it appears that you would need to use CreateTime and not
> > > > LogAppendTime. Also one of the rejected alternatives observes that
> > > changing
> > > > the existing configuration settings to try to flush ranges of a given
> > > > partition's log are problematic, but it seems to me you would have to
> > do
> > > > this in with timestamp-based deletion as well right? I think it would
> > be
> > > > useful for me if you or anyone else can go over the exact
> > > > mechanics/workflow for accomplishing precise purges at today's KIP
> > > meeting.
> > > >
> > > > Thanks,
> > > >
> > > > Joel
> > > >
> > > > On Monday, February 22, 2016, Bill Warshaw 
> > wrote:
> > > >
> > > > > Sounds good.  I'll hold off on sending out a VOTE thread until
> after
> > > the
> > > > > KIP meeting tomorrow.
> > > > >
> > > > > On Mon, Feb 22, 2016 at 12:56 PM, Becket Qin  >
> > > > wrote:
> > > > >
> > > > > > Hi Jun,
> > > > > >
> > > > > > I think it makes sense to implement KIP-47 after KIP-33 so we can
> > > make
> > > > it
> > > > > > work for both LogAppendTime and CreateTime.
> > > > > >
> > > > > > And yes, I'm actively working on KIP-33. I had a voting thread on
> > > > KIP-33
> > > > > > before and I'll bump it up.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jiangjie (Becket) Qin
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Feb 22, 2016 at 9:11 AM, Jun Rao 
> wrote:
> > > > > >
> > > > > > > Becket,
> > > > > > >
> > > > > > > Since you submitted KIP-33, are you actively working on that?
> If
> > > so,
> > > > it
> > > > > > > would make sense to implement KIP-47 after KIP-33 so that it
> > works
> > > > for
> > > > > > both
> > > > > > > CreateTime and LogAppendTime.
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > >
> > > > > > >
> > > 

[jira] [Resolved] (KAFKA-4274) KafkaConsumer.offsetsForTimes() hangs and times out on an empty map

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-4274.

Resolution: Fixed

Issue resolved by pull request 1993
[https://github.com/apache/kafka/pull/1993]

> KafkaConsumer.offsetsForTimes() hangs and times out on an empty map
> ---
>
> Key: KAFKA-4274
> URL: https://issues.apache.org/jira/browse/KAFKA-4274
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.1.0
>Reporter: Jun Rao
>Assignee: Jiangjie Qin
> Fix For: 0.10.1.0
>
>
> The following call will hang for the processing timeout in the consumer.
>consumer.offsetsForTimes(new util.HashMap[TopicPartition, 
> java.lang.Long]())



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4274) KafkaConsumer.offsetsForTimes() hangs and times out on an empty map

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15556903#comment-15556903
 ] 

ASF GitHub Bot commented on KAFKA-4274:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1993


> KafkaConsumer.offsetsForTimes() hangs and times out on an empty map
> ---
>
> Key: KAFKA-4274
> URL: https://issues.apache.org/jira/browse/KAFKA-4274
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.1.0
>Reporter: Jun Rao
>Assignee: Jiangjie Qin
> Fix For: 0.10.1.0
>
>
> The following call will hang for the processing timeout in the consumer.
>consumer.offsetsForTimes(new util.HashMap[TopicPartition, 
> java.lang.Long]())



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1993: KAFKA-4274: offsetsForTimes() hang on empty map.

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1993


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: kafka-trunk-jdk7 #1611

[Apache Jenkins Server]

See 

Changes:

[wangguoz] HOTFIX: recreate state.dir after cleanup

--
[...truncated 7844 lines...]

kafka.integration.SslTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup STARTED

kafka.integration.SslTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.integration.SslTopicMetadataTest > testBasicTopicMetadata STARTED

kafka.integration.SslTopicMetadataTest > testBasicTopicMetadata PASSED

kafka.integration.SslTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication STARTED

kafka.integration.SslTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication PASSED

kafka.integration.SslTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown STARTED

kafka.integration.SslTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooLow 
STARTED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooLow PASSED

kafka.integration.AutoOffsetResetTest > testResetToEarliestWhenOffsetTooLow 
STARTED

kafka.integration.AutoOffsetResetTest > testResetToEarliestWhenOffsetTooLow 
PASSED

kafka.integration.AutoOffsetResetTest > testResetToEarliestWhenOffsetTooHigh 
STARTED

kafka.integration.AutoOffsetResetTest > testResetToEarliestWhenOffsetTooHigh 
PASSED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooHigh 
STARTED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooHigh 
PASSED

kafka.integration.MinIsrConfigTest > testDefaultKafkaConfig STARTED

kafka.integration.MinIsrConfigTest > testDefaultKafkaConfig PASSED

kafka.integration.UncleanLeaderElectionTest > testUncleanLeaderElectionEnabled 
STARTED

kafka.integration.UncleanLeaderElectionTest > testUncleanLeaderElectionEnabled 
PASSED

kafka.integration.UncleanLeaderElectionTest > 
testCleanLeaderElectionDisabledByTopicOverride STARTED

kafka.integration.UncleanLeaderElectionTest > 
testCleanLeaderElectionDisabledByTopicOverride PASSED

kafka.integration.UncleanLeaderElectionTest > testUncleanLeaderElectionDisabled 
STARTED

kafka.integration.UncleanLeaderElectionTest > testUncleanLeaderElectionDisabled 
PASSED

kafka.integration.UncleanLeaderElectionTest > 
testUncleanLeaderElectionInvalidTopicOverride STARTED

kafka.integration.UncleanLeaderElectionTest > 
testUncleanLeaderElectionInvalidTopicOverride PASSED

kafka.integration.UncleanLeaderElectionTest > 
testUncleanLeaderElectionEnabledByTopicOverride STARTED

kafka.integration.UncleanLeaderElectionTest > 
testUncleanLeaderElectionEnabledByTopicOverride PASSED

kafka.integration.FetcherTest > testFetcher STARTED

kafka.integration.FetcherTest > testFetcher PASSED

kafka.integration.SaslSslTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack STARTED

kafka.integration.SaslSslTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack PASSED

kafka.integration.SaslSslTopicMetadataTest > testAutoCreateTopicWithCollision 
STARTED

kafka.integration.SaslSslTopicMetadataTest > testAutoCreateTopicWithCollision 
PASSED

kafka.integration.SaslSslTopicMetadataTest > testAliveBrokerListWithNoTopics 
STARTED

kafka.integration.SaslSslTopicMetadataTest > testAliveBrokerListWithNoTopics 
PASSED

kafka.integration.SaslSslTopicMetadataTest > testAutoCreateTopic STARTED

kafka.integration.SaslSslTopicMetadataTest > testAutoCreateTopic PASSED

kafka.integration.SaslSslTopicMetadataTest > testGetAllTopicMetadata STARTED

kafka.integration.SaslSslTopicMetadataTest > testGetAllTopicMetadata PASSED

kafka.integration.SaslSslTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup STARTED

kafka.integration.SaslSslTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.integration.SaslSslTopicMetadataTest > testBasicTopicMetadata STARTED

kafka.integration.SaslSslTopicMetadataTest > testBasicTopicMetadata PASSED

kafka.integration.SaslSslTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication STARTED

kafka.integration.SaslSslTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication PASSED

kafka.integration.SaslSslTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown STARTED

kafka.integration.SaslSslTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED

kafka.utils.ZkUtilsTest > testAbortedConditionalDeletePath STARTED

kafka.utils.ZkUtilsTest > testAbortedConditionalDeletePath PASSED

kafka.utils.ZkUtilsTest > testSuccessfulConditionalDeletePath STARTED

kafka.utils.ZkUtilsTest > testSuccessfulConditionalDeletePath PASSED

kafka.utils.ZkUtilsTest > testClusterIdentifierJsonParsing STARTED

kafka.utils.ZkUtilsTest > testClusterIdentifierJsonParsing PASSED

kafka.utils.timer.TimerTaskListTest > testAll STARTED

kafka.utils.timer.TimerTaskListTest > testAll PASSED

[jira] [Created] (KAFKA-4276) REST configuration not visible in connector properties config files

[Gwen Shapira (JIRA)]

Gwen Shapira created KAFKA-4276:
---

 Summary: REST configuration not visible in connector properties 
config files
 Key: KAFKA-4276
 URL: https://issues.apache.org/jira/browse/KAFKA-4276
 Project: Kafka
  Issue Type: Improvement
  Components: KafkaConnect
Reporter: Gwen Shapira
Assignee: Ewen Cheslack-Postava


REST host and port configs are not visible in connect-distributed.properties. 
I think this leads to some confusion as users don't know there's even a REST 
port and need to read the docs to find about it and the default (and these are 
marked as LOW configs).

We can easily improve that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-80: Kafka REST Server

[Andrew Schofield]

There's a massive difference between the governance of Kafka and the governance 
of the REST proxy.

In Kafka, there is a broad community of people contributing their opinions 
about future enhancements in the form of KIPs. There's some really deep 
consideration that goes into some of the trickier KIPs. There are people 
outside Confluent deeply knowledgeable  in Kafka and building the reputations 
to become committers. I get the impression that the roadmap of Kafka is not 
really community-owned (what's the big feature for Kafka 0.11, for example), 
but the conveyor belt of smaller features in the form of KIPs works  nicely. 
It's a good example of open-source working well.

The equivalent for the REST proxy is basically issues on GitHub. The roadmap is 
less clear. There's not really a community properly engaged in the way that 
there is with Kafka. So, you could say that it's clear that fewer people are 
interested, but I think  the whole governance thing is a big barrier to 
engagement. And it's looking like it's getting out of date.

In technical terms, I can think of two big improvements to the REST proxy. 
First, it needs to use the new consumer API so that it's possible to secure 
connections between the REST proxy and Kafka. I don't care too much which 
method calls it uses actually  uses to consume messages, but I do care that I 
cannot secure connections because of network security rules. Second, there's an 
affinity between a consumer and the instance of the REST proxy to which it 
first connected. Kafka itself avoids this kind of affinity for good reason, and 
in the name of availability the REST proxy should too. These are natural KIPs.

I think it would be good to have the code for the REST proxy contributed to 
Apache Kafka so that it would be able to be developed in the same way.

Andrew Schofield
  
From: Suresh Srinivas 
Sent: 07 October 2016 22:41:52
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-80: Kafka REST Server
    
ASF already gives us a clear framework and governance model for community
development. This is already understood by the people contributing to
Apache Kafka project, and they are the same people who want to contribute
to the REST server capability as well. Everyone is in agreement on the
need for collaborating on this effort. So why not contribute the code to
Apache Kafka. This will help avoid duplication of effort and forks that
may crop up, hugely benefitting the user community. This will also avoid
having to define a process similar to ASF on a GitHub project and instead
there is a single community with clear understanding community process as
defined in ASF.

As others have said, this is an important capability for Apache Kafka. It
is worth maintaining this as a part of the project.

Regards,
Suresh

On 10/6/16, 8:32 AM, "Ofir Manor"  wrote:

>I personally think it would be quite wasteful to re-implement the REST
>gateway just because that an actively-maintained piece of Apache-licensed
>software is not governed directly by the Apache Kafka community... While
>kafka-rest repo is owned by Confluent, the contributors including the main
>one are also part of the Apache Kafka  community, so there is a chance to
>work this out.
>
>However, there are two valid concerns here that could be addressed, around
>community and accessibility:
>>> What we are worried about is a project
>>> that's not maintained in a community. So the process of accepting
>>>patches
>>> and priorities is not clear, and it's not developed in Apache
>>>community.
>>> Not only that, existing REST API project doesn't support new client API
>and
>>> hence there is no security support either.
>
>This might be easy to fix. Maybe Confluent / kafka-rest community can
>clarify that - what is their contribution policy, dev style, roadmap etc.
>If they want, they can make an effort to encourage participation from
>people outside Confluent (easily accept contributions, invite external
>commiters or have open dev process similar to Apache Kafka etc), as there
>is definitely seems to be some interest on the list. That might clear the
>community concern and help kafka-rest project (but that is a calculation
>Confluent will have to make).
>
>The other, independent, concern is that REST is something that is expected
>to be available out of the box with Kafka. I personally don't feel
>strongly
>about it (better use proper, efficient APIs from day one), though it is
>definitely way smaller than adding a stream processing engine to the
>project :)
>Again,the kafka-rest "community" could take steps to make it even easier
>to
>install, configure and run kafka-rest for new users on vanilla Apache
>Kafka
>(outside the Confluent platform), if they wish that (or welcome
>contributions to that end), but that is up to them.
>Finally, if after the above steps were taken there would still a strong
>desire to include a great rest gateway with Apache Kafka, I assume the

[jira] [Commented] (KAFKA-3410) Unclean leader election and "Halting because log truncation is not allowed"

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15556670#comment-15556670
 ] 

Jun Rao commented on KAFKA-3410:


[~james cheng], the proposal in KAFKA-1211 should prevent the halting described 
in this jira. By knowing the leader generation, we can sync up the replicas 
better even when the log is manually deleted or truncated. The data loss won't 
be avoided though since at the Kafka level, we don't know the log is manually 
changed.

> Unclean leader election and "Halting because log truncation is not allowed"
> ---
>
> Key: KAFKA-3410
> URL: https://issues.apache.org/jira/browse/KAFKA-3410
> Project: Kafka
>  Issue Type: Bug
>Reporter: James Cheng
>
> I ran into a scenario where one of my brokers would continually shutdown, 
> with the error message:
> [2016-02-25 00:29:39,236] FATAL [ReplicaFetcherThread-0-1], Halting because 
> log truncation is not allowed for topic test, Current leader 1's latest 
> offset 0 is less than replica 2's latest offset 151 
> (kafka.server.ReplicaFetcherThread)
> I managed to reproduce it with the following scenario:
> 1. Start broker1, with unclean.leader.election.enable=false
> 2. Start broker2, with unclean.leader.election.enable=false
> 3. Create topic, single partition, with replication-factor 2.
> 4. Write data to the topic.
> 5. At this point, both brokers are in the ISR. Broker1 is the partition 
> leader.
> 6. Ctrl-Z on broker2. (Simulates a GC pause or a slow network) Broker2 gets 
> dropped out of ISR. Broker1 is still the leader. I can still write data to 
> the partition.
> 7. Shutdown Broker1. Hard or controlled, doesn't matter.
> 8. rm -rf the log directory of broker1. (This simulates a disk replacement or 
> full hardware replacement)
> 9. Resume broker2. It attempts to connect to broker1, but doesn't succeed 
> because broker1 is down. At this point, the partition is offline. Can't write 
> to it.
> 10. Resume broker1. Broker1 resumes leadership of the topic. Broker2 attempts 
> to join ISR, and immediately halts with the error message:
> [2016-02-25 00:29:39,236] FATAL [ReplicaFetcherThread-0-1], Halting because 
> log truncation is not allowed for topic test, Current leader 1's latest 
> offset 0 is less than replica 2's latest offset 151 
> (kafka.server.ReplicaFetcherThread)
> I am able to recover by setting unclean.leader.election.enable=true on my 
> brokers.
> I'm trying to understand a couple things:
> * In step 10, why is broker1 allowed to resume leadership even though it has 
> no data?
> * In step 10, why is it necessary to stop the entire broker due to one 
> partition that is in this state? Wouldn't it be possible for the broker to 
> continue to serve traffic for all the other topics, and just mark this one as 
> unavailable?
> * Would it make sense to allow an operator to manually specify which broker 
> they want to become the new master? This would give me more control over how 
> much data loss I am willing to handle. In this case, I would want broker2 to 
> become the new master. Or, is that possible and I just don't know how to do 
> it?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-82 - Add Record Headers

[Nacho Solis]

On Fri, Oct 7, 2016 at 8:45 AM, Jay Kreps  wrote:

> This discussion has come up a number of times and we've always passed.
>

​Hopefully this time the arguments will be convincing enough that Kafka can
decide to do something about it.
​



> One of things that has helped keep Kafka simple is not adding in new
> abstractions and concepts except when the proposal is really elegant and
> makes things simpler.
>

​I completely agree that we want things to be simple and elegant.  This is
exactly what headers provide.

Headers are a clean way to extend the system without sacrificing
performance or elegance. The are modular and backwards compatible.

​​

> Consider three use cases for headers:
>
>
> ​​
>  1. Kafka-scope: We want to add a feature to Kafka that needs a
>particular field.
>

​This is a _great_ use case for Kafka headers.  Having headers means that
you can have features that are optional.  Features that are slowly deployed
without needing to move everybody from one protocol version to another
protocol version. All clients don't have to change and all brokers don't
have to change.

Without headers you need to parse the messages differently.  With headers
you use the same parser.
I assume I don't need to get into how this makes the system extensible
without requiring others to use the same extensions you have.

​


>2. Company-scope: You want to add a header to be shared by everyone in
>your company.
>

​It is completely true that for client-side things you don't need a
architectural header system.  You could just write a wrapper and
encapsulate every message you send. You could achieve end-to-end.  Even if
this end-to-end exists, Kafka currently offers no way to identify the type
of a message (which I wish we could change), so we have to rely on some
magic number to identify the type. Once we have that we can have a header
system.

Avro is useful for encoding schema based systems, but it's not as useful
for modularity and it's not universal. We have a number of use cases that
don't use avro (and don't want to). They want to send binary data, but from
an org perspective still need some structure to be added for accounting,
tracing, auditing, security, etc.   There is some of this data that would
also be useful at the broker side. This is somewhat problematic at this
point (say, using a client side wrapper).



>3. World-wide scope: You are building a third party tool and want to add
>some kind of header.
>

​I understand that you see 3 as a niche case, trying to build a third party
tool. For us this is being a good community citizen.  Let's say that we
have a plugin for large-message support. If we wanted to make that
available to the community (as good citizens would), we could make our
header module open source and others could re-use it.  Why have to
re-implement something?   The same is true if some company decided to write
a geo-location header and we wanted to use it for some mobile product.  At
this point, it seems that at least a few organizations are looking for a
plugin system and it's likely that they'll have similar requirements.  For
example. it's possible many IoT companies would need similar features, or
maybe the self-driving cars need similar features, etc.  Something that
would benefit a community at large even if it didn't benefit all users.  So
maybe LinkedIn wouldn't care about the self-driving car style features but
we could care about the security features being worked on at BBVA.



>  1. A global registry of numeric keys is super super ugly. This seems
>silly compared to the Avro (or whatever) header solution which gives
> more
>compact encoding, rich types, etc.
>

​This seems like a perfectly reasonable thing to discuss.  I'm in favor of
this.  Avro is problematic for this, it implies you know the schema in
advance. You can't easily compose things. The richness of the types is a
matter of serialization so this would be a mute point. If you really wanted
avro, you could encode an avro object inside one of the headers and the
total overhead would be small.

Numeric ints as keys are used by many network protocols as an efficient way
to define the type of data carried. They have proven themselves.

As for keeping a registry, this is a simple thing. We already keep multiple
"registries", the Kafka ApiKeys and Error Codes are things we already
maintain.  Not to mention the "registries" of all the config variables.

​


>2. Using byte arrays for header values means they aren't really
>interoperable for case (3). E.g. I can't make a UI that displays
> headers,
>or allow you to set them in config. To work with third party headers,
> the
>only case I think this really helps, you need the union of all
>serialization schemes people have used for any tool.
>

​Byte arrays are serialized by the plugin in question.  If you don't have
that plugin (or the code to handle that specific header) then you won't
know what the data 

Re: [VOTE] 0.10.1.0 RC0

[Jason Gustafson]

Quick update: I'm planning to cut a new RC on Monday due to
https://issues.apache.org/jira/browse/KAFKA-4274. If you discover any new
problems in the meantime, let me know on this thread.

Thanks,
Jason

On Fri, Oct 7, 2016 at 9:36 AM, Vahid S Hashemian  wrote:

> Jason,
>
> Sure, I'll submit a patch for the trivial changes in the quick start.
> Do you recommend adding Windows version of commands along with the current
> commands?
>
> I'll also open a JIRA for the new consumer issue.
>
> --Vahid
>
>
>
> From:   Jason Gustafson 
> To: dev@kafka.apache.org
> Cc: Kafka Users 
> Date:   10/07/2016 08:57 AM
> Subject:Re: [VOTE] 0.10.1.0 RC0
>
>
>
> @Vahid Thanks, do you want to submit a patch for the quickstart fixes? We
> won't need another RC if it's just doc changes. The exception is a little
> more troubling. Perhaps open a JIRA and we can begin investigation? It's
> especially strange that you say it's specific to the new consumer.
>
> @Henry Actually that issue was resolved as "won't fix" since it pointed to
> an old version of the group coordinator design. But maybe it's misleading
> that we include JIRAs resolved as "won't fix" in the first place. At least
> they ought to be listed in a separate section?
>
> -Jason
>
> On Thu, Oct 6, 2016 at 5:27 PM, Henry Cai 
> wrote:
>
> > Why is this feature in the release note?
> >
> >
> >- [KAFKA-264 ] -
> > Change
> >the consumer side load balancing and distributed co-ordination to use
> a
> >consumer co-ordinator
> >
> > I thought this was already done in 2015.
> >
> > On Thu, Oct 6, 2016 at 4:55 PM, Vahid S Hashemian <
> > vahidhashem...@us.ibm.com
> > > wrote:
> >
> > > Jason,
> > >
> > > Thanks a lot for managing this release.
> > >
> > > I ran the quick start (Steps 2-8) with this release candidate on
> Ubuntu,
> > > Windows, and Mac and they mostly look great.
> > > These are some, hopefully, minor items and gaps I noticed with respect
> to
> > > the existing quick start documentation (and the updated quick start
> that
> > > leverages the new consumer).
> > > They may very well be carryovers from previous releases, or perhaps
> > > specific to my local environments.
> > > Hopefully others can confirm.
> > >
> > >
> > > Windows
> > >
> > > Since there are separate scripts on Windows platform, it probably
> would
> > > help if that is clarified in the quick start section. E.g. "On Windows
> > > platform replace `bin/` with `bin\windows\`". Or even have a separate
> > > quick start for Windows since a number of commands will be different
> on
> > > Windows.
> > > There is no `connect-standalone.sh` equivalent for Windows under
> > > bin\windows folder (Step 7).
> > > Step 8 is also not tailored for Windows terminals. I skipped this
> step.
> > > When I try to consume message using the new consumer (Step 5) I get an
> > > exception on the broker side. The old consumer works fine.
> > >
> > > java.io.IOException: Map failed
> > > at sun.nio.ch.FileChannelImpl.map(Unknown Source)
> > > at kafka.log.AbstractIndex.(AbstractIndex.scala:61)
> > > at kafka.log.OffsetIndex.(OffsetIndex.scala:51)
> > > at kafka.log.LogSegment.(LogSegment.scala:67)
> > > at kafka.log.Log.loadSegments(Log.scala:255)
> > > at kafka.log.Log.(Log.scala:108)
> > > at kafka.log.LogManager.createLog(LogManager.scala:362)
> > > at kafka.cluster.Partition.getOrCreateReplica(Partition.
> > scala:94)
> > > at
> > > kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.
> > > apply(Partition.scala:174)
> > > at
> > > kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.
> > > apply(Partition.scala:174)
> > > at scala.collection.mutable.HashSet.foreach(HashSet.scala:79)
> > > at
> kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:174)
> > > at
> kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:168)
> > > at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
> > > at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:242)
> > > at kafka.cluster.Partition.makeLeader(Partition.scala:168)
> > > at
> > > kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(
> > > ReplicaManager.scala:740)
> > > at
> > > kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(
> > > ReplicaManager.scala:739)
> > > at
> > > scala.collection.mutable.HashMap$$anonfun$foreach$1.
> > > apply(HashMap.scala:98)
> > > at
> > > scala.collection.mutable.HashMap$$anonfun$foreach$1.
> > > apply(HashMap.scala:98)
> > > at
> > > scala.collection.mutable.HashTable$class.foreachEntry(
> > HashTable.scala:226)
> > > at scala.collection.mutable.HashMap.foreachEntry(HashMap.
> > scala:39)
> > > at scala.collection.mutable.HashMap.foreach(HashMap.scala:98)
> > > 

[jira] [Commented] (KAFKA-4274) KafkaConsumer.offsetsForTimes() hangs and times out on an empty map

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15556565#comment-15556565
 ] 

ASF GitHub Bot commented on KAFKA-4274:
---

GitHub user becketqin opened a pull request:

https://github.com/apache/kafka/pull/1993

KAFKA-4274: offsetsForTimes() hang on empty map.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/becketqin/kafka KAFKA-4274

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1993.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1993


commit 8c6a1b7b0c219843f4785078f2d2ca0d0b2e9c24
Author: Jiangjie Qin 
Date:   2016-10-07T23:00:05Z

KAFKA-4274: offsetsForTimes() hang on empty map.




> KafkaConsumer.offsetsForTimes() hangs and times out on an empty map
> ---
>
> Key: KAFKA-4274
> URL: https://issues.apache.org/jira/browse/KAFKA-4274
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.1.0
>Reporter: Jun Rao
>Assignee: Jiangjie Qin
> Fix For: 0.10.1.0
>
>
> The following call will hang for the processing timeout in the consumer.
>consumer.offsetsForTimes(new util.HashMap[TopicPartition, 
> java.lang.Long]())



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1993: KAFKA-4274: offsetsForTimes() hang on empty map.

[becketqin]

GitHub user becketqin opened a pull request:

https://github.com/apache/kafka/pull/1993

KAFKA-4274: offsetsForTimes() hang on empty map.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/becketqin/kafka KAFKA-4274

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1993.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1993


commit 8c6a1b7b0c219843f4785078f2d2ca0d0b2e9c24
Author: Jiangjie Qin 
Date:   2016-10-07T23:00:05Z

KAFKA-4274: offsetsForTimes() hang on empty map.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-4274) KafkaConsumer.offsetsForTimes() hangs and times out on an empty map

[Jiangjie Qin (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15556529#comment-15556529
 ] 

Jiangjie Qin commented on KAFKA-4274:
-

[~junrao] Thanks for reporting the issue. Will submit a patch shortly.

> KafkaConsumer.offsetsForTimes() hangs and times out on an empty map
> ---
>
> Key: KAFKA-4274
> URL: https://issues.apache.org/jira/browse/KAFKA-4274
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.1.0
>Reporter: Jun Rao
>Assignee: Jiangjie Qin
> Fix For: 0.10.1.0
>
>
> The following call will hang for the processing timeout in the consumer.
>consumer.offsetsForTimes(new util.HashMap[TopicPartition, 
> java.lang.Long]())



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (KAFKA-4274) KafkaConsumer.offsetsForTimes() hangs and times out on an empty map

[Jiangjie Qin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jiangjie Qin reassigned KAFKA-4274:
---

Assignee: Jiangjie Qin

> KafkaConsumer.offsetsForTimes() hangs and times out on an empty map
> ---
>
> Key: KAFKA-4274
> URL: https://issues.apache.org/jira/browse/KAFKA-4274
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.1.0
>Reporter: Jun Rao
>Assignee: Jiangjie Qin
> Fix For: 0.10.1.0
>
>
> The following call will hang for the processing timeout in the consumer.
>consumer.offsetsForTimes(new util.HashMap[TopicPartition, 
> java.lang.Long]())



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-84: Support SASL/SCRAM mechanisms

[Gwen Shapira]

Can you talk more about rejecting the option of making the password
store pluggable? I am a bit uncomfortable with making ZK the one and
only password store...

On Tue, Oct 4, 2016 at 6:43 AM, Rajini Sivaram
 wrote:
> Hi all,
>
> I have just created KIP-84 to add SCRAM-SHA-1 and SCRAM-SHA-256 SASL
> mechanisms to Kafka:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-84%3A+Support+SASL+SCRAM+mechanisms
>
>
> Comments and suggestions are welcome.
>
> Thank you...
>
> Regards,
>
> Rajini



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [VOTE] KIP-47 - Add timestamp-based log deletion policy

[Gwen Shapira]

+1 (binding)

On Wed, Oct 5, 2016 at 1:55 PM, Bill Warshaw  wrote:
> Bumping for visibility.  KIP is here:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-47+-+Add+timestamp-based+log+deletion+policy
>
> On Wed, Aug 24, 2016 at 2:32 PM Bill Warshaw  wrote:
>
>> Hello Guozhang,
>>
>> KIP-71 seems unrelated to this KIP.  KIP-47 is just adding a new deletion
>> policy (minimum timestamp), while KIP-71 is allowing deletion and
>> compaction to coexist.
>>
>> They both will touch LogManager, but the change for KIP-47 is very
>> isolated.
>>
>> On Wed, Aug 24, 2016 at 2:21 PM Guozhang Wang  wrote:
>>
>> Hi Bill,
>>
>> I would like to reason if there is any correlation between this KIP and
>> KIP-71
>>
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-71%3A+Enable+log+compaction+and+deletion+to+co-exist
>>
>> I feel they are orthogonal but would like to double check with you.
>>
>>
>> Guozhang
>>
>>
>> On Wed, Aug 24, 2016 at 11:05 AM, Bill Warshaw 
>> wrote:
>>
>> > I'd like to re-awaken this voting thread now that KIP-33 has merged.
>> This
>> > KIP is now completely unblocked.  I have a working branch off of trunk
>> with
>> > my proposed fix, including testing.
>> >
>> > On Mon, May 9, 2016 at 8:30 PM Guozhang Wang  wrote:
>> >
>> > > Jay, Bill:
>> > >
>> > > I'm thinking of one general use case of using timestamp rather than
>> > offset
>> > > for log deletion, which is that for expiration handling in data
>> > > replication, when the source data store decides to expire some data
>> > records
>> > > based on their timestamps, today we need to configure the corresponding
>> > > Kafka changelog topic for compaction, and actively send a tombstone for
>> > > each expired record. Since expiration usually happens with a bunch of
>> > > records, this could generate large tombstone traffic. For example I
>> think
>> > > LI's data replication for Espresso is seeing similar issues and they
>> are
>> > > just not sending tombstone at all.
>> > >
>> > > With timestamp based log deletion policy, this can be easily handled by
>> > > simply setting the current expiration timestamp; but ideally one would
>> > > prefer to configure this topic to be both log compaction enabled as
>> well
>> > as
>> > > log deletion enabled. From that point of view, I feel that current KIP
>> > > still has value to be accepted.
>> > >
>> > > Guozhang
>> > >
>> > >
>> > > On Mon, May 2, 2016 at 2:37 PM, Bill Warshaw 
>> > wrote:
>> > >
>> > > > Yes, I'd agree that offset is a more precise configuration than
>> > > timestamp.
>> > > > If there was a way to set a partition-level configuration, I would
>> > rather
>> > > > use log.retention.min.offset than timestamp.  If you have an approach
>> > in
>> > > > mind I'd be open to investigating it.
>> > > >
>> > > > On Mon, May 2, 2016 at 5:33 PM, Jay Kreps  wrote:
>> > > >
>> > > > > Gotcha, good point. But barring that limitation, you agree that
>> that
>> > > > makes
>> > > > > more sense?
>> > > > >
>> > > > > -Jay
>> > > > >
>> > > > > On Mon, May 2, 2016 at 2:29 PM, Bill Warshaw 
>> > > > wrote:
>> > > > >
>> > > > > > The problem with offset as a config option is that offsets are
>> > > > > > partition-specific, so we'd need a per-partition config.  This
>> > would
>> > > > work
>> > > > > > for our particular use case, where we have single-partition
>> topics,
>> > > but
>> > > > > for
>> > > > > > multiple-partition topics it would delete from all partitions
>> based
>> > > on
>> > > > a
>> > > > > > global topic-level offset.
>> > > > > >
>> > > > > > On Mon, May 2, 2016 at 4:32 PM, Jay Kreps 
>> > wrote:
>> > > > > >
>> > > > > > > I think you are saying you considered a kind of trim() api that
>> > > would
>> > > > > > > synchronously chop off the tail of the log starting from a
>> given
>> > > > > offset.
>> > > > > > > That would be one option, but what I was saying was slightly
>> > > > different:
>> > > > > > in
>> > > > > > > the proposal you have where there is a config that controls
>> > > retention
>> > > > > > that
>> > > > > > > the user would update, wouldn't it make more sense for this
>> > config
>> > > to
>> > > > > be
>> > > > > > > based on offset rather than timestamp?
>> > > > > > >
>> > > > > > > -Jay
>> > > > > > >
>> > > > > > > On Mon, May 2, 2016 at 12:53 PM, Bill Warshaw <
>> > wdwars...@gmail.com
>> > > >
>> > > > > > wrote:
>> > > > > > >
>> > > > > > > > 1.  Initially I looked at using the actual offset, by adding
>> a
>> > > call
>> > > > > to
>> > > > > > > > AdminUtils to just delete anything in a given topic/partition
>> > to
>> > > a
>> > > > > > given
>> > > > > > > > offset.  I ran into a lot of trouble here trying to work out
>> > how
>> > > > the
>> > > > > > > system
>> > > > > > > > would recognize that every broker had successfully deleted
>> that
>> > 

[jira] [Updated] (KAFKA-4275) Check of State-Store-assignment to Processor-Nodes is not enabled

[Matthias J. Sax (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax updated KAFKA-4275:
---
Status: Patch Available  (was: In Progress)

> Check of State-Store-assignment to Processor-Nodes is not enabled
> -
>
> Key: KAFKA-4275
> URL: https://issues.apache.org/jira/browse/KAFKA-4275
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Matthias J. Sax
>Assignee: Matthias J. Sax
>
> In {{ProcessorContextImpl#getStateStores()}} we should check if a store was 
> connected to the processor and thus, if the processor is allowed to access 
> the store. This check is currently disabled.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-85: Dynamic JAAS configuration for Kafka clients

[Gwen Shapira]

Huge +1.

Thank you Rajini for all the hard work on security improvements. With
more Docker deployments of micro-services, getting rid of mandatory
config file will be super important.

Gwen

On Thu, Oct 6, 2016 at 5:43 AM, Edoardo Comar  wrote:
> Hi Rajini
> great improvement and I see you had the code ready ... !
>
> I now think it'd easier to implement a PR for KIP-83 (multiple logins per
> JVM) on top of yours,
> as you have now identified a client property that can be used for caching
> logins.
>
> I'm actually wondering if I caused you to cut down your KIP !!
>
> Also, I think your JIRA encompasses
> https://issues.apache.org/jira/browse/KAFKA-3302
>
> thanks,
> Edo
> --
> Edoardo Comar
> IBM MessageHub
> eco...@uk.ibm.com
> IBM UK Ltd, Hursley Park, SO21 2JN
>
> IBM United Kingdom Limited Registered in England and Wales with number
> 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6
> 3AU
>
>
>
> From:   Rajini Sivaram 
> To: dev@kafka.apache.org
> Date:   06/10/2016 10:49
> Subject:[DISCUSS] KIP-85: Dynamic JAAS configuration for Kafka
> clients
>
>
>
> Hi all,
>
> I have just created KIP-85 to enable JAAS login context for Kafka clients
> to be configured without a physical file:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-85%3A+Dynamic+JAAS+configuration+for+Kafka+clients
>
>
>
> Comments and suggestions are welcome.
>
> Thank you...
>
> Regards,
>
> Rajini
>
>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

[jira] [Commented] (KAFKA-4275) Check of State-Store-assignment to Processor-Nodes is not enabled

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15556448#comment-15556448
 ] 

ASF GitHub Bot commented on KAFKA-4275:
---

GitHub user mjsax opened a pull request:

https://github.com/apache/kafka/pull/1992

KAFKA-4275: Check of State-Store-assignment to Processor-Nodes is not 
enabled



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mjsax/kafka kafka-4275-stateStoreCheck

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1992.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1992


commit 5247403672f399a4631fd629f9077dc181461bbb
Author: Matthias J. Sax 
Date:   2016-10-07T22:29:09Z

KAFKA-4275: Check of State-Store-assignment to Processor-Nodes is not 
enabled




> Check of State-Store-assignment to Processor-Nodes is not enabled
> -
>
> Key: KAFKA-4275
> URL: https://issues.apache.org/jira/browse/KAFKA-4275
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Matthias J. Sax
>Assignee: Matthias J. Sax
>
> In {{ProcessorContextImpl#getStateStores()}} we should check if a store was 
> connected to the processor and thus, if the processor is allowed to access 
> the store. This check is currently disabled.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1992: KAFKA-4275: Check of State-Store-assignment to Pro...

[mjsax]

GitHub user mjsax opened a pull request:

https://github.com/apache/kafka/pull/1992

KAFKA-4275: Check of State-Store-assignment to Processor-Nodes is not 
enabled



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mjsax/kafka kafka-4275-stateStoreCheck

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1992.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1992


commit 5247403672f399a4631fd629f9077dc181461bbb
Author: Matthias J. Sax 
Date:   2016-10-07T22:29:09Z

KAFKA-4275: Check of State-Store-assignment to Processor-Nodes is not 
enabled




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] KIP-82 - Add Record Headers

[Dana Powers]

> I agree with the critique of compaction not having a value. I think we should 
> consider fixing that directly.

Agree that the compaction issue is troubling: compacted "null" deletes
are incompatible w/ headers that must be packed into the message
value. Are there any alternatives on compaction delete semantics that
could address this? The KIP wiki discussion I think mostly assumes
that compaction-delete is what it is and can't be changed/fixed.

-Dana

On Fri, Oct 7, 2016 at 1:38 PM, Michael Pearce  wrote:
>
> Hi Jay,
>
> Thanks for the comments and feedback.
>
> I think its quite clear that if a problem keeps arising then it is clear that 
> it needs resolving, and addressing properly.
>
> Fair enough at linkedIn, and historically for the very first use cases 
> addressing this maybe not have been a big priority. But as Kafka is now 
> Apache open source and being picked up by many including my company, it is 
> clear and evident that this is a requirement and issue that needs to be now 
> addressed to address these needs.
>
> The fact in almost every transport mechanism including networking layers in 
> the enterprise ive worked in, there has always been headers i think clearly 
> shows their need and success for a transport mechanism.
>
> I understand some concerns with regards to impact for others not needing it.
>
> What we are proposing is flexible solution that provides no overhead on 
> storage or network traffic layers if you chose not to use headers, but does 
> enable those who need or want it to use it.
>
>
> On your response to 1), there is nothing saying that it should be put in any 
> faster or without diligence and the same KIP process can still apply for 
> adding kafka-scope headers, having headers, just makes it easier to add, 
> without constant message and record changes. Timestamp is a clear real 
> example of actually what should be in a header (along with other fields) but 
> as such the whole message/record object needed to be changed to add this, as 
> will any further headers deemed needed by kafka.
>
> On response to 2) why within my company as a platforms designer should i 
> enforce that all teams use the same serialization for their payloads? But 
> what i do need is some core cross cutting concerns and information addressed 
> at my platform level and i don't want to impose onto my development teams. 
> This is the same argument why byte[] is the exposed value and key because as 
> a messaging platform you dont want to impose that on my company.
>
> On response to 3) Actually this isnt true, there are many 3rd party tools, we 
> need to hook into our messaging flows that they only build onto standardised 
> interfaces as obviously the cost to have a custom implementation for every 
> company would be very high.
> APM tooling is a clear case in point, every enterprise level APM tool on the 
> market is able to stitch in transaction flow end 2 end over a platform over 
> http, jms because they can stitch in some "magic" data in a 
> uniform/standardised for the two mentioned they stitch this into the headers. 
> It is current form they cannot do this with Kafka. Providing a standardised 
> interface will i believe actually benefit the project as commercial companies 
> like these will now be able to plugin their tooling uniformly, making it 
> attractive and possible.
>
> Some of you other concerns as Joel mentions these are more implementation 
> details, that i think should be agreed upon, but i think can be addressed.
>
> e.g. re your concern on the hashmap.
> it is more than possible not to have every record have to have a hashmap 
> unless it actually has a header (just like we have managed to do on the 
> serialized meesage) so if theres a concern on the in memory record size for 
> those using kafka without headers.
>
> On your second to last comment about every team choosing their own format, 
> actually we do want this a little, as very first mentioned, no we don't want 
> a free for all, but some freedom to have different serialization has 
> different benefits and draw backs across our business. I can iterate these if 
> needed. One of the use case for headers provided by linkedIn on top of my KIP 
> even shows where headers could be beneficial here as a header could be used 
> to detail which data format the message is serialized to allowing me to 
> consume different formats.
>
> Also we have some systems that we need to integrate that pretty near 
> impossible to wrap or touch their binary payloads, or we’re not allowed to 
> touch them (historic system, or inter/intra corporate)
>
> Headers really gives as a solution to provide a pluggable platform, and 
> standardisation that allows users to build platforms that adapt to their 
> needs.
>
>
> Cheers
> Mike
>
>
> 
> From: Jay Kreps 
> Sent: Friday, October 7, 2016 4:45 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] 

[jira] [Resolved] (KAFKA-3302) Pass kerberos keytab and principal as part of client config

[Gwen Shapira (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gwen Shapira resolved KAFKA-3302.
-
Resolution: Duplicate

Closing as duplicate since Rajini already has a PR out.

> Pass kerberos keytab and principal as part of client config 
> 
>
> Key: KAFKA-3302
> URL: https://issues.apache.org/jira/browse/KAFKA-3302
> Project: Kafka
>  Issue Type: Improvement
>  Components: security
>Reporter: Sriharsha Chintalapani
>Assignee: Sriharsha Chintalapani
> Fix For: 0.10.2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4217) KStream.transform equivalent of flatMap

[Elias Levy (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15556371#comment-15556371
 ] 

Elias Levy commented on KAFKA-4217:
---

That would work as well.

> KStream.transform equivalent of flatMap
> ---
>
> Key: KAFKA-4217
> URL: https://issues.apache.org/jira/browse/KAFKA-4217
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.0.1
>Reporter: Elias Levy
>Assignee: Guozhang Wang
>
> {{KStream.transform}} gives you access to state stores while allowing you to 
> return zero or one transformed {{KeyValue}}.  Alas, it is unclear what method 
> you should use if you want to access state stores and return zero or multiple 
> {{KeyValue}}.  Presumably you can use {{transform}}, always return {{null}}, 
> and use {{ProcessorContext.forward}} to emit {{KeyValues}}.
> It may be good to introduce a {{transform}}-like {{flatMap}} equivalent, or 
> allow store access from other {{KStream}} methods, such as {{flatMap}} itself.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-trunk-jdk8 #956

[Apache Jenkins Server]

See 

Changes:

[wangguoz] HOTFIX: move restoreConsumer.assign() to shutdownTasksAndState

[wangguoz] HOTFIX: recreate state.dir after cleanup

--
[...truncated 5086 lines...]

kafka.utils.ReplicationUtilsTest > testGetLeaderIsrAndEpochForPartition PASSED

kafka.utils.timer.TimerTest > testAlreadyExpiredTask STARTED

kafka.utils.timer.TimerTest > testAlreadyExpiredTask PASSED

kafka.utils.timer.TimerTest > testTaskExpiration STARTED

kafka.utils.timer.TimerTest > testTaskExpiration PASSED

kafka.utils.timer.TimerTaskListTest > testAll STARTED

kafka.utils.timer.TimerTaskListTest > testAll PASSED

kafka.utils.SchedulerTest > testMockSchedulerNonPeriodicTask STARTED

kafka.utils.SchedulerTest > testMockSchedulerNonPeriodicTask PASSED

kafka.utils.SchedulerTest > testMockSchedulerPeriodicTask STARTED

kafka.utils.SchedulerTest > testMockSchedulerPeriodicTask PASSED

kafka.utils.SchedulerTest > testNonPeriodicTask STARTED

kafka.utils.SchedulerTest > testNonPeriodicTask PASSED

kafka.utils.SchedulerTest > testRestart STARTED

kafka.utils.SchedulerTest > testRestart PASSED

kafka.utils.SchedulerTest > testReentrantTaskInMockScheduler STARTED

kafka.utils.SchedulerTest > testReentrantTaskInMockScheduler PASSED

kafka.utils.SchedulerTest > testPeriodicTask STARTED

kafka.utils.SchedulerTest > testPeriodicTask PASSED

kafka.utils.CommandLineUtilsTest > testParseEmptyArg STARTED

kafka.utils.CommandLineUtilsTest > testParseEmptyArg PASSED

kafka.utils.CommandLineUtilsTest > testParseSingleArg STARTED

kafka.utils.CommandLineUtilsTest > testParseSingleArg PASSED

kafka.utils.CommandLineUtilsTest > testParseArgs STARTED

kafka.utils.CommandLineUtilsTest > testParseArgs PASSED

kafka.utils.CommandLineUtilsTest > testParseEmptyArgAsValid STARTED

kafka.utils.CommandLineUtilsTest > testParseEmptyArgAsValid PASSED

kafka.utils.UtilsTest > testGenerateUuidAsBase64 STARTED

kafka.utils.UtilsTest > testGenerateUuidAsBase64 PASSED

kafka.utils.UtilsTest > testAbs STARTED

kafka.utils.UtilsTest > testAbs PASSED

kafka.utils.UtilsTest > testReplaceSuffix STARTED

kafka.utils.UtilsTest > testReplaceSuffix PASSED

kafka.utils.UtilsTest > testCircularIterator STARTED

kafka.utils.UtilsTest > testCircularIterator PASSED

kafka.utils.UtilsTest > testReadBytes STARTED

kafka.utils.UtilsTest > testReadBytes PASSED

kafka.utils.UtilsTest > testCsvList STARTED

kafka.utils.UtilsTest > testCsvList PASSED

kafka.utils.UtilsTest > testReadInt STARTED

kafka.utils.UtilsTest > testReadInt PASSED

kafka.utils.UtilsTest > testUrlSafeBase64EncodeUUID STARTED

kafka.utils.UtilsTest > testUrlSafeBase64EncodeUUID PASSED

kafka.utils.UtilsTest > testCsvMap STARTED

kafka.utils.UtilsTest > testCsvMap PASSED

kafka.utils.UtilsTest > testInLock STARTED

kafka.utils.UtilsTest > testInLock PASSED

kafka.utils.UtilsTest > testSwallow STARTED

kafka.utils.UtilsTest > testSwallow PASSED

kafka.utils.JsonTest > testJsonEncoding STARTED

kafka.utils.JsonTest > testJsonEncoding PASSED

kafka.utils.IteratorTemplateTest > testIterator STARTED

kafka.utils.IteratorTemplateTest > testIterator PASSED

kafka.metrics.KafkaTimerTest > testKafkaTimer STARTED

kafka.metrics.KafkaTimerTest > testKafkaTimer PASSED

kafka.metrics.MetricsTest > testMetricsReporterAfterDeletingTopic STARTED

kafka.metrics.MetricsTest > testMetricsReporterAfterDeletingTopic PASSED

kafka.metrics.MetricsTest > 
testBrokerTopicMetricsUnregisteredAfterDeletingTopic STARTED

kafka.metrics.MetricsTest > 
testBrokerTopicMetricsUnregisteredAfterDeletingTopic PASSED

kafka.metrics.MetricsTest > testClusterIdMetric STARTED

kafka.metrics.MetricsTest > testClusterIdMetric PASSED

kafka.metrics.MetricsTest > testMetricsLeak STARTED

kafka.metrics.MetricsTest > testMetricsLeak PASSED

kafka.consumer.TopicFilterTest > testWhitelists STARTED

kafka.consumer.TopicFilterTest > testWhitelists PASSED

kafka.consumer.TopicFilterTest > 
testWildcardTopicCountGetTopicCountMapEscapeJson STARTED

kafka.consumer.TopicFilterTest > 
testWildcardTopicCountGetTopicCountMapEscapeJson PASSED

kafka.consumer.TopicFilterTest > testBlacklists STARTED

kafka.consumer.TopicFilterTest > testBlacklists PASSED

kafka.consumer.PartitionAssignorTest > testRoundRobinPartitionAssignor STARTED

kafka.consumer.PartitionAssignorTest > testRoundRobinPartitionAssignor PASSED

kafka.consumer.PartitionAssignorTest > testRangePartitionAssignor STARTED

kafka.consumer.PartitionAssignorTest > testRangePartitionAssignor PASSED

kafka.consumer.ZookeeperConsumerConnectorTest > testBasic STARTED

kafka.consumer.ZookeeperConsumerConnectorTest > testBasic PASSED

kafka.consumer.ZookeeperConsumerConnectorTest > testCompressionSetConsumption 
STARTED

kafka.consumer.ZookeeperConsumerConnectorTest > testCompressionSetConsumption 
PASSED

kafka.consumer.ZookeeperConsumerConnectorTest > testLeaderSelectionForPartition 
STARTED

[jira] [Updated] (KAFKA-4117) Cleanup StreamPartitionAssignor behavior

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-4117:
-
Description: 
I went through the whole assignment logic once again and I feel the logic has 
now becomes a bit lossy, and I want to clean them up probably in another PR but 
just dump my thoughts here on the appropriate logic:

Some background:

1. Each {{KafkaStreams}} instance contains a clientId, and if not specified 
default value is applicationId-1/2/etc if there are multiple instances inside 
the same JVM. One instance contains multiple threads where the thread-clientId 
is constructed as clientId-StreamThread-1/2/etc, and the thread-clientId is 
used as the embedded consumer clientId as well as metrics tag.

2. However, since one instance can contain multiple threads, and hence multiple 
consumers, and when considering partition assignment, the streams library need 
to take the capacity into consideration based on the granularity of instance 
not on threads. Therefore we create a 4byte {{UUID.randomUUID()}} as the 
processId and encode that in the subscription metadata bytes, and the leader 
then knows if multiple consumer members are actually belong to the same 
instance (i.e. belong to threads of that instance), so that when assigning 
partitions it can balance among instances. NOTE that in production we recommend 
one thread per instance, so consumersByClient will only have one consumer per 
client (i.e. instance).

3. In addition, historically we hard-code the partition grouper logic, where 
for each task, it is assigned only with one partition of its subscribed topic. 
For example, if we have topicA with 5 partitions and topicB with 10 partitions, 
we will create 10 tasks, with the first five tasks containing one of the 
partitions each, while the last five tasks contain only one partition from 
topicB. And therefore the TaskId class contains the groupId of the sub-topology 
and the partition, so that taskId(group, 1) gets partition1 of topicA and 
partition1 of topicB. We later expose this to users to customize so that more 
than one partitions of the topic can be assigned to the same task, so that the 
partition field in the TaskId no longer indicate anything about which 
partitions are assigned, and we add {{AssignedPartitions}} to capture which 
partitions are assigned to which tasks.

4. While doing the assignment, the leader is also responsible for creating 
these changelog / repartition topics, and the number of partitions of these 
topics are equal to the number of tasks that needs to write to these topics, 
which are wrapped in {{stateChangelogTopicToTaskIds}} and 
{{internalSourceTopicToTaskIds}} respectively. After such topics are created, 
the leader also needs to "augment" the received cluster metadata with these 
topics to 1) check for copartitioning, and 2) maintained for QueryableState's 
discovery function.

The current implementation is mixed with all these legacy logic and gets quite 
messy, and I'm thinking to make a pass over the StreamPartitionAssignor and 
cleaning up it bit. More precisely:

1. Read and parse the subscription information to construct the clientMetadata 
map, where each metadata contains the {{Set consumerMemberIds}}, 
{{ClientState state}}, and {{HostInfo hostInfo}}.

2. Access the (sub-)topology to create the corresponding changelog / 
repartition topics and construct the {{stateChangelogTopicToTaskIds}} and 
{{internalSourceTopicToTaskIds}}.

3. Call {{streamThread.partitionGrouper.partitionGroups}} to get the map from 
created tasks to their assigned partitions.

4. Call {{TaskAssignor.assign}} (which now takes the whole clientMetadata map) 
to assign tasks to clients, and hence we get the assigned partitions to clients.

5. For each client, use some round-robin manner (as we did now) to assign tasks 
to their hosted consumers with the {{clientMetadata.consumerMemberIds}} map.

6. Check co-partitioning of assigned partitions, and maintain the {{Cluster}} 
metadata locally on the leader.

7. Construct the assignment info, where activeTasks is also a map from 
{{TaskId}} to list of {{TopicPartitions}} since otherwise we will not know 
which partitions are assigned to which tasks.

8. For non-leaders, when getting the assignment, also construct the Cluster 
metadata from the decoded assignment information; and also maintain the 
AssignmentInfo locally for constructing the tasks.

And some minor improvements:

1. The default {{thread-clientIds applicationId-x-StreamThread-y}} may still be 
conflicting to each other with multiple JVMs / machines, which is bad for 
metrics collection / debugging across hosts. We can modify the default clientId 
to {{applicationId-processId}} whereprocessId is the generated UUID, hence the 
default thread-clientId is {{applicationId-UUID-StreamThread-y}}.

2. The {{TaskId.partition}} field no longer indicate which partitions 

[jira] [Commented] (KAFKA-4217) KStream.transform equivalent of flatMap

[Matthias J. Sax (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15556349#comment-15556349
 ] 

Matthias J. Sax commented on KAFKA-4217:


Would it not be simpler to change `process(...)` to return a `KStream`  instead 
of `void` ?

> KStream.transform equivalent of flatMap
> ---
>
> Key: KAFKA-4217
> URL: https://issues.apache.org/jira/browse/KAFKA-4217
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.0.1
>Reporter: Elias Levy
>Assignee: Guozhang Wang
>
> {{KStream.transform}} gives you access to state stores while allowing you to 
> return zero or one transformed {{KeyValue}}.  Alas, it is unclear what method 
> you should use if you want to access state stores and return zero or multiple 
> {{KeyValue}}.  Presumably you can use {{transform}}, always return {{null}}, 
> and use {{ProcessorContext.forward}} to emit {{KeyValues}}.
> It may be good to introduce a {{transform}}-like {{flatMap}} equivalent, or 
> allow store access from other {{KStream}} methods, such as {{flatMap}} itself.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to normal : kafka-0.10.1-jdk7 #55

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-80: Kafka REST Server

[Suresh Srinivas]

ASF already gives us a clear framework and governance model for community
development. This is already understood by the people contributing to
Apache Kafka project, and they are the same people who want to contribute
to the REST server capability as well. Everyone is in agreement on the
need for collaborating on this effort. So why not contribute the code to
Apache Kafka. This will help avoid duplication of effort and forks that
may crop up, hugely benefitting the user community. This will also avoid
having to define a process similar to ASF on a GitHub project and instead
there is a single community with clear understanding community process as
defined in ASF.

As others have said, this is an important capability for Apache Kafka. It
is worth maintaining this as a part of the project.

Regards,
Suresh

On 10/6/16, 8:32 AM, "Ofir Manor"  wrote:

>I personally think it would be quite wasteful to re-implement the REST
>gateway just because that an actively-maintained piece of Apache-licensed
>software is not governed directly by the Apache Kafka community... While
>kafka-rest repo is owned by Confluent, the contributors including the main
>one are also part of the Apache Kafka  community, so there is a chance to
>work this out.
>
>However, there are two valid concerns here that could be addressed, around
>community and accessibility:
>>> What we are worried about is a project
>>> that's not maintained in a community. So the process of accepting
>>>patches
>>> and priorities is not clear, and it's not developed in Apache
>>>community.
>>> Not only that, existing REST API project doesn't support new client API
>and
>>> hence there is no security support either.
>
>This might be easy to fix. Maybe Confluent / kafka-rest community can
>clarify that - what is their contribution policy, dev style, roadmap etc.
>If they want, they can make an effort to encourage participation from
>people outside Confluent (easily accept contributions, invite external
>commiters or have open dev process similar to Apache Kafka etc), as there
>is definitely seems to be some interest on the list. That might clear the
>community concern and help kafka-rest project (but that is a calculation
>Confluent will have to make).
>
>The other, independent, concern is that REST is something that is expected
>to be available out of the box with Kafka. I personally don't feel
>strongly
>about it (better use proper, efficient APIs from day one), though it is
>definitely way smaller than adding a stream processing engine to the
>project :)
>Again,the kafka-rest "community" could take steps to make it even easier
>to
>install, configure and run kafka-rest for new users on vanilla Apache
>Kafka
>(outside the Confluent platform), if they wish that (or welcome
>contributions to that end), but that is up to them.
>Finally, if after the above steps were taken there would still a strong
>desire to include a great rest gateway with Apache Kafka, I assume the
>community could hypothetically fork the existing kafka-rest into an Apache
>Kafka subproject and maintain it "within Apache" instead of implementing
>it
>from scratch (though I'm not a lawyer etc) - but I cannot imagine it
>happen
>without Confluent blessing, and I think that is likely much less optimal
>(pulling in other Confluent / Apache licensed dependencies) than having a
>separate external community around kafka-rest.
>
>
>Just my two cents,
>
>
>Ofir Manor
>
>Co-Founder & CTO | Equalum
>
>Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io
>
>On Sun, Oct 2, 2016 at 11:23 PM, Harsha Chintalapani 
>wrote:
>
>> Neha & Jay,
>>  We did look at the open source alternatives. Our
>>concern
>> is what's the patch acceptance and adding features/ bug-fixes to the
>> existing project under a Github (although it's licensed under Apache
>>2.0).
>> It would be great if that project made available under Apache and
>>driven by
>> the community.  Adding to the above, not all Kafka users are interested
>>in
>> using the Java client API, they would like to have simple REST API where
>> they can code against using any language. I do believe this adds value
>>to
>> Apache Kafka in itself.
>>
>> "For 1, I don't think there is value in giving in to the NIH syndrome
>>and
>> reinventing the wheel. What I'm looking for is a detailed comparison of
>>the
>> gaps and why those can't be improved in the REST proxy that already
>>exists
>> and is actively maintained."
>>
>> We are not looking at this as  NIH. What we are worried about is a
>>project
>> that's not maintained in a community. So the process of accepting
>>patches
>> and priorities is not clear, and it's not developed in Apache community.
>> Not only that, existing REST API project doesn't support new client API
>>and
>> hence there is no security support either.
>> We don't know the timeline when that's made available. We would like to
>>add
>> admin functionality into the REST API. So the Roadmap of 

[jira] [Work started] (KAFKA-4275) Check of State-Store-assignment to Processor-Nodes is not enabled

[Matthias J. Sax (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on KAFKA-4275 started by Matthias J. Sax.
--
> Check of State-Store-assignment to Processor-Nodes is not enabled
> -
>
> Key: KAFKA-4275
> URL: https://issues.apache.org/jira/browse/KAFKA-4275
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Matthias J. Sax
>Assignee: Matthias J. Sax
>
> In {{ProcessorContextImpl#getStateStores()}} we should check if a store was 
> connected to the processor and thus, if the processor is allowed to access 
> the store. This check is currently disabled.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-4275) Check of State-Store-assignment to Processor-Nodes is not enabled

[Matthias J. Sax (JIRA)]

Matthias J. Sax created KAFKA-4275:
--

 Summary: Check of State-Store-assignment to Processor-Nodes is not 
enabled
 Key: KAFKA-4275
 URL: https://issues.apache.org/jira/browse/KAFKA-4275
 Project: Kafka
  Issue Type: Bug
  Components: streams
Reporter: Matthias J. Sax
Assignee: Matthias J. Sax


In {{ProcessorContextImpl#getStateStores()}} we should check if a store was 
connected to the processor and thus, if the processor is allowed to access the 
store. This check is currently disabled.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4274) KafkaConsumer.offsetsForTimes() hangs and times out on an empty map

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson updated KAFKA-4274:
---
Fix Version/s: 0.10.1.0

> KafkaConsumer.offsetsForTimes() hangs and times out on an empty map
> ---
>
> Key: KAFKA-4274
> URL: https://issues.apache.org/jira/browse/KAFKA-4274
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.1.0
>Reporter: Jun Rao
> Fix For: 0.10.1.0
>
>
> The following call will hang for the processing timeout in the consumer.
>consumer.offsetsForTimes(new util.HashMap[TopicPartition, 
> java.lang.Long]())



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to normal : kafka-trunk-jdk7 #1610

[Apache Jenkins Server]

See 

[jira] [Commented] (KAFKA-4271) The console consumer fails on Windows with new consumer is used

[Vahid Hashemian (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15556219#comment-15556219
 ] 

Vahid Hashemian commented on KAFKA-4271:


The command {{bin\windows\kafka-run-class.bat 
org.apache.kafka.streams.examples.wordcount.WordCountDemo}} in Step 8 of the 
Quickstart also fails with a similar error (likely because it leverages the new 
consumer).

> The console consumer fails on Windows with new consumer is used 
> 
>
> Key: KAFKA-4271
> URL: https://issues.apache.org/jira/browse/KAFKA-4271
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.0.1
>Reporter: Vahid Hashemian
>
> When I try to consume message using the new consumer (Quickstart Step 5) I 
> get an exception on the broker side. The old consumer works fine.
> {code}
> java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(Unknown Source)
> at kafka.log.AbstractIndex.(AbstractIndex.scala:61)
> at kafka.log.OffsetIndex.(OffsetIndex.scala:51)
> at kafka.log.LogSegment.(LogSegment.scala:67)
> at kafka.log.Log.loadSegments(Log.scala:255)
> at kafka.log.Log.(Log.scala:108)
> at kafka.log.LogManager.createLog(LogManager.scala:362)
> at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:94)
> at 
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
> at 
> kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
> at scala.collection.mutable.HashSet.foreach(HashSet.scala:79)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:174)
> at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:168)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
> at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:242)
> at kafka.cluster.Partition.makeLeader(Partition.scala:168)
> at 
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:740)
> at 
> kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:739)
> at 
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
> at 
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
> at 
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:98)
> at kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:739)
> at 
> kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:685)
> at 
> kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:148)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:82)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method)
> ... 29 more
> {code}
> This issue seems to break the broker and I have to clear out the logs so I 
> can bring the broker back up again.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4274) KafkaConsumer.offsetsForTimes() hangs and times out on an empty map

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15556217#comment-15556217
 ] 

Jun Rao commented on KAFKA-4274:


[~becket_qin], do you want to fix this? Thanks.

> KafkaConsumer.offsetsForTimes() hangs and times out on an empty map
> ---
>
> Key: KAFKA-4274
> URL: https://issues.apache.org/jira/browse/KAFKA-4274
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.1.0
>Reporter: Jun Rao
>
> The following call will hang for the processing timeout in the consumer.
>consumer.offsetsForTimes(new util.HashMap[TopicPartition, 
> java.lang.Long]())



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-4274) KafkaConsumer.offsetsForTimes() hangs and times out on an empty map

[Jun Rao (JIRA)]

Jun Rao created KAFKA-4274:
--

 Summary: KafkaConsumer.offsetsForTimes() hangs and times out on an 
empty map
 Key: KAFKA-4274
 URL: https://issues.apache.org/jira/browse/KAFKA-4274
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.10.1.0
Reporter: Jun Rao


The following call will hang for the processing timeout in the consumer.

   consumer.offsetsForTimes(new util.HashMap[TopicPartition, java.lang.Long]())




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-82 - Add Record Headers

[Michael Pearce]

Hi Jay,

Thanks for the comments and feedback.

I think its quite clear that if a problem keeps arising then it is clear that 
it needs resolving, and addressing properly.

Fair enough at linkedIn, and historically for the very first use cases 
addressing this maybe not have been a big priority. But as Kafka is now Apache 
open source and being picked up by many including my company, it is clear and 
evident that this is a requirement and issue that needs to be now addressed to 
address these needs.

The fact in almost every transport mechanism including networking layers in the 
enterprise ive worked in, there has always been headers i think clearly shows 
their need and success for a transport mechanism.

I understand some concerns with regards to impact for others not needing it.

What we are proposing is flexible solution that provides no overhead on storage 
or network traffic layers if you chose not to use headers, but does enable 
those who need or want it to use it.


On your response to 1), there is nothing saying that it should be put in any 
faster or without diligence and the same KIP process can still apply for adding 
kafka-scope headers, having headers, just makes it easier to add, without 
constant message and record changes. Timestamp is a clear real example of 
actually what should be in a header (along with other fields) but as such the 
whole message/record object needed to be changed to add this, as will any 
further headers deemed needed by kafka.

On response to 2) why within my company as a platforms designer should i 
enforce that all teams use the same serialization for their payloads? But what 
i do need is some core cross cutting concerns and information addressed at my 
platform level and i don't want to impose onto my development teams. This is 
the same argument why byte[] is the exposed value and key because as a 
messaging platform you dont want to impose that on my company.

On response to 3) Actually this isnt true, there are many 3rd party tools, we 
need to hook into our messaging flows that they only build onto standardised 
interfaces as obviously the cost to have a custom implementation for every 
company would be very high.
APM tooling is a clear case in point, every enterprise level APM tool on the 
market is able to stitch in transaction flow end 2 end over a platform over 
http, jms because they can stitch in some "magic" data in a 
uniform/standardised for the two mentioned they stitch this into the headers. 
It is current form they cannot do this with Kafka. Providing a standardised 
interface will i believe actually benefit the project as commercial companies 
like these will now be able to plugin their tooling uniformly, making it 
attractive and possible.

Some of you other concerns as Joel mentions these are more implementation 
details, that i think should be agreed upon, but i think can be addressed.

e.g. re your concern on the hashmap.
it is more than possible not to have every record have to have a hashmap unless 
it actually has a header (just like we have managed to do on the serialized 
meesage) so if theres a concern on the in memory record size for those using 
kafka without headers.

On your second to last comment about every team choosing their own format, 
actually we do want this a little, as very first mentioned, no we don't want a 
free for all, but some freedom to have different serialization has different 
benefits and draw backs across our business. I can iterate these if needed. One 
of the use case for headers provided by linkedIn on top of my KIP even shows 
where headers could be beneficial here as a header could be used to detail 
which data format the message is serialized to allowing me to consume different 
formats.

Also we have some systems that we need to integrate that pretty near impossible 
to wrap or touch their binary payloads, or we’re not allowed to touch them 
(historic system, or inter/intra corporate)

Headers really gives as a solution to provide a pluggable platform, and 
standardisation that allows users to build platforms that adapt to their needs.


Cheers
Mike



From: Jay Kreps 
Sent: Friday, October 7, 2016 4:45 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-82 - Add Record Headers

Hey guys,

This discussion has come up a number of times and we've always passed.

One of things that has helped keep Kafka simple is not adding in new
abstractions and concepts except when the proposal is really elegant and
makes things simpler.

Consider three use cases for headers:

   1. Kafka-scope: We want to add a feature to Kafka that needs a
   particular field.
   2. Company-scope: You want to add a header to be shared by everyone in
   your company.
   3. World-wide scope: You are building a third party tool and want to add
   some kind of header.

For the case of (1) you should not use headers, you should just add a field
to the record format. Having a 

[GitHub] kafka pull request #1940: HOTFIX: recreate state.dir after cleanup

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1940


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-4273) Streams DSL - Add TTL / retention period support for intermediate topics and state stores

[Davor Poldrugo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Poldrugo updated KAFKA-4273:
--
Description: 
Hi!
I'm using Streams DSL (0.10.0.1), which can only use RocksDB for local state as 
far as I know - it's not configurable.
In my use case my data has TTL / retnetion period. It's 48 hours. After that - 
data can be discarded.

I join two topics: "messages" and "prices" using windowed inner join.
The two intermediate Kafka topics for this join are named:
 * messages-prices-join-this-changelog
 * messages-prices-join-other-changelog

Since these topics are created as compacted by Kafka Streams, and I don't wan't 
to keep data forever, I have altered them to not use compaction. Right now my 
RocksDB state stores grow indefinitely, and I don't have any options to define 
TTL, or somehow periodically clean the older data. I 

A "hack" that I use to keep my disk usage low - I have schedulled a job to 
periodically stop Kafka Streams instances - one at the time. This triggers a 
rebalance, and partitions migrate to other instances. When the instance is 
started again, there's another rebalance, and sometimes this instance starts 
processing partitions that wasn't processing before the stop - which leads to 
deletion of the RocksDB state store for those partitions 
(state.cleanup.delay.ms). In the next rebalance the local store is recreated 
with a restore consumer - which reads data from - as previously mentioned - a 
non compacted topic. And this effectively leads to a "hacked TTL support" in 
Kafka Streams DSL.

Questions:
 * Do you think would be reasonable to add support in the DSL api to define TTL 
for local store?
 * Which opens another question - there are use cases which don't need the 
intermediate topics to be created as "compact". Could also this be added to the 
DSL api? Maybe only this could be added, and this flag should also be used for 
the RocksDB TTL. Of course in this case another config would be mandatory - the 
retention period or TTL for the intermediate topics and the state stores. I saw 
there is a new cleanup.policy - compact_and_delete - added with KAFKA-4015.
 * Which also leads to another question, maybe some intermediate topics / state 
stores need different TTL, so a it's not as simple as that. But after 
KAFKA-3870, it will be easier.

RocksDB supports TTL:
 * 
https://github.com/apache/kafka/blob/0.10.0.1/streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBStore.java#L166
 * https://github.com/facebook/rocksdb/wiki/Time-to-Live
 * 
https://github.com/facebook/rocksdb/blob/master/java/src/main/java/org/rocksdb/TtlDB.java

A somehow similar issue: KAFKA-4212



  was:
Hi!
I'm using Streams DSL (0.10.0.1), which can only use RocksDB for local state as 
far as I know - it's not configurable.
In my use case my data has TTL / retnetion period. It's 48 hours. After that - 
data can be discarded.

I join two topics: "messages" and "prices" using windowed inner join.
The two intermediate Kafka topics for this join are named:
 * messages-prices-join-this-changelog
 * messages-prices-join-other-changelog

Since these topics are created as compacted by Kafka Streams, and I don't wan't 
to keep data forever, I have altered them to not use compaction. Right now my 
RocksDB state stores grow indefinitely, and I don't have any options to define 
TTL, or somehow periodically clean the older data. I 

A "hack" that I use to keep my disk usage low - I have schedulled a job to 
periodically stop Kafka Streams instances - one at the time. This triggers a 
rebalance, and partitions migrate to other instances. When the instance is 
started again, there's another rebalance, and sometimes this instance starts 
processing partitions that wasn't processing before the stop - which leads to 
deletion of the RocksDB state store for those partitions 
(state.cleanup.delay.ms). In the next rebalance the local store is recreated 
with a restore consumer - which reads data from - as previously mentioned - a 
non compacted topic. And this effectively leads to a "hacked TTL support" in 
Kafka Streams DSL.

Questions:
 * Do you think would be reasonable to add support in the DSL api to define TTL 
for local store?
 * Which opens another question - there are use cases which don't need the 
intermediate topics to be created as "compact". Could also this be added to the 
DSL api? Maybe only this could be added, and this flag should also be used for 
the RocksDB TTL. Of course in this case another config would be mandatory - the 
retention period or TTL for the intermediate topics and the state stores.
 * Which also leads to another question, maybe some intermediate topics / state 
stores need different TTL, so a it's not as simple as that.

RocksDB supports TTL:
 * 

[jira] [Updated] (KAFKA-4273) Streams DSL - Add TTL / retention period support for intermediate topics and state stores

[Davor Poldrugo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Poldrugo updated KAFKA-4273:
--
Description: 
Hi!
I'm using Streams DSL (0.10.0.1), which can only use RocksDB for local state as 
far as I know - it's not configurable.
In my use case my data has TTL / retnetion period. It's 48 hours. After that - 
data can be discarded.

I join two topics: "messages" and "prices" using windowed inner join.
The two intermediate Kafka topics for this join are named:
 * messages-prices-join-this-changelog
 * messages-prices-join-other-changelog

Since these topics are created as compacted by Kafka Streams, and I don't wan't 
to keep data forever, I have altered them to not use compaction. Right now my 
RocksDB state stores grow indefinitely, and I don't have any options to define 
TTL, or somehow periodically clean the older data. I 

A "hack" that I use to keep my disk usage low - I have schedulled a job to 
periodically stop Kafka Streams instances - one at the time. This triggers a 
rebalance, and partitions migrate to other instances. When the instance is 
started again, there's another rebalance, and sometimes this instance starts 
processing partitions that wasn't processing before the stop - which leads to 
deletion of the RocksDB state store for those partitions 
(state.cleanup.delay.ms). In the next rebalance the local store is recreated 
with a restore consumer - which reads data from - as previously mentioned - a 
non compacted topic. And this effectively leads to a "hacked TTL support" in 
Kafka Streams DSL.

Questions:
 * Do you think would be reasonable to add support in the DSL api to define TTL 
for local store?
 * Which opens another question - there are use cases which don't need the 
intermediate topics to be created as "compact". Could also this be added to the 
DSL api? Maybe only this could be added, and this flag should also be used for 
the RocksDB TTL. Of course in this case another config would be mandatory - the 
retention period or TTL for the intermediate topics and the state stores.
 * Which also leads to another question, maybe some intermediate topics / state 
stores need different TTL, so a it's not as simple as that.

RocksDB supports TTL:
 * 
https://github.com/apache/kafka/blob/0.10.0.1/streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBStore.java#L166
 * https://github.com/facebook/rocksdb/wiki/Time-to-Live
 * 
https://github.com/facebook/rocksdb/blob/master/java/src/main/java/org/rocksdb/TtlDB.java

A somehow similar issue: KAFKA-4212



  was:
Hi!
I'm using Streams DSL (0.10.0.1), which can only use RocksDB for local state as 
far as I know - it's not configurable.
In my use case my data has TTL / retnetion period. It's 48 hours. After that - 
data can be discarded.

I join two topics: "messages" and "prices" using windowed inner join.
The two intermediate Kafka topics for this join are named:
 * messages-prices-join-this-changelog
 * messages-prices-join-other-changelog

Since these topics are created as compacted by Kafka Streams, and I don't wan't 
to keep data forever, I have altered them to not use compaction. Right now my 
RocksDB state stores grow indefinitely, and I don't have any options to define 
TTL, or somehow periodically clean the older data. I 

A "hack" that I use to keep my disk usage low - I have schedulled a job to 
periodically stop Kafka Streams instances - one at the time. This triggers a 
rebalance, and partitions migrate to other instances. When the instance is 
started again, there's another rebalance, and sometimes this instance starts 
processing partitions that wasn't processing before the stop - which leads to 
deletion of the RocksDB state store for those partitions 
(state.cleanup.delay.ms). In the next rebalance the local store is recreated 
with a restore consumer - which reads data from - as previously mentioned - a 
non compacted topic. And this effectively leads to a "hacked TTL support" in 
Kafka Streams DSL.

Questions:
 * Do you think would be reasonable to add support in the DSL api to define TTL 
for local store?
 * Which opens another question - there are use cases which don't need the 
intermediate topics to be created as "compact". Could also this be added to the 
DSL api? Maybe only this could be added, and this flag should also be used for 
the RocksDB TTL. Of course in this case another config would be mandatory - the 
retention period or TTL for the intermediate topics and the state stores.
 * Which also leads to another question, maybe some intermediate topics / state 
stores need different TTL, so a it's not as simple as that.

RocksDB supports TTL:
 * 
https://github.com/apache/kafka/blob/0.10.0.1/streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBStore.java#L166
 * https://github.com/facebook/rocksdb/wiki/Time-to-Live
 * 

[jira] [Created] (KAFKA-4273) Streams DSL - Add TTL / retention period support for intermediate topics and state stores

[Davor Poldrugo (JIRA)]

Davor Poldrugo created KAFKA-4273:
-

 Summary: Streams DSL - Add TTL / retention period support for 
intermediate topics and state stores
 Key: KAFKA-4273
 URL: https://issues.apache.org/jira/browse/KAFKA-4273
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Affects Versions: 0.10.0.1
Reporter: Davor Poldrugo
Assignee: Guozhang Wang


Hi!
I'm using Streams DSL (0.10.0.1), which can only use RocksDB for local state as 
far as I know - it's not configurable.
In my use case my data has TTL / retnetion period. It's 48 hours. After that - 
data can be discarded.

I join two topics: "messages" and "prices" using windowed inner join.
The two intermediate Kafka topics for this join are named:
 * messages-prices-join-this-changelog
 * messages-prices-join-other-changelog

Since these topics are created as compacted by Kafka Streams, and I don't wan't 
to keep data forever, I have altered them to not use compaction. Right now my 
RocksDB state stores grow indefinitely, and I don't have any options to define 
TTL, or somehow periodically clean the older data. I 

A "hack" that I use to keep my disk usage low - I have schedulled a job to 
periodically stop Kafka Streams instances - one at the time. This triggers a 
rebalance, and partitions migrate to other instances. When the instance is 
started again, there's another rebalance, and sometimes this instance starts 
processing partitions that wasn't processing before the stop - which leads to 
deletion of the RocksDB state store for those partitions 
(state.cleanup.delay.ms). In the next rebalance the local store is recreated 
with a restore consumer - which reads data from - as previously mentioned - a 
non compacted topic. And this effectively leads to a "hacked TTL support" in 
Kafka Streams DSL.

Questions:
 * Do you think would be reasonable to add support in the DSL api to define TTL 
for local store?
 * Which opens another question - there are use cases which don't need the 
intermediate topics to be created as "compact". Could also this be added to the 
DSL api? Maybe only this could be added, and this flag should also be used for 
the RocksDB TTL. Of course in this case another config would be mandatory - the 
retention period or TTL for the intermediate topics and the state stores.
 * Which also leads to another question, maybe some intermediate topics / state 
stores need different TTL, so a it's not as simple as that.

RocksDB supports TTL:
 * 
https://github.com/apache/kafka/blob/0.10.0.1/streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBStore.java#L166
 * https://github.com/facebook/rocksdb/wiki/Time-to-Live
 * 
https://github.com/facebook/rocksdb/blob/master/java/src/main/java/org/rocksdb/TtlDB.java





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-80: Kafka REST Server

[Neha Narkhede]

Harsha/Mani,

I completely agree that adding admin API support and security are important
features for the Kafka REST proxy. Luckily the roadmap items that you
mentioned as being important for a Kafka REST proxy server are exactly the
ones the community working on this REST proxy want to add to it :-)

But I haven't seen any community emails or patches being submitted by you
guys, so I'm wondering why you are concerned about whether the community is
open to accepting patches or not.

Does that make sense?

On Fri, Oct 7, 2016 at 12:03 PM Harsha Chintalapani  wrote:

> Ofir,
> …
> " personally think it would be quite wasteful to re-implement the REST
> gateway just because that an actively-maintained piece of Apache-licensed
> software is not governed directly by the Apache Kafka community... While
> kafka-rest repo is owned by Confluent, the contributors including the main
> one are also part of the Apache Kafka  community, so there is a chance to
> work this out."
> It is important for the code to be under Apache guidelines for
> contributions so that it is driven by the community.
> As part of Kafka community, we would like to improve the project and add
> new features. If kafka-rest repo can be donated to Apache Kafka project
> we will be more than happy to welcome that and add our features on top of
> it.
>
> Thanks,
> harsha
>
> On Thu, Oct 6, 2016 at 9:50 AM Jay Kreps  wrote:
>
> > Hi Manikumar,
> >
> > I agree totally agree that REST is important. What I don't understand is
> > why we'd duplicate the existing REST interface inside the Kafka project.
> > That seems to needlessly fragment things.
> >
> > -Jay
> >
> > On Sat, Oct 1, 2016 at 5:38 AM, Manikumar 
> > wrote:
> >
> > > Hi Jay,
> > >
> > > Thanks for your reply.
> > >
> > > I agree that we can not add all the clients/tools available in
> ecosystem
> > > page to Kafka repo itself. But we feel REST Interface is different from
> > > other clients/tools. Since any language that can work with HTTP can
> > > easily integrate with this interface, Having an "official"  REST
> > > interface helps user community. This also helps us to integrate well
> > > with external management and provisioning tools.  Apache Kafka release
> > > with Java clients + REST interface is sufficient for most of the user
> > > deployments/requirements. This helps users to deal with less number
> > > of distributions/builds.
> > >
> > > Thanks,
> > > Manikumar
> > >
> > >
> > > On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps  wrote:
> > >
> > > > Hey guys,
> > > >
> > > > There's already a REST interface maintained as a separate
> project--it's
> > > > open source and apache licensed and actively maintained (
> > > > https://github.com/confluentinc/kafka-rest). What is wrong with
> that?
> > > You
> > > > mentioned that there was some compatibility concern, but
> compatibility
> > > has
> > > > to do with the consumer protocol guarantees not the repo the code is
> > in,
> > > > right? Not sure that concern makes sense.
> > > >
> > > > We could argue for adding pretty much anything and everything in the
> > > > ecosystem page in Kafka itself but I'm not sure that would make the
> > > project
> > > > more agile.
> > > >
> > > > -Jay
> > > >
> > > > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar <
> manikumar.re...@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Hi Kafka Devs,
> > > > >
> > > > > I created KIP-80 to add Kafka REST Server to Kafka Repository.
> > > > >
> > > > > There are already open-source alternatives are available.  But we
> > would
> > > > > like to add REST server that
> > > > > many users ask for under Apache Kafka repo. Many data Infra tools
> > comes
> > > > up
> > > > > with Rest Interface.
> > > > > It is useful to have inbuilt Rest API support for Produce, Consume
> > > > messages
> > > > > and admin interface for
> > > > > integrating with external management and provisioning tools.This
> will
> > > > also
> > > > > allow the maintenance of
> > > > > REST server and adding new features makes it easy because apache
> > > > community.
> > > > >
> > > > > The KIP wiki is the following:
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > 80%3A+Kafka+Rest+Server
> > > > >
> > > > > Your comments and feedback are welcome.
> > > > >
> > > > > Thanks,
> > > > > Manikumar
> > > > >
> > > >
> > >
> >
>
-- 
Thanks,
Neha

Build failed in Jenkins: kafka-trunk-jdk8 #955

[Apache Jenkins Server]

See 

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on jenkins-test-39e (Ubuntu ubuntu jenkins-cloud-8GB 
jenkins-cloud-4GB cloud-slave) in workspace 

Cloning the remote Git repository
Cloning repository https://git-wip-us.apache.org/repos/asf/kafka.git
 > git init  # timeout=10
ERROR: Error cloning remote repo 'origin'
hudson.plugins.git.GitException: Could not init 

at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl$5.execute(CliGitAPIImpl.java:652)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:463)
at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:152)
at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:145)
at hudson.remoting.UserRequest.perform(UserRequest.java:153)
at hudson.remoting.UserRequest.perform(UserRequest.java:50)
at hudson.remoting.Request$2.run(Request.java:332)
at 
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
at ..remote call to jenkins-test-39e(Native Method)
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
at hudson.remoting.Channel.call(Channel.java:781)
at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:145)
at sun.reflect.GeneratedMethodAccessor384.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:131)
at com.sun.proxy.$Proxy161.execute(Unknown Source)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1046)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1086)
at hudson.scm.SCM.checkout(SCM.java:485)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1269)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:604)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
at hudson.model.Run.execute(Run.java:1741)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:410)
Caused by: hudson.plugins.git.GitException: Command "git init 
 returned status code 1:
stdout: 
stderr: : No space left 
on device

at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1723)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1699)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1695)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1317)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl$5.execute(CliGitAPIImpl.java:650)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:463)
at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:152)
at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:145)
at hudson.remoting.UserRequest.perform(UserRequest.java:153)
at hudson.remoting.UserRequest.perform(UserRequest.java:50)
at hudson.remoting.Request$2.run(Request.java:332)
at 
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
ERROR: null
Retrying after 10 seconds
Cloning the remote Git repository
Cloning repository 

Build failed in Jenkins: kafka-trunk-jdk8 #954

[Apache Jenkins Server]

See 

Changes:

[jason] MINOR: Use `hiResClockMs` in `testRequestExpiry` to fix transient test

--
[...truncated 14017 lines...]

org.apache.kafka.streams.state.internals.CompositeReadOnlyKeyValueStoreTest > 
shouldThrowInvalidStoreExceptionOnRangeDuringRebalance PASSED

org.apache.kafka.streams.state.internals.CompositeReadOnlyKeyValueStoreTest > 
shouldSupportRangeAcrossMultipleKVStores STARTED

org.apache.kafka.streams.state.internals.CompositeReadOnlyKeyValueStoreTest > 
shouldSupportRangeAcrossMultipleKVStores PASSED

org.apache.kafka.streams.state.internals.CompositeReadOnlyKeyValueStoreTest > 
shouldSupportAllAcrossMultipleStores STARTED

org.apache.kafka.streams.state.internals.CompositeReadOnlyKeyValueStoreTest > 
shouldSupportAllAcrossMultipleStores PASSED

org.apache.kafka.streams.state.internals.MergedSortedCacheKeyValueStoreIteratorTest
 > shouldIterateOverRange STARTED

org.apache.kafka.streams.state.internals.MergedSortedCacheKeyValueStoreIteratorTest
 > shouldIterateOverRange PASSED

org.apache.kafka.streams.state.internals.WrappingStoreProviderTest > 
shouldThrowInvalidStoreExceptionIfNoStoreOfTypeFound STARTED

org.apache.kafka.streams.state.internals.WrappingStoreProviderTest > 
shouldThrowInvalidStoreExceptionIfNoStoreOfTypeFound PASSED

org.apache.kafka.streams.state.internals.WrappingStoreProviderTest > 
shouldFindWindowStores STARTED

org.apache.kafka.streams.state.internals.WrappingStoreProviderTest > 
shouldFindWindowStores PASSED

org.apache.kafka.streams.state.internals.WrappingStoreProviderTest > 
shouldFindKeyValueStores STARTED

org.apache.kafka.streams.state.internals.WrappingStoreProviderTest > 
shouldFindKeyValueStores PASSED

org.apache.kafka.streams.state.internals.CachingWindowStoreTest > 
shouldPutFetchFromCache STARTED

org.apache.kafka.streams.state.internals.CachingWindowStoreTest > 
shouldPutFetchFromCache PASSED

org.apache.kafka.streams.state.internals.CachingWindowStoreTest > 
shouldIterateCacheAndStore STARTED

org.apache.kafka.streams.state.internals.CachingWindowStoreTest > 
shouldIterateCacheAndStore PASSED

org.apache.kafka.streams.state.internals.CachingWindowStoreTest > 
shouldTakeValueFromCacheIfSameTimestampFlushedToRocks STARTED

org.apache.kafka.streams.state.internals.CachingWindowStoreTest > 
shouldTakeValueFromCacheIfSameTimestampFlushedToRocks PASSED

org.apache.kafka.streams.state.internals.CachingWindowStoreTest > 
shouldForwardDirtyItemsWhenFlushCalled STARTED

org.apache.kafka.streams.state.internals.CachingWindowStoreTest > 
shouldForwardDirtyItemsWhenFlushCalled PASSED

org.apache.kafka.streams.state.internals.CachingWindowStoreTest > 
shouldIterateAcrossWindows STARTED

org.apache.kafka.streams.state.internals.CachingWindowStoreTest > 
shouldIterateAcrossWindows PASSED

org.apache.kafka.streams.state.internals.CachingWindowStoreTest > 
shouldFlushEvictedItemsIntoUnderlyingStore STARTED

org.apache.kafka.streams.state.internals.CachingWindowStoreTest > 
shouldFlushEvictedItemsIntoUnderlyingStore PASSED

org.apache.kafka.streams.state.internals.CachingWindowStoreTest > 
shouldForwardOldValuesWhenEnabled STARTED

org.apache.kafka.streams.state.internals.CachingWindowStoreTest > 
shouldForwardOldValuesWhenEnabled PASSED

org.apache.kafka.streams.state.internals.CachingWindowStoreTest > 
shouldForwardDirtyItemToListenerWhenEvicted STARTED

org.apache.kafka.streams.state.internals.CachingWindowStoreTest > 
shouldForwardDirtyItemToListenerWhenEvicted PASSED

org.apache.kafka.streams.state.internals.OffsetCheckpointTest > testReadWrite 
STARTED

org.apache.kafka.streams.state.internals.OffsetCheckpointTest > testReadWrite 
PASSED

org.apache.kafka.streams.state.internals.InMemoryLRUCacheStoreTest > testEvict 
STARTED

org.apache.kafka.streams.state.internals.InMemoryLRUCacheStoreTest > testEvict 
PASSED

org.apache.kafka.streams.state.internals.InMemoryLRUCacheStoreTest > testSize 
STARTED

org.apache.kafka.streams.state.internals.InMemoryLRUCacheStoreTest > testSize 
PASSED

org.apache.kafka.streams.state.internals.InMemoryLRUCacheStoreTest > 
testPutIfAbsent STARTED

org.apache.kafka.streams.state.internals.InMemoryLRUCacheStoreTest > 
testPutIfAbsent PASSED

org.apache.kafka.streams.state.internals.InMemoryLRUCacheStoreTest > 
testRestoreWithDefaultSerdes STARTED

org.apache.kafka.streams.state.internals.InMemoryLRUCacheStoreTest > 
testRestoreWithDefaultSerdes PASSED

org.apache.kafka.streams.state.internals.InMemoryLRUCacheStoreTest > 
testRestore STARTED

org.apache.kafka.streams.state.internals.InMemoryLRUCacheStoreTest > 
testRestore PASSED

org.apache.kafka.streams.state.internals.InMemoryLRUCacheStoreTest > 
testPutGetRange STARTED

org.apache.kafka.streams.state.internals.InMemoryLRUCacheStoreTest > 
testPutGetRange PASSED

org.apache.kafka.streams.state.internals.InMemoryLRUCacheStoreTest > 
testPutGetRangeWithDefaultSerdes STARTED

Build failed in Jenkins: kafka-0.10.1-jdk7 #54

[Apache Jenkins Server]

See 

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on jenkins-test-39e (Ubuntu ubuntu jenkins-cloud-8GB 
jenkins-cloud-4GB cloud-slave) in workspace 

java.io.IOException: Failed to mkdirs: 

at hudson.FilePath.mkdirs(FilePath.java:1191)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1267)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:604)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
at hudson.model.Run.execute(Run.java:1741)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:410)
Retrying after 10 seconds
java.io.IOException: Failed to mkdirs: 

at hudson.FilePath.mkdirs(FilePath.java:1191)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1267)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:604)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
at hudson.model.Run.execute(Run.java:1741)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:410)
Retrying after 10 seconds
java.io.IOException: Failed to mkdirs: 

at hudson.FilePath.mkdirs(FilePath.java:1191)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1267)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:604)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
at hudson.model.Run.execute(Run.java:1741)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:410)
Recording test results
ERROR: Build step failed with exception
 does not exist.
at 
org.apache.tools.ant.types.AbstractFileSet.getDirectoryScanner(AbstractFileSet.java:483)
at 
org.apache.tools.ant.types.AbstractFileSet.getDirectoryScanner(AbstractFileSet.java:460)
at 
hudson.tasks.junit.JUnitParser$ParseResultCallable.invoke(JUnitParser.java:127)
at 
hudson.tasks.junit.JUnitParser$ParseResultCallable.invoke(JUnitParser.java:107)
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2772)
at hudson.remoting.UserRequest.perform(UserRequest.java:153)
at hudson.remoting.UserRequest.perform(UserRequest.java:50)
at hudson.remoting.Request$2.run(Request.java:332)
at 
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
at ..remote call to jenkins-test-39e(Native Method)
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
at hudson.remoting.Channel.call(Channel.java:781)
at hudson.FilePath.act(FilePath.java:1007)
at hudson.FilePath.act(FilePath.java:996)
at hudson.tasks.junit.JUnitParser.parseResult(JUnitParser.java:103)
at 
hudson.tasks.junit.JUnitResultArchiver.parse(JUnitResultArchiver.java:128)
at 
hudson.tasks.junit.JUnitResultArchiver.perform(JUnitResultArchiver.java:149)
at 
hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:78)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:779)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:720)
at hudson.model.Build$BuildExecution.post2(Build.java:185)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:665)
  

Re: [DISCUSS] KIP-80: Kafka REST Server

[Harsha Chintalapani]

Ofir,
…
" personally think it would be quite wasteful to re-implement the REST
gateway just because that an actively-maintained piece of Apache-licensed
software is not governed directly by the Apache Kafka community... While
kafka-rest repo is owned by Confluent, the contributors including the main
one are also part of the Apache Kafka  community, so there is a chance to
work this out."
It is important for the code to be under Apache guidelines for
contributions so that it is driven by the community.
As part of Kafka community, we would like to improve the project and add
new features. If kafka-rest repo can be donated to Apache Kafka project
we will be more than happy to welcome that and add our features on top of
it.

Thanks,
harsha

On Thu, Oct 6, 2016 at 9:50 AM Jay Kreps  wrote:

> Hi Manikumar,
>
> I agree totally agree that REST is important. What I don't understand is
> why we'd duplicate the existing REST interface inside the Kafka project.
> That seems to needlessly fragment things.
>
> -Jay
>
> On Sat, Oct 1, 2016 at 5:38 AM, Manikumar 
> wrote:
>
> > Hi Jay,
> >
> > Thanks for your reply.
> >
> > I agree that we can not add all the clients/tools available in ecosystem
> > page to Kafka repo itself. But we feel REST Interface is different from
> > other clients/tools. Since any language that can work with HTTP can
> > easily integrate with this interface, Having an "official"  REST
> > interface helps user community. This also helps us to integrate well
> > with external management and provisioning tools.  Apache Kafka release
> > with Java clients + REST interface is sufficient for most of the user
> > deployments/requirements. This helps users to deal with less number
> > of distributions/builds.
> >
> > Thanks,
> > Manikumar
> >
> >
> > On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps  wrote:
> >
> > > Hey guys,
> > >
> > > There's already a REST interface maintained as a separate project--it's
> > > open source and apache licensed and actively maintained (
> > > https://github.com/confluentinc/kafka-rest). What is wrong with that?
> > You
> > > mentioned that there was some compatibility concern, but compatibility
> > has
> > > to do with the consumer protocol guarantees not the repo the code is
> in,
> > > right? Not sure that concern makes sense.
> > >
> > > We could argue for adding pretty much anything and everything in the
> > > ecosystem page in Kafka itself but I'm not sure that would make the
> > project
> > > more agile.
> > >
> > > -Jay
> > >
> > > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar  >
> > > wrote:
> > >
> > > > Hi Kafka Devs,
> > > >
> > > > I created KIP-80 to add Kafka REST Server to Kafka Repository.
> > > >
> > > > There are already open-source alternatives are available.  But we
> would
> > > > like to add REST server that
> > > > many users ask for under Apache Kafka repo. Many data Infra tools
> comes
> > > up
> > > > with Rest Interface.
> > > > It is useful to have inbuilt Rest API support for Produce, Consume
> > > messages
> > > > and admin interface for
> > > > integrating with external management and provisioning tools.This will
> > > also
> > > > allow the maintenance of
> > > > REST server and adding new features makes it easy because apache
> > > community.
> > > >
> > > > The KIP wiki is the following:
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > 80%3A+Kafka+Rest+Server
> > > >
> > > > Your comments and feedback are welcome.
> > > >
> > > > Thanks,
> > > > Manikumar
> > > >
> > >
> >
>

Build failed in Jenkins: kafka-trunk-jdk7 #1609

[Apache Jenkins Server]

See 

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on jenkins-test-39e (Ubuntu ubuntu jenkins-cloud-8GB 
jenkins-cloud-4GB cloud-slave) in workspace 

java.io.IOException: Failed to mkdirs: 

at hudson.FilePath.mkdirs(FilePath.java:1191)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1267)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:604)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
at hudson.model.Run.execute(Run.java:1741)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:410)
Retrying after 10 seconds
java.io.IOException: Failed to mkdirs: 

at hudson.FilePath.mkdirs(FilePath.java:1191)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1267)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:604)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
at hudson.model.Run.execute(Run.java:1741)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:410)
Retrying after 10 seconds
java.io.IOException: Failed to mkdirs: 

at hudson.FilePath.mkdirs(FilePath.java:1191)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1267)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:604)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
at hudson.model.Run.execute(Run.java:1741)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:410)
Recording test results
ERROR: Build step failed with exception
 does not exist.
at 
org.apache.tools.ant.types.AbstractFileSet.getDirectoryScanner(AbstractFileSet.java:483)
at 
org.apache.tools.ant.types.AbstractFileSet.getDirectoryScanner(AbstractFileSet.java:460)
at 
hudson.tasks.junit.JUnitParser$ParseResultCallable.invoke(JUnitParser.java:127)
at 
hudson.tasks.junit.JUnitParser$ParseResultCallable.invoke(JUnitParser.java:107)
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2772)
at hudson.remoting.UserRequest.perform(UserRequest.java:153)
at hudson.remoting.UserRequest.perform(UserRequest.java:50)
at hudson.remoting.Request$2.run(Request.java:332)
at 
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
at ..remote call to jenkins-test-39e(Native Method)
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
at hudson.remoting.Channel.call(Channel.java:781)
at hudson.FilePath.act(FilePath.java:1007)
at hudson.FilePath.act(FilePath.java:996)
at hudson.tasks.junit.JUnitParser.parseResult(JUnitParser.java:103)
at 
hudson.tasks.junit.JUnitResultArchiver.parse(JUnitResultArchiver.java:128)
at 
hudson.tasks.junit.JUnitResultArchiver.perform(JUnitResultArchiver.java:149)
at 
hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:78)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:779)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:720)
at hudson.model.Build$BuildExecution.post2(Build.java:185)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:665)
  

Re: [DISCUSS] KIP-80: Kafka REST Server

[Harsha Ch]

Ofir,
" personally think it would be quite wasteful to re-implement the REST
gateway just because that an actively-maintained piece of Apache-licensed
software is not governed directly by the Apache Kafka community... While
kafka-rest repo is owned by Confluent, the contributors including the main
one are also part of the Apache Kafka  community, so there is a chance to
work this out."
It is important for the code to be under Apache guidelines for
contributions so that it is driven by the community.
As part of Kafka community, we would like to improve the project and add
new features. If kafka-rest repo can be donated to Apache Kafka project
we will be more than happy to welcome that and add our features on top of
it.

Thanks,
harsha

On Thu, Oct 6, 2016 at 9:50 AM Jay Kreps  wrote:

Hi Manikumar,

I agree totally agree that REST is important. What I don't understand is
why we'd duplicate the existing REST interface inside the Kafka project.
That seems to needlessly fragment things.

-Jay

On Sat, Oct 1, 2016 at 5:38 AM, Manikumar  wrote:

> Hi Jay,
>
> Thanks for your reply.
>
> I agree that we can not add all the clients/tools available in ecosystem
> page to Kafka repo itself. But we feel REST Interface is different from
> other clients/tools. Since any language that can work with HTTP can
> easily integrate with this interface, Having an "official"  REST
> interface helps user community. This also helps us to integrate well
> with external management and provisioning tools.  Apache Kafka release
> with Java clients + REST interface is sufficient for most of the user
> deployments/requirements. This helps users to deal with less number
> of distributions/builds.
>
> Thanks,
> Manikumar
>
>
> On Sat, Oct 1, 2016 at 4:24 AM, Jay Kreps  wrote:
>
> > Hey guys,
> >
> > There's already a REST interface maintained as a separate project--it's
> > open source and apache licensed and actively maintained (
> > https://github.com/confluentinc/kafka-rest). What is wrong with that?
> You
> > mentioned that there was some compatibility concern, but compatibility
> has
> > to do with the consumer protocol guarantees not the repo the code is in,
> > right? Not sure that concern makes sense.
> >
> > We could argue for adding pretty much anything and everything in the
> > ecosystem page in Kafka itself but I'm not sure that would make the
> project
> > more agile.
> >
> > -Jay
> >
> > On Wed, Sep 28, 2016 at 12:04 AM, Manikumar 
> > wrote:
> >
> > > Hi Kafka Devs,
> > >
> > > I created KIP-80 to add Kafka REST Server to Kafka Repository.
> > >
> > > There are already open-source alternatives are available.  But we
would
> > > like to add REST server that
> > > many users ask for under Apache Kafka repo. Many data Infra tools
comes
> > up
> > > with Rest Interface.
> > > It is useful to have inbuilt Rest API support for Produce, Consume
> > messages
> > > and admin interface for
> > > integrating with external management and provisioning tools.This will
> > also
> > > allow the maintenance of
> > > REST server and adding new features makes it easy because apache
> > community.
> > >
> > > The KIP wiki is the following:
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 80%3A+Kafka+Rest+Server
> > >
> > > Your comments and feedback are welcome.
> > >
> > > Thanks,
> > > Manikumar
> > >
> >
>

Jenkins build is back to normal : kafka-trunk-jdk7 #1608

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-4272) Kafka Connect batch scripts are missing under `bin/windows/`

[Vahid Hashemian (JIRA)]

Vahid Hashemian created KAFKA-4272:
--

 Summary: Kafka Connect batch scripts are missing under 
`bin/windows/`
 Key: KAFKA-4272
 URL: https://issues.apache.org/jira/browse/KAFKA-4272
 Project: Kafka
  Issue Type: Improvement
Reporter: Vahid Hashemian


There are no {{connect-distributed.bat}} and {{connect-standalone.bat}} scripts 
under {{bin/windows/}} folder.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1986: HOTFIX: move restoreConsumer.assign() to shutdownT...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1986


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (KAFKA-4271) The console consumer fails on Windows with new consumer is used

[Vahid Hashemian (JIRA)]

Vahid Hashemian created KAFKA-4271:
--

 Summary: The console consumer fails on Windows with new consumer 
is used 
 Key: KAFKA-4271
 URL: https://issues.apache.org/jira/browse/KAFKA-4271
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.10.0.1
Reporter: Vahid Hashemian


When I try to consume message using the new consumer (Quickstart Step 5) I get 
an exception on the broker side. The old consumer works fine.

{code}
java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(Unknown Source)
at kafka.log.AbstractIndex.(AbstractIndex.scala:61)
at kafka.log.OffsetIndex.(OffsetIndex.scala:51)
at kafka.log.LogSegment.(LogSegment.scala:67)
at kafka.log.Log.loadSegments(Log.scala:255)
at kafka.log.Log.(Log.scala:108)
at kafka.log.LogManager.createLog(LogManager.scala:362)
at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:94)
at 
kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
at 
kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.apply(Partition.scala:174)
at scala.collection.mutable.HashSet.foreach(HashSet.scala:79)
at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:174)
at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:168)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:242)
at kafka.cluster.Partition.makeLeader(Partition.scala:168)
at 
kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:740)
at 
kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(ReplicaManager.scala:739)
at 
scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
at 
scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
at 
scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
at scala.collection.mutable.HashMap.foreach(HashMap.scala:98)
at kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:739)
at 
kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:685)
at kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:148)
at kafka.server.KafkaApis.handle(KafkaApis.scala:82)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method)
... 29 more
{code}

This issue seems to break the broker and I have to clear out the logs so I can 
bring the broker back up again.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-4257) Inconsistencies in 0.10.1 upgrade docs

[Jeff Klukas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Klukas resolved KAFKA-4257.

Resolution: Fixed

Ismael's PR addresses these questions.

> Inconsistencies in 0.10.1 upgrade docs 
> ---
>
> Key: KAFKA-4257
> URL: https://issues.apache.org/jira/browse/KAFKA-4257
> Project: Kafka
>  Issue Type: Bug
>  Components: documentation
>Affects Versions: 0.10.1.0
>Reporter: Jeff Klukas
>Priority: Minor
> Fix For: 0.10.1.0
>
>
> There are several inconsistencies in the 0.10.1.0 upgrade docs that make it 
> difficult to determine what client versions are compatible with what broker 
> versions.
> The initial heading in the upgrade docs is "Upgrading from 0.10.0.X to 
> 0.10.1.0", but it includes clauses about versions prior to 0.10. Is the 
> intention for these instructions to be valid for upgrading from brokers as 
> far back as 0.8? Should this section simply be called "Upgrading to 0.10.1.0"?
> I cannot tell from the docs whether I can upgrade to 0.10.1.0 clients on top 
> of 0.10.0.X brokers. In particular, step 5 of the upgrade instructions 
> mentions "Once all consumers have been upgraded to 0.10.0". Should that read 
> 0.10.1, or is the intention here truly that clients on 9.X or below need to 
> be at version 0.10.0.0 at a minimum?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4265) Intermittent test failure ReplicationQuotasTest.shouldBootstrapTwoBrokersWithFollowerThrottle

[Rajini Sivaram (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajini Sivaram updated KAFKA-4265:
--
Description: 
https://builds.apache.org/job/kafka-trunk-git-pr-jdk7/6085/testReport/junit/kafka.server/ReplicationQuotasTest/shouldBootstrapTwoBrokersWithFollowerThrottle

{quote}
java.lang.AssertionError: Offsets did not match for partition 0 on broker 100
at org.junit.Assert.fail(Assert.java:88)
at kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:752)
at 
kafka.server.ReplicationQuotasTest.kafka$server$ReplicationQuotasTest$$waitForOffsetsToMatch(ReplicationQuotasTest.scala:223)
at 
kafka.server.ReplicationQuotasTest$$anonfun$shouldMatchQuotaReplicatingThroughAnAsymmetricTopology$3.apply$mcZI$sp(ReplicationQuotasTest.scala:130)
at 
kafka.server.ReplicationQuotasTest$$anonfun$shouldMatchQuotaReplicatingThroughAnAsymmetricTopology$3.apply(ReplicationQuotasTest.scala:130)
at 
kafka.server.ReplicationQuotasTest$$anonfun$shouldMatchQuotaReplicatingThroughAnAsymmetricTopology$3.apply(ReplicationQuotasTest.scala:130)
at scala.collection.immutable.Range.foreach(Range.scala:141)
at 
kafka.server.ReplicationQuotasTest.shouldMatchQuotaReplicatingThroughAnAsymmetricTopology(ReplicationQuotasTest.scala:130)
at 
kafka.server.ReplicationQuotasTest.shouldBootstrapTwoBrokersWithFollowerThrottle(ReplicationQuotasTest.scala:73)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
{quote}

  was:
https://builds.apache.org/job/kafka-trunk-git-pr-jdk7/6085/testReport/junit/kafka.server/ReplicationQuotasTest/shouldBootstrapTwoBrokersWithFollowerThrottle:

{quote}
java.lang.AssertionError: Offsets did not match for partition 0 on broker 100
at org.junit.Assert.fail(Assert.java:88)
at kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:752)
at 
kafka.server.ReplicationQuotasTest.kafka$server$ReplicationQuotasTest$$waitForOffsetsToMatch(ReplicationQuotasTest.scala:223)
at 
kafka.server.ReplicationQuotasTest$$anonfun$shouldMatchQuotaReplicatingThroughAnAsymmetricTopology$3.apply$mcZI$sp(ReplicationQuotasTest.scala:130)
at 
kafka.server.ReplicationQuotasTest$$anonfun$shouldMatchQuotaReplicatingThroughAnAsymmetricTopology$3.apply(ReplicationQuotasTest.scala:130)
at 
kafka.server.ReplicationQuotasTest$$anonfun$shouldMatchQuotaReplicatingThroughAnAsymmetricTopology$3.apply(ReplicationQuotasTest.scala:130)
at scala.collection.immutable.Range.foreach(Range.scala:141)
at 
kafka.server.ReplicationQuotasTest.shouldMatchQuotaReplicatingThroughAnAsymmetricTopology(ReplicationQuotasTest.scala:130)
at 
kafka.server.ReplicationQuotasTest.shouldBootstrapTwoBrokersWithFollowerThrottle(ReplicationQuotasTest.scala:73)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
{quote}


> Intermittent test failure 
> ReplicationQuotasTest.shouldBootstrapTwoBrokersWithFollowerThrottle
> -
>
> Key: KAFKA-4265
> URL: https://issues.apache.org/jira/browse/KAFKA-4265
> Project: Kafka
>  Issue Type: Bug
>  Components: unit tests
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
>
> https://builds.apache.org/job/kafka-trunk-git-pr-jdk7/6085/testReport/junit/kafka.server/ReplicationQuotasTest/shouldBootstrapTwoBrokersWithFollowerThrottle
> {quote}
> java.lang.AssertionError: Offsets did not match for partition 0 on broker 100
>   at org.junit.Assert.fail(Assert.java:88)
>   at kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:752)
>   at 
> kafka.server.ReplicationQuotasTest.kafka$server$ReplicationQuotasTest$$waitForOffsetsToMatch(ReplicationQuotasTest.scala:223)
>   at 
> kafka.server.ReplicationQuotasTest$$anonfun$shouldMatchQuotaReplicatingThroughAnAsymmetricTopology$3.apply$mcZI$sp(ReplicationQuotasTest.scala:130)
>   at 
> kafka.server.ReplicationQuotasTest$$anonfun$shouldMatchQuotaReplicatingThroughAnAsymmetricTopology$3.apply(ReplicationQuotasTest.scala:130)
>   at 
> kafka.server.ReplicationQuotasTest$$anonfun$shouldMatchQuotaReplicatingThroughAnAsymmetricTopology$3.apply(ReplicationQuotasTest.scala:130)
>   at scala.collection.immutable.Range.foreach(Range.scala:141)
>   at 
> kafka.server.ReplicationQuotasTest.shouldMatchQuotaReplicatingThroughAnAsymmetricTopology(ReplicationQuotasTest.scala:130)
>   at 
> kafka.server.ReplicationQuotasTest.shouldBootstrapTwoBrokersWithFollowerThrottle(ReplicationQuotasTest.scala:73)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4265) Intermittent test failure ReplicationQuotasTest.shouldBootstrapTwoBrokersWithFollowerThrottle

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1779#comment-1779
 ] 

ASF GitHub Bot commented on KAFKA-4265:
---

GitHub user rajinisivaram opened a pull request:

https://github.com/apache/kafka/pull/1991

KAFKA-4265: Run replication quotas test with producer acks=1

Test expects all records to be published successfully, which cannot be 
guaranteed with acks=0 since failures are not retried.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rajinisivaram/kafka KAFKA-4265

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1991.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1991


commit ec3cc4adcebdce213a023c4ec8081aa1132f78cf
Author: Rajini Sivaram 
Date:   2016-10-07T17:32:29Z

KAFKA-4265: Run replication quotas test with producer acks=1




> Intermittent test failure 
> ReplicationQuotasTest.shouldBootstrapTwoBrokersWithFollowerThrottle
> -
>
> Key: KAFKA-4265
> URL: https://issues.apache.org/jira/browse/KAFKA-4265
> Project: Kafka
>  Issue Type: Bug
>  Components: unit tests
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
>
> https://builds.apache.org/job/kafka-trunk-git-pr-jdk7/6085/testReport/junit/kafka.server/ReplicationQuotasTest/shouldBootstrapTwoBrokersWithFollowerThrottle:
> {quote}
> java.lang.AssertionError: Offsets did not match for partition 0 on broker 100
>   at org.junit.Assert.fail(Assert.java:88)
>   at kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:752)
>   at 
> kafka.server.ReplicationQuotasTest.kafka$server$ReplicationQuotasTest$$waitForOffsetsToMatch(ReplicationQuotasTest.scala:223)
>   at 
> kafka.server.ReplicationQuotasTest$$anonfun$shouldMatchQuotaReplicatingThroughAnAsymmetricTopology$3.apply$mcZI$sp(ReplicationQuotasTest.scala:130)
>   at 
> kafka.server.ReplicationQuotasTest$$anonfun$shouldMatchQuotaReplicatingThroughAnAsymmetricTopology$3.apply(ReplicationQuotasTest.scala:130)
>   at 
> kafka.server.ReplicationQuotasTest$$anonfun$shouldMatchQuotaReplicatingThroughAnAsymmetricTopology$3.apply(ReplicationQuotasTest.scala:130)
>   at scala.collection.immutable.Range.foreach(Range.scala:141)
>   at 
> kafka.server.ReplicationQuotasTest.shouldMatchQuotaReplicatingThroughAnAsymmetricTopology(ReplicationQuotasTest.scala:130)
>   at 
> kafka.server.ReplicationQuotasTest.shouldBootstrapTwoBrokersWithFollowerThrottle(ReplicationQuotasTest.scala:73)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1991: KAFKA-4265: Run replication quotas test with produ...

[rajinisivaram]

GitHub user rajinisivaram opened a pull request:

https://github.com/apache/kafka/pull/1991

KAFKA-4265: Run replication quotas test with producer acks=1

Test expects all records to be published successfully, which cannot be 
guaranteed with acks=0 since failures are not retried.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rajinisivaram/kafka KAFKA-4265

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1991.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1991


commit ec3cc4adcebdce213a023c4ec8081aa1132f78cf
Author: Rajini Sivaram 
Date:   2016-10-07T17:32:29Z

KAFKA-4265: Run replication quotas test with producer acks=1




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: kafka-0.10.1-jdk7 #53

[Apache Jenkins Server]

See 

Changes:

[jason] MINOR: Use `hiResClockMs` in `testRequestExpiry` to fix transient test

--
[...truncated 1120 lines...]
kafka.utils.timer.TimerTest > testTaskExpiration PASSED

kafka.utils.timer.TimerTaskListTest > testAll STARTED

kafka.utils.timer.TimerTaskListTest > testAll PASSED

kafka.utils.SchedulerTest > testMockSchedulerNonPeriodicTask STARTED

kafka.utils.SchedulerTest > testMockSchedulerNonPeriodicTask PASSED

kafka.utils.SchedulerTest > testMockSchedulerPeriodicTask STARTED

kafka.utils.SchedulerTest > testMockSchedulerPeriodicTask PASSED

kafka.utils.SchedulerTest > testNonPeriodicTask STARTED

kafka.utils.SchedulerTest > testNonPeriodicTask PASSED

kafka.utils.SchedulerTest > testRestart STARTED

kafka.utils.SchedulerTest > testRestart PASSED

kafka.utils.SchedulerTest > testReentrantTaskInMockScheduler STARTED

kafka.utils.SchedulerTest > testReentrantTaskInMockScheduler PASSED

kafka.utils.SchedulerTest > testPeriodicTask STARTED

kafka.utils.SchedulerTest > testPeriodicTask PASSED

kafka.utils.CommandLineUtilsTest > testParseEmptyArg STARTED

kafka.utils.CommandLineUtilsTest > testParseEmptyArg PASSED

kafka.utils.CommandLineUtilsTest > testParseSingleArg STARTED

kafka.utils.CommandLineUtilsTest > testParseSingleArg PASSED

kafka.utils.CommandLineUtilsTest > testParseArgs STARTED

kafka.utils.CommandLineUtilsTest > testParseArgs PASSED

kafka.utils.CommandLineUtilsTest > testParseEmptyArgAsValid STARTED

kafka.utils.CommandLineUtilsTest > testParseEmptyArgAsValid PASSED

kafka.utils.UtilsTest > testGenerateUuidAsBase64 STARTED

kafka.utils.UtilsTest > testGenerateUuidAsBase64 PASSED

kafka.utils.UtilsTest > testAbs STARTED

kafka.utils.UtilsTest > testAbs PASSED

kafka.utils.UtilsTest > testReplaceSuffix STARTED

kafka.utils.UtilsTest > testReplaceSuffix PASSED

kafka.utils.UtilsTest > testCircularIterator STARTED

kafka.utils.UtilsTest > testCircularIterator PASSED

kafka.utils.UtilsTest > testReadBytes STARTED

kafka.utils.UtilsTest > testReadBytes PASSED

kafka.utils.UtilsTest > testCsvList STARTED

kafka.utils.UtilsTest > testCsvList PASSED

kafka.utils.UtilsTest > testReadInt STARTED

kafka.utils.UtilsTest > testReadInt PASSED

kafka.utils.UtilsTest > testUrlSafeBase64EncodeUUID STARTED

kafka.utils.UtilsTest > testUrlSafeBase64EncodeUUID PASSED

kafka.utils.UtilsTest > testCsvMap STARTED

kafka.utils.UtilsTest > testCsvMap PASSED

kafka.utils.UtilsTest > testInLock STARTED

kafka.utils.UtilsTest > testInLock PASSED

kafka.utils.UtilsTest > testSwallow STARTED

kafka.utils.UtilsTest > testSwallow PASSED

kafka.utils.JsonTest > testJsonEncoding STARTED

kafka.utils.JsonTest > testJsonEncoding PASSED

kafka.utils.IteratorTemplateTest > testIterator STARTED

kafka.utils.IteratorTemplateTest > testIterator PASSED

kafka.metrics.KafkaTimerTest > testKafkaTimer STARTED

kafka.metrics.KafkaTimerTest > testKafkaTimer PASSED

kafka.metrics.MetricsTest > testMetricsReporterAfterDeletingTopic STARTED

kafka.metrics.MetricsTest > testMetricsReporterAfterDeletingTopic PASSED

kafka.metrics.MetricsTest > 
testBrokerTopicMetricsUnregisteredAfterDeletingTopic STARTED

kafka.metrics.MetricsTest > 
testBrokerTopicMetricsUnregisteredAfterDeletingTopic PASSED

kafka.metrics.MetricsTest > testClusterIdMetric STARTED

kafka.metrics.MetricsTest > testClusterIdMetric PASSED

kafka.metrics.MetricsTest > testMetricsLeak STARTED

kafka.metrics.MetricsTest > testMetricsLeak PASSED

kafka.consumer.TopicFilterTest > testWhitelists STARTED

kafka.consumer.TopicFilterTest > testWhitelists PASSED

kafka.consumer.TopicFilterTest > 
testWildcardTopicCountGetTopicCountMapEscapeJson STARTED

kafka.consumer.TopicFilterTest > 
testWildcardTopicCountGetTopicCountMapEscapeJson PASSED

kafka.consumer.TopicFilterTest > testBlacklists STARTED

kafka.consumer.TopicFilterTest > testBlacklists PASSED

kafka.consumer.PartitionAssignorTest > testRoundRobinPartitionAssignor STARTED

kafka.consumer.PartitionAssignorTest > testRoundRobinPartitionAssignor PASSED

kafka.consumer.PartitionAssignorTest > testRangePartitionAssignor STARTED

kafka.consumer.PartitionAssignorTest > testRangePartitionAssignor PASSED

kafka.consumer.ZookeeperConsumerConnectorTest > testBasic STARTED

kafka.consumer.ZookeeperConsumerConnectorTest > testBasic PASSED

kafka.consumer.ZookeeperConsumerConnectorTest > testCompressionSetConsumption 
STARTED

kafka.consumer.ZookeeperConsumerConnectorTest > testCompressionSetConsumption 
PASSED

kafka.consumer.ZookeeperConsumerConnectorTest > testLeaderSelectionForPartition 
STARTED

kafka.consumer.ZookeeperConsumerConnectorTest > testLeaderSelectionForPartition 
PASSED

kafka.consumer.ZookeeperConsumerConnectorTest > testConsumerDecoder STARTED

kafka.consumer.ZookeeperConsumerConnectorTest > testConsumerDecoder PASSED

kafka.consumer.ZookeeperConsumerConnectorTest > testConsumerRebalanceListener 
STARTED

[GitHub] kafka pull request #1990: MINOR: Update Quickstart in documentation to accou...

[vahidhashemian]

GitHub user vahidhashemian opened a pull request:

https://github.com/apache/kafka/pull/1990

MINOR: Update Quickstart in documentation to account for Windows platforms



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vahidhashemian/kafka 
doc/quickstart_update_windows

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1990.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1990


commit fe5712354d137b6a46b0531678b0d7eb90bd4637
Author: Vahid Hashemian 
Date:   2016-10-07T17:48:03Z

MINOR: Update Quickstart in documentation to account for Windows platforms




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-4257) Inconsistencies in 0.10.1 upgrade docs

[Jason Gustafson (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1692#comment-1692
 ] 

Jason Gustafson commented on KAFKA-4257:


[~jeff.klu...@gmail.com] Can we resolve this issue with Ismael's improvements 
merged?

> Inconsistencies in 0.10.1 upgrade docs 
> ---
>
> Key: KAFKA-4257
> URL: https://issues.apache.org/jira/browse/KAFKA-4257
> Project: Kafka
>  Issue Type: Bug
>  Components: documentation
>Affects Versions: 0.10.1.0
>Reporter: Jeff Klukas
>Priority: Minor
> Fix For: 0.10.1.0
>
>
> There are several inconsistencies in the 0.10.1.0 upgrade docs that make it 
> difficult to determine what client versions are compatible with what broker 
> versions.
> The initial heading in the upgrade docs is "Upgrading from 0.10.0.X to 
> 0.10.1.0", but it includes clauses about versions prior to 0.10. Is the 
> intention for these instructions to be valid for upgrading from brokers as 
> far back as 0.8? Should this section simply be called "Upgrading to 0.10.1.0"?
> I cannot tell from the docs whether I can upgrade to 0.10.1.0 clients on top 
> of 0.10.0.X brokers. In particular, step 5 of the upgrade instructions 
> mentions "Once all consumers have been upgraded to 0.10.0". Should that read 
> 0.10.1, or is the intention here truly that clients on 9.X or below need to 
> be at version 0.10.0.0 at a minimum?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4217) KStream.transform equivalent of flatMap

[Elias Levy (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1654#comment-1654
 ] 

Elias Levy commented on KAFKA-4217:
---

It would seem to be the same request: allow a transform that emits multiple 
values.

Like Fodor, I am using {{ProcessorContext.forward}} to emit multiple values 
from {{Transformer.transform}}. Unlike him, I return {{null}} from the method 
instead of returning dummy values that must be filtered.

At the very least, it should be documented that one can use 
{{ProcessorContext.forward}} to emit multiple values from 
{{Transformer.transform}}.  Ideally, {{Transformer.transform}} would be 
modified to allow returning multiple values, or a variant of {{Transformer}} 
would allow you to do so.

> KStream.transform equivalent of flatMap
> ---
>
> Key: KAFKA-4217
> URL: https://issues.apache.org/jira/browse/KAFKA-4217
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.0.1
>Reporter: Elias Levy
>Assignee: Guozhang Wang
>
> {{KStream.transform}} gives you access to state stores while allowing you to 
> return zero or one transformed {{KeyValue}}.  Alas, it is unclear what method 
> you should use if you want to access state stores and return zero or multiple 
> {{KeyValue}}.  Presumably you can use {{transform}}, always return {{null}}, 
> and use {{ProcessorContext.forward}} to emit {{KeyValues}}.
> It may be good to introduce a {{transform}}-like {{flatMap}} equivalent, or 
> allow store access from other {{KStream}} methods, such as {{flatMap}} itself.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1629) Replica fetcher thread need to back off upon getting errors on partitions

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson updated KAFKA-1629:
---
Fix Version/s: (was: 0.10.1.0)

> Replica fetcher thread need to back off upon getting errors on partitions
> -
>
> Key: KAFKA-1629
> URL: https://issues.apache.org/jira/browse/KAFKA-1629
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
>  Labels: newbie++
>
> ReplicaFetcherThread's handlePartitionsWithErrors() function needs to be 
> implemented (currently it is an empty function) such that upon getting errors 
> on these partitions, the fetcher thread will back off the corresponding 
> simple consumer to retry fetching that partition.
> This can happen when there is leader migration, the replica may get a bit 
> delayed receiving the leader ISR update request before keeping retry fetching 
> the old leader.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1183) DefaultEventHandler causes unbalanced distribution of messages across partitions

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson updated KAFKA-1183:
---
Fix Version/s: (was: 0.10.1.0)
   (was: 0.8.1)

> DefaultEventHandler causes unbalanced distribution of messages across 
> partitions
> 
>
> Key: KAFKA-1183
> URL: https://issues.apache.org/jira/browse/KAFKA-1183
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Affects Versions: 0.8.0
>Reporter: Dragos Dena
>Assignee: Jun Rao
> Attachments: KAFKA-1183-trunk.patch
>
>
> KAFKA-959 introduced an optimisation in {{DefaultEventHandler}} that was 
> supposed to have the effect of sending all messages from the same batch to a 
> single partition if no key is specified.
> The problem is that the {{sendPartitionPerTopicCache}} cache, which holds the 
> current selected partition for each topic, isn't actually invalided at the 
> start or end of each batch.
> The observed result is that, after the first request chooses a random 
> partition, all subsequent messages from that producer land in the same 
> partition. If you have a large number of producers, then it should be fine, 
> but if your producer count is comparable to the partition count, then it will 
> get unbalanced.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1255) Offset in RecordMetadata is Incorrect with New Producer Ack = -1

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson updated KAFKA-1255:
---
Fix Version/s: (was: 0.10.1.0)

> Offset in RecordMetadata is Incorrect with New Producer Ack = -1
> 
>
> Key: KAFKA-1255
> URL: https://issues.apache.org/jira/browse/KAFKA-1255
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
>Assignee: Jay Kreps
> Attachments: sendwithAckMinusOne, sendwithAckOne
>
>
> With the new producer's integration test, one observation is that when 
> producer ack = -1, the returned offset is incorrect.
> Output files with two scenarios (send 100 messages with ack = 1 and -1) 
> attached.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-0.10.1-jdk7 #52

[Apache Jenkins Server]

See 

Changes:

[ismael] KAFKA-4262; Increase data volume in replication test

--
[...truncated 3832 lines...]

kafka.coordinator.GroupCoordinatorResponseTest > 
testSyncGroupLeaderAfterFollower STARTED

kafka.coordinator.GroupCoordinatorResponseTest > 
testSyncGroupLeaderAfterFollower PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testSyncGroupFromUnknownMember 
STARTED

kafka.coordinator.GroupCoordinatorResponseTest > testSyncGroupFromUnknownMember 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testValidLeaveGroup STARTED

kafka.coordinator.GroupCoordinatorResponseTest > testValidLeaveGroup PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testDescribeGroupInactiveGroup 
STARTED

kafka.coordinator.GroupCoordinatorResponseTest > testDescribeGroupInactiveGroup 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testSyncGroupNotCoordinator 
STARTED

kafka.coordinator.GroupCoordinatorResponseTest > testSyncGroupNotCoordinator 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testHeartbeatUnknownConsumerExistingGroup STARTED

kafka.coordinator.GroupCoordinatorResponseTest > 
testHeartbeatUnknownConsumerExistingGroup PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testValidHeartbeat STARTED

kafka.coordinator.GroupCoordinatorResponseTest > testValidHeartbeat PASSED

kafka.coordinator.GroupMetadataTest > testDeadToAwaitingSyncIllegalTransition 
STARTED

kafka.coordinator.GroupMetadataTest > testDeadToAwaitingSyncIllegalTransition 
PASSED

kafka.coordinator.GroupMetadataTest > testOffsetCommitFailure STARTED

kafka.coordinator.GroupMetadataTest > testOffsetCommitFailure PASSED

kafka.coordinator.GroupMetadataTest > 
testPreparingRebalanceToStableIllegalTransition STARTED

kafka.coordinator.GroupMetadataTest > 
testPreparingRebalanceToStableIllegalTransition PASSED

kafka.coordinator.GroupMetadataTest > testStableToDeadTransition STARTED

kafka.coordinator.GroupMetadataTest > testStableToDeadTransition PASSED

kafka.coordinator.GroupMetadataTest > testInitNextGenerationEmptyGroup STARTED

kafka.coordinator.GroupMetadataTest > testInitNextGenerationEmptyGroup PASSED

kafka.coordinator.GroupMetadataTest > testCannotRebalanceWhenDead STARTED

kafka.coordinator.GroupMetadataTest > testCannotRebalanceWhenDead PASSED

kafka.coordinator.GroupMetadataTest > testInitNextGeneration STARTED

kafka.coordinator.GroupMetadataTest > testInitNextGeneration PASSED

kafka.coordinator.GroupMetadataTest > testPreparingRebalanceToEmptyTransition 
STARTED

kafka.coordinator.GroupMetadataTest > testPreparingRebalanceToEmptyTransition 
PASSED

kafka.coordinator.GroupMetadataTest > testSelectProtocol STARTED

kafka.coordinator.GroupMetadataTest > testSelectProtocol PASSED

kafka.coordinator.GroupMetadataTest > testCannotRebalanceWhenPreparingRebalance 
STARTED

kafka.coordinator.GroupMetadataTest > testCannotRebalanceWhenPreparingRebalance 
PASSED

kafka.coordinator.GroupMetadataTest > 
testDeadToPreparingRebalanceIllegalTransition STARTED

kafka.coordinator.GroupMetadataTest > 
testDeadToPreparingRebalanceIllegalTransition PASSED

kafka.coordinator.GroupMetadataTest > testCanRebalanceWhenAwaitingSync STARTED

kafka.coordinator.GroupMetadataTest > testCanRebalanceWhenAwaitingSync PASSED

kafka.coordinator.GroupMetadataTest > 
testAwaitingSyncToPreparingRebalanceTransition STARTED

kafka.coordinator.GroupMetadataTest > 
testAwaitingSyncToPreparingRebalanceTransition PASSED

kafka.coordinator.GroupMetadataTest > testStableToAwaitingSyncIllegalTransition 
STARTED

kafka.coordinator.GroupMetadataTest > testStableToAwaitingSyncIllegalTransition 
PASSED

kafka.coordinator.GroupMetadataTest > testEmptyToDeadTransition STARTED

kafka.coordinator.GroupMetadataTest > testEmptyToDeadTransition PASSED

kafka.coordinator.GroupMetadataTest > testSelectProtocolRaisesIfNoMembers 
STARTED

kafka.coordinator.GroupMetadataTest > testSelectProtocolRaisesIfNoMembers PASSED

kafka.coordinator.GroupMetadataTest > testStableToPreparingRebalanceTransition 
STARTED

kafka.coordinator.GroupMetadataTest > testStableToPreparingRebalanceTransition 
PASSED

kafka.coordinator.GroupMetadataTest > testPreparingRebalanceToDeadTransition 
STARTED

kafka.coordinator.GroupMetadataTest > testPreparingRebalanceToDeadTransition 
PASSED

kafka.coordinator.GroupMetadataTest > testStableToStableIllegalTransition 
STARTED

kafka.coordinator.GroupMetadataTest > testStableToStableIllegalTransition PASSED

kafka.coordinator.GroupMetadataTest > testAwaitingSyncToStableTransition STARTED

kafka.coordinator.GroupMetadataTest > testAwaitingSyncToStableTransition PASSED

kafka.coordinator.GroupMetadataTest > testOffsetCommitFailureWithAnotherPending 
STARTED

kafka.coordinator.GroupMetadataTest > testOffsetCommitFailureWithAnotherPending 
PASSED

kafka.coordinator.GroupMetadataTest > testDeadToStableIllegalTransition STARTED

[jira] [Updated] (KAFKA-1592) Some INFO level logging needs to be DEBUG

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson updated KAFKA-1592:
---
Fix Version/s: (was: 0.10.1.0)

> Some INFO level logging needs to be DEBUG
> -
>
> Key: KAFKA-1592
> URL: https://issues.apache.org/jira/browse/KAFKA-1592
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
>Assignee: Mayuresh Gharat
>  Labels: newbie
>
> Some of the INFO level log4j entries are not really useful, for example in 
> SocketServer.Processor, due to metadata requests that reply on a separate and 
> short-lived socket, the following log can be constantly printed:
> info("Closing socket connection to 
> %s.".format(channelFor(key).socket.getInetAddress)) 
> We'd better move them to DEBUG if they are expected in normal state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1554) Corrupt index found on clean startup

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson updated KAFKA-1554:
---
Fix Version/s: (was: 0.10.0.0)
   (was: 0.10.1.0)

> Corrupt index found on clean startup
> 
>
> Key: KAFKA-1554
> URL: https://issues.apache.org/jira/browse/KAFKA-1554
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.1
> Environment: ubuntu 12.04, oracle jdk 1.7
>Reporter: Alexis Midon
>Assignee: Mayuresh Gharat
>Priority: Critical
> Attachments: KAFKA-1554.patch
>
>
> On a clean start up, corrupted index files are found.
> After investigations, it appears that some pre-allocated index files are not 
> "compacted" correctly and the end of the file is full of zeroes.
> As a result, on start up, the last relative offset is zero which yields an 
> offset equal to the base offset.
> The workaround is to delete all index files of size 10MB (the size of the 
> pre-allocated files), and restart. Index files will be re-created.
> {code}
> find $your_data_directory -size 10485760c -name *.index #-delete
> {code}
> This is issue might be related/similar to 
> https://issues.apache.org/jira/browse/KAFKA-1112
> {code}
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,696 
> INFO main kafka.server.KafkaServer.info - [Kafka Server 847605514], starting
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,698 
> INFO main kafka.server.KafkaServer.info - [Kafka Server 847605514], 
> Connecting to zookeeper on 
> zk-main0.XXX:2181,zk-main1.XXX:2181,zk-main2.:2181/production/kafka/main
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,708 
> INFO 
> ZkClient-EventThread-14-zk-main0.XXX.com:2181,zk-main1.XXX.com:2181,zk-main2.XXX.com:2181,zk-main3.XXX.com:2181,zk-main4.XXX.com:2181/production/kafka/main
>  org.I0Itec.zkclient.ZkEventThread.run - Starting ZkClient event thread.
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,714 
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
> environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,714 
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
> environment:host.name=i-6b948138.inst.aws.airbnb.com
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,714 
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
> environment:java.version=1.7.0_55
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,715 
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
> environment:java.vendor=Oracle Corporation
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,715 
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
> environment:java.home=/usr/lib/jvm/jre-7-oracle-x64/jre
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,715 
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
> environment:java.class.path=libs/snappy-java-1.0.5.jar:libs/scala-library-2.10.1.jar:libs/slf4j-api-1.7.2.jar:libs/jopt-simple-3.2.jar:libs/metrics-annotation-2.2.0.jar:libs/log4j-1.2.15.jar:libs/kafka_2.10-0.8.1.jar:libs/zkclient-0.3.jar:libs/zookeeper-3.3.4.jar:libs/metrics-core-2.2.0.jar
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,715 
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,716 
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
> environment:java.io.tmpdir=/tmp
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,716 
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
> environment:java.compiler=
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,716 
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
> environment:os.name=Linux
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,716 
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
> environment:os.arch=amd64
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,717 
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
> environment:os.version=3.2.0-61-virtual
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,717 
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
> environment:user.name=kafka
> 2014-07-11T00:53:17+00:00 i-6b948138 local3.info 2014-07-11 - 00:53:17,717 
> INFO main org.apache.zookeeper.ZooKeeper.logEnv - Client 
> environment:user.home=/srv/kafka
> 

[jira] [Updated] (KAFKA-2658) Implement SASL/PLAIN

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson updated KAFKA-2658:
---
Fix Version/s: (was: 0.10.0.0)
   (was: 0.10.1.0)

> Implement SASL/PLAIN
> 
>
> Key: KAFKA-2658
> URL: https://issues.apache.org/jira/browse/KAFKA-2658
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Affects Versions: 0.9.0.0
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
>Priority: Critical
>
> KAFKA-1686 supports SASL/Kerberos using GSSAPI. We should enable more SASL 
> mechanisms. SASL/PLAIN would enable a simpler use of SASL, which along with 
> SSL provides a secure Kafka that uses username/password for client 
> authentication.
> SASL/PLAIN protocol and its uses are described in 
> [https://tools.ietf.org/html/rfc4616]. It is supported in Java.
> This should be implemented after KAFKA-1686. This task should also hopefully 
> enable simpler unit testing of the SASL code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-trunk-jdk7 #1607

[Apache Jenkins Server]

See 

Changes:

[ismael] KAFKA-4267; Quota initialization for  uses incorrect ZK

--
[...truncated 12106 lines...]

org.apache.kafka.clients.producer.internals.SenderTest > testSendInOrder PASSED

org.apache.kafka.clients.producer.internals.SenderTest > testSimple STARTED

org.apache.kafka.clients.producer.internals.SenderTest > testSimple PASSED

org.apache.kafka.clients.producer.internals.BufferPoolTest > 
testStressfulSituation STARTED

org.apache.kafka.clients.producer.internals.BufferPoolTest > 
testStressfulSituation PASSED

org.apache.kafka.clients.producer.internals.BufferPoolTest > 
testCleanupMemoryAvailabilityWaiterOnInterruption STARTED

org.apache.kafka.clients.producer.internals.BufferPoolTest > 
testCleanupMemoryAvailabilityWaiterOnInterruption PASSED

org.apache.kafka.clients.producer.internals.BufferPoolTest > testBlockTimeout 
STARTED

org.apache.kafka.clients.producer.internals.BufferPoolTest > testBlockTimeout 
PASSED

org.apache.kafka.clients.producer.internals.BufferPoolTest > 
testCantAllocateMoreMemoryThanWeHave STARTED

org.apache.kafka.clients.producer.internals.BufferPoolTest > 
testCantAllocateMoreMemoryThanWeHave PASSED

org.apache.kafka.clients.producer.internals.BufferPoolTest > 
testCleanupMemoryAvailabilityWaiterOnBlockTimeout STARTED

org.apache.kafka.clients.producer.internals.BufferPoolTest > 
testCleanupMemoryAvailabilityWaiterOnBlockTimeout PASSED

org.apache.kafka.clients.producer.internals.BufferPoolTest > testSimple STARTED

org.apache.kafka.clients.producer.internals.BufferPoolTest > testSimple PASSED

org.apache.kafka.clients.producer.internals.BufferPoolTest > 
testDelayedAllocation STARTED

org.apache.kafka.clients.producer.internals.BufferPoolTest > 
testDelayedAllocation PASSED

org.apache.kafka.clients.MetadataTest > testListenerCanUnregister STARTED

org.apache.kafka.clients.MetadataTest > testListenerCanUnregister PASSED

org.apache.kafka.clients.MetadataTest > testTopicExpiry STARTED

org.apache.kafka.clients.MetadataTest > testTopicExpiry PASSED

org.apache.kafka.clients.MetadataTest > testFailedUpdate STARTED

org.apache.kafka.clients.MetadataTest > testFailedUpdate PASSED

org.apache.kafka.clients.MetadataTest > testMetadataUpdateWaitTime STARTED

org.apache.kafka.clients.MetadataTest > testMetadataUpdateWaitTime PASSED

org.apache.kafka.clients.MetadataTest > testUpdateWithNeedMetadataForAllTopics 
STARTED

org.apache.kafka.clients.MetadataTest > testUpdateWithNeedMetadataForAllTopics 
PASSED

org.apache.kafka.clients.MetadataTest > testClusterListenerGetsNotifiedOfUpdate 
STARTED

org.apache.kafka.clients.MetadataTest > testClusterListenerGetsNotifiedOfUpdate 
PASSED

org.apache.kafka.clients.MetadataTest > testMetadata STARTED

org.apache.kafka.clients.MetadataTest > testMetadata PASSED

org.apache.kafka.clients.MetadataTest > testListenerGetsNotifiedOfUpdate STARTED

org.apache.kafka.clients.MetadataTest > testListenerGetsNotifiedOfUpdate PASSED

org.apache.kafka.clients.MetadataTest > testNonExpiringMetadata STARTED

org.apache.kafka.clients.MetadataTest > testNonExpiringMetadata PASSED

org.apache.kafka.clients.ClientUtilsTest > testOnlyBadHostname STARTED

org.apache.kafka.clients.ClientUtilsTest > testOnlyBadHostname PASSED

org.apache.kafka.clients.ClientUtilsTest > testParseAndValidateAddresses STARTED

org.apache.kafka.clients.ClientUtilsTest > testParseAndValidateAddresses PASSED

org.apache.kafka.clients.ClientUtilsTest > testNoPort STARTED

org.apache.kafka.clients.ClientUtilsTest > testNoPort PASSED

org.apache.kafka.clients.NetworkClientTest > testSimpleRequestResponse STARTED

org.apache.kafka.clients.NetworkClientTest > testSimpleRequestResponse PASSED

org.apache.kafka.clients.NetworkClientTest > 
testDisconnectDuringUserMetadataRequest STARTED

org.apache.kafka.clients.NetworkClientTest > 
testDisconnectDuringUserMetadataRequest PASSED

org.apache.kafka.clients.NetworkClientTest > testClose STARTED

org.apache.kafka.clients.NetworkClientTest > testClose PASSED

org.apache.kafka.clients.NetworkClientTest > testLeastLoadedNode STARTED

org.apache.kafka.clients.NetworkClientTest > testLeastLoadedNode PASSED

org.apache.kafka.clients.NetworkClientTest > testConnectionDelayConnected 
STARTED

org.apache.kafka.clients.NetworkClientTest > testConnectionDelayConnected PASSED

org.apache.kafka.clients.NetworkClientTest > testRequestTimeout STARTED

org.apache.kafka.clients.NetworkClientTest > testRequestTimeout PASSED

org.apache.kafka.clients.NetworkClientTest > 
testSimpleRequestResponseWithStaticNodes STARTED

org.apache.kafka.clients.NetworkClientTest > 
testSimpleRequestResponseWithStaticNodes PASSED

org.apache.kafka.clients.NetworkClientTest > testConnectionDelay STARTED

org.apache.kafka.clients.NetworkClientTest > testConnectionDelay PASSED

org.apache.kafka.clients.NetworkClientTest > testSendToUnreadyNode STARTED

[jira] [Updated] (KAFKA-3511) Add common aggregation functions like Sum and Avg as build-ins in Kafka Streams DSL

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson updated KAFKA-3511:
---
Fix Version/s: (was: 0.10.1.0)

> Add common aggregation functions like Sum and Avg as build-ins in Kafka 
> Streams DSL
> ---
>
> Key: KAFKA-3511
> URL: https://issues.apache.org/jira/browse/KAFKA-3511
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Guozhang Wang
>Assignee: Eno Thereska
>  Labels: api
>
> Currently we have the following aggregation APIs in the Streams DSL:
> {code}
> KStream.aggregateByKey(..)
> KStream.reduceByKey(..)
> KStream.countByKey(..)
> KTable.groupBy(...).aggregate(..)
> KTable.groupBy(...).reduce(..)
> KTable.groupBy(...).count(..)
> {code}
> And it is better to add common aggregation functions like Sum and Avg as 
> built-in into the Streams DSL. A few questions to ask though:
> 1. Should we add those built-in functions as, for example 
> {{KTable.groupBy(...).sum(...)} or {{KTable.groupBy(...).aggregate(SUM, 
> ...)}}. Please see the comments below for detailed pros and cons.
> 2. If we go with the second option above, should we replace the countByKey / 
> count operators with aggregate(COUNT) as well? Personally I (Guozhang) feel 
> it is not necessary, as COUNT is a special aggregate function since we do not 
> need to map on any value fields; this is the same approach as in Spark as 
> well, where Count is built-in as first-citizen in the DSL, and others are 
> built-in as {{aggregate(SUM)}}, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-777) Add system tests for important tools

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson updated KAFKA-777:
--
Fix Version/s: (was: 0.10.1.0)

> Add system tests for important tools
> 
>
> Key: KAFKA-777
> URL: https://issues.apache.org/jira/browse/KAFKA-777
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.0
>Reporter: Sriram Subramanian
>Assignee: John Fung
>  Labels: kafka-0.8, p2, replication-testing
> Attachments: KAFKA-777.patch, KAFKA-777.patch
>
>
> Few tools were broken after the zk format change. It would be great to catch 
> these issues during system tests. Some of the tools are 
> 1. ShudownBroker
> 2. PreferredReplicaAssignment
> 3. ConsumerOffsetChecker
> There might be a few more for which we need tests. Need to add them once 
> identified.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1589) Strengthen System Tests

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson updated KAFKA-1589:
---
Fix Version/s: (was: 0.10.1.0)

> Strengthen System Tests
> ---
>
> Key: KAFKA-1589
> URL: https://issues.apache.org/jira/browse/KAFKA-1589
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
>Assignee: Ewen Cheslack-Postava
>  Labels: newbie
>
> Although the system test code is also part of the open source repository, not 
> too much attention is paid to this module today. The incurred results is that 
> we keep breaking the system tests with either changes on the admin tools, or 
> library upgrades that change the APIs like Zookeeper. And when the system 
> tests breaks / hangs / etc, it is also hard to debug the issue. We need to 
> treat the system test suite just as part of the open source code. 
> Based on my personal experience trouble shooting system tests, I would 
> propose doing at least the follow enhancement around system tests.
> 1. Add unit tests for all system util test tools, for example:
> kafka_system_test_utils.get_controller_attributes
> kafka_system_test_utils.get_leader_for
> 2. Add exception handling logic in the python test framework to clean-up the 
> testbed upon failures, so that the subsequent test cases will not be affected.
> 3. Remove timing based mechanism such as "sleep(5000) to wait for metadata to 
> be propagated" as much as possible to avoid transient failures.
> After those enhancements, we should probably also pick a very small subset 
> (say one from each suite) of the system test cases into the patch reviewing 
> process along with the unit tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-478) Move start_consumer & start_producer inside "start_entity_in_background"

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson updated KAFKA-478:
--
Fix Version/s: (was: 0.10.1.0)

> Move start_consumer & start_producer inside "start_entity_in_background"
> 
>
> Key: KAFKA-478
> URL: https://issues.apache.org/jira/browse/KAFKA-478
> Project: Kafka
>  Issue Type: Sub-task
>Affects Versions: 0.8.0
>Reporter: John Fung
>Assignee: John Fung
>  Labels: replication-testing
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-564) Wildcard-based topic consumption should assign partitions to threads uniformly

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson updated KAFKA-564:
--
Fix Version/s: (was: 0.10.1.0)

> Wildcard-based topic consumption should assign partitions to threads uniformly
> --
>
> Key: KAFKA-564
> URL: https://issues.apache.org/jira/browse/KAFKA-564
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.1
>Reporter: Joel Koshy
>
> Right now, if when a client uses createMessageStreamsByFilter and specifies 
> 'n' streams (threads), 'n' should be <= the max partition count of any topic. 
> If it is greater than that, the excess threads will be idle. However, it 
> would be better to allow a greater number of threads and spread all the 
> available partitions across the threads.
> This should not be too difficult, but may require significant refactoring.
> Although it is relevant to current trunk/0.7, we will target this for 
> post-0.8.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-288) java impacted changes from new producer and consumer request format

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson updated KAFKA-288:
--
Fix Version/s: (was: 0.10.1.0)

> java impacted changes from new producer and consumer request format
> ---
>
> Key: KAFKA-288
> URL: https://issues.apache.org/jira/browse/KAFKA-288
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.0
>Reporter: Joe Stein
>  Labels: replication, wireprotocol
>
> 1) javaapi.SyncProducer: We should get rid of send(topic: String, messages: 
> ByteBufferMessageSet) and only keep send(producerRequest: 
> kafka.javaapi.ProducerRequest). 
> This affects KafkaRecordWriter and DataGenerator
> 2)  javaapi.ProducerRequest: We will need to define a java version of 
> TopicData so that java producers can create request conveniently. The java 
> version of TopicData will use the java version of ByteBufferMessageSet. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-558) KafkaETLContext should use getTopicMetadata before sending offset requests

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson updated KAFKA-558:
--
Fix Version/s: (was: 0.10.1.0)

> KafkaETLContext should use getTopicMetadata before sending offset requests
> --
>
> Key: KAFKA-558
> URL: https://issues.apache.org/jira/browse/KAFKA-558
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.0
>Reporter: Joel Koshy
>Assignee: Joel Koshy
>
> Filing this or I may forget.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-289) reuse topicdata when sending producerrequest

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson updated KAFKA-289:
--
Fix Version/s: (was: 0.10.1.0)

> reuse topicdata when sending producerrequest
> 
>
> Key: KAFKA-289
> URL: https://issues.apache.org/jira/browse/KAFKA-289
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.0
>Reporter: Joe Stein
>  Labels: optimization, replication, wireprotocol
>
> The way that SyncProducer sends a ProducerRequest over socket is to first 
> serialize the whole request in a bytebuffer and then sends the bytebuffer 
> through socket. An alternative is to send the request like FetchReponse, 
> using a ProduceRequestSend that reuses TopicDataSend. This avoids code 
> duplication and is more efficient since it sends data in 
> ByteBufferMessagesSet directly to socket and avoids extra copying from 
> messageset to bytebuffer. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3637) Add method that checks if streams are initialised

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson updated KAFKA-3637:
---
Fix Version/s: (was: 0.10.1.0)

> Add method that checks if streams are initialised
> -
>
> Key: KAFKA-3637
> URL: https://issues.apache.org/jira/browse/KAFKA-3637
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams
>Affects Versions: 0.10.1.0
>Reporter: Eno Thereska
>Assignee: Liquan Pei
>  Labels: newbie
>
> Currently when streams are initialised and started with streams.start(), 
> there is no way for the caller to know if the initialisation procedure 
> (including starting tasks) is complete or not. Hence, the caller is forced to 
> guess for how long to wait. It would be good to have a way to return the 
> state of the streams to the caller.
> One option would be to follow a similar approach in Kafka Server 
> (BrokerStates.scala).
> As part of this change, we must remove the Thread.sleep() call in the Kafka 
> Streams integration tests and substitute it with TestUtils.waitUntilTrue().



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3873) Gradle Test Executor non-zero exit code when running streams tests

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson updated KAFKA-3873:
---
Fix Version/s: (was: 0.10.1.0)

> Gradle Test Executor non-zero exit code when running streams tests
> --
>
> Key: KAFKA-3873
> URL: https://issues.apache.org/jira/browse/KAFKA-3873
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams, unit tests
>Reporter: Ismael Juma
>Assignee: Damian Guy
>  Labels: transient-unit-test-failure
>
> This happened in a couple of builds:
> {code}
> org.apache.kafka.streams.integration.KStreamRepartitionJoinTest > 
> shouldFlatMapJoin PASSED
> :streams:test FAILED
> FAILURE: Build failed with an exception.
> * What went wrong:
> Execution failed for task ':streams:test'.
> > Process 'Gradle Test Executor 4' finished with non-zero exit value 1
> {code}
> https://builds.apache.org/job/kafka-trunk-jdk7/1369/console
> {code}
> org.apache.kafka.streams.integration.KGroupedStreamIntegrationTest > 
> shouldAggregateWindowed PASSED
> :streams:test FAILED
> FAILURE: Build failed with an exception.
> * What went wrong:
> Execution failed for task ':streams:test'.
> > Process 'Gradle Test Executor 4' finished with non-zero exit value 1
> {code}
> https://builds.apache.org/job/kafka-trunk-jdk7/1374/console



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3759) Incorrect JDBC credentials cause Connect worker to permanently fail

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson updated KAFKA-3759:
---
Fix Version/s: (was: 0.10.1.0)

> Incorrect JDBC credentials cause Connect worker to permanently fail
> ---
>
> Key: KAFKA-3759
> URL: https://issues.apache.org/jira/browse/KAFKA-3759
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 0.9.0.1
> Environment: I work on a cloudera quickstart VM. I installed kafka / 
> confluent via yum :
>  
> [cloudera@quickstart confluent]$ yum list installed | grep confluent
> confluent-camus.noarch  2.0.1-1   @confluent-2.0  
>  
> confluent-common.noarch 2.0.1-1   @confluent-2.0  
>  
> confluent-kafka-2.11.7.noarch   0.9.0.1-1 @confluent-2.0  
>  
> confluent-kafka-connect-hdfs.noarch 2.0.1-1   @confluent-2.0  
>  
> confluent-kafka-connect-jdbc.noarch 2.0.1-1   @confluent-2.0  
>  
> confluent-kafka-rest.noarch 2.0.1-1   @confluent-2.0  
>  
> confluent-rest-utils.noarch 2.0.1-1   @confluent-2.0  
>  
> confluent-schema-registry.noarch2.0.1-1   @confluent-2.0  
>  
>Reporter: Lars George
>Assignee: Ewen Cheslack-Postava
>
> As reported by our peers:
> All the following steps are executed in one cloudera instance, each in a 
> different terminal.
> - I start the kafka server with the (unchanged) properties file in 
> /etc/kafka/server.properties
> - I start the schema-registry with the (unchanged) properties file in 
> /etc/schema-registry/schema-registry.properties
> - I start the connect worker with the properties file 
> /etc/schema-registry/connect-avro-distributed.properties
> HOWEVER I have changed the following 2 properties for my test:
> config.storage.topic=connect-configs-broken
> offset.storage.topic=connect-offsets-broken
>  
> Now I execute a shell script that uses the REST API to post a connector 
> configuration to the worker. Here the snippet from the script
> that shows the connector config:
> {noformat} 
> ==
> read -resp $'please enter the password for the postgresql user \n' 
> postgres_password
>  
> echo posting connector config into the kafka cluster
> data="{
> \"name\": \"projectx-postgres-test\",
> \"config\": {
>
> \"connection.url\":\"jdbc:postgresql://$postgres_db_url/$postgres_db_name?user=$postgres_user=$postgres_password\",
>
> \"connector.class\":\"io.confluent.connect.jdbc.JdbcSourceConnector\",
>\"tasks.max\":\"1\",
>\"mode\":\"incrementing\",
>\"incrementing.column.name\":\"id\",
>\"topic.prefix\":\"test-postgres-jdbc-\"
> }
> }"
> url=http://$kafka_connect_rest_url/connectors
> ctype="Content-Type: application/json"
> curl -X POST $url -H "$ctype" --data "$data"
>  
> ==
> {noformat}
>  
> - I exectute the script with correct connection settings, but incorrect 
> postgresql username / password.
> - I receive the following answer from the REST API:
> {noformat}
> {"name":"projectx-postgres-test","config":{"connection.url":"jdbc:postgresql://localhost/connect_test?user=cloudera=wrongPW","connector.class":"io.confluent.connect.jdbc.JdbcSourceConnector","tasks.max":"1","mode":"incrementing","incrementing.column.name":"id","topic.prefix":"test-postgres-jdbc-","name":"projectx-postgres-test"},"tasks":[]}
> {noformat}
>  
> Now the connect worker stops and I see the following error and stack trace:
>  
> {noformat}
> [2016-05-23 05:53:43,966] INFO 127.0.0.1 - - [23/May/2016:12:53:41 +] 
> "POST /connectors HTTP/1.1" 201 343  2500 
> (org.apache.kafka.connect.runtime.rest.RestServer:60)
> [2016-05-23 05:53:44,656] ERROR Couldn't open connection to 
> jdbc:postgresql://localhost/connect_test?user=cloudera=wrongPW: 
> org.postgresql.util.PSQLException: FATAL: password authentication failed for 
> user "cloudera" (io.confluent.connect.jdbc.JdbcSourceConnector:76)
> [2016-05-23 05:53:44,663] ERROR Uncaught exception in herder work thread, 
> exiting:  (org.apache.kafka.connect.runtime.distributed.DistributedHerder:166)
> org.apache.kafka.connect.errors.ConnectException: Connector threw an 
> exception while starting
> at 
> org.apache.kafka.connect.runtime.Worker.addConnector(Worker.java:188)
> at 
> org.apache.kafka.connect.runtime.distributed.DistributedHerder.startConnector(DistributedHerder.java:670)
> at 

[jira] [Updated] (KAFKA-2901) Extend ListGroups and DescribeGroup APIs to cover offsets

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson updated KAFKA-2901:
---
Fix Version/s: (was: 0.10.1.0)

> Extend ListGroups and DescribeGroup APIs to cover offsets
> -
>
> Key: KAFKA-2901
> URL: https://issues.apache.org/jira/browse/KAFKA-2901
> Project: Kafka
>  Issue Type: Improvement
>  Components: consumer
>Affects Versions: 0.9.0.0
>Reporter: Andy Coates
>Assignee: Jason Gustafson
>
> The {{ListGroupsRequest}} and {{DescribeGroupsRequest}} added to 0.9.0.0 
> allow admin tools to get details of consumer groups now that this information 
> is not stored in ZK.
> The brokers also now store offset information for consumer groups. At 
> present, there is no API for admin tools to discover the groups that brokers 
> are storing offsets for.
> For example, if a consumer is using the new consumer api, is storing offsets 
> in Kafka under the groupId 'Bob', but is using manual partition assignment, 
> then the {{groupCache}} in the {{GroupMetadataManager}} will not contain any 
> information about the group 'Bob'. However, the {{offsetCache}} in the 
> {{GroupMetadataManager}} will contain information about 'Bob'.
> Currently the only way for admin tools to know the full set of groups being 
> managed by Kafka, i.e. those storing offsets in Kafka, those using Kafka for 
> balancing of consumer groups, and those using Kafka for both, is to consume 
> the offset topic.
> We need to extend the List/Describe groups API to allow admin tools to 
> discover 'Bob' and allow the partition offsets to be retrieved.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3183) Add metrics for persistent store caching layer

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson updated KAFKA-3183:
---
Fix Version/s: (was: 0.10.1.0)

> Add metrics for persistent store caching layer
> --
>
> Key: KAFKA-3183
> URL: https://issues.apache.org/jira/browse/KAFKA-3183
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Guozhang Wang
>  Labels: user-experience
>
> We need to add the metrics collection such as cache hits / misses, cache 
> size, dirty key size, etc for the RocksDBStore. However this may need to 
> refactor the RocksDBStore a little bit since currently caching is not exposed 
> to the MeteredKeyValueStore, and it uses an LRUCacheStore as the cache that 
> does not keep the dirty key set.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3101) Optimize Aggregation Outputs

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson updated KAFKA-3101:
---
Fix Version/s: (was: 0.10.1.0)

> Optimize Aggregation Outputs
> 
>
> Key: KAFKA-3101
> URL: https://issues.apache.org/jira/browse/KAFKA-3101
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Guozhang Wang
>  Labels: architecture
>
> Today we emit one output record for each incoming message for Table / 
> Windowed Stream Aggregations. For example, say we have a sequence of 
> aggregate outputs computed from the input stream (assuming there is no agg 
> value for this key before):
> V1, V2, V3, V4, V5
> Then the aggregator will output the following sequence of Change oldValue>:
> , , , , 
> where could cost a lot of CPU overhead computing the intermediate results. 
> Instead if we can let the underlying state store to "remember" the last 
> emitted old value, we can reduce the number of emits based on some configs. 
> More specifically, we can add one more field in the KV store engine storing 
> the last emitted old value, which only get updated when we emit to the 
> downstream processor. For example:
> At Beginning: 
> Store: key => empty (no agg values yet)
> V1 computed: 
> Update Both in Store: key => (V1, V1), Emit 
> V2 computed: 
> Update NewValue in Store: key => (V2, V1), No Emit
> V3 computed: 
> Update NewValue in Store: key => (V3, V1), No Emit
> V4 computed: 
> Update Both in Store: key => (V4, V4), Emit 
> V5 computed: 
> Update NewValue in Store: key => (V5, V4), No Emit
> One more thing to consider is that, we need a "closing" time control on the 
> not-yet-emitted keys; when some time has elapsed (or the window is to be 
> closed), we need to check for any key if their current materialized pairs 
> have not been emitted (for example  in the above example). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3913) Old consumer's metrics error when using IPv6

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson updated KAFKA-3913:
---
Fix Version/s: (was: 0.10.1.0)

> Old consumer's metrics error when using IPv6 
> -
>
> Key: KAFKA-3913
> URL: https://issues.apache.org/jira/browse/KAFKA-3913
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.9.0.1, 0.10.0.0
>Reporter: Pengwei
>
> The error is below:
> [2016-05-09 15:49:20,096] WARN Property security.protocol is not valid 
> (kafka.utils.VerifiableProperties)
> [2016-05-09 15:49:20,882] WARN Error processing 
> kafka.server:type=FetcherStats,name=RequestsPerSec,clientId=console-consumer-32775,brokerHost=fe80::92e2:baff:fe07:51cc,brokerPort=9093
>  (com.yammer.metrics.reporting.JmxReporter)
> javax.management.MalformedObjectNameException: Invalid character ':' in value 
> part of property
> at javax.management.ObjectName.construct(ObjectName.java:618)
> at javax.management.ObjectName.(ObjectName.java:1382)
> at 
> com.yammer.metrics.reporting.JmxReporter.onMetricAdded(JmxReporter.java:395)
> at 
> com.yammer.metrics.core.MetricsRegistry.notifyMetricAdded(MetricsRegistry.java:516)
> at com.yammer.metrics.core.MetricsRegistry.getOrAdd(MetricsRegistry.java:491)
> at com.yammer.metrics.core.MetricsRegistry.newMeter(MetricsRegistry.java:240)
> at kafka.metrics.KafkaMetricsGroup$class.newMeter(KafkaMetricsGroup.scala:79)
> at kafka.server.FetcherStats.newMeter(AbstractFetcherThread.scala:264)
> at kafka.server.FetcherStats.(AbstractFetcherThread.scala:269)
> at kafka.server.AbstractFetcherThread.(AbstractFetcherThread.scala:55)
> at kafka.consumer.ConsumerFetcherThread.(ConsumerFetcherThread.scala:38)
> at 
> kafka.consumer.ConsumerFetcherManager.createFetcherThread(ConsumerFetcherManager.scala:118)
> at 
> kafka.server.AbstractFetcherManager$$anonfun$addFetcherForPartitions$2.apply(AbstractFetcherManager.scala:83)
> at 
> kafka.server.AbstractFetcherManager$$anonfun$addFetcherForPartitions$2.apply(AbstractFetcherManager.scala:78)
> at 
> scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:778)
> at scala.collection.immutable.Map$Map1.foreach(Map.scala:116)
> at 
> scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:777)
> at 
> kafka.server.AbstractFetcherManager.addFetcherForPartitions(AbstractFetcherManager.scala:78)
> at 
> kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:95)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4074) Deleting a topic can make it unavailable even if delete.topic.enable is false

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson updated KAFKA-4074:
---
Fix Version/s: (was: 0.10.1.0)

> Deleting a topic can make it unavailable even if delete.topic.enable is false
> -
>
> Key: KAFKA-4074
> URL: https://issues.apache.org/jira/browse/KAFKA-4074
> Project: Kafka
>  Issue Type: Bug
>  Components: controller
>Reporter: Joel Koshy
>Assignee: Manikumar Reddy
>
> The {{delete.topic.enable}} configuration does not completely block the 
> effects of delete topic since the controller may (indirectly) query the list 
> of topics under the delete-topic znode.
> To reproduce:
> * Delete topic X
> * Force a controller move (either by bouncing or removing the controller 
> znode)
> * The new controller will send out UpdateMetadataRequests with leader=-2 for 
> the partitions of X
> * Producers eventually stop producing to that topic
> The reason for this is that when ControllerChannelManager adds 
> UpdateMetadataRequests for brokers, we directly use the partitionsToBeDeleted 
> field of the DeleteTopicManager (which is set to the partitions of the topics 
> under the delete-topic znode on controller startup).
> In order to get out of the situation you have to remove X from the znode and 
> then force another controller move.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1324) Debian packaging

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson updated KAFKA-1324:
---
Fix Version/s: (was: 0.10.1.0)

> Debian packaging
> 
>
> Key: KAFKA-1324
> URL: https://issues.apache.org/jira/browse/KAFKA-1324
> Project: Kafka
>  Issue Type: Improvement
>  Components: packaging
> Environment: linux
>Reporter: David Stendardi
>Priority: Minor
>  Labels: deb, debian, fpm, newbie, packaging
> Attachments: packaging.patch
>
>
> The following patch add a task releaseDeb to the gradle build :
> ./gradlew releaseDeb
> This task should create a debian package in core/build/distributions using 
> fpm :
> https://github.com/jordansissel/fpm.
> We decided to use fpm so other package types would be easy to provide in 
> further iterations (eg : rpm).
> *Some implementations details* :
> - We splitted the releaseTarGz in two tasks : distDir, releaseTarGz.
> - We tried to use gradle builtin variables (project.name etc...)
> - By default the service will not start automatically so the user is free to 
> setup the service with custom configuration.
> Notes : 
>  * FPM is required and should be in the path.
>  * FPM does not allow yet to declare /etc/default/kafka as a conffiles (see : 
> https://github.com/jordansissel/fpm/issues/668)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1613) Improve system test documentation

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson updated KAFKA-1613:
---
Fix Version/s: (was: 0.10.1.0)

> Improve system test documentation
> -
>
> Key: KAFKA-1613
> URL: https://issues.apache.org/jira/browse/KAFKA-1613
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Gwen Shapira
>Assignee: Ewen Cheslack-Postava
>Priority: Minor
>  Labels: newbie
>
> Few things to improve the docs:
> 1. Include python pre-requisites (matplotlib for example) and instructions 
> how to install them.
> 2. The README needs some cleanup, and reference to the wiki. For example, I'm 
> pretty sure system tests works with OS X.
> 3. Extra documentation about metrics and charts - they seem missing in action
> 4. Extra documentation about the cluster config - when to touch it, when not 
> to, why we have different levels, etc. Maybe example of cluster_conf.json 
> configured to run with remote hosts.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3801) Provide static serialize() and deserialize() for use as method references

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson updated KAFKA-3801:
---
Fix Version/s: (was: 0.10.1.0)

> Provide static serialize() and deserialize() for use as method references
> -
>
> Key: KAFKA-3801
> URL: https://issues.apache.org/jira/browse/KAFKA-3801
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients, streams
>Reporter: Jeff Klukas
>Assignee: Guozhang Wang
>Priority: Minor
>
> While most calls to {{Serializer.serialize}} and {{Deserializer.deserialize}} 
> are abstracted away in Kafka Streams through the use of `Serdes` classes, 
> there are some instances where developers may want to call them directly. The 
> serializers and deserializers for simple types don't require any 
> configuration and could be static, but currently it's necessary to create an 
> instance to use those methods.
> I'd propose moving serialization logic into a {{static public byte[] 
> serialize(? data)}} method and deserialization logic into a {{static public ? 
> deserialize(byte[] data)}} method. The existing instance methods would simply 
> call the static versions.
> See a full example for LongSerializer and LongDeserializer here:
> https://github.com/apache/kafka/compare/trunk...jklukas:static-serde-methods?expand=1
> In Java 8, these static methods then become available for method references 
> in code like {{kstream.mapValues(LongDeserializer::deserialize)}} without the 
> user needing to create an instance of {{LongDeserializer}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-789) Producer-side persistence for delivery guarantee

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson updated KAFKA-789:
--
Fix Version/s: (was: 0.10.1.0)

> Producer-side persistence for delivery guarantee
> 
>
> Key: KAFKA-789
> URL: https://issues.apache.org/jira/browse/KAFKA-789
> Project: Kafka
>  Issue Type: Improvement
>  Components: producer 
>Reporter: Matan Safriel
>Assignee: Jun Rao
>Priority: Minor
>
> A suggestion for higher guarantee for the part of entering messages into 
> Kafka through it's producer. It aims to address the case that the entire set 
> of broker replicas for a topic and partition is not available. Currently, in 
> that case, data is lost. When a message set exhausts the send retry counter, 
> the message set will be simply dropped. It would be nice being able to 
> provide higher guarantee that a message passed to the producer would 
> eventually be received by the broker. 
> In an environment with some disk space to spare for this on the producer 
> side, persisting to disk would seem to enable keeping messages for later 
> retry (until defined space limits are exhausted). Thus somewhat elevating the 
> level of guarantee. 
> One way to facilitate this would be capitalizing on 
> https://issues.apache.org/jira/browse/KAFKA-496, as the feedback it will add 
> will enable knowing what needs to be retried again later. Changes to the 
> producer or a wrapper around it (that may require access to the partitioning 
> functions) would be able to persist failed message sets and manage delivery 
> with a nice level of guarantee. As it would affect performance and use disks, 
> should probably be a non-default option.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1040) ConsumerConfig and ProducerConfig do "work" in the Constructor

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson updated KAFKA-1040:
---
Fix Version/s: (was: 0.10.1.0)

> ConsumerConfig and ProducerConfig do "work" in the Constructor
> --
>
> Key: KAFKA-1040
> URL: https://issues.apache.org/jira/browse/KAFKA-1040
> Project: Kafka
>  Issue Type: Improvement
>  Components: config, consumer, producer 
>Affects Versions: 0.8.0
> Environment: Java 1.7
> Linux Mint 14 (64bit)
>Reporter: Sharmarke Aden
>Assignee: Neha Narkhede
>Priority: Minor
>  Labels: config, newbie
>
> It appears that validation of configuration properties is performed in the 
> ConsumerConfig and ProducerConfig constructors. This is generally bad 
> practice as it couples object construction and validation. It also makes it 
> difficult to mock these objects in unit tests. 
> Ideally validation of the configuration properties should be separated from 
> object construction and initiated by those that rely/use these config objects.
> http://misko.hevery.com/code-reviewers-guide/flaw-constructor-does-real-work/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] 0.10.1.0 RC0

[Vahid S Hashemian]

Jason,

Sure, I'll submit a patch for the trivial changes in the quick start.
Do you recommend adding Windows version of commands along with the current 
commands?

I'll also open a JIRA for the new consumer issue.

--Vahid



From:   Jason Gustafson 
To: dev@kafka.apache.org
Cc: Kafka Users 
Date:   10/07/2016 08:57 AM
Subject:Re: [VOTE] 0.10.1.0 RC0



@Vahid Thanks, do you want to submit a patch for the quickstart fixes? We
won't need another RC if it's just doc changes. The exception is a little
more troubling. Perhaps open a JIRA and we can begin investigation? It's
especially strange that you say it's specific to the new consumer.

@Henry Actually that issue was resolved as "won't fix" since it pointed to
an old version of the group coordinator design. But maybe it's misleading
that we include JIRAs resolved as "won't fix" in the first place. At least
they ought to be listed in a separate section?

-Jason

On Thu, Oct 6, 2016 at 5:27 PM, Henry Cai 
wrote:

> Why is this feature in the release note?
>
>
>- [KAFKA-264 ] -
> Change
>the consumer side load balancing and distributed co-ordination to use 
a
>consumer co-ordinator
>
> I thought this was already done in 2015.
>
> On Thu, Oct 6, 2016 at 4:55 PM, Vahid S Hashemian <
> vahidhashem...@us.ibm.com
> > wrote:
>
> > Jason,
> >
> > Thanks a lot for managing this release.
> >
> > I ran the quick start (Steps 2-8) with this release candidate on 
Ubuntu,
> > Windows, and Mac and they mostly look great.
> > These are some, hopefully, minor items and gaps I noticed with respect 
to
> > the existing quick start documentation (and the updated quick start 
that
> > leverages the new consumer).
> > They may very well be carryovers from previous releases, or perhaps
> > specific to my local environments.
> > Hopefully others can confirm.
> >
> >
> > Windows
> >
> > Since there are separate scripts on Windows platform, it probably 
would
> > help if that is clarified in the quick start section. E.g. "On Windows
> > platform replace `bin/` with `bin\windows\`". Or even have a separate
> > quick start for Windows since a number of commands will be different 
on
> > Windows.
> > There is no `connect-standalone.sh` equivalent for Windows under
> > bin\windows folder (Step 7).
> > Step 8 is also not tailored for Windows terminals. I skipped this 
step.
> > When I try to consume message using the new consumer (Step 5) I get an
> > exception on the broker side. The old consumer works fine.
> >
> > java.io.IOException: Map failed
> > at sun.nio.ch.FileChannelImpl.map(Unknown Source)
> > at kafka.log.AbstractIndex.(AbstractIndex.scala:61)
> > at kafka.log.OffsetIndex.(OffsetIndex.scala:51)
> > at kafka.log.LogSegment.(LogSegment.scala:67)
> > at kafka.log.Log.loadSegments(Log.scala:255)
> > at kafka.log.Log.(Log.scala:108)
> > at kafka.log.LogManager.createLog(LogManager.scala:362)
> > at kafka.cluster.Partition.getOrCreateReplica(Partition.
> scala:94)
> > at
> > kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.
> > apply(Partition.scala:174)
> > at
> > kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.
> > apply(Partition.scala:174)
> > at scala.collection.mutable.HashSet.foreach(HashSet.scala:79)
> > at 
kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:174)
> > at 
kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:168)
> > at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
> > at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:242)
> > at kafka.cluster.Partition.makeLeader(Partition.scala:168)
> > at
> > kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(
> > ReplicaManager.scala:740)
> > at
> > kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(
> > ReplicaManager.scala:739)
> > at
> > scala.collection.mutable.HashMap$$anonfun$foreach$1.
> > apply(HashMap.scala:98)
> > at
> > scala.collection.mutable.HashMap$$anonfun$foreach$1.
> > apply(HashMap.scala:98)
> > at
> > scala.collection.mutable.HashTable$class.foreachEntry(
> HashTable.scala:226)
> > at scala.collection.mutable.HashMap.foreachEntry(HashMap.
> scala:39)
> > at scala.collection.mutable.HashMap.foreach(HashMap.scala:98)
> > at
> > kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:739)
> > at
> > kafka.server.ReplicaManager.becomeLeaderOrFollower(
> > ReplicaManager.scala:685)
> > at
> > kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:148)
> > at kafka.server.KafkaApis.handle(KafkaApis.scala:82)
> > at
> > kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
> > at java.lang.Thread.run(Unknown Source)
> > Caused by: java.lang.OutOfMemoryError: Map failed

Re: [DISCUSS] KIP-82 - Add Record Headers

[Joel Koshy]

Hi Jay,

Couple of comments inline:

One of things that has helped keep Kafka simple is not adding in new
> abstractions and concepts except when the proposal is really elegant and
> makes things simpler.
>

I don't quite see how this impacts simplicity because (per your taxonomy)
the scope is "company" and "world". So the decision to use it is or not is
really up to companies/individuals.


> Consider three use cases for headers:
>
>1. Kafka-scope: We want to add a feature to Kafka that needs a
>particular field.
>2. Company-scope: You want to add a header to be shared by everyone in
>your company.
>3. World-wide scope: You are building a third party tool and want to add
>some kind of header.
>
> For the case of (1) you should not use headers, you should just add a field
> to the record format.


Agreed - which is what we have been doing so far.


> sense. Occasionally people have complained that adding to the record format
> is hard and it would be nice to just shove lots of things in quickly. I

think a better solution would be to make it easy to add to the record
> format, and I think we've made progress on that. I also think we should be
>

The intent is not to shove things in quickly but to make it possible to
augment the streaming system with infrastructure features such as
audit/call tracing and more without invading the record format for all
users who may not need some of those features.


> earlier proposals. These things end up being long term commitments so it's
> really worth being thoughtful.
>

Which is another reason I think it is helpful to have such headers. If a
certain header type is clearly useful for the vast majority of users then
there is a case for it being integrated directly into the record format.
However, there could always be a large segment of users for whom certain
headers are irrelevant and they should have the ability to opt out of it.

For case (2) just use the body of the message. You don't need a globally
> agreed on definition of headers, just standardize on a header you want to
> include in the value in your company.


This works - but as I described in an earlier email in this thread has
drawbacks.

   1. A global registry of numeric keys is super super ugly. This seems
>silly compared to the Avro (or whatever) header solution which gives
> more
>compact encoding, rich types, etc.
>

Agreed - I would really like us to avoid the burden of being a registrar.

   2. Using byte arrays for header values means they aren't really
>interoperable for case (3). E.g. I can't make a UI that displays
> headers,
>or allow you to set them in config. To work with third party headers,
> the
>only case I think this really helps, you need the union of all
>serialization schemes people have used for any tool.
>

I don't quite see why - the user would need to have the suitable
interceptors in their classpath. Headers that it does not understand are
simply ignored.

   3. For case (2) and (3) your key numbers are going to collide like
>crazy. I don't think a global registry of magic numbers maintained
> either
>by word of mouth or checking in changes to kafka source is the right
> thing
>to do.


Agreed (~ point 1 above)


>4. We are introducing a new serialization primitive which makes fields
>disappear conditional on the contents of other fields. This breaks the
>whole serialization/schema system we have today.
>

I don't quite see why this is so.


>6. This proposes making the ProducerRecord and ConsumerRecord mutable
>and adding setters and getters (which we try to avoid).
>

This is another part of the proposal I don't really like although there are
use cases where it helps.


> For context on LinkedIn: I set up the system there, but it may have changed
> since i left. The header is maintained with the record schemas in the avro
> schema registry and is required for all records. Essentially all messages
>


> Not allowing teams to chose a data format other than avro was considered a
> feature, not a bug, since the whole point was to be able to share data,
> which doesn't work if every team chooses their own format.
>

It is pretty much the same - not much has changed since you left, but see
my earlier comments on this: http://markmail.org/message/3ln5mruxqfhbewgz
The proposal does not mean it empowers applications to use non-Avro for the
data plane. The feature supports a much clearer separation of the user's
data from typically infra-related headers (which could also be Avro-based).

At this point I think we should focus the discussion not on the specifics
(implementation) of the proposal but on the motivation. Email is fine but
it may be better to discuss in a hangout and circle back to the thread.

Thanks,

Joel


> On Thu, Sep 22, 2016 at 12:31 PM, Michael Pearce 
> wrote:
>
> > Hi All,
> >
> >
> > I would like to discuss the following KIP proposal:
> >
> > 

Build failed in Jenkins: kafka-trunk-jdk8 #953

[Apache Jenkins Server]

See 

Changes:

[ismael] KAFKA-4262; Increase data volume in replication test

--
[...truncated 3832 lines...]

kafka.coordinator.GroupMetadataManagerTest > testExpireGroup STARTED

kafka.coordinator.GroupMetadataManagerTest > testExpireGroup PASSED

kafka.coordinator.GroupMetadataManagerTest > testAddGroup STARTED

kafka.coordinator.GroupMetadataManagerTest > testAddGroup PASSED

kafka.coordinator.GroupMetadataManagerTest > testCommitOffset STARTED

kafka.coordinator.GroupMetadataManagerTest > testCommitOffset PASSED

kafka.coordinator.GroupMetadataManagerTest > testStoreGroupErrorMapping STARTED

kafka.coordinator.GroupMetadataManagerTest > testStoreGroupErrorMapping PASSED

kafka.coordinator.GroupMetadataManagerTest > testCommitOffsetFailure STARTED

kafka.coordinator.GroupMetadataManagerTest > testCommitOffsetFailure PASSED

kafka.coordinator.GroupMetadataManagerTest > testExpireOffset STARTED

kafka.coordinator.GroupMetadataManagerTest > testExpireOffset PASSED

kafka.coordinator.GroupMetadataManagerTest > testExpireOffsetsWithActiveGroup 
STARTED

kafka.coordinator.GroupMetadataManagerTest > testExpireOffsetsWithActiveGroup 
PASSED

kafka.coordinator.GroupMetadataManagerTest > testStoreEmptyGroup STARTED

kafka.coordinator.GroupMetadataManagerTest > testStoreEmptyGroup PASSED

kafka.coordinator.MemberMetadataTest > testMatchesSupportedProtocols STARTED

kafka.coordinator.MemberMetadataTest > testMatchesSupportedProtocols PASSED

kafka.coordinator.MemberMetadataTest > testMetadata STARTED

kafka.coordinator.MemberMetadataTest > testMetadata PASSED

kafka.coordinator.MemberMetadataTest > testMetadataRaisesOnUnsupportedProtocol 
STARTED

kafka.coordinator.MemberMetadataTest > testMetadataRaisesOnUnsupportedProtocol 
PASSED

kafka.coordinator.MemberMetadataTest > testVoteForPreferredProtocol STARTED

kafka.coordinator.MemberMetadataTest > testVoteForPreferredProtocol PASSED

kafka.coordinator.MemberMetadataTest > testVoteRaisesOnNoSupportedProtocols 
STARTED

kafka.coordinator.MemberMetadataTest > testVoteRaisesOnNoSupportedProtocols 
PASSED

kafka.coordinator.GroupMetadataTest > testDeadToAwaitingSyncIllegalTransition 
STARTED

kafka.coordinator.GroupMetadataTest > testDeadToAwaitingSyncIllegalTransition 
PASSED

kafka.coordinator.GroupMetadataTest > testOffsetCommitFailure STARTED

kafka.coordinator.GroupMetadataTest > testOffsetCommitFailure PASSED

kafka.coordinator.GroupMetadataTest > 
testPreparingRebalanceToStableIllegalTransition STARTED

kafka.coordinator.GroupMetadataTest > 
testPreparingRebalanceToStableIllegalTransition PASSED

kafka.coordinator.GroupMetadataTest > testStableToDeadTransition STARTED

kafka.coordinator.GroupMetadataTest > testStableToDeadTransition PASSED

kafka.coordinator.GroupMetadataTest > testInitNextGenerationEmptyGroup STARTED

kafka.coordinator.GroupMetadataTest > testInitNextGenerationEmptyGroup PASSED

kafka.coordinator.GroupMetadataTest > testCannotRebalanceWhenDead STARTED

kafka.coordinator.GroupMetadataTest > testCannotRebalanceWhenDead PASSED

kafka.coordinator.GroupMetadataTest > testInitNextGeneration STARTED

kafka.coordinator.GroupMetadataTest > testInitNextGeneration PASSED

kafka.coordinator.GroupMetadataTest > testPreparingRebalanceToEmptyTransition 
STARTED

kafka.coordinator.GroupMetadataTest > testPreparingRebalanceToEmptyTransition 
PASSED

kafka.coordinator.GroupMetadataTest > testSelectProtocol STARTED

kafka.coordinator.GroupMetadataTest > testSelectProtocol PASSED

kafka.coordinator.GroupMetadataTest > testCannotRebalanceWhenPreparingRebalance 
STARTED

kafka.coordinator.GroupMetadataTest > testCannotRebalanceWhenPreparingRebalance 
PASSED

kafka.coordinator.GroupMetadataTest > 
testDeadToPreparingRebalanceIllegalTransition STARTED

kafka.coordinator.GroupMetadataTest > 
testDeadToPreparingRebalanceIllegalTransition PASSED

kafka.coordinator.GroupMetadataTest > testCanRebalanceWhenAwaitingSync STARTED

kafka.coordinator.GroupMetadataTest > testCanRebalanceWhenAwaitingSync PASSED

kafka.coordinator.GroupMetadataTest > 
testAwaitingSyncToPreparingRebalanceTransition STARTED

kafka.coordinator.GroupMetadataTest > 
testAwaitingSyncToPreparingRebalanceTransition PASSED

kafka.coordinator.GroupMetadataTest > testStableToAwaitingSyncIllegalTransition 
STARTED

kafka.coordinator.GroupMetadataTest > testStableToAwaitingSyncIllegalTransition 
PASSED

kafka.coordinator.GroupMetadataTest > testEmptyToDeadTransition STARTED

kafka.coordinator.GroupMetadataTest > testEmptyToDeadTransition PASSED

kafka.coordinator.GroupMetadataTest > testSelectProtocolRaisesIfNoMembers 
STARTED

kafka.coordinator.GroupMetadataTest > testSelectProtocolRaisesIfNoMembers PASSED

kafka.coordinator.GroupMetadataTest > testStableToPreparingRebalanceTransition 
STARTED

kafka.coordinator.GroupMetadataTest > testStableToPreparingRebalanceTransition 
PASSED

[GitHub] kafka pull request #1890: MINOR: Use `hiResClockMs` in `testRequestExpiry` t...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1890


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-264) Change the consumer side load balancing and distributed co-ordination to use a consumer co-ordinator

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson updated KAFKA-264:
--
Fix Version/s: (was: 0.10.1.0)

> Change the consumer side load balancing and distributed co-ordination to use 
> a consumer co-ordinator
> 
>
> Key: KAFKA-264
> URL: https://issues.apache.org/jira/browse/KAFKA-264
> Project: Kafka
>  Issue Type: New Feature
>  Components: core
>Affects Versions: 0.7, 0.8.0
>Reporter: Neha Narkhede
>Assignee: Neha Narkhede
> Attachments: KAFKA-264.v1.patch
>
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> A high level design for the zookeeper consumer is here - 
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Detailed+Consumer+Coordinator+Design



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] 0.10.1.0 RC0

[Jason Gustafson]

>
> I suggest not having a "Fix version" set for issues that don't fix anything
> (it's not part of any release really).


Yeah, good call.

On Fri, Oct 7, 2016 at 8:59 AM, Ismael Juma  wrote:

> On Fri, Oct 7, 2016 at 4:56 PM, Jason Gustafson 
> wrote:
>
> > @Vahid Thanks, do you want to submit a patch for the quickstart fixes? We
> > won't need another RC if it's just doc changes. The exception is a little
> > more troubling. Perhaps open a JIRA and we can begin investigation? It's
> > especially strange that you say it's specific to the new consumer.
> >
>
> Also worth checking if the behaviour is any different than 0.10.0.0. That
> looks like one of the known issues with the way we mmap on Windows.
>
> @Henry Actually that issue was resolved as "won't fix" since it pointed to
> > an old version of the group coordinator design. But maybe it's misleading
> > that we include JIRAs resolved as "won't fix" in the first place. At
> least
> > they ought to be listed in a separate section?
> >
>
> I suggest not having a "Fix version" set for issues that don't fix anything
> (it's not part of any release really).
>
> Ismael
>

Re: [VOTE] 0.10.1.0 RC0

[Ismael Juma]

On Fri, Oct 7, 2016 at 4:56 PM, Jason Gustafson  wrote:

> @Vahid Thanks, do you want to submit a patch for the quickstart fixes? We
> won't need another RC if it's just doc changes. The exception is a little
> more troubling. Perhaps open a JIRA and we can begin investigation? It's
> especially strange that you say it's specific to the new consumer.
>

Also worth checking if the behaviour is any different than 0.10.0.0. That
looks like one of the known issues with the way we mmap on Windows.

@Henry Actually that issue was resolved as "won't fix" since it pointed to
> an old version of the group coordinator design. But maybe it's misleading
> that we include JIRAs resolved as "won't fix" in the first place. At least
> they ought to be listed in a separate section?
>

I suggest not having a "Fix version" set for issues that don't fix anything
(it's not part of any release really).

Ismael

Re: [VOTE] 0.10.1.0 RC0

[Jason Gustafson]

@Vahid Thanks, do you want to submit a patch for the quickstart fixes? We
won't need another RC if it's just doc changes. The exception is a little
more troubling. Perhaps open a JIRA and we can begin investigation? It's
especially strange that you say it's specific to the new consumer.

@Henry Actually that issue was resolved as "won't fix" since it pointed to
an old version of the group coordinator design. But maybe it's misleading
that we include JIRAs resolved as "won't fix" in the first place. At least
they ought to be listed in a separate section?

-Jason

On Thu, Oct 6, 2016 at 5:27 PM, Henry Cai 
wrote:

> Why is this feature in the release note?
>
>
>- [KAFKA-264 ] -
> Change
>the consumer side load balancing and distributed co-ordination to use a
>consumer co-ordinator
>
> I thought this was already done in 2015.
>
> On Thu, Oct 6, 2016 at 4:55 PM, Vahid S Hashemian <
> vahidhashem...@us.ibm.com
> > wrote:
>
> > Jason,
> >
> > Thanks a lot for managing this release.
> >
> > I ran the quick start (Steps 2-8) with this release candidate on Ubuntu,
> > Windows, and Mac and they mostly look great.
> > These are some, hopefully, minor items and gaps I noticed with respect to
> > the existing quick start documentation (and the updated quick start that
> > leverages the new consumer).
> > They may very well be carryovers from previous releases, or perhaps
> > specific to my local environments.
> > Hopefully others can confirm.
> >
> >
> > Windows
> >
> > Since there are separate scripts on Windows platform, it probably would
> > help if that is clarified in the quick start section. E.g. "On Windows
> > platform replace `bin/` with `bin\windows\`". Or even have a separate
> > quick start for Windows since a number of commands will be different on
> > Windows.
> > There is no `connect-standalone.sh` equivalent for Windows under
> > bin\windows folder (Step 7).
> > Step 8 is also not tailored for Windows terminals. I skipped this step.
> > When I try to consume message using the new consumer (Step 5) I get an
> > exception on the broker side. The old consumer works fine.
> >
> > java.io.IOException: Map failed
> > at sun.nio.ch.FileChannelImpl.map(Unknown Source)
> > at kafka.log.AbstractIndex.(AbstractIndex.scala:61)
> > at kafka.log.OffsetIndex.(OffsetIndex.scala:51)
> > at kafka.log.LogSegment.(LogSegment.scala:67)
> > at kafka.log.Log.loadSegments(Log.scala:255)
> > at kafka.log.Log.(Log.scala:108)
> > at kafka.log.LogManager.createLog(LogManager.scala:362)
> > at kafka.cluster.Partition.getOrCreateReplica(Partition.
> scala:94)
> > at
> > kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.
> > apply(Partition.scala:174)
> > at
> > kafka.cluster.Partition$$anonfun$4$$anonfun$apply$2.
> > apply(Partition.scala:174)
> > at scala.collection.mutable.HashSet.foreach(HashSet.scala:79)
> > at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:174)
> > at kafka.cluster.Partition$$anonfun$4.apply(Partition.scala:168)
> > at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:234)
> > at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:242)
> > at kafka.cluster.Partition.makeLeader(Partition.scala:168)
> > at
> > kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(
> > ReplicaManager.scala:740)
> > at
> > kafka.server.ReplicaManager$$anonfun$makeLeaders$4.apply(
> > ReplicaManager.scala:739)
> > at
> > scala.collection.mutable.HashMap$$anonfun$foreach$1.
> > apply(HashMap.scala:98)
> > at
> > scala.collection.mutable.HashMap$$anonfun$foreach$1.
> > apply(HashMap.scala:98)
> > at
> > scala.collection.mutable.HashTable$class.foreachEntry(
> HashTable.scala:226)
> > at scala.collection.mutable.HashMap.foreachEntry(HashMap.
> scala:39)
> > at scala.collection.mutable.HashMap.foreach(HashMap.scala:98)
> > at
> > kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:739)
> > at
> > kafka.server.ReplicaManager.becomeLeaderOrFollower(
> > ReplicaManager.scala:685)
> > at
> > kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:148)
> > at kafka.server.KafkaApis.handle(KafkaApis.scala:82)
> > at
> > kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
> > at java.lang.Thread.run(Unknown Source)
> > Caused by: java.lang.OutOfMemoryError: Map failed
> > at sun.nio.ch.FileChannelImpl.map0(Native Method)
> > ... 29 more
> >
> > This issue seems to break the broker and I have to clear out the logs so
> I
> > can bring the broker back up again.
> >
> >
> > Ubuntu / Mac
> >
> > At Step 8, the output I'm seeing after going through the instructions in
> > sequence is this (with unique words)
> >
> > all 1
> > lead1
> > to  1
> > hello   1
> > 

Re: [DISCUSS] KIP-82 - Add Record Headers

[Jay Kreps]

Hey guys,

This discussion has come up a number of times and we've always passed.

One of things that has helped keep Kafka simple is not adding in new
abstractions and concepts except when the proposal is really elegant and
makes things simpler.

Consider three use cases for headers:

   1. Kafka-scope: We want to add a feature to Kafka that needs a
   particular field.
   2. Company-scope: You want to add a header to be shared by everyone in
   your company.
   3. World-wide scope: You are building a third party tool and want to add
   some kind of header.

For the case of (1) you should not use headers, you should just add a field
to the record format. Having a second way of encoding things doesn't make
sense. Occasionally people have complained that adding to the record format
is hard and it would be nice to just shove lots of things in quickly. I
think a better solution would be to make it easy to add to the record
format, and I think we've made progress on that. I also think we should be
insanely focused on the simplicity of the abstraction and not adding in new
thingies often---we thought about time for years before adding a timestamp
and I guarantee you we would have goofed it up if we'd gone with the
earlier proposals. These things end up being long term commitments so it's
really worth being thoughtful.

For case (2) just use the body of the message. You don't need a globally
agreed on definition of headers, just standardize on a header you want to
include in the value in your company. Since this is just used by code in
your company having a more standard header format doesn't really help you.
In fact by using something like Avro you can define exactly the types you
want, the required header fields, etc.

The only case that headers help is (3). This is a bit of a niche case and i
think is easily solved just making the reading and writing of given
required fields pluggable to work with the header you have.

A couple of specific problems with this proposal:

   1. A global registry of numeric keys is super super ugly. This seems
   silly compared to the Avro (or whatever) header solution which gives more
   compact encoding, rich types, etc.
   2. Using byte arrays for header values means they aren't really
   interoperable for case (3). E.g. I can't make a UI that displays headers,
   or allow you to set them in config. To work with third party headers, the
   only case I think this really helps, you need the union of all
   serialization schemes people have used for any tool.
   3. For case (2) and (3) your key numbers are going to collide like
   crazy. I don't think a global registry of magic numbers maintained either
   by word of mouth or checking in changes to kafka source is the right thing
   to do.
   4. We are introducing a new serialization primitive which makes fields
   disappear conditional on the contents of other fields. This breaks the
   whole serialization/schema system we have today.
   5. We're adding a hashmap to each record
   6. This proposes making the ProducerRecord and ConsumerRecord mutable
   and adding setters and getters (which we try to avoid).

For context on LinkedIn: I set up the system there, but it may have changed
since i left. The header is maintained with the record schemas in the avro
schema registry and is required for all records. Essentially all messages
must have a field named "header" of type EventHeader which is itself a
record schema with a handful of fields (time, host, etc). The header
follows the same compatibility rules as other avro fields, so it can be
evolved in a compatible way gradually across apps. Avro is typed and
doesn't require deserializing the full record to read the header. The
header information is (timestamp, host, etc) is important and needs to
propagate into other systems like Hadoop which don't have a concept of
headers for records, so I doubt it could move out of the value in any case.
Not allowing teams to chose a data format other than avro was considered a
feature, not a bug, since the whole point was to be able to share data,
which doesn't work if every team chooses their own format.

I agree with the critique of compaction not having a value. I think we
should consider fixing that directly.

-Jay

On Thu, Sep 22, 2016 at 12:31 PM, Michael Pearce 
wrote:

> Hi All,
>
>
> I would like to discuss the following KIP proposal:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 82+-+Add+Record+Headers
>
>
>
> I have some initial ?drafts of roughly the changes that would be needed.
> This is no where finalized and look forward to the discussion especially as
> some bits I'm personally in two minds about.
>
> https://github.com/michaelandrepearce/kafka/tree/kafka-headers-properties
>
>
>
> Here is a link to a alternative option mentioned in the kip but one i
> would personally would discard (disadvantages mentioned in kip)
>
> https://github.com/michaelandrepearce/kafka/tree/kafka-headers-full?
>
>