Build failed in Jenkins: kafka-trunk-jdk8 #2705

[Apache Jenkins Server]

See 


Changes:

[me] KAFKA-6997: Exclude test-sources.jar when $INCLUDE_TEST_JARS is FALSE

[ismael] MINOR: Fix for the location of the trogdor.sh executable file in the

--
[...truncated 954.55 KB...]
kafka.message.ByteBufferMessageSetTest > testWrittenEqualsRead PASSED

kafka.message.ByteBufferMessageSetTest > testWriteTo STARTED

kafka.message.ByteBufferMessageSetTest > testWriteTo PASSED

kafka.message.ByteBufferMessageSetTest > testEquals STARTED

kafka.message.ByteBufferMessageSetTest > testEquals PASSED

kafka.message.ByteBufferMessageSetTest > testSizeInBytes STARTED

kafka.message.ByteBufferMessageSetTest > testSizeInBytes PASSED

kafka.message.ByteBufferMessageSetTest > testIterator STARTED

kafka.message.ByteBufferMessageSetTest > testIterator PASSED

kafka.metrics.KafkaTimerTest > testKafkaTimer STARTED

kafka.metrics.KafkaTimerTest > testKafkaTimer PASSED

kafka.metrics.MetricsTest > testMetricsReporterAfterDeletingTopic STARTED

kafka.metrics.MetricsTest > testMetricsReporterAfterDeletingTopic PASSED

kafka.metrics.MetricsTest > testSessionExpireListenerMetrics STARTED

kafka.metrics.MetricsTest > testSessionExpireListenerMetrics PASSED

kafka.metrics.MetricsTest > 
testBrokerTopicMetricsUnregisteredAfterDeletingTopic STARTED

kafka.metrics.MetricsTest > 
testBrokerTopicMetricsUnregisteredAfterDeletingTopic PASSED

kafka.metrics.MetricsTest > testClusterIdMetric STARTED

kafka.metrics.MetricsTest > testClusterIdMetric PASSED

kafka.metrics.MetricsTest > testControllerMetrics STARTED

kafka.metrics.MetricsTest > testControllerMetrics PASSED

kafka.metrics.MetricsTest > testWindowsStyleTagNames STARTED

kafka.metrics.MetricsTest > testWindowsStyleTagNames PASSED

kafka.metrics.MetricsTest > testBrokerTopicMetricsBytesInOut STARTED

kafka.metrics.MetricsTest > testBrokerTopicMetricsBytesInOut PASSED

kafka.javaapi.consumer.ZookeeperConsumerConnectorTest > testBasic STARTED

kafka.javaapi.consumer.ZookeeperConsumerConnectorTest > testBasic PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testSizeInBytesWithCompression 
STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testSizeInBytesWithCompression 
PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > 
testIteratorIsConsistentWithCompression STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > 
testIteratorIsConsistentWithCompression PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testIteratorIsConsistent 
STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testIteratorIsConsistent PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testEqualsWithCompression 
STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testEqualsWithCompression 
PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testWrittenEqualsRead STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testWrittenEqualsRead PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testEquals STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testEquals PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testSizeInBytes STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testSizeInBytes PASSED

kafka.security.token.delegation.DelegationTokenManagerTest > 
testPeriodicTokenExpiry STARTED

kafka.security.token.delegation.DelegationTokenManagerTest > 
testPeriodicTokenExpiry PASSED

kafka.security.token.delegation.DelegationTokenManagerTest > 
testTokenRequestsWithDelegationTokenDisabled STARTED

kafka.security.token.delegation.DelegationTokenManagerTest > 
testTokenRequestsWithDelegationTokenDisabled PASSED

kafka.security.token.delegation.DelegationTokenManagerTest > testDescribeToken 
STARTED

kafka.security.token.delegation.DelegationTokenManagerTest > testDescribeToken 
PASSED

kafka.security.token.delegation.DelegationTokenManagerTest > testCreateToken 
STARTED

kafka.security.token.delegation.DelegationTokenManagerTest > testCreateToken 
PASSED

kafka.security.token.delegation.DelegationTokenManagerTest > testExpireToken 
STARTED

kafka.security.token.delegation.DelegationTokenManagerTest > testExpireToken 
PASSED

kafka.security.token.delegation.DelegationTokenManagerTest > testRenewToken 
STARTED

kafka.security.token.delegation.DelegationTokenManagerTest > testRenewToken 
PASSED

kafka.security.auth.ResourceTypeTest > testJavaConversions STARTED

kafka.security.auth.ResourceTypeTest > testJavaConversions PASSED

kafka.security.auth.ResourceTypeTest > testFromString STARTED

kafka.security.auth.ResourceTypeTest > testFromString PASSED

kafka.security.auth.OperationTest > testJavaConversions STARTED

kafka.security.auth.OperationTest > testJavaConversions PASSED

kafka.security.auth.PermissionTypeTest > testJavaConversions STARTED

kafka.security.auth.PermissionTypeTest > testJavaConversions PASSED

kafka.security.auth.PermissionTypeTest > testFromString STARTED

kafka.security.auth.Pe

[jira] [Created] (KAFKA-7004) Support configurable restore consumer poll timeout

[Shawn Nguyen (JIRA)]

Shawn Nguyen created KAFKA-7004:
---

 Summary: Support configurable restore consumer poll timeout
 Key: KAFKA-7004
 URL: https://issues.apache.org/jira/browse/KAFKA-7004
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Affects Versions: 1.1.0
Reporter: Shawn Nguyen


In the StateChangelogReader, the restore consumer is currently hard coded to 
poll for 10ms at most per call.
{noformat}
public Collection restore(final RestoringTasks active) { if 
(!needsInitializing.isEmpty()) { initialize(); } if (needsRestoring.isEmpty()) 
{ restoreConsumer.unsubscribe(); return completed(); } final 
Set restoringPartitions = new 
HashSet<>(needsRestoring.keySet()); try { final ConsumerRecords 
allRecords = restoreConsumer.poll(10); for (final TopicPartition partition : 
restoringPartitions) { restorePartition(allRecords, partition, 
active.restoringTaskFor(partition)); }
{noformat}
It'd be nice to be able to configure the restore consumer to poll for a larger 
timeout (e.g. 500ms) to give it more time to accumulate records for the 
restoration task. In the main event loop for polling in StreamThread.java, the 
main consumer uses the POLL_MS_CONFIG set in StreamsConfig.java to configure 
the max poll timeout. We could construct a similar config in the StreamsConfig 
class, but prefixed with the consumer type (restore in this case).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-7003) Add headers with error context in messages written to the Connect DeadLetterQueue topic

[Arjun Satish (JIRA)]

Arjun Satish created KAFKA-7003:
---

 Summary: Add headers with error context in messages written to the 
Connect DeadLetterQueue topic
 Key: KAFKA-7003
 URL: https://issues.apache.org/jira/browse/KAFKA-7003
 Project: Kafka
  Issue Type: Task
Reporter: Arjun Satish


This was added to the KIP after the feature freeze. 

If the property {{errors.deadletterqueue.}}{{context.headers.enable}} is set to 
{{*true*}}, the following headers will be added to the produced raw message 
(only if they don't already exist in the message). All values will be 
serialized as UTF-8 strings.
||Header Name||Description||
|__connect.errors.topic|Name of the topic that contained the message.|
|__connect.errors.task.id|The numeric ID of the task that encountered the error 
(encoded as a UTF-8 string).|
|__connect.errors.stage|The name of the stage where the error occurred.|
|__connect.errors.partition|The numeric ID of the partition in the original 
topic that contained the message (encoded as a UTF-8 string).|
|__connect.errors.offset|The numeric value of the message offset in the 
original topic (encoded as a UTF-8 string).|
|__connect.errors.exception.stacktrace|The stacktrace of the exception.|
|__connect.errors.exception.message|The message in the exception.|
|__connect.errors.exception.class.name|The fully qualified classname of the 
exception that was thrown during the execution.|
|__connect.errors.connector.name|The name of the connector which encountered 
the error.|
|__connect.errors.class.name|The fully qualified name of the class that caused 
the error.|



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-7002) Allow replication factor to be set via a configuration property for the Connect DLQ topic

[Arjun Satish (JIRA)]

Arjun Satish created KAFKA-7002:
---

 Summary: Allow replication factor to be set via a configuration 
property for the Connect DLQ topic
 Key: KAFKA-7002
 URL: https://issues.apache.org/jira/browse/KAFKA-7002
 Project: Kafka
  Issue Type: Task
  Components: KafkaConnect
Affects Versions: 2.0.0
Reporter: Arjun Satish
Assignee: Arjun Satish
 Fix For: 2.0.0


Currently, the replication factor is hardcoded to a value of 3. This means that 
we cannot use a DLQ in any cluster setup with less than three brokers. It is 
better to have the user specify this value if the default value does meet the 
requirements.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-7001) Rename `errors.allowed.max` in Connect to `errors.tolerance`

[Arjun Satish (JIRA)]

Arjun Satish created KAFKA-7001:
---

 Summary: Rename `errors.allowed.max` in Connect to 
`errors.tolerance`
 Key: KAFKA-7001
 URL: https://issues.apache.org/jira/browse/KAFKA-7001
 Project: Kafka
  Issue Type: Task
  Components: KafkaConnect
Affects Versions: 2.0.0
Reporter: Arjun Satish
Assignee: Arjun Satish
 Fix For: 2.0.0






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: kafka-trunk-jdk8 #2704

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: Rename package `internal` to `internals` for consistency (#5137)

[mjsax] [KAFKA-6730] Simplify State Store Recovery (#5013)

[me] KAFKA-6981: Move the error handling configuration properties into the

[wangguoz] MINOR: docs should point to latest version (#5132)

--
[...truncated 953.81 KB...]

kafka.controller.PartitionStateMachineTest > 
testOfflinePartitionToOnlinePartitionTransitionZkUtilsExceptionFromStateLookup 
PASSED

kafka.controller.PartitionStateMachineTest > 
testInvalidOnlinePartitionToNonexistentPartitionTransition STARTED

kafka.controller.PartitionStateMachineTest > 
testInvalidOnlinePartitionToNonexistentPartitionTransition PASSED

kafka.controller.PartitionStateMachineTest > 
testInvalidOfflinePartitionToNewPartitionTransition STARTED

kafka.controller.PartitionStateMachineTest > 
testInvalidOfflinePartitionToNewPartitionTransition PASSED

kafka.controller.PartitionStateMachineTest > 
testOfflinePartitionToOnlinePartitionTransition STARTED

kafka.controller.PartitionStateMachineTest > 
testOfflinePartitionToOnlinePartitionTransition PASSED

kafka.javaapi.consumer.ZookeeperConsumerConnectorTest > testBasic STARTED

kafka.javaapi.consumer.ZookeeperConsumerConnectorTest > testBasic PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testSizeInBytesWithCompression 
STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testSizeInBytesWithCompression 
PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > 
testIteratorIsConsistentWithCompression STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > 
testIteratorIsConsistentWithCompression PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testIteratorIsConsistent 
STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testIteratorIsConsistent PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testEqualsWithCompression 
STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testEqualsWithCompression 
PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testWrittenEqualsRead STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testWrittenEqualsRead PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testEquals STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testEquals PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testSizeInBytes STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testSizeInBytes PASSED

kafka.zookeeper.ZooKeeperClientTest > testZNodeChangeHandlerForDataChange 
STARTED

kafka.zookeeper.ZooKeeperClientTest > testZNodeChangeHandlerForDataChange PASSED

kafka.zookeeper.ZooKeeperClientTest > testZooKeeperSessionStateMetric STARTED

kafka.zookeeper.ZooKeeperClientTest > testZooKeeperSessionStateMetric PASSED

kafka.zookeeper.ZooKeeperClientTest > testExceptionInBeforeInitializingSession 
STARTED

kafka.zookeeper.ZooKeeperClientTest > testExceptionInBeforeInitializingSession 
PASSED

kafka.zookeeper.ZooKeeperClientTest > testGetChildrenExistingZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testGetChildrenExistingZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testConnection STARTED

kafka.zookeeper.ZooKeeperClientTest > testConnection PASSED

kafka.zookeeper.ZooKeeperClientTest > testZNodeChangeHandlerForCreation STARTED

kafka.zookeeper.ZooKeeperClientTest > testZNodeChangeHandlerForCreation PASSED

kafka.zookeeper.ZooKeeperClientTest > testGetAclExistingZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testGetAclExistingZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testSessionExpiryDuringClose STARTED

kafka.zookeeper.ZooKeeperClientTest > testSessionExpiryDuringClose PASSED

kafka.zookeeper.ZooKeeperClientTest > testSetAclNonExistentZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testSetAclNonExistentZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testConnectionLossRequestTermination 
STARTED

kafka.zookeeper.ZooKeeperClientTest > testConnectionLossRequestTermination 
PASSED

kafka.zookeeper.ZooKeeperClientTest > testExistsNonExistentZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testExistsNonExistentZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testGetDataNonExistentZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testGetDataNonExistentZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testConnectionTimeout STARTED

kafka.zookeeper.ZooKeeperClientTest > testConnectionTimeout PASSED

kafka.zookeeper.ZooKeeperClientTest > 
testBlockOnRequestCompletionFromStateChangeHandler STARTED

kafka.zookeeper.ZooKeeperClientTest > 
testBlockOnRequestCompletionFromStateChangeHandler PASSED

kafka.zookeeper.ZooKeeperClientTest > testUnresolvableConnectString STARTED

kafka.zookeeper.ZooKeeperClientTest > testUnresolvableConnectString PASSED

kafka.zookeeper.ZooKeeperClientTest > testGetChildrenNonExistentZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testGetChildrenNonExi

Re: [VOTE] KIP-266: Add TimeoutException for KafkaConsumer#position

[Ted Yu]

I see where the 0.5 in your previous response came about.

The reason I wrote 'request.timeout.ms + 15000' was that I treat this value
in place of the default.block.ms
According to your earlier description:

bq. request.timeout.ms controls something different: the amount of time
we're willing to wait for an RPC to complete.

Basically we're in agreement. It is just that figuring out good default is
non-trivial.

On Tue, Jun 5, 2018 at 4:44 PM, Colin McCabe  wrote:

> On Tue, Jun 5, 2018, at 16:35, Ted Yu wrote:
> > bq. could probably come up with a good default
> >
> > That's the difficult part.
> >
> > bq. max(1000, 0.5 * request.timeout.ms)
> >
> > Looking at some existing samples:
> > In tests/kafkatest/tests/connect/templates/connect-distributed.properties
> ,
> > we have:
> >   request.timeout.ms=3
> >
> > Isn't the above formula putting an upper bound 15000 for the RPC timeout
> ?
>
> Correct.  It would put a 15 second default on the RPC timeout in this
> case.  If that's too short, the user has the option to change it.
>
> If we feel that 15 seconds is too short, we could put a floor of 30
> seconds or whatever on the RPC timeout, instead of 1 second.
>
> >
> > By fixed duration, I meant something like
> >   request.timeout.ms + 15000
>
> The RPC timeout should be shorter than the request timeout, so that we get
> multiple tries if the RPC hangs due to network issues.  It should not be
> longer.
>
> best,
> Colin
>
> >
> > Cheers
> >
> > On Tue, Jun 5, 2018 at 4:27 PM, Colin McCabe  wrote:
> >
> > > I don't think it can be fixed.  The RPC duration is something that you
> > > might reasonably want to tune.  For example, if it's too low, you
> might not
> > > be able to make progress at all on a heavily loaded server.
> > >
> > > We could probably come up with a good default, however.
> rpc.timeout.ms
> > > could be set to something like
> > > max(1000, 0.5 * request.timeout.ms)
> > >
> > > best,
> > > Colin
> > >
> > >
> > > On Tue, Jun 5, 2018, at 16:21, Ted Yu wrote:
> > > > bq. we must make the API timeout longer than the RPC timeout
> > > >
> > > > I agree with the above.
> > > >
> > > > How about adding a fixed duration on top of request.timeout.ms ?
> > > >
> > > > Thanks
> > > >
> > > > On Tue, Jun 5, 2018 at 4:18 PM, Colin McCabe 
> wrote:
> > > >
> > > > > As Jason noted earlier, request.timeout.ms controls something
> > > different:
> > > > > the amount of time we're willing to wait for an RPC to complete.
> > > > >
> > > > > Empirically, RPCs sometimes hang for a long time.  If the API
> timeout
> > > is
> > > > > the same as the RPC timeout, we are not robust against this failure
> > > > > condition.  The whole call fails rather than trying another server
> or
> > > a new
> > > > > socket.
> > > > >
> > > > > In order to fix this, we must make the API timeout longer than the
> RPC
> > > > > timeout.
> > > > >
> > > > > However, we have a lot of users who have been trained to use
> > > > > request.timeout.ms to make their clients time out earlier.  If we
> > > > > introduce a new config to do this instead, it's kind of a breaking
> > > change
> > > > > for them.  Perhaps we should go the other direction and create a
> new
> > > > > configuration like rpc.timeout.ms to do what request.timeout.ms is
> > > doing
> > > > > now.  Then request.timeout.ms can become what users already think
> it
> > > is:
> > > > > the timeout for their API calls.
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > >
> > > > > On Tue, Jun 5, 2018, at 15:29, Ted Yu wrote:
> > > > > > bq. we were already doing with request.timeout.ms
> > > > > >
> > > > > > I would vote for using existing config.
> > > > > >
> > > > > > Any new config parameter needs to go thru long process of
> digestion:
> > > > > > documentation, etc in order for users to understand and
> familiarize.
> > > > > >
> > > > > > The existing config would have lower mismatch of impedance.
> > > > > >
> > > > > > Cheers
> > > > > >
> > > > > > On Tue, Jun 5, 2018 at 3:17 PM, Jason Gustafson <
> ja...@confluent.io>
> > > > > wrote:
> > > > > >
> > > > > > > Thanks for the comments. I'm not sure I understand why we want
> to
> > > > > avoid the
> > > > > > > term "timeout." Semantically, that's what it is. If we don't
> want
> > > > > another
> > > > > > > timeout config, we could avoid it and hard-code a reasonable
> > > default
> > > > > or try
> > > > > > > to wrap the behavior into one of the other timeouts (which is
> sort
> > > of
> > > > > what
> > > > > > > we were already doing with request.timeout.ms). But I'm not
> too
> > > sure
> > > > > > > calling it something else addresses the problem.
> > > > > > >
> > > > > > > -Jason
> > > > > > >
> > > > > > > On Tue, Jun 5, 2018 at 2:59 PM, Dhruvil Shah <
> dhru...@confluent.io
> > > >
> > > > > wrote:
> > > > > > >
> > > > > > > > I agree that using `default.timeout.ms` could cause
> confusion
> > > since
> > > > > we
> > > > > > > > already have other timeout configurations in the con

Re: [VOTE] KIP-266: Add TimeoutException for KafkaConsumer#position

[Colin McCabe]

On Tue, Jun 5, 2018, at 16:35, Ted Yu wrote:
> bq. could probably come up with a good default
> 
> That's the difficult part.
> 
> bq. max(1000, 0.5 * request.timeout.ms)
> 
> Looking at some existing samples:
> In tests/kafkatest/tests/connect/templates/connect-distributed.properties ,
> we have:
>   request.timeout.ms=3
> 
> Isn't the above formula putting an upper bound 15000 for the RPC timeout ?

Correct.  It would put a 15 second default on the RPC timeout in this case.  If 
that's too short, the user has the option to change it.

If we feel that 15 seconds is too short, we could put a floor of 30 seconds or 
whatever on the RPC timeout, instead of 1 second.

> 
> By fixed duration, I meant something like
>   request.timeout.ms + 15000

The RPC timeout should be shorter than the request timeout, so that we get 
multiple tries if the RPC hangs due to network issues.  It should not be longer.

best,
Colin

> 
> Cheers
> 
> On Tue, Jun 5, 2018 at 4:27 PM, Colin McCabe  wrote:
> 
> > I don't think it can be fixed.  The RPC duration is something that you
> > might reasonably want to tune.  For example, if it's too low, you might not
> > be able to make progress at all on a heavily loaded server.
> >
> > We could probably come up with a good default, however.  rpc.timeout.ms
> > could be set to something like
> > max(1000, 0.5 * request.timeout.ms)
> >
> > best,
> > Colin
> >
> >
> > On Tue, Jun 5, 2018, at 16:21, Ted Yu wrote:
> > > bq. we must make the API timeout longer than the RPC timeout
> > >
> > > I agree with the above.
> > >
> > > How about adding a fixed duration on top of request.timeout.ms ?
> > >
> > > Thanks
> > >
> > > On Tue, Jun 5, 2018 at 4:18 PM, Colin McCabe  wrote:
> > >
> > > > As Jason noted earlier, request.timeout.ms controls something
> > different:
> > > > the amount of time we're willing to wait for an RPC to complete.
> > > >
> > > > Empirically, RPCs sometimes hang for a long time.  If the API timeout
> > is
> > > > the same as the RPC timeout, we are not robust against this failure
> > > > condition.  The whole call fails rather than trying another server or
> > a new
> > > > socket.
> > > >
> > > > In order to fix this, we must make the API timeout longer than the RPC
> > > > timeout.
> > > >
> > > > However, we have a lot of users who have been trained to use
> > > > request.timeout.ms to make their clients time out earlier.  If we
> > > > introduce a new config to do this instead, it's kind of a breaking
> > change
> > > > for them.  Perhaps we should go the other direction and create a new
> > > > configuration like rpc.timeout.ms to do what request.timeout.ms is
> > doing
> > > > now.  Then request.timeout.ms can become what users already think it
> > is:
> > > > the timeout for their API calls.
> > > >
> > > > best,
> > > > Colin
> > > >
> > > >
> > > > On Tue, Jun 5, 2018, at 15:29, Ted Yu wrote:
> > > > > bq. we were already doing with request.timeout.ms
> > > > >
> > > > > I would vote for using existing config.
> > > > >
> > > > > Any new config parameter needs to go thru long process of digestion:
> > > > > documentation, etc in order for users to understand and familiarize.
> > > > >
> > > > > The existing config would have lower mismatch of impedance.
> > > > >
> > > > > Cheers
> > > > >
> > > > > On Tue, Jun 5, 2018 at 3:17 PM, Jason Gustafson 
> > > > wrote:
> > > > >
> > > > > > Thanks for the comments. I'm not sure I understand why we want to
> > > > avoid the
> > > > > > term "timeout." Semantically, that's what it is. If we don't want
> > > > another
> > > > > > timeout config, we could avoid it and hard-code a reasonable
> > default
> > > > or try
> > > > > > to wrap the behavior into one of the other timeouts (which is sort
> > of
> > > > what
> > > > > > we were already doing with request.timeout.ms). But I'm not too
> > sure
> > > > > > calling it something else addresses the problem.
> > > > > >
> > > > > > -Jason
> > > > > >
> > > > > > On Tue, Jun 5, 2018 at 2:59 PM, Dhruvil Shah  > >
> > > > wrote:
> > > > > >
> > > > > > > I agree that using `default.timeout.ms` could cause confusion
> > since
> > > > we
> > > > > > > already have other timeout configurations in the consumer.
> > > > > > >
> > > > > > > +1 for using `default.block.ms`.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Dhruvil
> > > > > > >
> > > > > > > On Tue, Jun 5, 2018 at 11:48 AM, Bill Bejeck 
> > > > wrote:
> > > > > > >
> > > > > > > > Hi Jason,
> > > > > > > >
> > > > > > > > At first, I thought the same name between the producer and the
> > > > consumer
> > > > > > > was
> > > > > > > > ideal.
> > > > > > > >
> > > > > > > > But your comment makes me realize consistent names with
> > different
> > > > > > > semantics
> > > > > > > > is even more confusing.
> > > > > > > >
> > > > > > > > I'm +1 for not using `max.block.ms`.  I like Guozhang's
> > > > suggestion of
> > > > > > `
> > > > > > > > default.block.ms` for the name.
> > > > > > > >
> > > > > > > > Thanks,
> >

Re: [VOTE] KIP-266: Add TimeoutException for KafkaConsumer#position

[Ted Yu]

bq. could probably come up with a good default

That's the difficult part.

bq. max(1000, 0.5 * request.timeout.ms)

Looking at some existing samples:
In tests/kafkatest/tests/connect/templates/connect-distributed.properties ,
we have:
  request.timeout.ms=3

Isn't the above formula putting an upper bound 15000 for the RPC timeout ?

By fixed duration, I meant something like
  request.timeout.ms + 15000

Cheers

On Tue, Jun 5, 2018 at 4:27 PM, Colin McCabe  wrote:

> I don't think it can be fixed.  The RPC duration is something that you
> might reasonably want to tune.  For example, if it's too low, you might not
> be able to make progress at all on a heavily loaded server.
>
> We could probably come up with a good default, however.  rpc.timeout.ms
> could be set to something like
> max(1000, 0.5 * request.timeout.ms)
>
> best,
> Colin
>
>
> On Tue, Jun 5, 2018, at 16:21, Ted Yu wrote:
> > bq. we must make the API timeout longer than the RPC timeout
> >
> > I agree with the above.
> >
> > How about adding a fixed duration on top of request.timeout.ms ?
> >
> > Thanks
> >
> > On Tue, Jun 5, 2018 at 4:18 PM, Colin McCabe  wrote:
> >
> > > As Jason noted earlier, request.timeout.ms controls something
> different:
> > > the amount of time we're willing to wait for an RPC to complete.
> > >
> > > Empirically, RPCs sometimes hang for a long time.  If the API timeout
> is
> > > the same as the RPC timeout, we are not robust against this failure
> > > condition.  The whole call fails rather than trying another server or
> a new
> > > socket.
> > >
> > > In order to fix this, we must make the API timeout longer than the RPC
> > > timeout.
> > >
> > > However, we have a lot of users who have been trained to use
> > > request.timeout.ms to make their clients time out earlier.  If we
> > > introduce a new config to do this instead, it's kind of a breaking
> change
> > > for them.  Perhaps we should go the other direction and create a new
> > > configuration like rpc.timeout.ms to do what request.timeout.ms is
> doing
> > > now.  Then request.timeout.ms can become what users already think it
> is:
> > > the timeout for their API calls.
> > >
> > > best,
> > > Colin
> > >
> > >
> > > On Tue, Jun 5, 2018, at 15:29, Ted Yu wrote:
> > > > bq. we were already doing with request.timeout.ms
> > > >
> > > > I would vote for using existing config.
> > > >
> > > > Any new config parameter needs to go thru long process of digestion:
> > > > documentation, etc in order for users to understand and familiarize.
> > > >
> > > > The existing config would have lower mismatch of impedance.
> > > >
> > > > Cheers
> > > >
> > > > On Tue, Jun 5, 2018 at 3:17 PM, Jason Gustafson 
> > > wrote:
> > > >
> > > > > Thanks for the comments. I'm not sure I understand why we want to
> > > avoid the
> > > > > term "timeout." Semantically, that's what it is. If we don't want
> > > another
> > > > > timeout config, we could avoid it and hard-code a reasonable
> default
> > > or try
> > > > > to wrap the behavior into one of the other timeouts (which is sort
> of
> > > what
> > > > > we were already doing with request.timeout.ms). But I'm not too
> sure
> > > > > calling it something else addresses the problem.
> > > > >
> > > > > -Jason
> > > > >
> > > > > On Tue, Jun 5, 2018 at 2:59 PM, Dhruvil Shah  >
> > > wrote:
> > > > >
> > > > > > I agree that using `default.timeout.ms` could cause confusion
> since
> > > we
> > > > > > already have other timeout configurations in the consumer.
> > > > > >
> > > > > > +1 for using `default.block.ms`.
> > > > > >
> > > > > > Thanks,
> > > > > > Dhruvil
> > > > > >
> > > > > > On Tue, Jun 5, 2018 at 11:48 AM, Bill Bejeck 
> > > wrote:
> > > > > >
> > > > > > > Hi Jason,
> > > > > > >
> > > > > > > At first, I thought the same name between the producer and the
> > > consumer
> > > > > > was
> > > > > > > ideal.
> > > > > > >
> > > > > > > But your comment makes me realize consistent names with
> different
> > > > > > semantics
> > > > > > > is even more confusing.
> > > > > > >
> > > > > > > I'm +1 for not using `max.block.ms`.  I like Guozhang's
> > > suggestion of
> > > > > `
> > > > > > > default.block.ms` for the name.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Bill
> > > > > > >
> > > > > > > On Tue, Jun 5, 2018 at 1:33 PM, Guozhang Wang <
> wangg...@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Jason,
> > > > > > > >
> > > > > > > > Yeah I agree that "max.block.ms" makes people thinking of
> the
> > > > > > producer's
> > > > > > > > config with the same name, but their semantics are different.
> > > > > > > >
> > > > > > > > On the other hand, I'm a bit concerned with the reusing of
> the
> > > term
> > > > > > > > `timeout` as we already have `session.timeout.ms` and `
> > > > > > > request.timeout.ms`
> > > > > > > > in ConsumerConfig.. How about using the name `
> > > default.api.block.ms`
> > > > > or
> > > > > > > > simply `default.block.ms`?
> > > > > > > >
> > > > > > > >

Re: Someone to review KAFKA-6919, one line change for faulty documentation

[Colin McCabe]

Thanks, Koen.  I was on vacation, so I missed this originally.  But I'll review 
it now.

cheers,
Colin


On Fri, Jun 1, 2018, at 13:02, Koen De Groote wrote:
> Greetings,
> 
> Poking for someone to have a quick look at this, It's a one-line change. I
> noticed the documentation about trogdor was pointing to a non-existing
> folder.
> 
> Ticket: https://issues.apache.org/jira/browse/KAFKA-6919
> 
> PR: https://github.com/apache/kafka/pull/5040
> 
> Thanks.

[jira] [Resolved] (KAFKA-6997) Kafka run class doesn't exclude test-sources jar

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava resolved KAFKA-6997.
--
Resolution: Fixed

Issue resolved by pull request 5139
[https://github.com/apache/kafka/pull/5139]

> Kafka run class doesn't exclude test-sources jar
> 
>
> Key: KAFKA-6997
> URL: https://issues.apache.org/jira/browse/KAFKA-6997
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Affects Versions: 2.0.0
>Reporter: Magesh kumar Nandakumar
>Assignee: Magesh kumar Nandakumar
>Priority: Minor
> Fix For: 2.0.0
>
>
> kafka-run-class.sh has a flag INCLUDE_TEST_JAR. This doesn't exclude 
> test-sources jar files when the flag is set to false. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] KIP-266: Add TimeoutException for KafkaConsumer#position

[Colin McCabe]

I don't think it can be fixed.  The RPC duration is something that you might 
reasonably want to tune.  For example, if it's too low, you might not be able 
to make progress at all on a heavily loaded server.

We could probably come up with a good default, however.  rpc.timeout.ms could 
be set to something like
max(1000, 0.5 * request.timeout.ms)

best,
Colin


On Tue, Jun 5, 2018, at 16:21, Ted Yu wrote:
> bq. we must make the API timeout longer than the RPC timeout
> 
> I agree with the above.
> 
> How about adding a fixed duration on top of request.timeout.ms ?
> 
> Thanks
> 
> On Tue, Jun 5, 2018 at 4:18 PM, Colin McCabe  wrote:
> 
> > As Jason noted earlier, request.timeout.ms controls something different:
> > the amount of time we're willing to wait for an RPC to complete.
> >
> > Empirically, RPCs sometimes hang for a long time.  If the API timeout is
> > the same as the RPC timeout, we are not robust against this failure
> > condition.  The whole call fails rather than trying another server or a new
> > socket.
> >
> > In order to fix this, we must make the API timeout longer than the RPC
> > timeout.
> >
> > However, we have a lot of users who have been trained to use
> > request.timeout.ms to make their clients time out earlier.  If we
> > introduce a new config to do this instead, it's kind of a breaking change
> > for them.  Perhaps we should go the other direction and create a new
> > configuration like rpc.timeout.ms to do what request.timeout.ms is doing
> > now.  Then request.timeout.ms can become what users already think it is:
> > the timeout for their API calls.
> >
> > best,
> > Colin
> >
> >
> > On Tue, Jun 5, 2018, at 15:29, Ted Yu wrote:
> > > bq. we were already doing with request.timeout.ms
> > >
> > > I would vote for using existing config.
> > >
> > > Any new config parameter needs to go thru long process of digestion:
> > > documentation, etc in order for users to understand and familiarize.
> > >
> > > The existing config would have lower mismatch of impedance.
> > >
> > > Cheers
> > >
> > > On Tue, Jun 5, 2018 at 3:17 PM, Jason Gustafson 
> > wrote:
> > >
> > > > Thanks for the comments. I'm not sure I understand why we want to
> > avoid the
> > > > term "timeout." Semantically, that's what it is. If we don't want
> > another
> > > > timeout config, we could avoid it and hard-code a reasonable default
> > or try
> > > > to wrap the behavior into one of the other timeouts (which is sort of
> > what
> > > > we were already doing with request.timeout.ms). But I'm not too sure
> > > > calling it something else addresses the problem.
> > > >
> > > > -Jason
> > > >
> > > > On Tue, Jun 5, 2018 at 2:59 PM, Dhruvil Shah 
> > wrote:
> > > >
> > > > > I agree that using `default.timeout.ms` could cause confusion since
> > we
> > > > > already have other timeout configurations in the consumer.
> > > > >
> > > > > +1 for using `default.block.ms`.
> > > > >
> > > > > Thanks,
> > > > > Dhruvil
> > > > >
> > > > > On Tue, Jun 5, 2018 at 11:48 AM, Bill Bejeck 
> > wrote:
> > > > >
> > > > > > Hi Jason,
> > > > > >
> > > > > > At first, I thought the same name between the producer and the
> > consumer
> > > > > was
> > > > > > ideal.
> > > > > >
> > > > > > But your comment makes me realize consistent names with different
> > > > > semantics
> > > > > > is even more confusing.
> > > > > >
> > > > > > I'm +1 for not using `max.block.ms`.  I like Guozhang's
> > suggestion of
> > > > `
> > > > > > default.block.ms` for the name.
> > > > > >
> > > > > > Thanks,
> > > > > > Bill
> > > > > >
> > > > > > On Tue, Jun 5, 2018 at 1:33 PM, Guozhang Wang 
> > > > > wrote:
> > > > > >
> > > > > > > Hi Jason,
> > > > > > >
> > > > > > > Yeah I agree that "max.block.ms" makes people thinking of the
> > > > > producer's
> > > > > > > config with the same name, but their semantics are different.
> > > > > > >
> > > > > > > On the other hand, I'm a bit concerned with the reusing of the
> > term
> > > > > > > `timeout` as we already have `session.timeout.ms` and `
> > > > > > request.timeout.ms`
> > > > > > > in ConsumerConfig.. How about using the name `
> > default.api.block.ms`
> > > > or
> > > > > > > simply `default.block.ms`?
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Guozhang
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Jun 5, 2018 at 8:57 AM, Jason Gustafson <
> > ja...@confluent.io>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hey All,
> > > > > > > >
> > > > > > > > One more minor follow-up. As I was reviewing the change
> > mentioned
> > > > > > above,
> > > > > > > I
> > > > > > > > felt the name `max.block.ms` was a little bit misleading
> > since it
> > > > > only
> > > > > > > > applies to methods which do not have an explicit timeout. A
> > clearer
> > > > > > name
> > > > > > > > given its usage might be `default.timeout.ms`. It is the
> > default
> > > > > > timeout
> > > > > > > > for any blocking API which does not have a timeout. I'm leaning
> > > > > toward
> > > > >

Re: [VOTE] KIP-266: Add TimeoutException for KafkaConsumer#position

[Ted Yu]

bq. we must make the API timeout longer than the RPC timeout

I agree with the above.

How about adding a fixed duration on top of request.timeout.ms ?

Thanks

On Tue, Jun 5, 2018 at 4:18 PM, Colin McCabe  wrote:

> As Jason noted earlier, request.timeout.ms controls something different:
> the amount of time we're willing to wait for an RPC to complete.
>
> Empirically, RPCs sometimes hang for a long time.  If the API timeout is
> the same as the RPC timeout, we are not robust against this failure
> condition.  The whole call fails rather than trying another server or a new
> socket.
>
> In order to fix this, we must make the API timeout longer than the RPC
> timeout.
>
> However, we have a lot of users who have been trained to use
> request.timeout.ms to make their clients time out earlier.  If we
> introduce a new config to do this instead, it's kind of a breaking change
> for them.  Perhaps we should go the other direction and create a new
> configuration like rpc.timeout.ms to do what request.timeout.ms is doing
> now.  Then request.timeout.ms can become what users already think it is:
> the timeout for their API calls.
>
> best,
> Colin
>
>
> On Tue, Jun 5, 2018, at 15:29, Ted Yu wrote:
> > bq. we were already doing with request.timeout.ms
> >
> > I would vote for using existing config.
> >
> > Any new config parameter needs to go thru long process of digestion:
> > documentation, etc in order for users to understand and familiarize.
> >
> > The existing config would have lower mismatch of impedance.
> >
> > Cheers
> >
> > On Tue, Jun 5, 2018 at 3:17 PM, Jason Gustafson 
> wrote:
> >
> > > Thanks for the comments. I'm not sure I understand why we want to
> avoid the
> > > term "timeout." Semantically, that's what it is. If we don't want
> another
> > > timeout config, we could avoid it and hard-code a reasonable default
> or try
> > > to wrap the behavior into one of the other timeouts (which is sort of
> what
> > > we were already doing with request.timeout.ms). But I'm not too sure
> > > calling it something else addresses the problem.
> > >
> > > -Jason
> > >
> > > On Tue, Jun 5, 2018 at 2:59 PM, Dhruvil Shah 
> wrote:
> > >
> > > > I agree that using `default.timeout.ms` could cause confusion since
> we
> > > > already have other timeout configurations in the consumer.
> > > >
> > > > +1 for using `default.block.ms`.
> > > >
> > > > Thanks,
> > > > Dhruvil
> > > >
> > > > On Tue, Jun 5, 2018 at 11:48 AM, Bill Bejeck 
> wrote:
> > > >
> > > > > Hi Jason,
> > > > >
> > > > > At first, I thought the same name between the producer and the
> consumer
> > > > was
> > > > > ideal.
> > > > >
> > > > > But your comment makes me realize consistent names with different
> > > > semantics
> > > > > is even more confusing.
> > > > >
> > > > > I'm +1 for not using `max.block.ms`.  I like Guozhang's
> suggestion of
> > > `
> > > > > default.block.ms` for the name.
> > > > >
> > > > > Thanks,
> > > > > Bill
> > > > >
> > > > > On Tue, Jun 5, 2018 at 1:33 PM, Guozhang Wang 
> > > > wrote:
> > > > >
> > > > > > Hi Jason,
> > > > > >
> > > > > > Yeah I agree that "max.block.ms" makes people thinking of the
> > > > producer's
> > > > > > config with the same name, but their semantics are different.
> > > > > >
> > > > > > On the other hand, I'm a bit concerned with the reusing of the
> term
> > > > > > `timeout` as we already have `session.timeout.ms` and `
> > > > > request.timeout.ms`
> > > > > > in ConsumerConfig.. How about using the name `
> default.api.block.ms`
> > > or
> > > > > > simply `default.block.ms`?
> > > > > >
> > > > > >
> > > > > >
> > > > > > Guozhang
> > > > > >
> > > > > >
> > > > > > On Tue, Jun 5, 2018 at 8:57 AM, Jason Gustafson <
> ja...@confluent.io>
> > > > > > wrote:
> > > > > >
> > > > > > > Hey All,
> > > > > > >
> > > > > > > One more minor follow-up. As I was reviewing the change
> mentioned
> > > > > above,
> > > > > > I
> > > > > > > felt the name `max.block.ms` was a little bit misleading
> since it
> > > > only
> > > > > > > applies to methods which do not have an explicit timeout. A
> clearer
> > > > > name
> > > > > > > given its usage might be `default.timeout.ms`. It is the
> default
> > > > > timeout
> > > > > > > for any blocking API which does not have a timeout. I'm leaning
> > > > toward
> > > > > > > using this name since the current one seems likely to cause
> > > > confusion.
> > > > > > Any
> > > > > > > thoughts?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Jason
> > > > > > >
> > > > > > >
> > > > > > > On Thu, May 31, 2018 at 6:09 PM, Dong Lin  >
> > > > wrote:
> > > > > > >
> > > > > > > > Thanks for the KIP! I am in favor of the option 1.
> > > > > > > >
> > > > > > > > +1 as well.
> > > > > > > >
> > > > > > > > On Thu, May 31, 2018 at 6:00 PM, Jason Gustafson <
> > > > ja...@confluent.io
> > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Thanks everyone for the feedback. I've updated the KIP and
> > > added
> > > > > > > > > KAFKA-6979.
>

Re: [VOTE] KIP-266: Add TimeoutException for KafkaConsumer#position

[Colin McCabe]

As Jason noted earlier, request.timeout.ms controls something different: the 
amount of time we're willing to wait for an RPC to complete.

Empirically, RPCs sometimes hang for a long time.  If the API timeout is the 
same as the RPC timeout, we are not robust against this failure condition.  The 
whole call fails rather than trying another server or a new socket.

In order to fix this, we must make the API timeout longer than the RPC timeout.

However, we have a lot of users who have been trained to use request.timeout.ms 
to make their clients time out earlier.  If we introduce a new config to do 
this instead, it's kind of a breaking change for them.  Perhaps we should go 
the other direction and create a new configuration like rpc.timeout.ms to do 
what request.timeout.ms is doing now.  Then request.timeout.ms can become what 
users already think it is: the timeout for their API calls.

best,
Colin


On Tue, Jun 5, 2018, at 15:29, Ted Yu wrote:
> bq. we were already doing with request.timeout.ms
> 
> I would vote for using existing config.
> 
> Any new config parameter needs to go thru long process of digestion:
> documentation, etc in order for users to understand and familiarize.
> 
> The existing config would have lower mismatch of impedance.
> 
> Cheers
> 
> On Tue, Jun 5, 2018 at 3:17 PM, Jason Gustafson  wrote:
> 
> > Thanks for the comments. I'm not sure I understand why we want to avoid the
> > term "timeout." Semantically, that's what it is. If we don't want another
> > timeout config, we could avoid it and hard-code a reasonable default or try
> > to wrap the behavior into one of the other timeouts (which is sort of what
> > we were already doing with request.timeout.ms). But I'm not too sure
> > calling it something else addresses the problem.
> >
> > -Jason
> >
> > On Tue, Jun 5, 2018 at 2:59 PM, Dhruvil Shah  wrote:
> >
> > > I agree that using `default.timeout.ms` could cause confusion since we
> > > already have other timeout configurations in the consumer.
> > >
> > > +1 for using `default.block.ms`.
> > >
> > > Thanks,
> > > Dhruvil
> > >
> > > On Tue, Jun 5, 2018 at 11:48 AM, Bill Bejeck  wrote:
> > >
> > > > Hi Jason,
> > > >
> > > > At first, I thought the same name between the producer and the consumer
> > > was
> > > > ideal.
> > > >
> > > > But your comment makes me realize consistent names with different
> > > semantics
> > > > is even more confusing.
> > > >
> > > > I'm +1 for not using `max.block.ms`.  I like Guozhang's suggestion of
> > `
> > > > default.block.ms` for the name.
> > > >
> > > > Thanks,
> > > > Bill
> > > >
> > > > On Tue, Jun 5, 2018 at 1:33 PM, Guozhang Wang 
> > > wrote:
> > > >
> > > > > Hi Jason,
> > > > >
> > > > > Yeah I agree that "max.block.ms" makes people thinking of the
> > > producer's
> > > > > config with the same name, but their semantics are different.
> > > > >
> > > > > On the other hand, I'm a bit concerned with the reusing of the term
> > > > > `timeout` as we already have `session.timeout.ms` and `
> > > > request.timeout.ms`
> > > > > in ConsumerConfig.. How about using the name `default.api.block.ms`
> > or
> > > > > simply `default.block.ms`?
> > > > >
> > > > >
> > > > >
> > > > > Guozhang
> > > > >
> > > > >
> > > > > On Tue, Jun 5, 2018 at 8:57 AM, Jason Gustafson 
> > > > > wrote:
> > > > >
> > > > > > Hey All,
> > > > > >
> > > > > > One more minor follow-up. As I was reviewing the change mentioned
> > > > above,
> > > > > I
> > > > > > felt the name `max.block.ms` was a little bit misleading since it
> > > only
> > > > > > applies to methods which do not have an explicit timeout. A clearer
> > > > name
> > > > > > given its usage might be `default.timeout.ms`. It is the default
> > > > timeout
> > > > > > for any blocking API which does not have a timeout. I'm leaning
> > > toward
> > > > > > using this name since the current one seems likely to cause
> > > confusion.
> > > > > Any
> > > > > > thoughts?
> > > > > >
> > > > > > Thanks,
> > > > > > Jason
> > > > > >
> > > > > >
> > > > > > On Thu, May 31, 2018 at 6:09 PM, Dong Lin 
> > > wrote:
> > > > > >
> > > > > > > Thanks for the KIP! I am in favor of the option 1.
> > > > > > >
> > > > > > > +1 as well.
> > > > > > >
> > > > > > > On Thu, May 31, 2018 at 6:00 PM, Jason Gustafson <
> > > ja...@confluent.io
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Thanks everyone for the feedback. I've updated the KIP and
> > added
> > > > > > > > KAFKA-6979.
> > > > > > > >
> > > > > > > > -Jason
> > > > > > > >
> > > > > > > > On Wed, May 30, 2018 at 3:50 PM, Guozhang Wang <
> > > wangg...@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Thanks Jason. I'm in favor of option 1 as well.
> > > > > > > > >
> > > > > > > > > On Wed, May 30, 2018 at 1:37 PM, Bill Bejeck <
> > > bbej...@gmail.com>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > For what it's worth I'm +1 on Option 1 and the default
> > value
> > > > for
> > > > > > the
> > > > 

[jira] [Created] (KAFKA-7000) KafkaConsumer.position should wait for assignment metadata

[John Roesler (JIRA)]

John Roesler created KAFKA-7000:
---

 Summary: KafkaConsumer.position should wait for assignment metadata
 Key: KAFKA-7000
 URL: https://issues.apache.org/jira/browse/KAFKA-7000
 Project: Kafka
  Issue Type: Improvement
Reporter: John Roesler
Assignee: John Roesler


While updating Kafka Streams to stop using the deprecated Consumer.poll(long), 
I found that this code unexpectedly throws an exception:
{code:java}
consumer.subscribe(topics);
// consumer.poll(0); <- I've removed this line, which shouldn't be necessary 
here.

final Set partitions = new HashSet<>();
for (final String topic : topics) {
for (final PartitionInfo partition : consumer.partitionsFor(topic)) {
partitions.add(new TopicPartition(partition.topic(), 
partition.partition()));
}
}

for (final TopicPartition tp : partitions) {
final long offset = consumer.position(tp);
committedOffsets.put(tp, offset);
}{code}
Here is the exception:
{code:java}
Exception in thread "main" java.lang.IllegalStateException: You can only check 
the position for partitions assigned to this consumer.
   at 
org.apache.kafka.clients.consumer.KafkaConsumer.position(KafkaConsumer.java:1620)
   at 
org.apache.kafka.clients.consumer.KafkaConsumer.position(KafkaConsumer.java:1586)
   at 
org.apache.kafka.streams.tests.EosTestDriver.getCommittedOffsets(EosTestDriver.java:275)
   at 
org.apache.kafka.streams.tests.EosTestDriver.verify(EosTestDriver.java:148)
   at 
org.apache.kafka.streams.tests.StreamsEosTest.main(StreamsEosTest.java:69){code}
 

As you can see in the commented code in my snippet, we used to block for 
assignment with a poll(0), which is now deprecated.

It seems reasonable to me for position() to do the same thing that poll() does, 
which is call `coordinator.poll(timeout.toMillis())` early in processing to 
ensure an up-to-date assignment.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-6999) Document read-write lock usage of caching enabled stores

[Matthias J. Sax (JIRA)]

Matthias J. Sax created KAFKA-6999:
--

 Summary: Document read-write lock usage of caching enabled stores
 Key: KAFKA-6999
 URL: https://issues.apache.org/jira/browse/KAFKA-6999
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: Matthias J. Sax


>From the mailing list
{quote}Hello again fellow Kafkans,
 
Yesterday we observed a production deadlock take down one of our instances. 
Upon digging, it's clear that our usage of Kafka is the proximate cause, but 
the danger of our approach is not clear at all just from the Javadocs.
 
We have stream processors that read off an incoming KStream, possibly 
cross-reference some data from an auxiliary table via 
ReadOnlyKeyValueStore.get()
 
This is done via custom logic rather than a direct KTable join because which 
index is consulted may change depending on the shape of incoming data.
 
The ROKVS docs state,
 * A key value store that only supports read operations.
 * Implementations should be thread-safe as concurrent reads and writes
 * are expected.
 
They do **not** indicate that the CachingKVS layer uses a ReadWriteLock. If you 
have one CachingKVS flush a record cause a read from another CKVS, you are 
suddenly vulnerable to classic lock order reversals! Surprise!
 
A partial stack trace highlighting the problem, with many uninteresting frames 
removed, is inline at the bottom of this mail.
 
You could probably rightly point to us allowing a "observer" pattern to 
callback from within streams processing code is dangerous. We might move this 
off to an auxiliary thread to alleviate this problem.
 
But it still remains -- when you go an read that ROKVS documentation, it sure 
doesn't prepare you to this possibility!
 {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-6998) Remove caching wrapper stores if cache-size is configured to zero bytes

[Matthias J. Sax (JIRA)]

Matthias J. Sax created KAFKA-6998:
--

 Summary: Remove caching wrapper stores if cache-size is configured 
to zero bytes
 Key: KAFKA-6998
 URL: https://issues.apache.org/jira/browse/KAFKA-6998
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: Matthias J. Sax


Users can disable caching globally by setting the cache size to zero in their 
config. However, this does only effectively disable the caching layer, but the 
code is still in place.

We should consider to remove the caching wrappers completely for this case. The 
tricky part is, that we insert the caching layer at compile time, ie, when 
calling `StreamsBuilder#build()` – at this point, we don't know the 
configuration yet. Thus, we need to find a way to rewrite the topology after it 
is passed to `KafkaStreams` if case caching size is set to zero.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] KIP-266: Add TimeoutException for KafkaConsumer#position

[Ted Yu]

bq. we were already doing with request.timeout.ms

I would vote for using existing config.

Any new config parameter needs to go thru long process of digestion:
documentation, etc in order for users to understand and familiarize.

The existing config would have lower mismatch of impedance.

Cheers

On Tue, Jun 5, 2018 at 3:17 PM, Jason Gustafson  wrote:

> Thanks for the comments. I'm not sure I understand why we want to avoid the
> term "timeout." Semantically, that's what it is. If we don't want another
> timeout config, we could avoid it and hard-code a reasonable default or try
> to wrap the behavior into one of the other timeouts (which is sort of what
> we were already doing with request.timeout.ms). But I'm not too sure
> calling it something else addresses the problem.
>
> -Jason
>
> On Tue, Jun 5, 2018 at 2:59 PM, Dhruvil Shah  wrote:
>
> > I agree that using `default.timeout.ms` could cause confusion since we
> > already have other timeout configurations in the consumer.
> >
> > +1 for using `default.block.ms`.
> >
> > Thanks,
> > Dhruvil
> >
> > On Tue, Jun 5, 2018 at 11:48 AM, Bill Bejeck  wrote:
> >
> > > Hi Jason,
> > >
> > > At first, I thought the same name between the producer and the consumer
> > was
> > > ideal.
> > >
> > > But your comment makes me realize consistent names with different
> > semantics
> > > is even more confusing.
> > >
> > > I'm +1 for not using `max.block.ms`.  I like Guozhang's suggestion of
> `
> > > default.block.ms` for the name.
> > >
> > > Thanks,
> > > Bill
> > >
> > > On Tue, Jun 5, 2018 at 1:33 PM, Guozhang Wang 
> > wrote:
> > >
> > > > Hi Jason,
> > > >
> > > > Yeah I agree that "max.block.ms" makes people thinking of the
> > producer's
> > > > config with the same name, but their semantics are different.
> > > >
> > > > On the other hand, I'm a bit concerned with the reusing of the term
> > > > `timeout` as we already have `session.timeout.ms` and `
> > > request.timeout.ms`
> > > > in ConsumerConfig.. How about using the name `default.api.block.ms`
> or
> > > > simply `default.block.ms`?
> > > >
> > > >
> > > >
> > > > Guozhang
> > > >
> > > >
> > > > On Tue, Jun 5, 2018 at 8:57 AM, Jason Gustafson 
> > > > wrote:
> > > >
> > > > > Hey All,
> > > > >
> > > > > One more minor follow-up. As I was reviewing the change mentioned
> > > above,
> > > > I
> > > > > felt the name `max.block.ms` was a little bit misleading since it
> > only
> > > > > applies to methods which do not have an explicit timeout. A clearer
> > > name
> > > > > given its usage might be `default.timeout.ms`. It is the default
> > > timeout
> > > > > for any blocking API which does not have a timeout. I'm leaning
> > toward
> > > > > using this name since the current one seems likely to cause
> > confusion.
> > > > Any
> > > > > thoughts?
> > > > >
> > > > > Thanks,
> > > > > Jason
> > > > >
> > > > >
> > > > > On Thu, May 31, 2018 at 6:09 PM, Dong Lin 
> > wrote:
> > > > >
> > > > > > Thanks for the KIP! I am in favor of the option 1.
> > > > > >
> > > > > > +1 as well.
> > > > > >
> > > > > > On Thu, May 31, 2018 at 6:00 PM, Jason Gustafson <
> > ja...@confluent.io
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Thanks everyone for the feedback. I've updated the KIP and
> added
> > > > > > > KAFKA-6979.
> > > > > > >
> > > > > > > -Jason
> > > > > > >
> > > > > > > On Wed, May 30, 2018 at 3:50 PM, Guozhang Wang <
> > wangg...@gmail.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > Thanks Jason. I'm in favor of option 1 as well.
> > > > > > > >
> > > > > > > > On Wed, May 30, 2018 at 1:37 PM, Bill Bejeck <
> > bbej...@gmail.com>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > For what it's worth I'm +1 on Option 1 and the default
> value
> > > for
> > > > > the
> > > > > > > > > timeout.
> > > > > > > > >
> > > > > > > > > In addition to reasons outlined above by Jason, I think it
> > will
> > > > > help
> > > > > > to
> > > > > > > > > reason about consumer behavior (with respect to blocking)
> > > having
> > > > > the
> > > > > > > > > configuration and default value aligned with the producer.
> > > > > > > > >
> > > > > > > > > -Bill
> > > > > > > > >
> > > > > > > > > On Wed, May 30, 2018 at 3:43 PM, Ismael Juma <
> > > ism...@juma.me.uk>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Sounds good to me,
> > > > > > > > > >
> > > > > > > > > > On Wed, May 30, 2018 at 12:40 PM Jason Gustafson <
> > > > > > ja...@confluent.io
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Perhaps one minute? That is the default used by the
> > > producer.
> > > > > > > > > > >
> > > > > > > > > > > -Jason
> > > > > > > > > > >
> > > > > > > > > > > On Wed, May 30, 2018 at 9:50 AM, Ismael Juma <
> > > > > ism...@juma.me.uk>
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Option 1 sounds good to me provided that we can come
> up
> > > > with
> > > > > a
> > > > > > > good
> > > > > > > > > > > > default. What would

Re: [VOTE] KIP-266: Add TimeoutException for KafkaConsumer#position

[Jason Gustafson]

Thanks for the comments. I'm not sure I understand why we want to avoid the
term "timeout." Semantically, that's what it is. If we don't want another
timeout config, we could avoid it and hard-code a reasonable default or try
to wrap the behavior into one of the other timeouts (which is sort of what
we were already doing with request.timeout.ms). But I'm not too sure
calling it something else addresses the problem.

-Jason

On Tue, Jun 5, 2018 at 2:59 PM, Dhruvil Shah  wrote:

> I agree that using `default.timeout.ms` could cause confusion since we
> already have other timeout configurations in the consumer.
>
> +1 for using `default.block.ms`.
>
> Thanks,
> Dhruvil
>
> On Tue, Jun 5, 2018 at 11:48 AM, Bill Bejeck  wrote:
>
> > Hi Jason,
> >
> > At first, I thought the same name between the producer and the consumer
> was
> > ideal.
> >
> > But your comment makes me realize consistent names with different
> semantics
> > is even more confusing.
> >
> > I'm +1 for not using `max.block.ms`.  I like Guozhang's suggestion of `
> > default.block.ms` for the name.
> >
> > Thanks,
> > Bill
> >
> > On Tue, Jun 5, 2018 at 1:33 PM, Guozhang Wang 
> wrote:
> >
> > > Hi Jason,
> > >
> > > Yeah I agree that "max.block.ms" makes people thinking of the
> producer's
> > > config with the same name, but their semantics are different.
> > >
> > > On the other hand, I'm a bit concerned with the reusing of the term
> > > `timeout` as we already have `session.timeout.ms` and `
> > request.timeout.ms`
> > > in ConsumerConfig.. How about using the name `default.api.block.ms` or
> > > simply `default.block.ms`?
> > >
> > >
> > >
> > > Guozhang
> > >
> > >
> > > On Tue, Jun 5, 2018 at 8:57 AM, Jason Gustafson 
> > > wrote:
> > >
> > > > Hey All,
> > > >
> > > > One more minor follow-up. As I was reviewing the change mentioned
> > above,
> > > I
> > > > felt the name `max.block.ms` was a little bit misleading since it
> only
> > > > applies to methods which do not have an explicit timeout. A clearer
> > name
> > > > given its usage might be `default.timeout.ms`. It is the default
> > timeout
> > > > for any blocking API which does not have a timeout. I'm leaning
> toward
> > > > using this name since the current one seems likely to cause
> confusion.
> > > Any
> > > > thoughts?
> > > >
> > > > Thanks,
> > > > Jason
> > > >
> > > >
> > > > On Thu, May 31, 2018 at 6:09 PM, Dong Lin 
> wrote:
> > > >
> > > > > Thanks for the KIP! I am in favor of the option 1.
> > > > >
> > > > > +1 as well.
> > > > >
> > > > > On Thu, May 31, 2018 at 6:00 PM, Jason Gustafson <
> ja...@confluent.io
> > >
> > > > > wrote:
> > > > >
> > > > > > Thanks everyone for the feedback. I've updated the KIP and added
> > > > > > KAFKA-6979.
> > > > > >
> > > > > > -Jason
> > > > > >
> > > > > > On Wed, May 30, 2018 at 3:50 PM, Guozhang Wang <
> wangg...@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Thanks Jason. I'm in favor of option 1 as well.
> > > > > > >
> > > > > > > On Wed, May 30, 2018 at 1:37 PM, Bill Bejeck <
> bbej...@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > > For what it's worth I'm +1 on Option 1 and the default value
> > for
> > > > the
> > > > > > > > timeout.
> > > > > > > >
> > > > > > > > In addition to reasons outlined above by Jason, I think it
> will
> > > > help
> > > > > to
> > > > > > > > reason about consumer behavior (with respect to blocking)
> > having
> > > > the
> > > > > > > > configuration and default value aligned with the producer.
> > > > > > > >
> > > > > > > > -Bill
> > > > > > > >
> > > > > > > > On Wed, May 30, 2018 at 3:43 PM, Ismael Juma <
> > ism...@juma.me.uk>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Sounds good to me,
> > > > > > > > >
> > > > > > > > > On Wed, May 30, 2018 at 12:40 PM Jason Gustafson <
> > > > > ja...@confluent.io
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Perhaps one minute? That is the default used by the
> > producer.
> > > > > > > > > >
> > > > > > > > > > -Jason
> > > > > > > > > >
> > > > > > > > > > On Wed, May 30, 2018 at 9:50 AM, Ismael Juma <
> > > > ism...@juma.me.uk>
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Option 1 sounds good to me provided that we can come up
> > > with
> > > > a
> > > > > > good
> > > > > > > > > > > default. What would you suggest?
> > > > > > > > > > >
> > > > > > > > > > > Ismael
> > > > > > > > > > >
> > > > > > > > > > > On Wed, May 30, 2018 at 9:41 AM Jason Gustafson <
> > > > > > > ja...@confluent.io>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Everyone,
> > > > > > > > > > > >
> > > > > > > > > > > > There remains some inconsistency in the timeout
> > behavior
> > > of
> > > > > the
> > > > > > > > > > consumer
> > > > > > > > > > > > APIs which do not accept a timeout. Some of them
> block
> > > > > forever
> > > > > > > > (e.g.
> > > > > > > > > > > > position()) and some of them use request.timeout.ms
> > > (e.g.
> > > > > > > >

Re: [VOTE] KIP-266: Add TimeoutException for KafkaConsumer#position

[Dhruvil Shah]

I agree that using `default.timeout.ms` could cause confusion since we
already have other timeout configurations in the consumer.

+1 for using `default.block.ms`.

Thanks,
Dhruvil

On Tue, Jun 5, 2018 at 11:48 AM, Bill Bejeck  wrote:

> Hi Jason,
>
> At first, I thought the same name between the producer and the consumer was
> ideal.
>
> But your comment makes me realize consistent names with different semantics
> is even more confusing.
>
> I'm +1 for not using `max.block.ms`.  I like Guozhang's suggestion of `
> default.block.ms` for the name.
>
> Thanks,
> Bill
>
> On Tue, Jun 5, 2018 at 1:33 PM, Guozhang Wang  wrote:
>
> > Hi Jason,
> >
> > Yeah I agree that "max.block.ms" makes people thinking of the producer's
> > config with the same name, but their semantics are different.
> >
> > On the other hand, I'm a bit concerned with the reusing of the term
> > `timeout` as we already have `session.timeout.ms` and `
> request.timeout.ms`
> > in ConsumerConfig.. How about using the name `default.api.block.ms` or
> > simply `default.block.ms`?
> >
> >
> >
> > Guozhang
> >
> >
> > On Tue, Jun 5, 2018 at 8:57 AM, Jason Gustafson 
> > wrote:
> >
> > > Hey All,
> > >
> > > One more minor follow-up. As I was reviewing the change mentioned
> above,
> > I
> > > felt the name `max.block.ms` was a little bit misleading since it only
> > > applies to methods which do not have an explicit timeout. A clearer
> name
> > > given its usage might be `default.timeout.ms`. It is the default
> timeout
> > > for any blocking API which does not have a timeout. I'm leaning toward
> > > using this name since the current one seems likely to cause confusion.
> > Any
> > > thoughts?
> > >
> > > Thanks,
> > > Jason
> > >
> > >
> > > On Thu, May 31, 2018 at 6:09 PM, Dong Lin  wrote:
> > >
> > > > Thanks for the KIP! I am in favor of the option 1.
> > > >
> > > > +1 as well.
> > > >
> > > > On Thu, May 31, 2018 at 6:00 PM, Jason Gustafson  >
> > > > wrote:
> > > >
> > > > > Thanks everyone for the feedback. I've updated the KIP and added
> > > > > KAFKA-6979.
> > > > >
> > > > > -Jason
> > > > >
> > > > > On Wed, May 30, 2018 at 3:50 PM, Guozhang Wang  >
> > > > wrote:
> > > > >
> > > > > > Thanks Jason. I'm in favor of option 1 as well.
> > > > > >
> > > > > > On Wed, May 30, 2018 at 1:37 PM, Bill Bejeck 
> > > > wrote:
> > > > > >
> > > > > > > For what it's worth I'm +1 on Option 1 and the default value
> for
> > > the
> > > > > > > timeout.
> > > > > > >
> > > > > > > In addition to reasons outlined above by Jason, I think it will
> > > help
> > > > to
> > > > > > > reason about consumer behavior (with respect to blocking)
> having
> > > the
> > > > > > > configuration and default value aligned with the producer.
> > > > > > >
> > > > > > > -Bill
> > > > > > >
> > > > > > > On Wed, May 30, 2018 at 3:43 PM, Ismael Juma <
> ism...@juma.me.uk>
> > > > > wrote:
> > > > > > >
> > > > > > > > Sounds good to me,
> > > > > > > >
> > > > > > > > On Wed, May 30, 2018 at 12:40 PM Jason Gustafson <
> > > > ja...@confluent.io
> > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Perhaps one minute? That is the default used by the
> producer.
> > > > > > > > >
> > > > > > > > > -Jason
> > > > > > > > >
> > > > > > > > > On Wed, May 30, 2018 at 9:50 AM, Ismael Juma <
> > > ism...@juma.me.uk>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Option 1 sounds good to me provided that we can come up
> > with
> > > a
> > > > > good
> > > > > > > > > > default. What would you suggest?
> > > > > > > > > >
> > > > > > > > > > Ismael
> > > > > > > > > >
> > > > > > > > > > On Wed, May 30, 2018 at 9:41 AM Jason Gustafson <
> > > > > > ja...@confluent.io>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Everyone,
> > > > > > > > > > >
> > > > > > > > > > > There remains some inconsistency in the timeout
> behavior
> > of
> > > > the
> > > > > > > > > consumer
> > > > > > > > > > > APIs which do not accept a timeout. Some of them block
> > > > forever
> > > > > > > (e.g.
> > > > > > > > > > > position()) and some of them use request.timeout.ms
> > (e.g.
> > > > > > > > > > > parititonsFor()).
> > > > > > > > > > > I think we'd probably all agree that blocking forever
> is
> > > not
> > > > > > useful
> > > > > > > > > > > behavior and using request.timeout.ms has always been
> a
> > > hack
> > > > > > since
> > > > > > > > it
> > > > > > > > > > > controls a separate concern. I think there are
> basically
> > > two
> > > > > > > options
> > > > > > > > to
> > > > > > > > > > > address this:
> > > > > > > > > > >
> > > > > > > > > > > 1. We can add max.block.ms to match the producer and
> use
> > > it
> > > > as
> > > > > > the
> > > > > > > > > > default
> > > > > > > > > > > timeout when a timeout is not explicitly provided. This
> > > will
> > > > > fix
> > > > > > > the
> > > > > > > > > > > indefinite blocking behavior and avoid conflating
> > > > > > > request.timeout.ms
> > > > > > > > .
> > > > > > > > > > 

[jira] [Resolved] (KAFKA-6981) Missing Connector Config (errors.deadletterqueue.topic.name) kills Connect Clusters

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava resolved KAFKA-6981.
--
Resolution: Fixed

Issue resolved by pull request 5125
[https://github.com/apache/kafka/pull/5125]

> Missing Connector Config (errors.deadletterqueue.topic.name) kills Connect 
> Clusters
> ---
>
> Key: KAFKA-6981
> URL: https://issues.apache.org/jira/browse/KAFKA-6981
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Arjun Satish
>Assignee: Arjun Satish
>Priority: Major
> Fix For: 2.0.0
>
>
> The trunk version of AK currently tries to incorrectly read the property 
> (errors.deadletterqueue.topic.name) when starting a sink connector. This 
> fails no matter what the contents of the connector config are. The 
> ConnectorConfig does not define this property, and any calls to getString 
> will throw a ConfigException (since only known properties are retained by 
> AbstractConfig). 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: kafka-1.1-jdk7 #140

[Apache Jenkins Server]

See 


Changes:

[matthias] MINOR: Add missing configs for resilience settings

--
[...truncated 419.30 KB...]
kafka.admin.ResetConsumerGroupOffsetTest > 
testResetOffsetsNewConsumerExistingTopic PASSED

kafka.admin.ResetConsumerGroupOffsetTest > 
testResetOffsetsShiftByLowerThanEarliest STARTED

kafka.admin.ResetConsumerGroupOffsetTest > 
testResetOffsetsShiftByLowerThanEarliest PASSED

kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsByDuration STARTED

kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsByDuration PASSED

kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsToLocalDateTime 
STARTED

kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsToLocalDateTime 
PASSED

kafka.admin.ResetConsumerGroupOffsetTest > 
testResetOffsetsToEarliestOnTopicsAndPartitions STARTED

kafka.admin.ResetConsumerGroupOffsetTest > 
testResetOffsetsToEarliestOnTopicsAndPartitions PASSED

kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsToEarliestOnTopics 
STARTED

kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsToEarliestOnTopics 
PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > shouldFailIfBlankArg STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > shouldFailIfBlankArg PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowVerifyWithoutReassignmentOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowVerifyWithoutReassignmentOption PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithoutBrokersOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithoutBrokersOption PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowTopicsOptionWithVerify STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowTopicsOptionWithVerify PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithThrottleOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithThrottleOption PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > shouldFailIfNoArgs STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > shouldFailIfNoArgs PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowExecuteWithoutReassignmentOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowExecuteWithoutReassignmentOption PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowBrokersListWithVerifyOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowBrokersListWithVerifyOption PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldCorrectlyParseValidMinimumExecuteOptions STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldCorrectlyParseValidMinimumExecuteOptions PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithReassignmentOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithReassignmentOption PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldCorrectlyParseValidMinimumGenerateOptions STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldCorrectlyParseValidMinimumGenerateOptions PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithoutBrokersAndTopicsOptions STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithoutBrokersAndTopicsOptions PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowThrottleWithVerifyOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowThrottleWithVerifyOption PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > shouldUseDefaultsIfEnabled 
STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > shouldUseDefaultsIfEnabled 
PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldAllowThrottleOptionOnExecute STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldAllowThrottleOptionOnExecute PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowExecuteWithBrokers STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowExecuteWithBrokers PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowExecuteWithTopicsOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowExecuteWithTopicsOption PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldCorrectlyParseValidMinimumVerifyOptions STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldCorrectlyParseValidMinimumVerifyOptions PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithoutTopicsOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithoutTopicsOption PASSED

kafka.admin.BrokerApiVersionsCommandTest > checkBrokerApiVersionCommandOutput 
STARTED

kafka.admin.BrokerA

[jira] [Resolved] (KAFKA-6730) Simplify state store recovery

[Matthias J. Sax (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-6730.

Resolution: Fixed

> Simplify state store recovery
> -
>
> Key: KAFKA-6730
> URL: https://issues.apache.org/jira/browse/KAFKA-6730
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Matthias J. Sax
>Assignee: Richard Yu
>Priority: Major
> Fix For: 2.0.0
>
>
> In the current code base, we restore state stores in the main thread (in 
> contrast to older code that did restore state stored in the rebalance call 
> back). This has multiple advantages and allows us the further simplify 
> restore code.
> In the original code base, during a long restore phase, it was possible that 
> a instance misses a rebalance and drops out of the consumer group. To detect 
> this case, we apply a check during the restore phase, that the end-offset of 
> the changelog topic does not change. A changed offset implies a missed 
> rebalance as another thread started to write into the changelog topic (ie, 
> the current thread does not own the task/store/changelog-topic anymore).
> With the new code, that restores in the main-loop, it's ensured that `poll()` 
> is called regularly and thus, a rebalance will be detected automatically. 
> This make the check about an changing changelog-end-offset unnecessary.
> We can simplify the restore logic, to just consuming until `poll()` does not 
> return any data. For this case, we fetch the end-offset to see if we did 
> fully restore. If yes, we resume processing, if not, we continue the restore.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-6997) Kafka run class doesn't exclude test-sources jar

[Magesh kumar Nandakumar (JIRA)]

Magesh kumar Nandakumar created KAFKA-6997:
--

 Summary: Kafka run class doesn't exclude test-sources jar
 Key: KAFKA-6997
 URL: https://issues.apache.org/jira/browse/KAFKA-6997
 Project: Kafka
  Issue Type: Improvement
  Components: KafkaConnect
Affects Versions: 2.0.0
Reporter: Magesh kumar Nandakumar
Assignee: Magesh kumar Nandakumar
 Fix For: 2.0.0


kafka-run-class.sh has a flag INCLUDE_TEST_JAR. This doesn't exclude 
test-sources jar files when the flag is set to false. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Discount code for kafka summit SF 2018

[Aarti Gupta]

Hi folks,

Was wondering if this list has a discount code for kafka summit -San
francisco 2018.
I know there was one last year, so just wanted to check!

Thanks!
aarti

Jenkins build is back to normal : kafka-trunk-jdk8 #2702

[Apache Jenkins Server]

See 

Re: [VOTE] KIP-266: Add TimeoutException for KafkaConsumer#position

[Bill Bejeck]

Hi Jason,

At first, I thought the same name between the producer and the consumer was
ideal.

But your comment makes me realize consistent names with different semantics
is even more confusing.

I'm +1 for not using `max.block.ms`.  I like Guozhang's suggestion of `
default.block.ms` for the name.

Thanks,
Bill

On Tue, Jun 5, 2018 at 1:33 PM, Guozhang Wang  wrote:

> Hi Jason,
>
> Yeah I agree that "max.block.ms" makes people thinking of the producer's
> config with the same name, but their semantics are different.
>
> On the other hand, I'm a bit concerned with the reusing of the term
> `timeout` as we already have `session.timeout.ms` and `request.timeout.ms`
> in ConsumerConfig.. How about using the name `default.api.block.ms` or
> simply `default.block.ms`?
>
>
>
> Guozhang
>
>
> On Tue, Jun 5, 2018 at 8:57 AM, Jason Gustafson 
> wrote:
>
> > Hey All,
> >
> > One more minor follow-up. As I was reviewing the change mentioned above,
> I
> > felt the name `max.block.ms` was a little bit misleading since it only
> > applies to methods which do not have an explicit timeout. A clearer name
> > given its usage might be `default.timeout.ms`. It is the default timeout
> > for any blocking API which does not have a timeout. I'm leaning toward
> > using this name since the current one seems likely to cause confusion.
> Any
> > thoughts?
> >
> > Thanks,
> > Jason
> >
> >
> > On Thu, May 31, 2018 at 6:09 PM, Dong Lin  wrote:
> >
> > > Thanks for the KIP! I am in favor of the option 1.
> > >
> > > +1 as well.
> > >
> > > On Thu, May 31, 2018 at 6:00 PM, Jason Gustafson 
> > > wrote:
> > >
> > > > Thanks everyone for the feedback. I've updated the KIP and added
> > > > KAFKA-6979.
> > > >
> > > > -Jason
> > > >
> > > > On Wed, May 30, 2018 at 3:50 PM, Guozhang Wang 
> > > wrote:
> > > >
> > > > > Thanks Jason. I'm in favor of option 1 as well.
> > > > >
> > > > > On Wed, May 30, 2018 at 1:37 PM, Bill Bejeck 
> > > wrote:
> > > > >
> > > > > > For what it's worth I'm +1 on Option 1 and the default value for
> > the
> > > > > > timeout.
> > > > > >
> > > > > > In addition to reasons outlined above by Jason, I think it will
> > help
> > > to
> > > > > > reason about consumer behavior (with respect to blocking) having
> > the
> > > > > > configuration and default value aligned with the producer.
> > > > > >
> > > > > > -Bill
> > > > > >
> > > > > > On Wed, May 30, 2018 at 3:43 PM, Ismael Juma 
> > > > wrote:
> > > > > >
> > > > > > > Sounds good to me,
> > > > > > >
> > > > > > > On Wed, May 30, 2018 at 12:40 PM Jason Gustafson <
> > > ja...@confluent.io
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Perhaps one minute? That is the default used by the producer.
> > > > > > > >
> > > > > > > > -Jason
> > > > > > > >
> > > > > > > > On Wed, May 30, 2018 at 9:50 AM, Ismael Juma <
> > ism...@juma.me.uk>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Option 1 sounds good to me provided that we can come up
> with
> > a
> > > > good
> > > > > > > > > default. What would you suggest?
> > > > > > > > >
> > > > > > > > > Ismael
> > > > > > > > >
> > > > > > > > > On Wed, May 30, 2018 at 9:41 AM Jason Gustafson <
> > > > > ja...@confluent.io>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Everyone,
> > > > > > > > > >
> > > > > > > > > > There remains some inconsistency in the timeout behavior
> of
> > > the
> > > > > > > > consumer
> > > > > > > > > > APIs which do not accept a timeout. Some of them block
> > > forever
> > > > > > (e.g.
> > > > > > > > > > position()) and some of them use request.timeout.ms
> (e.g.
> > > > > > > > > > parititonsFor()).
> > > > > > > > > > I think we'd probably all agree that blocking forever is
> > not
> > > > > useful
> > > > > > > > > > behavior and using request.timeout.ms has always been a
> > hack
> > > > > since
> > > > > > > it
> > > > > > > > > > controls a separate concern. I think there are basically
> > two
> > > > > > options
> > > > > > > to
> > > > > > > > > > address this:
> > > > > > > > > >
> > > > > > > > > > 1. We can add max.block.ms to match the producer and use
> > it
> > > as
> > > > > the
> > > > > > > > > default
> > > > > > > > > > timeout when a timeout is not explicitly provided. This
> > will
> > > > fix
> > > > > > the
> > > > > > > > > > indefinite blocking behavior and avoid conflating
> > > > > > request.timeout.ms
> > > > > > > .
> > > > > > > > > > 2. We can deprecate the methods which don't accept a
> > timeout.
> > > > > > > > > >
> > > > > > > > > > I'm leaning toward the first solution because I think we
> > want
> > > > to
> > > > > > push
> > > > > > > > > users
> > > > > > > > > > to specifying timeouts through configuration rather than
> in
> > > > code
> > > > > > > (Jay's
> > > > > > > > > > original argument). I think the overloads are still
> useful
> > > for
> > > > > > > advanced
> > > > > > > > > > usage (e.g. in kafka streams), but we should give users
> an
> > > easy
> > > > > > > option
> > > > > > > > > 

Re: [VOTE] KIP-266: Add TimeoutException for KafkaConsumer#position

[Guozhang Wang]

Hi Jason,

Yeah I agree that "max.block.ms" makes people thinking of the producer's
config with the same name, but their semantics are different.

On the other hand, I'm a bit concerned with the reusing of the term
`timeout` as we already have `session.timeout.ms` and `request.timeout.ms`
in ConsumerConfig.. How about using the name `default.api.block.ms` or
simply `default.block.ms`?



Guozhang


On Tue, Jun 5, 2018 at 8:57 AM, Jason Gustafson  wrote:

> Hey All,
>
> One more minor follow-up. As I was reviewing the change mentioned above, I
> felt the name `max.block.ms` was a little bit misleading since it only
> applies to methods which do not have an explicit timeout. A clearer name
> given its usage might be `default.timeout.ms`. It is the default timeout
> for any blocking API which does not have a timeout. I'm leaning toward
> using this name since the current one seems likely to cause confusion. Any
> thoughts?
>
> Thanks,
> Jason
>
>
> On Thu, May 31, 2018 at 6:09 PM, Dong Lin  wrote:
>
> > Thanks for the KIP! I am in favor of the option 1.
> >
> > +1 as well.
> >
> > On Thu, May 31, 2018 at 6:00 PM, Jason Gustafson 
> > wrote:
> >
> > > Thanks everyone for the feedback. I've updated the KIP and added
> > > KAFKA-6979.
> > >
> > > -Jason
> > >
> > > On Wed, May 30, 2018 at 3:50 PM, Guozhang Wang 
> > wrote:
> > >
> > > > Thanks Jason. I'm in favor of option 1 as well.
> > > >
> > > > On Wed, May 30, 2018 at 1:37 PM, Bill Bejeck 
> > wrote:
> > > >
> > > > > For what it's worth I'm +1 on Option 1 and the default value for
> the
> > > > > timeout.
> > > > >
> > > > > In addition to reasons outlined above by Jason, I think it will
> help
> > to
> > > > > reason about consumer behavior (with respect to blocking) having
> the
> > > > > configuration and default value aligned with the producer.
> > > > >
> > > > > -Bill
> > > > >
> > > > > On Wed, May 30, 2018 at 3:43 PM, Ismael Juma 
> > > wrote:
> > > > >
> > > > > > Sounds good to me,
> > > > > >
> > > > > > On Wed, May 30, 2018 at 12:40 PM Jason Gustafson <
> > ja...@confluent.io
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Perhaps one minute? That is the default used by the producer.
> > > > > > >
> > > > > > > -Jason
> > > > > > >
> > > > > > > On Wed, May 30, 2018 at 9:50 AM, Ismael Juma <
> ism...@juma.me.uk>
> > > > > wrote:
> > > > > > >
> > > > > > > > Option 1 sounds good to me provided that we can come up with
> a
> > > good
> > > > > > > > default. What would you suggest?
> > > > > > > >
> > > > > > > > Ismael
> > > > > > > >
> > > > > > > > On Wed, May 30, 2018 at 9:41 AM Jason Gustafson <
> > > > ja...@confluent.io>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Everyone,
> > > > > > > > >
> > > > > > > > > There remains some inconsistency in the timeout behavior of
> > the
> > > > > > > consumer
> > > > > > > > > APIs which do not accept a timeout. Some of them block
> > forever
> > > > > (e.g.
> > > > > > > > > position()) and some of them use request.timeout.ms (e.g.
> > > > > > > > > parititonsFor()).
> > > > > > > > > I think we'd probably all agree that blocking forever is
> not
> > > > useful
> > > > > > > > > behavior and using request.timeout.ms has always been a
> hack
> > > > since
> > > > > > it
> > > > > > > > > controls a separate concern. I think there are basically
> two
> > > > > options
> > > > > > to
> > > > > > > > > address this:
> > > > > > > > >
> > > > > > > > > 1. We can add max.block.ms to match the producer and use
> it
> > as
> > > > the
> > > > > > > > default
> > > > > > > > > timeout when a timeout is not explicitly provided. This
> will
> > > fix
> > > > > the
> > > > > > > > > indefinite blocking behavior and avoid conflating
> > > > > request.timeout.ms
> > > > > > .
> > > > > > > > > 2. We can deprecate the methods which don't accept a
> timeout.
> > > > > > > > >
> > > > > > > > > I'm leaning toward the first solution because I think we
> want
> > > to
> > > > > push
> > > > > > > > users
> > > > > > > > > to specifying timeouts through configuration rather than in
> > > code
> > > > > > (Jay's
> > > > > > > > > original argument). I think the overloads are still useful
> > for
> > > > > > advanced
> > > > > > > > > usage (e.g. in kafka streams), but we should give users an
> > easy
> > > > > > option
> > > > > > > > with
> > > > > > > > > reasonable default behavior.
> > > > > > > > >
> > > > > > > > > If that sounds ok, I'd propose we add it to this KIP and
> fix
> > it
> > > > > now.
> > > > > > > This
> > > > > > > > > gives users an easy way to get the benefit of the
> > improvements
> > > > from
> > > > > > > this
> > > > > > > > > KIP without changing any code.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Jason
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Sun, May 13, 2018 at 7:58 PM, Richard Yu <
> > > > > > > yohan.richard...@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi,
> > >

Re: I'd like permission to create a KIP

[Guozhang Wang]

Hello Richard,

Done, cheers.

Guozhang

On Tue, Jun 5, 2018 at 10:09 AM, Richard Tjerngren 
wrote:

> my username on the wiki is richodemus
>
> thanks
>



-- 
-- Guozhang

Re: How to initiate a KIP?

[Guozhang Wang]

Hello Virgil,

I have granted you the permission, if you refresh your browser you should
see the link now.


Guozhang


On Tue, Jun 5, 2018 at 1:54 AM, Virgil Palanciuc 
wrote:

> Hi all,
>
> Here
> 
> it says that “anyone can initiate a KIP”, however the “Create KIP” link
> doesn’t work for me. How can I get the permissions to initiate one? (or if
> I can’t… what’s the process for contacting someone with a proposal?)
>
>
>
>
>
>
>
> Regards,
>
> Virgil.
>



-- 
-- Guozhang

I'd like permission to create a KIP

[Richard Tjerngren]

my username on the wiki is richodemus

thanks

Re: [VOTE] KIP-266: Add TimeoutException for KafkaConsumer#position

[Jason Gustafson]

Hey All,

One more minor follow-up. As I was reviewing the change mentioned above, I
felt the name `max.block.ms` was a little bit misleading since it only
applies to methods which do not have an explicit timeout. A clearer name
given its usage might be `default.timeout.ms`. It is the default timeout
for any blocking API which does not have a timeout. I'm leaning toward
using this name since the current one seems likely to cause confusion. Any
thoughts?

Thanks,
Jason


On Thu, May 31, 2018 at 6:09 PM, Dong Lin  wrote:

> Thanks for the KIP! I am in favor of the option 1.
>
> +1 as well.
>
> On Thu, May 31, 2018 at 6:00 PM, Jason Gustafson 
> wrote:
>
> > Thanks everyone for the feedback. I've updated the KIP and added
> > KAFKA-6979.
> >
> > -Jason
> >
> > On Wed, May 30, 2018 at 3:50 PM, Guozhang Wang 
> wrote:
> >
> > > Thanks Jason. I'm in favor of option 1 as well.
> > >
> > > On Wed, May 30, 2018 at 1:37 PM, Bill Bejeck 
> wrote:
> > >
> > > > For what it's worth I'm +1 on Option 1 and the default value for the
> > > > timeout.
> > > >
> > > > In addition to reasons outlined above by Jason, I think it will help
> to
> > > > reason about consumer behavior (with respect to blocking) having the
> > > > configuration and default value aligned with the producer.
> > > >
> > > > -Bill
> > > >
> > > > On Wed, May 30, 2018 at 3:43 PM, Ismael Juma 
> > wrote:
> > > >
> > > > > Sounds good to me,
> > > > >
> > > > > On Wed, May 30, 2018 at 12:40 PM Jason Gustafson <
> ja...@confluent.io
> > >
> > > > > wrote:
> > > > >
> > > > > > Perhaps one minute? That is the default used by the producer.
> > > > > >
> > > > > > -Jason
> > > > > >
> > > > > > On Wed, May 30, 2018 at 9:50 AM, Ismael Juma 
> > > > wrote:
> > > > > >
> > > > > > > Option 1 sounds good to me provided that we can come up with a
> > good
> > > > > > > default. What would you suggest?
> > > > > > >
> > > > > > > Ismael
> > > > > > >
> > > > > > > On Wed, May 30, 2018 at 9:41 AM Jason Gustafson <
> > > ja...@confluent.io>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Everyone,
> > > > > > > >
> > > > > > > > There remains some inconsistency in the timeout behavior of
> the
> > > > > > consumer
> > > > > > > > APIs which do not accept a timeout. Some of them block
> forever
> > > > (e.g.
> > > > > > > > position()) and some of them use request.timeout.ms (e.g.
> > > > > > > > parititonsFor()).
> > > > > > > > I think we'd probably all agree that blocking forever is not
> > > useful
> > > > > > > > behavior and using request.timeout.ms has always been a hack
> > > since
> > > > > it
> > > > > > > > controls a separate concern. I think there are basically two
> > > > options
> > > > > to
> > > > > > > > address this:
> > > > > > > >
> > > > > > > > 1. We can add max.block.ms to match the producer and use it
> as
> > > the
> > > > > > > default
> > > > > > > > timeout when a timeout is not explicitly provided. This will
> > fix
> > > > the
> > > > > > > > indefinite blocking behavior and avoid conflating
> > > > request.timeout.ms
> > > > > .
> > > > > > > > 2. We can deprecate the methods which don't accept a timeout.
> > > > > > > >
> > > > > > > > I'm leaning toward the first solution because I think we want
> > to
> > > > push
> > > > > > > users
> > > > > > > > to specifying timeouts through configuration rather than in
> > code
> > > > > (Jay's
> > > > > > > > original argument). I think the overloads are still useful
> for
> > > > > advanced
> > > > > > > > usage (e.g. in kafka streams), but we should give users an
> easy
> > > > > option
> > > > > > > with
> > > > > > > > reasonable default behavior.
> > > > > > > >
> > > > > > > > If that sounds ok, I'd propose we add it to this KIP and fix
> it
> > > > now.
> > > > > > This
> > > > > > > > gives users an easy way to get the benefit of the
> improvements
> > > from
> > > > > > this
> > > > > > > > KIP without changing any code.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Jason
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Sun, May 13, 2018 at 7:58 PM, Richard Yu <
> > > > > > yohan.richard...@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > With 3 binding votes and 6 non-binding, this KIP would be
> > > > accepted.
> > > > > > > > >
> > > > > > > > > Thanks for participating.
> > > > > > > > >
> > > > > > > > > On Thu, May 10, 2018 at 2:35 AM, Edoardo Comar <
> > > > edoco...@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > +1 (non-binding)
> > > > > > > > > >
> > > > > > > > > > On 10 May 2018 at 10:29, zhenya Sun 
> > wrote:
> > > > > > > > > >
> > > > > > > > > > > +1 non-binding
> > > > > > > > > > >
> > > > > > > > > > > > 在 2018年5月10日，下午5:19，Manikumar <
> > manikumar.re...@gmail.com
> > > >
> > > > > 写道：
> > > > > > > > > > > >
> > > > > > > > > > > > +1 (non-binding).
> > > > > > > > > > > > Thanks.
> > > > > > > > > > > >
> > > > > > > > 

[jira] [Resolved] (KAFKA-6996) ConsumerGroupCommand.scala uses deprecated components

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-6996.

Resolution: Duplicate

Duplicate of KAFKA-6884,

> ConsumerGroupCommand.scala uses deprecated components
> -
>
> Key: KAFKA-6996
> URL: https://issues.apache.org/jira/browse/KAFKA-6996
> Project: Kafka
>  Issue Type: Bug
>  Components: admin
>Affects Versions: 1.1.0
>Reporter: Zeynep Arikoglu
>Priority: Minor
>
> ConsumerGroupCommand.scala which is the only component to get the topic lags 
> still uses the deprecated components, such as SimpleConsumer and 
> ZkGroupTopicDirs have been deprecated with Kafka 0.11.0.0.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

KIP-176 and related PR : today deadline for merging in 2.0.0

[Paolo Patierno]

Hi,

because today, as far as I understood, should be the deadline for having the 
PRs related to KIPs merged for the 2.0.0, maybe any committers can take a look 
(review and maybe merge) the PR :

https://github.com/apache/kafka/pull/5097

related to the KIP-176 :

https://cwiki.apache.org/confluence/display/KAFKA/KIP-176%3A+Remove+deprecated+new-consumer+option+for+tools

Thanks


Paolo Patierno
Principal Software Engineer (IoT) @ Red Hat
Microsoft MVP on Azure & IoT
Microsoft Azure Advisor

Twitter : @ppatierno
Linkedin : paolopatierno
Blog : DevExperience

[jira] [Created] (KAFKA-6996) ConsumerGroupCommand.scala uses deprecated components

[Zeynep Arikoglu (JIRA)]

Zeynep Arikoglu created KAFKA-6996:
--

 Summary: ConsumerGroupCommand.scala uses deprecated components
 Key: KAFKA-6996
 URL: https://issues.apache.org/jira/browse/KAFKA-6996
 Project: Kafka
  Issue Type: Bug
  Components: admin
Affects Versions: 1.1.0
Reporter: Zeynep Arikoglu


ConsumerGroupCommand.scala which is the only component to get the topic lags 
still uses the deprecated components, such as SimpleConsumer and 
ZkGroupTopicDirs have been deprecated with Kafka 0.11.0.0.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-6995) Make config "internal.leave.group.on.close" public

[Boyang Chen (JIRA)]

Boyang Chen created KAFKA-6995:
--

 Summary: Make config "internal.leave.group.on.close" public
 Key: KAFKA-6995
 URL: https://issues.apache.org/jira/browse/KAFKA-6995
 Project: Kafka
  Issue Type: Improvement
  Components: consumer, streams
Reporter: Boyang Chen
Assignee: Boyang Chen


We are proposing to make the config "internal.leave.group.on.close" public. The 
reason is that for heavy state application the sticky assignment won't work 
because each stream worker will leave group during rolling restart, and there 
is a possibility that some members are left and rejoined while others are still 
awaiting restart. This would then cause multiple rebalance because after the 
ongoing rebalance is done, we are expecting late members to rejoin and move 
state from `stable` to `prepareBalance`. To solve this problem, heavy state 
application needs to use this config to avoid member list update, so that at 
most one rebalance will be triggered at a proper time when all the members are 
rejoined during rolling restart. This should just be one line change.

Code here:

* internal.leave.group.on.close
 * Whether or not the consumer should leave the group on close. If set to 
false then a rebalance
 * won't occur until session.timeout.ms expires.
 *
 * 
 * Note: this is an internal configuration and could be changed in the future 
in a backward incompatible way
 *
 */
 static final String LEAVE_GROUP_ON_CLOSE_CONFIG = 
"internal.leave.group.on.close";



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: kafka-trunk-jdk8 #2701

[Apache Jenkins Server]

See 


Changes:

[ismael] KAFKA-3665: Enable TLS hostname verification by default (KIP-294)

--
[...truncated 2.56 MB...]

org.apache.kafka.streams.integration.StreamTableJoinIntegrationTest > 
testInner[caching enabled = false] STARTED

org.apache.kafka.streams.integration.StreamTableJoinIntegrationTest > 
testInner[caching enabled = false] PASSED

org.apache.kafka.streams.integration.StreamTableJoinIntegrationTest > 
testLeft[caching enabled = false] STARTED

org.apache.kafka.streams.integration.StreamTableJoinIntegrationTest > 
testLeft[caching enabled = false] PASSED

org.apache.kafka.streams.integration.ResetIntegrationTest > 
shouldNotAllowToResetWhenInputTopicAbsent STARTED

org.apache.kafka.streams.integration.ResetIntegrationTest > 
shouldNotAllowToResetWhenInputTopicAbsent PASSED

org.apache.kafka.streams.integration.ResetIntegrationTest > 
shouldNotAllowToResetWhenIntermediateTopicAbsent STARTED

org.apache.kafka.streams.integration.ResetIntegrationTest > 
shouldNotAllowToResetWhenIntermediateTopicAbsent PASSED

org.apache.kafka.streams.integration.ResetIntegrationTest > 
testReprocessingByDurationAfterResetWithoutIntermediateUserTopic STARTED

org.apache.kafka.streams.integration.ResetIntegrationTest > 
testReprocessingByDurationAfterResetWithoutIntermediateUserTopic PASSED

org.apache.kafka.streams.integration.ResetIntegrationTest > 
testReprocessingFromScratchAfterResetWithIntermediateUserTopic STARTED

org.apache.kafka.streams.integration.ResetIntegrationTest > 
testReprocessingFromScratchAfterResetWithIntermediateUserTopic PASSED

org.apache.kafka.streams.integration.ResetIntegrationTest > 
testReprocessingFromScratchAfterResetWithoutIntermediateUserTopic STARTED

org.apache.kafka.streams.integration.ResetIntegrationTest > 
testReprocessingFromScratchAfterResetWithoutIntermediateUserTopic PASSED

org.apache.kafka.streams.integration.ResetIntegrationTest > 
testReprocessingFromFileAfterResetWithoutIntermediateUserTopic STARTED

org.apache.kafka.streams.integration.ResetIntegrationTest > 
testReprocessingFromFileAfterResetWithoutIntermediateUserTopic PASSED

org.apache.kafka.streams.integration.ResetIntegrationTest > 
testReprocessingFromDateTimeAfterResetWithoutIntermediateUserTopic STARTED

org.apache.kafka.streams.integration.ResetIntegrationTest > 
testReprocessingFromDateTimeAfterResetWithoutIntermediateUserTopic PASSED

org.apache.kafka.streams.integration.ResetIntegrationTest > 
shouldNotAllowToResetWhileStreamsRunning STARTED

org.apache.kafka.streams.integration.ResetIntegrationTest > 
shouldNotAllowToResetWhileStreamsRunning PASSED

org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testLeftInner[caching enabled = true] STARTED

org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testLeftInner[caching enabled = true] PASSED

org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testLeftOuter[caching enabled = true] STARTED

org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testLeftOuter[caching enabled = true] PASSED

org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testLeftLeft[caching enabled = true] STARTED

org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testLeftLeft[caching enabled = true] PASSED

org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testOuterLeft[caching enabled = true] STARTED

org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testOuterLeft[caching enabled = true] PASSED

org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testInner[caching enabled = true] STARTED

org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testInner[caching enabled = true] PASSED

org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testOuter[caching enabled = true] STARTED

org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testOuter[caching enabled = true] PASSED

org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testLeft[caching enabled = true] STARTED

org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testLeft[caching enabled = true] PASSED

org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testInnerInner[caching enabled = true] STARTED

org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testInnerInner[caching enabled = true] PASSED

org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testInnerOuter[caching enabled = true] STARTED

org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testInnerOuter[caching enabled = true] PASSED

org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testInnerLeft[caching enabled = true] STARTED

org.apache.kafka.streams.integration.TableTableJoinIntegrationTest > 
testInnerLeft

[jira] [Resolved] (KAFKA-6982) java.lang.ArithmeticException: / by zero

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-6982.
--
   Resolution: Fixed
Fix Version/s: 2.0.0

> java.lang.ArithmeticException: / by zero
> 
>
> Key: KAFKA-6982
> URL: https://issues.apache.org/jira/browse/KAFKA-6982
> Project: Kafka
>  Issue Type: Bug
>  Components: network
>Affects Versions: 1.1.0
> Environment: Environment: Windows 10. 
>Reporter: wade wu
>Priority: Major
> Fix For: 2.0.0
>
>
> Producer keeps sending messages to Kafka, Kafka is down. 
> Server.log shows: 
> ..
> [2018-06-01 17:01:33,945] WARN [Log partition=__consumer_offsets-6, 
> dir=D:\data\Kafka\kafka-logs] Found a corrupted index file corresponding to 
> log file 
> D:\data\Kafka\kafka-logs__consumer_offsets-6\.log due to 
> Corrupt index found, index file 
> (D:\data\Kafka\kafka-logs__consumer_offsets-6\.index) has 
> non-zero size but the last offset is 0 which is no greater than the base 
> offset 0.}, recovering segment and rebuilding index files... (kafka.log.Log)
>  [2018-06-01 17:01:33,945] WARN [Log partition=__consumer_offsets-6, 
> dir=D:\data\Kafka\kafka-logs] Found a corrupted index file corresponding to 
> log file 
> D:\data\Kafka\kafka-logs__consumer_offsets-6\.log due to 
> Corrupt index found, index file 
> (D:\data\Kafka\kafka-logs__consumer_offsets-6\.index) has 
> non-zero size but the last offset is 0 which is no greater than the base 
> offset 0.}, recovering segment and rebuilding index files... (kafka.log.Log)
>  [2018-06-01 17:01:34,664] ERROR Error while accepting connection 
> (kafka.network.Acceptor)
>  java.lang.ArithmeticException: / by zero
>  at kafka.network.Acceptor.run(SocketServer.scala:354)
>  at java.lang.Thread.run(Thread.java:748)
>  [2018-06-01 17:01:34,664] ERROR Error while accepting connection 
> (kafka.network.Acceptor)
>  java.lang.ArithmeticException: / by zero
>  at kafka.network.Acceptor.run(SocketServer.scala:354)
>  at java.lang.Thread.run(Thread.java:748)
>  [2018-06-01 17:01:34,664] ERROR Error while accepting connection 
> (kafka.network.Acceptor)
>  java.lang.ArithmeticException: / by zero
>  at kafka.network.Acceptor.run(SocketServer.scala:354)
>  at java.lang.Thread.run(Thread.java:748)
> ..
>  
> This line of code in SocketServer.scala causing the error: 
>                   {color:#33} currentProcessor = 
> currentProcessor{color:#d04437} % processors.size{color}{color}
>  
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-6972) Kafka ACL does not work expected with wildcard

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-6972.
--
Resolution: Information Provided

> Kafka ACL does not work expected with wildcard
> --
>
> Key: KAFKA-6972
> URL: https://issues.apache.org/jira/browse/KAFKA-6972
> Project: Kafka
>  Issue Type: Bug
>  Components: security
>Affects Versions: 0.11.0.0
> Environment: OS : CentOS 7, 64bit.
> Confluent : 3.3, Kafka 0.11.
>Reporter: Soyee Deng
>Assignee: Sönke Liebau
>Priority: Major
>
> Just started with Confluent 3.3 platform and Kafka 0.11 having SSL as 
> transportation security and Kerberos to restrict the access control based on 
> the holding principals. In order to make life easier, wildcard is extensively 
> used in my environment. But it turned out that is not working as expected. 
> My issue is that when I run the command _kafka-acls_ under one directory with 
> some files, this command would pick up the name of first file as the topic 
> name or group name. e.g. In my case, abcd.txt would be chosen while giving my 
> principal connect-consumer the permissions of consuming message from any 
> topic with any group Id.
> [quality@data-pipeline-1 test_dir]$ 
> KAFKA_OPTS=-Djava.security.auth.login.config='/etc/security/jaas/broker-jaas.conf'
>  kafka-acls --authorizer-properties 
> zookeeper.connect=data-pipeline-1.orion.com:2181 --add --allow-principal 
> User:connect-consumer --consumer --topic * --group *
>  Adding ACLs for resource `Topic:abcd.txt`:
>  User:connect-consumer has Allow permission for operations: Describe from 
> hosts: *
>  User:connect-consumer has Allow permission for operations: Read from hosts: *
> Adding ACLs for resource `Group:abcd.txt`:
>  User:connect-consumer has Allow permission for operations: Read from hosts: *
> Current ACLs for resource `Topic:abcd.txt`:
>  User:connect-consumer has Allow permission for operations: Describe from 
> hosts: *
>  User:connect-consumer has Allow permission for operations: Read from hosts: *
>  User:connect-consumer has Allow permission for operations: Write from hosts: 
> *
> Current ACLs for resource `Group:abcd.txt`:
>  User:connect-consumer has Allow permission for operations: Read from hosts: *
>  
> My current work around solution is changing command context to an empty 
> directory and run above command, it works as expected. 
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-5523) ReplayLogProducer not using the new Kafka consumer

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-5523.
--
   Resolution: Fixed
 Assignee: Manikumar
Fix Version/s: 2.0.0

> ReplayLogProducer not using the new Kafka consumer
> --
>
> Key: KAFKA-5523
> URL: https://issues.apache.org/jira/browse/KAFKA-5523
> Project: Kafka
>  Issue Type: Improvement
>  Components: tools
>Reporter: Paolo Patierno
>Assignee: Manikumar
>Priority: Minor
> Fix For: 2.0.0
>
>
> Hi,
> the ReplayLogProducer is using the latest Kafka producer but not the latest 
> Kafka consumer. Is this tool today deprecated ? I see that something like 
> that could be done using the MirrorMaker. [~ijuma] Does it make sense to 
> update the ReplayLogProducer to the latest Kafka consumer ?
> Thanks,
> Paolo



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-5940) kafka-delete-records.sh doesn't give any feedback when the JSON offset configuration file is invalid

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-5940.
--
Resolution: Fixed

Fixed in KAFKA-5919

> kafka-delete-records.sh doesn't give any feedback when the JSON offset 
> configuration file is invalid
> 
>
> Key: KAFKA-5940
> URL: https://issues.apache.org/jira/browse/KAFKA-5940
> Project: Kafka
>  Issue Type: Improvement
>  Components: tools
>Reporter: Jakub Scholz
>Assignee: Jakub Scholz
>Priority: Major
>
> When deleting records using {{bin/kafka-delete-records.sh}}, the user has to 
> pass a JSON file with the list of topics/partitions and the offset to which 
> the records should be deleted. However, currently when such file is invalid 
> the utility doesn't print any visible error:
> {code}
> $ bin/kafka-delete-records.sh --bootstrap-server localhost:9092 
> --offset-json-file offset.json
> Executing records delete operation
> Records delete operation completed:
> $
> {code}
> Instead, I would suggest that it throws an exception to make it clear that 
> the problem is the invalid JSON file:
> {code}
> $ bin/kafka-delete-records.sh --bootstrap-server localhost:9092 
> --offset-json-file offset.json
> Exception in thread "main" kafka.common.AdminCommandFailedException: Offset 
> json file doesn't contain valid JSON data.
>   at 
> kafka.admin.DeleteRecordsCommand$.parseOffsetJsonStringWithoutDedup(DeleteRecordsCommand.scala:54)
>   at 
> kafka.admin.DeleteRecordsCommand$.execute(DeleteRecordsCommand.scala:62)
>   at kafka.admin.DeleteRecordsCommand$.main(DeleteRecordsCommand.scala:37)
>   at kafka.admin.DeleteRecordsCommand.main(DeleteRecordsCommand.scala)
> $
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-6994) KafkaConsumer.poll throwing AuthorizationException timeout-dependent

[Edoardo Comar (JIRA)]

Edoardo Comar created KAFKA-6994:


 Summary: KafkaConsumer.poll throwing AuthorizationException 
timeout-dependent
 Key: KAFKA-6994
 URL: https://issues.apache.org/jira/browse/KAFKA-6994
 Project: Kafka
  Issue Type: Bug
Reporter: Edoardo Comar


With auto-topic creation enabled, when attempting to consume from a 
non-existent topic, the {{AuthorizationException}} may or may not be thrown 
from {{poll(timeout)}} depending on the {{timeout}} value.

The issue can be recreated modifying a test in {{AuthorizerIntegrationTest}} as 
below (see comment) to *not* add the needed acl and therefore expecting the 
test to fail.
 While the first {{poll}} call will always throw with a short timeout, the 
second {{poll}} will not throw with the short timeout.

{code:java}
  @Test
  def testCreatePermissionOnClusterToReadFromNonExistentTopic() {
testCreatePermissionNeededToReadFromNonExistentTopic("newTopic",
  Set(new Acl(userPrincipal, Allow, Acl.WildCardHost, Create)), 
  Cluster)
  }

  private def testCreatePermissionNeededToReadFromNonExistentTopic(newTopic: 
String, acls: Set[Acl], resType: ResourceType) {
val topicPartition = new TopicPartition(newTopic, 0)
val newTopicResource = new Resource(Topic, newTopic)
addAndVerifyAcls(Set(new Acl(userPrincipal, Allow, Acl.WildCardHost, 
Read)), newTopicResource)
addAndVerifyAcls(groupReadAcl(groupResource), groupResource)
this.consumers.head.assign(List(topicPartition).asJava)
try {
  this.consumers.head.poll(Duration.ofMillis(50L));
  Assert.fail("should have thrown Authorization Exception")
} catch {
  case e: TopicAuthorizationException =>
assertEquals(Collections.singleton(newTopic), e.unauthorizedTopics())
}

//val resource = if (resType == Topic) newTopicResource else 
Resource.ClusterResource
// addAndVerifyAcls(acls, resource)

// need to use a larger timeout in this subsequent poll else it may not 
cause topic auto-creation
// this can be verified by commenting the above addAndVerifyAcls line and 
expecting this test to fail
this.consumers.head.poll(Duration.ofMillis(50L));
  }

{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

How to initiate a KIP?

[Virgil Palanciuc]

Hi all,
Here
 it says that “anyone can initiate a KIP”, however the “Create KIP” link 
doesn’t work for me. How can I get the permissions to initiate one? (or if I 
can’t… what’s the process for contacting someone with a proposal?)

[cid:image001.png@01D3FCC3.EE0FEE00]


Regards,
Virgil.

[jira] [Created] (KAFKA-6993) Defective documentations for KStream/KTable methods

[Lee Dongjin (JIRA)]

Lee Dongjin created KAFKA-6993:
--

 Summary: Defective documentations for KStream/KTable methods
 Key: KAFKA-6993
 URL: https://issues.apache.org/jira/browse/KAFKA-6993
 Project: Kafka
  Issue Type: Bug
  Components: documentation, streams
Reporter: Lee Dongjin
Assignee: Lee Dongjin


Some KStream/KTable methods, mainly join methods, are providing same 
documentation for the default methods and overloaded methods:
 * KStream#join
 * KStream#leftJoin
 * KStream#outerJoin
 * KTable#filter
 * KTable#filterNot
 * KTable#mapValues
 * KTable#transformValues
 * KTable#join
 * KTable#leftJoin
 * KTable#outerJoin

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-6762) log-cleaner thread terminates due to java.lang.IllegalStateException

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-6762.

Resolution: Duplicate

Duplicate of KAFKA-6854.

> log-cleaner thread terminates due to java.lang.IllegalStateException
> 
>
> Key: KAFKA-6762
> URL: https://issues.apache.org/jira/browse/KAFKA-6762
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.0.0
> Environment: os: GNU/Linux 
> arch: x86_64 
> Kernel: 4.9.77 
> jvm: OpenJDK 1.8.0
>Reporter: Ricardo Bartolome
>Priority: Major
> Attachments: __consumer_offsets-9_.tar.xz
>
>
> We are experiencing some problems with kafka log-cleaner thread on Kafka 
> 1.0.0. We have planned to update this cluster to 1.1.0 by next week in order 
> to fix KAFKA-6683, but until then we can only confirm that it happens in 
> 1.0.0.
> log-cleaner thread crashes after a while with the following error:
> {code:java}
> [2018-03-28 11:14:40,199] INFO Cleaner 0: Beginning cleaning of log 
> __consumer_offsets-31. (kafka.log.LogCleaner)
> [2018-03-28 11:14:40,199] INFO Cleaner 0: Building offset map for 
> __consumer_offsets-31... (kafka.log.LogCleaner)
> [2018-03-28 11:14:40,218] INFO Cleaner 0: Building offset map for log 
> __consumer_offsets-31 for 16 segments in offset range [1612869, 14282934). 
> (kafka.log.LogCleaner)
> [2018-03-28 11:14:58,566] INFO Cleaner 0: Offset map for log 
> __consumer_offsets-31 complete. (kafka.log.LogCleaner)
> [2018-03-28 11:14:58,566] INFO Cleaner 0: Cleaning log __consumer_offsets-31 
> (cleaning prior to Tue Mar 27 09:25:09 GMT 2018, discarding tombstones prior 
> to Sat Feb 24 11:04:21 GMT 2018
> )... (kafka.log.LogCleaner)
> [2018-03-28 11:14:58,567] INFO Cleaner 0: Cleaning segment 0 in log 
> __consumer_offsets-31 (largest timestamp Fri Feb 23 11:40:54 GMT 2018) into 
> 0, discarding deletes. (kafka.log.LogClea
> ner)
> [2018-03-28 11:14:58,570] INFO Cleaner 0: Growing cleaner I/O buffers from 
> 262144bytes to 524288 bytes. (kafka.log.LogCleaner)
> [2018-03-28 11:14:58,576] INFO Cleaner 0: Growing cleaner I/O buffers from 
> 524288bytes to 112 bytes. (kafka.log.LogCleaner)
> [2018-03-28 11:14:58,593] ERROR [kafka-log-cleaner-thread-0]: Error due to 
> (kafka.log.LogCleaner)
> java.lang.IllegalStateException: This log contains a message larger than 
> maximum allowable size of 112.
> at kafka.log.Cleaner.growBuffers(LogCleaner.scala:622)
> at kafka.log.Cleaner.cleanInto(LogCleaner.scala:574)
> at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:459)
> at kafka.log.Cleaner.$anonfun$doClean$6(LogCleaner.scala:396)
> at kafka.log.Cleaner.$anonfun$doClean$6$adapted(LogCleaner.scala:395)
> at scala.collection.immutable.List.foreach(List.scala:389)
> at kafka.log.Cleaner.doClean(LogCleaner.scala:395)
> at kafka.log.Cleaner.clean(LogCleaner.scala:372)
> at 
> kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:263)
> at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:243)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:64)
> [2018-03-28 11:14:58,601] INFO [kafka-log-cleaner-thread-0]: Stopped 
> (kafka.log.LogCleaner)
> [2018-04-04 14:25:12,773] INFO The cleaning for partition 
> __broker-11-health-check-0 is aborted and paused (kafka.log.LogCleaner)
> [2018-04-04 14:25:12,773] INFO Compaction for partition 
> __broker-11-health-check-0 is resumed (kafka.log.LogCleaner)
> [2018-04-04 14:25:12,774] INFO The cleaning for partition 
> __broker-11-health-check-0 is aborted (kafka.log.LogCleaner)
> [2018-04-04 14:25:22,850] INFO Shutting down the log cleaner. 
> (kafka.log.LogCleaner)
> [2018-04-04 14:25:22,850] INFO [kafka-log-cleaner-thread-0]: Shutting down 
> (kafka.log.LogCleaner)
> [2018-04-04 14:25:22,850] INFO [kafka-log-cleaner-thread-0]: Shutdown 
> completed (kafka.log.LogCleaner)
> {code}
> What we know so far is:
>  * We are unable to reproduce it yet in a consistent manner.
>  * It only happens in the PRO cluster and not in the PRE cluster for the same 
> customer (which message payloads are very similar)
>  * Checking our Kafka logs, it only happened on the internal topics 
> *__consumer_offsets-**
>  * When we restart the broker process the log-cleaner starts working again 
> but it can take between 3 minutes and some hours to die again.
>  * We workaround it by temporary increasing the message.max.bytes and 
> replica.fetch.max.bytes values to 10485760 (10MB) from default 112 (~1MB).
> ** Before message.max.bytes = 10MB, we tried to match message.max.size with 
> the value of replica.fetch.max.size (1048576), but log-cleaned died with the 
> same error but different limit.
>  ** This