date:20170321

[GitHub] kafka pull request #2048: Pluggable verifiable clients

2017-03-21 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2048


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [VOTE] KIP-122: Add Reset Consumer Group Offsets tooling

2017-03-21 Thread Dong Lin

Thanks for the KIP!

+1 (non-binding)

On Tue, Mar 21, 2017 at 6:24 PM, Becket Qin  wrote:

> +1
>
> Thanks for the KIP. The tool is very useful.
>
> On Tue, Mar 21, 2017 at 4:46 PM, Jason Gustafson 
> wrote:
>
> > +1 This looks super useful! Might be worth mentioning somewhere
> > compatibility with the old consumer. It looks like offsets in zk are not
> > covered, which seems fine, but probably should be explicitly noted. Maybe
> > you can also add a note saying that the tool can be used for old
> consumers
> > which have offsets stored in Kafka, but it will not protect against an
> > active consumer group in that case?
> >
> > Thanks,
> > Jason
> >
> > On Tue, Mar 14, 2017 at 10:13 AM, Dong Lin  wrote:
> >
> > > +1 (non-binding)
> > >
> > > On Tue, Mar 14, 2017 at 8:53 AM, Bill Bejeck 
> wrote:
> > >
> > > > +1
> > > >
> > > > On Tue, Mar 14, 2017 at 11:50 AM, Grant Henke 
> > > wrote:
> > > >
> > > > > +1. Agreed. This is a great tool to have.
> > > > >
> > > > > On Tue, Mar 14, 2017 at 12:33 AM, Gwen Shapira 
> > > > wrote:
> > > > >
> > > > > > +1 (binding)
> > > > > >
> > > > > > Nice job - this is going to be super useful.
> > > > > >
> > > > > > On Thu, Feb 23, 2017 at 4:46 PM, Jorge Esteban Quilcate Otoya <
> > > > > > quilcate.jo...@gmail.com> wrote:
> > > > > >
> > > > > > > Hi All,
> > > > > > >
> > > > > > > It seems that there is no further concern with the KIP-122.
> > > > > > > At this point we would like to start the voting process.
> > > > > > >
> > > > > > > The KIP can be found here:
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > > > 122%3A+Add+Reset+Consumer+Group+Offsets+tooling
> > > > > > >
> > > > > > >
> > > > > > > Thanks!
> > > > > > >
> > > > > > > Jorge.
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > *Gwen Shapira*
> > > > > > Product Manager | Confluent
> > > > > > 650.450.2760 | @gwenshap
> > > > > > Follow us: Twitter  | blog
> > > > > > 
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Grant Henke
> > > > > Software Engineer | Cloudera
> > > > > gr...@cloudera.com | twitter.com/gchenke |
> > linkedin.com/in/granthenke
> > > > >
> > > >
> > >
> >
>

Build failed in Jenkins: kafka-0.10.2-jdk7 #108

2017-03-21 Thread Apache Jenkins Server

See 


Changes:

[wangguoz] MINOR: updated incorrect JavaDocs for joins

[wangguoz] MINOR: improve hanging ResetIntegrationTest

--
[...truncated 616.85 KB...]

org.apache.kafka.clients.producer.KafkaProducerTest > 
testInterceptorConstructClose PASSED

org.apache.kafka.clients.producer.KafkaProducerTest > testPartitionerClose 
STARTED

org.apache.kafka.clients.producer.KafkaProducerTest > testPartitionerClose 
PASSED

org.apache.kafka.clients.producer.KafkaProducerTest > testMetadataFetch STARTED

org.apache.kafka.clients.producer.KafkaProducerTest > testMetadataFetch PASSED

org.apache.kafka.clients.producer.KafkaProducerTest > 
testMetadataFetchOnStaleMetadata STARTED

org.apache.kafka.clients.producer.KafkaProducerTest > 
testMetadataFetchOnStaleMetadata PASSED

org.apache.kafka.clients.producer.ProducerRecordTest > testInvalidRecords 
STARTED

org.apache.kafka.clients.producer.ProducerRecordTest > testInvalidRecords PASSED

org.apache.kafka.clients.producer.ProducerRecordTest > testEqualsAndHashCode 
STARTED

org.apache.kafka.clients.producer.ProducerRecordTest > testEqualsAndHashCode 
PASSED

org.apache.kafka.clients.ClientUtilsTest > testOnlyBadHostname STARTED

org.apache.kafka.clients.ClientUtilsTest > testOnlyBadHostname PASSED

org.apache.kafka.clients.ClientUtilsTest > testParseAndValidateAddresses STARTED

org.apache.kafka.clients.ClientUtilsTest > testParseAndValidateAddresses PASSED

org.apache.kafka.clients.ClientUtilsTest > testNoPort STARTED

org.apache.kafka.clients.ClientUtilsTest > testNoPort PASSED

org.apache.kafka.clients.NodeApiVersionsTest > 
testUsableVersionCalculationNoKnownVersions STARTED

org.apache.kafka.clients.NodeApiVersionsTest > 
testUsableVersionCalculationNoKnownVersions PASSED

org.apache.kafka.clients.NodeApiVersionsTest > testVersionsToString STARTED

org.apache.kafka.clients.NodeApiVersionsTest > testVersionsToString PASSED

org.apache.kafka.clients.NodeApiVersionsTest > testUnsupportedVersionsToString 
STARTED

org.apache.kafka.clients.NodeApiVersionsTest > testUnsupportedVersionsToString 
PASSED

org.apache.kafka.clients.NodeApiVersionsTest > testUnknownApiVersionsToString 
STARTED

org.apache.kafka.clients.NodeApiVersionsTest > testUnknownApiVersionsToString 
PASSED

org.apache.kafka.clients.NodeApiVersionsTest > testUsableVersionCalculation 
STARTED

org.apache.kafka.clients.NodeApiVersionsTest > testUsableVersionCalculation 
PASSED

org.apache.kafka.clients.NodeApiVersionsTest > testUsableVersionLatestVersions 
STARTED

org.apache.kafka.clients.NodeApiVersionsTest > testUsableVersionLatestVersions 
PASSED

org.apache.kafka.clients.NetworkClientTest > testSimpleRequestResponse STARTED

org.apache.kafka.clients.NetworkClientTest > testSimpleRequestResponse PASSED

org.apache.kafka.clients.NetworkClientTest > 
testDisconnectDuringUserMetadataRequest STARTED

org.apache.kafka.clients.NetworkClientTest > 
testDisconnectDuringUserMetadataRequest PASSED

org.apache.kafka.clients.NetworkClientTest > testClose STARTED

org.apache.kafka.clients.NetworkClientTest > testClose PASSED

org.apache.kafka.clients.NetworkClientTest > testLeastLoadedNode STARTED

org.apache.kafka.clients.NetworkClientTest > testLeastLoadedNode PASSED

org.apache.kafka.clients.NetworkClientTest > testConnectionDelayConnected 
STARTED

org.apache.kafka.clients.NetworkClientTest > testConnectionDelayConnected PASSED

org.apache.kafka.clients.NetworkClientTest > testRequestTimeout STARTED

org.apache.kafka.clients.NetworkClientTest > testRequestTimeout PASSED

org.apache.kafka.clients.NetworkClientTest > 
testSimpleRequestResponseWithNoBrokerDiscovery STARTED

org.apache.kafka.clients.NetworkClientTest > 
testSimpleRequestResponseWithNoBrokerDiscovery PASSED

org.apache.kafka.clients.NetworkClientTest > 
testSimpleRequestResponseWithStaticNodes STARTED

org.apache.kafka.clients.NetworkClientTest > 
testSimpleRequestResponseWithStaticNodes PASSED

org.apache.kafka.clients.NetworkClientTest > testConnectionDelay STARTED

org.apache.kafka.clients.NetworkClientTest > testConnectionDelay PASSED

org.apache.kafka.clients.NetworkClientTest > testSendToUnreadyNode STARTED

org.apache.kafka.clients.NetworkClientTest > testSendToUnreadyNode PASSED

org.apache.kafka.clients.NetworkClientTest > testConnectionDelayDisconnected 
STARTED

org.apache.kafka.clients.NetworkClientTest > testConnectionDelayDisconnected 
PASSED
:clients:determineCommitId UP-TO-DATE
:clients:createVersionFile
:clients:jar UP-TO-DATE
:core:compileJava UP-TO-DATE
:core:compileScala
:79:
 value DEFAULT_TIMESTAMP in object OffsetCommitRequest is deprecated: see 
corresponding Javadoc for more information.

org.apache.kafka.common.requests.OffsetCommitRequest.DEFAULT_TIMESTAMP

[jira] [Commented] (KAFKA-4848) Stream thread getting into deadlock state while trying to get rocksdb lock in retryWithBackoff

2017-03-21 Thread Sachin Mittal (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-4848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15935744#comment-15935744
 ] 

Sachin Mittal commented on KAFKA-4848:
--

[~guozhang] We ran only once instance of streams application per machine. So if 
two threads would access same partition on the same machine then this issue 
would arise.

I am not sure what would happen with different threads of different instances 
(processes) on same machine would try to get lock of same partition.


> Stream thread getting into deadlock state while trying to get rocksdb lock in 
> retryWithBackoff
> --
>
> Key: KAFKA-4848
> URL: https://issues.apache.org/jira/browse/KAFKA-4848
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Sachin Mittal
>Assignee: Sachin Mittal
> Fix For: 0.11.0.0
>
> Attachments: thr-1
>
>
> We see a deadlock state when streams thread to process a task takes longer 
> than MAX_POLL_INTERVAL_MS_CONFIG time. In this case this threads partitions 
> are assigned to some other thread including rocksdb lock. When it tries to 
> process the next task it cannot get rocks db lock and simply keeps waiting 
> for that lock forever.
> in retryWithBackoff for AbstractTaskCreator we have a backoffTimeMs = 50L.
> If it does not get lock the we simply increase the time by 10x and keep 
> trying inside the while true loop.
> We need to have a upper bound for this backoffTimeM. If the time is greater 
> than  MAX_POLL_INTERVAL_MS_CONFIG and it still hasn't got the lock means this 
> thread's partitions are moved somewhere else and it may not get the lock 
> again.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: kafka-trunk-jdk8 #1365

2017-03-21 Thread Apache Jenkins Server

See 


Changes:

[wangguoz] MINOR: remove unused log field from KStreamTransformValuesProcessor

[wangguoz] MINOR: updated incorrect JavaDocs for joins

[me] KAFKA-4924: Fix Kafka Connect API findbugs warnings

--
[...truncated 938.12 KB...]
org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorTest > 
testJoinLeaderCannotAssign PASSED

org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorTest > 
testRejoinGroup STARTED

org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorTest > 
testRejoinGroup PASSED

org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorTest > 
testNormalJoinGroupLeader STARTED

org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorTest > 
testNormalJoinGroupLeader PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testPutConnectorConfig STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testPutConnectorConfig PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testCreateConnector STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testCreateConnector PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testAccessors STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testAccessors PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testJoinAssignment STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testJoinAssignment PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRebalance STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRebalance PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRebalanceFailedConnector STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRebalanceFailedConnector PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testHaltCleansUpWorker STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testHaltCleansUpWorker PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorNameConflictsWithWorkerGroupId STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorNameConflictsWithWorkerGroupId PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRestartUnknownConnector STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRestartUnknownConnector PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRestartConnectorRedirectToLeader STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRestartConnectorRedirectToLeader PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRestartConnectorRedirectToOwner STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRestartConnectorRedirectToOwner PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRestartUnknownTask STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRestartUnknownTask PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRequestProcessingOrder STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRequestProcessingOrder PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRestartTaskRedirectToLeader STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRestartTaskRedirectToLeader PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRestartTaskRedirectToOwner STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRestartTaskRedirectToOwner PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorConfigAdded STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorConfigAdded PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorConfigUpdate STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorConfigUpdate PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorPaused STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorPaused PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorResumed STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorResumed PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testUnknownConnectorPaused STARTED

org.apache.kafka.connect.runtime.distributed.DistributedHe

[jira] [Commented] (KAFKA-4890) State directory being deleted when another thread holds the lock

2017-03-21 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-4890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15935739#comment-15935739
 ] 

ASF GitHub Bot commented on KAFKA-4890:
---

Github user guozhangwang closed the pull request at:

https://github.com/apache/kafka/pull/2681


> State directory being deleted when another thread holds the lock
> 
>
> Key: KAFKA-4890
> URL: https://issues.apache.org/jira/browse/KAFKA-4890
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Damian Guy
> Attachments: logs2.tar.gz, logs3.tar.gz, logs.tar.gz
>
>
> Looks like a state directory is being cleaned up when another thread already 
> has the lock:
> {code}
> 2017-03-12 20:39:17 [StreamThread-1] DEBUG o.a.k.s.p.i.ProcessorStateManager 
> - task [0_6] Registering state store perGameScoreStore to its state manager
> 2017-03-12 20:40:21 [StreamThread-3] INFO  o.a.k.s.p.i.StateDirectory - 
> Deleting obsolete state directory 0_6 for task 0_6
> 2017-03-12 20:40:22 [StreamThread-1] ERROR o.a.k.c.c.i.ConsumerCoordinator - 
> User provided listener 
> org.apache.kafka.streams.processor.internals.StreamThread$1 for group 
> fireflyProd failed on partition assignment
> org.apache.kafka.streams.errors.ProcessorStateException: Error while 
> executing put key 
> \x00\x00\x00\x00}\xA2\x9E\x9D\x05\xF6\x95\xAB\x01\x12dayOfGame and value 
> \x00\x00\x00\x00z\x00\x00\x00\x00\x00\x80G@ from store perGameScoreStore
> at 
> org.apache.kafka.streams.state.internals.RocksDBStore.putInternal(RocksDBStore.java:248)
> at 
> org.apache.kafka.streams.state.internals.RocksDBStore.access$000(RocksDBStore.java:65)
> at 
> org.apache.kafka.streams.state.internals.RocksDBStore$1.restore(RocksDBStore.java:156)
> at 
> org.apache.kafka.streams.processor.internals.ProcessorStateManager.restoreActiveState(ProcessorStateManager.java:230)
> at 
> org.apache.kafka.streams.processor.internals.ProcessorStateManager.register(ProcessorStateManager.java:193)
> at 
> org.apache.kafka.streams.processor.internals.AbstractProcessorContext.register(AbstractProcessorContext.java:99)
> at 
> org.apache.kafka.streams.state.internals.RocksDBStore.init(RocksDBStore.java:152)
> at 
> org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStore.init(ChangeLoggingKeyValueBytesStore.java:39)
> at 
> org.apache.kafka.streams.state.internals.MeteredKeyValueStore$7.run(MeteredKeyValueStore.java:100)
> at 
> org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measureLatencyNs(StreamsMetricsImpl.java:188)
> at 
> org.apache.kafka.streams.state.internals.MeteredKeyValueStore.init(MeteredKeyValueStore.java:131)
> at 
> org.apache.kafka.streams.state.internals.CachingKeyValueStore.init(CachingKeyValueStore.java:62)
> at 
> org.apache.kafka.streams.processor.internals.AbstractTask.initializeStateStores(AbstractTask.java:86)
> at 
> org.apache.kafka.streams.processor.internals.StreamTask.(StreamTask.java:141)
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.createStreamTask(StreamThread.java:834)
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$TaskCreator.createTask(StreamThread.java:1207)
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$AbstractTaskCreator.retryWithBackoff(StreamThread.java:1180)
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.addStreamTasks(StreamThread.java:937)
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.access$500(StreamThread.java:69)
>  at 
> org.apache.kafka.streams.processor.internals.StreamThread$1.onPartitionsAssigned(StreamThread.java:236)
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:255)
> at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:339)
> at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:303)
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:286)
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1030)
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:995)
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:582)
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:368)
> Caused by: org.rocksdb.RocksDBException: `
> at org.rocksdb.RocksDB.put(Native Method)
> at org.

[GitHub] kafka pull request #2681: KAFKA-4890: Improve log4j for debugging [WIP]

2017-03-21 Thread guozhangwang

Github user guozhangwang closed the pull request at:

https://github.com/apache/kafka/pull/2681


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-4890) State directory being deleted when another thread holds the lock

2017-03-21 Thread Guozhang Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-4890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15935729#comment-15935729
 ] 

Guozhang Wang commented on KAFKA-4890:
--

[~yunolgun] Thanks for the updates. I tried to reproduce the above sequential 
unit test on a MacOS but it seems `FileLock` does prevent threads within the 
same JVM to simultaneously have locks on the same file, instead a 
`OverlappingFileLockException` will be thrown.

Googling more about `FileLocks` reveals that `FileLock` impl is OS-system 
dependent and also Java-version dependent. That is to say, it might not be 100% 
safe to rely on `FileLock` on all operating systems (btw which Java version are 
you using?)

Given this I will leave this JIRA open as is until we have more clues on how to 
reproduce this issue on a specific environment.

> State directory being deleted when another thread holds the lock
> 
>
> Key: KAFKA-4890
> URL: https://issues.apache.org/jira/browse/KAFKA-4890
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Damian Guy
> Attachments: logs2.tar.gz, logs3.tar.gz, logs.tar.gz
>
>
> Looks like a state directory is being cleaned up when another thread already 
> has the lock:
> {code}
> 2017-03-12 20:39:17 [StreamThread-1] DEBUG o.a.k.s.p.i.ProcessorStateManager 
> - task [0_6] Registering state store perGameScoreStore to its state manager
> 2017-03-12 20:40:21 [StreamThread-3] INFO  o.a.k.s.p.i.StateDirectory - 
> Deleting obsolete state directory 0_6 for task 0_6
> 2017-03-12 20:40:22 [StreamThread-1] ERROR o.a.k.c.c.i.ConsumerCoordinator - 
> User provided listener 
> org.apache.kafka.streams.processor.internals.StreamThread$1 for group 
> fireflyProd failed on partition assignment
> org.apache.kafka.streams.errors.ProcessorStateException: Error while 
> executing put key 
> \x00\x00\x00\x00}\xA2\x9E\x9D\x05\xF6\x95\xAB\x01\x12dayOfGame and value 
> \x00\x00\x00\x00z\x00\x00\x00\x00\x00\x80G@ from store perGameScoreStore
> at 
> org.apache.kafka.streams.state.internals.RocksDBStore.putInternal(RocksDBStore.java:248)
> at 
> org.apache.kafka.streams.state.internals.RocksDBStore.access$000(RocksDBStore.java:65)
> at 
> org.apache.kafka.streams.state.internals.RocksDBStore$1.restore(RocksDBStore.java:156)
> at 
> org.apache.kafka.streams.processor.internals.ProcessorStateManager.restoreActiveState(ProcessorStateManager.java:230)
> at 
> org.apache.kafka.streams.processor.internals.ProcessorStateManager.register(ProcessorStateManager.java:193)
> at 
> org.apache.kafka.streams.processor.internals.AbstractProcessorContext.register(AbstractProcessorContext.java:99)
> at 
> org.apache.kafka.streams.state.internals.RocksDBStore.init(RocksDBStore.java:152)
> at 
> org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStore.init(ChangeLoggingKeyValueBytesStore.java:39)
> at 
> org.apache.kafka.streams.state.internals.MeteredKeyValueStore$7.run(MeteredKeyValueStore.java:100)
> at 
> org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measureLatencyNs(StreamsMetricsImpl.java:188)
> at 
> org.apache.kafka.streams.state.internals.MeteredKeyValueStore.init(MeteredKeyValueStore.java:131)
> at 
> org.apache.kafka.streams.state.internals.CachingKeyValueStore.init(CachingKeyValueStore.java:62)
> at 
> org.apache.kafka.streams.processor.internals.AbstractTask.initializeStateStores(AbstractTask.java:86)
> at 
> org.apache.kafka.streams.processor.internals.StreamTask.(StreamTask.java:141)
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.createStreamTask(StreamThread.java:834)
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$TaskCreator.createTask(StreamThread.java:1207)
> at 
> org.apache.kafka.streams.processor.internals.StreamThread$AbstractTaskCreator.retryWithBackoff(StreamThread.java:1180)
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.addStreamTasks(StreamThread.java:937)
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.access$500(StreamThread.java:69)
>  at 
> org.apache.kafka.streams.processor.internals.StreamThread$1.onPartitionsAssigned(StreamThread.java:236)
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:255)
> at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:339)
> at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:303)
> at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinato

[jira] [Commented] (KAFKA-4870) A question about broker down , the server is doing partition master election,the client producer may send msg fail . How the producer deal with the situation ??

2017-03-21 Thread zhaoziyan (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-4870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15935727#comment-15935727
 ] 

zhaoziyan commented on KAFKA-4870:
--

thanks 

> A question about broker down , the server is doing partition master 
> election,the client producer may send msg fail . How the producer deal with 
> the situation ??
> 
>
> Key: KAFKA-4870
> URL: https://issues.apache.org/jira/browse/KAFKA-4870
> Project: Kafka
>  Issue Type: Test
>  Components: clients
> Environment: java client 
>Reporter: zhaoziyan
>Priority: Minor
>
> the broker down . The kafka cluster is doing partion  master election , the 
> producer send order msg or nomal msg ,the producer may send msg fail .How 
> client update metadata and deal with the msg send fail ?? 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-4919) Streams job fails with InvalidStateStoreException: Store is currently closed

2017-03-21 Thread Elias Levy (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-4919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15935711#comment-15935711
 ] 

Elias Levy commented on KAFKA-4919:
---

I believe I found the issue.  I am closing the store when my {{Processor}} is 
closed.  The {{Processor}} is closed during rebalance via 
{{onPartitionsRevoked}} -> {{suspendTasksAndStat}} -> 
{{closeAllTasksTopologies}} -> {{performOnAllTasks}} -> {{closeTopology}} -> 
{{ProcessorNode.close}}.

It was my understanding that it was proper to close state stores used by a 
{{Processor}} when the {{Processor}} is closed. The examples in the low-level 
API documentation follow this pattern (See 
[https://kafka.apache.org/0102/documentation/streams#streams_processor] and 
[https://github.com/apache/kafka/blob/trunk/streams/examples/src/main/java/org/apache/kafka/streams/examples/wordcount/WordCountProcessorDemo.java]).

Has this changed in 0.10.2.0?  Or is this a new issue only affecting the 
refactored segmented stores?

> Streams job fails with InvalidStateStoreException: Store is currently closed
> 
>
> Key: KAFKA-4919
> URL: https://issues.apache.org/jira/browse/KAFKA-4919
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Elias Levy
>
> I have a streams job, that previously worked, that consumes and writes to a 
> large number of topics with many partitions and that uses many threads.  I 
> upgraded the job to 0.10.2.0.  The job now fails after a short time running, 
> seemingly after a rebalance.
> {quote}
> WARN  2017-03-19 18:03:20,432 [StreamThread-10][StreamThread.java:160] : 
> Unexpected state transition from RUNNING to NOT_RUNNING
> {quote}
> The first observation is that Streams is no longer outputting exceptions and 
> backtraces.  I had to add code to get this information.
> The exception:
> {quote}
> Exception: org.apache.kafka.streams.errors.StreamsException: Exception caught 
> in process. taskId=1_225, processor=KSTREAM-SOURCE-03, 
> topic=some_topic, partition=225, offset=266411
> org.apache.kafka.streams.errors.StreamsException: Exception caught in 
> process. taskId=1_225, processor=KSTREAM-SOURCE-03, topic=some_topic, 
> partition=225, offset=266411
>   at 
> org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:216)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:641)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:368)
> Caused by: org.apache.kafka.streams.errors.InvalidStateStoreException: Store 
> someStore-201701060400 is currently closed
>   at 
> org.apache.kafka.streams.state.internals.RocksDBStore.validateStoreOpen(RocksDBStore.java:205)
>   at 
> org.apache.kafka.streams.state.internals.RocksDBStore.put(RocksDBStore.java:221)
>   at 
> org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStore.put(RocksDBSegmentedBytesStore.java:74)
>   at 
> org.apache.kafka.streams.state.internals.ChangeLoggingSegmentedBytesStore.put(ChangeLoggingSegmentedBytesStore.java:54)
>   at 
> org.apache.kafka.streams.state.internals.MeteredSegmentedBytesStore.put(MeteredSegmentedBytesStore.java:101)
>   at 
> org.apache.kafka.streams.state.internals.RocksDBWindowStore.put(RocksDBWindowStore.java:109)
>   ... X more
> {quote}
> The error occurs for many partitions.
> This was preceded by (for this partition):
> {quote}
> INFO  2017-03-19 18:03:16,403 [StreamThread-10][ConsumerCoordinator.java:393] 
> : Revoking previously assigned partitions [some_topic-225] for group some_job
> INFO  2017-03-19 18:03:16,403 [StreamThread-10][StreamThread.java:254] : 
> stream-thread [StreamThread-10] partitions [[some_topic-225]] revoked at the 
> beginning of consumer rebalance.
> INFO  2017-03-19 18:03:16,403 [StreamThread-10][StreamThread.java:1056] : 
> stream-thread [StreamThread-10] Closing a task's topology 1_225
> INFO  2017-03-19 18:03:17,887 [StreamThread-10][StreamThread.java:544] : 
> stream-thread [StreamThread-10] Flushing state stores of task 1_225
> INFO  2017-03-19 18:03:17,887 [StreamThread-10][StreamThread.java:534] : 
> stream-thread [StreamThread-10] Committing consumer offsets of task 1_225
> INFO  2017-03-19 18:03:17,891 [StreamThread-10][StreamThread.java:1012] : 
> stream-thread [StreamThread-10] Updating suspended tasks to contain active 
> tasks [[1_225, 0_445, 0_30]]
> INFO  2017-03-19 18:03:17,891 [StreamThread-10][StreamThread.java:1019] : 
> stream-thread [StreamThread-10] Removing all active tasks [[1_225, 0_445, 
> 0_30]]
> INFO  2017-03-19 18:03:19,925 [StreamThread-10][ConsumerCoordinator.java:252] 
> : Setting newly assigned partitions [some_tpoic-225] for group some_j

[jira] [Commented] (KAFKA-4848) Stream thread getting into deadlock state while trying to get rocksdb lock in retryWithBackoff

2017-03-21 Thread Guozhang Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-4848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15935696#comment-15935696
 ] 

Guozhang Wang commented on KAFKA-4848:
--

[~sjmittal] Could I ask you a follow-up question: are these two threads within 
the same process, or are they from different processes that sits in the same 
machine?

> Stream thread getting into deadlock state while trying to get rocksdb lock in 
> retryWithBackoff
> --
>
> Key: KAFKA-4848
> URL: https://issues.apache.org/jira/browse/KAFKA-4848
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Sachin Mittal
>Assignee: Sachin Mittal
> Fix For: 0.11.0.0
>
> Attachments: thr-1
>
>
> We see a deadlock state when streams thread to process a task takes longer 
> than MAX_POLL_INTERVAL_MS_CONFIG time. In this case this threads partitions 
> are assigned to some other thread including rocksdb lock. When it tries to 
> process the next task it cannot get rocks db lock and simply keeps waiting 
> for that lock forever.
> in retryWithBackoff for AbstractTaskCreator we have a backoffTimeMs = 50L.
> If it does not get lock the we simply increase the time by 10x and keep 
> trying inside the while true loop.
> We need to have a upper bound for this backoffTimeM. If the time is greater 
> than  MAX_POLL_INTERVAL_MS_CONFIG and it still hasn't got the lock means this 
> thread's partitions are moved somewhere else and it may not get the lock 
> again.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-4912) Add check for topic name length

2017-03-21 Thread Sharad (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-4912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15935694#comment-15935694
 ] 

Sharad commented on KAFKA-4912:
---

[~mjsax] Topic name lengths are being checked in Topic.scala and Topic.java. It 
seems that topic validation is funneled through the classes.
In the related JIRA the error was being thrown at renaming the dir to a length 
greater than 249 chars.
Not sure what needs to be done here. Please advise.

> Add check for topic name length
> ---
>
> Key: KAFKA-4912
> URL: https://issues.apache.org/jira/browse/KAFKA-4912
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Matthias J. Sax
>Assignee: Sharad
>Priority: Minor
>  Labels: newbie
>
> We should check topic name length (if internal topics, and maybe for source 
> topics? -> in cause, {{topic.auto.create}} is enabled this might prevent 
> problems), and raise an exception if they are too long. Cf. KAFKA-4893



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #2630: MINOR: improve hanging ResetIntegrationTest

2017-03-21 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2630


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #2718: MINOR: log start and end offsets for state store r...

2017-03-21 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2718


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-4919) Streams job fails with InvalidStateStoreException: Store is currently closed

2017-03-21 Thread Guozhang Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-4919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15935663#comment-15935663
 ] 

Guozhang Wang commented on KAFKA-4919:
--

[~damianguy] Could you take a look?

> Streams job fails with InvalidStateStoreException: Store is currently closed
> 
>
> Key: KAFKA-4919
> URL: https://issues.apache.org/jira/browse/KAFKA-4919
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Elias Levy
>
> I have a streams job, that previously worked, that consumes and writes to a 
> large number of topics with many partitions and that uses many threads.  I 
> upgraded the job to 0.10.2.0.  The job now fails after a short time running, 
> seemingly after a rebalance.
> {quote}
> WARN  2017-03-19 18:03:20,432 [StreamThread-10][StreamThread.java:160] : 
> Unexpected state transition from RUNNING to NOT_RUNNING
> {quote}
> The first observation is that Streams is no longer outputting exceptions and 
> backtraces.  I had to add code to get this information.
> The exception:
> {quote}
> Exception: org.apache.kafka.streams.errors.StreamsException: Exception caught 
> in process. taskId=1_225, processor=KSTREAM-SOURCE-03, 
> topic=some_topic, partition=225, offset=266411
> org.apache.kafka.streams.errors.StreamsException: Exception caught in 
> process. taskId=1_225, processor=KSTREAM-SOURCE-03, topic=some_topic, 
> partition=225, offset=266411
>   at 
> org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:216)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:641)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:368)
> Caused by: org.apache.kafka.streams.errors.InvalidStateStoreException: Store 
> someStore-201701060400 is currently closed
>   at 
> org.apache.kafka.streams.state.internals.RocksDBStore.validateStoreOpen(RocksDBStore.java:205)
>   at 
> org.apache.kafka.streams.state.internals.RocksDBStore.put(RocksDBStore.java:221)
>   at 
> org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStore.put(RocksDBSegmentedBytesStore.java:74)
>   at 
> org.apache.kafka.streams.state.internals.ChangeLoggingSegmentedBytesStore.put(ChangeLoggingSegmentedBytesStore.java:54)
>   at 
> org.apache.kafka.streams.state.internals.MeteredSegmentedBytesStore.put(MeteredSegmentedBytesStore.java:101)
>   at 
> org.apache.kafka.streams.state.internals.RocksDBWindowStore.put(RocksDBWindowStore.java:109)
>   ... X more
> {quote}
> The error occurs for many partitions.
> This was preceded by (for this partition):
> {quote}
> INFO  2017-03-19 18:03:16,403 [StreamThread-10][ConsumerCoordinator.java:393] 
> : Revoking previously assigned partitions [some_topic-225] for group some_job
> INFO  2017-03-19 18:03:16,403 [StreamThread-10][StreamThread.java:254] : 
> stream-thread [StreamThread-10] partitions [[some_topic-225]] revoked at the 
> beginning of consumer rebalance.
> INFO  2017-03-19 18:03:16,403 [StreamThread-10][StreamThread.java:1056] : 
> stream-thread [StreamThread-10] Closing a task's topology 1_225
> INFO  2017-03-19 18:03:17,887 [StreamThread-10][StreamThread.java:544] : 
> stream-thread [StreamThread-10] Flushing state stores of task 1_225
> INFO  2017-03-19 18:03:17,887 [StreamThread-10][StreamThread.java:534] : 
> stream-thread [StreamThread-10] Committing consumer offsets of task 1_225
> INFO  2017-03-19 18:03:17,891 [StreamThread-10][StreamThread.java:1012] : 
> stream-thread [StreamThread-10] Updating suspended tasks to contain active 
> tasks [[1_225, 0_445, 0_30]]
> INFO  2017-03-19 18:03:17,891 [StreamThread-10][StreamThread.java:1019] : 
> stream-thread [StreamThread-10] Removing all active tasks [[1_225, 0_445, 
> 0_30]]
> INFO  2017-03-19 18:03:19,925 [StreamThread-10][ConsumerCoordinator.java:252] 
> : Setting newly assigned partitions [some_tpoic-225] for group some_job
> INFO  2017-03-19 18:03:19,927 [StreamThread-10][StreamThread.java:228] : 
> stream-thread [StreamThread-10] New partitions [[some_topic-225]] assigned at 
> the end of consumer rebalance.
> INFO  2017-03-19 18:03:19,929 [StreamThread-10][StreamTask.java:333] : task 
> [1_225] Initializing processor nodes of the topology
> Something happens.  What ???
> INFO  2017-03-19 18:03:20,135 [StreamThread-10][StreamThread.java:1045] : 
> stream-thread [StreamThread-10] Closing a task 1_225
> INFO  2017-03-19 18:03:20,355 [StreamThread-10][StreamThread.java:544] : 
> stream-thread [StreamThread-10] Flushing state stores of task 1_225
> INFO  2017-03-19 18:03:20,355 [StreamThread-10][StreamThread.java:523] : 
> stream-thread [StreamThread-10] Closing the state manager of task 1_225

Jenkins build is back to normal : kafka-trunk-jdk8 #1364

2017-03-21 Thread Apache Jenkins Server

See

[jira] [Resolved] (KAFKA-4779) Failure in kafka/tests/kafkatest/tests/core/security_rolling_upgrade_test.py

2017-03-21 Thread Ismael Juma (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-4779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-4779.

Resolution: Fixed

> Failure in kafka/tests/kafkatest/tests/core/security_rolling_upgrade_test.py
> 
>
> Key: KAFKA-4779
> URL: https://issues.apache.org/jira/browse/KAFKA-4779
> Project: Kafka
>  Issue Type: Bug
>Reporter: Apurva Mehta
>Assignee: Rajini Sivaram
> Fix For: 0.11.0.0, 0.10.2.1
>
>
> This test failed on 01/29, on both trunk and 0.10.2, error message:
> {noformat}
> The consumer has terminated, or timed out, on node ubuntu@worker3.
> Traceback (most recent call last):
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka-0.10.2/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.6.0-py2.7.egg/ducktape/tests/runner_client.py",
>  line 123, in run
> data = self.run_test()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka-0.10.2/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.6.0-py2.7.egg/ducktape/tests/runner_client.py",
>  line 176, in run_test
> return self.test_context.function(self.test)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka-0.10.2/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.6.0-py2.7.egg/ducktape/mark/_mark.py",
>  line 321, in wrapper
> return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka-0.10.2/kafka/tests/kafkatest/tests/core/security_rolling_upgrade_test.py",
>  line 148, in test_rolling_upgrade_phase_two
> self.run_produce_consume_validate(self.roll_in_secured_settings, 
> client_protocol, broker_protocol)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka-0.10.2/kafka/tests/kafkatest/tests/produce_consume_validate.py",
>  line 100, in run_produce_consume_validate
> self.stop_producer_and_consumer()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka-0.10.2/kafka/tests/kafkatest/tests/produce_consume_validate.py",
>  line 87, in stop_producer_and_consumer
> self.check_alive()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka-0.10.2/kafka/tests/kafkatest/tests/produce_consume_validate.py",
>  line 79, in check_alive
> raise Exception(msg)
> Exception: The consumer has terminated, or timed out, on node ubuntu@worker3.
> {noformat}
> Looks like the console consumer times out: 
> {noformat}
> [2017-01-30 04:56:00,972] ERROR Error processing message, terminating 
> consumer process:  (kafka.tools.ConsoleConsumer$)
> kafka.consumer.ConsumerTimeoutException
> at kafka.consumer.NewShinyConsumer.receive(BaseConsumer.scala:90)
> at kafka.tools.ConsoleConsumer$.process(ConsoleConsumer.scala:120)
> at kafka.tools.ConsoleConsumer$.run(ConsoleConsumer.scala:75)
> at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:50)
> at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala)
> {noformat}
> A bunch of these security_rolling_upgrade tests failed, and in all cases, the 
> producer produced ~15k messages, of which ~7k were acked, and the consumer 
> only got around ~2600 before timing out. 
> There are a lot of messages like the following for different request types on 
> the producer and consumer:
> {noformat}
> [2017-01-30 05:13:35,954] WARN Received unknown topic or partition error in 
> produce request on partition test_topic-0. The topic/partition may not exist 
> or the user may not have Describe access to it 
> (org.apache.kafka.clients.producer.internals.Sender)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [VOTE] KIP-122: Add Reset Consumer Group Offsets tooling

2017-03-21 Thread Becket Qin

+1

Thanks for the KIP. The tool is very useful.

On Tue, Mar 21, 2017 at 4:46 PM, Jason Gustafson  wrote:

> +1 This looks super useful! Might be worth mentioning somewhere
> compatibility with the old consumer. It looks like offsets in zk are not
> covered, which seems fine, but probably should be explicitly noted. Maybe
> you can also add a note saying that the tool can be used for old consumers
> which have offsets stored in Kafka, but it will not protect against an
> active consumer group in that case?
>
> Thanks,
> Jason
>
> On Tue, Mar 14, 2017 at 10:13 AM, Dong Lin  wrote:
>
> > +1 (non-binding)
> >
> > On Tue, Mar 14, 2017 at 8:53 AM, Bill Bejeck  wrote:
> >
> > > +1
> > >
> > > On Tue, Mar 14, 2017 at 11:50 AM, Grant Henke 
> > wrote:
> > >
> > > > +1. Agreed. This is a great tool to have.
> > > >
> > > > On Tue, Mar 14, 2017 at 12:33 AM, Gwen Shapira 
> > > wrote:
> > > >
> > > > > +1 (binding)
> > > > >
> > > > > Nice job - this is going to be super useful.
> > > > >
> > > > > On Thu, Feb 23, 2017 at 4:46 PM, Jorge Esteban Quilcate Otoya <
> > > > > quilcate.jo...@gmail.com> wrote:
> > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > It seems that there is no further concern with the KIP-122.
> > > > > > At this point we would like to start the voting process.
> > > > > >
> > > > > > The KIP can be found here:
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > > 122%3A+Add+Reset+Consumer+Group+Offsets+tooling
> > > > > >
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > Jorge.
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > *Gwen Shapira*
> > > > > Product Manager | Confluent
> > > > > 650.450.2760 | @gwenshap
> > > > > Follow us: Twitter  | blog
> > > > > 
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Grant Henke
> > > > Software Engineer | Cloudera
> > > > gr...@cloudera.com | twitter.com/gchenke |
> linkedin.com/in/granthenke
> > > >
> > >
> >
>

[jira] [Commented] (KAFKA-4816) Message format changes for idempotent/transactional producer

2017-03-21 Thread Apurva Mehta (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-4816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15935601#comment-15935601
 ] 

Apurva Mehta commented on KAFKA-4816:
-

I compiled the perf numbers for this patch, results available here: 
https://docs.google.com/spreadsheets/d/1dHY6M7qCiX-NFvsgvaE0YoVdNq26uA8608XIh_DUpI4/edit?usp=sharing

There is no observable regression on produce or consume at varying message 
sizes, with different workloads, on different machines.

We do see significant on disk savings at smaller message sizes.

Given the nature of the tests, I think the margin of error in throughput 
results is 10%. Results which show a difference of more than 10% should be 
treated with a grain of salt: they generally don't reproduce across runs and 
across machines.

We see consistent producer side performance improvements at small sizes, 
correlated with lower produce request size.

We see general consume side improvements, but there is no pattern and there is 
significant variation across runs. So we can't make any claims on improving the 
consume performance with this message format. But we can claim safely that 
there is no regression since all runs were generally faster with the new format.

> Message format changes for idempotent/transactional producer
> 
>
> Key: KAFKA-4816
> URL: https://issues.apache.org/jira/browse/KAFKA-4816
> Project: Kafka
>  Issue Type: Sub-task
>  Components: clients, core, producer 
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
> Fix For: 0.11.0.0
>
>
> This task is for the implementation of the message format changes documented 
> here: 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging#KIP-98-ExactlyOnceDeliveryandTransactionalMessaging-MessageFormat.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #2721: Minor: Adding example to SMT documentation

2017-03-21 Thread gwenshap

GitHub user gwenshap opened a pull request:

https://github.com/apache/kafka/pull/2721

Minor: Adding example to SMT documentation



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gwenshap/kafka improve_smt_docs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2721.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2721


commit f063a10528a2c492637402bd4713dab7cd9f9d8a
Author: Gwen Shapira 
Date:   2017-03-22T00:30:41Z

Minor: Adding example to SMT documentation




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (KAFKA-4917) Our built-in file connector can't work with our built-in SMT

2017-03-21 Thread Gwen Shapira (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-4917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gwen Shapira resolved KAFKA-4917.
-
Resolution: Won't Fix

Hoist is fine :)

> Our built-in file connector can't work with our built-in SMT
> 
>
> Key: KAFKA-4917
> URL: https://issues.apache.org/jira/browse/KAFKA-4917
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Reporter: Gwen Shapira
>Assignee: Manasvi Gupta
>  Labels: newbie
>
> Our built-in file connector always returns STRING schema.
> All our transformations expect either STRUCT (if connectors return schema) or 
> a MAP (schemaless). 
> I understand why (how do you add a field to a STRING?), but it also means 
> that you can't have an example for SMT that works with Apache Kafka out of 
> the box. Which makes documentation kind of painful.
> Either we modify the file connector or we modify the SMTs to deal with STRING 
> better, but something gotta change.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [VOTE] KIP-82 Add Record Headers

2017-03-21 Thread Jason Gustafson

Thanks for the KIP! +1 (binding) from me. Just one nit: can we change
`Headers.header(key)` to be `Headers.lastHeader(key)`? It's not a
deal-breaker, but I think it's better to let the name reflect the actual
behavior as clearly as possible.

-Jason

On Wed, Feb 15, 2017 at 6:10 AM, Jeroen van Disseldorp 
wrote:

> +1 on introducing the concept of headers, neutral on specific
> implementation.
>
>
>
> On 14/02/2017 22:34, Jay Kreps wrote:
>
>> Couple of things I think we still need to work out:
>>
>> 1. I think we agree about the key, but I think we haven't talked about
>> the value yet. I think if our goal is an open ecosystem of these
>> header
>> spread across many plugins from many systems we should consider
>> making this
>> a string as well so it can be printed, set via a UI, set in config,
>> etc.
>> Basically encouraging pluggable serialization formats here will lead
>> to a
>> bit of a tower of babel.
>> 2. This proposal still includes a pretty big change to our
>> serialization
>> and protocol definition layer. Essentially it is introducing an
>> optional
>> type, where the format is data dependent. I think this is actually a
>> big
>> change though it doesn't seem like it. It means you can no longer
>> specify
>> this type with our type definition DSL, and likewise it requires
>> custom
>> handling in client libs. This isn't a huge thing, since the Record
>> definition is custom anyway, but I think this kind of protocol
>> inconsistency is very non-desirable and ties you to hand-coding
>> things. I
>> think the type should instead by [Key Value] in our BNF, where key and
>> value are both short strings as used elsewhere. This brings it in
>> line with
>> the rest of the protocol.
>> 3. Could we get more specific about the exact Java API change to
>> ProducerRecord, ConsumerRecord, Record, etc?
>>
>> -Jay
>>
>> On Tue, Feb 14, 2017 at 9:42 AM, Michael Pearce 
>> wrote:
>>
>> Hi all,
>>>
>>> We would like to start the voting process for KIP-82 – Add record
>>> headers.
>>> The KIP can be found
>>> at
>>>
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>> 82+-+Add+Record+Headers
>>>
>>> Discussion thread(s) can be found here:
>>>
>>> http://search-hadoop.com/m/Kafka/uyzND1nSTOHTvj81?subj=
>>> Re+DISCUSS+KIP+82+Add+Record+Headers
>>> http://search-hadoop.com/m/Kafka/uyzND1Arxt22Tvj81?subj=
>>> Re+DISCUSS+KIP+82+Add+Record+Headers
>>> http://search-hadoop.com/?project=Kafka&q=KIP-82
>>>
>>>
>>>
>>> Thanks,
>>> Mike
>>>
>>> The information contained in this email is strictly confidential and for
>>> the use of the addressee only, unless otherwise indicated. If you are not
>>> the intended recipient, please do not read, copy, use or disclose to
>>> others
>>> this message or any attachment. Please also notify the sender by replying
>>> to this email or by telephone (+44(020 7896 0011) and then delete the
>>> email
>>> and any copies of it. Opinions, conclusion (etc) that do not relate to
>>> the
>>> official business of this company shall be understood as neither given
>>> nor
>>> endorsed by it. IG is a trading name of IG Markets Limited (a company
>>> registered in England and Wales, company number 04008957) and IG Index
>>> Limited (a company registered in England and Wales, company number
>>> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
>>> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
>>> Index Limited (register number 114059) are authorised and regulated by
>>> the
>>> Financial Conduct Authority.
>>>
>>>
>

Re: [DISCUSS] KIP-129: Kafka Streams Exactly-Once Semantics

2017-03-21 Thread Sriram Subramanian

To add to this discussion, I do think we should think about this in
increments. We do a full recovery today and the EOS proposal will not make
this any worse. Using store snapshot is a good option to avoid store
recovery in the future but as Eno points out, all pluggable stores would
need to have this ability. W.r.t transaction failures, this should not be
an issue. We should be simply retrying. There is one optimization we can do
for clean shutdowns. We could store a clean shutdown file that contains the
input offsets. This file gets written when you close the streams instance.
On start, you could can check the offsets from the shutdown file and
compare it with the offsets we get from the consumer and ensure they match.
If they do, you could use the same store instead of recovering. However, if
we go with the snapshot approach, this will not be required. My vote would
be to implement V1 and solve the bootstrap problem which exist today in the
future versions.

On Tue, Mar 21, 2017 at 4:43 PM, Matthias J. Sax 
wrote:

> Thanks for your feedback Eno.
>
> For now, I still think that the KIP itself does not need to talk about
> this in more detail, because we apply the same strategy for EoS as for
> non-EoS (as of 0.10.2).
>
> Thus, in case of a clean shutdown, we write the checkpoint file for a
> store and thus know we can reuse the store. In case of failure, we need
> to recreate the store from the changelog.
>
> > Will a V1 design that relies on plain store recovery from Kafka for
> > each transaction abort be good enough, or usable?
>
> Why should it not be usable? It's the same strategy as used in 0.10.2
> and it runs in production in many companies already.
>
> > however it seems to me we might have a regression of sorts
> > Now we might pay it for a transaction failure.
>
> I would assume transaction failures to be quite rare. Maybe the core EoS
> folks can comment here, too.
>
>
>
> -Matthias
>
>
>
> On 3/20/17 3:16 PM, Eno Thereska wrote:
> > Hi Matthias,
> >
> > I'd like to see some more info on how you propose to handle transactions
> that involve state stores in the KIP itself. The design doc has info about
> various optimisations like RocksDb snapshots and transactions and such, but
> will there be a user-visible interface that indicates whether a store has
> snapshot and/or transactional capabilities? If a user plugs in another
> store, what guarantees are they expected to get?
> >
> > Will a V1 design that relies on plain store recovery from Kafka for each
> transaction abort be good enough, or usable? If your dataset is large
> (e.g., 200GB) the recovery time might be so large as to effectively render
> that Kafka Streams instance unavailable for tens of minutes. You mention
> that is not a regression to what we currently have, however it seems to me
> we might have a regression of sorts: currently we pay the recovery price
> for a Kafka Streams instance failure. Now we might pay it for a transaction
> failure. Will transaction failures be more or less common than the previous
> types of failures? I'd like to see this addressed.
> >
> > Thanks
> > Eno
> >
> >
> >
> >> On 15 Mar 2017, at 22:09, Matthias J. Sax 
> wrote:
> >>
> >> Just a quick follow up:
> >>
> >> Our overall proposal is, to implement KIP-129 as is as a “Stream EoS
> >> 1.0” version. The raised concerns are all valid, but hard to quantify at
> >> the moment. Implementing KIP-129, that provides a clean design, allows
> >> us to gain more insight in the performance implications. This enables
> >> us, to make an educated decision, if the “producer per task” model
> >> perform wells or not, and if a switch to a “producer per thread” model
> >> is mandatory.
> >>
> >> We also want to point out, that we can move incrementally from "producer
> >> per task" to "producer per thread" design or apply some incremental
> >> improvements to "producer per task" (as discussed in the doc). Thus,
> >> there is not issue with regard to upgrading.
> >>
> >>
> >> -Matthias
> >>
> >>
> >>
> >> On 3/15/17 2:36 PM, Matthias J. Sax wrote:
> >>> Hi,
> >>>
> >>> I want to pick up this thread again. As there are some concerns about
> >>> the "producer per task" design, we did write up an alternative
> "producer
> >>> per thread" design and discuss pros/cons of both approaches:
> >>>
> >>> https://docs.google.com/document/d/1CfOJaa6mdg5o7pLf_
> zXISV4oE0ZeMZwT_sG1QWgL4EE
> >>>
> >>>
> >>> Looking forward to your feedback.
> >>>
> >>>
> >>> -Matthias
> >>>
> >>>
> >>> On 3/10/17 3:24 AM, Damian Guy wrote:
>  Hi Matthias,
> 
>  Thanks for the response. I agree with you regarding the use of
>  PartitionGrouper to reduce the number of tasks. It would be good to
> have an
>  idea of any additional load on the brokers as we increase the number
> of
>  tasks and therefore producers.
> 
>  Thanks,
>  Damian
> 
>  On Wed, 8 Mar 2017 at 01:45 Matthias J. Sax 
> wrote:
> 
> > Damian, Jun,
> >
> > Thanks for y

[jira] [Commented] (KAFKA-4924) Fix findbugs warnings in Kafka-Connect-API

2017-03-21 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-4924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15935559#comment-15935559
 ] 

ASF GitHub Bot commented on KAFKA-4924:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2715


> Fix findbugs warnings in Kafka-Connect-API
> --
>
> Key: KAFKA-4924
> URL: https://issues.apache.org/jira/browse/KAFKA-4924
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Reporter: Colin P. McCabe
>Assignee: Colin P. McCabe
> Fix For: 0.11.0.0
>
>
> Fix findbugs warnings in Kafka-Connect-API



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (KAFKA-4924) Fix findbugs warnings in Kafka-Connect-API

2017-03-21 Thread Ewen Cheslack-Postava (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-4924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava resolved KAFKA-4924.
--
   Resolution: Fixed
Fix Version/s: 0.11.0.0

Issue resolved by pull request 2715
[https://github.com/apache/kafka/pull/2715]

> Fix findbugs warnings in Kafka-Connect-API
> --
>
> Key: KAFKA-4924
> URL: https://issues.apache.org/jira/browse/KAFKA-4924
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Reporter: Colin P. McCabe
>Assignee: Colin P. McCabe
> Fix For: 0.11.0.0
>
>
> Fix findbugs warnings in Kafka-Connect-API



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #2715: KAFKA-4924: Fix Kafka Connect API findbugs warning...

2017-03-21 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2715


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [VOTE] KIP-122: Add Reset Consumer Group Offsets tooling

2017-03-21 Thread Jason Gustafson

+1 This looks super useful! Might be worth mentioning somewhere
compatibility with the old consumer. It looks like offsets in zk are not
covered, which seems fine, but probably should be explicitly noted. Maybe
you can also add a note saying that the tool can be used for old consumers
which have offsets stored in Kafka, but it will not protect against an
active consumer group in that case?

Thanks,
Jason

On Tue, Mar 14, 2017 at 10:13 AM, Dong Lin  wrote:

> +1 (non-binding)
>
> On Tue, Mar 14, 2017 at 8:53 AM, Bill Bejeck  wrote:
>
> > +1
> >
> > On Tue, Mar 14, 2017 at 11:50 AM, Grant Henke 
> wrote:
> >
> > > +1. Agreed. This is a great tool to have.
> > >
> > > On Tue, Mar 14, 2017 at 12:33 AM, Gwen Shapira 
> > wrote:
> > >
> > > > +1 (binding)
> > > >
> > > > Nice job - this is going to be super useful.
> > > >
> > > > On Thu, Feb 23, 2017 at 4:46 PM, Jorge Esteban Quilcate Otoya <
> > > > quilcate.jo...@gmail.com> wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > It seems that there is no further concern with the KIP-122.
> > > > > At this point we would like to start the voting process.
> > > > >
> > > > > The KIP can be found here:
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > 122%3A+Add+Reset+Consumer+Group+Offsets+tooling
> > > > >
> > > > >
> > > > > Thanks!
> > > > >
> > > > > Jorge.
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > *Gwen Shapira*
> > > > Product Manager | Confluent
> > > > 650.450.2760 | @gwenshap
> > > > Follow us: Twitter  | blog
> > > > 
> > > >
> > >
> > >
> > >
> > > --
> > > Grant Henke
> > > Software Engineer | Cloudera
> > > gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
> > >
> >
>

Re: [DISCUSS] KIP-129: Kafka Streams Exactly-Once Semantics

2017-03-21 Thread Matthias J. Sax

Thanks for your feedback Eno.

For now, I still think that the KIP itself does not need to talk about
this in more detail, because we apply the same strategy for EoS as for
non-EoS (as of 0.10.2).

Thus, in case of a clean shutdown, we write the checkpoint file for a
store and thus know we can reuse the store. In case of failure, we need
to recreate the store from the changelog.

> Will a V1 design that relies on plain store recovery from Kafka for
> each transaction abort be good enough, or usable?

Why should it not be usable? It's the same strategy as used in 0.10.2
and it runs in production in many companies already.

> however it seems to me we might have a regression of sorts
> Now we might pay it for a transaction failure.

I would assume transaction failures to be quite rare. Maybe the core EoS
folks can comment here, too.



-Matthias



On 3/20/17 3:16 PM, Eno Thereska wrote:
> Hi Matthias,
> 
> I'd like to see some more info on how you propose to handle transactions that 
> involve state stores in the KIP itself. The design doc has info about various 
> optimisations like RocksDb snapshots and transactions and such, but will 
> there be a user-visible interface that indicates whether a store has snapshot 
> and/or transactional capabilities? If a user plugs in another store, what 
> guarantees are they expected to get? 
> 
> Will a V1 design that relies on plain store recovery from Kafka for each 
> transaction abort be good enough, or usable? If your dataset is large (e.g., 
> 200GB) the recovery time might be so large as to effectively render that 
> Kafka Streams instance unavailable for tens of minutes. You mention that is 
> not a regression to what we currently have, however it seems to me we might 
> have a regression of sorts: currently we pay the recovery price for a Kafka 
> Streams instance failure. Now we might pay it for a transaction failure. Will 
> transaction failures be more or less common than the previous types of 
> failures? I'd like to see this addressed.
> 
> Thanks
> Eno
> 
> 
> 
>> On 15 Mar 2017, at 22:09, Matthias J. Sax  wrote:
>>
>> Just a quick follow up:
>>
>> Our overall proposal is, to implement KIP-129 as is as a “Stream EoS
>> 1.0” version. The raised concerns are all valid, but hard to quantify at
>> the moment. Implementing KIP-129, that provides a clean design, allows
>> us to gain more insight in the performance implications. This enables
>> us, to make an educated decision, if the “producer per task” model
>> perform wells or not, and if a switch to a “producer per thread” model
>> is mandatory.
>>
>> We also want to point out, that we can move incrementally from "producer
>> per task" to "producer per thread" design or apply some incremental
>> improvements to "producer per task" (as discussed in the doc). Thus,
>> there is not issue with regard to upgrading.
>>
>>
>> -Matthias
>>
>>
>>
>> On 3/15/17 2:36 PM, Matthias J. Sax wrote:
>>> Hi,
>>>
>>> I want to pick up this thread again. As there are some concerns about
>>> the "producer per task" design, we did write up an alternative "producer
>>> per thread" design and discuss pros/cons of both approaches:
>>>
>>> https://docs.google.com/document/d/1CfOJaa6mdg5o7pLf_zXISV4oE0ZeMZwT_sG1QWgL4EE
>>>
>>>
>>> Looking forward to your feedback.
>>>
>>>
>>> -Matthias
>>>
>>>
>>> On 3/10/17 3:24 AM, Damian Guy wrote:
 Hi Matthias,

 Thanks for the response. I agree with you regarding the use of
 PartitionGrouper to reduce the number of tasks. It would be good to have an
 idea of any additional load on the brokers as we increase the number of
 tasks and therefore producers.

 Thanks,
 Damian

 On Wed, 8 Mar 2017 at 01:45 Matthias J. Sax  wrote:

> Damian, Jun,
>
> Thanks for your input.
>
>
> About Performance test:
>
> I can follow up with more performance tests using more partitions and
> also collecting broker metrics.
>
> However, I want to highlight again, that even if 1000+ partitions would
> be problematic, one can simply implement PartitionGrouper interface and
> reduce the number of tasks to 250 or 100... So I am not sure, if we
> should block this KIP, even if there might be some performance penalty
> for currently single partitioned tasks.
>
> About memory usage. JXM max-heap and max-off-heap did report 256MB and
> 133MB for all experiments (thus I did not put it in the spreadsheet).
> Thus, using 100 producers (each using a max of 32MB of memory) was not
> an issue with regard to memory consumption. I did not track "current
> head/off-heap" memory as this would require a more advance test setup to
> monitor it over time. If you think this would be required, we can do
> some tests though.
>
> However, as 256 MB was enough memory, and there are other components
> next to the producers using memory, I don't expect a severely increased
> memory usage. Pr

Re: [DISCUSS] KIP-120: Cleanup Kafka Streams builder API

2017-03-21 Thread Matthias J. Sax

@Guozhang:

I recognized that you want to have `Topology` in the name. But it seems
that more people preferred to not have it (Jay, Ram, Michael [?], myself).

@Michael:

You seemed to agree with Jay about not exposing the `Topology` concept
in our main entry class (ie, current KStreamBuilder), thus, I
interpreted that you do not want `Topology` in the name either (I am a
little surprised by your last response, that goes the opposite direction).

> StreamsBuilder builder = new StreamsBuilder();
> 
> // And here you'd define your...well, what actually?
> // Ah right, you are composing a topology here, though you are not
> aware of it.

Yes. You are not aware of if -- that's the whole point about it -- don't
put the Topology concept in the focus...

Furthermore,

>>> So what are you building here with StreamsBuilder?  Streams (hint: No)?
>>> And what about tables -- is there a TableBuilder (hint: No)?

I am not sure, if this is too much a concern. In contrast to
`KStreamBuilder` (singular) that contains `KStream` and thus puts
KStream concept in focus and thus degrade `KTable`, `StreamsBuilder`
(plural) focuses on "Streams API". IMHO, it does not put focus on
KStream. It's just a builder from the Streams API -- you don't need to
worry what you are building -- and you don't need to think about the
`Topology` concept (of course, you see that .build() return a Topology).


Personally, I see pros and cons for both `StreamsBuilder` and
`StreamsTopologyBuilder` and thus, I am fine either way. Maybe Jay and
Ram can follow up and share their thoughts?

I would also help a lot if other people put their vote for a name, too.



-Matthias



On 3/21/17 2:11 PM, Guozhang Wang wrote:
> Just to clarify, I did want to have the term `Topology` as part of the
> class name, for the reasons above. I'm not too worried about to be
> consistent with the previous names, but I feel the `XXTopologyBuilder` is
> better than `XXStreamsBuilder` since it's build() function returns a
> Topology object.
> 
> 
> Guozhang
> 
> 
> On Mon, Mar 20, 2017 at 12:53 PM, Michael Noll  wrote:
> 
>> Hmm, I must admit I don't like this last update all too much.
>>
>> Basically we would have:
>>
>> StreamsBuilder builder = new StreamsBuilder();
>>
>> // And here you'd define your...well, what actually?
>> // Ah right, you are composing a topology here, though you are not
>> aware of it.
>>
>> KafkaStreams streams = new KafkaStreams(builder.build(),
>> streamsConfiguration);
>>
>> So what are you building here with StreamsBuilder?  Streams (hint: No)?
>> And what about tables -- is there a TableBuilder (hint: No)?
>>
>> I also interpret Guozhang's last response as that he'd prefer to have
>> "Topology" in the class/interface names.  I am aware that we shouldn't
>> necessarily use the status quo to make decisions about future changes, but
>> the very first concept we explain in the Kafka Streams documentation is
>> "Stream Processing Topology":
>> https://kafka.apache.org/0102/documentation/streams#streams_concepts
>>
>> -Michael
>>
>>
>>
>> On Mon, Mar 20, 2017 at 7:55 PM, Matthias J. Sax 
>> wrote:
>>
>>> \cc users list
>>>
>>>
>>>  Forwarded Message 
>>> Subject: Re: [DISCUSS] KIP-120: Cleanup Kafka Streams builder API
>>> Date: Mon, 20 Mar 2017 11:51:01 -0700
>>> From: Matthias J. Sax 
>>> Organization: Confluent Inc
>>> To: dev@kafka.apache.org
>>>
>>> I want to push this discussion further.
>>>
>>> Guozhang's argument about "exposing" the Topology class is valid. It's a
>>> public class anyway, so it's not as issue. However, I think the question
>>> is not too much about exposing but about "advertising" (ie, putting it
>>> into the focus) or not at DSL level.
>>>
>>>
>>> If I interpret the last replies correctly, it seems that we could agree
>>> on "StreamsBuilder" as name. I did update the KIP accordingly. Please
>>> correct me, if I got this wrong.
>>>
>>>
>>> If there are not other objects -- this naming discussion was the last
>>> open point to far -- I would like the start the VOTE thread.
>>>
>>>
>>> -Matthias
>>>
>>>
>>> On 3/14/17 2:37 PM, Guozhang Wang wrote:
 I'd like to keep the term "Topology" inside the builder class since, as
 Matthias mentioned, this builder#build() function returns a "Topology"
 object, whose type is a public class anyways. Although you can argue to
>>> let
 users always call

 "new KafkaStreams(builder.build())"

 I think it is still more benefit to expose this concept.



 Guozhang

 On Tue, Mar 14, 2017 at 10:43 AM, Matthias J. Sax <
>> matth...@confluent.io

 wrote:

> Thanks for your input Michael.
>
>>> - KafkaStreams as the new name for the builder that creates the
>>> logical
>>> plan, with e.g. `KafkaStreams.stream("intput-topic")` and
>>> `KafkaStreams.table("input-topic")`.
>
> I don't thinks this is a good idea, for multiple reasons:
>
> (1) We would reuse a

[jira] [Updated] (KAFKA-4929) Transformation Key/Value type references should be to class name(), not canonicalName()

2017-03-21 Thread Guozhang Wang (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-4929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-4929:
-
Component/s: KafkaConnect

> Transformation Key/Value type references should be to class name(), not 
> canonicalName()
> ---
>
> Key: KAFKA-4929
> URL: https://issues.apache.org/jira/browse/KAFKA-4929
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 0.10.2.0
>Reporter: bruce szalwinski
>Priority: Minor
>
> The docs suggest that referencing the Key/Value transformations is done as 
> follows:
> {code}
> "transforms": "replaceFieldValue",
> "transforms.replaceFieldValue.type":  
> "org.apache.kafka.connect.transforms.ReplaceField.Value"
> {code}
> But that results in a validation failure saying that the class cannot be 
> found.
> {code}
> "value": {
> "errors": [
> "Invalid value 
> org.apache.kafka.connect.transforms.ReplaceField.Value for configuration 
> transforms.replaceFieldValue.type: Class 
> org.apache.kafka.connect.transforms.ReplaceField.Value could not be found.",
> "Invalid value null for configuration 
> transforms.replaceFieldValue.type: Not a Transformation"
> ],
> "name": "transforms.replaceFieldValue.type",
> "recommended_values": [
> "org.apache.kafka.connect.transforms.ExtractField.Key",
> "org.apache.kafka.connect.transforms.ExtractField.Value",
> "org.apache.kafka.connect.transforms.HoistField.Key",
> "org.apache.kafka.connect.transforms.HoistField.Value",
> "org.apache.kafka.connect.transforms.InsertField.Key",
> "org.apache.kafka.connect.transforms.InsertField.Value",
> "org.apache.kafka.connect.transforms.MaskField.Key",
> "org.apache.kafka.connect.transforms.MaskField.Value",
> "org.apache.kafka.connect.transforms.RegexRouter",
> "org.apache.kafka.connect.transforms.ReplaceField.Key",
> "org.apache.kafka.connect.transforms.ReplaceField.Value",
> 
> "org.apache.kafka.connect.transforms.SetSchemaMetadata.Key",
> 
> "org.apache.kafka.connect.transforms.SetSchemaMetadata.Value",
> "org.apache.kafka.connect.transforms.TimestampRouter",
> "org.apache.kafka.connect.transforms.ValueToKey"
> ],
> {code}
> Since the Key / Value transformations are defined as static nested classes, 
> the proper notation is
> {code}
> "transforms": "replaceFieldValue",
> "transforms.replaceFieldValue.type":  
> "org.apache.kafka.connect.transforms.ReplaceField$Value"
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #2670: MINOR: updated incorrect JavaDocs for joins

2017-03-21 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2670


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #2717: MINOR: remove unused log field from KStreamTransfo...

2017-03-21 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2717


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-4863) Querying window store may return unwanted keys

2017-03-21 Thread Guozhang Wang (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-4863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-4863:
-
Fix Version/s: 0.10.2.1

> Querying window store may return unwanted keys
> --
>
> Key: KAFKA-4863
> URL: https://issues.apache.org/jira/browse/KAFKA-4863
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Xavier Léauté
>Assignee: Damian Guy
>Priority: Critical
> Fix For: 0.11.0.0, 0.10.2.1
>
>
> Using variable length keys in a window store may cause unwanted results to be 
> returned when querying certain ranges.
> Below is a test case for {{RocksDBWindowStoreTest}} that shows the problem. 
> It fails, returning {{\[0001, 0003, 0002, 0004, 0005\]}} instead of {{\[0001, 
> 0003, 0005\]}}.
> {code:java}
> @Test
> public void testPutAndFetchSanity() throws IOException {
> final RocksDBWindowStoreSupplier supplier =
> new RocksDBWindowStoreSupplier<>(
> "window", 60 * 1000L * 2, 3,
> true, Serdes.String(), Serdes.String(),
> windowSize, true, Collections.emptyMap(), 
> false
> );
> final WindowStore store = supplier.get();
> store.init(context, store);
> try {
> store.put("a", "0001", 0);
> store.put("aa", "0002", 0);
> store.put("a", "0003", 1);
> store.put("aa", "0004", 1);
> store.put("a", "0005", 6);
> assertEquals(Utils.mkList("0001", "0003", "0005"), 
> toList(store.fetch("a", 0, Long.MAX_VALUE)));
> } finally {
> store.close();
> }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-4863) Querying window store may return unwanted keys

2017-03-21 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-4863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15935519#comment-15935519
 ] 

ASF GitHub Bot commented on KAFKA-4863:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2713


> Querying window store may return unwanted keys
> --
>
> Key: KAFKA-4863
> URL: https://issues.apache.org/jira/browse/KAFKA-4863
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Xavier Léauté
>Assignee: Damian Guy
>Priority: Critical
> Fix For: 0.11.0.0
>
>
> Using variable length keys in a window store may cause unwanted results to be 
> returned when querying certain ranges.
> Below is a test case for {{RocksDBWindowStoreTest}} that shows the problem. 
> It fails, returning {{\[0001, 0003, 0002, 0004, 0005\]}} instead of {{\[0001, 
> 0003, 0005\]}}.
> {code:java}
> @Test
> public void testPutAndFetchSanity() throws IOException {
> final RocksDBWindowStoreSupplier supplier =
> new RocksDBWindowStoreSupplier<>(
> "window", 60 * 1000L * 2, 3,
> true, Serdes.String(), Serdes.String(),
> windowSize, true, Collections.emptyMap(), 
> false
> );
> final WindowStore store = supplier.get();
> store.init(context, store);
> try {
> store.put("a", "0001", 0);
> store.put("aa", "0002", 0);
> store.put("a", "0003", 1);
> store.put("aa", "0004", 1);
> store.put("a", "0005", 6);
> assertEquals(Utils.mkList("0001", "0003", "0005"), 
> toList(store.fetch("a", 0, Long.MAX_VALUE)));
> } finally {
> store.close();
> }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #2713: KAFKA-4863: [Follow Up] Querying window store may ...

2017-03-21 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2713


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-4863) Querying window store may return unwanted keys

2017-03-21 Thread Guozhang Wang (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-4863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-4863:
-
   Resolution: Fixed
Fix Version/s: 0.11.0.0
   Status: Resolved  (was: Patch Available)

Issue resolved by pull request 2713
[https://github.com/apache/kafka/pull/2713]

> Querying window store may return unwanted keys
> --
>
> Key: KAFKA-4863
> URL: https://issues.apache.org/jira/browse/KAFKA-4863
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Xavier Léauté
>Assignee: Damian Guy
>Priority: Critical
> Fix For: 0.11.0.0
>
>
> Using variable length keys in a window store may cause unwanted results to be 
> returned when querying certain ranges.
> Below is a test case for {{RocksDBWindowStoreTest}} that shows the problem. 
> It fails, returning {{\[0001, 0003, 0002, 0004, 0005\]}} instead of {{\[0001, 
> 0003, 0005\]}}.
> {code:java}
> @Test
> public void testPutAndFetchSanity() throws IOException {
> final RocksDBWindowStoreSupplier supplier =
> new RocksDBWindowStoreSupplier<>(
> "window", 60 * 1000L * 2, 3,
> true, Serdes.String(), Serdes.String(),
> windowSize, true, Collections.emptyMap(), 
> false
> );
> final WindowStore store = supplier.get();
> store.init(context, store);
> try {
> store.put("a", "0001", 0);
> store.put("aa", "0002", 0);
> store.put("a", "0003", 1);
> store.put("aa", "0004", 1);
> store.put("a", "0005", 6);
> assertEquals(Utils.mkList("0001", "0003", "0005"), 
> toList(store.fetch("a", 0, Long.MAX_VALUE)));
> } finally {
> store.close();
> }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-4929) Transformation Key/Value type references should be to class name(), not canonicalName()

2017-03-21 Thread bruce szalwinski (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-4929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

bruce szalwinski updated KAFKA-4929:

Status: Patch Available  (was: Open)

> Transformation Key/Value type references should be to class name(), not 
> canonicalName()
> ---
>
> Key: KAFKA-4929
> URL: https://issues.apache.org/jira/browse/KAFKA-4929
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.2.0
>Reporter: bruce szalwinski
>Priority: Minor
>
> The docs suggest that referencing the Key/Value transformations is done as 
> follows:
> {code}
> "transforms": "replaceFieldValue",
> "transforms.replaceFieldValue.type":  
> "org.apache.kafka.connect.transforms.ReplaceField.Value"
> {code}
> But that results in a validation failure saying that the class cannot be 
> found.
> {code}
> "value": {
> "errors": [
> "Invalid value 
> org.apache.kafka.connect.transforms.ReplaceField.Value for configuration 
> transforms.replaceFieldValue.type: Class 
> org.apache.kafka.connect.transforms.ReplaceField.Value could not be found.",
> "Invalid value null for configuration 
> transforms.replaceFieldValue.type: Not a Transformation"
> ],
> "name": "transforms.replaceFieldValue.type",
> "recommended_values": [
> "org.apache.kafka.connect.transforms.ExtractField.Key",
> "org.apache.kafka.connect.transforms.ExtractField.Value",
> "org.apache.kafka.connect.transforms.HoistField.Key",
> "org.apache.kafka.connect.transforms.HoistField.Value",
> "org.apache.kafka.connect.transforms.InsertField.Key",
> "org.apache.kafka.connect.transforms.InsertField.Value",
> "org.apache.kafka.connect.transforms.MaskField.Key",
> "org.apache.kafka.connect.transforms.MaskField.Value",
> "org.apache.kafka.connect.transforms.RegexRouter",
> "org.apache.kafka.connect.transforms.ReplaceField.Key",
> "org.apache.kafka.connect.transforms.ReplaceField.Value",
> 
> "org.apache.kafka.connect.transforms.SetSchemaMetadata.Key",
> 
> "org.apache.kafka.connect.transforms.SetSchemaMetadata.Value",
> "org.apache.kafka.connect.transforms.TimestampRouter",
> "org.apache.kafka.connect.transforms.ValueToKey"
> ],
> {code}
> Since the Key / Value transformations are defined as static nested classes, 
> the proper notation is
> {code}
> "transforms": "replaceFieldValue",
> "transforms.replaceFieldValue.type":  
> "org.apache.kafka.connect.transforms.ReplaceField$Value"
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-2273) KIP-54: Add rebalance with a minimal number of reassignments to server-defined strategy list

2017-03-21 Thread Allen Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-2273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15935450#comment-15935450
 ] 

Allen Wang commented on KAFKA-2273:
---

Does the current implementation have any dependency on Kafka 0.10 APIs or 
protocols? Can the same code work with Kafka 0.9 client and broker?


> KIP-54: Add rebalance with a minimal number of reassignments to 
> server-defined strategy list
> 
>
> Key: KAFKA-2273
> URL: https://issues.apache.org/jira/browse/KAFKA-2273
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Olof Johansson
>Assignee: Vahid Hashemian
>  Labels: kip
> Fix For: 0.11.0.0
>
>
> Add a new partitions.assignment.strategy to the server-defined list that will 
> do reassignments based on moving as few partitions as possible. This should 
> be a quite common reassignment strategy especially for the cases where the 
> consumer has to maintain state, either in memory, or on disk.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Re: [DISCUSS] KIP-120: Cleanup Kafka Streams builder API

2017-03-21 Thread Guozhang Wang

Just to clarify, I did want to have the term `Topology` as part of the
class name, for the reasons above. I'm not too worried about to be
consistent with the previous names, but I feel the `XXTopologyBuilder` is
better than `XXStreamsBuilder` since it's build() function returns a
Topology object.


Guozhang


On Mon, Mar 20, 2017 at 12:53 PM, Michael Noll  wrote:

> Hmm, I must admit I don't like this last update all too much.
>
> Basically we would have:
>
> StreamsBuilder builder = new StreamsBuilder();
>
> // And here you'd define your...well, what actually?
> // Ah right, you are composing a topology here, though you are not
> aware of it.
>
> KafkaStreams streams = new KafkaStreams(builder.build(),
> streamsConfiguration);
>
> So what are you building here with StreamsBuilder?  Streams (hint: No)?
> And what about tables -- is there a TableBuilder (hint: No)?
>
> I also interpret Guozhang's last response as that he'd prefer to have
> "Topology" in the class/interface names.  I am aware that we shouldn't
> necessarily use the status quo to make decisions about future changes, but
> the very first concept we explain in the Kafka Streams documentation is
> "Stream Processing Topology":
> https://kafka.apache.org/0102/documentation/streams#streams_concepts
>
> -Michael
>
>
>
> On Mon, Mar 20, 2017 at 7:55 PM, Matthias J. Sax 
> wrote:
>
> > \cc users list
> >
> >
> >  Forwarded Message 
> > Subject: Re: [DISCUSS] KIP-120: Cleanup Kafka Streams builder API
> > Date: Mon, 20 Mar 2017 11:51:01 -0700
> > From: Matthias J. Sax 
> > Organization: Confluent Inc
> > To: dev@kafka.apache.org
> >
> > I want to push this discussion further.
> >
> > Guozhang's argument about "exposing" the Topology class is valid. It's a
> > public class anyway, so it's not as issue. However, I think the question
> > is not too much about exposing but about "advertising" (ie, putting it
> > into the focus) or not at DSL level.
> >
> >
> > If I interpret the last replies correctly, it seems that we could agree
> > on "StreamsBuilder" as name. I did update the KIP accordingly. Please
> > correct me, if I got this wrong.
> >
> >
> > If there are not other objects -- this naming discussion was the last
> > open point to far -- I would like the start the VOTE thread.
> >
> >
> > -Matthias
> >
> >
> > On 3/14/17 2:37 PM, Guozhang Wang wrote:
> > > I'd like to keep the term "Topology" inside the builder class since, as
> > > Matthias mentioned, this builder#build() function returns a "Topology"
> > > object, whose type is a public class anyways. Although you can argue to
> > let
> > > users always call
> > >
> > > "new KafkaStreams(builder.build())"
> > >
> > > I think it is still more benefit to expose this concept.
> > >
> > >
> > >
> > > Guozhang
> > >
> > > On Tue, Mar 14, 2017 at 10:43 AM, Matthias J. Sax <
> matth...@confluent.io
> > >
> > > wrote:
> > >
> > >> Thanks for your input Michael.
> > >>
> >  - KafkaStreams as the new name for the builder that creates the
> > logical
> >  plan, with e.g. `KafkaStreams.stream("intput-topic")` and
> >  `KafkaStreams.table("input-topic")`.
> > >>
> > >> I don't thinks this is a good idea, for multiple reasons:
> > >>
> > >> (1) We would reuse a name for a completely different purpose. The same
> > >> argument for not renaming KStreamBuilder to TopologyBuilder. The
> > >> confusion would just be too large.
> > >>
> > >> So if we would start from scratch, it might be ok to do so, but now we
> > >> cannot make this move, IMHO.
> > >>
> > >> Also a clarification question: do you suggest to have static methods
> > >> #stream and #table -- I am not sure if this would work?
> > >> (or was you code snippet just simplification?)
> > >>
> > >>
> > >> (2) Kafka Streams is basically a "processing client" next to consumer
> > >> and producer client. Thus, the name KafkaStreams aligns to the naming
> > >> schema of KafkaConsumer and KafkaProducer. I am not sure if it would
> be
> > >> a good choice to "break" this naming scheme.
> > >>
> > >> Btw: this is also the reason, why we have KafkaStreams#close() -- and
> > >> not KafkaStreams#stop() -- because #close() aligns with consumer and
> > >> producer client.
> > >>
> > >>
> > >> (3) On more argument against using KafkaStreams as DSL entry class
> would
> > >> be, that it would need to create a Topology that can be given to the
> > >> "runner/processing-client". Thus the pattern would be
> > >>
> > >>> Topology topology = streams.build();
> > >>> KafkaStramsRunner runner = new KafkaStreamsRunner(..., topology)
> > >>
> > >> (or of course as a one liner).
> > >>
> > >>
> > >>
> > >> On the other hand, there was the idea (that we intentionally excluded
> > >> from the KIP), to change the "client instantiation" pattern.
> > >>
> > >> Right now, a new client in actively instantiated (ie, by calling
> "new")
> > >> and the topology if provided as a constructor argument. However,
> > >> especially for DSL (not sure if i

[jira] [Commented] (KAFKA-4929) Transformation Key/Value type references should be to class name(), not canonicalName()

2017-03-21 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-4929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15935387#comment-15935387
 ] 

ASF GitHub Bot commented on KAFKA-4929:
---

GitHub user bruce-szalwinski opened a pull request:

https://github.com/apache/kafka/pull/2720

KAFKA-4929: Transformation Key/Value type references should be to class 
name(), not canonicalName()



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/CDKGlobal/kafka transforms

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2720.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2720


commit eba0af7a2de00c024095591482f439baa7de9bfd
Author: Bruce Szalwinski 
Date:   2017-03-21T21:28:16Z

Key / Value transformations are static nested classes and so are referenced 
using OuterClass$Key and OuterClass$Value.




> Transformation Key/Value type references should be to class name(), not 
> canonicalName()
> ---
>
> Key: KAFKA-4929
> URL: https://issues.apache.org/jira/browse/KAFKA-4929
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.2.0
>Reporter: bruce szalwinski
>Priority: Minor
>
> The docs suggest that referencing the Key/Value transformations is done as 
> follows:
> {code}
> "transforms": "replaceFieldValue",
> "transforms.replaceFieldValue.type":  
> "org.apache.kafka.connect.transforms.ReplaceField.Value"
> {code}
> But that results in a validation failure saying that the class cannot be 
> found.
> {code}
> "value": {
> "errors": [
> "Invalid value 
> org.apache.kafka.connect.transforms.ReplaceField.Value for configuration 
> transforms.replaceFieldValue.type: Class 
> org.apache.kafka.connect.transforms.ReplaceField.Value could not be found.",
> "Invalid value null for configuration 
> transforms.replaceFieldValue.type: Not a Transformation"
> ],
> "name": "transforms.replaceFieldValue.type",
> "recommended_values": [
> "org.apache.kafka.connect.transforms.ExtractField.Key",
> "org.apache.kafka.connect.transforms.ExtractField.Value",
> "org.apache.kafka.connect.transforms.HoistField.Key",
> "org.apache.kafka.connect.transforms.HoistField.Value",
> "org.apache.kafka.connect.transforms.InsertField.Key",
> "org.apache.kafka.connect.transforms.InsertField.Value",
> "org.apache.kafka.connect.transforms.MaskField.Key",
> "org.apache.kafka.connect.transforms.MaskField.Value",
> "org.apache.kafka.connect.transforms.RegexRouter",
> "org.apache.kafka.connect.transforms.ReplaceField.Key",
> "org.apache.kafka.connect.transforms.ReplaceField.Value",
> 
> "org.apache.kafka.connect.transforms.SetSchemaMetadata.Key",
> 
> "org.apache.kafka.connect.transforms.SetSchemaMetadata.Value",
> "org.apache.kafka.connect.transforms.TimestampRouter",
> "org.apache.kafka.connect.transforms.ValueToKey"
> ],
> {code}
> Since the Key / Value transformations are defined as static nested classes, 
> the proper notation is
> {code}
> "transforms": "replaceFieldValue",
> "transforms.replaceFieldValue.type":  
> "org.apache.kafka.connect.transforms.ReplaceField$Value"
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #2720: KAFKA-4929: Transformation Key/Value type referenc...

2017-03-21 Thread bruce-szalwinski

GitHub user bruce-szalwinski opened a pull request:

https://github.com/apache/kafka/pull/2720

KAFKA-4929: Transformation Key/Value type references should be to class 
name(), not canonicalName()



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/CDKGlobal/kafka transforms

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2720.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2720


commit eba0af7a2de00c024095591482f439baa7de9bfd
Author: Bruce Szalwinski 
Date:   2017-03-21T21:28:16Z

Key / Value transformations are static nested classes and so are referenced 
using OuterClass$Key and OuterClass$Value.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [VOTE] KIP-124: Request rate quotas

2017-03-21 Thread Guozhang Wang

+1. Thanks!

On Tue, Mar 21, 2017 at 2:00 PM, Becket Qin  wrote:

> +1
>
> Thanks for driving through this, Rajini :)
>
>
>
> On Tue, Mar 21, 2017 at 9:49 AM, Roger Hoover 
> wrote:
>
> > Rajini,
> >
> > This is great.  Thank you.  +1 (non-binding)
> >
> > Roger
> >
> > On Tue, Mar 21, 2017 at 8:55 AM, Ismael Juma  wrote:
> >
> > > Rajini,
> > >
> > > Thanks for the proposal and for addressing the (sometimes
> contradictory)
> > > feedback. :) +1 (binding) from me.
> > >
> > > Ismael
> > >
> > > On Mon, Mar 20, 2017 at 2:47 PM, Rajini Sivaram <
> rajinisiva...@gmail.com
> > >
> > > wrote:
> > >
> > > > I would like to initiate the voting process for KIP-124:
> > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > 124+-+Request+rate+quotas
> > > >
> > > > The KIP proposes to add request processing time quota to limit CPU
> > > > utilization at **, ** or ** levels
> > > > similar to the current bandwidth limiting quotas. The quota takes
> into
> > > > account the total time spent by the user/client on request handler
> and
> > > > network threads within a quota window. Quota will be configured as a
> > > > per-thread percentage value, which typically indicates the percentage
> > of
> > > > CPU cores allocated to the user/client.
> > > >
> > > > The discussion thread is here:
> > > >
> > > >
> > > > https://lists.apache.org/thread.html/46c7bbc8f381ebe718b3cce6ed8bdf
> > > > 3745df22b0bd88020d70c99813@%3Cdev.kafka.apache.org%3E
> > > >
> > > > Many thanks to everyone for the feedback and suggestions so far.
> > > >
> > > > Regards,
> > > >
> > > > Rajini
> > > >
> > >
> >
>



-- 
-- Guozhang

[jira] [Created] (KAFKA-4929) Transformation Key/Value type references should be to class name(), not canonicalName()

2017-03-21 Thread bruce szalwinski (JIRA)

bruce szalwinski created KAFKA-4929:
---

 Summary: Transformation Key/Value type references should be to 
class name(), not canonicalName()
 Key: KAFKA-4929
 URL: https://issues.apache.org/jira/browse/KAFKA-4929
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.10.2.0
Reporter: bruce szalwinski
Priority: Minor


The docs suggest that referencing the Key/Value transformations is done as 
follows:

{code}
"transforms": "replaceFieldValue",
"transforms.replaceFieldValue.type":  
"org.apache.kafka.connect.transforms.ReplaceField.Value"
{code}

But that results in a validation failure saying that the class cannot be found.

{code}
"value": {
"errors": [
"Invalid value 
org.apache.kafka.connect.transforms.ReplaceField.Value for configuration 
transforms.replaceFieldValue.type: Class 
org.apache.kafka.connect.transforms.ReplaceField.Value could not be found.",
"Invalid value null for configuration 
transforms.replaceFieldValue.type: Not a Transformation"
],
"name": "transforms.replaceFieldValue.type",
"recommended_values": [
"org.apache.kafka.connect.transforms.ExtractField.Key",
"org.apache.kafka.connect.transforms.ExtractField.Value",
"org.apache.kafka.connect.transforms.HoistField.Key",
"org.apache.kafka.connect.transforms.HoistField.Value",
"org.apache.kafka.connect.transforms.InsertField.Key",
"org.apache.kafka.connect.transforms.InsertField.Value",
"org.apache.kafka.connect.transforms.MaskField.Key",
"org.apache.kafka.connect.transforms.MaskField.Value",
"org.apache.kafka.connect.transforms.RegexRouter",
"org.apache.kafka.connect.transforms.ReplaceField.Key",
"org.apache.kafka.connect.transforms.ReplaceField.Value",
"org.apache.kafka.connect.transforms.SetSchemaMetadata.Key",

"org.apache.kafka.connect.transforms.SetSchemaMetadata.Value",
"org.apache.kafka.connect.transforms.TimestampRouter",
"org.apache.kafka.connect.transforms.ValueToKey"
],
{code}

Since the Key / Value transformations are defined as static nested classes, the 
proper notation is

{code}
"transforms": "replaceFieldValue",
"transforms.replaceFieldValue.type":  
"org.apache.kafka.connect.transforms.ReplaceField$Value"
{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (KAFKA-4808) send of null key to a compacted topic should throw error back to user

2017-03-21 Thread Mayuresh Gharat (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-4808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15935327#comment-15935327
 ] 

Mayuresh Gharat edited comment on KAFKA-4808 at 3/21/17 9:00 PM:
-

[~ijuma] Please find the KIP  here : 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-135+%3A+Send+of+null+key+to+a+compacted+topic+should+throw+non-retriable+error+back+to+user.



was (Author: mgharat):
[~ijuma] Please find the KIP  here : 
https://cwiki.apache.org/confluence/display/KAFKA/Send+of+null+key+to+a+compacted+topic+should+throw+non-retriable+error+back+to+user.


> send of null key to a compacted topic should throw error back to user
> -
>
> Key: KAFKA-4808
> URL: https://issues.apache.org/jira/browse/KAFKA-4808
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Affects Versions: 0.10.2.0
>Reporter: Ismael Juma
>Assignee: Mayuresh Gharat
> Fix For: 0.11.0.0
>
>
> If a message with a null key is produced to a compacted topic, the broker 
> returns `CorruptRecordException`, which is a retriable exception. As such, 
> the producer keeps retrying until retries are exhausted or request.timeout.ms 
> expires and eventually throws a TimeoutException. This is confusing and not 
> user-friendly.
> We should throw a meaningful error back to the user. From an implementation 
> perspective, we would have to use a non retriable error code to avoid this 
> issue.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [VOTE] KIP-124: Request rate quotas

2017-03-21 Thread Becket Qin

+1

Thanks for driving through this, Rajini :)



On Tue, Mar 21, 2017 at 9:49 AM, Roger Hoover 
wrote:

> Rajini,
>
> This is great.  Thank you.  +1 (non-binding)
>
> Roger
>
> On Tue, Mar 21, 2017 at 8:55 AM, Ismael Juma  wrote:
>
> > Rajini,
> >
> > Thanks for the proposal and for addressing the (sometimes contradictory)
> > feedback. :) +1 (binding) from me.
> >
> > Ismael
> >
> > On Mon, Mar 20, 2017 at 2:47 PM, Rajini Sivaram  >
> > wrote:
> >
> > > I would like to initiate the voting process for KIP-124:
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 124+-+Request+rate+quotas
> > >
> > > The KIP proposes to add request processing time quota to limit CPU
> > > utilization at **, ** or ** levels
> > > similar to the current bandwidth limiting quotas. The quota takes into
> > > account the total time spent by the user/client on request handler and
> > > network threads within a quota window. Quota will be configured as a
> > > per-thread percentage value, which typically indicates the percentage
> of
> > > CPU cores allocated to the user/client.
> > >
> > > The discussion thread is here:
> > >
> > >
> > > https://lists.apache.org/thread.html/46c7bbc8f381ebe718b3cce6ed8bdf
> > > 3745df22b0bd88020d70c99813@%3Cdev.kafka.apache.org%3E
> > >
> > > Many thanks to everyone for the feedback and suggestions so far.
> > >
> > > Regards,
> > >
> > > Rajini
> > >
> >
>

[jira] [Commented] (KAFKA-4808) send of null key to a compacted topic should throw error back to user

2017-03-21 Thread Mayuresh Gharat (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-4808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15935327#comment-15935327
 ] 

Mayuresh Gharat commented on KAFKA-4808:


[~ijuma] Please find the KIP  here : 
https://cwiki.apache.org/confluence/display/KAFKA/Send+of+null+key+to+a+compacted+topic+should+throw+non-retriable+error+back+to+user.


> send of null key to a compacted topic should throw error back to user
> -
>
> Key: KAFKA-4808
> URL: https://issues.apache.org/jira/browse/KAFKA-4808
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Affects Versions: 0.10.2.0
>Reporter: Ismael Juma
>Assignee: Mayuresh Gharat
> Fix For: 0.11.0.0
>
>
> If a message with a null key is produced to a compacted topic, the broker 
> returns `CorruptRecordException`, which is a retriable exception. As such, 
> the producer keeps retrying until retries are exhausted or request.timeout.ms 
> expires and eventually throws a TimeoutException. This is confusing and not 
> user-friendly.
> We should throw a meaningful error back to the user. From an implementation 
> perspective, we would have to use a non retriable error code to avoid this 
> issue.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-4908) consumer.properties logging warnings

2017-03-21 Thread Seweryn Habdank-Wojewodzki (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15935288#comment-15935288
 ] 

Seweryn Habdank-Wojewodzki commented on KAFKA-4908:
---

So even more. Deprecated configs shall be removed from release, or? :-)

> consumer.properties logging warnings
> 
>
> Key: KAFKA-4908
> URL: https://issues.apache.org/jira/browse/KAFKA-4908
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.2.0
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Minor
>
> default consumer.properties at startaup of the console consumer delivered 
> with Kafka package are logging warnings:
> [2017-03-15 16:36:57,439] WARN The configuration 
> 'zookeeper.connection.timeout.ms' was supplied but isn't a known config. 
> (org.apache.kafka.clients.consumer.ConsumerConfig)
> [2017-03-15 16:36:57,455] WARN The configuration 'zookeeper.connect' was 
> supplied but isn't a known config. 
> (org.apache.kafka.clients.consumer.ConsumerConfig)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-4870) A question about broker down , the server is doing partition master election,the client producer may send msg fail . How the producer deal with the situation ??

2017-03-21 Thread Colin P. McCabe (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-4870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15935281#comment-15935281
 ] 

Colin P. McCabe commented on KAFKA-4870:


This seems like a question, rather than a bug.  Did you ask about this on the 
mailing list?

> A question about broker down , the server is doing partition master 
> election,the client producer may send msg fail . How the producer deal with 
> the situation ??
> 
>
> Key: KAFKA-4870
> URL: https://issues.apache.org/jira/browse/KAFKA-4870
> Project: Kafka
>  Issue Type: Test
>  Components: clients
> Environment: java client 
>Reporter: zhaoziyan
>Priority: Minor
>
> the broker down . The kafka cluster is doing partion  master election , the 
> producer send order msg or nomal msg ,the producer may send msg fail .How 
> client update metadata and deal with the msg send fail ?? 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Features ideas - need feedback: message deletion and consumer improvements

2017-03-21 Thread Colin McCabe

On Wed, Mar 8, 2017, at 17:28, Maarek, Stephane wrote:
> Hi,
> 
> Two ideas that I would like to get feedback on before putting KIPs
> together.
> 
> 1) Ability to have the kafka client consumer “skip” data that can’t be
> de-serialized. it would be a consumer config such as
> “ignore.deserialization.errors” (got better naming?) that defaults to
> false to it’s backwards compatible, but if set to true, would produce a
> warning on the consumer client log but wouldn’t stop the processing - no
> errors thrown. The message would just be discarded. The use case is for
> example when reading an avro topic but someone pushes a message that’s
> not avro, currently consumers would break.

Interesting idea.  How about a Deserializer implementation that wraps
other Deserializer implementations in a try...catch block?

best,
Colin

> 
> 2) Ability to delete messages on topic. I believe log compaction already
> has a mechanism to do that so we would leverage that code. The idea would
> be to have an API to delete a message or a range of message based on
> topic / partition / offset. It would come with a command line tool. This
> would allow to delete messages from a topic so that if some bad data is
> pushed, it doesn’t break downstream consumers.
> 
> 
> Additionally, I may be able to write 1) by myself, but I believe I won’t
> have the capability to write 2), so I’d look for someone to help out
> there
> 
> Looking forward to feedback.
> 
> 
> Best regards,
> Stephane
> 
> This email, and any attachments, is confidential and may be covered by
> legal professional privilege or other legal rules. If you are not the
> intended recipient you must not disclose or use the information contained
> in it. If you have received this email in error please notify us
> immediately by return email or by calling our main switchboard on +613
> 9868 2100 and delete the email.

[jira] [Comment Edited] (KAFKA-4878) Kafka Connect does not log connector configuration errors

2017-03-21 Thread Armin Braun (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15935249#comment-15935249
 ] 

Armin Braun edited comment on KAFKA-4878 at 3/21/17 8:23 PM:
-

[~gwenshap] Np, debugged this on the weekend and was going to put in a PR later 
today or tomorrow morning. If you have any specific text you'd like to see in 
the error message, just let me know.
Also one question, you still want to mention the /config/validate in the error 
if when printing all error in the log? Yes right?


was (Author: original-brownbear):
[~gwenshap] debugged this on the weekend and was going to put in a PR later 
today or tomorrow morning. If you have any specific text you'd like to see in 
the error message, just let me know.
Also one question, you still want to mention the /config/validate in the error 
if when printing all error in the log? Yes right?

> Kafka Connect does not log connector configuration errors
> -
>
> Key: KAFKA-4878
> URL: https://issues.apache.org/jira/browse/KAFKA-4878
> Project: Kafka
>  Issue Type: Bug
>Reporter: Gwen Shapira
>Assignee: Armin Braun
>Priority: Blocker
> Fix For: 0.10.2.1
>
>
> Currently, on connector configuration error, Kafka Connect (both distributed 
> and stand alone) logs:
> org.apache.kafka.connect.runtime.rest.errors.BadRequestException: Connector 
> configuration is invalid (use the endpoint `/{connectorType}/config/validate` 
> to get a full list of errors)
> This is annoying because:
> 1. If I'm using stand-alone mode, I may have configured my connector via 
> configuration file and I don't want to know about the REST API at all.
> 2. The output of validate is rather annoying
> What I'd like to see in the output is:
> 1. number of errors in my configuration
> 2. at least one error, preferably all of them



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-4919) Streams job fails with InvalidStateStoreException: Store is currently closed

2017-03-21 Thread Elias Levy (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-4919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15935252#comment-15935252
 ] 

Elias Levy commented on KAFKA-4919:
---

The issue appears to be that a {{RocksDBStore}} segment of a 
{{RocksDBSegmentedBytesStore}} is closed during a call to {{put}} when the 
store is reused after a consumer rebalance.  I've reviewed the code, but I 
can't see how the segment could be closed after the rebalance.  If it is a 
newly crated segment, it will be open.  If it is closed, then it should have 
been destroyed and removed from the segments list in {{Segments.cleanup}}.

Thoughts?

> Streams job fails with InvalidStateStoreException: Store is currently closed
> 
>
> Key: KAFKA-4919
> URL: https://issues.apache.org/jira/browse/KAFKA-4919
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Elias Levy
>
> I have a streams job, that previously worked, that consumes and writes to a 
> large number of topics with many partitions and that uses many threads.  I 
> upgraded the job to 0.10.2.0.  The job now fails after a short time running, 
> seemingly after a rebalance.
> {quote}
> WARN  2017-03-19 18:03:20,432 [StreamThread-10][StreamThread.java:160] : 
> Unexpected state transition from RUNNING to NOT_RUNNING
> {quote}
> The first observation is that Streams is no longer outputting exceptions and 
> backtraces.  I had to add code to get this information.
> The exception:
> {quote}
> Exception: org.apache.kafka.streams.errors.StreamsException: Exception caught 
> in process. taskId=1_225, processor=KSTREAM-SOURCE-03, 
> topic=some_topic, partition=225, offset=266411
> org.apache.kafka.streams.errors.StreamsException: Exception caught in 
> process. taskId=1_225, processor=KSTREAM-SOURCE-03, topic=some_topic, 
> partition=225, offset=266411
>   at 
> org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:216)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:641)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:368)
> Caused by: org.apache.kafka.streams.errors.InvalidStateStoreException: Store 
> someStore-201701060400 is currently closed
>   at 
> org.apache.kafka.streams.state.internals.RocksDBStore.validateStoreOpen(RocksDBStore.java:205)
>   at 
> org.apache.kafka.streams.state.internals.RocksDBStore.put(RocksDBStore.java:221)
>   at 
> org.apache.kafka.streams.state.internals.RocksDBSegmentedBytesStore.put(RocksDBSegmentedBytesStore.java:74)
>   at 
> org.apache.kafka.streams.state.internals.ChangeLoggingSegmentedBytesStore.put(ChangeLoggingSegmentedBytesStore.java:54)
>   at 
> org.apache.kafka.streams.state.internals.MeteredSegmentedBytesStore.put(MeteredSegmentedBytesStore.java:101)
>   at 
> org.apache.kafka.streams.state.internals.RocksDBWindowStore.put(RocksDBWindowStore.java:109)
>   ... X more
> {quote}
> The error occurs for many partitions.
> This was preceded by (for this partition):
> {quote}
> INFO  2017-03-19 18:03:16,403 [StreamThread-10][ConsumerCoordinator.java:393] 
> : Revoking previously assigned partitions [some_topic-225] for group some_job
> INFO  2017-03-19 18:03:16,403 [StreamThread-10][StreamThread.java:254] : 
> stream-thread [StreamThread-10] partitions [[some_topic-225]] revoked at the 
> beginning of consumer rebalance.
> INFO  2017-03-19 18:03:16,403 [StreamThread-10][StreamThread.java:1056] : 
> stream-thread [StreamThread-10] Closing a task's topology 1_225
> INFO  2017-03-19 18:03:17,887 [StreamThread-10][StreamThread.java:544] : 
> stream-thread [StreamThread-10] Flushing state stores of task 1_225
> INFO  2017-03-19 18:03:17,887 [StreamThread-10][StreamThread.java:534] : 
> stream-thread [StreamThread-10] Committing consumer offsets of task 1_225
> INFO  2017-03-19 18:03:17,891 [StreamThread-10][StreamThread.java:1012] : 
> stream-thread [StreamThread-10] Updating suspended tasks to contain active 
> tasks [[1_225, 0_445, 0_30]]
> INFO  2017-03-19 18:03:17,891 [StreamThread-10][StreamThread.java:1019] : 
> stream-thread [StreamThread-10] Removing all active tasks [[1_225, 0_445, 
> 0_30]]
> INFO  2017-03-19 18:03:19,925 [StreamThread-10][ConsumerCoordinator.java:252] 
> : Setting newly assigned partitions [some_tpoic-225] for group some_job
> INFO  2017-03-19 18:03:19,927 [StreamThread-10][StreamThread.java:228] : 
> stream-thread [StreamThread-10] New partitions [[some_topic-225]] assigned at 
> the end of consumer rebalance.
> INFO  2017-03-19 18:03:19,929 [StreamThread-10][StreamTask.java:333] : task 
> [1_225] Initializing processor nodes of the topology
> Something happens.  What ???
> INFO  2017-03

[jira] [Commented] (KAFKA-4878) Kafka Connect does not log connector configuration errors

2017-03-21 Thread Armin Braun (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15935249#comment-15935249
 ] 

Armin Braun commented on KAFKA-4878:


[~gwenshap] debugged this on the weekend and was going to put in a PR later 
today or tomorrow morning. If you have any specific text you'd like to see in 
the error message, just let me know.
Also one question, you still want to mention the /config/validate in the error 
if when printing all error in the log? Yes right?

> Kafka Connect does not log connector configuration errors
> -
>
> Key: KAFKA-4878
> URL: https://issues.apache.org/jira/browse/KAFKA-4878
> Project: Kafka
>  Issue Type: Bug
>Reporter: Gwen Shapira
>Assignee: Armin Braun
>Priority: Blocker
> Fix For: 0.10.2.1
>
>
> Currently, on connector configuration error, Kafka Connect (both distributed 
> and stand alone) logs:
> org.apache.kafka.connect.runtime.rest.errors.BadRequestException: Connector 
> configuration is invalid (use the endpoint `/{connectorType}/config/validate` 
> to get a full list of errors)
> This is annoying because:
> 1. If I'm using stand-alone mode, I may have configured my connector via 
> configuration file and I don't want to know about the REST API at all.
> 2. The output of validate is rather annoying
> What I'd like to see in the output is:
> 1. number of errors in my configuration
> 2. at least one error, preferably all of them



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-4916) Add streams tests with brokers failing

2017-03-21 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-4916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15935102#comment-15935102
 ] 

ASF GitHub Bot commented on KAFKA-4916:
---

GitHub user enothereska opened a pull request:

https://github.com/apache/kafka/pull/2719

KAFKA-4916: test streams with brokers failing [WiP]



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/enothereska/kafka 
KAFKA-4916-broker-bounce-test

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2719.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2719


commit 9df10100d894364ceca3cb645c852b20938c00b3
Author: Eno Thereska 
Date:   2017-03-21T18:46:34Z

Initial test




> Add streams tests with brokers failing
> --
>
> Key: KAFKA-4916
> URL: https://issues.apache.org/jira/browse/KAFKA-4916
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Eno Thereska
>Assignee: Eno Thereska
>Priority: Critical
> Fix For: 0.11.0.0
>
>
> We need to add either integration or system tests with streams and have Kafka 
> brokers fail and come back up. A combination of transient and permanent 
> broker failures.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #2719: KAFKA-4916: test streams with brokers failing [WiP...

2017-03-21 Thread enothereska

GitHub user enothereska opened a pull request:

https://github.com/apache/kafka/pull/2719

KAFKA-4916: test streams with brokers failing [WiP]



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/enothereska/kafka 
KAFKA-4916-broker-bounce-test

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2719.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2719


commit 9df10100d894364ceca3cb645c852b20938c00b3
Author: Eno Thereska 
Date:   2017-03-21T18:46:34Z

Initial test




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

2017-03-21 Thread Dong Lin

Hey Jun,

Thanks for the explanation. Please see below my thoughts.

10. I see. So you are concerned with the potential implementation
complexity which I wasn't aware of. I think it is OK not to do log
cleaning on the .move log since there can be only one such log in each
directory. I have updated the KIP to specify this:

"The log segments in topicPartition.move directory will be subject to log
truncation, log retention in the same way as the log segments in the source
log directory. But we may not do log cleaning on the topicPartition.move to
simplify the implementation."

11.2 Now I get your point. I think we have slightly different expectation
of the order in which the reassignment tools updates reassignment node in
ZK and sends ChangeReplicaDirRequest.

I think the reassignment tool should first create reassignment znode and
then keep sending ChangeReplicaDirRequest until success. I think sending
ChangeReplicaDirRequest before updating znode has negligible impact on the
chance that the broker processes ChangeReplicaDirRequest before
LeaderAndIsrRequest from controller, because the time for controller to
receive ZK notification, handle state machine changes and send
LeaderAndIsrRequests should be much longer than the time for reassignment
tool to setup connection with broker and send ChangeReplicaDirRequest. Even
if broker receives LeaderAndIsrRequest a bit sooner, the data in the
original replica should be smaller enough for .move log to catch up very
quickly, so that broker can swap the log soon after it receives
ChangeReplicaDirRequest -- otherwise the intra.broker.throttled.rate is
probably too small. Does this address your concern with the performance?

One concern with the suggested approach is that the ChangeReplicaDirRequest
may be lost if broker crashes before it creates the replica. I agree it is
rare. But it will be confusing when it happens. Operators would have to
keep verifying reassignment and possibly retry execution until success if
they want to make sure that the ChangeReplicaDirRequest is executed.

Thanks,
Dong

On Tue, Mar 21, 2017 at 8:37 AM, Jun Rao  wrote:

> Hi, Dong,
>
> 10. I was mainly concerned about the additional complexity needed to
> support log cleaning in the .move log. For example, LogToClean is keyed off
> TopicPartition. To be able to support cleaning different instances of the
> same partition, we need additional logic. I am not how much additional
> complexity is needed and whether it's worth it. If we don't do log cleaning
> at all on the .move log, then we don't have to change the log cleaner's
> code.
>
> 11.2 I was thinking of the following flow. In the execute phase, the
> reassignment tool first issues a ChangeReplicaDirRequest to brokers where
> new replicas will be created. The brokers remember the mapping and return a
> successful code. The reassignment tool then initiates the cross broker
> movement through the controller. In the verify phase, in addition to
> checking the replica assignment at the brokers, it issues
> DescribeDirsRequest to check the replica to log dirs mapping. For each
> partition in the response, the broker returns a state to indicate whether
> the replica is final, temporary or pending. If all replicas are in the
> final state, the tool checks if all replicas are in the expected log dirs.
> If they are not, output a warning (and perhaps suggest the users to move
> the data again). However, this should be rare.
>
> Thanks,
>
> Jun
>
>
> On Mon, Mar 20, 2017 at 10:46 AM, Dong Lin  wrote:
>
> > Hey Jun,
> >
> > Thanks for the response! It seems that we have only two remaining issues.
> > Please see my reply below.
> >
> > On Mon, Mar 20, 2017 at 7:45 AM, Jun Rao  wrote:
> >
> > > Hi, Dong,
> > >
> > > Thanks for the update. A few replies inlined below.
> > >
> > > On Thu, Mar 16, 2017 at 12:28 AM, Dong Lin 
> wrote:
> > >
> > > > Hey Jun,
> > > >
> > > > Thanks for your comment! Please see my reply below.
> > > >
> > > > On Wed, Mar 15, 2017 at 9:45 PM, Jun Rao  wrote:
> > > >
> > > > > Hi, Dong,
> > > > >
> > > > > Thanks for the reply.
> > > > >
> > > > > 10. Could you comment on that?
> > > > >
> > > >
> > > > Sorry, I missed that comment.
> > > >
> > > > Good point. I think the log segments in topicPartition.move directory
> > > will
> > > > be subject to log truncation, log retention and log cleaning in the
> > same
> > > > way as the log segments in the source log directory. I just specified
> > > this
> > > > inthe KIP.
> > > >
> > > >
> > > This is ok, but doubles the overhead of log cleaning. We probably want
> to
> > > think a bit more on this.
> > >
> >
> > I think this is OK because the number of replicas that are being moved is
> > limited by the number of ReplicaMoveThread. The default number of
> > ReplicaMoveThread is the number of log directories, which mean we incur
> > these overhead for at most one replica per log directory at any time.
> > Suppose there are most than 100 replica in any log directory, the
> incre

[jira] [Comment Edited] (KAFKA-4674) Frequent ISR shrinking and expanding and disconnects among brokers

2017-03-21 Thread mjuarez (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-4674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15879553#comment-15879553
 ] 

mjuarez edited comment on KAFKA-4674 at 3/21/17 6:01 PM:
-

This is happening to us in one our clusters in prod today, across multiple 
topics, and basically on all nodes.  The most impacted topic/partitions seem to 
be the ones that have the highest volume, but every topic that has some 
activity is being impacted.

Unlike [~344277...@qq.com], we are not seeing any exceptions in the logs.  See 
attached broker log file.  Please let me know if you need more log files and/or 
details.

[Update 20170321] Just wanted to clarify that this is on Kafka 0.10.0.1.  We 
thought we would see this fixed after upgrading to Kafka 0.10.1.1, but that one 
is experiencing the same issues.  At this point, we're hesitant to upgrade.  
Does anybody have any idea if the fixes for 0.10.2.0 make this issue less 
frequent, or maybe even eliminate it?


was (Author: mjuarez):
This is happening to us in one our clusters in prod today, across multiple 
topics, and basically on all nodes.  The most impacted topic/partitions seem to 
be the ones that have the highest volume, but every topic that has some 
activity is being impacted.

Unlike [~344277...@qq.com], we are not seeing any exceptions in the logs.  See 
attached broker log file.  Please let me know if you need more log files and/or 
details.

> Frequent ISR shrinking and expanding and disconnects among brokers
> --
>
> Key: KAFKA-4674
> URL: https://issues.apache.org/jira/browse/KAFKA-4674
> Project: Kafka
>  Issue Type: Bug
>  Components: controller, core
>Affects Versions: 0.10.0.1
> Environment: OS: Redhat Linux 2.6.32-431.el6.x86_64
> JDK: 1.8.0_45
>Reporter: Kaiming Wan
> Attachments: controller.log.rar, kafkabroker.20170221.log.zip, 
> server.log.2017-01-11-14, zookeeper.out.2017-01-11.log
>
>
> We use a kafka cluster with 3 brokers in production environment. It works 
> well for several month. Recently, we get the UnderReplicatedPartitions>0 
> warning mail. When we check the log, we find that the partition is always 
> experience ISR shrinking and expanding. And the disconnection exception can 
> be found in controller's log.
> We also found some deviant output in zookeeper's log which point to a 
> consumer(using old API depends on zookeeper ) which has stopped its work with 
> many lags.
> Actually, it is not the first time we encounter this problem. When we 
> first met this problem, we also found the same phenomenon and the log output. 
> We solve the problem by deleting the consumer node info in zookeeper. Then 
> everything goes well.
> However, this time, after we deleting the consumer which already have 
> large lag, the frequent ISR shrinking and expanding didn't stop for a very 
> long time(serveral hours). Though, the issue didn't affect our consumer and 
> producer, we think it will make our cluster unstable. So at last, we solve 
> this problem by restart the controller broker.
> And now I wander what cause this problem. I check the source code and 
> only know poll timeout will cause disconnection and ISR shrinking. Is the 
> issue related to zookeeper because it will not hold too many metadata 
> modification and make the replication fetch thread take more time?
> I upload the log file in the attachment.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

2017-03-21 Thread Jun Rao

Hi, Dong,

10. I was mainly concerned about the additional complexity needed to
support log cleaning in the .move log. For example, LogToClean is keyed off
TopicPartition. To be able to support cleaning different instances of the
same partition, we need additional logic. I am not how much additional
complexity is needed and whether it's worth it. If we don't do log cleaning
at all on the .move log, then we don't have to change the log cleaner's
code.

11.2 I was thinking of the following flow. In the execute phase, the
reassignment tool first issues a ChangeReplicaDirRequest to brokers where
new replicas will be created. The brokers remember the mapping and return a
successful code. The reassignment tool then initiates the cross broker
movement through the controller. In the verify phase, in addition to
checking the replica assignment at the brokers, it issues
DescribeDirsRequest to check the replica to log dirs mapping. For each
partition in the response, the broker returns a state to indicate whether
the replica is final, temporary or pending. If all replicas are in the
final state, the tool checks if all replicas are in the expected log dirs.
If they are not, output a warning (and perhaps suggest the users to move
the data again). However, this should be rare.

Thanks,

Jun


On Mon, Mar 20, 2017 at 10:46 AM, Dong Lin  wrote:

> Hey Jun,
>
> Thanks for the response! It seems that we have only two remaining issues.
> Please see my reply below.
>
> On Mon, Mar 20, 2017 at 7:45 AM, Jun Rao  wrote:
>
> > Hi, Dong,
> >
> > Thanks for the update. A few replies inlined below.
> >
> > On Thu, Mar 16, 2017 at 12:28 AM, Dong Lin  wrote:
> >
> > > Hey Jun,
> > >
> > > Thanks for your comment! Please see my reply below.
> > >
> > > On Wed, Mar 15, 2017 at 9:45 PM, Jun Rao  wrote:
> > >
> > > > Hi, Dong,
> > > >
> > > > Thanks for the reply.
> > > >
> > > > 10. Could you comment on that?
> > > >
> > >
> > > Sorry, I missed that comment.
> > >
> > > Good point. I think the log segments in topicPartition.move directory
> > will
> > > be subject to log truncation, log retention and log cleaning in the
> same
> > > way as the log segments in the source log directory. I just specified
> > this
> > > inthe KIP.
> > >
> > >
> > This is ok, but doubles the overhead of log cleaning. We probably want to
> > think a bit more on this.
> >
>
> I think this is OK because the number of replicas that are being moved is
> limited by the number of ReplicaMoveThread. The default number of
> ReplicaMoveThread is the number of log directories, which mean we incur
> these overhead for at most one replica per log directory at any time.
> Suppose there are most than 100 replica in any log directory, the increase
> in overhead is less than 1%.
>
> Another way to look at this is that this is no worse than replica
> reassignment. When we reassign replica from one broker to another, we will
> double the overhread of log cleaning in the cluster for this replica. If we
> are OK with this then we are OK with replica movement between log
> directories.
>
>
> >
> >
> > >
> > > >
> > > > 11.2 "I am concerned that the ChangeReplicaDirRequest would be lost
> if
> > > > broker
> > > > restarts after it sends ChangeReplicaDirResponse but before it
> receives
> > > > LeaderAndIsrRequest."
> > > >
> > > > In that case, the reassignment tool could detect that through
> > > > DescribeDirsRequest
> > > > and issue ChangeReplicaDirRequest again, right? In the common case,
> > this
> > > is
> > > > probably not needed and we only need to write each replica once.
> > > >
> > > > My main concern with the approach in the current KIP is that once a
> new
> > > > replica is created in the wrong log dir, the cross log directory
> > movement
> > > > may not catch up until the new replica is fully bootstrapped. So, we
> > end
> > > up
> > > > writing the data for the same replica twice.
> > > >
> > >
> > > I agree with your concern. My main concern is that it is a bit weird if
> > > ChangeReplicaDirResponse can not guarantee success and the tool needs
> to
> > > rely on DescribeDirResponse to see if it needs to send
> > > ChangeReplicaDirRequest again.
> > >
> > > How about this: If broker doesn't not have already replica created for
> > the
> > > specified topicParition when it receives ChangeReplicaDirRequest, it
> will
> > > reply ReplicaNotAvailableException AND remember (replica, destination
> log
> > > directory) pair in memory to create the replica in the specified log
> > > directory.
> > >
> > >
> > I am not sure if returning ReplicaNotAvailableException is useful? What
> > will the client do on receiving ReplicaNotAvailableException in this
> case?
> >
> > Perhaps we could just replace the is_temporary field in
> > DescribeDirsRresponsePartition with a state field. We can use 0 to
> indicate
> > the partition is created, 1 to indicate the partition is temporary and 2
> to
> > indicate that the partition is pending.
> >
>
> ReplicaNotAvailableException is use

[GitHub] kafka pull request #2718: MINOR: log start and end offsets for state store r...

2017-03-21 Thread dguy

GitHub user dguy opened a pull request:

https://github.com/apache/kafka/pull/2718

MINOR: log start and end offsets for state store restoration

Debug loggin of the start and end offsets used during state store 
restoration

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dguy/kafka log-restore-offsets

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2718.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2718


commit c9d16b429b611e235291e917921420fa2139a435
Author: Damian Guy 
Date:   2017-03-21T17:08:31Z

log start and end offsets for state store restoration




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [VOTE] KIP-124: Request rate quotas

2017-03-21 Thread Roger Hoover

Rajini,

This is great.  Thank you.  +1 (non-binding)

Roger

On Tue, Mar 21, 2017 at 8:55 AM, Ismael Juma  wrote:

> Rajini,
>
> Thanks for the proposal and for addressing the (sometimes contradictory)
> feedback. :) +1 (binding) from me.
>
> Ismael
>
> On Mon, Mar 20, 2017 at 2:47 PM, Rajini Sivaram 
> wrote:
>
> > I would like to initiate the voting process for KIP-124:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 124+-+Request+rate+quotas
> >
> > The KIP proposes to add request processing time quota to limit CPU
> > utilization at **, ** or ** levels
> > similar to the current bandwidth limiting quotas. The quota takes into
> > account the total time spent by the user/client on request handler and
> > network threads within a quota window. Quota will be configured as a
> > per-thread percentage value, which typically indicates the percentage of
> > CPU cores allocated to the user/client.
> >
> > The discussion thread is here:
> >
> >
> > https://lists.apache.org/thread.html/46c7bbc8f381ebe718b3cce6ed8bdf
> > 3745df22b0bd88020d70c99813@%3Cdev.kafka.apache.org%3E
> >
> > Many thanks to everyone for the feedback and suggestions so far.
> >
> > Regards,
> >
> > Rajini
> >
>

[jira] [Commented] (KAFKA-4878) Kafka Connect does not log connector configuration errors

2017-03-21 Thread Gwen Shapira (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15934854#comment-15934854
 ] 

Gwen Shapira commented on KAFKA-4878:
-

[~original-brownbear] - Thank you. Let me know if you need help / pointers.

> Kafka Connect does not log connector configuration errors
> -
>
> Key: KAFKA-4878
> URL: https://issues.apache.org/jira/browse/KAFKA-4878
> Project: Kafka
>  Issue Type: Bug
>Reporter: Gwen Shapira
>Assignee: Armin Braun
>Priority: Blocker
> Fix For: 0.10.2.1
>
>
> Currently, on connector configuration error, Kafka Connect (both distributed 
> and stand alone) logs:
> org.apache.kafka.connect.runtime.rest.errors.BadRequestException: Connector 
> configuration is invalid (use the endpoint `/{connectorType}/config/validate` 
> to get a full list of errors)
> This is annoying because:
> 1. If I'm using stand-alone mode, I may have configured my connector via 
> configuration file and I don't want to know about the REST API at all.
> 2. The output of validate is rather annoying
> What I'd like to see in the output is:
> 1. number of errors in my configuration
> 2. at least one error, preferably all of them



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #2717: MINOR: remove unused log field from KStreamTransfo...

2017-03-21 Thread dguy

GitHub user dguy opened a pull request:

https://github.com/apache/kafka/pull/2717

MINOR: remove unused log field from KStreamTransformValuesProcessor

remove unused log field from KStreamTransformValuesProcessor

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dguy/kafka remove-unused-log-para

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2717.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2717


commit 30d914753759fcb73339a5df4e3529ac28dd7182
Author: Damian Guy 
Date:   2017-03-21T16:29:55Z

remove unused logger




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [VOTE] KIP-124: Request rate quotas

2017-03-21 Thread Ismael Juma

Rajini,

Thanks for the proposal and for addressing the (sometimes contradictory)
feedback. :) +1 (binding) from me.

Ismael

On Mon, Mar 20, 2017 at 2:47 PM, Rajini Sivaram 
wrote:

> I would like to initiate the voting process for KIP-124:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 124+-+Request+rate+quotas
>
> The KIP proposes to add request processing time quota to limit CPU
> utilization at **, ** or ** levels
> similar to the current bandwidth limiting quotas. The quota takes into
> account the total time spent by the user/client on request handler and
> network threads within a quota window. Quota will be configured as a
> per-thread percentage value, which typically indicates the percentage of
> CPU cores allocated to the user/client.
>
> The discussion thread is here:
>
>
> https://lists.apache.org/thread.html/46c7bbc8f381ebe718b3cce6ed8bdf
> 3745df22b0bd88020d70c99813@%3Cdev.kafka.apache.org%3E
>
> Many thanks to everyone for the feedback and suggestions so far.
>
> Regards,
>
> Rajini
>

[jira] [Commented] (KAFKA-4927) KStreamsTestDriver fails with NPE when KStream.to() sinks are used

2017-03-21 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-4927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15934793#comment-15934793
 ] 

ASF GitHub Bot commented on KAFKA-4927:
---

GitHub user wimvanleuven opened a pull request:

https://github.com/apache/kafka/pull/2716

KAFKA-4927 : KStreamsTestDriver fails with NPE when KStream.to() sinks are 
used

a KStream.to() sink is also a topic
... so the KStreamTestDriver to fetch it when required

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/wimvanleuven/kafka KAFKA-4927

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2716.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2716






> KStreamsTestDriver fails with NPE when KStream.to() sinks are used
> --
>
> Key: KAFKA-4927
> URL: https://issues.apache.org/jira/browse/KAFKA-4927
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Wim Van Leuven
>  Labels: test
> Fix For: 0.10.2.0
>
>
> *Context*
> KStreamsTestDriver allows to build integration tests of KStreamsTopologies. 
> This also includes topologies that sink data into outgoing topics by calling 
> the KStream.to() methods.
> *Problem*
> When a topic is added as a sink, the KStreamsTestDriver fails with 
> NullPointerExceptions. 
> *Solution*
> BugFix the method KStreamTestDriver.process() method to also lookup a topic  
> by topicName as a sink when a source has not been found.
> *PullRequest*



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #2716: KAFKA-4927 : KStreamsTestDriver fails with NPE whe...

2017-03-21 Thread wimvanleuven

GitHub user wimvanleuven opened a pull request:

https://github.com/apache/kafka/pull/2716

KAFKA-4927 : KStreamsTestDriver fails with NPE when KStream.to() sinks are 
used

a KStream.to() sink is also a topic
... so the KStreamTestDriver to fetch it when required

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/wimvanleuven/kafka KAFKA-4927

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2716.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2716






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-4927) KStreamsTestDriver fails with NPE when KStream.to() sinks are used

2017-03-21 Thread Wim Van Leuven (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-4927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wim Van Leuven updated KAFKA-4927:
--
Summary: KStreamsTestDriver fails with NPE when KStream.to() sinks are used 
 (was: KStreamsTestDriver fails with NPE when Topic sinks are used)

> KStreamsTestDriver fails with NPE when KStream.to() sinks are used
> --
>
> Key: KAFKA-4927
> URL: https://issues.apache.org/jira/browse/KAFKA-4927
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Wim Van Leuven
>  Labels: test
> Fix For: 0.10.2.0
>
>
> *Context*
> KStreamsTestDriver allows to build integration tests of KStreamsTopologies. 
> This also includes topologies that sink data into outgoing topics by calling 
> the KStream.to() methods.
> *Problem*
> When a topic is added as a sink, the KStreamsTestDriver fails with 
> NullPointerExceptions. 
> *Solution*
> BugFix the method KStreamTestDriver.process() method to also lookup a topic  
> by topicName as a sink when a source has not been found.
> *PullRequest*



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-4927) KStreamsTestDriver fails with NPE when Topic sinks are used

2017-03-21 Thread Wim Van Leuven (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-4927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wim Van Leuven updated KAFKA-4927:
--
Summary: KStreamsTestDriver fails with NPE when Topic sinks are used  (was: 
KStreamsTestDriver fails with NullPointerException when Topic sinks are used)

> KStreamsTestDriver fails with NPE when Topic sinks are used
> ---
>
> Key: KAFKA-4927
> URL: https://issues.apache.org/jira/browse/KAFKA-4927
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Wim Van Leuven
>  Labels: test
> Fix For: 0.10.2.0
>
>
> *Context*
> KStreamsTestDriver allows to build integration tests of KStreamsTopologies. 
> This also includes topologies that sink data into outgoing topics by calling 
> the KStream.to() methods.
> *Problem*
> When a topic is added as a sink, the KStreamsTestDriver fails with 
> NullPointerExceptions. 
> *Solution*
> BugFix the method KStreamTestDriver.process() method to also lookup a topic  
> by topicName as a sink when a source has not been found.
> *PullRequest*



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [VOTE] KIP-124: Request rate quotas

2017-03-21 Thread Jun Rao

Hi, Rajini,

Thanks for the proposal. +1 from me.

Jun

On Mon, Mar 20, 2017 at 7:47 AM, Rajini Sivaram 
wrote:

> I would like to initiate the voting process for KIP-124:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 124+-+Request+rate+quotas
>
> The KIP proposes to add request processing time quota to limit CPU
> utilization at **, ** or ** levels
> similar to the current bandwidth limiting quotas. The quota takes into
> account the total time spent by the user/client on request handler and
> network threads within a quota window. Quota will be configured as a
> per-thread percentage value, which typically indicates the percentage of
> CPU cores allocated to the user/client.
>
> The discussion thread is here:
>
>
> https://lists.apache.org/thread.html/46c7bbc8f381ebe718b3cce6ed8bdf
> 3745df22b0bd88020d70c99813@%3Cdev.kafka.apache.org%3E
>
> Many thanks to everyone for the feedback and suggestions so far.
>
> Regards,
>
> Rajini
>

[jira] [Updated] (KAFKA-4791) Kafka Streams - unable to add state stores when using wildcard topics on the source

2017-03-21 Thread Michael Noll (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-4791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Noll updated KAFKA-4791:

Affects Version/s: 0.10.2.0

> Kafka Streams - unable to add state stores when using wildcard topics on the 
> source
> ---
>
> Key: KAFKA-4791
> URL: https://issues.apache.org/jira/browse/KAFKA-4791
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.1.1, 0.10.2.0
> Environment: Java 8
>Reporter: Bart Vercammen
>Assignee: Bill Bejeck
>
> I'm trying to build up a topology (using TopologyBuilder) with following 
> components :
> {code}
> new TopologyBuilder()
>   .addSource("ingest", Pattern.compile( ... ))
>   .addProcessor("myprocessor", ..., "ingest")
>   .addStateStore(dataStore, "myprocessor")
> {code}
> Somehow this does not seem to work.
> When creating the topology with exact topic names, all works fine, but it 
> seems not possible to attach state stores when using wildcard topics on the 
> sources.
> Inside {{addStateStore}}, the processor gets connected to the state store 
> with {{connectProcessorAndStateStore}}, and there it will try to connect the 
> state store with the source topics from the processor: 
> {{connectStateStoreNameToSourceTopics}}  
> Here lies the problem: 
> {code}
> private Set findSourceTopicsForProcessorParents(String [] 
> parents) {
> final Set sourceTopics = new HashSet<>();
> for (String parent : parents) {
> NodeFactory nodeFactory = nodeFactories.get(parent);
> if (nodeFactory instanceof SourceNodeFactory) {
> sourceTopics.addAll(Arrays.asList(((SourceNodeFactory) 
> nodeFactory).getTopics()));
> } else if (nodeFactory instanceof ProcessorNodeFactory) {
> 
> sourceTopics.addAll(findSourceTopicsForProcessorParents(((ProcessorNodeFactory)
>  nodeFactory).parents));
> }
> }
> return sourceTopics;
> }
> {code}
> The call to {{sourceTopics.addAll(Arrays.asList(((SourceNodeFactory) 
> nodeFactory).getTopics()))}} will fail as there are no topics inside the 
> {{SourceNodeFactory}} object, only a pattern ({{.getTopics}} returns null)
> I also tried to search for some unit tests inside the Kafka Streams project 
> that cover this scenario, but alas, I was not able to find any.
> Only some tests on state stores with exact topic names, and some tests on 
> wildcard topics, but no combination of both ...



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (KAFKA-4928) Add integration test for DumpLogSegments

2017-03-21 Thread Ismael Juma (JIRA)

Ismael Juma created KAFKA-4928:
--

 Summary: Add integration test for DumpLogSegments
 Key: KAFKA-4928
 URL: https://issues.apache.org/jira/browse/KAFKA-4928
 Project: Kafka
  Issue Type: Test
Reporter: Ismael Juma
 Fix For: 0.11.0.0


DumpLogSegments is an important tool to analyse log files, but we have no JUnit 
tests for it. It would be good to have some tests that verify that the output 
is sane for a populated log.

Our system tests call DumpLogSegments, but we should be able to detect 
regressions via the JUnit test suite.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[DISCUSS] KAFKA-4911 Add ProducerRecord to ProducerInterceptor#onAcknowledgement

2017-03-21 Thread Kevin Conaway

Greetings all,

Please correct me if this is not the right place to discuss this.  I filed
a jira about this but I figured the mailing list might encourage better
collaboration.

I'd like to propose adding the ProducerRecord field to the
onAcknowledgement callback in the ProducerInterceptor interface that was
added in KAFKA-3162

The issue I'm having is that its not possible to determine if a specific
record was sent (based on the record content or key), only that some record
was sent based on the RecordMetadata.  From reading the KIP and the initial
mailing list discussion, it seems that the design was modeled after the
existing Producer.Callback interface. This model makes sense given that
callbacks are typically anonymous functions where you would have access to
the producer record:

ProducerRecord record = ...;
producer.send(record, new Callback() {
  @Override
  public void onCompletion(RecordMetadata recordMetadata, Exception e) {
// do something with record
  }
})

However, based on my understanding of the interceptor design, the
interceptor should be thread safe and thus stateless. Even if you wanted to
make it stateful, its not easy to tie together a record from onSend to
onAcknowledgement.  Exception handling also suffers from this if you want
to tie the exception to the specific record that failed

With that in mind, is there a way that the producer interceptor API can be
modified so that implementers can be aware of the key & content of the
record that was acknowledged or failed?

Thanks,

-- 
Kevin Conaway
http://www.linkedin.com/pub/kevin-conaway/7/107/580/
https://github.com/kevinconaway

[jira] [Created] (KAFKA-4927) KStreamsTestDriver fails with NullPointerException when Topic sinks are used

2017-03-21 Thread Wim Van Leuven (JIRA)

Wim Van Leuven created KAFKA-4927:
-

 Summary: KStreamsTestDriver fails with NullPointerException when 
Topic sinks are used
 Key: KAFKA-4927
 URL: https://issues.apache.org/jira/browse/KAFKA-4927
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 0.10.2.0
Reporter: Wim Van Leuven
 Fix For: 0.10.2.0


*Context*
KStreamsTestDriver allows to build integration tests of KStreamsTopologies. 
This also includes topologies that sink data into outgoing topics by calling 
the KStream.to() methods.

*Problem*
When a topic is added as a sink, the KStreamsTestDriver fails with 
NullPointerExceptions. 

*Solution*
BugFix the method KStreamTestDriver.process() method to also lookup a topic  by 
topicName as a sink when a source has not been found.

*PullRequest*




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: kafka-trunk-jdk8 #1363

2017-03-21 Thread Apache Jenkins Server

See 


Changes:

[ismael] MINOR: Fix zookeeper-security-migration documentation example

[ismael] KAFKA-4594; Annotate integration tests and provide gradle build targets

--
[...truncated 158.78 KB...]

kafka.security.auth.SimpleAclAuthorizerTest > testLoadCache STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testLoadCache PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithCollision STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithCollision PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokerListWithNoTopics STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokerListWithNoTopics PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testAutoCreateTopic STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > testAutoCreateTopic PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testGetAllTopicMetadata 
STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > testGetAllTopicMetadata 
PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testBasicTopicMetadata 
STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > testBasicTopicMetadata PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED

kafka.integration.PrimitiveApiTest > testMultiProduce STARTED

kafka.integration.PrimitiveApiTest > testMultiProduce PASSED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch STARTED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch PASSED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize 
STARTED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize PASSED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests STARTED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests PASSED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch STARTED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch PASSED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression STARTED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression PASSED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic STARTED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic PASSED

kafka.integration.PrimitiveApiTest > testEmptyFetchRequest STARTED

kafka.integration.PrimitiveApiTest > testEmptyFetchRequest PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack PASSED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopicWithCollision 
STARTED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopicWithCollision 
PASSED

kafka.integration.PlaintextTopicMetadataTest > testAliveBrokerListWithNoTopics 
STARTED

kafka.integration.PlaintextTopicMetadataTest > testAliveBrokerListWithNoTopics 
PASSED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopic STARTED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopic PASSED

kafka.integration.PlaintextTopicMetadataTest > testGetAllTopicMetadata STARTED

kafka.integration.PlaintextTopicMetadataTest > testGetAllTopicMetadata PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.integration.PlaintextTopicMetadataTest > testBasicTopicMetadata STARTED

kafka.integration.PlaintextTopicMetadataTest > testBasicTopicMetadata PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrok

Jenkins build is back to normal : kafka-trunk-jdk7 #2026

2017-03-21 Thread Apache Jenkins Server

See

Build failed in Jenkins: kafka-trunk-jdk8 #1362

2017-03-21 Thread Apache Jenkins Server

See 


Changes:

[wangguoz] KAFKA-4848: Fix retryWithBackoff deadlock issue

--
[...truncated 1.60 MB...]
org.apache.kafka.streams.state.internals.SerializedKeyValueIteratorTest > 
shouldReturnTrueOnHasNextWhenMoreResults PASSED

org.apache.kafka.streams.state.internals.SerializedKeyValueIteratorTest > 
shouldThrowUnsupportedOperationOnRemove STARTED

org.apache.kafka.streams.state.internals.SerializedKeyValueIteratorTest > 
shouldThrowUnsupportedOperationOnRemove PASSED

org.apache.kafka.streams.state.internals.SerializedKeyValueIteratorTest > 
shouldPeekNextKey STARTED

org.apache.kafka.streams.state.internals.SerializedKeyValueIteratorTest > 
shouldPeekNextKey PASSED

org.apache.kafka.streams.state.internals.SerializedKeyValueIteratorTest > 
shouldReturnFalseOnHasNextWhenNoMoreResults STARTED

org.apache.kafka.streams.state.internals.SerializedKeyValueIteratorTest > 
shouldReturnFalseOnHasNextWhenNoMoreResults PASSED

org.apache.kafka.streams.state.internals.CompositeReadOnlySessionStoreTest > 
shouldNotGetValueFromOtherStores STARTED

org.apache.kafka.streams.state.internals.CompositeReadOnlySessionStoreTest > 
shouldNotGetValueFromOtherStores PASSED

org.apache.kafka.streams.state.internals.CompositeReadOnlySessionStoreTest > 
shouldReturnEmptyIteratorIfNoData STARTED

org.apache.kafka.streams.state.internals.CompositeReadOnlySessionStoreTest > 
shouldReturnEmptyIteratorIfNoData PASSED

org.apache.kafka.streams.state.internals.CompositeReadOnlySessionStoreTest > 
shouldThrowInvalidStateStoreExceptionIfFetchThrows STARTED

org.apache.kafka.streams.state.internals.CompositeReadOnlySessionStoreTest > 
shouldThrowInvalidStateStoreExceptionIfFetchThrows PASSED

org.apache.kafka.streams.state.internals.CompositeReadOnlySessionStoreTest > 
shouldFindValueForKeyWhenMultiStores STARTED

org.apache.kafka.streams.state.internals.CompositeReadOnlySessionStoreTest > 
shouldFindValueForKeyWhenMultiStores PASSED

org.apache.kafka.streams.state.internals.CompositeReadOnlySessionStoreTest > 
shouldFetchResulstFromUnderlyingSessionStore STARTED

org.apache.kafka.streams.state.internals.CompositeReadOnlySessionStoreTest > 
shouldFetchResulstFromUnderlyingSessionStore PASSED

org.apache.kafka.streams.state.internals.CompositeReadOnlySessionStoreTest > 
shouldThrowInvalidStateStoreExceptionOnRebalance STARTED

org.apache.kafka.streams.state.internals.CompositeReadOnlySessionStoreTest > 
shouldThrowInvalidStateStoreExceptionOnRebalance PASSED

org.apache.kafka.streams.state.internals.RocksDBSessionStoreTest > shouldRemove 
STARTED

org.apache.kafka.streams.state.internals.RocksDBSessionStoreTest > shouldRemove 
PASSED

org.apache.kafka.streams.state.internals.RocksDBSessionStoreTest > 
shouldFindValuesWithinMergingSessionWindowRange STARTED

org.apache.kafka.streams.state.internals.RocksDBSessionStoreTest > 
shouldFindValuesWithinMergingSessionWindowRange PASSED

org.apache.kafka.streams.state.internals.RocksDBSessionStoreTest > 
shouldFetchAllSessionsWithSameRecordKey STARTED

org.apache.kafka.streams.state.internals.RocksDBSessionStoreTest > 
shouldFetchAllSessionsWithSameRecordKey PASSED

org.apache.kafka.streams.state.internals.RocksDBSessionStoreTest > 
shouldFindSessionsToMerge STARTED

org.apache.kafka.streams.state.internals.RocksDBSessionStoreTest > 
shouldFindSessionsToMerge PASSED

org.apache.kafka.streams.state.internals.RocksDBSessionStoreTest > 
shouldPutAndFindSessionsInRange STARTED

org.apache.kafka.streams.state.internals.RocksDBSessionStoreTest > 
shouldPutAndFindSessionsInRange PASSED

org.apache.kafka.streams.state.internals.WindowStoreUtilsTest > 
testSerialization STARTED

org.apache.kafka.streams.state.internals.WindowStoreUtilsTest > 
testSerialization PASSED

org.apache.kafka.streams.state.StoresTest > 
shouldCreateInMemoryStoreSupplierWithLoggedConfig STARTED

org.apache.kafka.streams.state.StoresTest > 
shouldCreateInMemoryStoreSupplierWithLoggedConfig PASSED

org.apache.kafka.streams.state.StoresTest > 
shouldCreatePersistenStoreSupplierNotLogged STARTED

org.apache.kafka.streams.state.StoresTest > 
shouldCreatePersistenStoreSupplierNotLogged PASSED

org.apache.kafka.streams.state.StoresTest > 
shouldCreatePersistenStoreSupplierWithLoggedConfig STARTED

org.apache.kafka.streams.state.StoresTest > 
shouldCreatePersistenStoreSupplierWithLoggedConfig PASSED

org.apache.kafka.streams.state.StoresTest > 
shouldCreateInMemoryStoreSupplierNotLogged STARTED

org.apache.kafka.streams.state.StoresTest > 
shouldCreateInMemoryStoreSupplierNotLogged PASSED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testShouldReadFromRegexAndNamedTopics STARTED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testShouldReadFromRegexAndNamedTopics PASSED

org.apache.kafka.streams.integration.RegexSourceIntegrationTest > 
testRegexMatchesTopicsAWhenCreated STARTE

[GitHub] kafka pull request #2695: KAFKA-4594: Annotate integration tests and provide...

2017-03-21 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2695


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #2714: Fix ZKSec Migrate example

2017-03-21 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2714


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

76 matches

Mail list logo