Build failed in Jenkins: kafka-trunk-jdk7 #3217

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-6534: Enforce a rebalance in the next poll call when encounter

--
[...truncated 608.37 KB...]

kafka.integration.TopicMetadataTest > classMethod FAILED
java.lang.AssertionError: Found unexpected threads during @BeforeClass, 
allThreads=Set(ThrottledRequestReaper-Produce, SessionTracker, 
kafka-request-handler-7, admin-client-network-thread, Reference Handler, 
kafka-network-thread-0-ListenerName(PLAINTEXT)-PLAINTEXT-0, 
kafka-request-handler-1, ThrottledRequestReaper-Request, 
kafka-network-thread-0-ListenerName(PLAINTEXT)-PLAINTEXT-2, 
ExpirationReaper-0-Heartbeat, kafka-request-handler-3, SyncThread:0, 
SensorExpiryThread, ProcessThread(sid:0 cport:36937):, 
kafka-log-cleaner-thread-0, kafka-scheduler-8, metrics-meter-tick-thread-1, 
Test worker-SendThread(127.0.0.1:36868), Signal Dispatcher, 
ExpirationReaper-0-Rebalance, controller-event-thread, 
transaction-log-manager-0, /0:0:0:0:0:0:0:1:35760 to /0:0:0:0:0:0:0:1:36770 
workers Thread 2, kafka-scheduler-4, /kafka-acl-changes-event-process-thread, 
kafka-scheduler-6, kafka-scheduler-0, ExpirationReaper-0-Fetch, Finalizer, 
kafka-scheduler-2, kafka-request-handler-6, ExpirationReaper-0-Produce, 
kafka-request-handler-0, 
kafka-network-thread-0-ListenerName(PLAINTEXT)-PLAINTEXT-1, 
kafka-request-handler-2, Test worker, /config/changes-event-process-thread, 
kafka-request-handler-4, NIOServerCxn.Factory:/127.0.0.1:0, kafka-scheduler-9, 
metrics-meter-tick-thread-2, main, 
kafka-socket-acceptor-ListenerName(PLAINTEXT)-PLAINTEXT-0, 
ExpirationReaper-0-DeleteRecords, kafka-scheduler-3, Test 
worker-SendThread(127.0.0.1:36937), ThrottledRequestReaper-Fetch, 
/0:0:0:0:0:0:0:1:35760 to /0:0:0:0:0:0:0:1:36770 workers Thread 3, 
kafka-scheduler-5, Controller-0-to-broker-0-send-thread, 
ForkJoinPool-1-worker-37, kafka-scheduler-7, group-metadata-manager-0, 
LogDirFailureHandler, TxnMarkerSenderThread-0, Test worker-EventThread, 
ExpirationReaper-0-topic, kafka-scheduler-1, kafka-request-handler-5)

kafka.integration.TopicMetadataTest > classMethod STARTED

kafka.integration.TopicMetadataTest > classMethod FAILED
java.lang.AssertionError: Found unexpected threads during @AfterClass, 
allThreads=Set(ThrottledRequestReaper-Produce, SessionTracker, 
kafka-request-handler-7, admin-client-network-thread, Reference Handler, 
kafka-network-thread-0-ListenerName(PLAINTEXT)-PLAINTEXT-0, 
kafka-request-handler-1, ThrottledRequestReaper-Request, 
kafka-network-thread-0-ListenerName(PLAINTEXT)-PLAINTEXT-2, 
ExpirationReaper-0-Heartbeat, kafka-request-handler-3, SyncThread:0, 
SensorExpiryThread, ProcessThread(sid:0 cport:36937):, 
kafka-log-cleaner-thread-0, kafka-scheduler-8, metrics-meter-tick-thread-1, 
Test worker-SendThread(127.0.0.1:36868), Signal Dispatcher, 
ExpirationReaper-0-Rebalance, controller-event-thread, 
transaction-log-manager-0, /0:0:0:0:0:0:0:1:35760 to /0:0:0:0:0:0:0:1:36770 
workers Thread 2, kafka-scheduler-4, /kafka-acl-changes-event-process-thread, 
kafka-scheduler-6, kafka-scheduler-0, ExpirationReaper-0-Fetch, Finalizer, 
kafka-scheduler-2, kafka-request-handler-6, ExpirationReaper-0-Produce, 
kafka-request-handler-0, 
kafka-network-thread-0-ListenerName(PLAINTEXT)-PLAINTEXT-1, 
kafka-request-handler-2, Test worker, /config/changes-event-process-thread, 
kafka-request-handler-4, NIOServerCxn.Factory:/127.0.0.1:0, kafka-scheduler-9, 
metrics-meter-tick-thread-2, main, 
kafka-socket-acceptor-ListenerName(PLAINTEXT)-PLAINTEXT-0, 
ExpirationReaper-0-DeleteRecords, kafka-scheduler-3, Test 
worker-SendThread(127.0.0.1:36937), ThrottledRequestReaper-Fetch, 
/0:0:0:0:0:0:0:1:35760 to /0:0:0:0:0:0:0:1:36770 workers Thread 3, 
kafka-scheduler-5, Controller-0-to-broker-0-send-thread, 
ForkJoinPool-1-worker-37, kafka-scheduler-7, group-metadata-manager-0, 
LogDirFailureHandler, TxnMarkerSenderThread-0, Test worker-EventThread, 
ExpirationReaper-0-topic, kafka-scheduler-1, kafka-request-handler-5)

kafka.integration.AutoOffsetResetTest > classMethod STARTED

kafka.integration.AutoOffsetResetTest > classMethod FAILED
java.lang.AssertionError: Found unexpected threads during @BeforeClass, 
allThreads=Set(ThrottledRequestReaper-Produce, SessionTracker, 
kafka-request-handler-7, admin-client-network-thread, Reference Handler, 
kafka-network-thread-0-ListenerName(PLAINTEXT)-PLAINTEXT-0, 
kafka-request-handler-1, ThrottledRequestReaper-Request, 
kafka-network-thread-0-ListenerName(PLAINTEXT)-PLAINTEXT-2, 
ExpirationReaper-0-Heartbeat, kafka-request-handler-3, SyncThread:0, 
SensorExpiryThread, ProcessThread(sid:0 cport:36937):, 
kafka-log-cleaner-thread-0, kafka-scheduler-8, metrics-meter-tick-thread-1, 
Test worker-SendThread(127.0.0.1:36868), Signal Dispatcher, 
ExpirationReaper-0-Rebalance, controller-event-thread, 
transaction-log-manager-0, /0:0:0:0:0:0:0:1:35760 to /0:0:0:0:0:0:0:1:36770 
workers 

Build failed in Jenkins: kafka-trunk-jdk8 #2442

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-6534: Enforce a rebalance in the next poll call when encounter

--
[...truncated 416.49 KB...]

kafka.zookeeper.ZooKeeperClientTest > testZNodeChildChangeHandlerForChildChange 
STARTED

kafka.zookeeper.ZooKeeperClientTest > testZNodeChildChangeHandlerForChildChange 
PASSED

kafka.zookeeper.ZooKeeperClientTest > testGetChildrenExistingZNodeWithChildren 
STARTED

kafka.zookeeper.ZooKeeperClientTest > testGetChildrenExistingZNodeWithChildren 
PASSED

kafka.zookeeper.ZooKeeperClientTest > testSetDataExistingZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testSetDataExistingZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testMixedPipeline STARTED

kafka.zookeeper.ZooKeeperClientTest > testMixedPipeline PASSED

kafka.zookeeper.ZooKeeperClientTest > testGetDataExistingZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testGetDataExistingZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testDeleteExistingZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testDeleteExistingZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testSessionExpiry STARTED

kafka.zookeeper.ZooKeeperClientTest > testSessionExpiry PASSED

kafka.zookeeper.ZooKeeperClientTest > testSetDataNonExistentZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testSetDataNonExistentZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testDeleteNonExistentZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testDeleteNonExistentZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testExistsExistingZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testExistsExistingZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testZooKeeperStateChangeRateMetrics 
STARTED

kafka.zookeeper.ZooKeeperClientTest > testZooKeeperStateChangeRateMetrics PASSED

kafka.zookeeper.ZooKeeperClientTest > testZNodeChangeHandlerForDeletion STARTED

kafka.zookeeper.ZooKeeperClientTest > testZNodeChangeHandlerForDeletion PASSED

kafka.zookeeper.ZooKeeperClientTest > testGetAclNonExistentZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testGetAclNonExistentZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testStateChangeHandlerForAuthFailure 
STARTED

kafka.zookeeper.ZooKeeperClientTest > testStateChangeHandlerForAuthFailure 
PASSED

kafka.network.SocketServerTest > testGracefulClose STARTED

kafka.network.SocketServerTest > testGracefulClose PASSED

kafka.network.SocketServerTest > controlThrowable STARTED

kafka.network.SocketServerTest > controlThrowable PASSED

kafka.network.SocketServerTest > testRequestMetricsAfterStop STARTED

kafka.network.SocketServerTest > testRequestMetricsAfterStop PASSED

kafka.network.SocketServerTest > testConnectionIdReuse STARTED

kafka.network.SocketServerTest > testConnectionIdReuse PASSED

kafka.network.SocketServerTest > testClientDisconnectionUpdatesRequestMetrics 
STARTED

kafka.network.SocketServerTest > testClientDisconnectionUpdatesRequestMetrics 
PASSED

kafka.network.SocketServerTest > testProcessorMetricsTags STARTED

kafka.network.SocketServerTest > testProcessorMetricsTags PASSED

kafka.network.SocketServerTest > testMaxConnectionsPerIp STARTED

kafka.network.SocketServerTest > testMaxConnectionsPerIp PASSED

kafka.network.SocketServerTest > testConnectionId STARTED

kafka.network.SocketServerTest > testConnectionId PASSED

kafka.network.SocketServerTest > 
testBrokerSendAfterChannelClosedUpdatesRequestMetrics STARTED

kafka.network.SocketServerTest > 
testBrokerSendAfterChannelClosedUpdatesRequestMetrics PASSED

kafka.network.SocketServerTest > testNoOpAction STARTED

kafka.network.SocketServerTest > testNoOpAction PASSED

kafka.network.SocketServerTest > simpleRequest STARTED

kafka.network.SocketServerTest > simpleRequest PASSED

kafka.network.SocketServerTest > closingChannelException STARTED

kafka.network.SocketServerTest > closingChannelException PASSED

kafka.network.SocketServerTest > testIdleConnection STARTED

kafka.network.SocketServerTest > testIdleConnection PASSED

kafka.network.SocketServerTest > 
testClientDisconnectionWithStagedReceivesFullyProcessed STARTED

kafka.network.SocketServerTest > 
testClientDisconnectionWithStagedReceivesFullyProcessed PASSED

kafka.network.SocketServerTest > testMetricCollectionAfterShutdown STARTED

kafka.network.SocketServerTest > testMetricCollectionAfterShutdown PASSED

kafka.network.SocketServerTest > testSessionPrincipal STARTED

kafka.network.SocketServerTest > testSessionPrincipal PASSED

kafka.network.SocketServerTest > configureNewConnectionException STARTED

kafka.network.SocketServerTest > configureNewConnectionException PASSED

kafka.network.SocketServerTest > testMaxConnectionsPerIpOverrides STARTED

kafka.network.SocketServerTest > testMaxConnectionsPerIpOverrides PASSED

kafka.network.SocketServerTest > processNewResponseException STARTED

kafka.network.SocketServerTest > processNewResponseException 

[jira] [Created] (KAFKA-6596) Why we can not see any consumer offset produce rate in kafka manager

[Rajendra Jangir (JIRA)]

Rajendra Jangir created KAFKA-6596:
--

 Summary: Why we can not see any consumer offset produce rate in 
kafka manager 
 Key: KAFKA-6596
 URL: https://issues.apache.org/jira/browse/KAFKA-6596
 Project: Kafka
  Issue Type: Bug
Reporter: Rajendra Jangir


I am using kafka v0.10.2  and kafka manager for monitoring.

But I am not able to see any consumer offset producer rate.

Please help me out. Thanks in advance



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-6597) Issues with Zookeeper and Kafka startup in Windows environment

[Alex Dunayevsky (JIRA)]

Alex Dunayevsky created KAFKA-6597:
--

 Summary: Issues with Zookeeper and Kafka startup in Windows 
environment
 Key: KAFKA-6597
 URL: https://issues.apache.org/jira/browse/KAFKA-6597
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.11.0.1, 0.10.0.1, 0.9.0.1
Reporter: Alex Dunayevsky


Inability to start Zookeeper and Kafka services using standard Kafka .bat 
utilities for Windows environment

*Problem 1:* CLASSPATH string not being formed correctly in 
bin\windows\kafka-run-class.bat.
|bin\windows\zookeeper-server-start.bat config\zookeeper.properties
*** ... class not found ...|

 

*Possible working solution*:

Assign CLASSPATH correctly in *bin\windows\kafka-run-class.bat:*
|set CLASSPATH=%~dp0..\..\libs\*|

 

 

*Problem 2:* In Kafka distro the *call :concat* may crash 
*bin\windows\kafka-run-class.bat* :
|rem Classpath addition for release
call :concat %BASE_DIR%\libs\*|

*Possible working solution:*

Comment or delete those lines of code.
|rem Classpath addition for release
rem call :concat %BASE_DIR%\libs\*|

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-6598) Kafka to support using ETCD beside Zookeeper

[Sebastian Toader (JIRA)]

Sebastian Toader created KAFKA-6598:
---

 Summary: Kafka to support using ETCD beside Zookeeper
 Key: KAFKA-6598
 URL: https://issues.apache.org/jira/browse/KAFKA-6598
 Project: Kafka
  Issue Type: New Feature
  Components: clients, core
Reporter: Sebastian Toader


The current Kafka implementation is bound to {{Zookeeper}} to store its 
metadata for forming a cluster of nodes (producer/consumer/broker). 
As Kafka is becoming popular for streaming in various environments where 
{{Zookeeper}} is either not easy to deploy/manage or there are better 
alternatives to it there is a need 
to run Kafka with other metastore implementation than {{Zookeeper}}.

{{etcd}} can provide the same semantics as {{Zookeeper}} for Kafka and since 
{{etcd}} is the favorable choice in certain environments (e.g. Kubernetes) 
Kafka should be able to run with {{etcd}}.
>From the user's point of view should be straightforward to configure to use 
>{{etcd}} by just simply specifying a connection string that point to {{etcd}} 
>cluster.

To avoid introducing instability the original interfaces should be kept and 
only the low level {{Zookeeper}} API calls should be replaced with \{{etcd}} 
API calls in case Kafka is configured 
to use {{etcd}}.

On the long run (which is out of scope of this jira) there should be an 
abstract layer in Kafka which then various metastore implementations would 
implement.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-6599) KTable KTable join semantics violated when caching enabled

[Jan Filipiak (JIRA)]

Jan Filipiak created KAFKA-6599:
---

 Summary: KTable KTable join semantics violated when caching enabled
 Key: KAFKA-6599
 URL: https://issues.apache.org/jira/browse/KAFKA-6599
 Project: Kafka
  Issue Type: Bug
  Components: streams
Reporter: Jan Filipiak


Say a tuple A,B got emmited after joining and the delete for A goes into the 
cache. After that the B record would be deleted aswell. B's join processor 
would look up A and see `null` while computing for old and new value (at this 
point we can execute joiner with A beeing null and still emit something, but 
its not gonna represent the actual oldValue) Then As cache flushes it doesn't 
see B so its also not gonna put a proper oldValue. The output can then not be 
used for say any aggregate as a delete would not reliably find its old 
aggregate where it needs to be removed from filter will also break as it stopps 
null,null changes from propagating. So for me it looks pretty clearly that 
Caching with Join breaks KTable semantics. be it my new join or the currently 
existing once.

 

this if branch here

[https://github.com/apache/kafka/blob/1.0/streams/src/main/java/org/apache/kafka/streams/state/internals/CachingKeyValueStore.java#L155]

is not usefull. I think its there because when one would delegate the true case 
to the underlying. One would get proper semantics for streams, but the 
weiredest cache I've seen.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-6600) Kafka Bytes Out lags behind Kafka Bytes In on all brokers when topics replicated with 3 and flume kafka consumer.

[kaushik srinivas (JIRA)]

kaushik srinivas created KAFKA-6600:
---

 Summary: Kafka Bytes Out lags behind Kafka Bytes In on all brokers 
when topics replicated with 3 and flume kafka consumer.
 Key: KAFKA-6600
 URL: https://issues.apache.org/jira/browse/KAFKA-6600
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.10.0.1
Reporter: kaushik srinivas


Below is the setup detail,

Kafka with 3 brokers (each broker with 10 cores and 32GBmem (12 GB heap)).

Created topic with 120 partitions and replication factor 3.

Throughput per broker is ~40k msgs/sec and bytes in ~8mb/sec.

Flume kafka source is used as the consumer.

Observations:

When the replication factor is kept 1, the bytes out and bytes in stops exactly 
at same timestamp(i.e when the producer to kafka is stopped).

But when the replication factor is increased to 3, there is a time lag observed 
in bytes out compared to bytes in. Flume kafka source is pulling data slowly. 
But flume is configured with very high memory and cpu configurations.

 

Tried increasing num.replica.fetchers from default value 1 to 10, 20, 50 etc 
and replica.fetch.max.bytes from default 1MB to 10MB,20MB. But no improvement 
is found to be observed in terms of the lag.

under repplicated partitions is observed to be zero using replica manager 
metrics in jmx.

Kafka brokers were monitored for cpu and memory, cpu is being used at 3% of 
total cores max and memory used at 4gb (32 Gb configured).

Flume kafka source has overriden kafka consumer properties : 
max.partition.fetch bytes is kept at default 1MB and fetch.max.bytes is kept at 
default 52MB. Flume kafka source batch size is kept at default value 1000.
 agent.sources..kafka.consumer.fetch.max.bytes = 10485760
 agent.sources..kafka.consumer.max.partition.fetch.bytes = 10485760
 agent.sources..batchSize = 1000
 

what more tuning is needed in order to reduce the lag between bytes in and 
bytes out  at kafka brokers with replication factor 3 or is there any 
configuration missed out?

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Wiki permission

[Matthias J. Sax]

User

Panuwat Anawatmongkhon (panuwat.anawatmonghkon) 

has already permissions to create and edit wiki pages.


-Matthias


On 2/26/18 11:23 PM, Panuwat Anawatmongkhon wrote:
> Hi all,
> Can I have permission for create and edit KIP?
> My profile name is Panuwat Anawatmongkhon.
> Cheers,
> Benz
> 



signature.asc
Description: OpenPGP digital signature

Hello

[程威]

Hello,we are a Open source Chinese document organization ,now we are spending 
time on translating kafka document into Chinese. This is 1.0.0 version link 
https://github.com/apachecn/kafka-doc-zh/tree/1.0.0。
We want to further cooperation with you, let more Chinese developers more 
convenient to use the document.

[DISCUSS] KIP-264: Add a consumer metric to record raw fetch size

[Vahid S Hashemian]

Hi all,

I started a short, and hopefully trivial, KIP to add a KafkaConsumer 
metric: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-264%3A+Add+a+consumer+metric+to+record+raw+fetch+size
Your feedback and suggestions are welcome!

Thanks.
--Vahid

Re: [DISCUSS] KIP-264: Add a consumer metric to record raw fetch size

[Ted Yu]

There is raw-bytes-consumed in the sample code.

Can you add description for that ?

Looks good overall.

On Tue, Feb 27, 2018 at 12:42 PM, Vahid S Hashemian <
vahidhashem...@us.ibm.com> wrote:

> Hi all,
>
> I started a short, and hopefully trivial, KIP to add a KafkaConsumer
> metric:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 264%3A+Add+a+consumer+metric+to+record+raw+fetch+size
> Your feedback and suggestions are welcome!
>
> Thanks.
> --Vahid
>
>

Re: [DISCUSS] KIP-264: Add a consumer metric to record raw fetch size

[Vahid S Hashemian]

Thanks Ted for taking a look.

I tried to follow the naming pattern used for 'bytes-fetched' metric.
For 'bytes-fetched' we have 'bytes-consumed-rate' (as the size of 
uncompressed fetched record size).
As for the raw (compressed) fetch size, we will have 
'raw-bytes-consumed-rate'.

I'll try to clarify this in the KIP.

Thanks.
--Vahid




From:   Ted Yu 
To: dev@kafka.apache.org
Date:   02/27/2018 02:40 PM
Subject:Re: [DISCUSS] KIP-264: Add a consumer metric to record raw 
fetch size



There is raw-bytes-consumed in the sample code.

Can you add description for that ?

Looks good overall.

On Tue, Feb 27, 2018 at 12:42 PM, Vahid S Hashemian <
vahidhashem...@us.ibm.com> wrote:

> Hi all,
>
> I started a short, and hopefully trivial, KIP to add a KafkaConsumer
> metric:
> 
https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_KAFKA_KIP-2D&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc&m=Dqe9lEqBEcReN5Sc4sQLomnD-KeYa5oof-kgEpEMofk&s=SJ1n-nKKAA4IKvuMC2DuXIfwGRHJW5dl9X_4_rx1s3E&e=

> 264%3A+Add+a+consumer+metric+to+record+raw+fetch+size
> Your feedback and suggestions are welcome!
>
> Thanks.
> --Vahid
>
>

Jenkins build is back to normal : kafka-trunk-jdk7 #3218

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-253: Support in-order message delivery with partition expansion

[Dong Lin]

Hey Allen,

Thanks for the comments.

On Mon, Feb 26, 2018 at 9:27 PM, Allen Wang 
wrote:

> Hi Dong,
>
> Please see my comments inline.
>
> Thanks,
> Allen
>
> On Sun, Feb 25, 2018 at 3:33 PM, Dong Lin  wrote:
>
> > Hey Allen,
> >
> > Thanks for your comment. I will comment inline.
> >
> > On Thu, Feb 22, 2018 at 3:05 PM, Allen Wang 
> > wrote:
> >
> > > Overall this is a very useful feature. With this we can finally scale
> > keyed
> > > messages.
> > >
> > > +1 on the ability to remove partitions. This will greatly increase
> > Kafka's
> > > scalability in cloud.
> > >
> > > For example, when there is traffic increase, we can add brokers and
> > assign
> > > new partitions to the new brokers. When traffic decreases, we can mark
> > > these new partitions as read only and remove them afterwards, together
> > with
> > > the brokers that host these partitions. This will be a light-weight
> > > approach to scale a Kafka cluster compared to partition reassignment
> > where
> > > you will always have to move data.
> > >
> > > I have some suggestions:
> > >
> > > - The KIP described each step in detail which is great. However, it
> lacks
> > > the "why" part to explain the high level goal we want to achieve with
> > each
> > > step. For example, the purpose of step 5 may be described as "Make sure
> > > consumers always first finish consuming all data prior to partition
> > > expansion to enforce message ordering".
> > >
> >
> > Yeah I think this is useful. This is a non-trivial KIP and it is useful
> to
> > explain the motivation of each change to help reading. I will added
> > motivation for each change in the KIP. Please let me know if there is
> > anything else that can make the KIP more readable.
> >
> >
> > >
> > > - The rejection of produce request at partition expansion should be
> > > configurable because it does not matter for non-keyed messages. Same
> with
> > > the consumer behavior for step 5. This will ensure that for non-keyed
> > > messages, partition expansion does not add the cost of possible message
> > > drop on producer or message latency on the consumer.
> > >
> >
> > Ideally we would like to avoid adding extra configs to keep the interface
> > simple. I think the current overhead in the producer is actually very
> > small. Partition expansion or deletion should happen very infrequently.
> > Note that our producer today needs to refresh metadata whenever there is
> > leadership movement, i.e. producer will receive
> > NotLeaderForPartitionException from the old leader and keep refreshing
> > metadata until it gets the new leader of the partition, which happens
> much
> > more frequently than Partition expansion or deletion. So I am not sure we
> > should add a config to optimize this.
> >
>
> I was concerned that at high message rate, rejecting requests could lead to
> producer side buffer full and lead to unnecessary message drop on producer
> side for non-keyed messages.
>
> What about the delay on consumer? It could be significant when one consumer
> is lagging for certain partitions and all consumers in the same group have
> to wait. This delay could be significant and again unnecessary for messages
> where the order does not matter.
>

I agree this may increase delay on the producer side. Consumer is not
impacted directly and any extra delay in consumer all comes from the extra
delay in producer.

Note that when broker has leadership change, if producer's metadata is
still using the old leader, producer will see
NotLeaderForPartitionException and will have to repeatedly update
metadadata until the metadata uses the new leader. Do you think the
metadata update after partition expansion, as introduced in this KIP, is
any worse than the metadata update required during leadership change? If
not, given that our user already needs to handle or tolerate the extra
delay during leadership change, I think the extra delay after partition
expansion should be fine.



>
>
> >
> >
> >
> > > - Since we now allow adding partitions for keyed messages while
> > preserving
> > > the message ordering on the consumer side, the default producer
> > partitioner
> > > seems to be inadequate as it rehashes all keys. As part of this KIP,
> > should
> > > we also include a partitioner that better handles partition changes,
> for
> > > example, with consistent hashing?
> > >
>
> I am not sure I understand the problem with the default partitioner. Can
> > you explain a bit more why default producer partitioner is inadequate
> with
> > this KIP? And why consistent hashing can be helpful?
> >
> >
> >
> The default partitioner use this algorithm for keyed messages:
>
> Utils.toPositive(Utils.murmur2(keyBytes)) % numPartitions
>
> As you can see, when the number of partitions are changed, there will be
> complete rehash of keys to partitions. This causes a lot of changes to key
> -> partition mapping and could be costly for stateful consumers. This is
> not an issue before since we cannot change number of partition

Re: [DISCUSS] KIP-253: Support in-order message delivery with partition expansion

[Dong Lin]

Hey Jan,

I am not sure if it is acceptable for producer to be stopped for a while,
particularly for online application which requires low latency. I am also
not sure how consumers can switch to a new topic. Does user application
needs to explicitly specify a different topic for producer/consumer to
subscribe to? It will be helpful for discussion if you can provide more
detail on the interface change for this solution.

Thanks,
Dong

On Mon, Feb 26, 2018 at 12:48 AM, Jan Filipiak 
wrote:

> Hi,
>
> just want to throw my though in. In general the functionality is very
> usefull, we should though not try to find the architecture to hard while
> implementing.
>
> The manual steps would be to
>
> create a new topic
> the mirrormake from the new old topic to the new topic
> wait for mirror making to catch up.
> then put the consumers onto the new topic
> (having mirrormaker spit out a mapping from old offsets to new offsets:
> if topic is increased by factor X there is gonna be a clean
> mapping from 1 offset in the old topic to X offsets in the new topic,
> if there is no factor then there is no chance to generate a
> mapping that can be reasonable used for continuing)
> make consumers stop at appropriate points and continue consumption
> with offsets from the mapping.
> have the producers stop for a minimal time.
> wait for mirrormaker to finish
> let producer produce with the new metadata.
>
>
> Instead of implementing the approach suggest in the KIP which will leave
> log compacted topic completely crumbled and unusable.
> I would much rather try to build infrastructure to support the mentioned
> above operations more smoothly.
> Especially having producers stop and use another topic is difficult and
> it would be nice if one can trigger "invalid metadata" exceptions for them
> and
> if one could give topics aliases so that their produces with the old topic
> will arrive in the new topic.
>
> The downsides are obvious I guess ( having the same data twice for the
> transition period, but kafka tends to scale well with datasize). So its a
> nicer fit into the architecture.
>
> I further want to argument that the functionality by the KIP can
> completely be implementing in "userland" with a custom partitioner that
> handles the transition as needed. I would appreciate if someone could point
> out what a custom partitioner couldn't handle in this case?
>
> With the above approach, shrinking a topic becomes the same steps. Without
> loosing keys in the discontinued partitions.
>
> Would love to hear what everyone thinks.
>
> Best Jan
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> On 11.02.2018 00:35, Dong Lin wrote:
>
>> Hi all,
>>
>> I have created KIP-253: Support in-order message delivery with partition
>> expansion. See
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-253%
>> 3A+Support+in-order+message+delivery+with+partition+expansion
>> .
>>
>> This KIP provides a way to allow messages of the same key from the same
>> producer to be consumed in the same order they are produced even if we
>> expand partition of the topic.
>>
>> Thanks,
>> Dong
>>
>>
>

Re: Question about developer documentation

[Ewen Cheslack-Postava]

The web page is more about general project info and might be of interest to
people beyond just developers. But I agree the wiki landing page could use
some updating. Even more than just the developer section as we're missing
several releases, the oldest ones are listed at the top, etc.

-Ewen

On Fri, Feb 9, 2018 at 10:53 AM, Ray Chiang  wrote:

> There is some documentation for developers at:
>
>   http://kafka.apache.org/project
>
> There's also another set of links at the bottom of this wiki page:
>
>   https://cwiki.apache.org/confluence/display/KAFKA/Index
>
> There's some minor duplication of information, but it's definitely not
> quite presented in a clean "step by step" manner.
>
> I think it could benefit from a reorganization of how the information is
> presented.  Before I start making suggestions, does anyone have any
> thoughts on the subject?
>
> -Ray
>
>

Re: Kafka 0.11.0.1 and filebeat 6.1 compatibility

[Ewen Cheslack-Postava]

filebeat is implemented using sarama last I checked, so presumably they are
on a version that doesn't know about Kafka 0.11.0.1 and therefore it
doesn't know which API versions to use.

Not sure if they support leaving it blank or exactly how the sarama config
works, but as far as I know sarama has support for the API compatibility
request that was added way back in 0.10.0.0, meaning it really shouldn't
need version info specified for anything newer than that unless they
specifically chose not to support some API versions.

In either case, this is probably better answered by folks from Elastic or
the sarama project.

-Ewen

On Fri, Feb 9, 2018 at 7:11 AM, Sandeep Sarkar 
wrote:

> Hi All,
>
>
>
> I am using filebeat 6.1 and kafka 0.11.0.1 to push logs. From filebeat
> logs I could see that communication is getting established but then logs
> are not getting pushed.
>
> Also when I updated filebeat's output.kafka.version to 0.11.0.1 I get
> error message :
>
>
>
> 2018/02/08 09:41:15.417691 beat.go:635: CRIT Exiting: error initializing
> publisher:  unknown/unsupported kafka version '0.11.0.1' accessing
> 'output.kafka' (source:'/etc/filebeat/filebeat.yml')
>
> Exiting: error initializing publisher: unknown/unsupported kafka version
> '0.11.0.1' accessing 'output.kafka' (source:'/etc/filebeat/filebeat.yml')
>
>
>
> So I used kafka 0.10.2.0 and updated filebeat's output.kafka.version to
> 0.10.2.0 and the error message is gone but I do not get any logs in kafka.
> Filebeate throws
>
> Kafka publish failed with: circuit breaker is open
>
>
>
> And after some time it throws below message in a loop:
>
> 2018/02/09 15:08:13.075932 client.go:234: DBG [kafka] Kafka publish failed
> with: circuit breaker is open
>
> 2018/02/09 15:08:13.110793 client.go:220: DBG [kafka] finished kafka batch
>
> 2018/02/09 15:08:13.110814 client.go:234: DBG [kafka] Kafka publish failed
> with: circuit breaker is open
>
> 2018/02/09 15:08:13.149482 metrics.go:39: INFO Non-zero metrics in the
> last 30s: beat.info.uptime.ms=3 beat.memstats.gc_next=18583840
> beat.memstats.memory_alloc=17667464 beat.memstats.memory_total=3883962925856
> filebeat.harvester.open_files=11 filebeat.harvester.running=10
> libbeat.config.module.running=0 libbeat.output.events.batches=832
> libbeat.output.events.failed=1703936 libbeat.output.events.total=1703936
> libbeat.pipeline.clients=2 libbeat.pipeline.events.active=4118
> libbeat.pipeline.events.retry=1703936 registrar.states.current=21
>
>
>
>
>
> Please tell me how to debug this.
>
>
>
> I am using logstash 6.1 as consumer.
>
>
>
> --
> Thanks
>
> http://www.oracle.com/
> Sandeep Sarkar | Member of Technical Staff
> Phone: HYPERLINK "tel:+918067445685"+918067445685
> Oracle CGBU PM&C
>
> Oracle India Bangalore
>
> http://www.oracle.com/commitment
>
> Oracle is committed to developing practices and products that help protect
> the environment
>
>
>
>
>

Jenkins build is back to normal : kafka-trunk-jdk8 #2443

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-6601) Kafka manager does not provide consumer offset producer rate with kafka v2.10-0.10.2.0

[Rajendra Jangir (JIRA)]

Rajendra Jangir created KAFKA-6601:
--

 Summary: Kafka manager does not provide consumer offset producer 
rate with kafka v2.10-0.10.2.0
 Key: KAFKA-6601
 URL: https://issues.apache.org/jira/browse/KAFKA-6601
 Project: Kafka
  Issue Type: Bug
Reporter: Rajendra Jangir


I am using kafka-manager and kafka version 2.10-0.10.2.

And I am not able to see producer rate for _consumer_offset topic._

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] 1.0.1 RC2

[Jason Gustafson]

+1. Verified artifacts and ran the basic quickstart.

-Jason

On Mon, Feb 26, 2018 at 1:08 AM, Manikumar 
wrote:

> +1 (non-binding)
> Built src and ran tests
> Ran core quick start
>
> On Sat, Feb 24, 2018 at 8:44 PM, Jakub Scholz  wrote:
>
> > +1 (non-binding) ... I used the Scala 2.12 binaries and run my tests with
> > producers / consumers.
> >
> > On Thu, Feb 22, 2018 at 1:06 AM, Ewen Cheslack-Postava <
> e...@confluent.io>
> > wrote:
> >
> > > Hello Kafka users, developers and client-developers,
> > >
> > > This is the third candidate for release of Apache Kafka 1.0.1.
> > >
> > > This is a bugfix release for the 1.0 branch that was first released
> with
> > > 1.0.0 about 3 months ago. We've fixed 49 issues since that release.
> Most
> > of
> > > these are non-critical, but in aggregate these fixes will have
> > significant
> > > impact. A few of the more significant fixes include:
> > >
> > > * KAFKA-6277: Make loadClass thread-safe for class loaders of Connect
> > > plugins
> > > * KAFKA-6185: Selector memory leak with high likelihood of OOM in case
> of
> > > down conversion
> > > * KAFKA-6269: KTable state restore fails after rebalance
> > > * KAFKA-6190: GlobalKTable never finishes restoring when consuming
> > > transactional messages
> > > * KAFKA-6529: Stop file descriptor leak when client disconnects with
> > staged
> > > receives
> > > * KAFKA-6238: Issues with protocol version when applying a rolling
> > upgrade
> > > to 1.0.0
> > >
> > > Release notes for the 1.0.1 release:
> > > http://home.apache.org/~ewencp/kafka-1.0.1-rc2/RELEASE_NOTES.html
> > >
> > > *** Please download, test and vote by Saturday Feb 24, 9pm PT ***
> > >
> > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > http://kafka.apache.org/KEYS
> > >
> > > * Release artifacts to be voted upon (source and binary):
> > > http://home.apache.org/~ewencp/kafka-1.0.1-rc2/
> > >
> > > * Maven artifacts to be voted upon:
> > > https://repository.apache.org/content/groups/staging/
> > >
> > > * Javadoc:
> > > http://home.apache.org/~ewencp/kafka-1.0.1-rc2/javadoc/
> > >
> > > * Tag to be voted upon (off 1.0 branch) is the 1.0.1 tag:
> > > https://github.com/apache/kafka/tree/1.0.1-rc2
> > >
> > > * Documentation:
> > > http://kafka.apache.org/10/documentation.html
> > >
> > > * Protocol:
> > > http://kafka.apache.org/10/protocol.html
> > >
> > > /**
> > >
> > > Thanks,
> > > Ewen Cheslack-Postava
> > >
> >
>

Re: [DISCUSS] KIP-253: Support in-order message delivery with partition expansion

[Stephane Maarek]

Sounds awesome !
Are you planning to have auto scaling of partitions in a following KIP ?
That would be the holy grail

On 28 Feb. 2018 5:13 pm, "Dong Lin"  wrote:

> Hey Jan,
>
> I am not sure if it is acceptable for producer to be stopped for a while,
> particularly for online application which requires low latency. I am also
> not sure how consumers can switch to a new topic. Does user application
> needs to explicitly specify a different topic for producer/consumer to
> subscribe to? It will be helpful for discussion if you can provide more
> detail on the interface change for this solution.
>
> Thanks,
> Dong
>
> On Mon, Feb 26, 2018 at 12:48 AM, Jan Filipiak 
> wrote:
>
> > Hi,
> >
> > just want to throw my though in. In general the functionality is very
> > usefull, we should though not try to find the architecture to hard while
> > implementing.
> >
> > The manual steps would be to
> >
> > create a new topic
> > the mirrormake from the new old topic to the new topic
> > wait for mirror making to catch up.
> > then put the consumers onto the new topic
> > (having mirrormaker spit out a mapping from old offsets to new
> offsets:
> > if topic is increased by factor X there is gonna be a clean
> > mapping from 1 offset in the old topic to X offsets in the new topic,
> > if there is no factor then there is no chance to generate a
> > mapping that can be reasonable used for continuing)
> > make consumers stop at appropriate points and continue consumption
> > with offsets from the mapping.
> > have the producers stop for a minimal time.
> > wait for mirrormaker to finish
> > let producer produce with the new metadata.
> >
> >
> > Instead of implementing the approach suggest in the KIP which will leave
> > log compacted topic completely crumbled and unusable.
> > I would much rather try to build infrastructure to support the mentioned
> > above operations more smoothly.
> > Especially having producers stop and use another topic is difficult and
> > it would be nice if one can trigger "invalid metadata" exceptions for
> them
> > and
> > if one could give topics aliases so that their produces with the old
> topic
> > will arrive in the new topic.
> >
> > The downsides are obvious I guess ( having the same data twice for the
> > transition period, but kafka tends to scale well with datasize). So its a
> > nicer fit into the architecture.
> >
> > I further want to argument that the functionality by the KIP can
> > completely be implementing in "userland" with a custom partitioner that
> > handles the transition as needed. I would appreciate if someone could
> point
> > out what a custom partitioner couldn't handle in this case?
> >
> > With the above approach, shrinking a topic becomes the same steps.
> Without
> > loosing keys in the discontinued partitions.
> >
> > Would love to hear what everyone thinks.
> >
> > Best Jan
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On 11.02.2018 00:35, Dong Lin wrote:
> >
> >> Hi all,
> >>
> >> I have created KIP-253: Support in-order message delivery with partition
> >> expansion. See
> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-253%
> >> 3A+Support+in-order+message+delivery+with+partition+expansion
> >> .
> >>
> >> This KIP provides a way to allow messages of the same key from the same
> >> producer to be consumed in the same order they are produced even if we
> >> expand partition of the topic.
> >>
> >> Thanks,
> >> Dong
> >>
> >>
> >
>

Re: [VOTE] KIP-171 - Extend Consumer Group Reset Offset for Stream Application

[Stephane Maarek]

Bootstrap servers vs broker list is the biggest hurdle when teaching to
beginners. A standardized set of parameters would incredibly help

On 27 Feb. 2018 9:44 am, "Matthias J. Sax"  wrote:

I agree on consistency, too.

However, I am not sure if we should introduce an explicit --execute
option. Anybody familiar with Linux tools will expect a command to
execute by default.

Thus, I would suggest to remove --execute for all tools that use this
option atm.

Btw: there is a related Jira:
https://issues.apache.org/jira/browse/KAFKA-1299

Furthermore, this also affect arguments like

--bootstrap-servers
vs
--broker-list

and maybe others.

IMHO, all tools should use the same names. Thus, it's a larger change...
But totally worth doing it.


-Matthias

On 2/26/18 10:09 AM, Guozhang Wang wrote:
> Hi Jorge,
>
> I agree on being consistent across our tools.
>
> Besides the kafka-consumer-groups and kafka-streams-application-reset, a
> couple of other classes to consider adding the --execute options for the
> next major release:
>
> 1. kafka-preferred-replica-elections
> 2. kafka-reassign-partitions
> 3. kafka-delete-records
> 4. kafka-topics
> 5. kafka-acls
> 6. kafka-configs
> 7. kafka-delegation-tokens
>
>
> Guozhang
>
> On Mon, Feb 26, 2018 at 3:03 AM, Jorge Esteban Quilcate Otoya <
> quilcate.jo...@gmail.com> wrote:
>
>> Hi all,
>>
>> Thanks for the feedback.
>>
>> I have updated the "Compatibility, Deprecation, and Migration Plan"
section
>> to document this to support the rollback. I probably should have handled
>> this change, as small as it looks, as a new KIP to avoid this issue.
>>
>> I like Colin's idea about asking for confirmation, although I'm not sure
if
>> another tool has already this behavior and could create more confusion
>> (e.g. why this command ask for confirmation and others don't). Maybe we
>> will require a more broad looks at the CLI tools to agree on this?
>>
>> Jorge.
>>
>> El jue., 22 feb. 2018 a las 21:09, Guozhang Wang ()
>> escribió:
>>
>>> Yup, agreed.
>>>
>>> On Thu, Feb 22, 2018 at 11:46 AM, Ismael Juma  wrote:
>>>
 Hi Guozhang,

 To clarify my comment: any change with a backwards compatibility impact
 should be mentioned in the "Compatibility, Deprecation, and Migration
>>> Plan"
 section (in addition to the deprecation period and only happening in a
 major release as you said).

 Ismael

 On Thu, Feb 22, 2018 at 11:10 AM, Guozhang Wang 
 wrote:

> Just to clarify, the KIP itself has mentioned about the change so the
>>> PR
> was not un-intentional:
>
> "
>
> 3. Keep execution parameters uniform between both tools: It will
>>> execute
 by
> default, and have a `dry-run` parameter just show the results. This
>>> will
> involve change current `ConsumerGroupCommand` to change execution
 options.
>
> "
>
> We were agreed that the proposed change is better than the current
 status,
> since may people not using "--execute" on consumer reset tool were
 actually
> surprised that nothing gets executed. What we were concerning as a
> hind-sight is that instead of doing such change in a minor release
>> like
> 1.1, we should consider only doing that in the next major release as
>> it
> breaks compatibility. In the past when we are going to remove /
>> replace
> certain option we would first add a going-to-be-deprecated warning in
>>> the
> previous releases until it was finally removed. So Jason's suggestion
>>> is
 to
> do the same: we are not reverting this change forever, but trying to
 delay
> it after 1.1.
>
>
> Guozhang
>
>
> On Thu, Feb 22, 2018 at 10:56 AM, Colin McCabe 
 wrote:
>
>> Perhaps, if the user doesn't pass the --execute flag, the tool
>> should
>> print a prompt like "would you like to perform this reset?" and
>> wait
 for
> a
>> Y / N (or yes or no) input from the command-line.  Then, if the
 --execute
>> flag is passed, we skip this.  That seems 99% compatible, and also
>> accomplishes the goal of making the tool less confusing.
>>
>> best,
>> Colin
>>
>>
>> On Thu, Feb 22, 2018, at 10:23, Ismael Juma wrote:
>>> Yes, let's revert the incompatible changes. There was no mention
>> of
>>> compatibility impact on the KIP and we should ensure that is the
>>> case
> for
>>> 1.1.0.
>>>
>>> Ismael
>>>
>>> On Thu, Feb 22, 2018 at 9:55 AM, Jason Gustafson <
>>> ja...@confluent.io
>
>> wrote:
>>>
 I know it's a been a while since this vote passed, but I think
>> we
> need
>> to
 reconsider the incompatible changes to the consumer reset tool.
 Specifically, we have removed the --execute option without
> deprecating
>> it
 first, and we have changed the default behavior to execute
>> rather
> than
>> do a
 dry run. The latter in particu

Re: [DISCUSS] KIP-263: Allow broker to skip sanity check of inactive segments on broker startup

[Stephane Maarek]

This is great and definitely needed. I'm not exactly sure of what goes in
the process of checking log files at startup, but is there something like
signature checks of files (especially closed, immutable ones) that can be
saved on disk and checked against at startup ? Wouldn't that help speed up
boot time, for all segments ?

On 26 Feb. 2018 5:28 pm, "Dong Lin"  wrote:

> Hi all,
>
> I have created KIP-263: Allow broker to skip sanity check of inactive
> segments on broker startup. See
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 263%3A+Allow+broker+to+skip+sanity+check+of+inactive+
> segments+on+broker+startup
> .
>
> This KIP provides a way to significantly reduce time to rolling bounce a
> Kafka cluster.
>
> Comments are welcome!
>
> Thanks,
> Dong
>