[ https://issues.apache.org/jira/browse/KAFKA-4547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15832805#comment-15832805 ]
ASF GitHub Bot commented on KAFKA-4547: --------------------------------------- GitHub user vahidhashemian opened a pull request: https://github.com/apache/kafka/pull/2415 KAFKA-4547 (0.10.1 hotfix): Avoid unnecessary offset commit that could lead to an invalid offset position if partition is paused You can merge this pull request into a Git repository by running: $ git pull https://github.com/vahidhashemian/kafka KAFKA-4547-0.10.1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/2415.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2415 ---- commit 6acbf27aebbe5b6b4f230ce44ea5151c4c53ffbd Author: Jason Gustafson <ja...@confluent.io> Date: 2016-09-20T02:59:03Z MINOR: Bump to version 0.10.2 commit f396fdac197409fb955f00a6f642f04e4926ba41 Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-09-20T06:17:23Z KAFKA-4193; Fix for intermittent failure in FetcherTest Author: Ben Stopford <benstopf...@gmail.com> Reviewers: Jason Gustafson <ja...@confluent.io> Closes #1881 from benstopford/KAFKA-4193 commit c195003cb6e05f2d8c49285ff7e77b1cb3aa4361 Author: Eno Thereska <eno.there...@gmail.com> Date: 2016-09-20T10:33:50Z HOTFIX: Added check for metadata unavailable Author: Eno Thereska <eno.there...@gmail.com> Reviewers: Damian Guy <damian....@gmail.com>, Ismael Juma <ism...@juma.me.uk> Closes #1887 from enothereska/hotfix-metadata-unavailable commit 3663275cf066b7715cc11b26fd9c144bbff1c373 Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-09-20T13:53:48Z KAFKA-4184; Intermittent failures in ReplicationQuotasTest.shouldBootstrapTwoBrokersWithFollowerThrottle Build is unstable, so it's hard to validate this change. Of the various builds up until 11am BST the test ran twice and passed twice. Author: Ben Stopford <benstopf...@gmail.com> Reviewers: Ismael Juma <ism...@juma.me.uk> Closes #1873 from benstopford/KAFKA-4184 commit 4f821830bc6b726cddf90999fff76006745b1a3f Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-09-20T14:41:14Z KAFKA-4197; Make ReassignPartitionsTest System Test move data The ReassignPartitionsTest system tests doesn't reassign any replicas (i.e. move data). This is a simple issue. It uses a 3 node cluster with replication factor of 3, so whilst the replicas are jumbled around, nothing actually is moved from machine to machine when the assignment is executed. This fix just ups the number of nodes to 4 so things move. Tests pass locally. There are runs pending on the two branch builders Passes: https://jenkins.confluent.io/job/system-test-kafka-branch-builder/551/ https://jenkins.confluent.io/job/system-test-kafka-branch-builder-2/94/ https://jenkins.confluent.io/job/system-test-kafka-branch-builder/553/ https://jenkins.confluent.io/job/system-test-kafka-branch-builder/554/ https://jenkins.confluent.io/job/system-test-kafka-branch-builder-2/95 Failures: https://jenkins.confluent.io/job/system-test-kafka-branch-builder/552 => _RuntimeError: There aren't enough available nodes to satisfy the resource request. Total cluster size: 1, Requested: 4, Already allocated: 1, Available: 0._ Which I assume to do with the test env. Author: Ben Stopford <benstopf...@gmail.com> Reviewers: Ismael Juma <ism...@juma.me.uk> Closes #1892 from benstopford/fix_reassignment_test commit 24f81ea764a493b4422b6a3ef6b3e771d0e4d63b Author: Damian Guy <damian....@gmail.com> Date: 2016-09-21T18:11:12Z MINOR: add javadoc comment to PersistenKeyValueFactory.enableCaching missing javadoc on public API method PersistenKeyValueFactory.enableCaching Author: Damian Guy <damian....@gmail.com> Reviewers: Eno Thereska, Guozhang Wang Closes #1891 from dguy/minor-java-doc commit a632716a3c9a871f325c6f13aefa9aed0add4b82 Author: Damian Guy <damian....@gmail.com> Date: 2016-09-21T18:13:39Z MINOR: remove unused code from InternalTopicManager Remove isValidCleanupPolicy and related fields as they are never used. Author: Damian Guy <damian....@gmail.com> Reviewers: Eno Thereska, Guozhang Wang Closes #1888 from dguy/minor-remove-unused commit 732fabf94ebc9631d31f2feb2116ee8b63beabef Author: Jason Gustafson <ja...@confluent.io> Date: 2016-09-22T17:07:50Z KAFKA-3782: Ensure heartbeat thread restarted after rebalance woken up Author: Jason Gustafson <ja...@confluent.io> Reviewers: Guozhang Wang Closes #1898 from hachikuji/KAFKA-3782 commit 27e3edc791760dea7ff4d048f87d1585f9e235d7 Author: Elias Levy <fearsome.lucid...@gmail.com> Date: 2016-09-22T17:33:23Z MINOR: Fix comments in KStreamKStreamJoinTest Minor comment fixes. Author: Elias Levy <fearsome.lucid...@gmail.com> Reviewers: Guozhang Wang Closes #1885 from eliaslevy/fix-test-comments commit 4112bc3af7bec63ae1248c1b9db745ba8ac4c63c Author: Eno Thereska <eno.there...@gmail.com> Date: 2016-09-22T19:45:43Z HOTFIX: Decrease commit interval The original commit interval of 30 seconds might be too large in some cases, e.g., when the verifier finishes before those 30 seconds have elapsed. Author: Eno Thereska <eno.there...@gmail.com> Reviewers: Damian Guy, Guozhang Wang Closes #1899 from enothereska/hotfix-smoke-test-commit-interval commit d32f3f2828c199a5a09bbba779d07430e43fb190 Author: Ismael Juma <ism...@juma.me.uk> Date: 2016-09-22T21:53:40Z MINOR: Increase `zkConnectionTimeout` and timeout in `testReachableServer` We had a number of failures recently due to these timeouts being too low. It's a particular problem if multiple forks are used while running the tests. Author: Ismael Juma <ism...@juma.me.uk> Reviewers: Jason Gustafson <ja...@confluent.io> Closes #1889 from ijuma/increase-zk-timeout-in-tests commit 106a7456060750ab0604d290b8c1e33a57adbf20 Author: Jason Gustafson <ja...@confluent.io> Date: 2016-09-22T22:03:49Z MINOR: Add test cases for delays in consumer rebalance listener Author: Jason Gustafson <ja...@confluent.io> Reviewers: Ismael Juma <ism...@juma.me.uk> Closes #1866 from hachikuji/rebalance-delay-test-cases commit 6a13a3dbaddf99850b2583007577fa2a6e1e6d3a Author: Jason Gustafson <ja...@confluent.io> Date: 2016-09-23T20:13:29Z KAFKA-3590; Handle not-enough-replicas errors when writing to offsets topic Author: Jason Gustafson <ja...@confluent.io> Reviewers: Ismael Juma <ism...@juma.me.uk>, Guozhang Wang <wangg...@gmail.com> Closes #1859 from hachikuji/KAFKA-3590 commit 0de807357be4d776747e960b5fd17ddeae9a6cad Author: Ryan Pridgeon <ryan.n.pridg...@gmail.com> Date: 2016-09-23T21:20:41Z KAFKA-3719; Allow underscores in hostname Technically this does not strictly adhere to RFC-952 however it is valid for domain names, urls and uris so we should loosen the requirements a tad. Author: Ryan Pridgeon <ryan.n.pridg...@gmail.com> Reviewers: Ismael Juma <ism...@juma.me.uk> Closes #1856 from rnpridgeon/KAFKA-3719 commit 36242b846a42b33d7d4c1931f2dae93ebe1547c7 Author: Sumit Arrawatia <sumit.arrawa...@gmail.com> Date: 2016-09-24T09:16:49Z KAFKA-4151; Update public docs for Cluster Id (KIP-78) - Updated implementation docs with details on Cluster Id generation. - Mention cluster id in "noteworthy changes for 0.10.1.0" in upgrade docs. Author: Sumit Arrawatia <sumit.arrawa...@gmail.com> Author: arrawatia <sumit.arrawa...@gmail.com> Reviewers: Ismael Juma <ism...@juma.me.uk> Closes #1895 from arrawatia/kip-78-docs commit 1d055f7551d138324d2540095a1cfc1c8f74d76f Author: Arun Mahadevan <ai...@hortonworks.com> Date: 2016-09-25T07:44:56Z KAFKA-3282; Change tools to use new consumer if zookeeper is not specified Author: Arun Mahadevan <ai...@hortonworks.com> Reviewers: Jason Gustafson <ja...@confluent.io>, Ismael Juma <ism...@juma.me.uk> Closes #1376 from arunmahadevan/cons-consumer-fix commit c0a62b70a8eadc550c937bb18e0203ab691618f5 Author: Rajini Sivaram <rajinisiva...@googlemail.com> Date: 2016-09-26T00:01:45Z KAFKA-4055; System tests for secure quotas Fix existing client-id quota test which currently don't configure quota overrides correctly. Add new tests for user and (user, client-id) quota overrides and default quotas. Author: Rajini Sivaram <rajinisiva...@googlemail.com> Reviewers: Jun Rao <jun...@gmail.com> Closes #1860 from rajinisivaram/KAFKA-4055 commit b75245cfbbefc712103b9329da0f27a205baa6aa Author: Jason Gustafson <ja...@confluent.io> Date: 2016-09-26T21:54:01Z MINOR: Wakeups propagated from commitOffsets in WorkerSinkTask should be caught Author: Jason Gustafson <ja...@confluent.io> Reviewers: Ewen Cheslack-Postava <e...@confluent.io> Closes #1907 from hachikuji/catch-wakeup-worker-sink-task commit be6056abc9970600347c95c4c8658799b76dbe6b Author: Apurva Mehta <apurva.1...@gmail.com> Date: 2016-09-27T00:18:18Z KAFKA-4214; kafka-reassign-partitions fails all the time when brokers are bounced during reassignment There is a corner case bug, where during partition reassignment, if the controller and a broker receiving a new replica are bounced at the same time, the partition reassignment is failed. The cause of this bug is a block of code in the KafkaController which fails the reassignment if the aliveNewReplicas != newReplicas, ie. if some of the new replicas are offline at the time a controller fails over. The fix is to have the controller listen for ISR change events even for new replicas which are not alive when the controller boots up. Once the said replicas come online, they will be in the ISR set, and the new controller will detect this, and then mark the reassignment as successful. Interestingly, the block of code in question was introduced in KAFKA-990, where a concern about this exact scenario was raised :) This bug was revealed in the system tests in https://github.com/apache/kafka/pull/1904. The relevant tests will be enabled in either this or a followup PR when PR-1904 is merged. Thanks to junrao identifying the issue and providing the patch. Author: Apurva Mehta <apurva.1...@gmail.com> Reviewers: Jun Rao <jun...@gmail.com> Closes #1910 from apurvam/KAFKA-4214 commit 60ad6d727861a87fa756918a7be7547e9b1f4c3d Author: Vahid Hashemian <vahidhashem...@us.ibm.com> Date: 2016-09-27T02:04:22Z KAFKA-3831; Prepare for updating new-consumer-based Mirror Maker's default partition assignment strategy to round robin This patch adds proper warning message and necessary doc updates for updating the default partition assignment strategy of Mirror Maker from range to round robin. The actual switch would occur as part of a major release cycle (to be scheduled). Author: Vahid Hashemian <vahidhashem...@us.ibm.com> Reviewers: Jason Gustafson <ja...@confluent.io> Closes #1499 from vahidhashemian/KAFKA-3831 commit a208478f37f216324a87b530f18f30ecd79601c8 Author: Magnus Reftel <magnus.ref...@skatteetaten.no> Date: 2016-09-27T04:15:25Z MINOR: Fix Javadoc of @throws AuthorizationException for KafkaConsumer.poll Author: Magnus Reftel <magnus.ref...@skatteetaten.no> Reviewers: Jason Gustafson <ja...@confluent.io> Closes #1901 from reftel/feature/poll_javadoc commit 5d6408f6cfda3f8ab366195f69e90de048cde25d Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-09-27T13:00:44Z KAFKA-4200; Fix throttle argument in kafka-reassign-partitions.sh Simple jira which alters two things: 1. kafka-reassign-partitions --verify prints Throttle was removed regardless of whether a throttle was applied. It should only print this if the value was actually changed. 2. --verify should exception if the —throttle argument. (check generate too) To test this I extracted all validation logic into a separate method and added a test which covers the majority of combinations. The validation logic was retained as is, other than implementing (2) and adding validation to the --broker-list option which you can currently apply to any of hte main actions (where it is ignored). Requirement 1 was tested manually (as it's just println). Testing: - Build passes locally. - System test reassign_partitions_test.py also passes. Author: Ben Stopford <benstopf...@gmail.com> Reviewers: Jun Rao <jun...@gmail.com> Closes #1896 from benstopford/KAFKA-4200 commit b8ed4a51134ca8b98d1445871a55ed33b6ad5b92 Author: Ben Stopford <benstopf...@gmail.com> Date: 2016-09-27T19:27:20Z KAFKA-4177; Remove ThrottledReplicationRateLimit from Server Config This small PR pulls ThrottledReplicationRateLimit out of KafkaConfig and puts it in a class that defines Dynamic Configs. Client configs are also placed in this class and validation added. Author: Ben Stopford <benstopf...@gmail.com> Reviewers: Jun Rao <jun...@gmail.com>, Ismael Juma <ism...@juma.me.uk> Closes #1864 from benstopford/KAFKA-4177 commit 3db752a565071c78e4b11eaafa739844fa785b04 Author: Jason Gustafson <ja...@confluent.io> Date: 2016-09-27T20:35:29Z MINOR: Make new consumer default for Mirror Maker Author: Jason Gustafson <ja...@confluent.io> Reviewers: Ismael Juma <ism...@juma.me.uk> Closes #1914 from hachikuji/mm-default-new-consumer commit 61d3378bc84914a521a65cdfffb7299928fa8671 Author: Ismael Juma <ism...@juma.me.uk> Date: 2016-09-27T23:49:46Z MINOR: Remove no longer required --new-consumer switch in docs Author: Ismael Juma <ism...@juma.me.uk> Reviewers: Jason Gustafson <ja...@confluent.io> Closes #1905 from ijuma/no-new-consumer-switch-in-examples commit 0c25c73782e6e70b8f37e3dda2fa2a5b0b1c8c65 Author: Damian Guy <damian....@gmail.com> Date: 2016-09-28T00:35:24Z HOTFIX: fix npe in StreamsMetadataState when onChange has not been called If some StreamsMetadataState methods are called before the onChange method is called a NullPointerException was being thrown. Added null check for cluster in isInitialized method Author: Damian Guy <damian....@gmail.com> Reviewers: Guozhang Wang <wangg...@gmail.com> Closes #1920 from dguy/fix-npe-streamsmetadata commit 0c4cc5a4466d0dfce01903c29d97af3e428597f3 Author: Guozhang Wang <wangg...@gmail.com> Date: 2016-09-28T00:37:12Z MINOR: fixes a few error logging formats Author: Guozhang Wang <wangg...@gmail.com> Reviewers: Eno Thereska <eno.there...@gmail.com> Closes #1919 from guozhangwang/minor-error-message-fixes commit c526c0c3f6f153ef8f84a3e66d9f997d8c31e20c Author: Damian Guy <damian....@gmail.com> Date: 2016-09-28T00:43:36Z MINOR: add test to make sure ProcessorStateManager can handle State Stores with logging disabled Adding the test so we know that the State Stores with logging disabled or without a topic don't throw any exceptions. Author: Damian Guy <damian....@gmail.com> Reviewers: Guozhang Wang <wangg...@gmail.com> Closes #1916 from dguy/state-store-logging-disabled commit d83cde7cabe4e86951c6760e68e65b99752cfe0e Author: Damian Guy <damian....@gmail.com> Date: 2016-09-28T18:07:44Z KAFKA-3708: Better exception handling in Kafka Streams KafkaExceptions currently thrown from within StreamThread/StreamTask currently bubble up without any additional context. This makes it hard to figure out where something went wrong, i.e, which topic had the serialization exception etc Author: Damian Guy <damian....@gmail.com> Reviewers: Guozhang Wang <wangg...@gmail.com> Closes #1819 from dguy/kafka-3708 and squashes the following commits: d6feaa8 [Damian Guy] address comments 15b89e7 [Damian Guy] merge trunk 6b8a8af [Damian Guy] catch exceptions in various places and throw more informative versions c86eeda [Damian Guy] fix conflicts 8f37e2c [Damian Guy] add some context to exceptions commit 281fac9ed626b16190d56e609c9bf04a3188374a Author: Rajini Sivaram <rajinisiva...@googlemail.com> Date: 2016-09-28T19:18:20Z KAFKA-4227; Shutdown AdminManager when KafkaServer is shutdown Terminate topic purgatory thread in AdminManager during server shutdown to avoid threads being left around in unit tests. Author: Rajini Sivaram <rajinisiva...@googlemail.com> Reviewers: Ismael Juma <ism...@juma.me.uk> Closes #1927 from rajinisivaram/KAFKA-4227 ---- > Consumer.position returns incorrect results for Kafka 0.10.1.0 client > --------------------------------------------------------------------- > > Key: KAFKA-4547 > URL: https://issues.apache.org/jira/browse/KAFKA-4547 > Project: Kafka > Issue Type: Bug > Components: clients > Affects Versions: 0.10.1.0, 0.10.0.2, 0.10.1.1 > Environment: Windows Kafka 0.10.1.0 > Reporter: Pranav Nakhe > Assignee: Vahid Hashemian > Priority: Blocker > Labels: clients > Fix For: 0.10.2.0 > > Attachments: issuerep.zip > > > Consider the following code - > KafkaConsumer<String, String> consumer = new > KafkaConsumer<String, String>(props); > List<TopicPartition> listOfPartitions = new ArrayList(); > for (int i = 0; i < > consumer.partitionsFor("IssueTopic").size(); i++) { > listOfPartitions.add(new TopicPartition("IssueTopic", > i)); > } > consumer.assign(listOfPartitions); > consumer.pause(listOfPartitions); > consumer.seekToEnd(listOfPartitions); > // consumer.resume(listOfPartitions); -- commented out > for(int i = 0; i < listOfPartitions.size(); i++) { > > System.out.println(consumer.position(listOfPartitions.get(i))); > } > > I have created a topic IssueTopic with 3 partitions with a single replica on > my single node kafka installation (0.10.1.0) > The behavior noticed for Kafka client 0.10.1.0 as against Kafka client > 0.10.0.1 > A) Initially when there are no messages on IssueTopic running the above > program returns > 0.10.1.0 > 0 > 0 > 0 > 0.10.0.1 > 0 > 0 > 0 > B) Next I send 6 messages and see that the messages have been evenly > distributed across the three partitions. Running the above program now > returns > 0.10.1.0 > 0 > 0 > 2 > 0.10.0.1 > 2 > 2 > 2 > Clearly there is a difference in behavior for the 2 clients. > Now after seekToEnd call if I make a call to resume (uncomment the resume > call in code above) then the behavior is > 0.10.1.0 > 2 > 2 > 2 > 0.10.0.1 > 2 > 2 > 2 > This is an issue I came across when using the spark kafka integration for > 0.10. When I use kafka 0.10.1.0 I started seeing this issue. I had raised a > pull request to resolve that issue [SPARK-18779] but when looking at the > kafka client implementation/documentation now it seems the issue is with > kafka and not with spark. There does not seem to be any documentation which > specifies/implies that we need to call resume after seekToEnd for position to > return the correct value. Also there is a clear difference in the behavior in > the two kafka client implementations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)